Do Vision-Language Models Really Understand Visual Language?
Authors: Yifan Hou
Buse Giledereli
Yilei Tu
Mrinmaya Sachan
Venue: ICML 2025 poster (2025)
Keywords: vision-language model
visual language
evaluation
interpretation
diagram
Primary area: general_machine_learning->evaluation

Abstract: Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicting complex concepts and their relationships in the form of an image. The symbolic nature of diagrams presents significant challenges for building models capable of understanding them. Yet, recent studies seem to suggest that Large Vision-Language Models (LVLMs) can even tackle complex reasoning tasks involving diagrams. In this paper, we investigate this phenomenon by developing a comprehensive test suite to evaluate the diagram comprehension capability of LVLMs. Our test suite uses a variety of questions focused on concept entities and their relationships over a set of synthetic as well as real diagrams across several domains to evaluate the recognition and reasoning abilities of models. Our evaluation of six LVLMs shows that while these models can accurately identify and reason about entities, their ability to understand relationships is notably limited. Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowledge as shortcuts to identify and reason about the relational information. Thus, we conclude that LVLMs have a limited capability for genuine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.

Do Vision-Language Models Really Understand Visual Language?
Yifan Hou Buse Giledereli
Yilei Tu Mrinmaya Sachan 1
{ Tyffnn hou, bgiledereli, yiletu, mr i imnaa.maaa....m sachan eiinf, ethz. ch
Abstract
1. Introduction
Symbolic signals such as language serve powerful tools in communication by abstracting and interpreting informa- tion. Visual language is a form of communication that uses symbols, shapes, and spatial arrangements to convey complex ideas (Greenspan & Shanker, 2009; Li, 2023). Di- agrams, which encapsulate symbolic information the vi- sual stream, prime example of visual language (Zdebik, 2012; Anderson et al., 2011) that are extensively used in practice across various domains, e.g., mathematics (Seo et al., 2015), science (Lu et al., 2022), education (Kembhavi et al., 2016; 2017), and illustrations (Hiippala & Orekhova, 2018; Lu et al., 2021). Developing models capable of un- derstanding symbolic information, e.g. in diagrams, is a critical milestone in advancing machine intelligence (Bauer & Johnson-Laird, 1993; de Rijke, 1999; Cromley et al., 2010). Even though recent Large Vision-Lannuage Mod- els (LVLMs, OpenAl, 2023; Anil et al., 2023) have demon- strated some success on diagram-based visual reasoning tasks (Lu al., 2023; Zhang et al., 2024; Chen et al., 2024), it remains unclear whether the performance on these tasks truly reflects the models' ability to comprehensively under- stand the symbolic information in diagrams.
Visual language is system of communica- tion that conveys information through symbols, shapes, and spatial arrangements. Diagrams are typical example of a visual language depicting complex concepts and their relationships in the form of image. The symbolic nature of dia- grams presents significant challenges for building models capable of understanding them. Recent studies suggest that Large Vision-Languaae Mod- els (L.V.Ms) can even tackle complex reasoning tasks involving diagrams. In this paper, we inves- tigate this phenomenon by developing a compre- hensive test suite evaluate the diagram compre- hension capability of LVLMs. Our test suite uses a variety of questions focused on concept entities and their relationships over set of synthetic as well as real diagrams across domains to evaluate the recognition and reasoning abilities of models. Our evaluation of LVLMs shows that while they can accurately identify and reason about entities, their ability to understand relationships is notably limited. Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowl- edge as shortcuts to identify and reason about the relational information. Thus, we conclude that LVLMs have limited capability for gen- uine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.
For this purpose, we design a comprehensive test suite that investigates the ability of LVLMs to understand diagrams. As defined by Foucault (1977) and Deleuze (1986), dia- grams are abstract tools that organize visual entities using relational information. Drawing inspiration from this, our test suite focuses on evaluating diagram understanding by assessing how well models can understand entities and re- lations in typical diagrams ($ 2.1). We evaluate diagram understanding by defining two types of tasks pertaining to fast recognition of entities and relations and slow multi-step reasoning (Kahneman, 2011) over them ($ 2.2). While we cannot cover every diagram type for practical reasons, we still cover diagrams across six domains. To ensure that our evaluation is both controlled and generalizable, our test suite includes both clean synthetic diagrams and 1, 001 annotated real diagrams carefully selected from existing datasets Kr- ishnamurthy et al. (2016); Kembhavi et al. (2016) ($ 2.3). Using our test suite, we conduct a detailed analysis of the
'Department of Computer Science, ETH Zürich. Correspon- dence to: Yifan Hou <yifan.hou in ethz.chh... Mrinmaya Sachan Emrinmaya.sachan @ inf. ethz.ch>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Do Vision-Language Models Really Understand Visual Language?
(a) Synthetic Diagram
Q: Which entity connects Gorilla? Wagon, B: Clock, C: Trash Bin D: Owl
Kilker whale
Lequed sol
GPT-4o: From the diagram you provided, the entity that directly connects
gorilla the clock (option B). There a visible arrow connecting the clock to the gorilla.
Emporn
X
Accuracy (%)
81.6
Q: Which creature is the predator of Krill? A: Phytoplankton, Whale, Fish, D: Krill
Squid
Krll
61.81
GPT-4o: From the image, the predator Krill both the Fish and Squid, as indicated by arrows pointing from Krill towards them. So, the correct answer would be C: Fish.
Synthetic
A Real
(c) Results on Diagram QA
(b) Real Diagram
Photoplaskton
Figure 1: The responses of GPT-4o diagram-related questions reveal notable pattern. The model struggles correctly the relation question the simple synthetic diagram, yet successfully understands the relationship in complex real diagram. We demonstrate that this pattern occurs consistently (Tabs. and 4),
model's strengths and weaknesses in understanding dia- grams and explore the following questions:
questions in practice? (Fig. 1c) One potential hypothesis is that the models leverage their background knowledge as shortcut to answer these questions. To test it, we explore the impact of knowledge on question-anssering (QA) and draw the following observations:
Q1: Can LVLMs understand diagrams? To assess the basic capabilities of LVLMs, first test them using clean synthetic diagrams, followed by evaluations in real-world scenarios for comparison. Our findings from these evalua- tions lead to three key observations:
Knowledge enhances model in relation recognition. We construct synthetic diagrams that incorporate semantic knowledge and observe notable improvement in rela- tion recognition questions, suggesting that models per- form better on wwwwledge-groundee diagrams ($ 4.1). Additionally, in real diagrams, we categorize questions based on whether they require background knowledge (e.g., commonsense) or can be answered independently of it. Results show that models excel at answering questions that draw upon background knowledge ($ 4.2).
LVLMs can identify entities and reason about them. By generating synthetic diagrams to evaluate models from multiple perspectives, we observe that models consistently perform well on entity- melate- questions. They curately identify and reason about entities in synthetic diagrams, regardless of whether the entity is represented textually or visually ($ 3.1).
LVLMs struggle with identifying and reasoning about relations (Fig. fa). In synthetic scenarios, the models exhibit significant difficulty in identifying relationships between depicted concepts and in performing reasoning tasks based on those relationships 3.2). This challenge persists across various diagram settings and prompting templates (Apps. F.2.3 and F.2.4).
LVLMs only answer relation questions correctly for sim- ple real diagrams. We classify diagrams by complexity that is based the number of entities, and analyze QA performance for simple and complex diagrams. Results reveal that while accuracy on entity questions remains consistent, that on relation questions drops significantly with the increase of complexity. This indicates that the models' seemingly good performance on relation ques- tions is primarily driven by handling simpler diagrams, rather than by genuine relation understanding ($ 5.1).
For real diagrams, LVLMs still understand entities and cannot reason about relations. But they can identify relations (Fig. 1b). We annotate real diagrams with ques- tions from multiple aspects and evaluate models on them. Unexpectedly, we find that the models perform signif- icantly better on relation recognition questions in real diagrams compared to those in synthetic diagrams ($ 3.3).
LVLMs rely on learned knowledge to hallucinate rela- tions. In the case study, we demonstrate that even when no relations are provided, LVLMs infer relations based on their learned knowledge. Furthermore, if provided rela- tions contradict the models' learned knowledge, they tend to disregard them and instead rely on their background knowledge to answer the questions ($ 5.2).
Q2: LVLMS cannot identify relations in simple syn- thetic diagrams, how do they manage answer complex
2

---

Do Vision-Languaaee Models Really Understand Visual Language?
Synthetic Diagram
Question
Example
(s(V KF, NR)
Which one of the entities exists in the diagram?
Entity
Qs(V/KF,NC NC)
How many text labels are there in the diagram?
(g(EE) KF)NR) (5(E)) KF)
Which one of the text labels is placed on the left of the entity cow?
Implicit Relation
How many text labels are placed on the left the entity cow?
2g(E) KF)N) Q5(E)) NC)
Which one of the pairs is connected in the diagram?
Explicit Relation
How many entities are connected to cow?
Real Diagram
Question
Example
Qn(V KF)
Which entity is in the diagram?
(n KF, NF)
How many entities are there in the diagram?
Entity
Qn(V KR, NR)
Which producer in the diagram?
QR KR, NC)
How many consumers are in the diagram?
(R( KF, NR)
Which entity is connected to Fish?
(R(E KF, NC)
How many arrows are linked Fish in the diagram?
Relation
(n(E) KR, NR)
Which is not the predator of Krill?
(R) KR, NC)
How many types of prey are consumed by Fish in the foodweb?
Table I: The template and example of question annotations. The underlined entity varies across diagrams. There are no KR questions for synthetic diagrams since they do not have background knowledge. Questions for real diagrams correspond to Fig. 1b. In terms of relations, "Explicit Relation" refers to relations that are clearly depicted through arrows or segments, while "Implicit Relation" refers to those that are conveyed indirectly, such as through relative position relationships.
Findings. We summarize our research findings as follows: While LVLMs can recognize and reason about entities, they struggle with relations. The models do not engage in gen- uine diagram parsing or reasoning; rather, their seemingly strong performance on various benchmarks is an illusion cre- ated by their reliance on knowledge shortcuts. Specifically, these models identify the entities depicted in the diagrams and simply retrieve relevant pre-learned knowledge.
diagram in Fig. lb). Each entity V E y could be represented in multiple ways, e.g., text and visuals in the diagram. Each relation E = (V, r") E connects two entities. Relations are either explicitly represented by arrows or via implicit relationships (e.g., relative positioning of entities).
2.2. How Do We Evaluate The Model's Diagram Understanding Ability?
In designing our test suite, we draw inspiration from (Kah- neman, 2011) who argues that the thinking process can be naturally divided into two modes: System 1, which han- dles automatic, quick, and intuitive thinking (e.g., pattern recognition and everyday decisions), and System 2, which is responsible for deliberate, slow, and logical thinking (e.g.. logical reasoning and critical analysis). We evaluate both the recognition and reasoning abilities of models via question- answering (QA). We carefully design a set of questions posed in a multiple-choice QA format with each question having one correct answer and three incorrect options and simply use the model's accuracy in answering the questions as an evaluation of the model's ability on that skill. Overall, we denote the set of questions as Q. We denote the set of questions related to understanding entities as Q(V), while those related to relations as Q(F). Additionally, we use sub- scripts to distinguish between different types of questions: questions on synthetic diagrams are denoted as Qs. whereas questions for real diagrams are denoted as Qn- Our test suite questions are categorized follows:
2. Test Suite Design
In this section, we provide a definition of a diagram and provide a desiderata for our evaluation suite.
2.1. Diagrams As Graphs
Diagrams work as an abstract tool to describe concepts and relationships (Foucault, 1977; Deleuze, 1986). In practice, we choose this representation as it is quite flexible and a broad set of diagrams can be represented in a format as shown in Fig. 1. For example, logical diagrams such as water cycles illustrate the process transitions (relations) among cycle stages (entities). Schematic diagrams such as circuits demonstrate the connections (relations) among electronic components (entities). Previous work (Song et al., 1995; Hiippala & Orekhova, 2018) also chose to annotate and model diagrams as concepts and their connections." 1
Following that, we define a diagram as a graph Here, is the set of entities (e.g., "Squid" in the example
{v,E).
GGiven the variety of diagram types, we leave certain cases that are challenging to represent in this way for future work.
Recognition vs. Reasoning Questions. To measure the two key abilities of models in recognition of entities and
3

---

Do Vision-Language Models Really Understand Visual Language?
relations vs. reasoning, we design two types of questions: Name Recognition (NR) and Number Counting (NC). The NR questions measure the recognition ability of models by verifying the existence of specific entities or relations. In contrast, NC. questions measure reasoning ability by asking for the number of certain types of entities or relations. We formally denote these question sets as Q(INNR) and Q(-INC).
Synthetic diagram set. We generate synthetic diagrams in two steps. First, we randomly create between 2 to 9 entities represented by images or text. Then, we randomly establish between 1 to maximum relations among them using directed arrows. To ensure clarity in the synthetic diagrams, we care- fully avoid situations when arrows cross over certain entities. In practice, we construct our entity set V by sampling from a pre-defined set containing 377 distinct entities provided by Lu et al. (2021). By default, we generate 1, 0,000 diagrams for each experiment.
Knowledge-Required vs. Knowledge-Free. Next, we test if LVLMs use any knowledge shortcuts (Ye & Kovashka, 2021; Tang et al., 2023) to answer questions without true diagram understanding. Diagrams often encode some background real-world knowledge, and the models may use their background knowledge as a shorteut to answer the questions. To further tease out the models' true understand- ing of diagrams, we design questions that both do and do not require background knowledge, allowing us to test the models' capabilities in each scenario. If a question requires the model to use background knowledge (e.g.. semantic or commonsense), we call it a knowledge-require (KR) question, which is denoted as Q( (.])k)). On the other hand, questions that do not rely on such external knowledge are termed knowledge-free (KF) questions, denoted as Q( (KF).... This distinction helps us clearly separate the model's 's capac- ity for pure visual processing from its ability incorporate and utilize prior knowledge when answering questions.
Real diagram set. We carefully filtered and curated se- lection of 1, 001 real-world diagrams from Krishnamurthy et al. (2016); Kembhavi et al. (2016) to include in our test suite. These diagrams span a diverse range of domains, including ecology, biology, physies, astronomy, chemistry, and geology. This selection ensures that our test suite covers broad spectrum of scientific disciplines, providing a com- prehensive evaluation of the models' capabilities. During the filtering process, diagrams are first categorized by do- main. Subsequently, low-quality diagrams, along with those considered too simplistic or excessively complex, were re- moved to ensure reliable annotations. Example questions are given in Tab. Detailed statistical information and an- notation details of these diagrams can be found in in App. E.
Templates and examples of each question type are presented in Tab. 1. Each question type targets a specific diagram component or model ability within the evaluation. For real diagrams, the question templates are tailored to the specific context of each domain. Details on the question design in each domain are in App. E.I. From these questions, we can derive the following intuition:
3. Do LVLMs Understand Diagrams?
In this section, we investigate whether LVLMs can under- stand entities ($ 3.1) and relations ($ 3.2) in synthetic dia- grams. Additionally, we present the evaluation results on real diagrams (5 3.3).
Intuition 1. KR questions in real diagrams are generally more challenging than KF questions in synthetic diagrams. The reasons are: Real diagrams are inherently more com- plex, containing a wider range of information compared to synthetic diagrams; 2) Beyond assessing basic abilities, answering KR questions also requires the integration of ad- ditional background knowledge.
3.1. Can LVLMs Understand Entities?
We begin by evaluating whether LVL.Ms can identify and reason about entities represented by text boxes (i.e., text en- tities) or visual icons (i.e., visual entities). Additionally, we examine the models' ability to correctly identify the spatial information (i.c., locations) of these entities in App. F.2.2.
2.3. Which Diagrams Do We Consider?
Diagrams are extensively utilized in various domains, ap- pearing in different forms and encompassing a wide range of information types. While evaluating models in a synthetic setting helps to reduce the impact of confounding factors, the resulting conclusions may not fully extend to real-world cases. Conversely, evaluating models on real diagrams al- lows for broader coverage of diagram types, though it may introduce biases due to the additional information or knowl- edge embedded in these diagrams. To address this challenge, our test suite incorporates both synthetic and real diagrams, providing balanced and comprehensive evaluation.
Preparation. We evaluate three open-source models: LLaMA-3.2-11B-Viiee (i.e., ILLMM,, Dubey et al., 2024)), LLaVA-One Vi Viiiio-- (i.e., LLaVA, Li et al., 2024a)), and Qwen2-VL-7B (i.c., Qwen2, Wang et al., 2024)), as well as three large models: GPT-4Vision (i.e., GPT-4V, OpenAI, 2023)), GPT-4o (OpenAI, 2024), and Gemini 1.5 Pro (i.e., Gemini, Anil et al., 2023)), where the evaluation takes place from June to September 2024. More details about the model configuration can be found in App. F.I. The prompting templates and demonstration examples for various models are given in Figs. 11 to 14 App. G.I.1.
4

---

Do Vision-Language Models Really Understand Visual Language?
Text Entity
Visual Entity
Implicit Relation
Explicit Relation
Acc (%)
Acc (%)
(V(r NR)
@=(ViKEF
@s(V'KK, NR) Qs(VjKE,NC
(5(*|K,, sc)
(s(A) (s(K]ke, NC)
LLaVA
38.9
26.6
46.4
30.8
LLaVA
30.2
27.5
35.1
Molmo
93.4
78.8
64.1
54.0
Molmo
71.7
31.2
59.1
LLaMA
91.3
90.9
72.7
70.1
LLaMA
75.4
32.0
55.2
46.1
Qwen-2B
82.3
73.2
63.1
53.5
Qwen2-2B
63.3
29.8
44,0
33.1
Qwen-7B
97.6
73.0
94.5
73.0
Qwen2-7B
74.4
59.0
59.8
51.5
Qwen-72B
99.1
97.9
90.6
86.4
Qwen2-72B
77.9
67.1
70.3
63.8
4PT-4V
97.8
99.6
85.7
93.7
GPT-4V
34.4
61.6
GPT-4o
99.2
100,0
92.6
94,9
GPT-4o
77.3
55.3
76.6
70.2
Gemini
88.1
95.8
87.7
86.5
Gemini
31.8
68.5
70.2
Average 87.5
81.8
77.5
71.4
Average
40.9
58.9
52.6
Table 2: Performance on QA in terms of entities in text labels or visual icons. LVLMs always identify entities correctly, and can also reason about them effectively.
Table Performance on QA relations. LVLMs struggle identify both implicit and explicit relations and are unable to reason about them effectively.
Results. We evaluate LVLMs under the Chain-of- Thhough prompting (CoT, Wei al., 2022) as in Tab. 2. Results are consistent under the zero-shot prompting (ZS) setting (App. 1.2.)). The results demonstrate that all LVLMs can easily recognize entities in both text boxes (> 85% accu- racy) and visual icons (z> 80% accuracy). The accuracies on NR questions, which assess entity recognition, remain consistently high. For the NC questions, which evaluate rea- soning ability, we find that LVLMs can answer them pretty well, achieving 80% accuracy for text entities and 75% for visual entities. Our findings entity recognition and reasoning can be summarized as follows:
Results. Tab. 3 presents the accuracies for relation ques- tions. Results indicate that all models generally struggle with relation recognition (NR questions), which leads to an average accuracy of around 65% and 60%. Models also have difficulty reasoning about relations (NC questions), with the average accuracy around 40% and 54%. Notably, even for GPT-4V, its performance on counting implicit rela- tions (i.e., relative positions) is nearly equivalent to random guessing (34.4%), and it also shows significant difficulty in recognizing or reasoning about explicit relations.
Observation (Ability understand entity). LVLMs can nearly perfectly identify entities in diagrams and demon- strate strong reasoning abilities regarding these entities.
Consistency Verification. Before proceeding, we validate our findings on relations to ensure their reliability. We adjust the diagram generation settings (e.g.. arrow features) and observe that the results remained consistent (App. F.2.3). Beyond diagram variations, we also test the robustness of our results by examining the consistency of LVLMs' per- formance with different prompting templates. First, results of zero-shot prompting is consistent, which also support our findings are valid (App. F.2.1). Second, we run our evaluations under the in-context learning (ICL) setting. The findings indicate that ICL does not improve the models' abil- ity to identify reason about relations (App. F.2.4). These results further confirm the reliability of our conclusions. Thus, we can summarize our observation as follows:
This ability is fundamental to various vision tasks. Our observation aligns with existing research, confirming that models possess the basic capability to perform simple object detection and count objects to some extent. Next, we turn our focus to complex relations to determine whether LVLMs can comprehend the intricate interactions between entities.
3.2. Can LVLMS Understand Relations?
We categorize relations into two types for our research: implicit relations (e.g., relative positions of entities) and explicit relations (e.g., arrows or segments).
Observation 2 (Ability to understand relations). LVLMs can barely identify both implicit and explicit relations, and they are unable reason about them.
Preparation. We generate synthetic diagrams following previous settings ($ 3.1). To reduce errors contributed by entity understanding, here we represent entities by text, which yields the best performance on corresponding NR and NC questions (Tab. 2). Example questions are Tab. 1. The prompting templates and demonstration examples are in Figs. 17 to 20 in App. 1.1.2.
This observation contradicts the remarkable success that LVLMs have demonstrated in understanding complex dia- grams. To further investigate their failures, we evaluate them on real diagrams to determine whether they can effectively comprehend more complex, real-world scenarios.
5

---

Do Vision-Language Models Really Understand Visual Language?
Acc (%)
Entity
Relation
Acc
Q5(E NR)
(E KF, NC)
Qg(V[Ka. NR)
(g() Qg(E|KR NR) (a(EEK,,N
Semantic Know.
w/o
W/t
w/o
w/at1
LLaVA
56.5
37.3
45.1
30.2
LLaVA
35.1
49.314.21
28.3
31.33.0t
Molmo
82.7
54.9
59.8
51.2
Molmo
59.1
69. 69.81a.7t
50.4
55.04.6t
LLaMA
87.3
59.7
73.7
51.2
LLaMA
55.2
73.318.1t
46.1
54.4s.3t
Qwen2-2B
66.6
40.7
45.7
39.3
Qwen2-7B
90.0
58.0
Qwen2-2B
44.0
60.7 60.716.7t
33.1
39.9a.8t
Qwen2-72B
93.7
77.8
79.4
69.8
Qwen2-7B
59.8
74.214.4f
51.5
42.5o.0u
Qwen2-72B
70.3
81.411.1t
63.8
60.73.11
GPT-4V
88.9
78.8
78.7
59.9
GPT-4o
93.1
82.3
84.1
72,9
GPT-4V
61.6
74.412.8t
59.5
57.91.63
Gemini
85.0
68.4
80.5
57.7
GPT-4o
76.6
82.86.21
70.2
72.92.71
Gemini
68.5
72.23.7t
70.2
64.8s.41
Average
82.6
61.8
68.7
54.5
Average
58.9
70.9
52.6
53.3
Table Performance on KR questions real diagrams. Results indicate models continue to recognize and large models can reason about entities effectively, and they struggle with reasoning about relations. However, surprisingly, models (exeept LLaVA) can recognize relations in real diagrams pretty well.
Table 5: Performance on synthetic diagrams without (Vanilla, w/) and with semantic knowledge KKnowleggeGGrundder w/). The accuracy change from Vanilla to Knowledge- Sralge-Grronnee reveals that models better identify relations in knowledge-grouuee grounded diagrams while still struggling to reason about them effectively.
3.3. Do LVLMs Understand Real Diagrams?
their construction, real diagrams often portray entities and relations that agree with commonsense knowledge about the underlying concepts ssuc as the stages of the water cycle or the predator-prey relations a food chain.
Synthetic diagrams are used evaluate the basic abilities. Next, we move to real diagrams to double-check how these models perform in practical diagram understanding.
Preparation. The evaluation follows similar settings to that on synthetic diagrams. We provide the performance on KR questions for both entities and relations real diagrams. Example questions are given in Tab. 1, and the question design for each domain in App. E.1. Prompt templates and examples are in Figs. 27 to 30 in App. G.2.
4. Knowledge Shortcut: Quantitative Analysis
Given that rich knowledge has been encoded into LVLMs during various training stages, possible hypothesis emerges: models may not truly understand diagrams but instead rely on their ingrained knowledge as shortcuts to provide answers. In this section, we explore the impact of knowledge on the models' ability to answer questions.
Results. Tab. 4 presents the performance of LVLMs on real diagrams. We observe that models continue to perform well in recognizing and reasoning about entities, with GPT- 4o achieving 93.1% 13.1% accuracy in recognition and 82.3% ac- curacy in reasoning. Besides, models still struggle to reason about relations in real diagrams, similar to their performance on synthetic diagrams. Notably, though, we find that three large models can recognize relations quite well in real di- agrams, with an average accuracy of 81.1%. From these results, we can obtain the following observation:
4.1. Knowledge Grounding of Diagrams Improves Relation Recognition
determine the effect of knowledge, we compare the mod- els' performance on diagrams with and without embedded knowledge. Our focus is on the explicit relation in synthetic diagrams. We construct relations that incorporate semantic knowledge to simulate practical conditions where models might use this knowledge as shortcuts. If our hypothesis is correct, we would expect to see an increase in accuracy on QA tasks for these specially constructed diagrams.
Observation (Performance real diagrams). LVLMS struggle to recognize relations in simple synthetic diagrams, yet they can recognize relations in complex real diagrams.
We substantiate that LVLMS cannot understand relations in synthetic diagrams, yet this finding reveals a contradiction. While the models do not inherently possess the ability to recognize relations, they are able to do in real diagrams. This outcome contradicts our initial intuition (Intuition I). Therefore, we further investigate these counterintuitive find- ings by examining the key difference between synthetic and real diagrams: the role knowledge. While synthetic diagrams contain entities and relations that are random in
Preparation. To construct diagrams grounded with se- mantic knowledge, we generate relations with real mean- ing behind them. Specifically, for each entity, we get its Word2Vec embedding (Mikolov et al., 2013) based on the text attribute, and use cosine similarity implemented by spaCy (Honnibal et al., 2020). If the similarity between entity text is larger than 0.5, we regard that there exists a relation. We construct a semantic graph on all the entities from Lu et al. (2021) with 377 entities and 1901 relations.
6

---

Do Vision-Language Models Really Understand Visual Language?
Acc (%)
[Rx(V/KF. N8) (p(() [<<(V|KFNN)) @x(V/KK NC)ATa
Ace (%)
Qg(E|KR N8)a1 a R2(E NC)
Qg(5)K A1
LLaVA
40.6
56.516.9t
31.0
37.3s.at
LLaVA
44.7
45.1o.4t
29.3
30.2
Molmo
79.4
82.7a.st
60.7
54.9..8
Molmo
54.4
59.8s.4t
49.4
51.2) 81
ILaMA
85.2
87.32.1f
59.8
59.70.14
LLaMA
63,4
73.710.at
56.7
51.20.51
Owen2-2B
62.2
66, 66.64.41
40.2
40.7c.6t
Qwen2-2B
42.6
45.7a.11
32.9
39.3a.41
Qwen2-7B
90.2
90.0b.24
60.6
56.14.6
Qwen2-7B
58.8
71.412.67
53.5
58.0&.51
Qwen2-72B
92.5
93.71.21
77.8
77.80.0-
Qwen2-72B
66.7
79.412.7t
62.7
69.87.11
GPT-4V
91.5
88.9q (o
75.2
78.5 78.8g
GPT-4V
69.7
78.7a.0t
57.1
59.92.st
GPT-4o
93.6
93. 13.1o.a
79.1
82.3g.21
GPT-4o
73.9
84.110.21
65.0
72.9
Gemini
82.0
85.0g.0r
75.5
68.47.11
Gemini
66.3
80.514.21
61.2
57.7a.51
Average
79.7
82.6
62.2
61.8
Average
60.0
68.7
52.0
54.5
2. B1
6.71
2.51
Table 6: Performance for questions on entities in real diagrams. The accuracy gap (from answering KF questions KR questions) indicates that models perform similarly on entity questions, regard- less of whether or not they require background knowledge.
Table 7: Performance for questions on relations in real diagrams. The accuracy gap (from answering questions to questions) suggests that models perform better on NR questions when these questions require knowledge to answer. However, there is no sig- nificant improvement in accuracy when it comes to NC questions.
We randomly generate diagrams following the settings men- tioned in 2.3, but we constrain the generated diagram as the subgraph in the constructed semantic gruph. See Figs. 25 and 26 in App. G.1.5 for the prompting templates and demonstration examples.
knowledge in questions related to entities. The results show that the gap between KF and KR questions for entities in real diagrams is negligible. The average accuracy increase in recognition is only smaller than 3% (except LLaVA achieves better performance when knowledge is required). The in- crease in reasoning is similarly minimal at 0.2%.
Results. Tab. 5 presents the evaluation accuracies di- agrams without and with semantic knowledge. For com- parison, we include the performance changes relative to the original results shown in Tab. 3. Overall, models ex- hibit improved relation recognition in diagrams containing semantic knowledge. The average accuracy improvement for NR questions is 11.6%. However, for NC questions, the performance remains largely unchanged. The findings are consistent with our hypothesis: models utilize knowledge in diagrams as relation recognition shorteuts. With these results, we have below observation:
Evaluation on relation questions. Similarly, Tab. 7 illus- trates the impact of knowledge on questions related to rela- tions. The results show that when questions require knowl- edge, models are better at recognizing relations (except LLaVA achieves roughly the same performance). However, their reasoning ability remains largely unchanged. Specif- ically. the average accuracy increase gap in recognition is 9.5%, while the gap in reasoning is only 1.2%. Thus, we have the observation below:
Observation 4 (Knowledge in diagrams). LVLMs are more effective at recognizing relations in diagrams that relate to background knowledge, they can use it as a shortcut.
Observation 5 (Knowledge shortcuts help). LVLMs are better recognizing relations in real diagrams when the question requires them to use background knowledge. How- ever, whether or not question requires knowledge does not significantly impact the models' performance in entity recognition, entity reasoning, or reasoning about relations.
This observation indicates that even when questions do not explicitly require background knowledge to answer them, the presence of knowledge in the diagram can still enhance the performance of relation recognition. Next, we investi- gate whether questions that require models to actively use knowledge could further improve their performance.
Observations 4 and 5 lead to conclusions that are entirely contrary our initial intuition (Intuition 1). In this section, quantitative analysis to support our hypothesis that knowledge acts shortcut for recognizing relations. In the next section, we provide qualitative analysis to further substantiate the validity of this hypothesis.
4.2. Relation Recognition Questions Requiring Knowledge Show Improvements
We then evaluate the models using questions that do not require background knowledge (KF questions) and compare their performance with questions that do require such knowl- edge (KR questions).
5. Knowledge Shortcut: Qualitative Analysis
We provide further evidence to confirm that LVLMs can only recognize and reason about entities but not relations in
Evaluation on entity questions We follow the same set- tings in previous experiments. Tab. 6 illustrates the role of
We presume that this is hecause LLLVV is weaker than other models such that it cannot handle relations in real diagrams well.
7

---

Do Vision-Language Models Really Understand Visual Language?
Onty (Kt, QRl (C) Raly (Kk. NK)
Gnlie an, sR) Qy(A NK)
(gRV
Accuracy
0.8
Accuracy
0.8
0.6
0.6
0.4
0.4
< 4
5-6
7-8
9-10
2
S 4
5-6
7-8
9-10
> 11
|>|
|
(a) Acc.: entity questions
(b) Acc.: relation questions
Figure 2: Performance of LVLMs (CoT) on answering questions for real diagrams with different complexities (i.e.. the number of entities in the diagram, [v|). Results show that models can always answer questions entity well but cannot handle questions on relations if the diagram complex.
real diagrams ($ 5.1). The ability to recognize relations in real diagrams appears to be an illusion driven by knowledge shortcuts rather than genuine understanding ($ 5.2).
no relational information or when the relations presented conflict with background knowledge. For evaluation, we pose an annotated question (QR(E] [K|K,, NR)) and a complex reasoning question involving food chain counting and use the configuration in App. F.I.
5.1. LVLMs Cannot Recognize and Reason About Relations in Real Diagrams
In Fig. 3, when comparing the responses in the left sub- figure (vanilla) with those in the middle subfigure (w/o relation), we observe that even in the absence of explicit relational information, the model still identifies the correct predator. Additionally, for the food chain counting question, the model continues to provide the original answers. This indicates that the model has pre-existing knowledge and it can use the knowledge as a shortcut for answering ques- tions. Similarly, when comparing the responses in the left subfigure (vanilla) with those in the right subfigure (with random relations), we find that the model provides the same answers despite the introduction of random relations. The new correct predator could be "A)" or "C)", and the new cor- rect food chain count 4. This demonstrates that the model relies on learned knowledge rather than parsing the diagram itself. With this additional evidence, we can reasonably conclude the following finding:
Preparation. We use the number of entities in a dia- gram as an indicator of its complexity, with the answers to (V]KF, NC) providing the entity count, as introduced in Tab. 1. We then divide all real diagrams into five bins based on their entity count, ensuring that each bin contains more than 100 diagrams (detailed statistics are provided in Fig. 9). For each subset of diagrams, we report the average accuracy for all questions under the CoT setting.
Results. Fig. 2 present the average accuracy of three models on all questions (detailed results are in Fig. 10 in App. F.3). Overall, the performance on entity questions remains consistent across different levels of diagram com- plexity, while there is noticeable decline in accuracy for relation questions as diagram complexity increases. These results further support that LVLMs can understand entities but struggle with understanding relations. Additionally, they suggest that the models' apparent success in recognizing and reasoning about relations in real diagrams is largely due their performance on simpler diagrams, rather than true comprehension of complex relational structures.
Finding: Current LVLMs can recognize and reason about entities diagrams but struggle with understanding rela- tions. However, they manage to answer diagram-related complex questions by identifying entities and leveraging relevant learned knowledge shortcut.
5.2. LVLMs Hallucinate Relations in Real Diagrams
6. Related Work
Next, we provide an intuitive and vivid case study to demon- strate that LVLMs hallucinate when interpreting relations in diagrams. Using the example diagram from Fig. 1, we construct two special cases for comparison: in the first case, we remove all relations from the diagram, and in the sec- ond case, we replace the relations with random ones. The goal is test the GPT-4o's 's response when there either
LVLM Evaluation. Recent benchmarks evaluate Large Vision-Language Models (LVLMss) on perception, knowl- edge, hallucination, and reasoning (Fu et al., 2023; Yu et al., 2023; Yue et al., 2024; Liu et al., 2023). These benchmarks use captioning visual question answering probe model capabilities, but often use natural or synthetic images that
8

---

Do Vision-Language Models Really Understand Visual Language?
whale
Loqped
Killer
Leqpand
Killer whole
Loqued
Tmpperor propris
Eneperos peoplls
peopiin
Squid
Squid
Squid
Xidt
Kill
Koll
Phynoplankton
Phytoplankoos
Phynoplankten
Q: Which creature the predator of Krill? A: Phytoplankton, B: Whale, C: Fish, D: Krill
GPT-4o: From the image, the
GPT Based on the correct
GPT-4o: Based the best answer would be: C: Fish
correct answer would be Fish. v
answer Fish.
X
Think step by step before answering the question. How many food chains there? A) C) 5
GPT-4o: Let's carefully analyze the diagram... Answer: A) 6
t GPT-4o: approach this
GPT-4o: Let's analyze the image answer is: A) 6
X
problem answer is: A) 6
Figure 3: The model response the example diagram and its variants. Results suggest that the model relies background knowledge a shortcut rather accurately recognizing and reasoning about relations.
differ from symbolic visual languages like diagrams.
shortcuts when answering complex diagram reasoning ques- tions. These results suggest that the apparent successes of LVLMs in diagram reasoning tasks create misleading impression of their true diagram understanding capabilities.
Diagram QA and Reasoning. Prior DQA work spans syn- thetic datasets like NLVR and CLEVR (Suhr et al., 2017; Johnson et al., 2017), textbook diagrams like A12D (Kem- bhavi et al., 2016), and statistical graphics (Masry et al., 2022). While some focus on layout or procedure under- standing, others probe commonsense reasoning. Our work uniquely investigates the gap between synthetic and real di- agrams, attributing LVLMs" diagram performance to knowl- edge shortcuts.
Impact Statement
Our study reveals a critical gap in the current evaluation of Large Vision-Language Models (LVLMs)) their apparent success in diagram reasoning tasks often stems from lever- aging background knowledge rather than genuine diagram comprehension. By introducing a comprehensive test suite that decouples visual recognition, relational reasoning, and knowledge reliance, we provide robust framework for ana- lyzing the symbolic reasoning capabilities of LVLMs. This insight challenges prevailing assumptions about multimodal understanding and offers practical implications for future benchmarks, model development, and safety-sensitive ap- plications where precise symbolic reasoning is essential.
LVLM Capabilities and Limits. While LVLMs show strong performance on QA and image reasoning (Chen et al., 2023; Fatemi al., 2024), recent studies highlight their limitations in basic reasoning and perception (Hu et al., 2024; Yang et al., 2024). Our work builds on this view, showing how diagram understanding is often an illusion created by leveraging pre-trained knowledge.
A cknnoweegment
7. Conclusion
We thank the reviewers for their constructive feedback. We also thank Chenxi Pang, Jingwei Ni, and Shaobo Cui for their valuable input during the early stages of this work. Yifan Hou is supported by the Swiss Data Science Center PhD Grant (P22-05).
We evaluate three LVLMs diagram understanding using our test suite, including synthetic and real diagrams. Our findings reveal that while these models can perfectly recog- nize and reason about entities depicted in the diagrams, they struggle with recognizing the depicted relations. Further- more, we demonstrate that the models rely on knowledge

---

Do Vision-Language Models Really Understand Visual Language?
References
Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C.,, Allonsuss, D., Song, D., Pintz, D.. Livshits, D., Esiobu, D., Choud- hary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hup- kes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Di- nan, E., Smith, E. M., Radenovic, F,, Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J. Mahadcokar, J... Shah, J.. van der Linde, J., Bil- lock, J., Hong, J., Lee, J., Fu, J.. Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J.. Saxe, J., Jia. J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The Ilama 3 herd of models. CoRR, abs/2407.2178. 2024. doi:
Anderson, M., Meyer, B., and Olivier, P. Diagrammatic rep- resentation and reasoning. Springer Science & Business Media, 2011.
Anil, R., Borgeaud, S., Wu, Y., Alayrae, J., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonogiou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lilli- crap, T. P., Lazaridou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee, B., Viola, F,, Reynolds, M., Xu, Y. Doherty, R., Collins, E., Meyer, C., Ruther- ford. E., Moreira, E., Ayoub, K., Goel, M., Tucker, G., Piqueras, E., Krikum, M., Barr, I., Savinov, N., Danihelka, I., Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M.. Syg- nowski, J., and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 12.3122.110,,,, 2023. doi: 10.48550/ARXI..2312.11 1105.
Fatemi, B., Halcrow, J., and Perozzi, B. Talk like a graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Represen- lations, ICIR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Bauer, M. and Johnson-Laird, P. N. How diagrams can improve reasoning. Psychological Science, 4(6):372-378, 1993. doi: 10. 10.1111/j.114 14679//46679780... tb00584.xx
Chen, L., Li, B., Shen, S., Yang. J., Li, C., Keutzer, K., Darrell, T., and Liu, Z. Large language models are visual reasoning coordinators. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
Foucault, M. Discipline and punish: The birth of the prison. (transsatdd from the french by alan m. sheridan]. London: Allen Lane. cone.OOrginal work published in 1975], 1977.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., and Ji, R. MME: A comprehensive evaluation benchmark for mul- timodal large language models. CoRR, abs/2306. 13394,
Chen, L., Li, J., Dong, X., Zhang, P., Zang. Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., and Zhao, F. Are we on the right way for evaluating large vision-language models? CoRR, abs/2403.203300 2024. doi: 10.48550/ ARXIV.2403.2033.
2023. doi: 00.48550/ARXI..2366..
Greenspan, S. I. and Shanker, S. The first idea: How sym- bols, language, and intelligence evolved from our primate ancestors to modern humans. Da Capo Press, 2009.
Cromley, J. G., Snyder-Hogan, I.. E., and Luciw-Dubas, U. A. Cognitive activitics in complex science text and diagrams. Contemporary Educational Psychology, 35(1): 59-74, 2010. ISSN 0360-07000 1466-4 doi: https://doi.or///I 10. 1016 1016/j.cedpsych.2000 w0t...00009.00000 10.002.
Gupta, T. and Kembhavi, A. Visual programming: Compo- sitional visual reasoning without training. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 14953-14962. IEEE, 2023. doi: 10.1 109/ CVPR52772.2023.0122
de Rijke, M. Logical reasoning with diagrams, gerard all- wein and jon barwise, eds. J. Log. Lang. Inf., 8(3):387- 390, 1999. doi: 10.1023/A:1008848911
Hippala,, T. and Orekhova, S. Enhancing the A12 diagrams dataset using rhetorical structure theory. Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S.,, Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S., and Tokunaga, T. (eds.), Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA), 2018
Deleuze, G. Foucault. Univ of Minnesota Press, 1986.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B.,
10

---

Do Vision-Language Models Really Understand Visual Language?
Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. spaCy: Industrial-strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo. 1212303.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F. Zhang, H., Zhang, K., Li, Y., Liu, Z.,, and Li, C. Llava-onevision: Easy visual task transfer. CoRR, 0tt//24080033266 2024a. doi: 10.48550ARXXI.240...33.
Hu, Y., Tang, X., Yang, H., and Zhang. M. Case-based or rule-based: How do transformers do the math? CoRR, abs/2402.17709 2024. doi: 10.4855/ARXIV.2402. 17709.
Li, F.F. The Worlds I See: Curiosity, Exploration, and Discovery at the Dawn of AI. Flatiron books: moment of lift book, 2023.
Islam, M. S., Rahman, R., Masry, A., Laskar, M. T. R., Nayeem, M. T., and Hoque, E. Are large vision language models up the challenge of chart comprehension and reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2024, pp. 3334-3368, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, Evaluating object hallucination in large vision- language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 292-305. Association for Computational Linguistics, 2023. doi: 10.18653/V 005533/V1/2022 1/2023. 2222002..MLL-MMAA..
Johnson, J., Hariharan, B., van der Maaten, I.., Fei-Fei, I.., Zitnick, C. L., and Girshick, R. B. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1988-1997 IEEE Computer Society, 2017. doi: 10.1 10.1009/CVP..221722255
Li, Y., Wang, I.,, Hu, B., Chen, X., Zhong, W., Lyu, C., Wang, W., and Zhang, M. A comprehensive evaluation of gpt-4v on knowledge-intensive visual question answering, 2024b.
Liu, F., Guan, T., Li, Z., Chen, L., Yacoob, Y., Manocha, D., and Zhou, T. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v(ision), 1lava-1.5, and other multi-modality models. CORR, abs/2310. 145666 2023. doi: 10.48550/ARXIV231100 14566.
Kahneman, D. Thinking. fast and slow. Farrar, Straus and Giroux, New York, 2011. ISBN 9780374275631 0374275637.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M. Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Leibe, B., Matas, J., Sebe, N., and Welling. M. (cds.), Computer Vision ECCV 2016 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 235-251. Springer, 2016. doi: 00.1007/978833319-44444 16.
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Vanschoren, and Yeung, S. (eds.), Pro- ceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Lu, P., Mishra, S., Xia, T., Qiu, I.., Chang, K., Zhu, S., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science ques- tion answering. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28. December 9, 2022, 2022.
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5376-5384, 2017.
Krishnamurthy, J., Tafjord, O., and Kembhavi, A. Semantic parsing to probabilistic programs for situated question answering, In Su, J., Duh, K., and Carreras, X. (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 160-170, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 111111/6//11--11111 1016.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K., Galley, M., and Gao, J. Mathvista: Evaluating math reasoning in visual contexts with gpt- 4v, bard, and other large multimodal models. CORR, abs/2310.02255, 2023. doi: 10.48550/ARXI..3310. 02255.
Kuhnle, A. and Copestake, A. Shapeworld a new test methodology for multimodal language understanding. CoRR, 7041144441111 2017.
Mao, C., Teotia, R., Sundar, A., Menon, S., Yang. J., Wang, X., and Vondrick, C. Doubly right object recognition:

---

Do Vision-Language Models Really Understand Visual Language?
A why prompt for visual rationales. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2722-222222 2732. IEEE, 2023. doi: 10. 10.1109/CVPR52229. 2023.00267.
Singh, S., Chaurasia, P., Varun, Y., Pandya, P., Gupta, V., Gupta, V., and Roth, D. FlowVQA: Mapping multimodal logic in visual question answering with flowcharts. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Find- ings of the Association for Computational Linguisties: ACL 2024, pp. 1330-1350, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 1..18663/1//2224ffndiigg. acl...
Masry, A., Do, X. L.,, Tan, J. Q.. Joty, S., and Hoque, E. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022. findings-acl.1 177.
Song, I.-Y., Evans, M., and Park, R. K. A comparative anal- ysis of entity-relaiionship diagrams. Journal of Computer and Software Engineering. 3(4))427 1995.
Suhr, A., Lewis, M., Yeh, J., and Artzi, Y. A Corpus of Nat- ural Language for Visual Reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 217-223, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10. 10.11665/vll 1/P13-23/
Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. In Bengio, Y. and LeCun, Y. (eds.), 1st International Confer- ence on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceed- 2013.
Tang, R., Kong, D., Huang, L., and Xue, H. Large lan- guage models can be lazy learners: Analyze shortcuts in in-context learning. In Rogers, A., Boyd-Graber J. L., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 4645-4657, Association for Com- putational Linguistics, 2023.
2013,
OpenAL. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 1445500/AXX..2233..07777777
OpenAI. Hello GPT-4o. 2024.
doi: 10.18653/V
Pan, H., Zhang, Q.. Caragea, C., Dragut, E., and Latecki, J. Flowlearn: Evaluating large vision-language models on flowchart understanding. In Endriss, U., Melo, F. S., Bach, K., Diz, J. J. B., Alonso-Moral, J. M., Barro, S., and Heintz, F. (eds.), ECAI 2024 27th European Con- ference on Artificial Intelligence, 19-24 October 2024, Santiago de Compostela, Spain Including 13th Confer- ence on Prestigious Applications of Intelligent Systems (PAIS 2024), volume 392 of Frontiers in Artificial Intelli- gence and Applications, pp. 73-80. IOS Press, 2024. doi: 10.3233/FA1A244473.
BADINGG-ACL.225
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., Fan, Y.. Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, Qwen2-vl: Enhaneing vision-language model's perception of the world at any resolution. CoRR, 12s/2409.11191, 2024. 12191.
doi: 10.48550/ARXVV2200099
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning models. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho. K., and Oh, A. (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
large language
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. Is chatgpt a general-purpose natural language processing task solver? In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp- 1339-1384. Association for Computational Linguistics, 2023. doi: 1/6633/11/2223.MMLPPPM
Yang, H., Meng, F., Lin, Z.,, and Zhang, M. Parrot mind: To- wards explaining the complex task reasoning of pretrained large language models with template-content structure, 2024.
Sco, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Mal- colm, C. Solving Geometry Problems: Combining Text and Diagram Interpretation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1666-177,, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 1
Ye, K. and Kovashka, A. A case study of the shortcut effects in visual commonsense reasoning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial
12

---

Do Vision-Language Models Really Understand Visual Language?
Intelligence, IAAI 2021, The Eleventh Symposium Ed- ucational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 3181-3189. AAAI Press, 2021. doi: 10.1609/AAALV33514.6222.
Yu, W., Yang, Z., Li, L., Wang. J., Lin, K., Liu, Z.,, Wang, X., and Wang, L.. Mm-vet: Evaluating large multimodal mod- els for integrated capabilitiis. CoRR, Abs/2308.02490 2023. doi: 11445500ARXXIV223800229999 02490.
Yue, X., Ni, Y., Zhang, K., Zheng, Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin. M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556 9566,9957,,66 June 2024.
Zdebik, J. Deleuze and the diagram. Deleuze and the Diagram, pp. 1-256, 2012.
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P.,
Zhou, A., Lu, P., Chang, K., Gao, P., and Li, H. Math- verse: Does your multi-modal LLM truly see the dia- grams visual math problems? CoRR, abs/2403.14624, 2024. doi: 1044550/ARXXV.2405..402 14624.
13

---

Do Vision-Language Models Really Understand Visual Language?
A. Limitations
Our work has two main limitations. First, we focus exclusively on diagrams that depict various entities and relationships. There may be other specialized types of diagrams that are not well-suited to this representation. We encourage future research explore and analyze model performance on such diagrams. Second, while we demonstrate that LVLMs have limited diagram understanding capabiliities and that their strong performance is largely due knowledge shortcuts, do not offer insights on how to address this issue. Future work could focus on developing strategies to enhance LVLMs' true diagram understanding abilities. Besides, we use the simple prompt to make sure the comparison is fair. Future work may explore more complex prompt templates and better answer (i.e., model output) parsers for better performance.
B. Extended Discussion on Related Work
Evaluation of LVLMs. A growing number of benchmarks assess the capabilities of multimodal models, particularly LVLMs, across various dimensions such as perception (Fu et al., 2023), factual knowledge (Yu et al., 2023; Li et al., 2024b), hallucination (Liu et al., 2023; Li et al., 2023), and reasoning ( (Yue et al., 2024). These benchmarks use image captioning or visual question answering (VQA) as evaluation formats, using images sourced from natural image datasets (Li et al., 2023), or generated specifically for probing model abilities (Liu et al., 2023; Zhang et al., 2024). While informative, they primarily focus on natural images, and thus may not reflect models' performance on more abstract visual representations like diagrams.
Diagram Question Answering (DQA). DQA studies have explored a range of tasks, from spatial reasoning with simple synthetic diagrams (e.g.. NLVR (Suhr al., 2017), ShapeWorld (Kuhnle & Copestake, 2017)) understanding textbook- style illustrations (e.g.. FoodWeb (Krishnamurthy al., 2016), A12D (Kembhavi et al., 2016), and TQA (Kembhavi et al., 2017)). While CLEVR (Johnson et al., 2017) and IconQA (Lu et al., 2021) involve spatial and logical reasoning over diagrams, many real-world datasets rely heavily on commonsense knowledge rather than visual reasoning. Recent work on statistical diagram understanding (e.g., charts and flowcharts) highlights LVLMs' limitations in parsing basic visual elements and relations (Masry et al., 2022; Islam et al., 2024; Singh et al., 2024; Pan et al., 2024). Our work aligns with these findings in synthetic settings and further explains why LVLMs excel in real-world diagram tasks by leveraging prior knowledge instead of true diagram parsing.
Capabilities and Limitations of LVLMs. LVLMs are often praised for their impressive performance on multimodal reasoning tasks, including diagram understanding (Chen et al., 2024), complex question answering (Qin et al., 2023), and visual reasoning (Chen et al., 2023; Gupta & Kembhavi, 2023). However, contrasting body of work points to fundamental limitations, such as hallucination, shortcut learning, and poor generalization on tasks requiring perception and logical consistency (Yang et al., 2024; Hu et al., 2024; Mao et al., 2023). Notably, LVLMs often fail in basic reasoning tasks like counting (Fu et al., 2023) and relation tracking (Yue et al., 2024). Our work reinforces this perspective by demonstrating that LVLMs' diagram reasoning often an illusion driven by background knowledge rather than actual -ymboliy reasoning.
C. Ethical Considerations
Our paper focuses on evaluation using synthetic and public datasets, and we thereby do not foresee any ethical issues originating from this work. The license of FoodWeb (Krishnamurthy et al., 2016) and A12D (Kembhavi et al., 2016) is BSD-2-Clause and Apache-2.0 respectively.
D. Reproducibility
We have tried our best to ensure the reproducibility of our work. The datasets and models that we use in this paper are publicly available. We present all the details about our evaluation and data including the annotation in the Appendix. At the same time, we have uploaded our evaluation code, generated synthetic diagram data, annotated real diagram data, as well as the responses of LVLMs in supplementary files. Every stage of our work ranging from code to results is introduced thoroughly and can be easily reproduced.
14

---

Do Vision-Language Models Really Understand Visual Language?
E. Supplementary Test Suite Details
We provide additional details about our test suite here.
E.1. Question Annotation Details
For questions on synthetic diagrams, we annotate them as described in Tab. 1. Regarding real diagrams, we introduce their annotation details concerning the domains respectively. For each domain, we choose a representative diagram to show how we annotate, as shown in Fig. 4.
For each question in real diagram set, we provide four options. We first annotate the correct option. The three incorrect options are sampled uniformly without replacement from the pool of correct answers for questions of the same type, excluding the current correct answer. Furthermore, we manually verify (and edit if necessary) the sampled negative options to ensure that they are not inadvertently correct answers. The four options are randomly shuffled before feeding into LVLMs. Tab. records some statistics of our real diagram set.
Domains
Ecology
Biology
Physics
Astronomy
Chemistry
Geology
Num. Entity Rep.
462
205
77 Visual Explicit
101
54 Text & Visual Explicit
102 Text & Visual Implicit
Text & Visual Explicit
Text & Visual Explicit
Text & Visual Implicit
Relation Rep.
Food Chain Food Web
Solar System
Water Cycle
Planet Structure
Topics
Life Cycle
Circuit
Satellite System
Carbon Cycle
Star Structure
Table 8: Details of the real diagram set. "Entity Rep." and "Relation Rep." refer to the way that entities and relations are represented. "Topics" introduces the typical types of diagrams in that domain.
E.1.1. ECOLOGY- FOOD WEB; FOOD CHAIN
Food web diagrams illustrate predatory relationships among animals (and between animals and plants or detritus) in the same environment (c.g. prairie, forests, sea, etc.). Any possible path from a plant to top animal is a single food chain.
QR(V/KF, KF, NR). This question type is consistently annotated as "Which entity is in the diagram?". For Fig, 4a, the correct option "Sparrow".
QR(ViKF, NC)). This question type is consistently annotated as "How many entities are in the diagram?". For Fig. 4a, the correct option is "10".
Qn(V|KR, KR, NR). The annotation of the question type is determined by the number of producers in the diagram: (a) If the diagram contains three or fewer producers, the question type is annotated as "Which producer is in the diagram?". (b) If the diagram includes more than three producers, the annotation changes to "Which producer is not in the diagram?". (c) Only in cases where the food web diagram contains no producers (i.e. in the detritus environment) do we annotate the question as "Which consumer is not in the diagram?", Fig. 4a contains two producers, "Grass" and "Blueberry Bush", which is applicable to category (a). The correct option we annotate is "Blueberry Bush". The (a)(b((c) question subtypes account for 85.71%, 11, 10.60%, and 3.68% respectively. As the LVLM must comprehend the concepts of producer and consumer correctly answer these questions, this question type is classified requiring knowledge.
QR(V/KR, NC). This question type is consistently annotated as "How many distinct consumers are there in the food web?", which represents a subset of (R(V/KF, Nc). For Fig. 4a, the correct option is "8". Since the LVLM needs to comprehend the meaning of consumer, this question type classified requiring knowledge.
QF(E|KF, NR). Depending on the number of arrows linked to an entity, for an entity with numerous connections, the question is annotated as "Which entity is not connected to Entity?" For an entity with fewer connections, the question is annotated as "Which entity is connected to Entity?". These two subtypes account for 48.05% and 51.95% respectively. For Fig. 4a, we ask "Which entity connected to Shrew", and the annotated correct option is "Great Horndd Owl". This
15

---

Do Vision-Languee Models Really Understand Visual Language?
WOODLAND ECOSYSTEM
Life Cycle of a Monarch Butterfly
Sparrow
Skunk
(chrysalis)
Rabbit
Shrew
adult butterfly
Field Mouse
(caterpilar)
Grashhopper
Blueberry
Bush
Grass
aa) Ecology
(b) Biology
Load Liig-tLbgbbubbb
Source-Sinne Single Dry Cell (chemical
(energy conversion
heat/ light)
23.5
September 22-23 Autumnal Equinox
Sonductors-Wires
December 21-22 Winter Solatice
June 20-22 Summer Soletice
March 20-21 Viemal Equinox
Control-On-Of Switch
(c) Physics
(d) Astronomy
Earth Internal
CO2
plants CO2
Combustion gives CO2
Crust
Mantle
Outer Core
Animals respire and gives off CO2
Inner Core
Aakur
Animels which contain carbon
Decompasing gives off CO2
(e) Chemistry
(f) Geology
Figure 4: The representative diagrams of domains.
16

---

Do Vision-Language Models Really Understand Visual Language?
question type only involves recognizing the relations between an entity of interest and others, and thus knowledge is required.
QR(FFKF, NC). This question type is consistently annotated as "How many arrows are linked to Entity in the diagram?". For each diagram, randomly select an entity that has both arrow pointing to it and an arrow pointing out of it the entity of interest. For example, in Fig. 4a, we annotate "How many arrows are linked to Skunk in the diagram?" with the correct option "3" This question type only involves counting the relations (represented by arrows) between an entity of interest and other entities, and thus no knowledge is required.
QR(F/KR, NR). Corresponding to QR( FKK,, NR), we translate "Which entity is not connected to Entity?" to "Which not the predator of Entity?" or "Which not the prey of Entity?".. Similarly, we translate "Which entity is connected to Entity?" to "Which is the predator of Entity?" or "Which the prey Entity?". Entity of interest is the same as Q (2EEE) (E|KF, KF, NR). These four question subtypes account for 3.98%, 38.94%, 30.75%, and 26.33% respectively. Refer our annotation of Fig. 4a QR( KF, NR), for this question type, we annotate question as "Which is the predator of Shrew?" with the correct option "Great Horndd Owl". Since the LVLM needs to comprehend the meaning of prey (arrow points to the entity) and predator (arrow points out of the entity), this question type requires knowledge.
QR(E|KR, NC). This question type is consistently annotated as "How many types of prey are consumed by Entity in the foodweb?" with the same entity of interest as ( NR). Accordingly, for Fig. 4a, we annotate "How many types of prey are consumed by Skunk in the foodweb?" with the correct option "2". Similarly, since the LVLM needs to comprehend the meaning prey, this question type requires knowledge.
E. BIOLOGY LIFE CYCLE
In contrast E.1.1, this domain delineates the developmental morphology of single species across its life stages (e.g.. for insects in diagrams like Fig. 4b, stages such as egg, pupa, and adult are depicted). These morphological stages are interconnected via arrows, forming directed cyclic graph. Notably, those diagrams include arrows pointing from the mature form (adult) to its offspring (egg, embryo, or seed). We claim that the mature form represents the final stage of the species' life cycle, while the offspring represents the first stage.
QR(V[KF) KF, NR). This question type is consistently annotated as "Which entity is in the diagram?". For Fig. 4b, the correct option "Larva'.
Qn(V/KF, (C). This question type is consistently annotated as "How many entities are in the diagram?". For Fig. 4b, the correct option is "4"
QR(V/KR, KR, NR). This question type is annotated as "What is the [firs//last] life stage for the creature in the diagram?", where we choose one two words manually, resulting 47. 47.80% for "first" and 52.20% for "last". For Fig, 4b, we annotate "first" with the correct option "Egg". Determining the first and last stage of lifecycle requires background knowledge in biology.
QR(V[KR, KR. NC). This question type is consistently annotated "How many life stages are after the stage Entity in the diagram?".. For Fig. 4b, we replace Entity with "Larva" and the correct option is "2" Since the LVL. needs to comprehend the meaning of life stage and the arrow direction, this question type requires knowledge.
(F This question type is consistently annotated as "Which entity is connected to the Entity?" For Fig. 4b, we replace Entity with "Larva" and the correct option is "Egg".
(F(E NC). This question type is consistently annotated as "How many arrows are in the diagram?", In Fig. 4b, the correct option is "4"
@R(E|KR, NR). This question type is consistently annotated as "Which stage is after the Entity stage in the diagram?". For Fig. 4b, we replace Entity with "Larva" and the correct option is "Pupa". This question type requires LVLMs to understand the meaning of stage and arrow directions, and thus knowledge required.
17

---

Do Vision-Language Models Really Understand Visual Language?
QR(E)KR, NC). This question type consistently annotated "How many stages can the creature change in the diagram?" Same (F(KK, NC), in Fig. 4b, the correct option is "4". Similarly, since the LVLM needs to comprehend the meaning of stage and creature, this question type requires knowledge. A few diagrams use the same lifecycle paradigm to describe multiple species, e.g. egg + tadpole/chick + frog/chicken. The correct answer to Q (E(EEKR, NC) in this case is 3 while that to (F|K,,, NC) is 6.
E.1. PHYSICS CIRCUIT
This domain contains simple middle-to-highe hiigh-so-oh--- circuit diagrams, typically containing only power, switches, wires, and a few appliances (e.g., light bulbs), and does not contain circuit diagrams for complex electronics that require knowledge beyond high school. Circuit diagrams can be abstracted undirected cyclic graphs. Note that in this domain, do not regard wires entities even if there is explicit text in the diagram. Instead, consider them to be the representations of relations.
QR(V KF, NR)) This question type is consistently annotated as "Which entity is in the diagram?". For Fig. 4c, the correct option is "Bulb".
This question type is consistently annotated "How many entities are in the diagram?". For Fig. 4c, the correct option is "3"
[F)]KK,NC)..
QR(V/KR, NR). This question type is annotated as "Which electronic component is in the diagram?". For Fig. 4c, the correct option is "Battery". Knowledge is required to understand the meaning of electronic component.
Qn(V|KK KR, NC). This question type is consistently annotated as "How many bulbs in the diagram will glow when the switch is closed?" In Fig. 4c, the correct option is "1" Since the LVLM needs understand the function of switch well as determine whether it is a closed circuit, this question type is classified as requiring knowledge. All the diagrams contain switches, but some do not contain light bulbs or they are intentionally short-circuited. In such cases, the correct answer is O.
QR(E|KF,NR) This question type is consistently annotated as "Which entity is connected to the Entity by the line?". For Fig. 4c, we replace Entity with "Bulb" and the correct option is "Battery".
(F, This question type is consistently annotated as "How many line segments are in the diagram?". For Fig. 4c, the correct option is "3"
(R(E/KR, NR). This question type is consistently annotated "Which electronic component connected to the Entity by the wire?" Same Q (R(EEK,,RR) NR), in Fig. 4c, we replace Entity with "Bulb"and the correct option is "Battery". This question type requires LVLMs to understand the meaning of electronic component and wire, and thus knowledge is required.
Qn(E)RR NC). This question type is consistently annotated "How many wires in the diagram?". Same as ( NC)), in Fig. 4c, the correct option is "3" Similarly, since the LVLM needs comprehend the meaning of wire, this question type requires knowledge.
1.4. ASTRONOMY SOLAR SYSTEM; SATELLITE SYSTEM
The subject of the diagrams in this domain encompasses seasonal changes caused by the Earth's revolution around the Sun, moon phase changes caused by the rotation of satellites around the planets, and planetary revolutions in the solar system. Diagrams describing phase changes can be regarded as directed cyclic graphs. A diagram depicting the solar system uses relative positions express the relation between astronomical objects without using arrows.
(R[V|KF,NR) NR). option "Sun".
This question type consistently annotated as "Which entity is in the diagram?", For Fig. 4d, the correct
Qn(v[KF KF, NC). correct option "5".
This question type is consistently annotated as "How many entities are in the diagram?". For Fig. 4d, the
18

---

Do Vision-Language Models Really Understand Visual Language?
Qn(ViKR, NR). This question type annotated "Which astronomical object is in the diagram?". For Fig. 4d, the correct option is "Sun". Knowledge is required understand the meaning of astronomical object.
Qn(VIKKRNC). This question type is consistently annotated as "How many planets or satellites are in the diagram?" For Fig. 4d, the correct option is "4". Since the LVLM needs to understand the meaning of planets and satellites, this question type requires background knowledge.
QR (e(K)),,NK) This question type is consistently annotated as "Which entity is connected to the Entity in the diagram?". For Fig. 4d, we replace Entity with "September" and the correct option is "December".
(R(F NC). This question type is consistently annotated as "How many arrows are the diagram?". For Fig. 4d, the correct option is "5". Some of the diagrams rely on relative positions rather than arrows to represent relations between entities, in which case the answer is O.
Qn(E|KR,NK) This question type is consistently annotated as "What is the next phase after the Entity in the diagram?". For Fig. 4d, we replace Entity with "Earth in Summer" and the correct option is "Earth in Fall" This question type requires LVLMs to understand the meaning of phase, and thus knowledge is required.
OR(E|KK,NO NC). This question type is consistently annotated as "How many times that the planets or satellites can change in the diagram?". For Fig. 4d, the correct option is "4". Similarly, since the LVLM needs to comprehend the meaning of planets and satellites, this question type requires knowledge.
1.5. CHEMISTRY WATER CYCLE; CARBON CYCLE
This domain includes topics such as the water cycle and the carbon cycle where various plants and animals participate, and photosynthesis and transpiration of a specific plant. Water or carbon enjoy multiple pathways to transfer between two phases, so that substantial diagrams can be viewed as directed multigraphs. Other diagrams that describe single cyclic pathway (such as photosynthesis in a single plant) can be viewed as directed cyclic graphs.
For example, the visual entities in the Fig. 4e are Sun, House Emissions, Carbon Dioxide, Cow, Ground (Soil), Tree, Worm, and Cloud. Among these, Smoke, Cow, Ground, Tree, and Worm are actively involved in the depicted carbon cycle. These entities are called cycle stages. Entities such as soil, lakes, and forests are, per image segmentation principles, considered as single entities despite their spatial extent.
QR(V[KF,NE NR). This question type is consistently annotated as "Which entity is in the diagram?". For Fig. 4e, the correct option is "Cow".
This question type is consistently annotated as "How many entities are in the diagram?". As mentioned in the domain summary, there are "8" visual entities in Fig. 4c.
QR(V[KF,NC)
Qn(v/KR NR). This question type is annotated as "Which cycle stage is described in the diagram?". For Fig. 4e, the correct option is "Animal".
QR(V/KR, NC). This question type is consistently annotated as "How many different cycle stages are in the diagram?" As mentioned before, there are "5" cycle stages in Fig, 4e. Since the LVLM needs to understand the function of cycle stages, this question type requires knowledge.
On(E|KF,,K) This question type is consistently annotated as "Which entity is connected to the Entity by the arrow?". For Fig. 4e, we replace Entity with "Cow" and the correct option is "Carbon Dioxide"
(R(E|KF,NC) NC). are "1" arrows.
This question type is consistently annotated as "How many arrows are in the diagram?". In Fig. 4c, there
19

---

Do Vision-Language Models Really Understand Visual Language?
(ERR) NR). This question type is consistently annotated "Which cycle stage will happen after the Entity in the diagram?". For Fig. 4e, replace Entity with "Animal" and the correct option is "Carbon Dioxide" This question type requires LVLMs to understand the meaning of cycle stage, and thus knowledge is required.
Qn(E/KR, NC). This question type is consistently annotated as "How many different processes of transitions are in the diagram?" For Fig. 4c, the correct option is "5". Similarly, since the LVLM needs to comprehend the meaning of processes of transitions, this question type requires knowledge.
E.1 GEOLOGY PLANET STRUCTURE; STAR STRUCTURE
Diagrams in this domain depict the geological structure of the Earth or other astronomical objects, showing the relationship between geological strata through their relative positions (typically nested structures) rather than explicit arrows.
QR(V/KF) (F(KK),,RR) NR). This question type consistently annotated as "Which layer is included the diagram?". For Fig. 4f, the correct option is "Crust".
(R(V|KF, NC). This question type is consistently annotated as "How many layers are the diagram?". For Fig. 4f, the correct option is "4".
Qn(v|KR, KR, NR). This question type is annotated as "Which stratification that is outside the core is included in the diagram?". For Fig. 4f, the correct option is 'Crust" Understanding the meaning of stratification and determining the containing relations between stratigraphic layers requires the use of background knowledge.
Qn([/KK KR. NC) This question type is consistently annotated as "How many stratifications outside the core in the diagram?". For Fig. 4f, the correct option is "2" ("Mantle" and "Crust"). Same as above, answering this question also requires background knowledge.
QR(F|KF,NH NR). This question type is consistently annotated as "Which layer is next to the Layer in the diagram?". For Fig. 4f, we replace Layer with "Crust" and the correct option is "Mantle". This question type only involves recognizing the adjacency between a layer interest and others, and thus no knowledge is required.
QR(F|KF,NC) This question type is consistently annotated as "How many layer boundaries are the diagram?". For Fig. 4f, the correct option is "3". We regard the boundary between layers as the representation of relation. This question only involves counting the number of boundaries, so no background knowledge is required.
Qn(E/KR, NR). This question type is consistently annotated as "Which stratification is the next outside layer of the Layer in the diagram?". For Fig. 4f, we replace Layer with "Mantle" and the correct option is "Crust". This question type requires LVLMs understand the meaning of stratification and the containing relations between layers, hence knowledge is required.
(n(E|KR, NC). This question type is consistently annotated "How many transition zones of the structure are in the diagram?". For Fig. 4f, the correct option is "3". Similarly, since the LVLM needs to comprehend the meaning of transition zones, this question type requires knowledge.
F. Supplementary Results
We introduce the details of the models we evaluate and provide additional results in this section.
F.1. Model Configurations
Generally, we spend around 8008 for all experiments. The LVLM models are provided with the system message: "You are a visual assistant answering multiple choice questions about diagrams. Read the question, inspect the diagram, and answer with the correct choice in the following format: 'A) 0
20

---

Do Vision-Language Models Really Understand Visual Language?
GPT-4V. Model used with the key gpt-4-visions - with the OpenAl API, Chat Completions. The cemperature parameter is to 0 ensure deterministic outputs and seed is given to the model to help with reproducibility. The max.tokens is limited to 600.
Model is used with the key got-4o with the OpenAl API, Chat Completions. The temperature parameter is set to 0 ensure deterministic outputs and a seed is given to the model to help with reproducibility. The max.tokens is limited to 600.
GPT-4o.
Gemini. Model is used with the key gemini -1. 5-pro with the VertexAI API, Generative Models. The temperature parameter is set to 0 to ensure deterministic outputs. The max output tokens is limited to 600.
E22 Synthetic Diagram
The additional results on the synthetic diagrams are given in this subsection.
F,2.1. ZERO-SHOT PROMPTING
Zero-Shot Accuracy (%)
LLaVA
Molmo
LLaMA Qwen-2B Qwen-7B Qwen-72B GPT-4V
GPT-4o
Gemini
Average
Qs(V(KF, NR)
35.5
73.0
87.5
97.4
97.5
98.3
97.4
91.6
86.9
85.0
Text Entiy
Qs(V[KF,
24.0
76.2
80.2
40.0
76.4
84.2
50.6
64.1
71.9
63.1
Visual Entity
Q5(V|KF, NR)
43.1
56.0
71.0
87.5
91.8
85.1
83.4
87.5
90.2
77.3
Qs(V[KF,NC)
34.1
62.5
77.5
42.9
68.5
81.7
32.4
46.7
67.2
57.1
Synthetic
Implicit Relation
(5(E(KF NR)
27.5
74.7
79.8
71.2
78.4
80.7
75.9
72.5
58.5
68.8
Qs(E(KF, NC)
26.3
29.5
28.8
32.7
35,0
44.9
30.4
37.0
30.4
32.8
Explicit Relation
(5(E(KF, NR)
31.1
55.0
49.7
49.7
58.2
68.3
57.6
61.8
61.0
54.7
Q5(E(KF, NC)
28.2
52.3
45.9
35.0
49.3
56.3
49.6
57.6
69.6
49.3
(5(E)KFF NR)
33.2
67.0
72.9
69.4
72.5
79.2
74.2
77.9
72.2
68.7
w/ Semantic Know.
(5(E/KF, NC)
25.2
51.6
50.7
34.0
45.9
51.8
55.8
60.7
68.1
49.3
Qx(V(KF, NR)
32.0
77.7
85.6
69.0
81.4
92.2
91.9
88.3
88.1
78.5
Entity
Qv KR, NR)
43.7
59.1
81.9
70.4
71.0
89.5
84.3
87.9
@R(V NC)
16.8
48.8
61.6
15.5
48.3
69.6
38.6
45.9
58.4
44.8
Real
(R(VKKR NC)
18.7
50.7
58.9
25.7
51.4
71.2
49.4
61.4
49.0
R KF, NR.
31.6
47.2
60.0
44.2
56.0
70.8
67.0
69.2
68.0
57.1
Relation
Q je (E KR, NR)
34.8
69.6
52.3
66.3
79.5
77.5
81.6
80.9
R KF, NC)
20.2
55.1
57.8
27.7
47.8
61.4
38.3
50.0
53.4
45.7
Qn KR, NC)
22.7
53.2
55.6
28.8
45.5
61.8
42.1
50.4
51.4
45.7
Table All the evaluation results under the zero-shot prompting setting. We find that these results consistent with those under the CoT prompting setting, and our conclusions are also supported by them.
We report all the results together that obtained under the zero-shot prompt setting in Tab. 9. We can find that the conclusions are consistent.
F.2.2, ENTITY POSITION AND SPATIAL. RELATION
Synthetic Diagram
Question
Question Example (with Answer Options)
(v(K,K,
Which of the text labels exists the top of the diagram?
Entity Position
Qs(V
How many text labels are there in the top row of the diagram?
Table 10: The template and example of questions for the evaluation of entity position and spatial relation in synthetic diagrams. The text with underline (e.g.. top row) is specific and varies across diagrams.
Preparation. We generate synthetic diagrams similar to the previous settings (Tab. 1). We introduce a grid structure to describe the absolute positions of entities. Specifically, we use a 3 x grid and gridlines to define the compartments the
21

---

Do Vision-Language Models Really Understand Visual Language?
canvas. The entities are placed in the center of them with generated arrows connecting them. We represent the entity via text. As depicted in the example in Tab. 10, use "top/center//oottom row/column" to describe the entity location.
Accuracy (%)
(s(ViKE,NR)
Qs(V ([V] NC)
LLaVA (ZS/CoT)
31.6 39.1
27.4 28.9
Molmo (ZS/CoT)
84.3 77.5
52.2 62.2/61.1
LLaMA (ZS/CoT)
74.0/710
39.1
Qwen2-2B (ZS/CoT)
64.4 66.6
24.1
Qwen2-7BB BB (ZS/CoT)
63.3 / 74.4
43.8 / 59.0
Qwen2-72B (7S/CoT)
89.4 83.5
77.2/
GPT-4V (ZS/CoT)
75.3 1/3/772 77.7
41.8 / 64.0
GPT-4o (ZS/CoT)
78.1 / 89.5
50.3 / 79.5
Gemini (ZS/CoT)
63.6 / 64.8
66.8 / 74.0
Average (ZS/CoT)
69.3
71.6
46.8 55.6
Table 11: Performance of LVLMs entity position QA. LVLMs can capture part of the position information and struggle with entity identification and reasoning.
Results. LVLMs start to struggle with identifying the entity's 's position attribute (Tab. 11). Even with CoT prompting, the average score of NR questions is 77. 73%,, which is worse compared to the entity text recognition accuracy in Tab. 2 (i.e., 95.02%). Since LVLMs only identify the position partially, the average accuracy on NC questions (i.e., 72.50% with CoT prompting) is also worse than that of text entity (i.e., 98.46%), where Gemini performs much worse than the other two models.
GPT-4V (CoT)
GPT-4o (CoT)
Gemini (CoT)
GPT-4V (CoT)
GPT-4o (CoT)
Gemini (CoT)
Accuracy
Accuracy
0.8
0.8
0.6
0.6
0.4
TopRow CenterRow BottomRow Row LeftColumn Center ConterColumn ColumColumn
TopRow
Center BottomRow Row
LeftColum Center olumn Right RolumColumn Column
(K)
(b) Qs(V(KF,Nc) NC)
Figure 5: Accuracies of LVLMs on @s(V KF, NR) and (V(K/KF, NC) with entities located in different positions (top row, center row, bottom row, left column, center column, and right column).
Analysis. We further analyze the results for more insights by visualizing the performance of LVLMs with entities in different compartments. Results (Fig. 5) show that LVLMs can answer NR and NC questions much better if the entities are not in the center area, where this phenomenon more obvious for two GPT models.
F.2.3. CONSISTENCY: DIAGRAM VARIATION
Preparation. We change the relation attributes, i.e., the arrow features in synthetic diagrams, to see if LVLMs can understand relations better. Specifically, we randomly change the arrowhead size to its 1.5, or 2 times, change the line width to its 0.5, 2, or 4 times, and the arrow color to black, red, or blue. Other settings remain the same as in $ 2.3. Then, we ask the same questions on these newly generated diagrams to observe how performance changes. For simplicity, we only consider the CoT prompting setting since achieves better results. The prompting templates as well as demonstration examples are shown Figs. 21 and 22.
22

---

Do Vision-Language Models Really Understand Visual Language?
Vanilla
New-Arrows
Vanilla
New-Arrows
80
80
Accuracy (%
60
60
40
40
20
20
GPT-4V
GPT-40
Gemini
Average
GPT-4V
GPT-40
Gemini
Average
(a) QN, [KF NR)
(b) Q(E, [KF,NC].
Figure 6: Accuracies of LVLMs the diagrams with modified arrow features (denoted by New-Arrows). New results are consistent with our previous findings (denoted by Vanilla) as in $ 2.3.
Results. We visualize the results in Fig. 6 comparing with the results in Tab. 3. We find that changing the relation attributes does not yet improve the accuracy of QA for both NR questions and NC questions. The maximum improvement is only 1%, while the average accuracies remain roughly the same (1.74% lower for NR questions and 0.3% higher for NC questions). These results further support that our findings on relations are valid and that LVLMs indeed do not understand relations even with different types of relations.
F.2.4. CONSISTENCY: PROMPT VARIATION
Preparation. We follow the settings in Li et al. (2024b) construct ICL prompts. We randomly select 4 examples and concatenate these diagrams as well questions and answers as few-shot examples (represented by image). Then, we modify the template to adapt to the ICL. examples under the setting of CoT prompting for evaluation. See Figs. 23 and 24 for the prompting templates and demonstration examples.
CoT
CoT with ICL
CoT CoT with ICL
80
80
Accuracy (%)
60
60
40
40
20
20
0
GPT-4V
GPT-40
Gemini
Average
GPT-4V
GPT-40
Gemini
Average
(a) QE [KF, NR)
(b) Q(E, NC))
Figure 7: Accuracies of LVLMs with ICL examples (with CoT prompting). Results are also consistent with our previous findings in $ 2..3.
We observe that ICL does not help with the relation identification (Fig. 7a) and reasoning (Fig. 7b) at all. Overall the average scores decrease, dropping 3% for NR questions and dropping 6% on NC questions. The findings further support that LVLMs can neither identify nor reason about relations even when provided few examples with answers in context.
Results.
F.3. Real Diagram
We provide the supplementary results for evaluations on real diagrams.
23

---

Do Vision-Language Models Really Understand Visual Language?
F. PRIOR KNOWLEDGE DOES NoT HELP
Accuracy (%) of GPT-40 (CoT)
Original
No-Prior
100
80
60
40
20
(s(E)KENR) (s(E|KENC) (R(VIKENR)
0
QR(ViKRAR)
CR(ViKENC)
QR(V(KRNC) QR(EİKENR) QR(E(KR,R) (R(E)KENC) (R(E(KR,NC)
Figure 8: Test accuracies of GPT-4o under the CoT settings on both synthetic and real diagrams for the original system message (i.e., "Original"" and no-prior knowledge-requin system message (i.e., "No-Prior"), Results show that asking the model do prior knowledge could not help the model better perform tasks.
To test if the prior knowledge in the model affect the performance, we adjust our system prompts to clearly let the model ignore its prior knowledge when answering the questions. The original system message (i.e., prompt instructions) well the new are listed below.
Original system message: "You are visual assistant answering multiple choice questions about diagrams. Read the question, inspect the diagram, and answer with the correct choice in the following format: 'A) 0?."
New system message that asks the model to ignore its prior knowledge: "You are visual assistant answering multiple choice questions about diagrams. Read the question, only inspect the diagram but do not use your prior knowledge, and answer with the correct choice the following format: 'A) 0"."
test GPT-4o model with these two prompts under the CoT setting. Fig. presents the test accuracies on both system messages are more or less the same, while our original one can achieve slightly better overall performance.
F.3.2. DIAGRAM DISTRIBUTION ON ENTITY NUMBER
198
200
182
Entity Number
99
100
90
77
65
62
42
42
15
18
14
10
10
6
2
0
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22 23 24
Figure 9: The distribution of the number of diagrams with respect to the number of entities in them.
For our 1, 001 real diagrams, we annotate diagrams with the number of entities in them. Thus, we visualize the distribution Fig. 9. We can find that it similar to the long-tail distribution, and most diagrams have 3 10 entities.
24

---

Do Vision-Language Models Really Understand Visual Language?
0.8
Accuracy
0.8
Accuracy
0.6
0.6
GPT-4V (CoT)
0.4
GPT-4V (CoT)
GPT-4o (CoT)
GPT-4o (CoT)
0.4
Gemini (CoT)
0.2
Gemini (CoT)
4
5-6
7-8
9-10
2
11
4
5-6
7-8
9-10
2
11
|
|
(a) (R(VVKK, KR, NR)
(b) QR(VQR((K))))) NC)
1
0.8
Accuracy
0.8
Accuracy
0.6
0.6
GPT-4V (CoT)
0.4
GPT-4V (CoT)
GPT-4o (CoT)
GPT44 (CrT)
0.4
Gemini (CoT)
0.2
Gemini (CoT)
5-6
7-8
9-10
2
5-6
7-8
9-10
2
M
|
(c) QI((VKRR NR)
(d) QR(( Q NC)
0.8
0.8
Accuracy
Accuracy
0.6
0.6
GPT-4V (CoT)
0.4
GPT-4V (CoT)
GPT-4o (CoT)
GPT-4o (CoT)
0.4
Gemini (CoT)
0.2
Gemini (CoT)
<
4
5-6
7-8
9-10
2 11
4
5-6
7-8
9-10
2 11
M
|
(e) (R(E)
(1) (R(K),,
0.8
0.8
Accuracy
Accuracy
0.6
0.6
GPT-4V (CoT)
0.4
GPT-4V (CoT)
GPT-4o (CoT)
GPT40 (CrT)
0.4
Gemini (CoT)
0.2
Gemini (CoT)
<4 4
5-6
7-8
9-10
2
<4 4
5-6
7-8
9-10
2
|v
|
(g) (R(ERRR NR)
(h) (E((E NC)
Figure 10: Accuracies of LVLMs on all questions with respect the number of entities in the diagram.
25

---

Do Vision-Language Models Really Understand Visual Language?
F.3.3. SUPPLEMENTARY RESULTS FOR STATISTICAL ANALYSIS
We provide all the detailed accuracies for three models on QA with respect the number of entities (Fig. 10), Generally, all these three models have similar tendencies, and the accuracy tendencies similar to their average in Fig. 2.
G. Examples of Prompts and Responses
G.1. Synthetic Diagrams
G.1.1. ENTITY
Prompting templates and demonstration examples for text entities and visual entities are shown in Figs. 11 to 16.
G.1.2. RELATION
Prompting templates and demonstration examples of explicit relations (i.e., arrows) and implicit relations Ep (i.e., spatial relations) shown in Figs. 17 to 20.
G.1.3. DIAGRAM GENERATION
Prompting templates and demonstration examples of synthetic diagrams with different arrow features are shown in Figs. 21 and 22.
1.1.4. IN-CONTEXT LEARNING (ICL) PROMPTING TEMPLATE
Four-shot ICL Prompting templates and demonstration examples are shown in Figs. 23 and 24.
G.1.5. DIAGRAMS WITH SEMANTIC KNOWLEDGE
Prompting templates and demonstration examples of synthetic diagrams containing semantic knowledge shown in Figs. 25 and 26.
G.2. Real Diagrams
Prompting templates and demonstration examples of real diagrams (Astronomy Solar System; Satellite System) for different question types are shown in Figs. 27 to 34.
26

---

Do Vision-Language Models Really Understand Visual Language?
(s(V(KF,N NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which one of the entities exists in the diagram? A) bus B) hexagon C) horse D) piggy bank
piggy bank
motorcycle
chicken
pine cone
desk
circle
The diagram contains labels for various objects. I will look for the entities listed in the multiple- choice options within the diagram.
GPT-4V:
A) bus - This label is not present in the diagram.
B) hexagon There no label for a hexagon in the diagram.
C) horse The label "horse" is not visible in the diagram.
D) piggy bank The label "piggy bank" is present in the top left corner of diagram.
The correct answer is:
D) piggy bank
GPT-4o: To determine which entity exists in the diagram, will examine each option and check present in the diagram.
A) bus There is mention of bus in the diagram.
B) hexagon There is no mention of hexagon in the diagram.
C) horse There is no mention of a horse in the diagram. D) piggy bank The term "piggy bank" is present in the diagram.
Therefore, the correct answer is:
D) piggy bank
Gemini 1.5 Pro: The diagram contains the word "chicken". The answer is: C) chicken.
Figure 11: An existence question example text entity.
27

---

Do Vision-Language Models Really Understand Visual Language?
@s(V/KF (F,NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many text labels there in the diagram? 2 B) 3 C) D
turkey
bird
chair
GPT-4V: To determine the number of labels in the diagram, will count each instance of text:
"turkey"
2. "bird"
3. "'cair"
There are three text labels in total.
The correct answer is: B)
GPT-4o: To determine the number of text labels in the diagram, let's count each distinct text label present:
"turkey"
"bird"
"chair"
There are three text labels in the diagram. Therefore, the correct answer is: B) 3
Gemini 1.5 Pro: The text labels in this image are "turkey", "bird", and "chair". There are 3 text labels. **C) 3**
Figure 12: A count question example for text entity.
28

---

Do Vision-Language Models Really Understand Visual Language?
Qs(V(KF, NR)
Question Prompt: Think step by step before answering the question and show your reasoning, Which one the entities exists in the diagram? A) lion B) tree house C) fence D) hay bale
GPT-4V:
To answer the question, will inspect each item in the diagram and compare it to the entities listed
in the options:
-A) lion: There no image of lion in the diagram.
(B) tree house: The image in the top right corner resembles structure with multiple windows and roof,
which could be interpreted as tree house.
C) fence: There is no image of a fence in the diagram.
D) hay bale: There is no image that resembles hay bale in the diagram.
Based on the inspection, the entity that exists in the diagram that most closely resembles one of the options
the tree house.
B) tree house
To determine which entity exists diagram, let's examine each option and compare images present in the diagram:
GPT-4o:
A) Lion There is image of lion in the diagram.
B) Tree house There is no image of tree house in the diagram.
C) Fence There is no image of a fence in the diagram.
D) Hay bale There is no image of a hay bale in the diagram.
The images present the diagram are: 1. Sushi
A cauldron
A building
A table tennis paddle and ball
None of these images correspond lion, tree house, fence, hay bale. Therefore, of the entities listed in the options exist in the diagram.
Gemini 1.5 Pro: The image contains fence. C) fence.
Figure 13: An existence question example for the visual entity.
29

---

Do Vision-Language Models Really Understand Visual Language?
Qs(vIKF, NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many icons are there in the diagram? A) B) 7 C) D)
DBDE
t
GPT-4V:
To determine the number of icons in the diagram, will count each distinct icon: The first icon at the top left appears be a loaf of bread.
The second icon at the top middle a keyboard.
3. The third icon at the top right is shopping cart with books.
4. The fourth icon at the bottom left is bag of coffee beans.
5. The fifth icon at the bottom middle is pair of tongs.
6. The sixth icon at the bottom right is pair of flip-flops.
7. The seventh icon in the middle a sign with arrows pointing up.
There are a total of icons in the diagram. The сoтrect answer is: B)
GPT-4o: To determine the number of icons the diagram, let's count each distinct icon:
Bread icon (top left)
2. Keyboard icon (top center)
3. Book icon (top right)
4. Milk and cookies icon (bottom left)
5. Fork icon (center)
6. Road sign icon (bottom center) Sandals icon (bottom right)
There are distinct icons in the diagram. Therefore, the correct answer is: B 7
Gemini 1.5 Pro: Let's count the icons in the image: bread, keyboard, book, fork, sign, sandals, milk and cookies. There 8 icons. As is not option, the closest one is 9.
Figure 14: A count question example for the visual entity.
30

---

Do Vision-Language Models Really Understand Visual Language?
@s(V|KF,NR NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which one of the text labels exists in the bottom row of the diagram? A) pool ice skate C) juice D) ruler
ice skate
nulor
paintbrush
juice
To find text label that exists in the bottom row of the diagram, will look at the labels positioned at the bottom the diagram. There are three labels in the bottom row: "paintbrush." " "juice," and "blanket." Now, will compare these labels with the options provided: A) pool This label is in the top row of the diagram.
GPT-4V:
B) ice skate This label is also in the top row of the diagram.
C) juice- This label is indeed in the bottom row of the diagram.
D) ruler - This label is in the middle row of the diagram.
The correct answer is the one that matches a label the bottom row, which is:
juice
GPT-4o: To determine which text label exists in the bottom row of the diagram, let's examine the positions of the labels:
The diagram is divided into grid with three rows.
2. The bottom row contains three labels: "paintbrush," "juice," and "blanket." Now, let's match these labels with the options provided:
A) pool This label is in the top row.
B) ice skate - This label is in the top row.
juice- - This label is in the bottom row. D) ruler - This label is in the middle row. Therefore, the correct answer is: juice
The text labels the bottom row are "paintbrush", "juice", and "blanket". Of these, "blanket" one of the choices. So the answer is D) ruler.
Gemini 1,5 Pro:
Figure 15: An existence question example to evaluate the position of entities.
31

---

Do Vision-Language Models Really Understand Visual Language?
@s(V/KF, KF,
Question Prompt: Think step by step before answering the question and show your reasoning. How many text
labels are there in the top row of the diagram? A) B) C) D)
candy
paper
castle
baby
stamp
To determine the number of text labels in the top row of the diagram, will inspect the top row from left to right.
GPT-4V:
-The first label on the left is "candy."
The second label in the middle is "paper."
There is no third label in the top row. Therefore, there are two text labels the top row of the diagram.
2
GPT-4o:
To determine the number of text labels in the top row of the diagram, follow these steps: Identify the top row the diagram. The top row is the uppermost horizontal section of the grid.
Count the text labels within this top row. the following text labels:
In the top row, '"cand""
"paper"
"castle"
There are three text labels in the top row. Therefore, the correct answer is: A) 3
Gemini 1.5 Pro: The top row contains the words "candy", "paper", and "castle". There are 3 words in the top row. A.
Figure 16: A count question example to evaluate the position of entities.
32

---

Do Vision-Language Models Really Understand Visual Language?
@S(EIKF KF, NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which one of the text labels placed on the right of the entity fishing pole? A) hummingbird B) nail C) van D) square
fishin pole
finger puppet
square
fork
To find text label that is placed on the right of the entity labeled "fishing pole," will look at the diagram and locate the "fishing pole" label. Then, I will identify which label directly its right. Upon inspecting the diagram, see that the "fishing pole" label is located towards the left side of the image. To the right of the "fishing pole" label, there is the label "van." Therefore, the correct answer is: C) van
GPT-4V:
GPT-4o: To determine which text label is placed to the right of the entity "fishing pole." let's analyze the diagram step by step:
1. Locate the "fishing pole" label in the diagram. It is positioned towards the left side of the diagram.
2. Identify the labels that are to the right of the "fishing pole" label.
From the diagram: "van" above and the right of "fishing pole."
"finger puppet" directly the right of "fishing pole."
"fork" is further to the right and slightly below "fishing pole."
"square" is below and to the left of "fishing pole."
Now, let's match the options with the labels: A) hummingbird Not present in the diagram.
B) nail Not present in the diagram.
C) van Present in the diagram, above and to the right "fishing pole."
D) square Present in the diagram, but the left of "fishing pole."
The correct answer is: C) van
Gemini 1.5 Pro: The text label "finger puppet" on the right of the entity "fishing pole". The answer **D) **
Figure 17: An existence question example from the Spatial Relation subtask.
33

---

Do Vision-Language Models Really Understand Visual Language?
@s(E KF, NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many text
labels are placed the left of the entity seal? A) B) 0 C) 3 D)
hackey ball
To determine how many text labels are placed on the left of the entity "seal," will look the positions of the text labels relative the word "seal" in the diagram.
GPT-4V:
1. The text label "hockey ball"" is placed above and to the left of the word "seal."
2. The text label "sauce" is placed below the word "seal" and does not appear to be on the left right, but rather directly below. Based on the positions of the text labels, there is only one text label ("hockey ball") that is placed on the left of the entity "seal."
D)
GPT-4o: To determine how many text labels are placed on the left of the entity "seal," let's examine the diagram step by step:
Identify the position of the entity "seal" in the diagram.
2. Look for any text labels that are positioned the left of "seal."
From the diagram:
"seal" is located towards the left side of the diagram.
The text labels "hockey ball" and "sauce" are both positioned to the right of "seal."
There are no text labels to the left of "seal."
Therefore, the correct answer is: 0
Gemini 1.5 Pro: The text label "seal" is on the left the entity "seal". There are no other text labels on the left of "seal". Therefore, the answer is D) 1. D)
Figure 18: A count question example from the Spatial Relation subtask.
34

---

Do Vision-Language Models Really Understand Visual Language?
Qs(E E KF, N
Think step by step before answering the question and show your reasoning. Which one of the pairs are connected the diagram? A) gorilla and sandwich B) penguin and sandwich C) award ribbon and sandwich D) award ribbon and penguin
Question Prompt:
penguin
gorilla
award
sandwich
GPT-4V:
To determine which pairs are connected in the diagram, let's look at each pair in the multiple-choice
options:
A) gorilla and sandwich There no direct line connecting the gorilla and the sandwich in the diagram.
B) penguin and sandwich There is no direct line connecting the penguin and the sandwich in the diagram.
C) award ribbon and sandwich There is no direct line connecting the award ribbon and the sandwich the diagram.
D) award ribbon and penguin There is direct line connecting the award ribbon and the penguin in the diagram.
Based on the connections shown in the diagram, the correct answer is:
D) award ribbon and penguin
To determine which pairs are connected in the diagram, need to look at the arrows and see which items are directly linked by them.
GPT-4o:
**Gorilla and Sandwich**: There no direct arrow connecting "gorilla" and "sandwich".
**Penguin and Sa There is a direct arrow from "penguin" to "sandwich".
3. ** Award Ribbon and Sandwich**; There is no direct arrow connecting "award ribbon" and "sandwich".
A*wad Ribbon and Penguin **; There is direct arrow connecting "award ribbon" and "penguin".
Based the diagram, the correct answer is:
B) penguin and sandwich
Gemini 1.5 Pro: The award ribbon points the gorilla and the penguin points the gorilla. D) award ribbon and penguin
Figure 19: An existence question example from the Explicit Relation subtask.
35

---

Do Vision-Language Models Really Understand Visual Language?
@s(E KF, KF)
Question Prompt: Think step by step before answering the question and show your reasoning. How many entities connected ring? A) 6 B) C) D)
train
stop light
birthday candle
ring
free house
circle
window
penci
To determine how many entities are connected to "ring," will inspect the diagram and count the lines that connect directly the word "ring." Upon inspection, can see that there three lines connecting to "ring." These lines connect "ring" to "stop light," "circle," and "pencil."
GPT-4V:
Therefore, the correct answer is: 3
To determine how many entities are connected to "ring," we need to count the number of arrows pointing to or from "ring."
GPT-4o:
There is an arrow connecting "ring" to "stop light."
There is an arrow connecting "ring" to "circle."
3. There is
connecting "ring" "pencil."
4. There is arrow connecting "ring" to "train."
5. There is an arrow connecting "ring" to "birthday candle."
6. There is an arrow connecting "ring" to "tree house."
So, there are entities connected to "ring."
The correct answer is: A) 6
Gemini 1.5 Pro: Ring is connected to stop light and pencil. Therefore, there are entities connected to ring.
**D) 2**
Figure 20: A count question example from the Explicit Relation subtask.
36

---

Do Vision-Language Models Really Understand Visual Language?
@s(E) KF, NR)
Question Trompt:Thini step by step before answering the question and show your reasoning. Which one of the pairs connected in the diagram? A) arrow and bird B) bird and domino C) arrow and river D) filing cabinet and river
river
domino
arrow
flinng cabinet
dog
dioe
GPT-4V: To determine which pairs connected in the diagram, let's look at each pair in the multiple-choice
options:
A) arrow and bird There is no line connecting the arrow to the bird.
B) bird and domino- - There is no line connecting the bird to the domino.
C) arrow and river There is a line connecting the arrow to the river.
D) filing cabinet and river There is no line connecting the filing cabinet to the river.
Based the connections shown in the diagram, the correct
is:
C) arrow and river
GPT-4o: To determine which pairs are connected the diagram, let's examine each pair one by one: A) Arrow and bird: The arrow labeled "arrow" is pointing towards the "dog". The "bird" is connected to the "domino". Therefore, the arrow and bird are not connected. B) Bird and domino: The "bird" is connected to the "domino" with a direct line. Therefore, the bird and domino are connected.
C) Arrow and river: The "arrow" is pointing towards the "dog". The "river" is not connected to the "arrow". Therefore, the arrow and river are not connected.
D) Filing cabinet and river: The "filing cabinet" connected the "dog" and "dice". The "river" connected the "filing cabinet".. Therefore, the filing cabinet and river not connected.
The correct answer is: B) bird and domino
Gemini 1.5 Pro: The arrow points the dog and the filing cabinet points the dog. The answer is D) filing cabinet and river.
Figure 21: An existence question example from the Explicit Relation subtask for different arrow feature.
37

---

Do Vision-Language Models Really Understand Visual Language?
@s(E KF, KF,NC)
Question Prompt:
Think step by step before answering the question and show your reasoning. How many entities
connected apricot? A) 4 B) 0 C) D)
apricot
sugar
parachute
bassball
To determine how many entities are connected "apricot," will count the lines that connect directly the word "apricot" in the diagram.
GPT-4V:
1. There is one line connecting "apricot" to "rubber duck."
2. There is another line connecting "apricot" to "sugar."
3. There is a third line connecting "apricot" to "parachute."
No other lines are connected to "apricot." " Therefore, there are three entities connected to "apricot." "
3
To determine how many entities are connected to "apricot," we need to count the number of arrows pointing to and from "apricot."
GPT-4o:
There is an arrow from "apricot" to "rubber duck."
2. There is
from "apricot" to "sugar."
3. There is arrow from "sugar" to "apricot."
4. There is an arrow from "parachute" to "apricot."
So, "apricot" is connected to four entities: "rubber "sugar, "sugar" (again, but counted once), and "parachute."
Therefore, the correct answer is: A) 4
Gemini 1.5 Pro: Apricot has connections to rubber duck, parachute, and sugar. There are 3 connections in total. Therefore, the answer is **C)3* 3*
Figure 22: A count question example from the Explicit Relation subtask for different arrow feature.
38

---

Do Vision-Language Models Really Understand Visual Language?
(E(E NR)
Question Prompt:
Following the in-context examples in the first image, answer the following question about the second image. Think step by step before answering the question and show your reasoning. Which one of the pairs are connected in the diagram? A) clock tower and flower B) swim fin and flower C) clock tower and piano D) flower and piano
In-Context Reference Examples
rable
somo
many are connected to finger?
Question: Which of
Question: many
Question:
Which
pains are connected in the diagram?
are connected to caterpillar?
pairs are connected in the diagram?
A)
A)
dack and
stake square
C)
3
B) clock and doorway
CI
snowilake and bug.
D)
4
() balloon and kitlen
D)
rose and bag.
balloon dourway
Answer: B) clock and doorway
Answer: C) rose and bug
To answer the question, I will examine the second image to determine which pairs of items are connected by lines.
GPT-4V:
The "clock tower" connected the "piano" by line.
Based these observations, the correct answer is: B) swim fin and flower
To determine which pairs connected in the diagram, let's examine the connections: **Clock to Connected to #*piano**, Since we need to find the correct pair, the answer is: B) Swim fin and flower
GPT-4o:
Gemini 1.5 Pro: The swim fin and clock tower are connected by a line. The answer is **C) clock tower and *
Figure 23; existence question example from the Explicit Relation subtask with few-shot examples.
39

---

Do Vision-Language Models Really Understand Visual Language?
@5(E) (F,NC)
Following the in-context examples in the first image, answer the following question about the second image. Think step by step before answering the question and show your reasoning. How many entities are connected to rectangle? A) 0 B) 3 C) 2 D)
Question Prompt:
In-Context Reference Examples
mont hee!
raller
deer
Question: How many entities connected finger?
Question: Which one of the pairs connected in the diagram?
Question: How many entities to caterpillar?
Which one of the pairs connected the diagram?
Question:
A)
B) 2
A} clock and kitten
B)
A] snake and square
B) clock and doorway
snowllake and bug.
C) balloon and kitten
D)
rose and bug
Answer: D)
D) balloon and doorway
Answer: B)
D) snake and racooon
clock and doorway
worge
In the second image, the entity labeled "rectangle" is connected two other entities: "pie" and "motorcycle." Therefore, Thereff,,, the correct answer to the question is: 2)
GPT-4V:
GPT-4o: To determine how many entities are connected to the rectangle in the second image, follow these
steps:
1. Identify the rectangle in the diagram.
2. Count the number arrows pointing to or from the rectangle.
Upon inspection: The rectangle has two arrows pointing to it from "motorcycle."
Therefore, the number of entities connected to the rectangle is 2.
The correct answer is: C) 2
Gemini 1.5 Pro: The entities connected rectangle are motorcycle and salad. Motorcycle is connected with single line, and salad is connected with line that goes through motorcycle. Therefore, there #42** entities connected to rectangle.
**C) 2**
Figure 24: count question example from Explicit Relation subtask with few-shot examples.
40

---

Do Vision-Language Models Really Understand Visual Language?
@ KR,NR)
Question Prompt:Think step by step before answering question and show your reasoning. Which one of the pairs are connected in the diagram? A) butterfly and skunk B) bug and skunk C) snake and goldfish D) caterpillar and snake
butterfly
snake
caterpillar
bug
goldfiih
skunk
GPT-4V:
To determine which pair of words are connected in the diagram, will look at each pair in the multiple-choice options and see if there is a line connecting them in the diagram. A) butterfly and skunk There no line directly connecting 'butterfly' and 'skunk' in the diagram.
B) bug and skunk There is no line directly connecting "bug' and 'skunk' in the diagram.
C) snake and goldfish There is no line directly connecting 'snake' and 'goldfish" in the diagram.
D) caterpillar and snake There is a line directly connecting 'aterpillar' and 'snake' in the diagram.
The correct answer is:
D) caterpillar and snake
To determine which pair is connected in the diagram, we need to follow the arrows between the pairs mentioned in the options.
GPT-4o:
A) Butterfly and skunk: There is no direct arrow connecting butterfly and skunk.
B) Bug and skunk: There is direct arrow connecting bug and skunk.
C) Snake and goldfish: There is direct arrow connecting snake and goldfish.
D) Caterpillar and snake: There is direct arrow from caterpillar to snake. Therefore, the correct answer is: D) caterpillar and snake
Gemini 1.5 Pro: The caterpillar has arrow pointing to the butterfly. The butterfly has arrow pointing to the snake. This means the caterpillar and snake connected. D) caterpillar and snake **D) caterpillar and **
Figure 25: An existence question example for a diagram generated from knowledge graph.

---

Do Vision-Language Models Really Understand Visual Language?
@s(E) KR,NK NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many entities
are connected to doll? A) 2 B) 0 1 3
oylinder
rectangle
doll
toy
To determine how many entities are connected "doll," let's look the diagram: There is line connecting "doll" to "toy." There other lines connecting "doll" to any other entities in the diagram. Therefore, "doll" is connected to only one entity.
GPT-4V:
The correct answer is: C)
GPT-4o: To determine how many entities are connected to "doll," let's examine the diagram step by step: 1. Identify the entities in the diagram: "cylinder," "rectangle," "doll," and "toy." 2. Look for arrows pointing to or from "doll."
From the diagram: There is one arrow pointing from "toy" to "doll."
Therefore, "doll" connected one entity.
The correct answer is: C)
Gemini 1.5 Pro: Doll is connected to toy. There an arrow pointing from toy to doll. Therefore, the answer I.
Figure 26: A count question example for a diagram generated from knowledge graph.
42

---

Do Vision-Languea Models Really Understand Visual Language?
QR(V/KR NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which astronomical object is in the diagram? A) earth B) moon C) belt D) jupiter
The diagram shows a central yellow sphere representing the Sun and blue-green sphere in four different positions around it, representing the Earth at various points in its orbit. The diagram includes labels for the equinoxes and solstices, which are events that occur due the Earth's tilt and orbit around the Sun. The Earth is depicted with tilt, and the different positions show how the sunlight hits the Earth differently throughout the year, leading to the changing seasons. Given the options provided and the context of the diagram... The correct answer is: A) earth
GPT-4V:
GPT-4o:
To determine which astronomical object is in the diagram, let's analyze the elements present:
The diagram shows a central yellow object, which likely the Sun. 2. Surrounding the Sun, there are four blue and green spheres with landmasses that resemble continents on Earth. The diagram labels specific dates: March 20 21 (equinox), June 20 21 (solstice), September 22 or 23 (equinox), and December 21 or 22 (solstice). These dates correspond to significant points in Earth's 's orbit around the Sun, marking the equinoxes and solstices.
Given these observations, the object in the diagram is Earth. Therefore, the correct answer is: A) earth
Gemini 1.5 Pro: The diagram shows the earth revolving around the sun. So the answer is A)
Figure 27: knowledge-rqquired entity recognition question example for a diagram.
43

---

Do Vision-Language Models Really Understand Visual Language?
CH(V/KR,NG
Question Prompt: Think step by step before answering the question and show your reasoning. How many planets or satellites are in the diagram? A) 3 B) 8 C) 10 4
Ongilnes
31
km))
The diagram shows four distinct celestial bodies: one is the Sun, and the other three are repre- sentations of Earth at different times of the year, indicating the position of Earth relative to the Sun during the equinoxes and solstices. These are not three different planets satellites, but rather one planet, Earth, shown at three different points its orbit around the Sun. Therefore, the answer to how many planets or satellites in the diagram is: 4
GPT-4V:
GPT-4o: To determine the number of planets satellites in the diagram, let's carefully analyze the image
step by step:
1. **Identify the objects in the diagram:** There is central yellow object, which represents the Sun. There are four blue and green objects surrounding the Sun, which represent the Earth at different positions in its orbit around the Sun.
**Count the distinct objects:** The Sun one object. The Earth is shown in four different positions, but it is still the same planet.
3. **Summarize the c Total number of distinct objects: 1 (Sun) (Earth) 2.
Since the question asks for the number of planets satellites, and we only have the Earth (a planet) and the Sun (which is star, not a planet or satellite), the correct count is planet. Therefore, the correct answer is: D) 4
Gemini 1.5 Pro: There are four images of the Earth and one of the Sun. The Earth is planet and the Sun is a star. A) 4
Figure 28: A knowledge-required entity reasoning question example for real diagram.
44

---

Do Vision-Languee Models Really Understand Visual Language?
QR(E NR)
Question Prompt: Think step by step before answering the question and show your reasoning. What is the next stage after the earth in summer in the diagram? A) full moon B) perigee C) earth fall D) earth in june
Supander
The diagram shows the Earth at different points in its orbit around the Sun, corresponding the seasons. The Earth in summer is labeled "June 20 21 (solstice)." Following the direction of the arrows, the next stage after the Earth in summer would be the Earth in fall. The Earth in fall is labeled "September 22 or 23 (equinox)." Therefore, the correct answer is: C) earth in fall
GPT-4V:
To determine the next stage after the Earth in summer in the diagram, let's follow these steps: Identify the current position the Earth in summer. The diagram shows that the Earth summer is labeled as "June or 21 (solstice)". 2. Follow the direction of the arrows to see the sequence of the Earth's positions around the Sun. 3. The arrow from "June 20 or 21 (solstice)" points to "September 22 or 23 (equinox)".
GPT-4o:
Therefore, the next stage after the Earth in summer is "earth in fall".
The correct answer is: C) earth in fall
Gemini 1.5 Pro: The diagram shows the earth different points in its orbit around the sun. The earth in summer (June 20 or 21) is followed by the earth in fall (September 22 or 23). **C) earth in fall**
Figure 29: knowledge-fin relation recognition question example for real diagram.
45

---

Do Vision-Language Models Really Understand Visual Language?
QR(EIKR, NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many times that the planets satellites change in the diagram? A) B) 12 C)
8
Dnvelee (e
The diagram shows the Earth four different points in its orbit around the Sun, corresponding to the equinoxes and solstices. These are the positions where the tilt of Earth's 's axis is oriented such that the sun's rays are directly over the equator (equinoxes) or over the Tropic of Cancer or Capricorn (solstices), leading to the start the different seasons.
GPT-4V:
The question seems to be asking how many times the planets satellites can change positions in the diagram. Since the diagram is specifically showing Earth's position at four distinct times of the year, the planets or satellites (in this case, Earth) change positions four times as it moves through its orbit from equinox solstice to the next.
Therefore, the answer is: C) 4
GPT-4o: To determine how many times the planets or satellites can change in the diagram, let's analyze the diagram step by step:
The diagram shows the Earth at four different positions in its orbit around the Sun, corresponding to the solstices and equinoxes. 2. The positions are: March or 21 (equinox) June 20 or 21 (solstice) September or 23 (equinox) December 22 (solstice) 3. The arrows indicate the transition from one position to the next.
Each transition from position the next represents change. There four positions, and each position transitions to the next:
March to June June to September September to December- December to March
There are 4 transitions in total.
Therefore, the number times the planets or satellites can change in the diagram is:
C) 4
Gemini 1.5 Pro: The diagram shows the Earth at four different points in its orbit around the sun. Therefore, the planets or satellites in the diagram can change 4 times. **C) 4**
Figure 30: A reguuiiree-rrurredd relation reasoning question example for real diagram.
46

---

Do Vision-Language Models Really Understand Visual Language?
QR(V NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which entity is in the diagram? A) earth B) belt C) jupiter D) moon
Onglanes
The diagram shows a central yellow sphere representing the Sun and four different positions of a blue and green sphere representing planet as orbits around the Sun. The four positions correspond to different dates and are labeled with specific astronomical events: two equinoxes (March 20 or 21 and September 22 23) and solstices (June 20 21 and December 21 22)...
GPT-4V:
Given the options provided:
A) Earth This is the correct answer because the diagram depicts Earth's orbit around the Sun and the changing seasons, which are specific to Earth.
B) Belt - This is not correct; there is no belt depicted in the diagram.
C) Jupiter- This not correct; the diagram does not represent Jupiter's orbit or its relation to the Sun.
D) Moon This not correct; the diagram does not show the Moon or phases.
The correct answer is:
A) Earth
GPT-4o: To determine which entity is in the diagram, let's analyze the elements present:
1. The diagram shows central yellow circle, which represents the Sun. Surrounding the Sun, there four blue and green spheres with landmasses that resemble Earth's continents. 3. The labels indicate specific dates related to equinoxes and solstices, which are events that occur due to Earth's orbit around the Sun. Given these observations, the entity in the diagram is: A) earth
Gemini 1.5 Pro: The diagram shows the earth at different points in its orbit around the sun. **A)**
Figure 31: knowledge-fiee free entity recognition question example real diagram.
47

---

Do Vision-Language Models Really Understand Visual Language?
QR(VIKF,NC) NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many entities are the diagram? A) 15 B) 5 C) 12 2
Nuck2forzl
Jebenn
To determine the number of entities in the diagram, let's count each distinct item: 1. The Sun (in the center) 2. Earth at the top (March equinox) 3. Earth the right (December solstice) 4. Earth at the bottom (September equinox) 5. Earth on the left (June solstice) Adding these up. we have a total of entities.
GPT-4V:
The correct answer is: B) 5
To determine the number of entities in the diagram, let's identify and count each distinct object: The Sun in the center. 2. Four Earth positions around the Sun, each representing different times of the year (March equinox, June solstice, September equinox, December solstice). So, we have: Sun Earth positions Adding these together, we get total of entities.
GPT-4o:
Therefore, the correct answer is: 5
Gemini 1.5 Pro: There 4 Earth illustrations and Sun illustration. 4+ 1 = **B) 5*
Figure 32: A knowledge-free entity reasoning question example for real diagram.
48

---

Do Vision-Language Models Really Understand Visual Language?
(R(E NR)
Question Prompt: Think step by step before answering the question and show your reasoning. Which entity is connected to the june in the diagram? A) waning crescent B) summer C) waxing crescent D) september
Orpannal
Decedle vi Jelliio
The diagram shows the Earth at four key points in its orbit around the Sun, corresponding to the equinoxes and solstices. The date "June 20 or 21" is labeled with "(solstice)," which refers to the summer solstice in the Northern Hemisphere. This is the time when the Northern Hemisphere tilted most directly towards the Sun and experiences the longest day of the year, marking the beginning of summer. Therefore, the entity connected June in the diagram is: B) summer
GPT-4V:
GPT-4o: To determine which entity is connected June in the diagram, let's analyze the diagram step by
step:
1. The diagram shows the Earth's 's position relative to the Sun at different times of the year.
2. The positions are labeled with specific dates and events...
3. The question asks for the entity connected to June. In the diagram, June 20 or 21 is labeled as a solstice.
The solstice in June is known as the summer solstice in the Northern Hemisphere, marking the beginning of summer.
Now, let's match this information with the given options:
A) waning crescent This is phase of the moon, not related to the solstice.
B) summer This is correct, as the June solstice marks the beginning of summer in the Northern Hemisphere.
C) waxing crescent This is phase of the moon, not related to the solstice.
D) September This is another month, not directly connected to June. Therefore, the correct answer is: B) summer
Gemini 1.5 Pro: The arrow points from June to September.
D) september**
Figure 33: knowledge-free relation recognition question example real diagram.
49

---

Do Vision-Language Models Really Understand Visual Language?
QR(E|KF, NC)
Question Prompt: Think step by step before answering the question and show your reasoning. How many arrows are in the diagram? A) 4 B) C) 13 5
Ongllnal
31
GPT-4V:
To determine the number of arrows in the diagram, will count each arrow that is depicted. Starting from the top right and moving clockwise: 1. There is one large arrow pointing from the globe labeled "December
Another or 22 (solstice)" the globe labeled "March 20 or (equinox)". large arrow points from the globe labeled "March 20 or 21 (equinox)" the globe labeled "June 20 21 (solstice)". 3. A third large arrow points from the globe labeled "June 20 21 (solstice)" to the globe labeled "September 22 or 23 (equinox)". 4. Finally, fourth large arrow points from the globe labeled "September 22 or 23 (equinox)" back to the globe labeled "December 21 or 22 (solstice)). There are no other arrows visible in the diagram. Therefore, the total count is four arrows. The correct answer is: A) 4
To determine the number of arrows in the diagram, let's carefully count each arrow step by step: Starting from the top Earth (March 20 or 21), there is an arrow pointing to the right towards the Earth labeled December 21 or 22. From the Earth labeled December 21 or 22, there an arrow pointing downwards towards the Earth labeled September or 23. From the Earth labeled September 22 23, there is an arrow pointing to the left towards the Earth labeled June 20 21. From the Earth labeled June 20 or 21, there is arrow pointing upwards towards the Earth labeled March 20 21. So, there are total of arrows in the diagram. The correct answer is: A) 4
GPT-4o:
Gemini 1.5 Pro: There is one arrow connecting each globe, There are four globes. Therefore, there are four
arrows, *) 4#*
Figure 34: knowledge-free relation reasoning question example for real diagram.
50


Reviews:
[{"review_id": "WosX26MuSg", "reviewer": "Reviewer_rNT4", "summary": "## update after rebuttal\n\nThe paper presents a comprehensive and detailed study on if LVLMs are able to understand visual diagrams. The authors develop a test suite of synthetic as well as real diagrams and test multiple LVLMs on it. The authors find that the models are able to identify identities well. However, they have limited abilities in recognizing relations, especially synthetic ones and the ones without background knowledge as shortcuts. This suggests that existing LVLMs have limited abilities on diagram reasoning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for acknowledging the clarity and robustness of our work. Below, we provide our responses to the concerns and questions raised. Additionally, in line with the suggestions from Reviewer 8S38 and Reviewer 3mRA, we have expanded our evaluation model set to include 9 models, and the results remain consistent, further reinforcing the validity of our findings.\n\n--- \n\n>**Point 1**: *the scope of the problem covered is limited, … if the author can discuss about more general and complex diagrams, …*\n\n>**Response**: We agree with the reviewer that expanding the range of diagram types discussed could be valuable for the community. In our work, we design two evaluation settings: one with synthetic data and another with real-world data. For the synthetic evaluation, we focus on a typical diagram type (entity-relation) to ensure clarity and intuitiveness. For the real-world evaluation, we gather data from six domains, incorporating various diagram types from existing works (Figure 4 and Table 8 in Appendix D provide additional details). While there is potential to include more general or complex diagram types, our work is constrained by the availability of existing diagram datasets. Nonetheless, we have aimed to cover as diverse a range of diagram types as possible.\n--- \n>**Question 1**: *... Llava performs very poorly on the benchmark compared with other models. However, I noticed that its benchmark score is similar to Llama and Qwen's. Any thoughts on this?*\n\n>**Response**: We appreciate the reviewer for carefully reviewing our paper. Indeed, the performance of LLaVA is significantly lower than that of the other models. We believe this can be attributed to differences in the training patterns. Unlike the other models, LLaVA is primarily trained with real-world images, while the other models incorporate augmented synthetic data and reinforcement learning techniques, such as DPO, which likely enhance their generalizability. We hypothesize that these additional training phases contribute to the stronger performance observed in these models on our benchmark.\n\n--- \n\n>**Question 2**: *... can the models do much better on \"relation\" with some SFT with your synthetic data?*\n\n>**Response**: Although SFT with our data could likely improve the evaluation accuracy on our benchmark, we do not believe it would lead to true diagram understanding ability or generalizability in practice. In our work, our primary focus is to highlight the weaknesses of existing Vision-Language Models (VLMs). Addressing how to improve these models is a broader topic, and we encourage future work to explore potential solutions.", "claims_and_evidence": "Yes the paper has clearly presents all the claims, together with clear evidence.", "methods_and_evaluation_criteria": "Yes, the evaluation makes sense for diagram understanding.", "theoretical_claims": "No theory claims.", "experimental_designs_or_analyses": "Yes, I checked each one and the paper is sound.", "relation_to_broader_scientific_literature": "There are many prior works that relate to diagram understanding in prior works on LVLM. For example, popular benchmarks like MMMU and ChartQA contain diagram understanding problems. The paper focuses on entity & relation and reveals that existing models still have a long way to go on recognizing and reasoning on relations.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is overall well-written and clearly structured. The conclusions are sound and interesting and reveal a particular weakness of existing LVLMs. The experiment design is interesting and informational. \n\nOne weakness is that the scope of the problem covered is limited, with only entities connected by relations. It would be nicer if the author can discuss about more general and complex diagrams, which would make bigger impact to the community.", "other_comments_or_suggestions": "page 4 line 172 on the right missing a space between \"detailsof\"", "questions_for_authors": "An interesting phenomenon I found in the numbers is that Llava performs very poorly on the benchmark compared with other models. However, I noticed that its benchmark score is similar to Llama and Qwen's. Any thoughts on this?\n\nAlso, can the models do much better on \"relation\" with some SFT with your synthetic data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "63ktxAYITY", "reviewer": "Reviewer_8S38", "summary": "This paper investigates the diagram comprehension capabilities of Large Vision-Language Models (LVLMs) by developing a comprehensive test suite across synthetic and real-world diagrams. The evaluation reveals that while LVLMs can accurately recognize entities, they struggle to understand relationships within diagrams. Further analysis suggests that their apparent reasoning ability is largely due to leveraging background knowledge as shortcuts, rather than genuine diagram understanding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our experimental design is comprehensive and convincing. Our responses to the proposed concerns are provided below.\n\n---\n\n>**Point 1**: *... raising the question of whether larger models (e.g., 72B) might exhibit improved relational reasoning and diagram perception....*\n\n>**Response**: We appreciate the constructive suggestions provided by the reviewer. We are pleased to report that we have now completed experiments on three additional models: Molmo-7B, Qwen2-VL-2B, and Qwen2-VL-72B. The results for these models have been incorporated into the paper (Tables 2, 3, 4, 5, 6, 7, 9, and 11). Overall, the trends and conclusions remain consistent (Molmo performs similarly to LLaMA, and Qwen-2B and Qwen-72B perform similarly to Qwen-7B). We will briefly present some of their scores here (Tables 2, 3, and 4).\n\n\n>**Table 2 (entity understanding on synthetic diagrams)**\n>| Acc (%)  | Text Entity (recognition) | Text Entity (counting) | Visual Entity (recognition) | Visual Entity (counting) |\n>|-------------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|\n>| LLaVA       | 38.9| 26.6| 46.4  | 30.8|\n>| Molmo       | 93.4| 78.8| 64.1 | 54.0 |\n>| LLaMA       | 91.3| 90.9| 72.7| 70.1|\n>| Qwen-2B     | 82.3 | 73.2 | 63.1| 53.5 |\n>| Qwen-7B     | 97.6 | 73.0 | 94.5  | 73.0 |\n>| Qwen-72B    | 99.1  | 97.9 | 90.6  | 86.4|\n>| GPT-4V      | 97.8| 99.6 | 85.7 | 93.7  |\n>| GPT-4o      | 99.2 | 100.0 | 92.6| 94.9 |\n>| Gemini      | 88.1| 95.8 | 87.7| 86.5|\n\n\n\n\n\n>**Table 3 (relation understanding on synthetic diagrams)**\n>| Acc (%)  | Implicit Relation (recognition) | Implicit Relation (counting) | Explicit Relation (recognition) | Explicit Relation (counting) |\n>|-------------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|\n>| LLaVA       | 30.2                           | 27.5                           | 35.1                           | 28.3                           |\n>| Molmo       | 71.7                           | 31.2                           | 59.1                           | 50.4                           |\n>| LLaMA       | 75.4                           | 32.0                           | 55.2                           | 46.1                           |\n>| Qwen-2B    | 63.3                           | 29.8                           | 44.0                           | 33.1                           |\n>| Qwen-7B    | 74.4                           | 59.0                           | 59.8                           | 51.5                           |\n>| Qwen-72B   | 77.9                           | 67.1                           | 70.3                           | 63.8                           |\n>| GPT-4V      | 72.3                           | 34.4                           | 61.6                           | 59.5                           |\n>| GPT-4o      | 77.3                           | 55.3                           | 76.6                           | 70.2                           |\n>| Gemini      | 60.9                           | 31.8                           | 68.5                           | 70.2                           |\n\n\n>**Table 4 (entity and relation understanding on real diagrams)**\n>| Model      | Entity (recognition) | Entity (counting) | Relation (recognition) | Relation (counting) |\n>|-----------|-------------|-------------|--------------|--------------|\n>| LLaVA  | 56.5        | 37.3        | 45.1         | 30.2         |\n>| Molmo  | 82.7        | 54.9        | 59.8         | 51.2         |\n>| LLaMA  | 87.3        | 59.7        | 73.7         | 51.2         |\n>| Qwen-2B  | 66.6    | 40.7        | 45.7         | 39.3         |\n>| Qwen-7B  | 90.0    | 56.1        | 71.4         | 58.0         |\n>| Qwen-72B | 93.7    | 77.8        | 79.4         | 69.8         |\n>| GPT-4V | 88.9        | 78.8        | 78.7         | 59.9         |\n>| GPT-4o | 93.1        | 82.3        | 84.1         | 72.9         |\n>| Gemini | 85.0        | 68.4        | 80.5         | 57.7         |\n\n---\n\n>**Point 2**: *There are more complex relationships that could be explored, such as logical, hierarchical, or causal relationships, …*\n\n>**Response**: While our synthetic diagrams may not include complex relationships, our real-world diagram dataset covers a wide range of relationship types (more details can be found in Figure 4 and Table 8 in Appendix D). Unfortunately, for rarer cases, we are constrained by the limitations of existing datasets, but we have included as much diversity in the relationships as possible.\n\n---\n\n>**Point 3**: *However, the authors fail to provide a direct comparison with these works. For example:...*\n\n>**Response**: We sincerely appreciate the reviewer for pointing out these relevant works. We will include them in our related work section and provide a discussion on them.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence. \n\n1.  LVLMs can accurately recognize entities, they struggle to understand relationships within diagrams: supported by Tables 2-3.\n2. Reasoning ability is largely due to leveraging background knowledge as shortcuts:supported by Tables 4-7.", "methods_and_evaluation_criteria": "This paper investigates the diagram comprehension capabilities of Large Vision-Language Models (LVLMs) by developing a comprehensive test suite across synthetic and real-world diagrams. The evaluation reveals that while LVLMs can accurately recognize entities, their ability to understand relationships is notably limited, often relying on background knowledge as shortcuts rather than genuinely perceiving diagram structures. However, the paper does not clearly specify the scale of evaluated models, raising the question of whether larger models (e.g., 72B) might exhibit improved relational reasoning and diagram perception or if these limitations persist—or even worsen—with increased model size. Addressing this would clarify whether scaling alone mitigates these issues or if fundamental architectural improvements are needed for genuine diagram understanding.", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "The experimental design is comprehensive and easy to follow, effectively supporting the observations and final conclusions. However, the relationships tested in both synthetic and real-world images are primarily spatial and follow arrow-based dependencies. There are more complex relationships that could be explored, such as logical, hierarchical, or causal relationships, to provide a more holistic evaluation of LVLMs’ diagram comprehension capabilities.", "relation_to_broader_scientific_literature": "Refer to Section of Essential References Not Discussed.", "essential_references_not_discussed": "Exploring the visual limitations of LVLMs is not a new topic, as many previous studies have addressed this issue. However, the authors fail to provide a direct comparison with these works. For example:\n\n[1] Hulingxiao He et. al., ANALYZING AND BOOSTING THE POWER OF FINEGRAINED VISUAL RECOGNITION FOR MULTI-MODAL LARGE LANGUAGE MODELS, ICLR 2025.\n\n [2] Shengbang Tong et. al., Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024. \n\n[3] Shengbang Tong et. al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024. [4] Xingyu Fu et. al., BLINK: Multimodal Large Language Models Can See but Not Perceive, ECCV 2024.", "other_strengths_and_weaknesses": "This paper contributes to diagram understanding of LVLMs, like visual rasoning and shortcut learning. The findings suggest that existing LVLMs lack true diagram understanding, opening avenues for research in improving relational reasoning in LVLMs through better graph-based representations. However, additional experiments are needed to make these conclusions more convincing: Evaluating larger models to determine whether scaling mitigates these limitations; Expanding benchmarks to test more complex, non-spatial relationships beyond arrows and spatial positioning.", "other_comments_or_suggestions": "Refer to Section of Other Strengths And Weaknesses.", "questions_for_authors": "I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJkyv3iGe5", "reviewer": "Reviewer_3mRA", "summary": "The paper introduces a new benchmark dataset for vision-language models. The benchmark is based on diagram understanding. Unlike most existing diagram understanding datasets, the dataset introduced in this work is specifically designed to study the understanding of relationships between depicted entities by using graphs whose vertices are entities (eg., objects) connected with lines (edges in the graph). The paper studies the performance of existing vision-language models on various question types and subsets (eg. real vs. synthetic images / entities vs. relations / edges random vs. semantic / etc.). The paper uses the results as evidence to argue that the existing models do not truly understand diagrams but that they use short-cuts to answer questions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very positive feedback. We have carefully designed our experiments to ensure they are comprehensive, and our conclusions are as convincing as possible. We will address the proposed concern about evaluation model set below.\n\n>**Concern Point**: *Not a lot of models are evaluated...*\n\n>**Response**: We are grateful for this constructive suggestion. As mentioned in our response to Reviewer 8S38, we have added three new models—Molmo-7B, Qwen2-VL-2B, and Qwen2-VL-72B—to all of our experiments. Our model set now includes three groups with 9 models: large models (GPT-4V, GPT-4o, and Gemini), small models (LLaVA-7B, Molmo-7B, LLaMA-11B), and models with three scales (Qwen2-VL with 2B, 7B, and 72B versions). The performance trends and our conclusions remain consistent, and we believe this expanded model set enhances the reliability of our conclusions.", "claims_and_evidence": "The claims are supported by clear evidence and are convincing to me.", "methods_and_evaluation_criteria": "The paper introduces a benchmark dataset itself. The paper uses the performance of models on this data to derive the main hypothesis in this paper, which is that existing vision-language models use shortcuts rather than truly understanding visual information in diagrams.", "theoretical_claims": "N/A \n\nThis is an empirical study.", "experimental_designs_or_analyses": "The experiments are fairly comprehensive and seem sound overall. The evidence brought forth in support of the paper's hypothesis is derived from multiple different performance evaluations on variations of the images and questions. \n\nThe paper uses a mix of graph images generated from synthetic and real imagery to support the hypothesis that real imagery is more amenable to learning shortcuts by leveraging background knowledge as opposed to the information derived from the image itself.", "relation_to_broader_scientific_literature": "All the relevant similar existing work is discussed in a Related Works section located in the Appendix. This is a bit strange and unconventional, but probably Ok.", "essential_references_not_discussed": "I am not an expert on recent diagram understanding benchmarks, but the presented related works section seems reasonably comprehensive.", "other_strengths_and_weaknesses": "The paper is very well written, and very well-structured, and exceptionally easy to follow as a result. \n\nNot a lot of models are evaluated (Llama, Llava, Qwen2 and three closed cloud models). As this is a benchmark paper, I would say this is the bare minimum. Considering the huge number of vision-language models that have been released in recent times, it would be great if the sample of tested models was larger, and if the results held up, that would increase the confidence in the hypothesis brought forth in this work. \n\nThe use of word2vec to generate synthetic relations to test for the use of background knowledge seems like a very nice and clever idea. \n\nThe results on knowledge inclusion are convincing enough in my opinion, but not as clear-cut as one would hope after reading the earlier parts of the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MY2IMxCVpS", "reviewer": "Reviewer_en7J", "summary": "This paper argues that LVLM's good performance in diagram reasoning mainly comes from pre-existing background knowledge rather than a genuine understanding of the diagrams' relational structure. \n\nThe authors develop a test suite specifically to assess diagram comprehension:\n- Their evaluations focus on two main tasks: fast recognition and slow, multi-step reasoning.\n- Their evaluations are not only performed on synthetic diagrams but also real diagrams. \n- Their evaluations are based on open-source models as well as large accessible models.\n\nAfter showing their experiments and analysis, the authors conclude that the apparent models can perfectly recognize and reason about entities in the diagrams, while they struggle with recognizing the relations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our claims are clear and convincing. We will address the proposed concerns and answer the questions below. Additionally, we are pleased to inform you that we have extended our evaluation model set to include 9 models, as suggested by Reviewer 8S38 and Reviewer 3mRA, with consistent conclusions, further enhancing the reliability of our work.\n\n---\n\n>**Point 1**: *Limited analysis in failure cases.*\n\n>**Response**: We agree with the reviewer that a more detailed failure case analysis would provide valuable insights into why the models fail. We would like to kindly remind the reviewer that Section 5 already includes a qualitative analysis. Additionally, we have provided examples, along with our prompts and the models' responses, for each experiment in the Appendix (Figures 11–34). While these figures do not provide a thorough analysis for each case, they offer readers a clearer understanding of why the models fail by showcasing their responses. We will also emphasize this point in the main paper.\n\n---\n\n>**Question 1**: *Can the authors explain why using text labels performs better on QA in Table 2?*\n\n>**Response**: We compare the performance of entity recognition on synthetic diagrams, distinguishing between text entities and visual entities. The results show that models can perfectly identify text entities. We believe this is due to the models being specifically trained on numerous OCR datasets (as mentioned in their technical reports). As a result, their ability to recognize and extract text is significantly stronger compared to their ability to recognize visual elements. \n\n---\n\n>**Question 3**: *Did the Table 4 test on text labels, visual icons, or both?*\n\n>**Response**: Based on the results in Table 2, we observe that the model performs well in recognizing text entities. To ensure that our subsequent experiments—such as exploring the models' abilities in more complex tasks like counting and relation recognition—are not influenced by entity recognition errors, we default to representing entities in text for the remainder of our experiments.\n\n---\n\n>**Question 2 and 4**: *Can the authors provide performance on QA for relations using visual icons, like Table 3? For the synthetic diagrams, do they only have text entities or visual entities? Can the authors provide some results on synthetic diagrams with both text entities and visual entities?*\n\n>**Response**: We appreciate the thoughtful questions and thank the reviewer for carefully reading our paper. The underlying logic is that since models can accurately recognize text entities, any failure to recognize relations likely stems from a lack of relation recognition ability, rather than being bottlenecked by entity recognition. We understand the reviewer’s concern regarding entity representation. To address this, we report the performance of entity/relation recognition on synthetic diagrams with entity representations of text, visual, and both. Additionally, to better align with the results, we reformat the multi-choice QA into binary classification (i.e., with only one negative option).\n\n\n>| GPT-4V    | Precision (Visual) | Recall (Visual) | F1 (Visual) | Accuracy (Visual) | Precision (Text) | Recall (Text) | F1 (Text) | Accuracy (Text) | Precision (Both) | Recall (Both) | F1 (Both) | Accuracy (Both) \n>|-------------|-----------|-----------|----------|-----------|----------|----------|---------|----------|----------|----------|---------|----------|\n>| **Entity recognition**  | 97.37 | 84.52     | 89.54    | 91.12     | 99.86 | 99.75 | 99.78 | 99.79 | 99.86 | 99.75 | 99.78 | 99.79 |\n>| **Relation recognition**| 55.98 | 48.82     | 51.58    | 55.77 | 54.82     | 40.33     | 45.28    | 55.83     | 55.05     | 47.00     | 49.99    | 56.42     |\n\n>The results reveal two key observations. First, representing entities with both text and visual information yields similar performance to text-only entities, and the model still struggles with relation recognition regardless of entity representation. Second, the recall for visual entity recognition is relatively lower, indicating that the model cannot perfectly identify the correct visual entities, rather than misclassifying wrong entities as correct ones. We hope these updated results help address the reviewer’s concern regarding entity representation.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. The authors show details about how they set up each experiment.", "methods_and_evaluation_criteria": "The proposed evaluation criteria make sense.", "theoretical_claims": "Their theoretical claim (LVLMs reply on knowledge shortcuts when answering complex diagram reasoning questions) is supported by their detailed experiment designs and analyses.", "experimental_designs_or_analyses": "The main contribution in this paper is their evaluation suite, from data creation to experiment implementations. \n\nBoth synthetic diagrams and real diagrams make sense. However, please answer my questions about the synthetic diagrams in the following parts.\n\nThe experimental design begins with simpler tasks—such as identifying entities and relationships in synthetic diagrams—and gradually progresses to more complex tasks involving real diagrams. Additionally, the authors conduct both quantitative and qualitative analyses on knowledge shortcut experiments, demonstrating that LVLMs may not truly understand diagrams but instead rely on ingrained shortcuts to generate answers.", "relation_to_broader_scientific_literature": "The key contributions of the paper shows the concerns about the LVLMs in complex diagram reasoning tasks, which is related to the findings of current LVLMs abilities in reasoning both in 2D and 3D images.", "essential_references_not_discussed": "The essential references are discussed by the authors in their supplementary materials.", "other_strengths_and_weaknesses": "Paper Strengths:\n- The paper is well-structured, and the topic is interesting.\n- The authors show the detailed implementation for each experiment.\n- The authors argue their claims logically,  from synthetic diagrams experiments to real diagrams experiments. \n\nMajor Weaknesses (please also refer to the **Questions For Authors**):\n- Limited analysis in failure cases. \n- Missing some details key settings.", "other_comments_or_suggestions": "Here are some minor suggestions:\n-  I highly suggest the authors could highlight some abbreviations, such as NR and NC.\n- L181, seems like you only have one **Intuition**, why do you label it as **Intuition** 1.", "questions_for_authors": "I have some questions regarding this paper:\n- Can the authors explain why using text labels performs better on QA in Table 2?\n- Can the authors provide performance on QA for relations using visual icons, like Table 3?\n- Did the Table 4 test on text labels, visual icons, or both?\n- For the synthetic diagrams, do they only have text entities or visual entities? Can the authors provide some results on synthetic diagrams with both text entities and visual entities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark for evaluating VLM performance on diagram comprehension, including both synthetic and real-world diagrams. Analysis on the benchmark with existing VLMs show that they mostly fail to represent relational information about entities in diagrams, and that current success in this task is mostly due to background knowledge rather than genuinely understanding diagram contents. The benchmark could be a useful addition to existing work in studying the limitations of modern VLMs, although as pointed out by a couple of reviewers, the scope of diagrams included in the dataset is relatively limited, focusing on relations between entities. To the extent that ICML is welcoming benchmark/dataset/analysis papers, I think this is a relatively solid contribution.
Decision: Accept (poster)