Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
Authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang
Accepted at: acl25

Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

Yingjie Zhu1,2, Xuefeng Bai1*, Kehai Chen1,2, Yang Xiang2*, Jun Yu3, Min Zhang1,2
1Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China
2Peng Cheng Laboratory, Shenzhen, China
3School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China
{baixuefeng,chenkehai,yujun,zhangmin2021}@hit.edu.cn
zhuyj@stu.hit.edu.cn, xiangy@pcl.ac.cn

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across diverse tasks. Despite great success, recent studies show that LVLMs encounter substantial limitations when engaging with visual graphs. To study the reason behind these limitations, we propose VGCURE, a comprehensive benchmark covering 22 tasks for examining the fundamental graph understanding and reasoning capacities of LVLMs. Extensive evaluations conducted on 14 LVLMs reveal that LVLMs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. Based on this observation, we propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks. Experiments validate the effectiveness of our method in improving LVLMs’ performance on fundamental and downstream graph learning tasks, as well as enhancing their robustness against complex visual graphs.1

1 Introduction

Graphs serve as a fundamental data structure across a wide range of domains, including social network analysis (Schweimer et al., 2022), recommendation systems (Zhang et al., 2023), knowledge graphs (Zhang et al., 2024b), chemistry (Cao et al., 2024), biomedical molecules (Liu et al., 2023) and semantic reasoning (Bai et al., 2022). Existing methods have achieved great success in enhancing understanding and reasoning abilities in graph-based tasks (Kim et al., 2023a; Chen et al., 2024). However, these approaches typically focus on specific graph types or tasks, posing challenges in designing versatile systems that are suitable for various tasks and graphs across diverse domains.

Recently, Large Vision-Language Models (LVLMs) have exhibited outstanding performance across a wide range of downstream tasks by unifying various inputs in the form of images and processing them with human-like understanding and reasoning abilities (Zhu et al., 2024; Zheng et al., 2024; Zhang et al., 2025). This triggers a growing interest in employing LVLMs for graph learning problems, as the vision modality offers a natural and intuitive way for comprehending structural information and facilitating general graph reasoning (Poklukar et al., 2022). However, recent studies (Wei et al., 2024; Li et al., 2024c; Ai et al., 2024) reveal significant challenges for LVLMs in graph-based learning tasks, where LVLMs achieve less than 15% accuracy on mathematical graph reasoning tasks, markedly below their performance in image and text reasoning (Li et al., 2024c). Therefore, a natural challenge arises: why do LVLMs fail in graph learning, and how to enhance LVLMs to process graphs like professionals?

To address this challenge, we begin by identi-

![Overall of our VGCURE benchmark.](page_1012_370_388_246.png)

Figure 1: Overall of our VGCURE benchmark.
<table>
  <tr>
    <th>Benchmark</th>
    <th>Evaluation Topic</th>
    <th># Tasks</th>
    <th>Graph Type</th>
    <th>Anonymity</th>
    <th># Scale</th>
  </tr>
  <tr>
    <td>VisionGraph</td>
    <td>Graph Theory Problems</td>
    <td>10</td>
    <td>Synthetic</td>
    <td>✓</td>
    <td>3,000</td>
  </tr>
  <tr>
    <td>Ai et al. (2024)</td>
    <td>Multi-hop Reasoning</td>
    <td>N/A</td>
    <td>Real-world</td>
    <td>✗</td>
    <td>1,355</td>
  </tr>
  <tr>
    <td>GVQA</td>
    <td>Graph Theory Problems</td>
    <td>7</td>
    <td>Synthetic</td>
    <td>✓</td>
    <td>157,896</td>
  </tr>
  <tr>
    <td><b>our VGCURe</b></td>
    <td>Fundamental Graph Understanding and Reasoning</td>
    <td>22</td>
    <td>Synthetic & Real-world</td>
    <td>✓</td>
    <td>223,646</td>
  </tr>
</table>

Table 1: Comparisons among different visual graph analysis benchmarks for LVLMs, where # Tasks and # Scale represent the number of task types and test samples, respectively.

fying a research gap in existing evaluations that limit our understanding of LVLMs. Current work primarily focuses on graph theory problems or multi-hop reasoning tasks (Wei et al., 2024; Ai et al., 2024), which are complex and require a combination of diverse cognitive abilities. Thus it remains uncertain whether LVLMs have strong fundamental understanding and reasoning capabilities in processing visual graphs—such as recognizing the basic components of a graph and making basic logical inferences based on graph data—which is crucial for determining whether the limitations stem from a lack of fundamental abilities or other higher-order capabilities.

To this end, we present the Vision Graph Comprehensive Understanding and REasoning benchmark, VGCURe, designed to thoroughly evaluate the fundamental understanding and reasoning capabilities of LVLMs on visual graphs. As shown in Fig.1, VGCURe evaluates the fundamental capabilities of LVLMs diverse challenges, including 9 graph understanding tasks and 13 graph reasoning tasks. Moreover, VGCURe features 10 anonymized graph structures from both synthetic and real-world sources, offering a robust testbed for assessing LVLMs’ proficiency in handling diverse graphs. Experiments on 14 representative LVLMs show that current LVLMs exhibit weak fundamental understanding and reasoning capabilities on visual graphs, especially in capturing relational information and dealing with structurally complex visual graphs.

Motivated by the above observations, we further introduce MCDGRAPH, a novel structure-aware fine-tuning framework to enhance structure learning capabilities of LVLMs through three self-supervised learning tasks: 1) masked graph infilling, 2) contrastive graph discrimination, and 3) graph description. Experiments show that MCDGRAPH significantly improves LVLMs’ graph understanding and reasoning abilities, especially on edge-related tasks and graph-related downstream tasks. Further analysis demonstrates that our method also enhances LVLMs’ robustness and generalization to visual graphs with complex structure and unseen styles. The contributions of this work can be summarized as follows:
• We introduce VGCURe, a comprehensive benchmark to systematically evaluate LVLMs’ fundamental understanding and reasoning abilities on visual graphs.
• Through extensive experiments on 14 LVLMs, we reveal LVLMs’ limitations in basic graph understanding and reasoning, especially for tasks concerning relational or structural information.
• We propose a self-supervised framework to enhance LVLMs’ ability to capture structural information in visual graphs. Experiments validate its effectiveness on both fundamental and downstream graph learning tasks.

2 The VGCURe Benchmark

To evaluate LVLMs’ fundamental graph understanding and reasoning capabilities, we introduce VGCURe, a large-scale multimodal graph benchmark with 22 challenging tasks. VGCURe features 10 graph types, both synthetic and real-world, to assess LVLMs’ performance on diverse graphs. The graphs are anonymized to minimize the impact of pre-existing LLM knowledge on core reasoning abilities, promoting knowledge-free reasoning (Hu et al., 2024). Tab.1 compares VGCURe with three recent benchmarks. It is evident that VGCURe excels in fundamental graph understanding and reasoning capabilities. Furthermore, VGCURe offers a comprehensive evaluation through more varied graph types, tasks, and test samples.

2.1 Graph Structure Generation

We begin by collecting a wide variety of graph structures for generating visual graphs and challenging tasks. Following Fatemi et al. (2024), we first employ NetworkX (Hagberg et al., 2008) to generate a diverse set of random synthetic structures, including Erdős–Rényi (ER) graphs (Erdős and Rényi, 1959), scale-free networks (SFN) (Barabási and Albert, 1999), Barabási–Albert (BA) model (Albert and Barabási, 2002), stochastic
<table>
  <tr>
    <th>Task</th>
    <th>QA sample</th>
    <th>FC sample (Label)</th>
  </tr>
  <tr>
    <td>NNu</td>
    <td>Q: How many nodes are there in this graph?<br>A: 12</td>
    <td>There are 12 nodes in this graph. (True)<br>There are 17 nodes in this graph. (False)</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>Q: How many edges are there in this graph?<br>A: 15</td>
    <td>There are 15 edges in this graph. (True)<br>There are 17 edges in this graph. (True)</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>-</td>
    <td>This graph is a directed graph. (True)<br>This graph is an undirected graph. (True)</td>
  </tr>
  <tr>
    <td>DQ</td>
    <td>Q: What is the degree of E9 in this graph?<br>A: 1</td>
    <td>The degree of E9 in this graph is 1. (True)<br>The degree of E9 in this graph is 2. (False)</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>Q: Which nodes are out-neighbors of E6 in this graph?<br>A: [E3]</td>
    <td>E3 is a out-neighbors of E6 in this graph. (True)<br>E7 is a out-neighbors of E6 in this graph. (False)</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>Q: Which entities are R6 of the entity that is R5 of E3?<br>A: [E4, E7]</td>
    <td>E4 is R6 of the entity that is R5 of E3. (True)<br>E1 is R6 of the entity that is R5 of E3. (False)</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>Q: Which entities are R4 of E9 as well as R8 of E1?<br>A: [E2]</td>
    <td>E2 is R4 of E9 as well as R8 of E1. (True)<br>E1 is R4 of E9 as well as R8 of E1. (False)</td>
  </tr>
  <tr>
    <td>CN</td>
    <td>-</td>
    <td>E8 and E1 share a common out-neighbor, i.e., common head entity. (True)<br>E10 and E12 share a common out-neighbor, i.e., common head entity. (False)</td>
  </tr>
  <tr>
    <td>RA</td>
    <td>Q: Which entities are connected to E3 via the same relation from E3 to E1?<br>A: [E2]</td>
    <td>E2 is connected to E3 via the same relation from E3 to E1. (True)<br>E6 is connected to E3 via the same relation from E3 to E1. (False)</td>
  </tr>
  <tr>
    <td>SRN</td>
    <td>Q: Which entities are both R2 of E10?<br>A: [E4, E12]</td>
    <td>E4 and E12 are both R2 of E10. (True)<br>E4 and E3 are both R2 of E10. (False)</td>
  </tr>
  <tr>
    <td>NR</td>
    <td>Q: What is the relation from the entity that is R5 of E3 to E2?<br>A: [R8]</td>
    <td>E2 is R8 of the entity that is R5 of E3. (True)<br>E2 is R4 of the entity that is R5 of E3. (False)</td>
  </tr>
  <tr>
    <td>CP</td>
    <td>Q: Is there a path from E3 to E4?<br>A: Yes. The paths are [[E3, E1, E2, E4], [E3, E1, E4]]</td>
    <td>There are 2 paths from E3 to E4. (True)<br>There are 3 paths from E3 to E4. (False)</td>
  </tr>
</table>

Table 2: Examples for each task in VGCURE. These samples all correspond to the graph shown in Fig.7(d).

block model (SBM) (Holland et al., 1983) and star graphs. In addition, we extract anonymized structures from real-world graphs in GraphArena (Tang et al., 2024), including DBLP, Social Network, DBpedia, Openflights and PubChemQC. All the entity and relation names in each graph are replaced with generic names to eliminate the impact of the model’s internal knowledge on reasoning. After initializing the graph structure, we use the Graphviz (Ellson et al., 2002) to generate concise directed and undirected visual graphs.

2.2 Tasks Design

To thoroughly assess the abilities of LVLMs in fundamental graph structure understanding and reasoning, the proposed VGCURE encompasses the following categories of tasks. Tab.2 presents examples for each task.

Graph Understanding: The graph understanding tasks involve analyzing and extracting structural, relational, and property-based information from the visual graph, which aims to gain insights into the composition and topology of the graph, including its nodes, edges, connectivity, and the relationships among its components.
• Node Number Query (NNu): Calculate the total number of nodes in the graph.
• Edge Number Query (EN): Determine the total number of edges in the graph.
• Directedness Check (DC): Verify whether the graph is undirected or directed.
• Degree Query (DQ): Calculate the degree of the specified node.
• Neighbor Query (NQ): Identify the nodes that are directly connected to the given node.

Graph Reasoning: The reasoning tasks focus on exploring the knowledge-free reasoning ability of LVLMs on visual graphs. To differentiate from graph understanding tasks, we designed a series of 2-hop reasoning tasks.
• Nested Node Query (NN): Identify the entities linked to the given entity through a composite chain involving the given relations.
• Conjunctive Relation Query (CR): Retrieve the entities satisfying both two independent relationship constraints with two distinct entities.
• Common Neighbor Check (CN): Determine whether two entities share at least one common neighbor in a 2-hop relational path.
• Relation Analogy Query (RA): Find the entities linked to a target entity via a relation identical to that links a given entity pair.
• Shared Relation Neighbor Query (SRN): Identify the set of entities that are connected to the given entity through the given relation.
• Nested Relation Query (NR): Identify the relation between a target entity and an intermediate entity obtained by traversing a specific relation path from a given entity.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="6">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>DQ</th>
    <th>NQ</th>
    <th>CR</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <th></th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>EM_F1</th>
    <th>Label_Acc</th>
  </tr>
  <tr>
    <td>SPHINX</td>
    <td>21.03</td>
    <td>11.38</td>
    <td>15.45</td>
    <td>12.84</td>
    <td>29.45</td>
    <td>6.05</td>
    <td>9.81</td>
    <td>7.28</td>
    <td>16.44</td>
    <td>14.62</td>
    <td>21.69</td>
    <td>11.62</td>
    <td>26.93</td>
    <td>1.54</td>
    <td>4.99</td>
    <td>0.68</td>
    <td><b>95.59</b></td>
  </tr>
  <tr>
    <td>Monkey</td>
    <td>40.09</td>
    <td>9.22</td>
    <td>17.81</td>
    <td>9.90</td>
    <td>21.88</td>
    <td>8.90</td>
    <td>12.96</td>
    <td>2.85</td>
    <td>4.62</td>
    <td>3.74</td>
    <td>4.13</td>
    <td>9.79</td>
    <td>21.61</td>
    <td>3.06</td>
    <td>7.98</td>
    <td>0.46</td>
    <td>5.11</td>
  </tr>
  <tr>
    <td>MiniGPT-v2</td>
    <td>11.80</td>
    <td>10.61</td>
    <td>18.03</td>
    <td>16.40</td>
    <td>27.08</td>
    <td>8.92</td>
    <td>18.02</td>
    <td>2.01</td>
    <td>2.65</td>
    <td>7.50</td>
    <td>14.87</td>
    <td>9.85</td>
    <td>27.47</td>
    <td>5.86</td>
    <td>13.87</td>
    <td>5.26</td>
    <td><b>95.59</b></td>
  </tr>
  <tr>
    <td>mPLUG-Owl3</td>
    <td>28.38</td>
    <td>8.84</td>
    <td>6.86</td>
    <td>20.32</td>
    <td>51.16</td>
    <td>8.68</td>
    <td>11.39</td>
    <td>5.42</td>
    <td>11.81</td>
    <td>11.07</td>
    <td>14.81</td>
    <td>6.48</td>
    <td>18.06</td>
    <td>2.03</td>
    <td>0.33</td>
    <td>10.87</td>
    <td>74.97</td>
  </tr>
  <tr>
    <td>LLaVA1.5-7B</td>
    <td>14.53</td>
    <td>7.56</td>
    <td>30.14</td>
    <td>11.43</td>
    <td>21.25</td>
    <td>1.80</td>
    <td>0.21</td>
    <td>2.22</td>
    <td>0.13</td>
    <td>6.36</td>
    <td>9.37</td>
    <td>5.38</td>
    <td>7.73</td>
    <td>0.84</td>
    <td>2.23</td>
    <td>2.63</td>
    <td>78.64</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT</td>
    <td>47.89</td>
    <td>9.59</td>
    <td>19.52</td>
    <td>23.27</td>
    <td>44.11</td>
    <td>14.77</td>
    <td>21.74</td>
    <td>6.60</td>
    <td>12.44</td>
    <td>10.14</td>
    <td>15.29</td>
    <td>12.68</td>
    <td>27.88</td>
    <td>9.88</td>
    <td>6.46</td>
    <td>8.27</td>
    <td>95.56</td>
  </tr>
  <tr>
    <td>LLaVA-OV</td>
    <td>23.47</td>
    <td>3.08</td>
    <td>23.80</td>
    <td>15.00</td>
    <td>39.12</td>
    <td>10.74</td>
    <td>19.87</td>
    <td>7.50</td>
    <td>16.61</td>
    <td>9.85</td>
    <td>19.30</td>
    <td>10.21</td>
    <td>30.99</td>
    <td>1.04</td>
    <td>1.36</td>
    <td>9.69</td>
    <td>94.83</td>
  </tr>
  <tr>
    <td>LLaVA1.5-13B</td>
    <td>17.08</td>
    <td>7.62</td>
    <td>26.33</td>
    <td>15.68</td>
    <td>32.66</td>
    <td>6.08</td>
    <td>7.83</td>
    <td>4.31</td>
    <td>13.23</td>
    <td>6.52</td>
    <td>10.80</td>
    <td>7.48</td>
    <td>12.86</td>
    <td>4.21</td>
    <td>6.21</td>
    <td>5.91</td>
    <td>82.41</td>
  </tr>
  <tr>
    <td>InternLM-XC2.5</td>
    <td>60.20</td>
    <td>10.53</td>
    <td>41.12</td>
    <td>18.90</td>
    <td>53.33</td>
    <td>14.64</td>
    <td>26.14</td>
    <td>19.19</td>
    <td>45.60</td>
    <td>11.50</td>
    <td>19.06</td>
    <td>14.12</td>
    <td>46.53</td>
    <td>3.09</td>
    <td>5.49</td>
    <td>28.82</td>
    <td>95.53</td>
  </tr>
  <tr>
    <td>Llama3.2</td>
    <td>77.31</td>
    <td>9.39</td>
    <td>35.56</td>
    <td>18.79</td>
    <td>42.34</td>
    <td><b>18.93</b></td>
    <td>29.43</td>
    <td>21.90</td>
    <td>55.00</td>
    <td><b>15.80</b></td>
    <td>27.51</td>
    <td>20.79</td>
    <td>61.30</td>
    <td><b>14.11</b></td>
    <td><b>24.05</b></td>
    <td>20.80</td>
    <td>94.70</td>
  </tr>
  <tr>
    <td>Qwen-VL</td>
    <td>42.45</td>
    <td>9.66</td>
    <td>20.56</td>
    <td>10.95</td>
    <td>21.25</td>
    <td>11.44</td>
    <td>15.51</td>
    <td>7.93</td>
    <td>18.30</td>
    <td>12.48</td>
    <td>17.36</td>
    <td>11.11</td>
    <td>22.79</td>
    <td>5.43</td>
    <td>4.01</td>
    <td>0.00</td>
    <td>4.48</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td><b>97.80</b></td>
    <td><b>16.38</b></td>
    <td>48.09</td>
    <td>16.18</td>
    <td>38.12</td>
    <td>16.52</td>
    <td>28.34</td>
    <td>21.02</td>
    <td>48.57</td>
    <td>14.19</td>
    <td><b>27.96</b></td>
    <td>19.48</td>
    <td>56.42</td>
    <td>12.73</td>
    <td>25.07</td>
    <td>12.90</td>
    <td>38.06</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>77.45</td>
    <td>9.78</td>
    <td>50.75</td>
    <td>25.01</td>
    <td>68.58</td>
    <td>18.30</td>
    <td><b>30.82</b></td>
    <td>24.87</td>
    <td>59.31</td>
    <td>10.83</td>
    <td>17.12</td>
    <td>20.72</td>
    <td>59.99</td>
    <td>10.58</td>
    <td>18.97</td>
    <td>14.53</td>
    <td>43.97</td>
  </tr>
  <tr>
    <td>GPT4o-mini*</td>
    <td>89.20</td>
    <td>15.40</td>
    <td><b>56.40</b></td>
    <td>30.81</td>
    <td><b>77.40</b></td>
    <td>17.01</td>
    <td>29.98</td>
    <td>22.33</td>
    <td>53.15</td>
    <td>15.47</td>
    <td>24.81</td>
    <td><b>22.25</b></td>
    <td>59.48</td>
    <td>8.62</td>
    <td>13.48</td>
    <td><b>42.55</b></td>
    <td>92.40</td>
  </tr>
</table>

Table 3: Model performance on QA samples across various tasks, where EM_F1 is the macro F1 score calculated based on the exact match between the predicted path and the ground truth path. Label_Acc measures the accuracy of the model’s prediction on whether a path exists or not. The best results are bolded.

• Connective Path Query (CP): Determine the existence of directed paths or shortest undirected paths between two given entities, and retrieve all possible paths if they exist.

For each task, we construct one QA sample and two fact checking (FC) samples (with labels of True and False, respectively) automatically based on the template, except for CN and DC which have only fact checking samples due to the strong similarity between the two samples. The total number of final samples is 223,646. More details about the design logic and statistics of VGCURE can be found in Appendix A and Tab.9.

3 Benchmarking LVLMs on VGCURE

3.1 Experimental Setup

We conduct evaluation on 13 open-source LVLMs, including InternLM-XComposer2.5-7B (Zhang et al., 2024a), InternVL2-8B, Llama3.2-11B-Vision-Instruct, LLaVA1.5-7B (Liu et al., 2024), LLaVA1.5-13B (Liu et al., 2024), LLaVA-NeXT-7B (Li et al., 2024b), LLaVA-OneVision-7B (Li et al., 2024a), MiniGPT-v2 (Chen et al., 2023), Monkey (Li et al., 2024d), mPLUG-Owl3-7B (Ye et al., 2024), Qwen-VL (Bai et al., 2023), Qwen2-VL-7B-Instruct (Wang et al., 2024) and SPHINX (Lin et al., 2023). Meanwhile, we also evaluate the performance of the GPT-4o-mini, which is a strong closed-source LVLM2. Due to the high cost of GPT-4o-mini, we randomly select 50 graphs from each graph structure for testing. For all methods, the zero-shot setting is adopted during evaluation. More details can be found in Appendix B.

2We excluded GPT-4o as a baseline due to its high cost.

3.2 Main Result

Graph Understanding Tab.3 and Tab.4 present the evaluation results for question answering (QA) and fact checking (FC), respectively. We report the averaged results across various graph structures. In general, most LVLMs struggle to precisely understand the structural and relational information in visual graphs. In specific, (I) Among the graph understanding tasks, Node Number Query (NNu) and Directedness Check (DC) are the easiest for most LVLMs. This indicates that most LVLMs can accurately capture the number of nodes and directedness information within the visual graph. (II) All LVLMs struggle with Edge Number Query (EN) and Neighbor Query (NQ) tasks, with the highest accuracy of 16.38% and F1 score of 30.81% , respectively. This indicates that current LVLMs are weak in understanding relational and structural information. (III) Even with a similar number of parameters, the graph understanding abilities of open-source LVLMs vary significantly, with Qwen2-VL and InternVL2 showing better performance in both QA and FC samples. (IV) For the same task, LVLMs perform differently on QA and FC samples, likely due to different ways in reasoning and understanding required by each task (Thorne et al., 2018). (V) The closed-source LVLM, GPT-4o-mini, offers no significant advantages and even underperforms open-source LVLMs, especially on FC tasks.

Graph Reasoning Based on graph reasoning results in Tabs.3 and 4, it can be observed that, (I) Compared to graph understanding, the graph reasoning tasks are more challenging and the overall performance of LVLMs is worse on both
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="6">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>DC</th>
    <th>DQ</th>
    <th>NQ</th>
    <th>NN</th>
    <th>CR</th>
    <th>CN</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <td>SPHINX</td>
    <td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td>
  </tr>
  <tr>
    <td>Monkey</td>
    <td>40.31</td><td>52.23</td><td>51.38</td><td>55.49</td><td>64.25</td><td>65.24</td><td>50.51</td><td>55.62</td><td>44.04</td><td>53.29</td><td>50.54</td><td>50.76</td>
  </tr>
  <tr>
    <td>MiniGPT-v2</td>
    <td>34.77</td><td>50.38</td><td>34.28</td><td>50.39</td><td>33.92</td><td>50.18</td><td>33.69</td><td>49.91</td><td>44.29</td><td>50.99</td><td>49.68</td><td>50.03</td>
  </tr>
  <tr>
    <td>mBert-wwm-ext</td>
    <td>37.53</td><td>50.80</td><td>31.66</td><td>58.59</td><td>33.04</td><td>74.76</td><td>39.44</td><td>46.92</td><td>33.15</td><td>50.09</td><td>34.75</td><td>49.86</td>
  </tr>
  <tr>
    <td>LLaVA-L-7B</td>
    <td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td>
  </tr>
  <tr>
    <td>LLaVA-NEXT</td>
    <td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td><td>33.33</td><td>50.00</td>
  </tr>
  <tr>
    <td>LLaVA-OV</td>
    <td>39.13</td><td>52.69</td><td>33.79</td><td>49.68</td><td>65.02</td><td>68.49</td><td>36.71</td><td>51.49</td><td>37.12</td><td>51.70</td><td>46.83</td><td>53.37</td>
  </tr>
  <tr>
    <td>InternLM-XC2S</td>
    <td>60.31</td><td>63.26</td><td>50.78</td><td>57.09</td><td>87.74</td><td>87.92</td><td>33.33</td><td>50.00</td><td>36.39</td><td>49.31</td><td>47.65</td><td>50.31</td>
  </tr>
  <tr>
    <td>InternLM-XC2L</td>
    <td>37.19</td><td>51.79</td><td>40.00</td><td>51.05</td><td>65.76</td><td>66.72</td><td>43.07</td><td>49.16</td><td>64.46</td><td>65.40</td><td>50.81</td><td>54.05</td>
  </tr>
  <tr>
    <td>Llama2</td>
    <td>87.66</td><td>87.66</td><td>47.26</td><td>59.67</td><td>43.27</td><td>51.83</td><td>42.23</td><td>51.39</td><td>65.88</td><td>66.03</td><td>53.47</td><td>56.40</td>
  </tr>
  <tr>
    <td>Qwen-VL</td>
    <td>76.50</td><td>77.67</td><td>68.26</td><td>68.27</td><td>94.92</td><td>94.94</td><td>64.32</td><td>67.40</td><td>67.34</td><td>68.76</td><td>44.17</td><td>53.77</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>68.63</td><td>71.18</td><td>36.82</td><td>50.23</td><td>93.17</td><td>93.18</td><td>63.28</td><td>63.84</td><td>72.81</td><td>73.27</td><td>62.46</td><td>63.62</td>
  </tr>
  <tr>
    <td>GPT-4o-mini*</td>
    <td>64.49</td><td>67.00</td><td>39.83</td><td>52.30</td><td>90.52</td><td>90.60</td><td>42.03</td><td>52.90</td><td>48.77</td><td>56.80</td><td>50.77</td><td>51.51</td>
  </tr>
</table>

Table 4: Model performance on FC samples across various tasks. The best results are bolded.

![Model performance (F1/Acc) on representative QA samples across various graph structures and tasks, where OF, PCQC and SN denotes Openflights, PubChemQC and Social network, respectively.](page_1012_1012_420_312.png)

Figure 2: Model performance (F1/Acc) on representative QA samples across various graph structures and tasks , where OF, PCQC and SN denotes Openflights, PubChemQC and Social network, respectively.

QA and FC samples. This may be a knock-on effect due to deficiencies in visual graph understanding. (II) Among all open-source LVLMs, InternVL2 and Llama3.2-Vision achieve better performance and LLaVA1.5-7B perform the worst on graph reasoning tasks. (III) All the LVLMs perform poorly on Nested Relation Query (NR) for both QA and FC samples, which is similar to the observation in the graph understanding task. This suggests that LVLMs are deficient in recognizing edges and understanding structural information within visual graphs. (IV) On the QA samples, the performance of different LVLMs on Connective Path Query (CP) varies widely. SPHINX, Monkey and Qwen-VL demonstrate almost no ability to recognize paths between two specific nodes in the visual graph. (V) Similarly, GPT-4o-mini underperforms in most tasks compared to open-source LVLMs, except for the Connective Path Query (CP) task.

3.3 Impact of Structures

Inspired by Fatemi et al. (2024), we explore how graph structure affects LVLMs’ ability to understand and reason on visual graphs. Fig.2 compares the performance of the top five LVLMs on QA samples across various structures. Obviously, the graph structure significantly impacts LVLMs’ performance on most tasks. Notably, all LVLMs perform well on PCQC, which has a simpler structure with fewer nodes and edges (averaging 5.45 nodes and 4.76 edges), and weaker on BA, which has the highest edge count in VGCURE (averaging 21.02 edges). Furthermore, on different tasks, the performance of LVLMs is affected differently by the graph structure. EN and CP show larger performance variations across graph structures, whereas NNu and NN show smaller differences. More results are shown in Figs.12 and 13, the overall trends are similar to above findings.

3.4 Impact of Complexity

We further discuss the impact of graph complexity on the LVLMs’ ability to understand and reason over the visual graph by considering the number of edges, number of nodes, and average degree. Fig.3 shows a performance comparison of five LVLMs on QA samples across representative tasks and complexity levels. As complexity increases, LVLMs’ performance declines, especially on EN, where results vary significantly. Besides, some LVLMs perform best at intermediate complexity, but struggle with more complex graphs. This reflects a balance between information richness and complexity of visual graphs, whereas higher complexity likely overwhelm the LVLMs’ abilities to reason or generalize due to the complex information within large graphs. In addition, different complexity dimensions affect LVLMs’
Figure 3: Model performance (Acc) comparison on QA samples across various dimensions of complexity.

Figure 4: Results comparison between LVLM and LLM on QA samples across various tasks.

<table>
  <tr>
    <th>Error Type</th>
    <th>R.M.</th>
    <th>C.L.</th>
    <th>S.H.</th>
    <th>E.A.</th>
    <th>Ot.G.</th>
    <th>F.E.</th>
  </tr>
  <tr>
    <td>Percentage</td>
    <td>54%</td>
    <td>7%</td>
    <td>17%</td>
    <td>9%</td>
    <td>5%</td>
    <td>8%</td>
  </tr>
</table>

Table 5: Frequency of each error type.

performance on various tasks differently, such as \( NNu \), where LVLMs are more influenced by the number of nodes and average degree than by edges. More results in Figs.14-19 show the similar trend.

3.5 Why LVLMs Fail on Fundamental Tasks?
To explore in depth whether the failure of LVLMs on the fundamental task is due to their weak perception or other higher-order ability, we compare the performance of LVLMs and the corresponding backbone LLMs that take the textual graph as input. As Fig.4 shows, despite the disparity in the performance of the individual models, LVLMs perform weaker than the corresponding backbone LLMs on both understanding and reasoning tasks. In particular, LVLMs’ performance degrades the most on Edge Number Query (EN), which also illustrates the lack of ability of LVLMs to capture structural information in the visual graph. This indicates that the failure of LVLMs is partly attributed to their weaker visual graph perception. In addition, the performance of the backbone LLMs on these tasks remains undesired, with all accuracies below 45%, this suggests that the backbone LLMs exhibit limited capabilities in understanding and reasoning on graph data. More details and results are presented in Appendix C.1.

3.6 Error Analysis
In addition, by analyzing a batch of results generated by Qwen2-VL and InternVL2 on QA samples in VGCURE, we find that their main errors can be grouped into the following six categories:
• Relation Misunderstanding (R.M.): Failure to properly understand the relations between entities, leading to erroneous reasoning.
• Complexity Limitation (C.L.): When faced with intricate or highly complex graph structures, LVLMs struggle to process and understand the information effectively, often resulting in incomplete or erroneous outputs.
• Structural Hallucination (S.H.): Generating or perceiving structures that do not exist, leading to erroneous or misleading results that do not match the actual visual graph.
• Entity-based Answering (E.A): Directly using the entities mentioned in the question as answers, thus ignoring deeper relation understanding or logical reasoning in the visual graph.
• Off-target Generation (Ot.G.): Deviation from the task or misunderstanding of the question, leading to the generation of irrelevant answers.
• Format Error (F.E.): The output of the model is incorrectly formatted or unexpected.

Examples of each error are shown in the Tab.11.
Error Distribution Analysis We also calculate the overall distribution of each error. The results are shown in Tab.5. It can be observed that, (I) Relation Misunderstanding appears most frequently due to LVLMs’ limited capacity to effectively capture structural and relational information in the visual graph. (II) Structural Hallucination also appears frequently due to the inherent hallucination tendency of LLMs. (III) Despite demonstrating robust instruction-following capabilities, LVLMs remain prone to errors like Off-target Generation and Format Error. (IV) Complexity Limitation and Entity-based Answering also account for a
certain percentage, i.e., LVLMs may be stumped when faced with overly complex visual graphs, or directly use the entities mentioned in the question as answers.

Primary Error Pattern Analysis We further analyze the primary error pattern of each task. We find that, (I) Generally, the primary error occurring in all tasks except the Node Number Query is Relation Misunderstanding. (II) Structural Hallucination occurs more frequently in Nested Node Query, Conjunctive Relation Query, Relation Analogy Query, Shared Relation Neighbor Query, and Nested Relation Query tasks. (III) Format Error is usually found in Connective Path Query task.

4 The MCDGRAPH Framework

To enhance the ability of LVLMs to understand and reason on visual graphs, we propose MCDGRAPH, a self-supervised fine-tuning framework designed to improve LVLMs’ ability to capture structural and relational information within visual graphs. As illustrated in Fig.5, MCDGRAPH comprises three key tasks: Masked Graph Infilling, Contrastive Graph Discrimination, and Graph Description.

4.1 Task1: Masked Graph Infilling

For this task, we randomly mask either nodes or edges in a visual graph and challenge the model to predict the masked element based on the partially observed graph. Since anonymized visual graphs contain no semantic information, we also provide the corresponding text triples as input.

\[
M = \text{LVLMs}(\hat{G}, I, T),
\]

where \( \hat{G} \), \( I \), \( T \) denotes the masked graph, task instruction, and text triples of the original graph, respectively. This task encourages LVLMs to infer missing structure information, improving their ability to understand graph structure and the relationships between elements.

4.2 Task2: Contrastive Graph Discrimination

To further refine the LVLMs’ understanding of graph structure, we introduce a contrastive learning task, which helps train the LVLMs to distinguish between two visual graphs that may either represent the same graph with different layouts or two distinct graphs with similar layouts.

\[
Y = \text{LVLMs}(G_1, G_2, I),
\]

where the answer \( Y \in \{Yes, No\} \), \( G_1, G_2 \) denotes two graphs, and \( I \) is the task instruction. By learning how to perform structural reasoning and graphical isomorphism detection, this task aims to improve LVLMs by recognizing subtle structural differences between two visual graphs.

4.3 Task3: Graph Description

Graph Description task requires LVLMs to generate a textual description of a given visual graph, including the total number of nodes and edges, as well as the names of all the nodes in the graph,

\[
D = \text{LVLMs}(G, I),
\]

where \( D \) represents the description, and \( G, I \) denotes the input graph and task instruction, respectively. This task ensures that the LVLMs develop a clear understanding of the graph’s composition, thereby enhancing their ability to interpret and summarize graph-based information.

5 Enhancing LVLMs with MCDGRAPH

5.1 Experimental Setup

We validate the effectiveness of MCDGRAPH on top two performing LVLMs on VGCURE, i.e., Qwen2-VL and InternVL2. We collect a new set of anonymized visual graphs beyond VGCURE with synthetic structures and automatically create 20k training samples for MCDGRAPH. To prevent catastrophic forgetting, we apply LoRA (Hu et al., 2022) to efficiently enhance the LVLMs’ abilities while preserving their original performance. More details about training samples and implementation are available in Appendix D.

5.2 Results and Analysis

Main Results Tab.6 compares the performance of Qwen2-VL and InternVL2 before and after applying MCDGRAPH on both visual graph understanding and reasoning tasks. We can observe
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>DC</th>
    <th>DQ</th>
    <th>NQ</th>
    <th>NN</th>
    <th>CR</th>
    <th>CN</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <td colspan="13"><b>QA Samples</b></td>
  </tr>
  <tr>
    <td><i>Qwen2-VL</i></td>
    <td>97.80</td>
    <td>16.38</td>
    <td>-</td>
    <td>48.09</td>
    <td>16.18</td>
    <td>16.52</td>
    <td>21.02</td>
    <td>-</td>
    <td>14.19</td>
    <td>19.48</td>
    <td>12.73</td>
    <td>12.90</td>
  </tr>
  <tr>
    <td><i>w MCDGRAPH</i></td>
    <td>98.34 ↑</td>
    <td>25.92 ↑</td>
    <td>-</td>
    <td>60.94 ↑</td>
    <td>25.44 ↑</td>
    <td>13.32</td>
    <td>26.14 ↑</td>
    <td>-</td>
    <td>13.14</td>
    <td>20.74 ↑</td>
    <td>14.44 ↑</td>
    <td>11.95</td>
  </tr>
  <tr>
    <td><i>InternVL2</i></td>
    <td>77.45</td>
    <td>9.78</td>
    <td>-</td>
    <td>50.75</td>
    <td>25.01</td>
    <td>18.30</td>
    <td>24.87</td>
    <td>-</td>
    <td>10.83</td>
    <td>20.72</td>
    <td>10.58</td>
    <td>14.53</td>
  </tr>
  <tr>
    <td><i>w MCDGRAPH</i></td>
    <td>95.68 ↑</td>
    <td>40.45 ↑</td>
    <td>-</td>
    <td>54.78 ↑</td>
    <td>28.80 ↑</td>
    <td>19.43 ↑</td>
    <td>28.53 ↑</td>
    <td>-</td>
    <td>11.67 ↑</td>
    <td>22.34 ↑</td>
    <td>16.50 ↑</td>
    <td>12.76</td>
  </tr>
  <tr>
    <td colspan="13"><b>FC Samples</b></td>
  </tr>
  <tr>
    <td><i>Qwen2-VL</i></td>
    <td>76.50</td>
    <td>68.26</td>
    <td>94.92</td>
    <td>64.32</td>
    <td>67.34</td>
    <td>44.17</td>
    <td>74.28</td>
    <td>38.52</td>
    <td>35.28</td>
    <td>57.07</td>
    <td>42.08</td>
    <td>42.25</td>
  </tr>
  <tr>
    <td><i>w MCDGRAPH</i></td>
    <td>89.58 ↑</td>
    <td>65.80</td>
    <td>95.84 ↑</td>
    <td>77.10 ↑</td>
    <td>79.75 ↑</td>
    <td>60.71 ↑</td>
    <td>83.07 ↑</td>
    <td>53.90 ↑</td>
    <td>53.48 ↑</td>
    <td>64.17 ↑</td>
    <td>64.12 ↑</td>
    <td>60.11 ↑</td>
  </tr>
  <tr>
    <td><i>InternVL2</i></td>
    <td>68.63</td>
    <td>36.82</td>
    <td>93.17</td>
    <td>63.28</td>
    <td>72.81</td>
    <td>62.46</td>
    <td>75.37</td>
    <td>47.12</td>
    <td>33.70</td>
    <td>56.98</td>
    <td>55.94</td>
    <td>54.71</td>
  </tr>
  <tr>
    <td><i>w MCDGRAPH</i></td>
    <td>76.55 ↑</td>
    <td>71.98 ↑</td>
    <td>90.04 ↑</td>
    <td>56.83 ↑</td>
    <td>80.98 ↑</td>
    <td>73.14 ↑</td>
    <td>80.23 ↑</td>
    <td>52.09 ↑</td>
    <td>52.81 ↑</td>
    <td>59.07 ↑</td>
    <td>69.03 ↑</td>
    <td>45.82</td>
  </tr>
</table>

Table 6: Model performance (Acc/F1/EM_F1 for QA and F1 for FC) on various tasks. ↑ indicates an improvement compared to the original model. The complete experimental results are shown in Tab.18 and 19.

![Model performance (F1/Acc) comparison on QA samples across representative tasks and edge ranges.](page_184_1012_563_377.png)

Figure 6: Model performance (F1/Acc) comparison on QA samples across representative tasks and edge ranges.

that (I) MCDGRAPH improves the performance of LVLMs on almost all tasks, demonstrating the effectiveness of the proposed method. (II) The improvement of LVLMs is particularly impressive on edge-related tasks, i.e., Edge Number Query (EN) and Nested Relation Query (NR), that are relatively difficult for LVLMs. This suggests that MCDGRAPH enhances LVLMs’ ability to capture structural information. (III) Although MCDGRAPH does not optimize for the tasks in VGCURE, it still shows obvious improvement in both QA and FC samples for most tasks in VGCURE. This demonstrates that the proposed method can improve the fundamental graph structure understanding capabilities of LVLMs, which leads to better performance on downstream tasks. We also present a case on NR to further understand the effectiveness of the proposed method, please refer to Appendix G for detailed information.

Ablation Study We also conduct ablation study on QA samples with Qwen2-VL and the results are shown in Tab.7. It can be observed that after removing different training tasks, although the LVLM may have a slight performance improvement on some tasks, it may lose certain capabilities causing its performance to plummet on some task (as evidenced by the red results in the Tab.7). Meanwhile, all three self-supervised tasks enhance the LVLMs’ ability to capture structural and relational information from different dimensions, complementing each other in order to comprehensively improve the LVLM’s overall performance on all tasks.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>DQ</th>
    <th>NQ</th>
    <th>NN</th>
    <th>CR</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <td><i>Qwen2-VL</i></td>
    <td>97.80</td>
    <td>16.38</td>
    <td>48.09</td>
    <td>16.18</td>
    <td>16.52</td>
    <td>21.02</td>
    <td>14.19</td>
    <td>19.48</td>
    <td>12.73</td>
    <td>12.90</td>
  </tr>
  <tr>
    <td><i>w MCDGraph</i></td>
    <td>98.34 ↑</td>
    <td>25.92 ↑</td>
    <td>60.94 ↑</td>
    <td>25.44 ↑</td>
    <td>13.32</td>
    <td>26.14 ↑</td>
    <td>13.14</td>
    <td>20.74 ↑</td>
    <td>14.44 ↑</td>
    <td>11.95</td>
  </tr>
  <tr>
    <td>- Model</td>
    <td>99.64</td>
    <td>28.70</td>
    <td>64.02</td>
    <td>25.35</td>
    <td>15.16</td>
    <td>25.13</td>
    <td>11.83</td>
    <td>21.11</td>
    <td>15.20</td>
    <td>5.33</td>
  </tr>
  <tr>
    <td>- Contrastive</td>
    <td>99.64</td>
    <td>28.70</td>
    <td>64.02</td>
    <td>25.35</td>
    <td>15.16</td>
    <td>25.13</td>
    <td>11.83</td>
    <td>21.11</td>
    <td>15.20</td>
    <td>5.33</td>
  </tr>
  <tr>
    <td>- Description</td>
    <td>100</td>
    <td>13.27</td>
    <td>57.80</td>
    <td>23.09</td>
    <td>14.14</td>
    <td>23.47</td>
    <td>10.16</td>
    <td>19.87</td>
    <td>16.99</td>
    <td>8.41</td>
  </tr>
</table>

Table 7: Ablation study on across various tasks.

Performance on Varying Complexity Fig.6 illustrates the performance improvements of MCDGRAPH on LVLMs across varying graph complexities on six representative tasks in VGCURE. Fine-tuning with MCDGRAPH consistently improves performance across most tasks and complexity levels, demonstrating its its effectiveness in enhancing the LVLMs’ ability to understand and reason over visual graphs. For simpler graphs, the improvement is smaller, where LVLMs already perform well, but as complexity increases, the performance gap between fine-tuned and baseline models becomes more pronounced, highlighting MCDGRAPH’s importance in handling more complex visual graphs. Due to the space limit, the results for other dimensions are shown in Figs.20-25 in the Appendix, with conclusions similar to those above.

Generalization of MCDGRAPH To validate the generalization of our method, we regenerate 50 visual graphs with different visual styles and naming conventions of nodes and edges from those in VGCURE for each graph structure. The details and results are presented in Appendix E.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">VisionGraph</th>
    <th colspan="4">FactKG</th>
  </tr>
  <tr>
    <th>Connect</th>
    <th>Cycle</th>
    <th>MaxFlow</th>
    <th>Accuracy</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Qwen-2-VL</td>
    <td>55.8</td>
    <td>52.88</td>
    <td>1.72</td>
    <td>79.60</td>
    <td>81.18</td>
    <td>79.13</td>
    <td>79.13</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td>
    <td>53.37</td>
    <td>52.88</td>
    <td>5.17</td>
    <td>80.15</td>
    <td>81.71</td>
    <td>79.96</td>
    <td>79.71</td>
  </tr>
  <tr>
    <td>Initial</td>
    <td>56.0</td>
    <td>52.88</td>
    <td>6.0</td>
    <td>80.73</td>
    <td>80.73</td>
    <td>78.95</td>
    <td>79.95</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td>
    <td>54.72</td>
    <td>52.88</td>
    <td>8.62</td>
    <td>79.78</td>
    <td>81.54</td>
    <td>79.32</td>
    <td>79.53</td>
  </tr>
</table>

Table 8: Model performance on downstream tasks.

5.3 Results on Downstream Reasoning Tasks

To further demonstrate the scalability and applicability of our method, we evaluate MCDGRAPH on graph-related downstream reasoning tasks.

VisionGraph We first evaluate the performance of MCDGRAPH on three representative graph theory problems in VisionGraph (Li et al., 2024c). As the results shown in Tab.8, MCDGRAPH generally improve the performance of LVLMs on these tasks, especially for the relatively difficult Maximum Flow task. This confirms the effectiveness and scalability of our method.

FactKG We then evaluate MCDGRAPH on FactKG (Kim et al., 2023b), a knowledge graph-based fact verification dataset collected from real-world data. As Tab.8 shows, MCDGRAPH gives consistently better performance than LVLMs on FactKG, suggesting that the proposed method can improve LVLMs in real-world graph-related tasks. Note that FactKG not only requires fundamental visual graph understanding and reasoning abilities but also relies on LVLMs’ understanding of semantic and logical relationships between entities, which our method does not address. Therefore, the improvement of LVLMs is not as significant as that of the graph theory problems. We also evaluate our method on two general vision reasoning tasks and the results are available in Appendix F.

6 Related Work

Multimodal Benchmark for Graphs
Li et al. (2024c) and Wei et al. (2024) introduce VisionGraph and GVLQA, respectively, for testing the problem-solving capabilities of LVLMs in graph theory. Both of them contain numerous synthetic visual graphs and complex graph theory problems. Besides, Ai et al. (2024) propose a novel instruction-following benchmark for multimodal graph understanding and reasoning, which contains a number of real-world graph images with diverse structures across various domains. However, these benchmarks focus on specific downstream tasks where LVLMs perform poorly. The goal of VGCURE is to assess LVLMs’ fundamental understanding and reasoning abilities on visual graphs to identify the reasons for their failures.

Boosting LVLMs for Visual Graph Reasoning
Li et al. (2024c) propose a Description-Program-Reasoning (DPR) chain to enhance logical accuracy through graphical structure description and multi-step reasoning. Wei et al. (2024) introduce GITA, an end-to-end framework integrating visual information into instruction-based graph reasoning. Additionally, Deng et al. (2024) present GraphVis, which uses curriculum fine-tuning for training LVLMs on feature recognition and visual graph QA tasks. Unlike current methods, our MCDGRAPH is a general-purpose, self-supervised approach that improves the fundamental understanding and reasoning of LVLMs on visual graphs, making it adaptable to most graph-related downstream tasks.

Graph Benchmarks for GNNs Rozemberczki et al. (2021) construct Wikipedia-based graphs with pages as nodes and hyperlinks as edges. Hu et al. (2020) present realistic datasets spanning social, biological, molecular, code, and knowledge graphs. Memyei and Cangea (2020) propose Wiki-CS, a GNN benchmark based on Computer Science articles. Morris et al. (2020) release TUDataset, covering 120 datasets across multiple domains for graph classification and regression. To address homophily limitations, Lim et al. (2021) introduce large-scale, non-homophilous graph datasets. Dwivedi et al. (2022) propose the Long Range Graph Benchmark to evaluate models on long-range interaction reasoning tasks. Different from these graph benchmarking approach, the proposed VGCURE focuses on exploring the fundamental graph understanding and reasoning capabilities of LVLMs, showcasing their potential to unify multimodal information processing through a unified visual learning paradigm.

7 Conclusion

This paper introduces VGCURE, a comprehensive benchmark comprising 22 tasks to evaluate LVLMs’ fundamental understanding and reasoning capabilities on visual graphs. Experiments on 14 LVLMs reveal significant limitations, especially in capturing structural information. To this end, we propose MCDGRAPH, a structure-aware self-supervised method to enhance open-source LVLMs’ structure learning abilities. Extensive experiments validate the effectiveness of our method across a wide range of graph-related tasks.
Limitations

• Complexity of Visual Graphs. Due to the limitations of current LVLMs’ performance on visual graph tasks, we restrict the number of nodes in the synthetic graph structure to between 7 and 15, potentially limiting the exploration and improvement of the LVLMs’ performance on more complex visual graphs.

• Experiments on Larger LVLMs. Due to limited resources, the majority of our experiments are performed only on LVLMs with around 7B parameters, lacking performance evaluation and improvement of larger models with more parameters.

Acknowledgments

We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions. This work was partly supported by the National Natural Science Foundation of China (62406091, 62276077, 62125201, U23B2055, U24A20328, U24B20174, 62350710797), the National Science and Technology Innovation 2030 Major program (2024ZD01NL00101), and Shenzhen Science and Technology Program (KQTD20240729102154066, ZDSYS20230626091203008).

References

Qihang Ai, Jiafan Li, Jincheng Dai, Jianwu Zhou, Lemao Liu, Haiyun Jiang, and Shuming Shi. 2024. Advancement in graph understanding: A multimodal benchmark and fine-tuning of vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7485–7501.

Réka Albert and Albert-László Barabási. 2002. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6001–6015.

Albert-László Barabási and Réka Albert. 1999. Emergence of scaling in random networks. science, 286(5439):509–512.

He Cao, Yanjun Shao, Zhiyuan Liu, Zijing Liu, Xiangru Tang, Yuan Yao, and Yu Li. 2024. PRESTO: Progressive pretraining enhances synthetic chemistry outcomes. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10197–10224.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghu Ram Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.

Runjin Chen, Tong Zhao, AJAY KUMAR JAISWAL, Neil Shah, and Zhangyang Wang. 2024. LLaGA: Large language and graph assistant. In Forty-first International Conference on Machine Learning.

Yihe Deng, Chenchen Ye, Zijie Huang, Mingyu Derek Ma, Yiwen Kou, and Wei Wang. 2024. Graphvis: Boosting LLMs with visual knowledge graph integration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. 2022. Long range graph benchmark. Advances in Neural Information Processing Systems, 35:22326–22340.

John Ellson, Emden Gansner, Lefteris Koutsofios, Stephen C North, and Gordon Woodhull. 2002. Graphviz—open source graph drawing tools. In Graph Drawing: 9th International Symposium, GD 2001 Vienna, Austria, September 23–26, 2001 Revised Papers 9, pages 483–484. Springer.

Paul Erdős and Alfréd Rényi. 1959. On random graphs I. Publicationes Mathematicae Debrecen, 6:290–297.

Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. arXiv preprint arXiv:2406.09170.

Aric Hagberg, Pieter J Swart, and Daniel A Schult. 2008. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Laboratory (LANL), Los Alamos, NM (United States).

Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. 1983. Stochastic blockmodels: First steps. Social networks, 5(2):109–137.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. 2024. Large language models are cross-lingual knowledge-free reasoners. arXiv preprint arXiv:2406.16655.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133.

Jiho Kim, Yeonsu Kwon, Yohan Jo, and Edward Choi. 2023a. KG-GPT: A general framework for reasoning on knowledge graphs using large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9410–9421.

Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. 2023b. FactKG: Fact verification via reasoning on knowledge graphs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16190–16206, Toronto, Canada. Association for Computational Linguistics.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava-onenision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024b. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895.

Yunxin Li, Baotian Hu, Haoyuan Shi, Wei Wang, Longyue Wang, and Min Zhang. 2024c. Visiongraph: Leveraging large multimodal models for graph theory problems in visual context. In Forty-first International Conference on Machine Learning.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024d. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26763–26773.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. 2021. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in neural information processing systems, 34:20887–20902.

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. 2023. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575.

Haotian Liu, Chunyuan Li, Yuhehng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306.

Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. 2023. MoIca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15623–15638.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, volume 35, pages 2507–2521.

Péter Mernyei and Cătălina Cangea. 2020. Wikics: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. 2020. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663.

Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S. Melo, Ana Paiva, and Danica Kragic. 2022. Geometric multimodal contrastive representation learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 17782–17800.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. 2021. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2):cnab014.

Christoph Schweimer, Christine Gfrerer, Florian Lugstein, David Pape, Jan A. Velimsky, Robert Elsässer, and Bernhard C. Geiger. 2022. Generating simple directed social network graphs for information spreading. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 1475–1485.

Dustin Schwenk, Apoorv Khadelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162.

Jiasheng Si, Yibo Zhao, Yingjie Zhu, Haiyang Zhu, Wenpeng Lu, and Deyu Zhou. 2024a. CHECKWHY: Causal fact verification via argument structure. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15636–15659.
Jiasheng Si, Yingjie Zhu, Wenpeng Lu, and Deyu Zhou. 2024b. Denoising rationalization for multi-hop fact verification via multi-granular explainer. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12593–12608.

Jianheng Tang, Qifan Zhang, Yuhai Li, and Jia Li. 2024. Grapharena: Benchmarking large language models on graph computational problems. arXiv preprint arXiv:2407.00379.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Yanbin Wei, Shuai Fu, Weisen Jiang, Zejian Zhang, Zhixiong Zeng, Qi Wu, James Kwok, and Yu Zhang. 2024. GITA: Graph to visual and textual integration for vision-language graph reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840.

Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. 2024a. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320.

Xin Zhang, Linhai Zhang, and Deyu Zhou. 2023. Sentiment analysis on streaming user reviews via dual-channel dynamic graph neural network. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7208–7220.

Yu Zhang, Kehai Chen, Xuefeng Bai, Zhao Kang, Quanjiang Guo, and Min Zhang. 2024b. Question-guided knowledge graph re-scoring and injection for knowledge graph question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8972–8985.

Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, and Min Zhang. 2025. Evaluating and steering modality preferences in multimodal large language model. arXiv preprint arXiv:2505.20977.

Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).

Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal table understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9102–9124.

Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, and Nicholas Jing Yuan. 2024. Multi-modal knowledge graph construction and application: A survey. IEEE Transactions on Knowledge and Data Engineering, 36(2):715–735.

A VGCCURE Construction

A.1 Graphs Generation

For synthetic graph structures, we use the NetworkX library for random generation and employ hyperparameters to control the expected macroscopic properties of each graph:

• ER: This structure takes an edge probability parameter \( p \), which we choose randomly from \( \{0.2, 0.3, 0.4\} \) during generation.

• BA: This structure takes the parameter \( m \), which denotes the number of edges to attach from a new node to existing nodes. We choose randomly from \( \{2, 3\} \) during generation.

• SFN: For this structure, we use the default parameters provided by NetworkX except for the number of nodes during generation.

• SBM: This structure takes the sizes of blocks \( s \) and the density of edges going from the nodes of one group to nodes of another group \( p \) as parameters. During generation, we set \( s \) to \( [n, m] \) and \( p \) to \( [[p_1, p_2], [p_2, p_3]] \), where \( n \) and \( m \) are a random integer from \( [3, 7] \) and \( [4, 8] \), respectively, and \( p_1, p_2, p_3 \) are all randomly selected from \( \{0.2, 0.3, 0.4\} \).

• Star: This structure requires no parameters other than the number of nodes.

For all the above structures except SBM, the number of nodes during generation is an arbitrary integer in the range \( [7, 15] \).

To anonymize the visual graph, we use the unique name \( Ex \) containing no information to name the nodes in the graph structure, where \( x \in \{1, 2, \ldots, n\} \) and \( n \) is the number of nodes in the graph. For edges, we choose a random identify
from {R1, R2, ..., R10} to name them. The name can be repeated for each edge. The examples of synthetic visual graph are shown in Fig.7.

A.2 Tasks Generation
To ensure the correctness of the generated samples, we first search for relevant paths in the given graph that satisfy the conditions of the task. Then the final QA samples and FC samples are generated based on the paths and corresponding templates. If no path exists, the generation of samples for the task is skipped. The final statistics of VGCURe and the example samples with undirected graph for each task are shown in Tab.9 and Tab.10, respectively.

B Experimental Setup for Evaluation

B.1 Prompts
To facilitate the LVLMs to understand the content in the visual graph, we take a visual graph description in addition to the input during test.
• Directed Visual Graph Description: The given image shows a graph where circles represent nodes, with the content inside indicating the node names. The arrowed lines connecting two nodes represent edges, and the content in the middle of the edges indicates the edge names.
• Undirected Visual Graph Description: The given image shows a graph where circles represent nodes, with the content inside being the node names. The lines connecting two nodes represent edges, and the content in the middle of the edges represents the edge names.
The complete prompt for QA samples are as follow:
• NN, CR, RA, SRN, NQ: [Visual Graph Description] Answer the given questions based on the graph in the image.\nQuestion: [question]\nPlease provide the answer directly without the reasoning process and present your answer in the LIST format: [Entity1, Entity2, ...].
• NR: [Visual Graph Description] Answer the given questions based on the graph in the image.\nQuestion: [question]\nPlease provide the answer directly without the reasoning process and present your answer in the LIST format: [Relation1, Relation2, ...].
• CP: [Visual Graph Description] Answer the given questions based on the graph in the image.\nQuestion: [question]\nIf yes, please output all the shortest paths in the LIST Format and conclude your answer with "Yes. The shortest paths are [[Entity1, Entity2,...], [Entity3, Entity4,...], ...]". If no path exists, please answer "No".
• NNu, EN, DQ: [Visual Graph Description] Answer the given questions based on the graph in the image.\nQuestion: [question]\nPlease provide the answer directly without the reasoning process.
The complete prompt for FC samples are as follow:
• [Visual Graph Description] Verify the truth of the given claim against the graph in the image.\nClaim: [claim]\nThe answer should contain only "True" or "False", without reasoning process.

B.2 Evaluation Metrics
For the QA samples of NQ, NN, CR, RA, SRN, and NR tasks, we use (macro-averaged) F1 score and Hits@1 as in the previous QA benchmarks (Rajpurkar et al., 2016; Zhang et al., 2018). For the QA samples of CP task, we employ EM_F1, which is the macro F1 score calculated based on the exact match between the predicted path and the ground truth path, and Label_Acc, which measures the accuracy of the model’s prediction on whether a path exists or not. For the QA samples of NNu, EN and DQ tasks, we compute the accuracy between predicted answers and ground truth. For the FC samples of all tasks, following Si et al. (2024a,b), we use macro F1 and accuracy as the metrics.

C Why LVLMs Fail on Fundamental Tasks?

C.1 Comparison with Backbone LLMs
To compare the performance of the LVLMs with the corresponding LLMs on the VGCURe task, we first randomly select 1000 samples for each task and convert the corresponding visual graphs into text triples to construct the text version of the samples, and then evaluate them with the corresponding backbone LLMs.
The complete prompts for QA samples are as follows:
• NN, CR, RA, SRN, NQ: Given a set of triples representing an directed/undirected graph, where each triple denotes [Node, Edge, Node], answer the given questions based on the graph.\nTriples: [Triples of visual graph]\nQuestion: [question]\nPlease provide
(a) Directed BA    (b) Directed ER    (c) Directed SBM    (d) Directed SFN    (e) Directed Star

(f) Undirected BA    (g) Undirected ER    (h) Undirected SBM    (i) Undirected SFN    (j) Undirected Star

Figure 7: Examples of synthetic visual graphs.

(a)    (b)

(c)    (d)

Figure 8: Visual graphs corresponding to the examples of error type.

the answer directly without the reasoning process and present your answer in the LIST format: [Entity1, Entity2, ...].
• NR: Given a set of triples representing an directed/undirected graph, where each triple denotes [Node, Edge, Node], answer the given questions based on the graph.\nTriples: [Triples of visual graph]\nQuestion: [question]\nPlease provide the answer directly without the reasoning process and present your answer in the LIST format: [Relation1, Relation2, ...].
• NNu, EN, DQ: Given a set of triples representing an directed/undirected graph, where each triple denotes [Node, Edge, Node], answer the given questions based on the graph.\nTriples: [Triples of visual graph]\nQuestion: [question]\nPlease provide the answer directly without the reasoning process.
The complete prompt for FC samples are as follow:
• [Visual Graph Description] Given a set of triples representing an directed graph, where each triple denotes [Node, Edge, Node], verify the truth of the given claim against the graph.\nTriples: [Triples of visual graph]\nClaim: [claim]\nThe answer should contain only "True" or "False", without reasoning process.

The Connective Path Query (CP) task is ignored here due to LLMs’ poor instruction-following ability on this task.

D Experimental Setup for Training

D.1 Training Samples

Graphs Generation For Masked Graph Infilling and Graph Description task, we use the same synthetic visual graph generation strategy as VGCURE. As for the Contrastive Graph Discrimination, in order to reduce the difficulty, we limited the number of nodes per graph structure to [4, 8] during generation.

Task Instruction To increase the diversity of samples, we designed various instructions with similar semantics for each task in the MCDGRAPH.
• Masked Graph Infilling
– Using the given set of triples, predict the word that should fill the [MASK] position in the knowledge graph in the image.
<table>
  <tr>
    <th rowspan="2">Structure Type</th>
    <th rowspan="2"># Graphs</th>
    <th colspan="13">Number of QA Samples</th>
    <th rowspan="2">Avg. Nodes</th>
    <th rowspan="2">Avg. Edges</th>
  </tr>
  <tr>
    <th>NN</th><th>CR</th><th>CN</th><th>RA</th><th>SRN</th><th>NR</th><th>CP</th><th>NNu</th><th>EN</th><th>DC</th><th>DQ</th><th>NQ</th>
  </tr>
  <tr>
    <td>BA</td><td>Directed</td><td>400</td><td>400</td><td>400</td><td>400</td><td>370</td><td>324</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>10.98</td><td>20.99</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>11.02</td><td>21.04</td>
  </tr>
  <tr>
    <td>ER</td><td>Directed</td><td>400</td><td>387</td><td>377</td><td>377</td><td>304</td><td>229</td><td>387</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>11.06</td><td>17.69</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>396</td><td>395</td><td>396</td><td>346</td><td>346</td><td>395</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>11.00</td><td>17.41</td>
  </tr>
  <tr>
    <td>SBM</td><td>Directed</td><td>400</td><td>399</td><td>392</td><td>392</td><td>314</td><td>399</td><td>400</td><td>253</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>10.99</td><td>17.13</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>399</td><td>399</td><td>399</td><td>399</td><td>400</td><td>371</td><td>371</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>11.04</td><td>17.32</td>
  </tr>
  <tr>
    <td>SFN</td><td>Directed</td><td>400</td><td>400</td><td>400</td><td>400</td><td>331</td><td>149</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>10.95</td><td>13.86</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>379</td><td>379</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>11.05</td><td>12.80</td>
  </tr>
  <tr>
    <td>Star</td><td>Directed</td><td>400</td><td>396</td><td>388</td><td>388</td><td>350</td><td>288</td><td>396</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>12.05</td><td>11.05</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>400</td><td>400</td><td>400</td><td>393</td><td>393</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>12.11</td><td>11.11</td>
  </tr>
  <tr>
    <td>DBLP</td><td>Directed</td><td>200</td><td>196</td><td>192</td><td>192</td><td>140</td><td>196</td><td>200</td><td>145</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>7.85</td><td>17.76</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>200</td><td>200</td><td>200</td><td>200</td><td>173</td><td>173</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>7.85</td><td>17.76</td>
  </tr>
  <tr>
    <td>Dhpedia</td><td>Directed</td><td>200</td><td>186</td><td>185</td><td>185</td><td>93</td><td>186</td><td>200</td><td>101</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>8.13</td><td>11.36</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>200</td><td>200</td><td>200</td><td>200</td><td>155</td><td>155</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>200</td><td>8.13</td><td>11.36</td>
  </tr>
  <tr>
    <td>Openflights</td><td>Directed</td><td>100</td><td>100</td><td>100</td><td>100</td><td>65</td><td>100</td><td>100</td><td>63</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>5.51</td><td>13.59</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>100</td><td>100</td><td>100</td><td>100</td><td>90</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td><td>5.51</td><td>13.59</td>
  </tr>
  <tr>
    <td>PubChemQC</td><td>Directed</td><td>400</td><td>400</td><td>138</td><td>400</td><td>400</td><td>119</td><td>119</td><td>37</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>5.45</td><td>4.76</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>400</td><td>400</td><td>398</td><td>400</td><td>398</td><td>400</td><td>151</td><td>151</td><td>400</td><td>400</td><td>400</td><td>400</td><td>400</td><td>5.45</td><td>4.76</td>
  </tr>
  <tr>
    <td>Social Network</td><td>Directed</td><td>300</td><td>262</td><td>254</td><td>254</td><td>134</td><td>262</td><td>300</td><td>124</td><td>300</td><td>300</td><td>300</td><td>300</td><td>300</td><td>7.57</td><td>9.25</td>
  </tr>
  <tr>
    <td></td><td>Undirected</td><td>300</td><td>299</td><td>297</td><td>299</td><td>238</td><td>238</td><td>297</td><td>300</td><td>300</td><td>300</td><td>300</td><td>300</td><td>300</td><td>7.57</td><td>9.25</td>
  </tr>
  <tr>
    <td>Total</td><td></td><td>6400</td><td>6320</td><td>6015</td><td>6282</td><td>5493</td><td>5247</td><td>5795</td><td>5224</td><td>6400</td><td>6400</td><td>6400</td><td>6400</td><td>6400</td><td>-</td><td>-</td>
  </tr>
</table>

Table 9: Statistics of VGCURE benchmark, where # Graphs represents the number of visual graphs, Avg.Nodes and Avg.Edges denote the average number of nodes and edges in the graph, respectively. For each task, the number of QA samples is the values in the table, except for CN and DC, which have no QA samples, and the number of FC samples is twice the value in the table.

– Based on the provided triples, determine the correct word to complete the [MASK] position in the knowledge graph shown in the image.
– Given the set of triples, predict the word that should be placed in the [MASK] position within the knowledge graph in the image.
– Use the given triples to predict the appropriate word for the [MASK] position in the knowledge graph depicted in the image.
– Using the set of triples, identify the word that should fill the [MASK] position in the knowledge graph in the image.
– Based on the set of triples, predict the correct word for the [MASK] position in the knowledge graph in the image.
– Given the triples, predict the word that fits the [MASK] position in the knowledge graph present in the image.
– Using the triples provided, determine the word that should be used to fill the [MASK] position in the knowledge graph in the image.
– Predict the word that should occupy the [MASK] position in the knowledge graph in the image, based on the given triples.
– Using the provided triples, identify the word that should complete the [MASK] position in the knowledge graph in the image.

• Contrastive Graph Discrimination
– Determine whether Graph A and Graph B in the given image are identical.
– Assess if Graph A and Graph B depicted in the image are equivalent.
– Evaluate whether the structures of Graph A and Graph B in the provided image match.
– Identify if there are any differences between Graph A and Graph B in the shown image.
– Check if Graph A and Graph B illustrated in the image are the same.
– Analyze the image to determine if Graph A is identical to Graph B.
– Investigate whether Graph A and Graph B in the given image are congruent.
– Examine the provided image to see if Graph A and Graph B are equivalent.
– Compare Graph A and Graph B in the image to establish their similarity.
– Confirm if Graph A and Graph B presented in
<table>
  <tr>
    <th>Task</th>
    <th>QA sample</th>
    <th>FC sample (Label)</th>
  </tr>
  <tr>
    <td>NNu</td>
    <td>Q: How many nodes are there in this graph?<br>A: 11</td>
    <td>There are 11 nodes in this graph. (True)<br>There are 15 nodes in this graph. (False)</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>Q: How many edges are there in this graph?<br>A: 15</td>
    <td>There are 15 edges in this graph. (True)<br>There are 19 edges in this graph. (True)</td>
  </tr>
  <tr>
    <td>DC</td>
    <td>-</td>
    <td>This graph is an undirected graph. (True)<br>This graph is a directed graph. (True)</td>
  </tr>
  <tr>
    <td>DQ</td>
    <td>Q: What is the degree of E7 in this graph?<br>A: 2</td>
    <td>The degree of E7 in this graph is 2. (True)<br>The degree of E7 in this graph is 7. (False)</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>Q: Which nodes are neighbors of E6 in this graph?<br>A: [E1, E2, E7, E9]</td>
    <td>E7 is a neighbors of E6 in this graph. (True)<br>E10 is a neighbors of E6 in this graph. (False)</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>Q: Which entities are connected to the entity that has R10 with E2 via R9?<br>A: [E11, E4]</td>
    <td>E4 is connected to the entity that has R10 with E2 via R9. (True)<br>E3 is connected to the entity that has R10 with E2 via R9. (False)</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>Q: Which entities are connected to E8 via R3 as well as connected E1 via R10?<br>A: [E2]</td>
    <td>E2 is connected to E8 via R3 as well as connected E1 via R10. (True)<br>E3 is connected to E8 via R3 as well as connected E1 via R10. (False)</td>
  </tr>
  <tr>
    <td>CN</td>
    <td>-</td>
    <td>E5 and E2 share a common neighbor. (True)<br>E10 and E11 share a common neighbor. (False)</td>
  </tr>
  <tr>
    <td>RA</td>
    <td>Q: Which entities are connected to E1 via the same relation between E11 and E17?<br>A: [E4]</td>
    <td>E4 is connected to E1 via the same relation between E11 and E1. (True)<br>E2 is connected to E1 via the same relation between E11 and E1. (False)</td>
  </tr>
  <tr>
    <td>SRN</td>
    <td>Q: Which entities are both connected to E2 via R10?<br>A: [E1, E5]</td>
    <td>E5 and E1 both connected to E2 via R10. (True)<br>E5 and E3 are both connected to E2 via R10. (False)</td>
  </tr>
  <tr>
    <td>NR</td>
    <td>Q: What is the relation between E2 and the entity that is connected to E6 via R8?<br>A: [R10]</td>
    <td>The relation between E2 and the entity that is connected to E6 via R8 is R10. (True)<br>The relation between E2 and the entity that is connected to E6 via R8 is R4. (False)</td>
  </tr>
  <tr>
    <td>CP</td>
    <td>Q: Is there a path between E5 and E3?<br>A: Yes. The shortest paths are [[E5, E1, E3], [E5, E2, E3]]</td>
    <td>[E5, E1, E3] is one of the shortest path between E5 and E3. (True)<br>[E5, E11, E2, E3] is one of the shortest path between E5 and E3. (False)</td>
  </tr>
</table>

Table 10: Examples with undirected graph for each task in VGCURE. These samples all correspond to the SFN graph shown in Fig.7(i).

the image are indistinguishable.

• Graph Description
– Describe the given graph, including the number of nodes, the number of edges, and the names of all the nodes.
– Provide a description of the given graph, specifying the number of nodes, edges, and listing all the node names.
– Analyze the given graph by stating the number of nodes, edges, and enumerating the names of all the nodes.
– Summarize the graph by detailing the number of nodes, edges, and listing the names of each node.
– Explain the graph, including the count of nodes and edges, and provide the names of all the nodes.
– Describe the graph, indicating how many nodes and edges it contains, and listing all the node names.
– Provide an overview of the graph, mentioning the number of nodes, edges, and the names of all nodes.
– Characterize the given graph, noting the number of nodes, edges, and listing all the node names.
– Detail the structure of the given graph, including node and edge counts, and providing a list of all node names.
– Give a description of the graph, including the total number of nodes, edges, and the names of all the nodes.

Similar to VGCURE, we include a visual graph description in input as well. Thus, the complete task instruction \( I \) for each training sample in MCDGRAPH is "[Visual Graph Description] [Instruction]".

Number of Samples For Masked Graph Infilling task, we generate **10,000 samples**, with half of the samples masking nodes and the other half masking edges. For Contrastive Graph Discrimination tasks, **5,000 samples**, where each sample consists of two visual graphs, are generated automatically. Similarly, the Graph Description task also contains **5,000 samples**, each corresponding to a unique visual graphs.

D.2 Implementation Details

For Qwen2-VL, we employ the LoRA-based supervised fine-tuning scripts provided by LLaMA-Factory\(^3\). For InternVL2, we perform LoRA-based fine-tuning based on the code and documentation provided officially\(^4\). The hyperparameters used for training are shown in Tab.12. All the experiments are finished on 4 A100 GPUs with 80GB memory.

\(^3\)https://github.com/hiyouga/LLaMA-Factory
\(^4\)https://github.com/OpenGVLab/InternVL
<table>
  <tr>
    <th>Error Type</th>
    <th>QA samples</th>
    <th>Visual Graph</th>
  </tr>
  <tr>
    <td><b>Relation Misunderstanding</b></td>
    <td>Q: What is the relation from the entity that is R5 of E3 to E1?<br>A: [R1]<br>P: [R6]</td>
    <td>Fig.8(a)</td>
  </tr>
  <tr>
    <td><b>Complexity Limitation</b></td>
    <td>Q: What is the relation from the entity that is R7 of E1 to E7?<br>A: [R5]<br>P: [R9]</td>
    <td>Fig.8(b)</td>
  </tr>
  <tr>
    <td><b>Structural Hallucination</b></td>
    <td>Q: Is there a path between E1 and E3?<br>A: Yes. The shortest paths are [[E1, E6, E3]]<br>P: Yes. The shortest paths are [E1, E3] and [E1, E4, E3].</td>
    <td>Fig.8(c)</td>
  </tr>
  <tr>
    <td><b>Entity-based Answering</b></td>
    <td>Q: Which entities are R5 of E2 as well as R1 of E1?<br>A: [E11]<br>P: [E2, E1]</td>
    <td>Fig.8(b)</td>
  </tr>
  <tr>
    <td><b>Off-target Generation</b></td>
    <td>Q: What is the relation between E2 and the entity that is connected to E4 via R9?<br>A: [R2]<br>P: [Edge, Node]</td>
    <td>Fig.8(d)</td>
  </tr>
  <tr>
    <td><b>Format Error</b></td>
    <td>Q: Is there a path between E4 and E1?<br>A: Yes. The shortest paths are [[E4, E3, E2, E1]]<br>P: Yes. The shortest paths are [[E4, R9, E3, R2, E2, R6, E1]].</td>
    <td>Fig.8(d)</td>
  </tr>
</table>

Table 11: Examples of each error type, where \( Q \) denotes question, \( A \) denotes gold answer and \( P \) denotes prediction.

<table>
  <tr>
    <th>Model</th>
    <th>Lora_rank</th>
    <th>Lora_alpha</th>
    <th>Global Batch Size</th>
    <th>Learning rate</th>
    <th>Epoch</th>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>8</td>
    <td>16</td>
    <td>64</td>
    <td>1e-4</td>
    <td>5</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>64</td>
    <td>128</td>
    <td>64</td>
    <td>4e-5</td>
    <td>1</td>
  </tr>
</table>

Table 12: Hyperparameters for training

<table>
  <tr>
    <th>Models</th>
    <th>ScienceQA</th>
    <th>AOKVQA</th>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>85.13</td>
    <td>84.37</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td>
    <td>81.31</td>
    <td>83.67</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>97.07</td>
    <td>85.41</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td>
    <td>96.88</td>
    <td>84.45</td>
  </tr>
</table>

Table 13: Model performance (Acc) on general VQA tasks.

E Generalization of MCDGRAPH

E.1 Impact on Visual Styles

To validate the generalization of our method, we employ NetworkX and Matplotlib to regenerate 50 visual graphs with different visual styles from those in VGCURe for each graph structure to explore the impact of visual graph styles. The examples of the same visual graph with different styles are illustrated in Fig.10 and Fig.11.

As the results shown in Tabs.14 and 15, although LVLMs never encounter the different style of visual graph during fine-tuning, our method can still improve the performance of the LVLMs on almost all tasks. This demonstrates the ability of our method to enhance LVLMs’ ability in capturing the structural information in visual graphs with excellent generalization. In addition, compared to Tab.6, it can be noticed that the experimental results before and after the change of style are similar, which indicates that the style itself has no effect on the evaluation of LVLMs.

E.2 Impact of Naming Conventions

To explore the impact of altering the naming conventions, we regenerated the visual graphs and samples using the following new naming conventions.

• Node-Edge: The nodes in the visual graph are renamed Node\( x \), where \( x \in \{1, 2, \ldots, n\} \) and \( n \) is the number of nodes in the graph, and the edges are renamed Edge\( y \), where \( y \in \{1, 2, \ldots, 10\} \).

• Name-R: The nodes in the visual graph are renamed to a simple and common human name without any semantic bias, like “John”, “Jane”, “Mike”, “Mary”, etc. The names of the edges remain as they are, i.e., R1, R2, ..., R10.

The results are shown in Tabs.16 and 17. We can observe that after altering the naming conventions, LVLMs continue to show similar trends on most of tasks. Therefore, the experimental analysis in the main text still holds. Meanwhile, the performance of LVLMs decreases on most of the tasks when confronted with different names. This might be due to the fact that the new naming conventions gives longer names to nodes and edges and recognizing these information increases the difficulty of the task. It is worth noting that in the face of the new naming conventions, our MCDGRAPH still improves the performance of LVLMs on most tasks, although the fine-tuning still uses the original
Visual Graph

Question & Predictions
What is the relation from the entity that is R1 of E9 to E3?

Qwen2-VL [R1]
InternLM-XC2.5 [R9, R2]
LLaVA-13B [R1, E9, E3]
mPLUG-Owl3 [E9, R1, E3]
Qwen2-VL w MCDGraph [R9]

Figure 9: A case of Nested Relation Query task.

naming conventions. This strongly demonstrates the effectiveness as well as the generalization of our method.

F Results on General VQA Tasks

We evaluate our MCDGRAPH on two general VQA tasks, i.e., ScienceQA (Lu et al., 2022) and AOKVQA (Schwenk et al., 2022). According to the results shown in Tab.13, we can observe that after MCDGraph, LVLMs perform worse on these two VQA tasks. This is due to the fact that our MCDGRAPH is proposed for visual graph understanding and reasoning tasks, which are very different from general VQA tasks. And it is worth noting that our method does not overly compromise the LVLMs’ performance on general VQA tasks, which we consider acceptable.

G Case Study

Fig.9 illustrates a case on Nested Relation Query task. We can observe that Qwen2-VL, LLaVA-13B and mPLUG-Owl3 all make the error of Entity-based Answering when confronted with this question, i.e., using “R1”, “E9”, and “E3” mentioned in the question as the generated answer. InternLM-XC2.5 makes the error of Relation Misunderstanding, i.e., the edge pointing from E3 to E1 is also used as the answer. However, Qwen2-VL after applying MCDGRAPH can answer this question correctly. This demonstrates that the proposed method can improve the fundamental graph structure understanding of LVLMs, thus avoiding the occurrence of the previously mentioned errors.

Figure 10: An example of visual graph with different style for the experimental results in Tab.14

Figure 11: An example of visual graph with different style for the experimental results in Tab.15
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th><th>EN</th><th>DC</th><th>DQ</th><th>NQ</th>
    <th>NN</th><th>CR</th><th>CN</th><th>RA</th><th>SRN</th><th>NR</th><th>CP</th>
  </tr>
  <tr>
    <td colspan="13"><b>QA Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>98.20</td><td>18.10</td><td>-</td><td>50.70</td><td>18.64</td><td>15.66</td><td>22.66</td><td>-</td><td>14.83</td><td>20.45</td><td>16.95</td><td>11.04</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>99.40↑</td><td>40.20↑</td><td>-</td><td>56.80↑</td><td>28.44↑</td><td>14.64</td><td>27.02↑</td><td>-</td><td>13.69</td><td>22.22↑</td><td>14.73</td><td>11.99↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>73.70</td><td>8.60</td><td>-</td><td>48.40</td><td>26.33</td><td>18.26</td><td>26.19</td><td>-</td><td>9.85</td><td>21.75</td><td>11.21</td><td>1.60</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>93.80↑</td><td>35.70↑</td><td>-</td><td>52.90↑</td><td>28.89↑</td><td>19.90↑</td><td>29.24↑</td><td>-</td><td>11.73↑</td><td>23.23↑</td><td>18.04↑</td><td>14.49↑</td>
  </tr>
  <tr>
    <td colspan="13"><b>FC Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>71.37</td><td>61.78</td><td>86.53</td><td>73.82</td><td>73.03</td><td>48.00</td><td>77.35</td><td>39.98</td><td>36.04</td><td>60.22</td><td>45.31</td><td>42.93</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>88.45↑</td><td>63.14↑</td><td>94.43↑</td><td>76.26↑</td><td>78.59↑</td><td>63.17↑</td><td>82.63↑</td><td>53.78↑</td><td>54.33↑</td><td>66.26↑</td><td>65.95↑</td><td>56.30↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>75.43</td><td>34.80</td><td>86.85</td><td>63.68</td><td>74.52</td><td>61.12</td><td>74.23</td><td>47.22</td><td>33.48</td><td>55.85</td><td>55.32</td><td>53.15</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>83.82↑</td><td>70.14↑</td><td>83.48↑</td><td>60.77↑</td><td>79.69↑</td><td>71.39↑</td><td>78.30↑</td><td>52.38↑</td><td>51.24↑</td><td>59.83↑</td><td>68.58↑</td><td>44.73↑</td>
  </tr>
</table>

Table 14: Model performance (Acc/F1/EM_F1 for QA and F1 for FC) on various tasks with different visual styles. The example of the corresponding visual style is shown in Fig.10

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th><th>EN</th><th>DC</th><th>DQ</th><th>NQ</th>
    <th>NN</th><th>CR</th><th>CN</th><th>RA</th><th>SRN</th><th>NR</th><th>CP</th>
  </tr>
  <tr>
    <td colspan="13"><b>QA Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>95.30</td><td>20.80</td><td>-</td><td>40.00</td><td>13.95</td><td>8.73</td><td>19.04</td><td>-</td><td>11.73</td><td>14.89</td><td>13.16</td><td>3.17</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>98.80↑</td><td>17.30↑</td><td>-</td><td>50.50↑</td><td>22.33↑</td><td>11.56↑</td><td>21.92↑</td><td>-</td><td>11.05</td><td>17.84↑</td><td>11.60</td><td>0.12</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>85.20</td><td>17.90</td><td>-</td><td>39.60</td><td>24.21</td><td>19.99</td><td>22.26</td><td>-</td><td>8.06</td><td>19.46</td><td>12.03</td><td>2.25</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>97.20↑</td><td>35.70↑</td><td>-</td><td>43.00↑</td><td>26.46↑</td><td>19.12</td><td>23.54↑</td><td>-</td><td>11.00↑</td><td>20.45↑</td><td>14.80↑</td><td>12.11↑</td>
  </tr>
  <tr>
    <td colspan="13"><b>FC Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>70.88</td><td>68.06</td><td>66.94</td><td>57.90</td><td>65.09</td><td>43.62</td><td>67.32</td><td>34.28</td><td>34.79</td><td>53.23</td><td>42.54</td><td>42.89</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>76.36↑</td><td>58.47↑</td><td>85.20↑</td><td>74.38↑</td><td>73.69↑</td><td>56.97↑</td><td>75.05↑</td><td>48.68↑</td><td>53.84↑</td><td>60.99↑</td><td>60.44↑</td><td>56.93↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>68.42</td><td>38.35</td><td>91.84</td><td>64.58</td><td>65.45</td><td>62.91</td><td>72.42</td><td>48.17</td><td>34.60</td><td>57.88</td><td>57.77</td><td>52.48</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>69.45↑</td><td>69.11↑</td><td>84.10</td><td>62.09</td><td>74.68↑</td><td>71.52↑</td><td>75.19↑</td><td>52.97↑</td><td>50.89↑</td><td>61.00↑</td><td>66.44↑</td><td>43.19↑</td>
  </tr>
</table>

Table 15: Model performance (Acc/F1/EM_F1 for QA and F1 for FC) on various tasks with different visual styles. The example of the corresponding visual style is shown in Fig.11

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th><th>EN</th><th>DC</th><th>DQ</th><th>NQ</th>
    <th>NN</th><th>CR</th><th>CN</th><th>RA</th><th>SRN</th><th>NR</th><th>CP</th>
  </tr>
  <tr>
    <td colspan="13"><b>QA Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>94.30</td><td>18.10</td><td>-</td><td>48.40</td><td>17.55</td><td>11.34</td><td>13.72</td><td>-</td><td>10.10</td><td>11.82</td><td>5.34</td><td>10.63</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>94.10</td><td>16.00</td><td>-</td><td>63.50↑</td><td>17.25</td><td>8.50</td><td>14.56↑</td><td>-</td><td>8.29</td><td>12.81↑</td><td>12.71↑</td><td>13.00↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>65.61</td><td>19.15</td><td>-</td><td>47.94</td><td>17.45</td><td>12.11</td><td>15.75</td><td>-</td><td>8.87</td><td>13.54</td><td>8.25</td><td>8.12</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>89.30↑</td><td>34.60↑</td><td>-</td><td>49.20↑</td><td>17.59↑</td><td>12.88↑</td><td>14.63</td><td>-</td><td>8.29</td><td>13.75↑</td><td>4.52</td><td>5.45</td>
  </tr>
  <tr>
    <td colspan="13"><b>FC Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>74.18</td><td>66.99</td><td>91.90</td><td>58.77</td><td>58.34</td><td>37.99</td><td>59.80</td><td>34.25</td><td>34.51</td><td>45.99</td><td>35.39</td><td>39.40</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>85.24↑</td><td>66.42</td><td>90.93</td><td>81.77↑</td><td>77.30↑</td><td>56.79↑</td><td>76.47↑</td><td>53.06↑</td><td>52.85↑</td><td>59.45↑</td><td>58.51↑</td><td>58.72↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>72.82</td><td>41.92</td><td>96.19</td><td>65.34</td><td>73.36</td><td>61.51</td><td>77.15</td><td>52.21</td><td>34.79</td><td>55.23</td><td>57.50</td><td>53.17</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>73.86↑</td><td>72.70↑</td><td>94.28</td><td>59.99</td><td>78.23↑</td><td>69.02↑</td><td>74.32↑</td><td>52.12</td><td>54.71↑</td><td>57.41↑</td><td>68.51↑</td><td>44.20↑</td>
  </tr>
</table>

Table 16: Model performance (Acc/F1/EM_F1 for QA and F1 for FC) on various tasks with Node-Edge naming convention.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th><th>EN</th><th>DC</th><th>DQ</th><th>NQ</th>
    <th>NN</th><th>CR</th><th>CN</th><th>RA</th><th>SRN</th><th>NR</th><th>CP</th>
  </tr>
  <tr>
    <td colspan="13"><b>QA Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>21.00</td><td>22.90</td><td>-</td><td>51.10</td><td>17.78</td><td>11.80</td><td>18.31</td><td>-</td><td>11.50</td><td>15.33</td><td>8.66</td><td>14.02</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>13.50</td><td>25.70↑</td><td>-</td><td>63.30↑</td><td>18.66↑</td><td>9.56</td><td>18.91↑</td><td>-</td><td>10.13</td><td>15.90↑</td><td>10.81↑</td><td>11.72↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>32.80</td><td>17.70</td><td>-</td><td>55.10</td><td>19.77</td><td>14.61</td><td>21.60</td><td>-</td><td>10.69</td><td>16.20</td><td>11.29</td><td>11.79</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>24.40</td><td>36.00↑</td><td>-</td><td>54.20</td><td>21.10↑</td><td>13.32</td><td>21.44</td><td>-</td><td>8.15</td><td>16.01</td><td>17.77↑</td><td>11.84↑</td>
  </tr>
  <tr>
    <td colspan="13"><b>FC Samples</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>86.35</td><td>67.03</td><td>93.73</td><td>69.39</td><td>74.05</td><td>41.30</td><td>71.54</td><td>43.59</td><td>38.54</td><td>58.05</td><td>42.43</td><td>44.18</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>68.68</td><td>69.10↑</td><td>96.45↑</td><td>78.20↑</td><td>76.96↑</td><td>57.61↑</td><td>82.66↑</td><td>56.07↑</td><td>55.19↑</td><td>63.51↑</td><td>63.17↑</td><td>56.45↑</td>
  </tr>
  <tr>
    <td>InternVL2</td><td>69.34</td><td>39.80</td><td>87.75</td><td>62.98</td><td>76.17</td><td>69.06</td><td>72.14</td><td>57.15</td><td>39.82</td><td>56.87</td><td>65.83</td><td>48.31</td>
  </tr>
  <tr>
    <td>w MCDGRAPH</td><td>78.05↑</td><td>71.65↑</td><td>75.85</td><td>55.84</td><td>81.23↑</td><td>71.50↑</td><td>79.29↑</td><td>52.43</td><td>57.76↑</td><td>53.85</td><td>67.60↑</td><td>41.99↑</td>
  </tr>
</table>

Table 17: Model performance (Acc/F1/EM_F1 for QA and F1 for FC) on various tasks with Name-R naming convention.
Figure 12: Model performance (Acc/F1) on QA samples across various graph structures and tasks.

Figure 13: Model performance (F1) on FC samples across various graph structures and tasks.

Figure 14: Model performance (F1/Acc) comparison on QA samples across various tasks and edge ranges.
Figure 15: Model performance (F1) comparison on FC samples across various tasks and edge ranges.

Figure 16: Model performance (F1/Acc) comparison on QA samples across various tasks and node ranges.

Figure 17: Model performance (F1) comparison on FC samples across various tasks and node ranges.
Figure 18: Model performance (F1/Acc) comparison on QA samples across various tasks and average degree.

Figure 19: Model performance (F1) comparison on FC samples across various tasks and average degree.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="6">QA Samples</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>IDQ</th>
    <th>NQ</th>
    <th>NN</th>
    <th>CR</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <th></th>
    <th>Ace</th>
    <th>Ace</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>EM_F1</th>
    <th>Label_Acc</th>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>97.80</td>
    <td>16.38</td>
    <td>48.09</td>
    <td>16.18</td>
    <td>38.12</td>
    <td>16.52</td>
    <td>28.34</td>
    <td>21.02</td>
    <td>48.57</td>
    <td>14.19</td>
    <td>27.96</td>
    <td>19.48</td>
    <td>56.42</td>
    <td>12.73</td>
    <td>25.07</td>
    <td>12.90</td>
    <td>38.06</td>
  </tr>
  <tr>
    <td>w/MCDGRAPH</td>
    <td>98.56</td>
    <td>23.92</td>
    <td>60.01</td>
    <td>25.44</td>
    <td>63.41</td>
    <td>32.32</td>
    <td>29.97</td>
    <td>26.14</td>
    <td>63.84</td>
    <td>13.14</td>
    <td>28.42</td>
    <td>20.41</td>
    <td>62.21</td>
    <td>14.41</td>
    <td>25.71</td>
    <td>11.95</td>
    <td>63.98↑</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>77.45</td>
    <td>17.73</td>
    <td>47.61</td>
    <td>18.58</td>
    <td>32.42</td>
    <td>18.53</td>
    <td>20.72</td>
    <td>19.59</td>
    <td>30.77</td>
    <td>14.53</td>
    <td>25.97</td>
    <td>15.43</td>
    <td>33.97</td>
    <td>12.76</td>
    <td>25.23</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/MCDGRAPH</td>
    <td>95.68</td>
    <td>40.45</td>
    <td>54.78</td>
    <td>28.80</td>
    <td>72.77</td>
    <td>19.43</td>
    <td>27.86</td>
    <td>28.53</td>
    <td>67.61</td>
    <td>11.67↑</td>
    <td>19.81</td>
    <td>22.34↑</td>
    <td>61.07↑</td>
    <td>16.50↑</td>
    <td>40.21↑</td>
    <td>12.76</td>
    <td>25.23</td>
  </tr>
</table>

Table 18: Performance Improvement of MCDGRAPH on QA samples across various tasks. ↑ indicates an improvement compared to the original model.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="6">Understanding</th>
    <th colspan="6">Reasoning</th>
  </tr>
  <tr>
    <th>NNu</th>
    <th>EN</th>
    <th>IDQ</th>
    <th>NQ</th>
    <th>NN</th>
    <th>CR</th>
    <th>RA</th>
    <th>SRN</th>
    <th>NR</th>
    <th>CP</th>
  </tr>
  <tr>
    <th></th>
    <th>F1</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Ace</th>
    <th>F1</th>
    <th>Ace</th>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>76.50</td>
    <td>77.60</td>
    <td>68.27</td>
    <td>94.94</td>
    <td>64.32</td>
    <td>67.40</td>
    <td>87.74</td>
    <td>86.76</td>
    <td>44.17</td>
    <td>53.77</td>
    <td>74.26</td>
    <td>75.24</td>
    <td>38.52</td>
    <td>51.10</td>
    <td>38.28</td>
    <td>50.93</td>
    <td>19.18</td>
    <td>42.06</td>
    <td>52.93</td>
    <td>42.23</td>
    <td>53.38</td>
  </tr>
  <tr>
    <td>w/MCDGRAPH</td>
    <td>89.58</td>
    <td>89.97</td>
    <td>85.80</td>
    <td>68.64</td>
    <td>95.84</td>
    <td>77.10</td>
    <td>77.25</td>
    <td>79.75</td>
    <td>80.05</td>
    <td>67.01</td>
    <td>68.35</td>
    <td>83.07</td>
    <td>81.80</td>
    <td>53.04</td>
    <td>54.47</td>
    <td>53.69</td>
    <td>64.17</td>
    <td>44.21</td>
    <td>64.12</td>
    <td>65.51</td>
    <td>60.11↑</td>
    <td>60.17↑</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>88.43</td>
    <td>71.18</td>
    <td>76.82</td>
    <td>90.23</td>
    <td>91.17</td>
    <td>91.88</td>
    <td>63.28</td>
    <td>61.84</td>
    <td>72.81</td>
    <td>73.27</td>
    <td>62.46</td>
    <td>63.62</td>
    <td>75.37</td>
    <td>76.18</td>
    <td>47.42</td>
    <td>50.40</td>
    <td>35.70</td>
    <td>56.14</td>
    <td>57.42</td>
    <td>55.94</td>
    <td>58.42</td>
    <td>54.71</td>
    <td>56.71</td>
  </tr>
  <tr>
    <td>w/MCDGRAPH</td>
    <td>88.55</td>
    <td>77.51</td>
    <td>71.08</td>
    <td>72.02</td>
    <td>90.36</td>
    <td>90.58</td>
    <td>58.85</td>
    <td>60.87</td>
    <td>75.41</td>
    <td>75.87</td>
    <td>60.23</td>
    <td>60.66</td>
    <td>82.09</td>
    <td>82.81</td>
    <td>57.01</td>
    <td>58.07</td>
    <td>60.07</td>
    <td>60.07</td>
    <td>60.07</td>
    <td>60.07</td>
    <td>60.07</td>
    <td>60.07</td>
    <td>60.07</td>
  </tr>
</table>

Table 19: Performance Improvement of MCDGRAPH on FC samples across various tasks. ↑ indicates an improvement compared to the original model.
Figure 20: Performance Improvement of MCDGRAPH (F1/Acc) on QA samples across various tasks and edge ranges.

Figure 21: Performance Improvement of MCDGRAPH (F1) comparison on FC samples across various tasks and edge ranges.

Figure 22: Performance Improvement of MCDGRAPH (F1/Acc) on QA samples across various tasks and node ranges.
Figure 23: Performance Improvement of MCDGRAPH (F1) comparison on FC samples across various tasks and node ranges.

Figure 24: Performance Improvement of MCDGRAPH (F1/Acc) on QA samples across various tasks and average degree.

Figure 25: Performance Improvement of MCDGRAPH (F1) comparison on FC samples across various tasks and average degree.

Reviewer paper summaries: This paper introduces VGCURE, a comprehensive benchmark for evaluating Large Vision Language Models (LVLMs), covering 22 tasks aimed at examining the fundamental graph understanding and reasoning capabilities of LVLMs. VGCURE evaluates the core capabilities of LVLMs through various challenges, including 9 graph understanding tasks and 13 graph reasoning tasks. Furthermore, the paper proposes MCDGRAPH, a fine-tuning framework that enhances LVLMs via three self-supervised learning tasks, demonstrating that MCDGRAPH significantly improves the performance of LVLMs on both the fundamental tasks in the benchmark and downstream graph learning tasks.
Summary of strengths: VGCURE represents the first benchmark focused on evaluating the graph understanding and reasoning capabilities of LVLMs, marking a significant step towards advancing their comprehension and reasoning capabilities. VGCURE systematically assesses these capabilities across 22 diverse tasks, comprising more than 20,000 samples and various graph types, making it a robust and comprehensive benchmark. The proposed MCDGRAPH framework also addresses a clear gap identified through the benchmark. The three self-supervised tasks introduced provide substantial improvements across numerous tasks, demonstrating the framework’s robustness.
Summary of weaknesses: **W1** The VGCURE benchmark focuses exclusively on simplified graphs with 7–15 nodes, but I’m curious what would happen if it encountered challenges such as large-scale graphs. This may limit our understanding of how these models might scale to larger, more complex graphs common in real-world applications.

**W2** It would be great to delve deeper into the taxonomy, providing examples of each type of error for different task categories. Analysing these could provide more targeted insights.

**W3** While the MCDGRAPH framework demonstrates a significant improvement, I don’t see any ablation studies or thorough evaluation on the individual contribution of each of the three self-supervised tasks. Could we omit one of them? If so, how would it impact performance? What led to the choice of these three tasks? Addressing these questions would be extremely helpful.
Comments, suggestions and typos: **S1** While MCDGRAPH improves performance on the proposed dataset, could it be transferred to other structured data beyond graphs, such as hierarchical trees or in understanding relationships between elements?

**S2** It would also be helpful to analyse some complex cases beyond the 7–15 node range. A few will be very helpful. See **W2**

See **W2**

See **W3**
Overall assessment scores: 3.5