Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024)

Abstract: August 11-16, 2024 ©2024 Association for Computational Linguistics Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models Qihang Ai1∗, Jiafan Li2,3∗, Jincheng Dai, Jianwu Zhou, Lemao Liu, Haiyun Jiang5†, Shuming Shi Beijing Institute of Technology Institute of Software, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China Zhejiang University Sun Yat-sen University Wechat AI, Tencent, China Tencent AI Lab, China qihang-ai@bit.edu.cn, lijiafan23@mails.ucas.ac.cn, jincheng_dai@zju.edu.cn, 1120211477@bit.edu.cn, redmondliu@tencent.com, jianghy66@mail.sysu.edu.cn, shumingshi@tencent.com Abstract

heading: 1 Introduction
text: Graph is an important form of structured data, which is capable of storing and representing the complex relationships between objects. Downstream tasks performed on graph data include node classification (Xiao et al., 2022), link prediction (Zhang and Chen, 2018), graph reasoning (Chen et al., 2020) etc. In early days, graph learning is commonly modelled using graph networks, with a wealth of seminal literature supporting this approach. In recent years, graph neural networks(Li et al., 2015; Dai et al., 2018; Battaglia et al., 2018; Fan et al., 2019; Zhang and Chen, 2018) provide a more flexible and effective means of dealing with
*Equal contribution †Corresponding author(jianghy66@mail.sysu.edu.cn)
the diversity and complexity in graph structures. By propagating and aggregating node information within the graph structure, GNNs capture complex graph features.
With the rise of Large Language Models (LLMs), there has been extensive research in the field of natural language-guided interactive graph data understanding, showcasing two main strategies. The first approach constructs prompts by explicitly representing graph structures in a sequential format. Constructing prompts for LLMs involves innovative techniques such as self-prompting (Guo et al., 2023), graph-syntax trees (Zhao et al., 2023b), natural language descriptions of graphs’ structures and features (Ye et al., 2023b), and graph-structure prompting in various modalities (Das et al., 2023). The second strategy transforms GNN-learned features into LLM-comprehensible tokens. Models like GIT-Former (Liu et al., 2024a) GIMLET (Zhao et al., 2023a) and MolCA (Liu et al., 2024b) consider graph as a mode and integrate all modality data into a unified latent space.
However, due to the inherent differences between graph structures and language models, current integration methods face significant limitations. First, sequence-based prompt representations inevitably lose crucial graph structural information, which is vital for complex graph understanding and reasoning(Ge et al., 2023; Li et al., 2024). Second, integrating GNN-learned features into LLMs presents alignment challenges, as the representation learning space of GNNs cannot be easily mapped onto the token space of LLMs through simple function mappings(Xue et al., 2023).
Considering these limitations, this work introduces a new paradigm for graph data understanding based on VLMs. Our basic idea is to transform graph data into images, either through their natural representations like maps or visualization
7485
methods†. This is followed by utilizing an image encoder to interpret the visual information, thereby understanding the semantic information of nodes and relationships within the graph data. The final step of this process involves the fusion of image and text encoders, which is inherently supported in specific VLMs (Kim et al., 2021; Li et al., 2022a; Liu et al., 2023), offering a novel approach to graph data comprehension. Figure 1 shows an example of graph data access based on visual language models.
Utilizing visual language models for graph data comprehension offers friendly and natural interaction, reducing manual labor costs, and unifies diverse graph structures globally, handling various graph-related tasks through natural language instructions. Leveraging state-of-the-art visual language models trained on extensive datasets facilitates knowledge transfer and promises high performance potential in this paradigm of graph understanding and reasoning.
This work introduces a novel dataset on various graph formats for multimodal graph understanding and reasoning in English and Chinese. Instructions involve simple queries and complex multi-hop reasoning on the graphs, with difficulty levels varying from simple to medium and difficult. Additionally, based on this dataset, we conducted a comprehensive evaluation of current open models from various perspectives. In conclusion, the current models exhibit significantly better performance in English compared to Chinese. Even the most powerful models like Genimi or GPT-4V achieve accuracy rates below 30%. Finally, in fine-tuning LLaVA and MiniGPT-4 on our dataset, experimental results revealed that the fine-tuned LLaVA model on English&Chinese datasets outperformed base-
†https://www.ownthink.com/
line LLaVA on both English and Chinese dataset and achieved comparable performance with Genimi in Chinese. To enhance the open-source models’ ability in recognizing Chinese characters, we also proposed incorporating OCR data transformed into instruction data into our Chinese datasets. The model refined through this approach exhibited significant improvements on the Chinese dataset after fine-tuning.
Our main contributions are as follows:
• We constructed a dataset for multimodal graph understanding and reasoning, providing instructions for graph-related questions and corresponding ground truth answers.
• We evaluated the capabilities of public VLMs in graph learning, analyzing their performance and limitations across various graph tasks.
• We fine-tuned the open-source models LLaVA and MiniGPT-4 using the constructed dataset, resulting in significant improvements on both Chinese and English datasets.

heading: 2 Related Work

heading: 2.1 Integrate LLMs with Graph Learning
text: With the emergence of LLMs, the applications in graph-related tasks have surpassed traditional GNN-based methods. Currently, there are three main ways of integrating large language models with graph data.
LLMs-as-Enhancers. This research line aims to enrich the node attribute and relation representations using the capabilities of LLMs. For example, TAPE (He et al., 2023) leverages the knowledge of large language models to generate high-quality node features, thereby enhancing the quality of initial node embeddings in GNNs. KnowledgeEnhanced Augmentation (KEA) (Chen et al., 2024) enriches text attributes by providing additional information, stimulating LLMs to generate lists of knowledge entities and their descriptions and encoding them through fine-tuned PLMs and deep sentence embedding models.
LLMs-as-Predictors. This method represents graph node attributes and structures in the form of prompts, using LLMs to directly generate predictive patterns. For instance, InstructGLM (Ye et al., 2023b)replaces GNN’s predictors with LLMs.
GPT4Graph (Guo et al., 2023) adopts graph description language of prompt engineering, improving collaborative working methods in various situations. GraphGPT (Tang et al., 2023) aligns LLMs with graph structural knowledge through a graphguided tuning paradigm.
LLM and Graph Collaboration. Aligning the embedding spaces of graph models and LLMs achieves the integration of graph and text modalities. Text2Mol (Edwards et al., 2021) proposes a cross-modal attention mechanism using transformer decoder for early fusion of graph and text embeddings. "Think on Graph" (Sun et al., 2024) presents the "LLM ⊗ KG" paradigm, a new approach integrating LLMs and knowledge graphs (KG). It treats large language models as agents for interactively exploring entities and relations in KGs. RLMRec (Ren et al., 2023) suggests aligning the semantic space of LLMs with the representation space of collaborative relational signals in recommendation systems through contrastive modeling.

heading: 2.2 Multimodal Large Language Models
text: Recently, with the rapid advancement of large language models and their demonstrated powerful interactive capabilities, a new paradigm has been proposed for the vision-language tasks. Based on an encoder-decoder framework and utilizing LLMs as decoders (Liu et al., 2023; Li et al., 2023; Wang et al., 2023; Su et al., 2023), Multimodal Large Language Models (MLLMs) exhibit significant multimodal capabilities across various benchmarks. This approach leverages cross-modal transfer, enabling the sharing of knowledge between language and multimodal domains (Zhu et al., 2023).
Visual instruction tuning (Liu et al., 2023) is employed to develop a MLLM that is adept at general-purpose visual and language understanding. LLaVA (Liu et al., 2023) extends the
self-instruction (Wang et al., 2022) approach to the multimodal field by translating images into texts with captions and bounding boxes. Besides LLaVA, many other powerful MLLMs have also emerged built upon LLMs (Yin et al., 2023), including the open-source ones, e.g., MiniGPT-4 (Zhu et al., 2023), mPLUG-Owl2 (Ye et al., 2023a), Multimodal-GPT (Gong et al., 2023), and the commercial models, e.g., GPT-4V (OpenAI, 2023) and Gemini (Team et al., 2023).

heading: 3 Dataset
text: We constructed two multimodal instructionfollowing datasets on graphs in English and Chinese respectively. Each dataset includes five types of graphs: knowledge graph, route map, mind map, flowchart, and Gantt chart. The images were crawled from search engines and filtered for relevance and accuracy by humans. The construction of multimodal instructions involves two steps. The first step prompts GPT-4V to generate candidate instructions and responses for each image. The second step involves human evaluation and annotation to ensure the validation of instructions and accuracy of responses. For images containing mixed language content, such as English text in Chinese images or vice versa, we have already discarded these during the manual filtering process to ensure language consistency in our dataset.

heading: 3.1 Data Annotation
text: After manually filtering out unclear and irrelevant images which don’t belong to any graph type†, We obtained a total of 2,807 images. Among these, 1,493 are Chinese images, including 517 knowledge graphs, 335 mind maps, 584 route maps, 49 flowcharts, and 8 Gantt charts. The 1,314 English
†Images containing mixed language content, such as English in Chinese images or vice versa, were discarded.
images are comprised of 251 flowcharts, 208 Gantt charts, 257 knowledge graphs, 216 mind maps, and 382 route maps. The proportion of Chinese to English annotated images is approximately 1.14:1.
Given a valid image, we first annotated it by GPT-4V under our elaborated prompts as shown in Figure 14, during which Chinese images were annotated using prompts in Chinese and English images were annotated with English prompts. For each image, under our prompt, GPT-4V usually posed three questions: simple, medium, and complex, as shown in Figure 19 and Figure 20 in Appendix A.4.
Following this automatic annotation process, we conducted an evaluation of the relevance of the instructions generated by GPT-4V in relation to the images. This critical analysis aimed to ascertain the extent to which GPT-4V’s output aligned with the images. In our in-depth examination, the instructions generated by GPT-4V might be invalid due to either of the following two aspect:
• Instructions are completely unrelated to the image’s content.
• Instructions are related to the image’s theme but involving nodes or edges not present in the image.
For each image with several instruction-response pairs, we then implemented a manual fineannotation strategy. Valid instructions were retained while invalid ones were discarded. Subsequently, we checked the correctness of responses corresponding to the valid instructions and corrected any incorrect responses. In cases where all instructions for an image were deemed invalid, we manually supplemented two instruction-response pairs: one simple instruction and one complex instruction. This approach ensures a high standard of instruction-response pairs in our benchmarks.

heading: 3.2 Data Statistics and Analysis
text: Table 1 presents the statistical information of our multimodal instruction-following benchmark dataset. All dataset construction processes undergo
rigorous manual evaluation to ensure high quality. To demonstrate this, we asked an unseen annotator to randomly select a sample of 50 Chinese and English samples (i.e., image, instruction, response), respectively. Each image was subjected to a rigorous quality assessment within the selected samples. Our analysis revealed that out of the 100 samples, an impressive 97 were classified as qualified, highlighting the dataset’s high standard of clarity and uniqueness.

heading: 3.3 Evaluation Protocols
text: To facilitate the evaluation of model performance using our graph vision-language dataset, we provided two evaluation protocols. The first is a manual assessment of model accuracy, where human evaluators were asked to determine whether the model response satisfies the instruction conditioned on an image (Zhu et al., 2023). The second evaluation protocol we used is a GPT-assisted visual instruction assessment. Building on a previous work (Liu et al., 2023) that employed GPT-4 (textonly) to score answers by comparing the output of GPT-4 with that of another model, we have adopted a similar approach for GPT-4V evaluation and designed an English prompt shown in Figure 16 and a Chinese prompt in Figure 17 in Appendix A.3.

heading: 4 Evaluation on Public Vision-Language Models
text: 4.1 What Abilities Do We Focus On?
The initial phase of our study involved a manual evaluation of GPT-4V (OpenAI, 2023), Gemini (Team et al., 2023), LLaVA (Liu et al., 2023), and MiniGPT-4 (Zhu et al., 2023)’s capability to generate and follow instructions in response to images.
Instruction Generation Ability. Given an image of graph, we first evaluate the capability of VLMs on generating graph-related instructions. We determined the validity of each instruction as described in Section 4.3.
Various Instruction Following Abilities. Given a valid instruction of an image, we also evaluated the correctness of the response generated by existing VLMs. We assessed various ability dimensions, including simple and intermediate-level instruction following ability, complex instruction following ability, multi-hop reasoning ability, robustness to noise, performance across different information densities, and an ablation study on different types of graphs. These dimensions were explored by providing corresponding instructions (see Section 4.4, 4.5, 4.6, 4.7, 4.8, 4.9).

heading: 4.2 Evaluation Settings
text: Instruction Validity. We sampled a set of 50 images for the experiment, encompassing five types of graphs with a balanced distribution between English and Chinese. The prompts we used are shown in Figure 14 in Appendix A.3. As mentioned in Section 3.1, there may exist two types of invalid instructions. The distribution of valid rate for different models is detailed in Table 2.
Response to Various Instructions. For each ability dimension, we randomly sampled 25 instructions for evaluation from our dataset. To enhance the quality of responses, we utilized pre-established prompts as detailed in Figure 15 in Appendix A.3.
The evaluation metric focused on the accuracy of the VLMs’ responses to the instructions. This was determined by human evaluators. To provide a comprehensive understanding, we separately evaluated the performance of VLMs on images with content in Chinese and English †. The assessment
†In cases where images contain both Chinese and English content, classification as Chinese or English was determined
result is presented in Table 3.

heading: 4.3 Instruction Generation Ability
text: During the Chinese image instruction generation, we observed low performance for all the models, with less than 40% valid rate. The main reason for the low performance was the misrecognition of node names. Specifically they struggled to correctly identify the Chinese characters within them. As a result, these models tend to generate erroneous node and relation names based on their internal knowledge. We illustrate this hallucination by an example in Figure 21 in Appendix A.4.

heading: 4.4 Simple and Medium Instruction Following Ability
text: As mentioned in Section 3.1, the instructions of our dataset were initially generated by GPT-4V, where each sample was additionally labeled with simple, medium, or complex. The accuracy results for simple and medium instructions are shown in Table 3.
MiniGPT-4 lacked basic cognition in discerning graph types and was unable to differentiate between types of graphs. A common challenge faced by all these models, including GPT-4V, Gemini, LLaVA, and MiniGPT-4, was their struggle with identifying the number of nodes in graph-type data.

heading: 4.5 Complex Instruction Following Ability
text: Complex instruction following tasks require understanding intricate commands but may result in simple answers. The corresponding evaluation results are shown in Table 3.
based on the predominant language used in the nodes and edges of the graph.
When processing images containing English content, the performance of Gemini, LLaVA, and MiniGPT-4 in complex instructions surpasses their accuracy in medium instructions. This improved performance can be attributed to the nature of complex instructions, which some encompass openended questions relying on coarse-grained visual information. Such scenarios play to the strengths of LLMs, leveraging their robust linguistic capabilities to effectively interpret and respond to these complex instructions.

heading: 4.6 Multi-hop Reasoning Ability
text: As mentioned in Section 4.5, complex instructions may result in simple answers. In contrast, multihop reasoning instructions involve linking several logical steps, usually leading to more elaborate responses. This posed a significant challenge for VLMs in providing completely accurate responses to multi-hop reasoning tasks. The corresponding evaluation results are shown in Table 3.

heading: 4.7 Robustness to Noise
text: As mentioned in Section 3.1, we have excluded blurry images from our dataset. In this part, we have additionally collected a set of noisy images, comprising 25 instructions each for both Chinese and English content. The corresponding results are shown in Table 3.
MiniGPT-4 occasionally struggled with noisy images, which was evident in instances of language confusion, such as responding in English to prompts and images that are in Chinese, and producing meaningless repetitive answers. GPT-4V tended to offer vague responses or guiding suggestions, stating that due to image quality limitations,
it couldn’t give valid responses.

heading: 4.8 Performance across Different Information Densities
text: Different images can contain varying content and details, i.e., different information densities. Information density, typically judged by the number of nodes and edges in a graph, ranged from sparse to dense and is assessed manually. This section explores how the VLMs perform with images of varying information densities. The corresponding evaluation results are shown in Table 3 and GPT4V outperformed other VLMs in processing image information densities in both English and Chinese contexts.

heading: 4.9 Ablation Study on Different Types of Graphs
text: To comprehensively assess how the VLMs perform across various types of graphs, we randomly sampled 25 instructions for each graph type, with the related results being displayed in Table 4.
In the analysis of images containing English content, all four VLMs displayed a relative familiarity with the structure of knowledge graphs, achieving their highest accuracy in this category. In route maps, GPT-4V not only led in accuracy for both Chinese and English content images but also provided more precise and standardized responses, using both absolute directions(north, south, east, west) and relative directions(up, down, left, right). In contrast, Gemini used only absolute directions, while LLaVA and MiniGPT-4 depended entirely on relative directions.

heading: 5 Experiment

heading: 5.1 Baseline Models and Settings
text: MiniGPT-4 consists of a vision encoder with a pretrained ViT (Dosovitskiy et al., 2021) and QFormer (Li et al., 2022b), a single linear projection layer, and an advanced Vicuna (Chiang et al., 2023) large language model. During the training process, only the parameters of the Q-Former and linear projection layers are fine-tuned, while the parameters of the language and visual models are kept unchanged.
LLaVA uses language-only GPT-4 to generate multimodal language-image instruction-following data. This approach allows LLaVA to connect a vision encoder and a language model for generalpurpose applications. The fine-tuning stage involved updating both the pre-trained weights of the projection layer and LLM in LLaVA, while keeping the vision encoder fixed.
Both LLaVA and MiniGPT-4 were trained for 10 epochs. Performance was evaluated after all the intervals, and the model demonstrating the best performance was then selected for data generation.
Experiment Details We divided the training set and the test set according to the ratio of 4:1. Specific data is shown in Table 1. We used the English training sets, the Chinese training sets and the combination of Chinese and English training sets for fine tuning, respectively.

heading: 5.2 Results

heading: 5.2.1 Qualitative Analysis
text: After fine-tuning, the model demonstrates many advanced abilities compared to the baseline model.
Here, we will analyze and explain in detail with specific examples based on the best-performing model, named LLaVA FT. These cases are shown in Appendix A.1.
In Figure 2, LLaVA FT effectively utilizes visual cues to discern pertinent information within images, enabling accurate interpretation of depicted processes. In constrast, the baseline LLaVA model relies solely on contextual understanding, thereby neglecting crucial image details in its responses. The scenario depicted in Figure 3 highlights LLaVA FT’s proficiency in accurately arranging tasks based on their sequence in Gantt charts, resulting in precise identifications. Conversely, the baseline LLaVA model struggles to precisely identify the sequence and frequently misinterprets presented tasks. And Figure 4 demonstrates LLaVA FT’s proficiency in identifying and categorizing relationships within a knowledge graph, while Figure 5 showcases its enhanced ability to determine the quantity of nodes within a given graph, providing specific and relevant responses.
These cases demonstrate the enhanced capabilities of the fine-tuned model in understanding flowcharts, Gantt charts, knowledge graphs, and mind maps. The fine-tuned model’s ability to answer questions based on graphical data in both English and Chinese has been significantly improved.

heading: 5.2.2 Quantitative Analysis
text: We used the evaluation method described in Section 3.3. In Manual Evaluation, we randomly sampled 100 instructions from the test sets. Human evaluators assessed the correctness and reasonableness of the model-generated responses. In GPT-assisted Evaluation, we calculated the average value of GPT-
4V scores for each model to measure the effect of fine-tuning models.The results were compared with the baseline and are presented in Table 5.
After fine-tuning, MiniGPT-4 improved by 8%, while LLaVA improved by 15% over the baseline. Mixed-language fine-tuning in both Chinese and English showed similar effectiveness to singlelanguage fine-tuning. The most effective model, LLaVA+English&Chinese, achieved significant improvements, with 28% accuracy on the Chinese test set, 54% on the English test set, and 39.4% overall. Particularly noteworthy is its performance on the Chinese test set, matching that of Gemini.

heading: 5.3 Ablation Study

heading: 5.3.1 OCR Instructions
text: In our evaluation in Section 4.4, both MiniGPT4 and LLaVA exhibited a significant decrease in accuracy when providing responses to Chinese images. To enhance their ability to recognize Chinese characters in image datasets, we have devised targeted OCR instructions. We utilized the training set provided in the Chinese Scene Text Recognition Technology Innovation Competition.† The training dataset consists of 212,023 images containing textual information. We tokenized OCR text from images and selected 2,224 closely aligned images for fine-tuning. We created the instruction set by using the prompt "Please recognize the text in the image" and pairing it with the text recognition results from the training dataset. The prompts we used are shown in Figure 18 in Appendix A.3. Subsequently, we combined the OCR instructions with Chinese fine-tuning instructions, adhering to a data augmentation model, to enhance the OCR capabilities. We separately added the OCR instruction to the Chinese fine-tuning dataset and the English&Chinese fine-tuning dataset.
†https://aistudio.baidu.com/datasetdetail/8429

heading: 5.3.2 CoT Augumented
text: In our model, we generated a series of intermediate reasoning steps, referred to as Chain-ofThought(CoT), to enhance the capability for complex reasoning problems. For complex questions, it is necessary to integrate information from both text and images for reasoning, so we used Multimodal CoT (Zhang et al., 2024), which consists of two stages: Rationale Generation and Answer Inference. Initially, caption text and visual features are utilized to obtain a more accurate Rationale (R). Subsequently, the final Answer is derived from R, along with text and visual features.
Our Ablation results are presented in Table 6. There was a noticeable improvement in both the Chinese test set and the Chinese-English mixed test set after incorporating OCR fine-tuning. Particularly, on the mixed-language test set, the performance increased to 46%, marking a substantial improvement of 21% compared to the baseline LLaVA model. The multimodal CoT approach resuls in some improvement compared to the original model. However, there is still a need to design a more suitable multimodal fusion method tailored for graph-related problems, involving the identification of nodes and relationships within the graph.

heading: 6 Conclusion and Future Work
text: This study focuses on natural language-guided interactive graph data understanding, distinguishing itself from traditional graph network modeling by utilizing VLMs to enable flexible interactions with graph data. We constructed a benchmark dataset to address the lack of available data in this domain, further supporting the development and evaluation of models in this field. By fine-tuning LLaVA and MiniGPT-4 with our dataset, we observed notable improvements in performance on both Chinese and English datasets, marking a significant advancement in the field of graph data understanding. In the future, we will advance this work from the following two directions. (1) Designing dedicated im-
age encoders to capture the pixels related to nodes and edges more sensitively. (2) Supporting large graph (with thousands of nodes or edges) understanding by splitting a big image into a sequence of sub-images. It requires that the VLMs have the ability to understand the internal content in a single image as well as the contents across different sub-images, for a more accurate understanding of the original large graph content.

heading: 7 Limitations

heading: 7.1 Imbalanced Dataset
text: While there is an abundance of available graphic data resources, obtaining them in the form of images can be particularly challenging. In our constructed dataset, the samples for Gantt charts and flowcharts are notably scarce. This scarcity is primarily due to the limited availability of these specific types of images on the internet, resulting in fewer instructions in our dataset compared to the other three graph types. Despite our efforts to manually filter out noisy images, as highlighted in Section 3.2, where we found that out of 100 randomly sampled images, an impressive 97 were deemed of high quality, our dataset may still contain images of duplication and blurriness. Hence, there’s a necessity to explore more reliable automated data cleaning pipelines.

heading: 7.2 Hallucination
text: Our model is built upon LLMs and inherits its limitations. It may suffer from illusions when faced with non-existent knowledge. When querying nodes and relationships in images, the model may respond with nodes not present in the image, relying on its own common sense for answers. As shown in Figure 21 in Appendix A.4, this example demonstrates the model’s hallucination in interpreting Chinese instructions, where it incorrectly recognizes "Apply for a refund" as "Select a product" .

heading: A Appendix
text: A.1 More Qualitative Results Here we list some qualitative results. Yellow highlights correct but irrelevant information. Green highlights the correct response and Red highlights the wrong response.
A.2 More Quantitative Results A.2.1 Manual Evaluation Here we list some quantitative results. Yellow highlights correct but irrelevant information. Green highlights the correct response and Red highlights the wrong response.
A.2.2 GPT-assisted Evaluation Here, we list some GPT-assisted score results of the VLMs, including the best-performing fine-tuned models.
A.3 Prompts
The prompt used to generate image-based conversation from GPT-4V is shown in Figure 14. Under this prompt, GPT-4V usually posed three questions for each image: simple, medium, or complex.
route map. Yellow highlights the expert identification. Green highlights ensuring correctness and Red highlights the response form.
The prompt used for prompting VLMs to answes the instructions based on an given image is showed in Figure 15.
route map. Yellow highlights the expert identification. Green highlights ensuring correctness.
The prompt used for a GPT-4V-assisted visual
instruction assessment are shown in Figure 16 and Figure 17.
A.4 Data
Figure 19 presents an example of a knowledge graph with English content, where the instructions and responses were generated by GPT-4V following the specified English prompt.
Similarly, Figure 20 illustrates a flowchart with Chinese content, with instructions and responses produced by GPT-4V in accordance with the adapted Chinese prompt.
structions. Red highligthts the hallucination in the response of the model. The meaning of the Chinese characters in the first step is "Apply for a refund" in Chinese while LLaVA recognized it as "Select a product" in Chinese.

heading: Future Work
text: In the future, we will advance this work from the following two directions. (1) Designing dedicated im-
age encoders to capture the pixels related to nodes and edges more sensitively. (2) Supporting large graph (with thousands of nodes or edges) understanding by splitting a big image into a sequence of sub-images. It requires that the VLMs have the ability to understand the internal content in a single image as well as the contents across different sub-images, for a more accurate understanding of the original large graph content