GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
Authors: Yanbin Wei
Shuai Fu
Weisen Jiang
Zejian Zhang
Zhixiong Zeng
Qi Wu
James Kwok
Yu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph Reasoning
Visual Question Answering
Large Multimodal Model
Primary area: other
TL;DR: Pioneeringly establish a vision-language graph reasoning setting for VLMs to compete with LLMs, showcasing vision's benefits for language-based graph reasoning, and provide a framework to derive graph VQA benchmarks from existing data.

Abstract: Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish  $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
Yanbin Weil! Shuai Fu 1.3, Weisen Jiang Zejian Zhang', Zhixiong Zeng', Qi Wu², James T. Kwok?, Yu Zhang'
Department Computer Science and Engineering. Southern University of Science and Technology Department of Computer Science and Engineering, Hong Kong University of Science and Technology
Australia Institute for Machine Learning, University of Adelaide "Tencent [yanbin. ust, fus. jayce, zejianzhang33, yu. zhang. ust, baysonkong)@gnaal com barretzeng@tencent. com, qi wu01 @adelaide edu. au, jameskåcse ust hk
Abstract
Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in the textual format, they overlook the rich vision modality, which is intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures visual images (i.c., visual graph) are still unexplored. To fill the gap, we innovatively propose end-to-end framework, called Graph vIsual and Textual IntegrAtion (GITA), which incorporates visual graphs into general graph reasoning. Besides, construct the Graph-based Vision-Language Question Answering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs on general graph reasoning. Moreover, experimental results demonstrate the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.
Introduction
60 social network analysis [9]. and knowledge graph reasoning [72] 46 62). Various architectures have been developed, from shallow embedding methods 6 53 to advanced Graph Neural Networks
Graph reasoning tasks are pivotal in domains such as recommendation systems 25
(GNNs) [37 (4) and graph Transformers [71] 40, g. While these models excel in graph reasoning tasks, they often lack generalizability, flexibility, and neer-freenllnness Achieving good performance with these models typically requires domain-specific tuning, which limits their abilities to generalize across different domains. Additionally, these models struggle to handle diverse tasks with the same architecture. Each task often requires specialized design, including task-specific data processing and decoder, leading to limited flcxibility. Lastly, unlike the Large Language Models (LLMs) that engage in conversations with users, these models less explainable and user-friinne -rienfrienlly.
In contrast, LLMs have shown great generalization capabilities across wide variety of reasoning 33 by encapsulating task-specific demands within cohesive and interpretable mechanism text prompts, under a unified architecture with minimal domain- -ppcciff adjustments. These advantages have sparked investigations into the potential of LLMs for graph reasoning. Recent developments lend credence to the notion that LLMs can indeed interpret and manipulate graph-
tasks 61 67
74
32
"Equal contribution. CCoresponning authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

structured data through textual representations. For example, InstructGLM [66], GPT4Graph [2]] and LLMtoGraph 44] convert graphs into textual descriptions and then use these descriptions paired with queries to enable L.MMs to generate accurate responses for graph reasoning tasks. Furthermore, the introduction of benchmarks such as GraphQA [44 and NI.Graph [59] is a testament to the growing interest in evaluating LLMs' effectiveness on graph reasoning tasks framed in natural languages.
Despite the development of numerous methods and benchmarks for graph reasoning on LLMs, they often overlook the valuable vision modality, which a natural means for humans to comprehend structural information and has demonstrated its success in various visual reasoning scenarios [30, 68
69 (4 3 [54]. Consequently, the following questions arise: (1) Can incorporating visual information be beneficial for general graph reasoning scenarios? (2) If so, how can we effectively integrate the vision modality into graph reasoning? To the best of our knowledge, these questions remain
unexplored.
To answer them, we first propose an end-to-end framework called Graph to vlsual and Textual Atin (GITA ill that systematically integrates visual information into instruction-baser graph reasoning, by rendering graph structures customized visual images which are called visual graph. Specifically, the GITA framework has four components: a graph visualizer for generating visual graphs, a graph describer for producing textual descriptions of the graph structure, a task-based questioner that organizes the description and requirements of the current task into prompt instruction, and Vision-Language Model (VLM) to perform vision- languge graph reasoning. In the proposed GITA framework, visual information can be incorporated into many tasks with explicit or implicit graph structures, without sacrificing its versatility, flexibility, or frenfriendlliness Besides, since there is no dataset for vision-language dataset for general graph reasoning purposes called Graph-based Vision-Lange Langun-Languag Question Answering (GVLQA) based on the proposed GITA framework. The GVLQA dataset consists of 526K instances covering seven representative graph reasoning tasks, aiming to thoroughly evaluate the structure-based graph reasoning abilities of VLMs and LLMs. Extensive experiments on the GVLQA dataset and five real-world datasets demonstrate the effectiveness of the proposed GITA model. Furthermore, we delve into the effects of visual graph augmentation strategies and find that layout augmentation can dramatically boost vision-bace based graph reasoning performance.
general graph reasoning capabilities, we construct the first
Our main contributions are summarized as follows.
We introduce an end-to-end GITA framework, innovatively integrating vision modality to boost the graph reasoning abilities of language models.
We establish GVLQA, the first vision-language question-answe answering dataset for general graph reasoning purposes. It can be used to thoroughly evaluate the dtructure-based graph reasoning abilities of LLMs//LMM and can also be used as pretraining data to boost the performance of downstream tasks.
Extensive experiments on benchmark datasets across various graph reasoning tasks demon- strate the effectiveness of the proposed GITA framework and the benefits of layout augmen- tation on visual graphs.
2 Related Work
Graph Reasoning. Graph reasoning 63 aims to answer questions based graphs, which involves
utilizing graph structures to guide the reasoning process to generate answers. Graph reasoning has a
wide variety of applications in social network analysis [47 41), bioinformatics [31] 18) chemistry
[19] physics [4). knowledge graph reasoning [6. and recommendation systems (39. 26 Many graph
reasoning methods have been proposed. Early attempts (3] learn node and edge representations
through shallow modules, which may have only limited expressive power. Graph Neural Networks
(GNNs) such as GCN (37, GAT (58), GraphSAGE (33) MPNN [19], and GIN 64 use message-
passing paradigm
19 to model graph dependencies and update node features. Transformer based
Project Homepage: v- graph. github.
1 'Code Repository: httppp///gthaaaaccc////eaaaanaan
5 'Dataset: http:////gggggggeeooooooooolllla

---

graph models 71 40 8 further propose to use self-attention to increase the expressiveness and long- range dependency. However, as discussed in Sec I these models may exhibit limited generalizability. flexibility, and uree-frien
LLMs on Graph Reasoning. There have been many attempts use LLMS in graph reasoning. Depending on how they align the input spaces of graphs and LLMs, we categorize them into two types: Graph-to-text and Graph-to-token, Graph-to-text methods transform a graph into textual descriptions, which are concatenated with the instructions and fed to the LLM for querying, For example, InstructGLM [66] uses natural language describe the graph and proposes instruction prompts to fine-ture tune-tne the LLM. He et.al [27] applies LLMs to explain graphs for training GNNs, while Chen et.al [10] treat LLMs as enhancers exploit text attributes or as predictors for node classification on text-attributed graphs. GPT4Graph [21] and LLMtoGraph [44] convert graphs
into specific code or natural language formats by the powerful ChatGPT
48 49]. On the other
hand, Graph-to-token methods include GraphGPT (55), GraphToken 50) and LLaGA (9] For these methods, the graph is represented as a specially designed token sequence, which is projected or merged into the LLM's token space for text-based reasoning. However, none of the aforementioned methods represent the graph structure information as images, highlighting the uniqueness of the proposed GITA framework and GVLQA dataset.
Large Vision- Vision-Language Models. Large VLMs have significantly expanded the cognitive abilities of LLMs by integrating the vision modality to address Aision-language tasks. Many methods have been proposed. Some early explorations like Flamingo (2). CLIP (ST] and BLIP-2 [43 use a visual encoder for processing images and align the visual and textual embeddings. Subsequent models like LLaVA 45 and MiniGPT-4 (75 combine visual and textual inputs single LLM for solving multimodal tasks. InstructBlip lip 11 proposes an instruction-awee aware query transformer and trains vision- language model by instruction tuning. However, despite progress in a wide range of vision- language tasks 770 [6], using visual information in graph reasoning remains overlooked. We take the first step in this field, pushing the boundaries of VLMs in graph reasoning.
3 GITA: Graph to Visual and Textual Integration
3.1 Preliminary
Graph Reasoning. In traditional graph reasoning settings, models typically rely on two main inputs: (i) the graph structure G = {C, E}, where C' and E are the set of vertices and edges, respectively; (ii) the task requirement T, encompassing specific operations or questions pertaining to the graph. Based on the information provided in G and a specific task requirement T, models are expected to output a reasonable answer A. On the other hand, in the context of instruction-based graph reasoning methods, it is necessary to convert these inputs into textual form. This transformation facilities graph reasoning within natural language, allowing for improved interpretation and harnessing the formidable reasoning capabilities of large language models.
3.2 Architecture
Overview. Different from the above graph reasoning methods, we propose a Graph to Image- -xt Assistant (GITA), which is the first attempt to perform graph reasoning in a vision-text-based manner. GITA comprises four pivotal components: a task-agnostic graph visualizer V, a graph describer D, a task-specific questioner Q, and a VLM reasoner Rg. as illustrated in Figure I Firstly, V and G are designed to produce visual depictions (i.e., visual graphs) and textual descriptions of the graph structure inputs, respectively. Then, given the task requirement T and the textual description produced by D. Q is designed to form a task-specific query. Finally, R& receives the visual input IG from V based on the visual graph and the textual input Q7 from Q, then generates answers A in natural language. In the following, we introduce the four components in detail.
Graph Visualizer. The role of the graph visualizer is to generate visual graphs from structural graphs. The image representation of a structural graph is not unique, as there can be variations in many aspects, such as backdrop colors, layouts, and node shapes. These variations may enhance the robustness of models through effective training but simultaneously increase the learning difficulty for models. Therefore, balancing consistency and variety is necessary during the graph visualization process. This trade-off is reflected in our design of graph visualizer, by maintaining consistency in
3

---

Traditional L.M Solutions
Framcoook
Reppener
Graph Vinsaloee
Viural Graghh
Coolgaratios
1000
Bad Buige Styles
VIN
Greph Desrriptioss
Inovertions
Testeal Desctipiia
Teakopecilec Querr aalld
Gragh Dnerrieer
185)
equxas::
Telloos pils aud Oandaat
Dasiikag
calkeced
it iepulaged centing
0.6
Treplem
Tack-haned
Qualieer
numplc
Fless aende aocotii sipe onder
Test Kopuealile
Dusa
Dura
Tel-agansia dootiption
Anerptioes
llgr))
Ouep Foaton
Figure 1: The architecture of the GITA framework with comparison to existing LLM solution.
basic image styles common to general images (i.e., size, resolution, backdrop) and only introducing customizable variations in four geapherelated image styles unique to visual graphs (i.e., layout, node shapes. node outline styles, and edge thickness). Graph visualization in V can be formulated by the following equation:
l = VVG,TT,,,),
where 1ا denotes the visual graph derived from graph G, while I' and A are the fixed basic image styles and customizable graph-related image styles, respectively.
Visualizing the entire graph can be challenging when the number of nodes or edges is very large. affecting the clarity of the images. To address this, our graph visualizer adopts the standard strategy of k-hop subgraph sampling. Specifically, k-hop subgraph sampling for node in the of vertices C involves selecting a subgraph G, = (Cn, S N&(u), F,, C F.,, where N₂ (u) includes nodes within k steps from and each edge (i in F,, connects nodes within C., To generate the visual graph of the k--opp subgraph G., centered on ur the nodes within G, are relabeled from 0 to |Cal] 1 to facilitate the generalization of visual graphs. Subsequently, this relabeled subgraph G,, is fed the graph visualizer to generate its visual graph Ic. by Eq. T..
In practice. the graph visualizer can be implemented by a variety of graphic visualization tools, such as Graphviz [17]. Matplotlib (56), and NetworkX [22]. Among them, Graphviz can automatically design the layouts of visual graphs, and is especially suitable for building large-scale datasets. Matplotlib is excellent for customizable plots with fine-graine line-graieed control, and NetworkX excels in complex network analysis. We have implemented various graph visualizers using modular, plug-in architecture in GITA. Specific examples of the visual graphs generated with these tools can be found in Appendix [D]
The graph describer D is tasked with generating task-agnostic textual descriptions of a given graph G. To ensure clarity and fidelity of these descriptions, we meticulously craft a templates. The graph description templates outlined in Appendix E
Graph Describer.
curated set of graph-describie are designed to cover a broad spectrum of scenarios, accommodating various graph configurations including directed undirected graphs and those with or without node or edge weights. To generate the description for a given graph, the graph describer initially selects an appropriate template based on the graph's characteristics, such its directionality and whether it includes node attributes edge weights. Subsequently, this template used by replacing placeholders with actual data, such as the number of nodes, the number of edges, and the endpoints of each edge, craft detailed descriptions tailored to the specific graph in question. The process for D to generate textual descriptions can be formulated as follows:
De = D(G, P);
where De denotes the textual description generated by graph describer, and P the graph-describing template of the graph G.
By introducing these unified and structured graph-describig templates, the graph describer is empowered to generate coherent and informative descriptions that focus on the inherent characteristics of the graph itself, independent of specific task requirements.
Questioner. The questioner Q is tailored to capture the intricate requirements of specific tasks and reflect them in its output task-specific query. In detail, Q receives the task-agnostic textual

---

descriptions from the graph describer and refines them to align with the task context by elucidating the concrete meanings of nodes and edges. These refined descriptions are then enriched with task responsibilities and input/output specifications to form task- spec-specific queries. The formulation of the questioner to generate the task-specific queries can be represented as follows:
O Q(T, Da),
(3)
where Q6 represents the task-specific query generated by the questioner with given the task require- ment T' and the textual description Dc. The construction of task-specific queries can be approached in two main ways: manual template-based construction and bootstrapping LLM agents. Manual template-based construction enriches DG with task-specifis manual templates, which preferred for tasks with precise requirements, such the Traveling Salesman Problem (TSP) [12]. where accuracy is critical and the task definitions are well-understood. This is because it can ensure clarity and reduce the risk of errors due to its meticulous attention to details. On the other hand, bootstrapping LLM agents for automated synthesis is more economical and suitable for dynamic or bespoke tasks, such as robotic planning complex gaming scenarios, it can take advantage of the speed and adaptability of LLM agents to interpret context and generate appropriate queries, minimizing manual effort and enhancing responsiveness changing conditions. Both methods are illustrated with examples in Appendix F showcasing their applications and benefits in different scenarios.
VLM Reasoner. The VLM reasoner Rs performs final graph reasoning with visual inputs Ig from V and textual inputs O from Q, and outputs responses in natural language. This reasoning process can be represented as the following:
A R(IG. (a)
where A is the answer generated by the vision- lasion-lagguage model R. In this work, we adopt GPT-4V and LLaVA-7B/13B as VLM reasoners. These models are regarded as representatives in the realm of closed-source and open-source VLMs, respectively.
In summary, GITA systematically incorporates the vision modality into instruction- -basd graph reasoning. In Appendix B, we discuss the characteristics of GITA, in aspects of generalizability. flexibility and user-friendliness
3.3 Visual Graph Augmentation
Visual graphs generated for the same graph G can be considered as an unique data augmentation technique. Building on the four graph-related image styles introduced in the graph visualizer part of Sec 3.2 we propose the following augmentation strategies: layout augmentation, node shape augmentation, node outline style augmentation, and edge thickness augmentation. Specifically, layout augmentation involves altering the layout styles while keeping all the other settings constant. Similarly, by changing only the respective attributes, we can implement node shape augmentation, node outline style augmentation, and edge thickness augmentation. These four proposed augmentation strategies facilitate studies on the importance of each in enhancing the graph reasoning abilities of VLM reasoners.
3.4 Training
Given a visual graph Ig and text-specific query Q along with the target answer A,, the VLM reasoner of GITA is trained to generate answers A. Specifically, IG is input into the vision encoder of the VLM reasoner, resulting in set of visual features F. If there is dimension difference between F, and the pretrained word embeddings, these F will be aligned with the pretrained word embedding space of the text decoder by vision-to-text projector. Finally, the aligned visual features F Faligned and Q7 are concatenated as input sequences of the text decoder.
Formally, given Ic. QT. and A,, the VLM reasoner is trained by minimizing the following negative log-likelihood:
C. = Σ log Po(A Faligned, L6 Act),
(5)
where di is the trainable parameter and A, denotes the prediction token at the i-th position. Besides, Aci represents the first predicted tokens. During the inference process, GITA is capable of accepting structure graphs inputs and performing graph reasoning in an end-to-end manner.
5

---

4 GVLQA Dataset
In this section, we introduce the GVLQA dataset to fill the absence of a vision-languaa ler--aaggg--asseeeeee based general graph reasoning dataset. It is designed to: 1) evaluate the graph reasoning capabilities of VLMs or LLMs; 2) help models acquire fundamental graph comprehension and reasoning abilities as a pretraining dataset.
4.1
Construction
The GVLQA dataset is created by utilizing the graph visualizerthe graph describer, and questioner in GITA to generate aasion-languagebassee question-answa pairs for graph reasoning on an open-source graph dataset. Specifically, we first extract both the original graph structures and the ground-truth outputs from the NLGraph-full dataset [59]. Then the graph visualizer (detailed Sec 3.2) and the graph describer (outlined in Sec 3.2 are used to generate visual graphs and textual descriptions for these original graph structures, respectively. Afterwards, the questioner (described in Sec (.2) further improves and enriches the textual descriptions by converting them into textual queries. At the same time, it transforms the ground-truth output into text-based answers, following specific output requirements. By combining these visual graphs, textual queries, and text-based answers, we obtain the Graph-based Vision-Languaae Question Answering (GVLQA) dataset.
In the process of establishing GVLQA, we employed graphviz 17 to instantiate the graph visualizer. This choice is made due to its multitude of pre-defined layout algorithms, which enable convenient adjustment of visual graph layouts. Additionally, manual template-basee constructed queries are utilized as the questioner because these tasks are famous with well-defined requirements.
4.2 Structure
The GVLQA dataset comprises 526K samples, each consisting of a visual graph, textual query, and its corresponding answer. It is divided into five subsets: WEVLQA-BASER and four augmentation subsets GVLQA-AUGLY GVLQA-AUGNS, GVLQA-AUGN, and GVLQA-AUGET. In GVLQA- BASE, the visual graphs are uniformly styled. The remaining four augmentation subsets are derived from GVLQA-BAS through the four visual graph augmentations (Sec varying in six different layouts, three node shapes, four node outline styles, and four degrees of edge thickness, respectively. Detailed statistics of the four subsets are shown in Table 6 of Appendix C
Each GVLQA subset undergoes evaluation across seven graph reasoning tasks, outlined as follows.
Connectivity [52 (denoted Connect): Determine whether two randomly selected nodes and v in an undirected graph are connected.
Cycle (52]: Identify whether a cycle exists in an undirected graph.
Topological Sort 335 (denoted TS): Find valid topological sort for directed acyelic graph. Here, topological sort outputs a linear ordering of the nodes such that for every directed edge u c, node comes before in the ordering.
Shortest Path 133 (denoted SP): Find the shortest path between two nodes in a weighted undirected graph. The shortest path between two nodes the path connecting the two nodes with the minimum sum of edge weights along the path.
Maximum Flow 15] (denoted MaxFlow): Calculate the maximum flow from a source node
to a sink node in a network graph.
Bipartite Graph Matching (36 (denoted BGM): Find a matching set in a bipartite graph with the largest number of edges. A matching set is collection of edges in which no two edges share any common node.
Hamilton Path 20 (denoted HP): Find valid Hamilton path in an undirected graph. A
Hamiltonian path a path that traverses each node graph exactly once.
Figure 6 offers illustrations for these tasks in the GVIQQA-BASE dataset. Illustrations of all the GVLQA subsets are provided in Appendix H

---

Experiments
In this section, we extensively evaluate the performance LLM baselines and the proposed GITA on the GVLQA-BASE and five real-world datasets. To better clarify the reasoning capabilities of solely visual graphs, we also test GITA without the textual descriptions of graphs, which can be considered variant of GITA and denoted vision-only (VO). In this case, the visual graph the only information source for graph reasoning. Additionally. we investigate the importance of visual graph augmentation (See 3.3) strategies. by comparison GITA-7B trained on GVLQA-BASE and on the other augmentation subsets of GVLQA 4.2c Lastly, we investigate the effectiveness of using GVLQA as the pretrained dataset on real- world datasets. The evaluation metrics for all experiments are accuracy by exact matching. For the fine-tuning setting, we fine-tune the LoRA adapters [28] for all weight matrices in the text decoder of the VLM reasoner, while keeping the vision encoder in the VLM reasoner frozen. More detailed experimental settings are in Appendix G
5.1 Evaluation on the GVLQA-BASE Dataset
In this subsection, we perform experiments the EVLQA-BASE dataset to compare GITA with popular LLMs including GPT-4 Turbo [49], LLaMA2-7B/ [37], and Vicuna-7B/1 7B/13B (73), under both zero-shot and fine-tuning settings. The experimental results shown in Table Based on these results, we can obtain the following observations.
Observation 1: GITA Outperforms LLM Baselines. As can be seen in Table GITA consistently outperforms the LLM baselines under the same setting. This underscores its SOTA effectiveness in instruction- baasud graph reasoning tasks, showing robust capabilities across different parameter scales under both fine-tuning and zero-shot settings. Moreover, under the fine-tuning setting, incorporating the vision modality consistently benefits 7B models. But for the 13B models, the performance of some tasks may degrade. This could be attributed to the greater challenge of aligning representations of the visual and textual modalities in the larger 13B models compared the 7B models, in the case of only iin-ttnnig LoRA adapters in the text decoder. We speculate that full training could potentially address this issue. However, we leave this as future work due resource constraints.
Observation 2: Mainstream Open-source VLM/LLMS Lack Fundamental Graph Reasoning Abilities. The zero-shot results illustrate that prominent open-source LLMs VLMs, including LLaMA2, Vicuma, and LLaVA, exhibit minimal graph reasoning capabilities on the GVLQA-BASE dataset. Specifically, these models produce random answers, i.e., randomly responding with either "Yes." or "No." " for tasks involving Connect and Cycle, resulting in a performance close to 50%. Cur-
Table 1: Accuracy (%) comparisons on GVLQA-BASH under zero-shot and fine-tuning settings, where "VO" denotes variant of GITA using only the vision modality.
Models
Connect
Cycle
TS
SP
MaxFlow
BGM
HP
Avg
Zero-shot
LaMMA27B
50.06
49.43
0.00
0.00
0.00
0.00
0.00
14,21
Vicuna-7B
50.06
49.43
0.00
0.00
0.00
0.00
14.21
GITA-7B (VO)
50.06
50.33
0.00
0.00
0.00
0.00
0.00
14.34
GITA-7B
50.06
49.43
0.00
0.00
0.00
0.00
0.00
14.21
GPT-4 Turbo
76.70
49.51
19.59
35.35
6.89
42.11
47.04
39.60
GITA-ZS (VO)
57.76
63.34
5.34
4.88
1.59
46.60
10.74
27.18
GITA-ZS
82.58
51.46
19.71
37.69
6.00
52.21
50.00
42.81
Fine-tuning
LLaMA2-7B
97.33
94.63
33.26
26.01
9.56
90.86
23.95
53.66
Vicuna-7B
97.58
95.04
34.46
25.98
9.33
91.04
25.55
54.15
GITA-7B (VO)
59.97
96.34
13.30
5.72
2.89
93.01
1.11
38.91
GITA-7B
98.95
96.67
41.12
32.15
20.00
93.19
29.26
58.76
L.LMA2-13B
98.79
93.36
33.83
27.93
12.22
91.34
33.46
55.85
Vicuna-
99.35
94.39
36.73
28.53
11.34
92.65
34.81
56.83
GITA-13R (VO)
58.00
96.91
14.45
5.72
4.89
93.19
1.85
39.29
GITA-13B
99.14
95.60
38.69
40.47
20.66
92.12
33.33
60.00

---

Table 2:
Accuracy (%) comparisons across GVLQA subsets using GITA-7B (VO). denotes dramatic
performance improvement.
Connect
Cycle
TS
SP
MaxFlow
BGM
HP
Ayg
GVLQA-BASE
59.97
96.34
13.30
5.72
2.89
93.01
1.11
38.91
AVLQA-AUGNS
59.85
96.75
14.17
6.61
3.78
91.58
1.48
39.17
GVLQA-AUGNO
54.87
96.50
14.29
5.54
3,94
92.83
1.11
38.44
GVLQA-AUGET
57.98
96.91
13.37
5.97
3.11
91.76
0.74
38.55
GVI.QA-AUGIY
87.18 1
97.07
14,86
76.55 f
3.94
93.19
70.74
63.36 1
Table 3: Accuracy (%) comparisons on real-world datasets under zero-shot and fine-tuning settings,
where indicates the usage of checkpoint pretrained in the Cycle task of GVLQA-BASE. Models
ca-GrQe
ca-HepTTh
PolBlogs
Cora
CiteSeer
Avg
Zero-shot
LLaMA2-7B
40.59
48.89
10.74
24.35
30.33
30.98
Vicuna-7B
41.35
50.00
8.72
26.94
29.13
31.22
GITA-7B
71.95
86.06
46.98
31.37
30.63
53.40
GITA-7B'
72.02
86.08
48.32
32.10
31.83
54.07
fine-tuning
LLaMA2-7B
76.57
89.06
80.54
83.76
73.27
80.64
Vicuna-7B
78.95
89.85
80.54
84.87
74.17
81.68
GITA-7B
79.70
91.13
84.56
85.24
75.07
83.14
GITA-7B (w/ AUGLY)
79.77
91.21
85.23
85.24
75.68
83.43
GITA-7B*
80.46
91.68
85.23
86.35
76.57
84.06
rent SOTA closed-source LLMs or VLMs, including GPT-4 Turbo and GPT-4V, demonstrate superior zero-shot performance compared with the aforementioned open-source models. This observation implies that current open-sourc LLMs and VLMs lack basic graph reasoning ability, which may be attributed to the insufficient availability relevant training data. Such observation also enhances our motivation to propose the GVLQA dataset, with the aim of improving the graph reasoning capabilities of VLMs/LLMS.
Observation 3: Increasing Model Size Leads to Better Graph Reasoning Capabilities. The comparison of VLMs/LLMs with different parameter sizes, specifically 7B and 13B models, verify the benefits of increasing the model size for graph reasoning capabilities. In this regard, GITA-13B outperforms its counterpart with 7B parameters (GITA-7B) both on average and across four of the seven tasks. However, it is worth noting that GITA-13B does not outperform GITA-7B on the other three tasks. We hypothesize that this discrepancy may be attributed to insufficient modality alignment due to LoRA fine-tuning.
Observation 4: Vision and Text Modalities Proficient in Diffeernt Types of Graph Reasoning Tasks. We explore the individual capabilities of the visual and textual modalities within the GITA framework. The results indicate that the text and vision modalities can complement each other and contribute to better performance than individual ones, as removing either modality leads to performance drops most cases (Vicuna & GITA (VO) and GPT-4 Turbo & GITA (VO) Table [ While the graph reasoning capability provided by the vision modality may not be as strong as that of the text modality most cases, relying solely on vision still enables the model to possess basic graph reasoning abilitics. Specifically, the model outperforms text-based LLMs in 2 of the 7 tasks (Cycle and BGM) when relying solely on vision. This consistent improvement across all comparison groups demonstrates the potential of the vision modality to excel in certain graph reasoning tasks, leveraging contrast,
its ability to capture visual patterns like cycles and graph properties such as bipartition. text exhibits higher proficiency than vision modality in sequence-relate relented graph reasoning problems, particularly on tasks such as TS, SP, and HP, which require constructing ordered node sequences.
5.2 Evaluation for the Visual Graph Augmentations
To assess the impact of the proposed visual graph augmentation strategies (including layout, node shape, node outline style, and edge thickness augmentations), we compare the performance of vision- only GITA-7B models trained the four augmented subsets of GVLQA and on GVLOA-BASE
8

---

(without augmentation). The results are presented in Table 2 To fully utilize the visual information in visual graphs, we fine-tune the visual encoder of VLMs addition to the vision-to-texx projector and the LoRA adapters within the text decoder in this experiment.
As can be seen from the results, a significant enhancement in overall performance is observed with the introduction of layout augmentation (GVLQA-AUGLY), The average performance improves remarkably from 38.91% % to 63.36%. Notably, significant improvements are observed on SP (5.72% to 76.55%) HP (1.11% to 70.74%), and Connect (59. 97% to 87.18%). These findings highlight the critical role of layout augmentation suggests the potential for creating larger-scale datasets for dasion-language which could significantly contribute to advancing this field. Conversely, the other three augmentations do not yield such substantial performance improvements, further emphasizing the importance of layout augmentation in aiioo-language-basee based reasoning.
generating visual graphs. In other words, this observation
graph reasoning.
5.3 Evaluation on RorldWorld Datasets
In this section, we study the effectiveness of GITA on the ca- GrQ 42 and ca-HepTh 42 datasets for the link prediction task, and on the PolBlog [. Cora [65] and CiteSeer [65] datasets for the node classification task. Table g in the appendix C presents the statistics of these datasets. The graph can have thousands of nodes/edges, making it infeasible to feed the entire graph into the model.
Consequently, we employ k-hop subgraph sampling (with = 2) discussed in See 3.2 to satisfy the token length restriction of LLMs and visual graph scope effectively.
The experimental results are presented in Table It is evident that GITA consistently outperforms the LLM baselines, and its performance progressively improves with the addition of layout augmentation and the use of the GVLQA checkpoint. Notably, we emphasize the advantages of using GVLQA- BASE as pretrained dataset by comparing it with GITA-7b. Performance improvements of 0.67% % and 0.92% are observed in the zero-shot and fine- Ane-tuning settings, respectively. This highlights the potential application value of the proposed GVLQA dataset.
5.4 Comparison of GITA with Dedicated Graph Baselines
Table 4: Accuracy (%) comparisons among dedicated GNNs and GITAs on GVLQA-Baa -Base.
Connect
Cycle
TS
SP
MaxHlow
BGM
HP
Avg
GCN
79.65
70.89
45.71
44.56
56.44
76.70
32.22
58.02
SAGE
82.72
73.58
44.51
49.25
50.67
81.00
36.67
59.78
GITA-7B
98.95
96.67
41.12
32.15
20.00
93.19
29.26
58.76
GITA-131 99.14
95.60
38.69
40.47
20.66
92.12
33.33
60.00
Though GITA is designed for language- based general graph reasoning settings, which are much more user-friendly (by user-beed natural language) and general (unique model architecture for various scenarios) than the typical application of dedicated GNNs, it remains essential to conduct comprehensive comparison with specialized GNNs to elucidate the strengths and limitations of GITA'S applicbillity and capabilities. To this end, we assess the graph reasoning abilities of GITA against dedicated GNNs, including GCN [8 and SAGE (4],, using the GVLQA-Bass dataset, as detailed in Table 4 In addition, we explore and compare the effects of k--hop subgraph sampling on the proposed GITA and GNN baselines. Using the ca-Hepth dataset, we analyze the impact of increasing the number of hops & on the reasoning time and performance of both GITA and GNNs. The results are in Table 5
Overall Graph Reasoning Ability Comparison. As shown in Table compared to the dedicated GNNs, the fine-tuned GITA-7B models have comparable average graph reasoning performance, with the larger GATA-13B model performs slightly better. In particular, compared to GNNs, the GITA model shows stronger ability in recognizing local structures in the graphs (Connect and Cycle) and to accomplish tasks with obvious layout heuristics (BGM). We believe that this advantage comes from GITA's visual perception. For SP and MaxFlow, GITA'S performance is inferior GNNs. This may be because GNNs process edge weight more effeetively through the message-passing mechanism.
Scalability and Performance Variation with Different Numbers of Hops k. The inference time results are shown in Table [ As can be seen, GITA demonstrates inferior scalability compared to the GNN baselines. Its scalability remains stable the sampled graph size (i.e., k) increases.

---

From the aceuracy results in T GITA, GCN, and SAGE achieve their peak perfor- mance at k = 2, suggesting that a small sampled graph size suffices for optimal per- formance. Though the dedicated GNNs attain higher peak performance than GITA, they exhibit performance declines as k in- creases (e.g., 3 or 4), while GITA's perfor- mance is more stable w.r.t. k.
Table 5: Accuracy (%) and Inference Time (in parenthe-
ses) for GNNs and GITA on ca-Hepth Dataset with dif- ferent subgraph sampling hop number & E {1, 2, 4].
GCN
SAGE
GITA-7B
93.27 (0.02s)
94.40 (0.03s)
90.33 (17.23min)
k=2
94.49 (0.04s)
94.43 (0.04s)
91.13 117.66min)
91.10 (0,04s)
90.95 (0.18s)
90.31 (17. 22min)
81.92 (0.05s)
83.60 (0.22s)
86.10 (17.01min)
5.5
Case Study
In this section, we present examples graph information provided in both visual and textual formats, which offer some intuitive interpreta- tions for experimental results. Figure (a) shows an example where the GITA-7B (VO) model outperforms its LLMs-based counterpart, and Figure (b) shows an opposite scenario.
Head
oe precc deloo Bukkl
RERAATAAASRO Lndioded graat
hol tode midee
Omane Tratt
Cooant Th 4----o
with in oity 20
The task depicted in Figureßiar is cycle detec- tion and the correct answer is 'No'. This is predicted successfully by the vision-only GITA- 7B model, while the text-based Vicuna-7B fails. In this example, recognizing cycle patterns is much easier in visual graphs, whereas text- LLMs struggle with disordered textual descriptions of edges, which could inherently involve greater complexity and more challenges.
(a)
(b)
Figure 2: A comparative case study of graph repre- sentation in vision and text modalities. All meth- ods are trained on the EVLQA-BASS dataset.
On the other hand, the fixed layout of visual graphs presented in EVLQA-BASE may impede the visual encoder in identifying the shortest path between two nodes, although we have verified layout augmentation can greatly improve the graph reasoning abilities of models, as shown in Sec. [.22 This limitation might arise from the confusion caused by the visual distance within an image, without considering the weights between the nodes. For instance, in Figure 2b), the correct answer is '4->6->0', which visually appears as a more convoluted path but numerically has a shorter path length of 3 1 2. In contrast, the incorrect answer given by GITA-7B (vision-only) is '4->2->0', which has higher path length cost of = 1 but visually seems like a more direct shortcut. This observation further validates the effectiveness of employing layout augmentation to enhance performance in this task. Layout variations of visual graphs play a crucial role in mitigating the visual confusion caused by the spatial arrangement within visual graph. However, it seems more effective for text- based LLMs to handle explicitly separated nodes and weights, as illustrated by the text (in red) in Figure b).
6 Conclusion
In this paper, propose an end-to-end framework called GITA for vision-language-based graph reasoning. Extensive experiments validate the superiority of incorporating visual information into instruction-based graph reasoning. Furthermore, we conduct comparative analysis of the four proposed visual graph augmentations and identify layout augmentation as the most effective approach for enhancing visual graphs. This finding offers valuable insights for the development of larger- scale datasets aimed at facilitating vision-land saiio-language-been graph reasoning. Lastly, we highlight the potential application value of the proposed GVLQA dataset pretrained dataset.
Acknowledgements
This work was supported by NSFC key grant 62136005 and NSFC general grant 62076118 18, and in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grants 16200021 and 16202523).
10

---

References
[1] Lada A Adamic and Natalie Glance. The political blogosphere and the 2004 us election: divided they blog. In Proceedings of the 3rd international workshop Link discovery, pages 36-43, 2005.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in Neural Information Processing Systems, 35:23716-23736 2022.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39 49-48, 2016.
[4] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[5] Peter W Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01266 2018.
Vinicius
[6] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[7] Qi Cao, Huawei Shen, Jinhua Gao, Bingzheng Wei, and Xueqi Cheng. Popularity prediction on social platforms with coupled graph neural networks. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 70-78, 2020.
[8] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference Machine Learning, pages 3469-3489. PMLR, 2022.
[9] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.08/70, 2024.
[10] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (LLMs) learning on graphs. arXiv preprint arXiv:2307.03393 2023.
[11] W Dai, J Li. Li, AMH Tiong, Zhao, W Wang, B Li, Fung, and S Hoi. Instructblip: Towards general-purpose Information Processing Systems, 2023.
purpose dision-languag models with instruction tuning. Advances in Neural
[12] George Dantzig, Ray Fulkerson, and Selmer Johnson. Solution of a large-scale traveling- salesman problem. Journal of the operations research society of Americu, 2(4):393-410, 1954.
[13] Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1959.
[14] Bahare Fatemi, Jonathan Halerow, and Bryan Perozzi. Talk like graph: Encoding graphs for large language models. In Proceedings of International Conference on Learning Representations, 2024.
[15] Lester Randolph Ford and Delbert R Fulkerson. Maximal flow through network. Canadian Journal of Mathematics, 1956.
[16] Shuai Fu, Xiequn Wang, Qiushi Huang, and Yu Zhang. Nemesis: Normalizing the soft-prompt vectors of vision-language models. In Proceedings of International Conference on Learning Representations, 2024.
11

---

[17] Emden R Gansner and Stephen C North. An open graph visualization system and its applications software engineering. Software: practice and experience. 30(11):1203-1233 2000.
[18] Anne-Claude Gavin, Patrick Aloy, Paola Grandi, Roland Krause, Markus Boesche, Martina Marzioch, Christina Rau, Lars Juhl Jensen, Sonja Bastuck, Birgit Dümpelfeld, et al. Proteome survey reveals modularity of the yeast cell machinery. Nature, 440(7084))631-636 2006.
[19] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pages 1263-1272 1272. PMLR, 2017.
[20] Ronald J Gould. Advances on the hamiltonian problem-a survey. Graphs and Combinatorics, 2003.
[21] Jiayan Guo, Lun Du, and Hengyu Liu. GPT4Graph: Can large language models under- stand graph structured data? an empirical evaluation and benchmarking. arXiv preprint 2rXiv:2305. 15066, 2023.
[22] Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab,LLA Los Alamos, NM (United States), 2008
[23] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[24] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[25] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639 648, 2020.
(26) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pages 173-182, 2017.
[27] Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph repre- sentation learning. In Proceedings of International Conference on Learning Representations, 2024.
[28] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[29] Chao Huang, Huance Xu, Yong Xu, Peng Dai, Lianghao Xia, Mengyin Lu, Liefeng Bo, Hao Xing, Xiaoping Lai, and Yanfang Ye. Knowledge-aware coupled graph neural network for social recommendation. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 11-1112,,, 2021.
[30] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700-6709, 2019.
[31] Hawoong Jeong, Sean P Mason, A-L Barabási, and Zoltan N Oltvai. Lethality and centrality in protein networks. Nature, 41 (4683))444-22 2001.
[32] Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James Kwok. Forward-backwant reasoning in large language models for mathematical verification. In Findings of the Association for Computational Linguistics, 2024.
[33] Weisen Jiang, Yu Zhang, and James Kwok. Effective structured- sructured-promppiing by meta-learning and representitive verbalizer. In International Conference on Machine Learning, 2023.
12

---

[34] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2901-2910, 2017.
[35] Arthur B Kahn. Topological sorting of large networks. Communications ACM, 1962.
[36] Richard M Karp, Umesh V Vazirani, and Vijay V Vazirani. An optimal algorithm for on-line bipartite matching. In Proceedings of the twenty-second annual ACM symposium Theory of computing, 1990.
[37] Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint ar0iv: 2016.
[38] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1609. 2016.
[39] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[40] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létoumeau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Pro- cessing Systems, 34:21618 21629, 2021.
[41] Jure Leskovec, Lars Backstrom, Ravi Kumar, and Andrew Tomkins. Microscopic evolution of social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 462-470, 2008.
[42] Jure Leskovee, Jon Kleinberg, and Christos Faloutsos. Graph evolution: Densification and shrinking diameters. ACM transactions Knowledge Discovery from Data (TKDD), 1(1):2ess, 2007.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 2023.
[44] Chang Liu and Bo Wu. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308. /224, 2023.
[45] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in Neural Information Processing Systems, 2023.
[46] Xiao Liu, Shiyu Zhao, Kai Su. Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. Mask and reason: Pre-training knowledge graph transformers for complex logical queries. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1120-11330 1130, 2022.
[47] Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167- 256, 2003.
[48] OpenAl. GPT-3.5. Technical report, 2022.
[49] OpenAl. GPT-4 Turbo. Technical report, 2023.
[50] Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mchran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for Ilms. arXiv preprint arXiv:2402.05882. 2024.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[52] Robert Sedgewick. Algorithms in C, part 5: graph algorithms. Pearson Education, 2001.
13

---

[53] Richard Socher. Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013.
[54] Alane Suhr, Stephanie Zhou, Ally Zhang. Iris Zhang, Huajun Bai, and Yoav Artzi, corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, 2019.
[55] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng. Dawei Yin, and Chao Huang. Graphgpt: Graph instruction uning for large language models. arXiv preprint arXiv:2310. 13023, 2023.
[56] Sandro Tosi. Matplotlib for Python developers. Packt Publishing Ltd, 2009.
[57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arX1i:2307.09220 2023.
[58] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, Yoshua Bengio, et al. Graph attention networks. stat, 1050(20):00-4555)) 2017.
[59] Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In NeurlPS, 2023.
[60] Xiang Wang, Tinglin Huang, Dingxian Wang. Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua. Learning intents behind interactions with knowledge graph for recommen- dation. In Proceedings of the web conference 2021, pages 878-887, 2021.
[61] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Proceedings of Neural Information Processing Systems, 2022.
[62] Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. Kicgpt: Large language model with knowledge in context for knowledge graph completion. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8667-8683, 2023.
[63] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[64] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 2018
[65] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48, PMLR, 2016.
[66] Ruosong Ye, Caiqi Zhang, Runhui Wang. Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.00 071344 2023.
[67] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models. In Proceedings of International Conference on Learning Representations, 2024.
[68] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 6720-6731, 2019.
[69] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 53 5317-5327, 2019.
14

---

[70] Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm- Ilms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 2024.
[71] Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.
[72] Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs. Advances in Neural Information Processing 2021.
Systems, 14:19172-191
[73] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Sto- ica. Judging LLM-as-a- jujdge with MT-bench and chatbot arena. In NeurlPS (Datasets and Benchmarks Track), 2023.
[74] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompt- ing enables complex reasoning in large language models. In Proceedings of International Conference Learning Representations, 2023.
[75] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing langua-a understanding with advanced large language models. arXiv preprint 2304 /0592, 2023.
15

---

A Visual Modality Enhances Effectiveness by Uncovering Critical Substructures
In this section, present study to highlight the complementary role of the visual modality in graph reasoning tasks. The visual modality excels at recognizing beneficial substructures or local patterns, which erucial for effective graph reasoning. For instance, identifying the "hop number" essential for shortest path calculations, recognizing "leaf nodes" is vital for topological sorting, and detecting "cycles" is necessary to avoid in Hamilton path construction. We extracted these substructures from the GVLQA-Bass dataset and manually labeled them. By employing a frozen Vision Transformer (ViT) the LLaVA framework with a trainable Multi-Layer Perceptron (MLP) decoder, we achieved identification accuracies of 89.92%, 95. 16%. and 92.39% for hop number counting, leaf node identification, and cycle detection, respectively. In contrast, using pre-trained BERT model with the same trainable MLP decoder resulted in significantly lower accuracies of 55.47%, 26.33%, and 60.32% for the same tasks. Therefore, the enhanced effectiveness of integrating visual and textual modalities can be attributed to the additional structural information provided by the visual modality, which facilitates the identification of these critical substructures.
B Advantages of GITA Over Traditional Graph Neural Networks
GITA offers several advantages over traditional Graph Neural Networks (GNNs) in terms of general- izability, flexibility, and freendiienlinsss::
Unlike GNNs, which require task-specific feature engineering and architecture adjustments, GITA employs unified model architecture for all tasks, demonstrating its generalizability. By separating task specifications from graph structures, GITA can handle various graph reasoning tasks seamlessly. Additionally, it exhibits strong zero-shot capabilities, allowing it to perform well on tasks it has not been explicitly trained on, which a feature not commonly found traditional GNNs.
Besides, traditional GNNs often demand specialized knowledge in model architectures and coding to accommodate diverse tasks, posing a challenge for non-experts. In contrast, GITA overcomes this barrier by employing language-basse based templates for task adaptation, enhancing its flexibility. This flexibility enables GITA to effectively handle broad spectrum of tasks, offering a framework that can be customized to specific requirements using daily language, without the necessity of profound expertise in graph neural networks.
Moreover, by leveraging existing VLMs, GITA can respond in natural language, allowing for intuitive graph reasoning with simple queries like "Is there cycle in this graph?" This stands in contrast to the unreadable vector representations typically used in GNNs, significantly enhancing GITA's user-friendliness
C Datasets Statistics
Table 6: Statistics of the GVLQA dataset.
Subset
Connect
Cycle
TS
SP
MaxFlow
BMG
HP
Total
BASE
16,410
4,100
2,910
1,560
1,500
1.860
900
29,240
AUGLY
98,460
24,600
17,460
9,360
9,000
11,160
5,400
175,440
AUGNS
49,230
12,300
8,730
4,680
4,500
5,580
2,700
87,720
AUGNO
65,640
16,400
11,640
6,240
6,000
7,440
3,600
116,960
AUGET
65,640
16,400
11,640
6,240
6,000
7,440
3,600
116,960
Total
295,380
73,800
52,380
28,080
27,000
33,480
16,200
526,320
D Illustrations for Visualization Tools in Graph Visualizer
The GITA graph visualizer incorporates a variety implementations for existing visualization tools such Graphviz, Matplotlib with NetworkX, and igraph, each selected for their unique capabilities
16

---

Table 7: Average numbers of nodes and edges for each task in GVLQA.
Average
Task
Connect
Cycle
TS
SP
MaxFlow
BGM
HP
#nodes #edges
25.01 95.46
23.42
21.86
13.65
13,90 49.16
21.13 51.03
13.24
23.66
114.10
23.99
45.05
Table Statistics of real-world datasets
ca-GrQC
ca-HepTh
PolBlogs
Cora
CiteSeer
# Nodes # Edges
5,242
9,877 25,998
1,490
2,708
3,327
14,496
19,025
5,278
4,676
domain
collaboration 5.53
collaboration 5.26
social
citation 3.9
citation
average degree
25.54
2.74
in graph rendering. These tools are implemented our code as interchangeable modules, enhancing flexibility based the requirements of different projects.
(s) Grppviiz
(b) Networkx
Mappooiib
Igraph Malplotibb
Figure Examples the visual graph generated by various visualization tools.
Figure showcases some visual graphs produced by these different graph visualizer implementations.
E
Graph-describing Templates
The graph describer relics on a set unified structured templates designed to generate coherent and informative descriptions that emphasize the inherent characteristics of the graph itself, regardless of specific task requirements. These graph-describing templates cover various scenarios, includ- ing directed graphs, undirected graphs, graphs with node identities or features, and graphs with edge weights or capacities. provides an illustration these templates, where [P] denotes placeholders required be filled by corresponding graph information.
Examples of Manual-template-hann and LLMaaggatt-botttrapped Query Generation
Manual-template-basee Query Generation. The queries Q can be generated by fask-specife man- ual templates. These templates are manually crafted by human supplement descriptions/instruiionn about 1) concrete meanings of nodes and edges, 2) task responsibilities and 3) input/output specifica- tions into task-agnostic graph description DG. Therefore, the precision and faith of generated task-specific queries Q% are guaranteed by human calibrations. An example of manual-template based query generation for topological sorting illustrated in Figure In this example, placeholders [P] are used to represent information that scripts will automatically fill in.
17

---

Graph categories
Undireeted
Directed
Prototype
In undirected graph, (ij) means that node and node directed graph, (ij) that node and node connected with un undirected edge. The nodes are numbered connected with directed edge from mode to mode j. The from [P] to [P), and the edges (P)). [P]) ((P). [P]...
nodes numbered from [P] to (P), and the edges ([)).P) (PD. ([P). [P]...
W/ Node Atributes
In undirected graph, the nodes are numhered from [P] to In directed graph, the modes are numbered from (P) (P), every
attribute. (i.j) meuns and every and node j are connected with an undirected edge. attributes notle node [P]: [P] node [P]
attribute. (ij) means node node are connected with directed edge from mode: to node
j.
The attributes of nodes are:
node [P]:
node [P]: [P]
The edges (PPMP) (PPP))
The edges nre: ([P)PP) ([P)P)))....
W/ Edge Weights
In undirected graph, the nodes are numbered from [P] to In directed graph, the nodes are numbered from (P) (P).
[P], the edges an edge between node [P] and mode [P] with weight [P),
edges are:
edge from node [P] to node (P) with weight (P).
edge between and
weight (P),
odgo from node [P)
weight (P).
W/ Both
In undirected graph, the nodes are numbered from [P] to In directed graph, the nodes are numbered from [P] [P).
[PI. and every node has an attribute.
and every node has an attribute. The attributes of nodes arc: [P]:
The attributes of nodes node [P]: [P] node [P]: [P]
node [P]: (P]
And the edges are:
And the edges are: [P)
weight [P), an edge betwoea node [P] and node [P] with weight (P),
an edge between
edge an edge from mode [P] to node (P) with weight (P).
weight [PI.
Table 9; Graph-describing Templates for various categories.
Textual Description
Manus--templat--based Questioner
Task-specific Query
In directed graph with 7 nodes numbered ed from node should be vialled before node 2.
graph, (ij) means that noce and node j are connected with
Manual Template:
directed graph with nodes numbered from
undrecttdd edge.
[P:
node [P] should be visited before nade (P).
This to find valld topological
The nodes are numbered from to 6, and the edges (0.2) (2.6)
sorting for this directed graph: Please provide possible topolopical ordering path, example: 0..a 2 3 Q:
topological sorting
directed graph
Please provide a possible topological ordering path, for
example: > + 2-3 Q: The topological order directed graph
The topological order the directed
Task-agnostic textual description meanings nodes/edges Figure 4: Examples of the manual manual-template-basee query generation, where [P] denotes the placehold-
Task Responsibillty
Output Specification
18

---

Query Generation. Figure 5 presents an example of employing bootstrapped LLM agent, such as 48, for monster-hunting gaming. By incorporating task-specific information into the prompt, including node/edge meanings and task responsibilities, the L.LM agent automatically generates a response that serves as the desired task-specific query. Compared to using manual templates, bootstrapping LLM agents for automated synthesis is more flexible and economic as it can take advantage of LLM agents to automatically interpret context and generate appropriate queries for various scenarios and minimize manual effort with changing conditions. Such properties make it suitable for dynamic or bespoke tasks, such as robotic planning or complex gaming scenarios.
Questioner
Task-spaciflc Query
Prompt
in directed graph with node represents monster, and undirected signites that the
description: In directed (L))
made and madee connected with continccted
coupled If fone monstar hurted, will iniuta the other monalar it coupled the notes numbered from to defined follows
Textual Description
numbered from 0: and
edges
edges are
Agent (ChatGPT] y
69.11 (1.2) (2,00
(0.1) Monster and Monster 1 ar coupledt
You are required help refine the groph deser ption with 1) coneweta medeeshedggs maveing. Aldd =o Manalers Edga-> Two monatwra coupled and hunting one of them will oviaao another. 2) lask responsiblity huniing monater eamh by each wehoot mitetnn mome mait M0 short sentanie to rule inpuroutput on, for nuepu shoule only number" or "hee cutput should be only or No. Directl givw me refined gr graph description:
(1.2) Monster and Mo Monster 2 are coupted
(2.5) (33))-Mae Monster 2 and Mar Moreter 3 are coupled.
Tect-spacific
information
The lask responsibiity to huntt each monster one by one witthot taiing more than one monster at tim
exampie, a sertence
InpusCutbut Specification: output should sequence numbers reprosenninn order than one monster at time.
monslers can hunted without inttaling
Task-agnostic textual description
meenings of rodes/edges Figure 5: Examples of the
Task Responabillly
Output Specification
query generation.
G Experiment Settings
For all fine-tuning experiments, we batch size of 128 and adopt the AdamW optimizer (with learning rate of 0.0002 and 0.00002 for the LoRA adapters within the text decoder and vision-to-text projector, respectively).
Detailed Settings for GVLQA Dataset During the evaluation, the temperature is set to for all baselines. All fine-tuning experiments are conducted on NVIDIA DGX station with 8xA100 GPUs. We split the GVLQA dataset in the ratio of 7:3 for training and testing, respectively. The accuracy (%) metries computed by comparing the prediction and ground truths with exact matching. We use the next-token prediction projector. Visual graphs are encoded as visual embeddings by visual encoder. Visual embeddings are concatenated with the embeddings of textual descriptions and instructions (i.e., questions), then fed to the text decoder to generate the answer.
taa-took--rrrditioon loss to fine-tune the LoRA [28] adapters of LLMs and the vision-to-text
weal-world Datasets Here we provide more details about the five real-world datasets used Sec 5.3 The datasets ca-GrQC and Ha-HepTTh represent collaboration networks from the arXiv sections of General Relativity and Quantum Cosmology, and High Energy Physics - Theory, respectively, featuring nodes authors and edges co-authorships. They be downloaded from Stanford Network Analysis Project (SNAP) website PolBlogs is a network of U.S. political blogs from February 2005, categorized by political alignment and linked by blog references. Cora and CiteSeer are both citation networks, where nodes correspond to scientific papers and edges to citations, utilized for tasks such document classification and citation analysis, with papers categorized into various research fields. Statistics of the datasets are shown in Table g For each dataset, 80% 11t///11///// 10% of the edges randomly used for saininngyalidaaionntsstngg respectively.
Detailed Settings for Real-world Benchmarks the conventional semi-supervised node classi- fication setting, class labels are available for some nodes, which is reflected in the visual graph by coloring the nodes with a unique random color for each class. To focus on evaluating the model's ability to capture structural information, GITA filters out the influence of node features in Cora and CiteSeer datasets. For link prediction tasks on ca-GrQC and ca- ca-HepTh datasets, GITA treats the graphs undirected. In the test split, both the original links and their reverse links do not appear in the train and valid splits. During training and evaluation, an equal number of negative sampled links used alongside the positive links. These negative links sampled at each training epoch
"https://sappssaann stanford...ttooordo..oooo www.nooo.ddddddddde...it.....
19

---

but remain fixed during evaluation. For the GVLQA pretrain checkpoint, GITA adopts the 7B cycle checkpoint finetuned on GVLQA-BASE, where the performance is nearly mature. Hyperparameter combinations for each model are determined through grid search, and the specific combinations can be found in the provided code.
H Illustrations of GVLQA subsets
provides In this section, present the illustrations of the GVLQA subsets. Figure Provieds an overview of we showcase the augmented visual graphs in GVLQA-AUGLY (augment layouts), GVLQA-AUGNS (augment node styles), GVLOA-AUGNO (augment node outline styles), and GVLQA-AUGET (augment edge thickness), respectively.
GVLQA-BASE Subsequently, from Figure to Figure T0,
Connectivtt
2. Dycle Visail
Topollng Sert
@Shortal Patss
graph
Teduaal desce
Teatual description
Teskaal description
Textua description
tris divecuu nodel and rode cor macdud indrucddd The pocia an siband onters (1M0
This dlig parn depara
Burnntrr hecied
(drroara rodel and not en cor recied
uach diveted noce aga leo aveer roce krit divelled sundered 0ri sodn cloakn rarad belowe roce
with each inger'
rock ane numbered Irom 11.01001) 21.011111
indicated in unditscid gasp, redio ure and us beeween sote And reds will soight
Feno path batvoen rode" antrees undincced propei?
beeloghol dioccdd praph 1,3022
One shortel made H0-0
Vist
Wessl graph
Tectual desci grlon
Visuil graps
Tediual description
Viuit grapt
Todaal des descrition
dineadd graph, with each
numborad ham to
B-C riaar shal ride and undirccedd wiga The
narsrrical lobal in doss gropn the hodks adges ane udge noci adge from vosk ta made mode 2/a rade with dapiiclyy with ciearr anionge kerm
from Oto 2 Each hoal
100 varad in Hout lood la intensted bost of ball and senturr.
lo 10, and edges and IS SP 21th 100D ES 12 (6-00-2
nedved by
Bogi rech In thwr part indi grash puth Has Butin pan aegaser nodic conat be adger D-b--207----00 20--0-0-----00 2--------4
wovr the mooinam number boo
A.2.
Figure 6: An overview of the GVLQA-BASE. Each figure depicts the tasks involving graph-based reasoning, showcasing visual graph, textual question, and the corresponding answer.
I Limitation
The GITA framework proposed in the paper, along with its experimental results, exhibits certain limitations that should be acknowledged. Firstly, when dealing with large-scale graphs, the con- ventional subgraph sampling strategy employed by GITA may result in imbalanced and insufficient sampling, leading to the loss of critical graph structural information. This compromise is necessary accommodate the limited contextual length of the text-based LLM and the restricted scope of the visual graph. Secondly, due to computational constraints, the fine-tuning procedures in the paper were restricted to the LoRA framework. While this approach offers advantages, more comprehensive fine-tuning process that considers both visual and text modalities is expected to better align the two and potentially enhance performance. Addressing these limitations should be considered as a future research direction in this field.
20

---

Consecivity
Cycle
Topology Sant
Shartett Pats
Teaked
the und sccied greet. undresse recrrdee and oue seckec The recey rur burdd owtent ca (1.0
The dhip depett graah eitt usery divettu roce aga leo fied. rapdlegool snce. moce kilf's rocas surbered mode chould be vereea befoww
The cridil de Bucsules
Fed modi and
with much adbars
comsdded with
indicdtd bya peoskery lus recres are nambore odgas are dn acga
sldkw are lo the engee (1.01)
neds will waight
unciteedd proph?
Pen dyd undrnded
spplogkal onder discadd pope lo
Giv the worket palh from noce mode
Sestaal desci
Vinual graps
Tertual dxncription
Visaal grapi
Todtal descrittion
aetrace divered giiet sich
doo brlk, and booke
F.E. menicccton cooiicttoonnn ncln and
pradt
by
mode endin dertees The
exveniic the in chiie proat rayy in diwccdd lou fireen Proppeccl edgs wate eods Prs rate uain cipiillyy 3 wit coevii J. enorga
hood fis of todde mot
indorishst in kee 2. Heel Movewar outh rostre capulle le
modes are pntile enges BLR Ph B.B 11.1100 It PUR 2320g220 (L,20122
sdge
one
held
Dogh with necee thers part that every ra:h yooef (yes, Has
What mosoimmn number andior bank thay manniid
corvecd arjaset
Wh mastrna Bos frmmeecs
Figure 7: An overview of the GVLQA-AUGLY. Figures are akin to GVLQA-BASE but vary only in layouts.
Consectvity
2. Dycle
Topologe Sect
Shortett Patk
andinast
morhed graen, and recko iou edgs-
Thas vlkag can depices suo prooned roce ska lus thet rapotopioll snae nore Wiit nonic aiterotingg fron mode virad below
groche i amoons
sode and
with bach rigers
conme with
the rocey are bared edgee (1
mnde ald by's
The sodee
prodiity lia
lnd edggee (L1) (1.
nedee an Aundaves oetee ore de negs nede woight
bataose rade and meds cirecdedd grape??
deneaa dy tha undruded grash?
tppelgial onder of dineced graph by:
Cive ahortet pach fhom noca mode
Wausl
Textul deserption
Vaaul greph
Testual
Viual grash
Toxtaal descrittion
puiphe
nurearrd bow Dito
moki nodee undiniiddd wegg The
nuneeical beel cille
from Ors Sach hod
(rock nodie diraceed
hate basts criverte Her Hevd Ploutendtt
10 pntrie adges ane IL KOVIRAGI 21 12 IN 1031220 (6.20122 PMRICAA)
wdys troe code
enge boww nado to node code lonees with capacry ride wil capudly an wit
cad tesd repilie bats, and sinilar mat and
Dugi with rotte tans Hote with edge
man number
Figure 8: An overview of the GVI.QA-AUGNO. Figures are akin to GVLQA-BASE but vary only in node outline styles.
21

---

Canseciviyy
Cycle
Topalogy Sait
Pats
Teaked
the undtrodbed
the und accied greet undrest recerde and oue socked The rocey rur burdd oxtpet wn (1
Then sur depeti eitt usery divettu roce age leo fied rapolqpoo sraer noce kill's nocas surbered mode chould be veree beflow
The pridil de Buroddes
Frdt mods and
with mach adgers
conneded with
indiudtd bya peoskery lus redles are numborr edgas de acga rads wilh waight
The sldkw are lo the engen (1
uncileded prope?
hhe dyd undrededd grash?
spelogkal onder discaed pope to
Give the shoreet palt from noce mode
Testaal descr
Visual grapa
Viuaa graph
Toxtall description
Rustadar cinndee grich, eich
F.E. menicccton soa nnls
sege's capett by roceenal ciite
mode secadaae
fram (1r Sisih hood
ondi derleig The
prouenty diucked rodor
hat lasis enevected Heat tit intrpaaee
modes are
pntine edgee BLRIIIL IN BUB 1L.100 Et 1N
edge hare rode nadi ugge node rode vode a linven with chpoery nede eel capurte au wit
houl Homewer
pao, and antiina
end
Dogle with part
that every nalhe emolf tyes,
What the ropiia ropiiamn number host unaoo mrnniidd
rich arjaset
Wh mastraa los fomreecs
Figure 9: An overview of the GVLQA-AUGNS. Figures are akin to GVLQA-BASE but vary only in node shapes.
Consectvity
2. Dycle
Topologo Sect
Sbottet Patk
andinase
the sorhed graen, and racks iou edge- the mocey are bard wdgee (1.0 (P
Thai vlkeg can depicss san procted roce sur bos thet tapolopioll snae noc bittis nonec aiteroniggg hhon mode viradd befow
grochi d amoons
sode and
with sach rigers
comme aod with
rede ala byra peodiity lia
sodee lnd edgge (U1) (1.1
neder on Aundaves oate ore de negs neds soight
Iataose made meds cirecctdd grape??
Berea dyde tha undruded gragh?
tppelgial onder of dineced graph by:
Oiv amoriat palh from noca mode
Wausl
Textul descr
Vaaui greph
Testaal
Viual gresh
Toxtual descrittion
puiph
nureaerd brow Dito
moki node
nuneeical beel cilse
Tron Ors Sach hoo
undiniiddd rego The
prohk nodie dircced
hate basts criverte Her Hevd Ploutenddt cal tesd repilie tals, and ainilaa
10 putrie adges ane IL KEVIRAG E1 2012
wdys tine code enge boww nodo nede rede lonees with capacry ride wif capadt an win.
IN 1021220 96.20122 PARIAAA
mat and by can
Bugi with witte Pare Hote with edgan
rapimmm number
Figure 10: An overview of the GVI.QA-AUGET Figures are akin to GVLQA-BASE but vary only in edge thicknesses.
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Please refer to the abstract part and the contribution enumeration at the tail of the introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification:
Please refer to Appendix !!
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not involve any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We show fundamental experiment settings in Section 5 and more details for ex- periments settings in Appendix Besides, we provide the complete codes as supplementary materials.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The complete codes are included, and the proposed GVLQA dataset is released with common access.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.ccppullc//guus http://////uiiee///deuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment details are given in both Section 5 and Appendix G Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the machine (type and storage) requirements in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: We make sure the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research does not have concerns about societal impacts because it is designed for general-purpose purpose graph reasoning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
27

---

Answer: [Yes]
Justification: The paper includes using an graph visualizer to generate abstract graph images, however, these images are focus graph structure, without any sensitive information.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited necessary assets and conduct CC-BY 4.0 for our codes and datasets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pappeswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and other supplementary materials are followed with readme and instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
28

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "hCDdO2XB6z", "reviewer": "Reviewer_SxbK", "summary": "This paper introduces Graph to Visual and Textual Integration (GITA), a novel framework that enhances graph reasoning by integrating visual representations with traditional text-based processing. GITA's innovation lies in using both visual and textual inputs to address graph-structured data, a significant deviation from the typical text-only approaches in Large Language Models (LLMs). Key contributions include the development of GITA, the creation of the Graph-based Vision-Language Question Answering (GVLQA) dataset—the first for visual-textual graph reasoning—and extensive evaluations showing GITA's superior performance over existing LLMs across various datasets, highlighting the advantage of incorporating visual information into graph reasoning tasks.", "questions": "please refer to the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces the novel idea of integrating visual graphs with textual descriptions for enhancing graph reasoning tasks. \n2. The creation of the GVLQA dataset is a significant contribution, as it is the first vision-language dataset designed specifically for general graph reasoning.", "weaknesses": "1. While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with the visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well.\n2. I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else.\n3. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional questions or require further clarification, please feel free to let us know. \n\n**Note: Some tables mentioned in this rebuttal are contained in our rebuttal supplement PDF.**\n\n> **W1**: While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well.\n\nWe agree with you that only the visual modality may be not so good for graph reasoning tasks. Please note that our work is to show that **integrating the visual modality with the textual modality** could achieve better performance than a single modality, as evidenced by our submitted paper Tables 1 and 3.\n\nWhile \"VO\" demonstrates limited performance in Table 1 for most tasks, we have **successfully enhanced its capabilities** through **layout augmentation** in GITA, as evidenced in Table 2.\n\nThe visual modality could **complement** the textual modality in graph reasoning tasks. Here we provide **a case study**. Typically, the visual modality is more excellent in recognizing beneficial substructures/local patterns compared to the textual modality, and some of them are crucial in graph reasoning. For instance, \"hop number\" serves as a heuristic in shortest path calculations, \"leaf nodes\" are critical in topological sorting, and \"cycles\" must be prevented in Hamiltonian path construction. We extracted these substructures inside the GVLQA-Base and manually labeled them. Employing the frozen ViT in LLaVA with a trainable MLP decoder, we achieved identification accuracies of 89.92\\%, 95.16\\%, and 92.39\\%, respectively for hop number counting, leaf nodes identification, and cycle identification. In contrast, using a pre-trained Bert with the same trainable MLP decoder, the accuracies are significantly lower (i.e., 55.47\\%, 26.33\\%, and 60.32\\%). Therefore, the effectiveness of the integration of the visual and textual modalities may be due to that the visual modality provides extra **beneficial structural information** such as those substructures.\n\nBesides, The benefits of vision do not conflict with the benefits that come from the ability to step-by-step reasoning (e.g. Chain-of-thought, CoT). **Table 1 in our rebuttal supplement PDF file** shows the model benefits from them both simultaneously.\n\n\n\n> **W2**: I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else.\n\nWe would like to point out\n\n1. GITA is the **first** to enable the use of MLLMs in graph reasoning. This is a significant advance because nearly all existing graph reasoning works do not utilize the visual modality, which could help improve the performance.\n2. GITA is general for **almost all existing graph reasoning data** because it only requires graph information in bare structures $G=\\{V, E\\}$.\n3. GITA **solves technical problems** including: how to use MLLM for graph reasoning based on bare graph structure, how to maintain vision graph usability and keep context length in large-scale graphs, how to trade-off the consistency and variability of vision graphs, special augmentations for vision graphs and their impacts, etc. \n4. GVLQA is just a **by-product** of our core contribution, GITA.\n   Therefore, the methodologies inside GITA are not limited to any individual benchmark or scenario but are significant and valuable to graph reasoning. As a result, we think our work fits the main track instead of the dataset track.\n\n> **W3**. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.\n\nWe would like to point out in our experiments, LLaVa has been utilized as the VLM Reasoner for both the GITA and GITA (VO) configurations. As one of the components of GITA, LLaVa receives the visual and textual inputs from the other components of GITA (though we store them as GVLQA) and the results are shown in the Table as GITA and GITA (VO) terms. Therefore, LLaVa is not necessary to be compared again as an individual baseline because it is **a special case of GITA**. Similarly, GPT-4V serves as the VLM Reasoner for the GITA-ZS and GITA-ZS (VO) configurations in Table 1 and is not considered a baseline. \nAccording to your suggestion, we utilized **GPT-4o** (not released since the submission of our work) and **Gemini**, as the MLLM reasoner and recorded the results in **Table 2 in our rebuttal supplement PDF file** , which shows that the **benefits** of incorporating vision with GITA are **consistent across** various VLM reasoner choices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYFAOu90iw", "reviewer": "Reviewer_6MUx", "summary": "The paper introduces an end-to-end framework called Graph to Visual and Textual Integration (GITA) to visualize graphs in order to improve LLMs’ reasoning capabilities on graph tasks. GITA consists of three main components: a Graph Describer to translate a graph into a natural language description, a Graph Visualizer to visualize the graph as an image, and a Questioner to generate task-specific queries conditioned on the task information. GITA shows improvements compared to vanilla LLMs and VLMs on graph reasoning tasks and some real-world graph datasets.", "questions": "1. Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset?\n2. In Table 1, is GITA-7B (VO) equivalent to Llava-7B?\n3. In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?", "limitations": "See weekness above", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is convincing. Just as humans reason over structured data, it is natural to visualize the structure first. The additional visual modality can provide rich information that can assist in reasoning over structured data. Therefore, generalizing this human behavior to LLMs/VLMs makes a lot of sense.\n\n2. The method is simple yet effective for some graph reasoning tasks on small graphs. The idea of visualization is a form of data augmentation.", "weaknesses": "1. The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value.\n2. The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets.\n3. Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken [1-3], etc.\n\n[1] Graph Chain-of-Thought, https://arxiv.org/abs/2404.07103\n\n[2] GraphLLM, https://arxiv.org/abs/2310.05845\n\n[3] GraphToken, https://arxiv.org/abs/2402.05862", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional question or require further clarification, please feel free to let us know.\n\n> **W1**: The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value.\n\nIn fact, k-hop subgraph sampling is a common practice in graph learning. Typically, a k-hop sampling may not lead to performance degradation. On the contrary, k-hop subgraphs show more dedicated awareness of the local structure details. For example, ShaDow-GNN [1] indicates from both theoretical and empirical analysis, that in graph data, one can ignore distant neighbors ($k >= 4$), and the most effective $k$ is 2 or 3. \n\nTo further illustrate, We conducted experimented to show the performance of both GITA and Vicuna by varying $k$ on large graph datasets ca-GrQc and ca-HepTh and show the results in the following table. It is evident that both GITA and Vicuna reach their good performance at $k = 2$. Thus, increasing $k$ does not necessarily enhance the performance, and k-hop sampling does not lead to a decline in performance. \n**Based on this observation, we think that the proposed GITA method is applicable to general graph reasoning tasks.**\n| Model        | **ca-GrQc** | **ca-HepTh** |\n| ------------ | ----------- | ------------ |\n| Vicuna (k=2) | **78.95**   | **89.85**    |\n| Vicuna (k=3) | 78.95       | 89.66        |\n| Vicuna (k=4) | 76.53       | 85.24        |\n| GITA (k=2)   | **79.70**   | **91.13**    |\n| GITA (k=3)   | 79.67       | 90.31        |\n| GITA (k=4)   | 75.47       | 86.10        |\n\n[1] Decoupling the Depth and Scope of Graph Neural Networks. NeurIPS, 2021.\n\n\n\n> **W2**: The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets.\n\nWe think that the requirements of human efforts in GITA are negligible based on the following aspects. \n1. For a new dataset, the Graph Visualizer and Graph Describer in GITA can be directly used in a **function-invoking** manner. We provide the default setting. As a result, users do not need to design them.\n2. The task-specific template inside the **Questioner** is **the only component requiring human efforts**. It only requires users to describe the task definition and the concrete meanings of the graph elements in user-friendly natural language.\n3. We also offer **an automated approach** that allows users to generate task-specific queries by prompting an agent like ChatGPT. An example of the query generation for a custom gaming scenario has been provided in **Appendix E**. Though this automated approach still necessitates human input to describe the task initially, it is a minimum and unavoidable requirement for any language-based method.\n\n\n> **W3**: Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken.\n\nGraph Chain-of-Thought is not applicable to the general graph reasoning tasks, because it is built on knowledge graph (KG), but general graph reasoning tasks usually do not have the corresponding KG. GraphToken does not provide its data and code. Hence, in the following table, we here provide the performance comparison with GraphLLM on a randomly chosen half-size GVLQA-Base subset.\n| Model       | Connect   | Cycle     | TS        | SP        | MaxFlow  | BGM       | HP        | Avg       |\n| ----------- | --------- | --------- | --------- | --------- | -------- | --------- | --------- | --------- |\n| GraphLLM-7B | 94.74     | 92.36     | 42.17     | **56.72** | **52.0** | 58.24     | 26.3      | **60.36** |\n| GITA-7B     | **99.05** | **97.48** | **44.11** | 33.05     | 24.89    | **93.37** | **28.15** | 60.01     |\n\nBased on the results, for **substructure-awareness tasks** such as **Connect** and **Cycle** and tasks with beneficial visual heuristics such as **BGM**, the visual modality introduced by GITA is more advantageous than the graph modality introduced by GraphLLM. GraphLLM shows its superiority in MaxFlow and SP, which may be due to their graph transformer encoder being more effective in processing edge weights. Finally, GITA and GraphLLM show similar abilities on sequential ordering tasks TS and HP, and **comparable average performance**. We will include this comparison in the revision.\n\n> **Q1**: Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset?\n\nThe average numbers of nodes and edges for each task in GVLQA are shown in the following table.\n| Average / Task | Connect | Cycle | TS     | SP    | MaxFlow | BGM   | HP    |\n| -------------- | ------- | ----- | ------ | ----- | ------- | ----- | ----- |\n| #nodes         | 25.01   | 23.42 | 21.86  | 13.65 | 13.90   | 21.13 | 13.24 |\n| #edges         | 95.46   | 23.66 | 114.10 | 23.99 | 49.16   | 51.03 | 45.05 |\n\n\n>**Q2**: In Table 1, is GITA-7B (VO) equivalent to LLaVa-7B?\n\nGITA-7B (VO) represents **a variant of GITA** using LLaVa-7B reasoning on the vision graph generated by GITA Graph Visualizer and a direct question like \"Is there a cycle in this undirected graph\", but **without the textual descriptions** of the graph generated by GITA Graph Describer.\n\n>**Q3**: In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?\n\nWe have introduced the detailed fine-tuning settings in both Section 5 and Appendix F in our submission.  For the fine-tuning setting in **Table 1**, we **fine-tune the LoRA adapters** for all weight matrices within the text decoder and **the alignment projector**, while keeping the vision encoder in the VLM reasoner frozen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RSJ43lLmqk", "reviewer": "Reviewer_gmAJ", "summary": "To fill the gap that LLM overlook the rich vision modality with graph structure, this paper proposes GITA to incorporates visual graphs into general graph reasoning. A large graph vision-language dataset called GVLQA is designed to boost the general graph reasoning capabilities.", "questions": "The dataset will be released in the future? If so, I appreciate it.", "limitations": "See the weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper may involve a relatively large workload, proposing a large vision-language dataset, which can greatly promote the development of VLM.\n2. Integrating visual graphs into VLM holds significant value.", "weaknesses": "1. This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology.\n2. This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance.\n3. In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional questions or require further clarification, please feel free to let us know.\n\n> **W1**: This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology.\n\nWe acknowledge that our work builds upon existing backbone models and tools. Incorporating visual information into general graph reasoning, however, is an interesting idea that has not been explored before. As a result, we have encountered **several challenging issues** during our work. For instance, \n\n1) how to maintain the usability of vision graphs while managing context length in large-scale graphs;\n\n2. how to balance consistency and variability in vision graphs, and the impact of specialized augmentations for vision graphs, etc. \n\n**To handle those issues that have never been explored before**, we proposed the GITA framework and provided valuable empirical findings.\n\n> **W2**: This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance.\n\nThe illustrations of the graphs have been provided in **Figure 2** as part of the case study for readers to understand how vision plays a role in graph reasoning. We have also included examples of visual graph generated by different graph visualizer tools implemented by us in **Appendix C**, as well as illustrations of these visual graphs for each subset of GVLQA in Figures 6-9 in **Appendix G**. We will highlight them in the revision.\n\n>**W3**: In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.\n\nThank you for pointing out this typo. The 'G' in the end should be 'D'. We will correct it in the revision.\n\n>**Q1**: Will the dataset be released in the future? If so, I appreciate it.\n\nOf course! The dataset will been released soon. Due to the submission policy, we do not provide the link in the submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PuZ44whJRz", "reviewer": "Reviewer_oE8d", "summary": "The paper introduces the GITA framework, which innovatively integrates visual graphs into general graph reasoning tasks for LLMs. By combining visual and textual information of a graph, GITA improves the comprehensibility and flexibility of graph reasoning, outperforming other LLM approaches. The authors also develop the Graph-based Vision-Language Question Answering (GVLQA) dataset, the first vision-language dataset for graph reasoning. The study also highlights the benefits of layout augmentation in visual graphs and pretraining on the GVLQA dataset.", "questions": "Please see the above weaknesses.", "limitations": "The authors have acknowledged several key limitations of their work but could benefit from providing more comprehensive solutions and experimental evidence.  As for my suggestion for improvement, please see the “weaknessess” part.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Integrating visual graphs into graph reasoning tasks for LLMs, is a novel approach. This represents a creative combination of visual and textual modalities to improve graph comprehension and reasoning. The creation of the GVLQA dataset is original and fills a gap in the current datasets available for graph reasoning by incorporating both visual and textual elements.\n\n2.  Extensive experiments on both the GVLQA and five real-world datasets validate the effectiveness of the proposed framework, providing strong empirical evidence for the paper's claims.\n\n3. The paper is clearly written, with well-defined sections and logical flow. The detailed explanations of the GITA framework's components and the dataset construction process enhance understanding.\n\n4.  By successfully integrating visual information into graph reasoning tasks, the paper addresses a significant limitation of existing LLMs and VLMs. This has the potential to substantially advance the field of graph reasoning.", "weaknesses": "1. The motivation of the framework is challenged. As for the commonly used graph reasoning methods such as GNNs, the authors state that \"These methods often lack generalizability, flexibility, and user-friendliness.\" However, the advantages of GITA in these areas are not directly demonstrated. The zero-shot performance of LLM-based methods is poor for many questions (such as MaxFlow, SP, TS). Considering this, users need to fine-tune when addressing questions on a new dataset. Additionally, GITA requires data augmentation and manual template-based construction for task-specific queries. Therefore, it is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified.\n\n2. While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages.\n\n3. The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios. \n\n4. The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed.\n\n5. Some places need double-check. For example, the meaning of GITA needs to be unified: the authors mention GITA as \"Graph to vIsual and Textual Integration\" in line 44, while they refer to it as \"Graph to Image-Txt Assistant\" in line 117. Another example is in line 120, which needs to be changed to \"Firstly, V and D are designed to produce visual depictions and textual descriptions.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough comments and insightful suggestions, we here address your concerns and adopt your suggestions as follows: \n\n> **W1**: It is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified.\n\nIn Appendix A, we have detailed GITA's flexibility, user-friendliness, and generalizability. Unlike GNNs, whose task-specific feature engineering and architecture adjustments require professional techniques on model architectures and coding, GITA employs a consistent architecture that simplifies adaptation to new tasks using language-based templates. This makes it accessible even to non-experts, significantly enhancing **flexibility**. Moreover, GITA utilizes natural language for input and output, enabling intuitive graph reasoning through simple queries like 'Is there a cycle in this graph?' and providing straightforward 'Yes' or 'No' answers, thereby improving **user-friendliness** compared to the human-unreadable vector representations/embeddings in GNNs.\n\n\nBesides, we further demonstrate that GITA-ZS (zero-shot) also has promising performance. According to the results shown in the **Table 3 in the rebuttal supplement PDF**, where 'PT' represents advanced prompting techniques including in-context-learning (ICL), chain-of-thought (CoT), and self-consistency (SC), its performance can be further improved with both the evolution of the VLM reasoner (i.e., from GPT-4v to GPT-4o) and advanced prompting techniques including ICL, CoT, and SC. Based on this result and the results reported in the submission, our GITA method exhibits promising zero-shot capabilities (**generalizability on tasks**).\n\n> **W2**: While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages.\n\nAccording to your suggestion, we compare with dedicated GNNs, including GCN and SAGE, and present the results in **Table 4 in rebuttal supplement PDF**. Compared to dedicated GNNs, zero-shot GITA (with prompt techniques) and fine-tuned GITA-7B model have similar average graph reasoning performance. The larger GATA-13B model performs slightly better.\n\nIn particular, compared to GNNs, the GITA model shows a stronger ability to recognize local structures in graphs (Connect and Cycle) and to accomplish tasks with obvious layout heuristics (BGM). We believe that this advantage comes from  GITA's visual perception. For SP and MaxFlow, GITA performs inferior to GNNs. This may be because GNNs process edge weight more effectively through its message passing mechanism. For HP and TS, GITA-ZS performs best. Those results will be put in the revision.\n\n> **W3**: The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios.\n\nAccording to your suggestion, we conducted experiment to evaluate the scalability and performance for GITA and dedicated GNNs while increasing the number of hops $k$ in the ca-HepTh dataset.\n\nThe results shown in **Table 5 in supplement PDF** demonstrate the scalability of GITA is kept stable with the sampled graph size (i.e., $k$) increasing. \nAccording to the accuracy reported in **Table 6 in supplement PDF**, GITA, GCN, and SAGE achieve their respective peak performance when $k$ equals 2, demonstrating that a small sampled graph size is enough for the performance. Dedicated GNNs show higher peak performance than GITA, but also show worse performance when $k$ becomes larger (e.g., 3 or 4), which demonstrates the performance of GITA is more stable than dedicated GNNs w.r.t. $k$. We will include those results in the revision.\n\n> **W4**: The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed.\n\nBecause existing Multilingual Large Language Models (MLLMs) are not inherently attuned to graph data, they require fine-tuning to effectively align vision and text inputs in a graph context. The effectiveness of this alignment during fine-tuning is influenced by the proportion of tunable parameters. **Following LLaVA**, for the larger GITA-13B, the trainable parameter ratio is only 0.78\\%, which is much smaller than the 1.46\\% for GITA-7B. This **limitation in tunable parameters** may result in **less effective alignment** for GITA-13B compared to GITA-7B, potentially leading to poorer performance of GITA-13B on some tasks. \n\nTo address this issue, two solutions are proposed: one is to employ **full-finetuning**, which directly tunes all trainable parameters to align the modalities. Another approach involves applying the proposed GITA to **more diverse graph data**, thereby obtaining a richer source of vision-language data for alignment ([r1] illustrates diverse and rich data can hugely improve alignment). These additional vision-language data can be fine-tuned together with the current task data, or used as pretraining (as demonstrated in Section 5.3, using GVLQA checkpoints for real-world datasets).\n\n[r1] Visual Instruction Tuning. NeurIPS, 2023\n> **W5**: Some places need double-check, including the inconsistent full name of GITA and a typo in line 120.\n\nThank you for pointing out the inconsistent terms and notations. We will fix them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for your valuable feedback and thoughtful comments on our manuscript. We have carefully reviewed each of your concerns and have addressed them individually in the responses below. We believe that these revisions and clarifications have strengthened our manuscript, and we hope that our responses meet your satisfaction.

Additionally, due to space constraints, we have included some of the larger tables in an attached **PDF** document for your convenience.

Please find our detailed replies to your specific points below.

Best regards,  
Authors

Meta-review: The paper presents GITA, a framework that integrates visual graphs generated from text inputs to enhance graph reasoning tasks in LLMs. The creation of the GVLQA dataset, specifically designed for vision-language graph reasoning, is a valuable contribution that fills a gap in the current landscape of graph reasoning datasets. The paper demonstrates promising results across multiple datasets, showcasing GITA’s effectiveness, particularly in local structure recognition and layout-sensitive tasks. Although some concerns were raised regarding scalability and comparisons to GNNs, the authors effectively addressed these by providing additional experiments and detailed explanations in their rebuttal. 

All the reviewers comments have been addressed. All the scores recommend acceptance. Congrats to the authors.
Decision: Accept (poster)