Title: GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Reasoning

Visual Question Answering

Large Multimodal Model
Abstract: Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish  $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Abstract

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in the textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., visual graph ) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called G raph to v I sual and T extual Integr A tion (GITA), which incorporates visual graphs into general graph reasoning. Besides, we construct the G raph-based V isionL anguage Q uestion A nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs on general graph reasoning. Moreover, experimental results demonstrate the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

1 Introduction

Graph reasoning tasks are pivotal in domains such as recommendation systems [25, 60], social network analysis [7, 29], and knowledge graph reasoning [72, 46, 62]. Various architectures have been developed, from shallow embedding methods [6, 53] to advanced Graph Neural Networks (GNNs) [37, 64] and graph Transformers [71, 40, 8]. While these models excel in graph reasoning tasks, they often lack generalizability, flexibility, and user-friendliness. Achieving good performance with these models typically requires domain-specific tuning, which limits their abilities to generalize across different domains. Additionally, these models struggle to handle diverse tasks with the same architecture. Each task often requires a specialized design, including task-specific data processing and decoder, leading to limited flexibility. Lastly, unlike the Large Language Models (LLMs) that can engage in conversations with users, these models are less explainable and user-friendly.

In contrast, LLMs have shown great generalization capabilities across a wide variety of reasoning tasks [61, 67, 74, 32, 33] by encapsulating task-specific demands within a cohesive and interpretable mechanism - text prompts, under a unified architecture with minimal domain-specific adjustments. These advantages have sparked investigations into the potential of LLMs for graph reasoning. Recent developments lend credence to the notion that LLMs can indeed interpret and manipulate graph-

structured data through textual representations. For example, InstructGLM [66], GPT4Graph [21], and LLMtoGraph [44] convert graphs into textual descriptions and then use these descriptions paired with queries to enable LLMs to generate accurate responses for graph reasoning tasks. Furthermore, the introduction of benchmarks such as GraphQA [14] and NLGraph [59] is a testament to the growing interest in evaluating LLMs' effectiveness on graph reasoning tasks framed in natural languages.

Despite the development of numerous methods and benchmarks for graph reasoning on LLMs, they often overlook the valuable vision modality, which is a natural means for humans to comprehend structural information and has demonstrated its success in various visual reasoning scenarios [30, 68, 69, 34, 3, 54]. Consequently, the following questions arise: (1) Can incorporating visual information be beneficial for general graph reasoning scenarios? (2) If so, how can we effectively integrate the vision modality into graph reasoning? To the best of our knowledge, these questions remain unexplored.

To answer them, we first propose an end-to-end framework called G raph to v I sual and T extual Integr A tion (GITA) 34 that systematically integrates visual information into instruction-based graph reasoning, by rendering graph structures to customized visual images which are called visual graph . Specifically, the GITA framework has four components: a graph visualizer for generating visual graphs, a graph describer for producing textual descriptions of the graph structure, a task-based questioner that organizes the description and requirements of the current task into prompt instruction, and a Vision-Language Model (VLM) to perform vision-language graph reasoning. In the proposed GITA framework, visual information can be incorporated into many tasks with explicit or implicit graph structures, without sacrificing its versatility, flexibility, or user-friendliness. Besides, since there is no dataset for vision-supported general graph reasoning capabilities, we construct the first vision-language dataset for general graph reasoning purposes called G raph-based V isionL anguage Q uestion A nswering (GVLQA) 5 based on the proposed GITA framework. The GVLQA dataset consists of 526K instances covering seven representative graph reasoning tasks, aiming to thoroughly evaluate the structure-based graph reasoning abilities of VLMs and LLMs. Extensive experiments on the GVLQA dataset and five real-world datasets demonstrate the effectiveness of the proposed GITA model. Furthermore, we delve into the effects of visual graph augmentation strategies and find that layout augmentation can dramatically boost vision-based graph reasoning performance.

Our main contributions are summarized as follows.

We introduce an end-to-end GITA framework, innovatively integrating vision modality to boost the graph reasoning abilities of language models.

We establish GVLQA, the first vision-language question-answering dataset for general graph reasoning purposes. It can be used to thoroughly evaluate the structure-based graph reasoning abilities of LLMs/VLMs and can also be used as pretraining data to boost the performance of downstream tasks.

Extensive experiments on benchmark datasets across various graph reasoning tasks demonstrate the effectiveness of the proposed GITA framework and the benefits of layout augmentation on visual graphs.

2 Related Work

Graph Reasoning. Graph reasoning [5, 63] aims to answer questions based on graphs, which involves utilizing graph structures to guide the reasoning process to generate answers. Graph reasoning has a wide variety of applications in social network analysis [47, 41], bioinformatics [31, 18], chemistry [19], physics [4], knowledge graph reasoning [6], and recommendation systems [39, 26]. Many graph reasoning methods have been proposed. Early attempts [6, 53] learn node and edge representations through shallow modules, which may have only limited expressive power. Graph Neural Networks (GNNs) such as GCN [37], GAT [58], GraphSAGE [23], MPNN [19], and GIN [64] use messagepassing paradigm [19] to model graph dependencies and update node features. Transformer-based

graph models [71, 40, 8] further propose to use self-attention to increase the expressiveness and longrange dependency. However, as discussed in Sec 1, these models may exhibit limited generalizability, flexibility, and user-friendliness.

LLMs on Graph Reasoning. There have been many attempts to use LLMs in graph reasoning. Depending on how they align the input spaces of graphs and LLMs, we categorize them into two types: Graph-to-text and Graph-to-token. Graph-to-text methods transform a graph into textual descriptions, which are concatenated with the instructions and fed to the LLM for querying. For example, InstructGLM [66] uses natural language to describe the graph and proposes instruction prompts to fine-tune the LLM. He et.al [27] applies LLMs to explain graphs for training GNNs, while Chen et.al [10] treat LLMs as enhancers to exploit text attributes or as predictors for node classification on text-attributed graphs. GPT4Graph [21] and LLMtoGraph [44] convert graphs into specific code or natural language formats by the powerful ChatGPT [48, 49]. On the other hand, Graph-to-token methods include GraphGPT [55], GraphToken [50] and LLaGA [9]. For these methods, the graph is represented as a specially designed token sequence, which is projected or merged into the LLM's token space for text-based reasoning. However, none of the aforementioned methods represent the graph structure information as images, highlighting the uniqueness of the proposed GITA framework and GVLQA dataset.

Large Vision-Language Models. Large VLMs have significantly expanded the cognitive abilities of LLMs by integrating the vision modality to address vision-language tasks. Many methods have been proposed. Some early explorations like Flamingo [2], CLIP [51], and BLIP-2 [43] use a visual encoder for processing images and align the visual and textual embeddings. Subsequent models like LLaVA [45] and MiniGPT-4 [75] combine visual and textual inputs in a single LLM for solving multimodal tasks. InstructBlip [11] proposes an instruction-aware query transformer and trains a vision-language model by instruction tuning. However, despite progress in a wide range of visionlanguage tasks [70, 16], using visual information in graph reasoning remains overlooked. We take the first step in this field, pushing the boundaries of VLMs in graph reasoning.

3 GITA: Graph to Visual and Textual Integration

3.1 Preliminary

Graph Reasoning. In traditional graph reasoning settings, models typically rely on two main inputs: (i) the graph structure G = { C, E } , where C and E are the set of vertices and edges, respectively; (ii) the task requirement T , encompassing specific operations or questions pertaining to the graph. Based on the information provided in G and a specific task requirement T , models are expected to output a reasonable answer A . On the other hand, in the context of instruction-based graph reasoning methods, it is necessary to convert these inputs into textual form. This transformation facilities graph reasoning within natural language, allowing for improved interpretation and harnessing the formidable reasoning capabilities of large language models.

3.2 Architecture

Overview. Different from the above graph reasoning methods, we propose a G raph to I mageT xt A ssistant (GITA), which is the first attempt to perform graph reasoning in a vision-text-based manner. GITA comprises four pivotal components: a task-agnostic graph visualizer V , a graph describer D , a task-specific questioner Q , and a VLM reasoner R ϕ , as illustrated in Figure 1. Firstly, V and G are designed to produce visual depictions (i.e., visual graphs ) and textual descriptions of the graph structure inputs, respectively. Then, given the task requirement T and the textual description produced by D , Q is designed to form a task-specific query. Finally, R ϕ receives the visual input I G from V based on the visual graph and the textual input Q T G from Q , then generates answers A in natural language. In the following, we introduce the four components in detail.

Graph Visualizer. The role of the graph visualizer is to generate visual graphs from structural graphs. The image representation of a structural graph is not unique, as there can be variations in many aspects, such as backdrop colors, layouts, and node shapes. These variations may enhance the robustness of models through effective training but simultaneously increase the learning difficulty for models. Therefore, balancing consistency and variety is necessary during the graph visualization process. This trade-off is reflected in our design of graph visualizer, by maintaining consistency in

Graph

Data

Configurations

Base Image Styles

Graph-related

Image Styles

Textual Description

In a directed graph,

(i,j) means that node i and

node j are connected with

an undirected edge.

The nodes are  numbered

from 0 to 6, and the edges

are:

(0,2) (2,6) ...

Task-agnostic Graph

Describing

Visual Graph

2

6

1

3

5

4

0

OR

OR ...

(w/ diff

structure-

aware configs)

2

6

1

3

5

4

0

Input tokens

...

The nodes are

numbered from

0 to 6, and the

edges are:

(0,2)

(2,6)

(1,4)

...

Traditional LLM Solutions

Graph

Data

Instructions

Q: Is there a cycle

in this graph?

Q: Is there a path

between

node 0 and node 4?

...

Response:

Yes.

Task

...

Task-based

Questioner

Task

Task-specific Query

In a directed graph with 7 nodes numbered from 0 to 6:

node 0 should be visited before node 2,

...

This task is to find a valid topological sorting for this

directed graph.

Please provide a possible topological ordering path, for

example: 0 -> 1 -> 2 -> 3 -> 4. Q: The topological order

of the directed graph is:

VLM   Reasoner

Task-agnostic descriptions

Response:

0 -> 2 -> 6 -> 3 -> 1 -> 4 -> 5.

Customizable Image Styles

Task Responsibility

Output Specification

Refined descriptions

(w/ the meanings of nodes/edges)

basic image styles common to general images (i.e., size, resolution, backdrop) and only introducing customizable variations in four graph-related image styles unique to visual graphs (i.e., layout, node shapes, node outline styles, and edge thickness). Graph visualization in V can be formulated by the following equation:

where I G denotes the visual graph derived from graph G , while Γ and ∆ are the fixed basic image styles and customizable graph-related image styles, respectively.

Visualizing the entire graph can be challenging when the number of nodes or edges is very large, affecting the clarity of the images. To address this, our graph visualizer adopts the standard strategy of k -hop subgraph sampling. Specifically, k -hop subgraph sampling for a node u in the set of vertices C involves selecting a subgraph G u = { C u ⊆ N k ( u ) , E u ⊆ E } , where N k ( u ) includes nodes within k steps from u and each edge ( i, j ) in E u connects nodes within C u . To generate the visual graph of the k -hop subgraph G u centered on u , the nodes within G u are relabeled from 0 to | C u | -1 to facilitate the generalization of visual graphs. Subsequently, this relabeled subgraph G u is fed to the graph visualizer to generate its visual graph I G u by Eq. (1).

In practice, the graph visualizer can be implemented by a variety of graphic visualization tools, such as Graphviz [17], Matplotlib [56], and NetworkX [22]. Among them, Graphviz can automatically design the layouts of visual graphs, and is especially suitable for building large-scale datasets. Matplotlib is excellent for customizable plots with fine-grained control, and NetworkX excels in complex network analysis. We have implemented various graph visualizers using modular, plug-in architecture in GITA. Specific examples of the visual graphs generated with these tools can be found in Appendix D.

Graph Describer. The graph describer D is tasked with generating task-agnostic textual descriptions of a given graph G . To ensure clarity and fidelity of these descriptions, we meticulously craft a curated set of graph-describing templates . The graph description templates outlined in Appendix E are designed to cover a broad spectrum of scenarios, accommodating various graph configurations including directed or undirected graphs and those with or without node or edge weights. To generate the description for a given graph, the graph describer initially selects an appropriate template based on the graph's characteristics, such as its directionality and whether it includes node attributes or edge weights. Subsequently, this template is used by replacing placeholders with actual data, such as the number of nodes, the number of edges, and the endpoints of each edge, to craft detailed descriptions tailored to the specific graph in question. The process for D to generate textual descriptions can be formulated as follows:

where D G denotes the textual description generated by graph describer, and P is the graph-describing template of the graph G .

By introducing these unified and structured graph-describing templates, the graph describer is empowered to generate coherent and informative descriptions that focus on the inherent characteristics of the graph itself, independent of specific task requirements.

Questioner. The questioner Q is tailored to capture the intricate requirements of specific tasks and reflect them in its output task-specific query. In detail, Q receives the task-agnostic textual

descriptions from the graph describer and refines them to align with the task context by elucidating the concrete meanings of nodes and edges. These refined descriptions are then enriched with task responsibilities and input/output specifications to form task-specific queries. The formulation of the questioner to generate the task-specific queries can be represented as follows:

where Q T G represents the task-specific query generated by the questioner with given the task requirement T and the textual description D G . The construction of task-specific queries can be approached in two main ways: manual template-based construction and bootstrapping LLM agents. Manual template-based construction enriches D G with task-specific manual templates, which is preferred for tasks with precise requirements, such as the Traveling Salesman Problem (TSP) [12], where accuracy is critical and the task definitions are well-understood. This is because it can ensure clarity and reduce the risk of errors due to its meticulous attention to details. On the other hand, bootstrapping LLM agents for automated synthesis is more economical and suitable for dynamic or bespoke tasks, such as robotic planning or complex gaming scenarios, as it can take advantage of the speed and adaptability of LLM agents to interpret context and generate appropriate queries, minimizing manual effort and enhancing responsiveness to changing conditions. Both methods are illustrated with examples in Appendix F, showcasing their applications and benefits in different scenarios.

VLMReasoner. The VLM reasoner R ϕ performs final graph reasoning with visual inputs I G from V and textual inputs Q T G from Q , and outputs responses in natural language. This reasoning process can be represented as the following:

where A is the answer generated by the vision-language model R . In this work, we adopt GPT-4V and LLaVA-7B/13B as VLM reasoners. These models are regarded as representatives in the realm of closed-source and open-source VLMs, respectively.

In summary, GITA systematically incorporates the vision modality into instruction-based graph reasoning. In Appendix B, we discuss the characteristics of GITA, in aspects of generalizability, flexibility and user-friendliness.

3.3 Visual Graph Augmentation

Visual graphs generated for the same graph G can be considered as an unique data augmentation technique. Building on the four graph-related image styles introduced in the graph visualizer part of Sec 3.2, we propose the following augmentation strategies: layout augmentation , node shape augmentation , node outline style augmentation , and edge thickness augmentation . Specifically, layout augmentation involves altering the layout styles while keeping all the other settings constant. Similarly, by changing only the respective attributes, we can implement node shape augmentation, node outline style augmentation, and edge thickness augmentation. These four proposed augmentation strategies facilitate studies on the importance of each in enhancing the graph reasoning abilities of VLM reasoners.

3.4 Training

Given a visual graph I G and a text-specific query Q T G , along with the target answer A t , the VLM reasoner of GITA is trained to generate answers A . Specifically, I G is input into the vision encoder of the VLM reasoner, resulting in a set of visual features F . If there is a dimension difference between F v and the pretrained word embeddings, these F will be aligned with the pretrained word embedding space of the text decoder by a vision-to-text projector. Finally, the aligned visual features F aligned and Q T G are concatenated as input sequences of the text decoder.

Formally, given I G , Q T G , and A t , the VLM reasoner is trained by minimizing the following negative log-likelihood:

where ϕ is the trainable parameter and A i denotes the prediction token at the i -th position. Besides, A <i represents the first i -1 predicted tokens. During the inference process, GITA is capable of accepting structure graphs as inputs and performing graph reasoning in an end-to-end manner.

4 GVLQA Dataset

In this section, we introduce the GVLQA dataset to fill the absence of a vision-language-based general graph reasoning dataset. It is designed to: 1) evaluate the graph reasoning capabilities of VLMs or LLMs; 2) help models acquire fundamental graph comprehension and reasoning abilities as a pretraining dataset.

4.1 Construction

The GVLQA dataset is created by utilizing the graph visualizerthe graph describer, and questioner in GITA to generate vision-language-based question-answer pairs for graph reasoning on an open-source graph dataset. Specifically, we first extract both the original graph structures and the ground-truth outputs from the NLGraph-full dataset [59]. Then the graph visualizer (detailed in Sec 3.2) and the graph describer (outlined in Sec 3.2) are used to generate visual graphs and textual descriptions for these original graph structures, respectively. Afterwards, the questioner (described in Sec 3.2) further improves and enriches the textual descriptions by converting them into textual queries. At the same time, it transforms the ground-truth output into text-based answers, following specific output requirements. By combining these visual graphs, textual queries, and text-based answers, we obtain the Graph-based Vision-Language Question Answering (GVLQA) dataset.

In the process of establishing GVLQA, we employed graphviz [17] to instantiate the graph visualizer. This choice is made due to its multitude of pre-defined layout algorithms, which enable convenient adjustment of visual graph layouts. Additionally, manual template-based constructed queries are utilized as the questioner because these tasks are famous with well-defined requirements.

4.2 Structure

The GVLQA dataset comprises 526K samples, each consisting of a visual graph, a textual query, and its corresponding answer. It is divided into five subsets: GVLQA-BASE, and four augmentation subsets GVLQA-AUGLY, GVLQA-AUGNS, GVLQA-AUGNO, and GVLQA-AUGET. In GVLQABASE, the visual graphs are uniformly styled. The remaining four augmentation subsets are derived from GVLQA-BASE through the four visual graph augmentations (Sec 3.3), varying in six different layouts, three node shapes, four node outline styles, and four degrees of edge thickness, respectively. Detailed statistics of the four subsets are shown in Table 6 of Appendix C.

Each GVLQA subset undergoes evaluation across seven graph reasoning tasks, outlined as follows.

Connectivity [52] (denoted Connect): Determine whether two randomly selected nodes u and v in an undirected graph are connected.

Cycle [52]: Identify whether a cycle exists in an undirected graph.

Topological Sort [35] (denoted TS): Find a valid topological sort for a directed acyclic graph. Here, topological sort outputs a linear ordering of the nodes such that for every directed edge u ← v , node u comes before v in the ordering.

Shortest Path [13] (denoted SP): Find the shortest path between two nodes in a weighted undirected graph. The shortest path between two nodes is the path connecting the two nodes with the minimum sum of edge weights along the path.

Maximum Flow [15] (denoted MaxFlow): Calculate the maximum flow from a source node to a sink node in a network graph.

Bipartite Graph Matching [36] (denoted BGM): Find a matching set in a bipartite graph with the largest number of edges. A matching set is a collection of edges in which no two edges share any common node.

Hamilton Path [20] (denoted HP): Find a valid Hamilton path in an undirected graph. A Hamiltonian path is a path that traverses each node in a graph exactly once.

Figure 6 offers illustrations for these tasks in the GVLQA-BASE dataset. Illustrations of all the GVLQA subsets are provided in Appendix H.

5 Experiments

In this section, we extensively evaluate the performance of LLM baselines and the proposed GITA on the GVLQA-BASE and five real-world datasets. To better clarify the reasoning capabilities of solely visual graphs, we also test GITA without the textual descriptions of graphs, which can be considered as a variant of GITA and denoted as vision-only (VO). In this case, the visual graph is the only information source for graph reasoning. Additionally, we investigate the importance of visual graph augmentation (Sec 3.3) strategies, by comparison GITA-7B trained on GVLQA-BASE and on the other augmentation subsets of GVLQA (Sec 4.2). Lastly, we investigate the effectiveness of using GVLQA as the pretrained dataset on real-world datasets. The evaluation metrics for all experiments are accuracy by exact matching. For the fine-tuning setting, we fine-tune the LoRA adapters [28] for all weight matrices in the text decoder of the VLM reasoner, while keeping the vision encoder in the VLM reasoner frozen. More detailed experimental settings are in Appendix G.

5.1 Evaluation on the GVLQA-BASE Dataset

In this subsection, we perform experiments on the GVLQA-BASE dataset to compare GITA with popular LLMs including GPT-4 Turbo [49], LLaMA2-7B/13B [57], and Vicuna-7B/13B [73], under both zero-shot and fine-tuning settings. The experimental results are shown in Table 1. Based on these results, we can obtain the following observations.

Observation 1: GITA Outperforms LLM Baselines. As can be seen in Table 1, GITA consistently outperforms the LLM baselines under the same setting. This underscores its SOTA effectiveness in instruction-based graph reasoning tasks, showing robust capabilities across different parameter scales under both fine-tuning and zero-shot settings. Moreover, under the fine-tuning setting, incorporating the vision modality consistently benefits 7B models. But for the 13B models, the performance of some tasks may degrade. This could be attributed to the greater challenge of aligning representations of the visual and textual modalities in the larger 13B models compared to the 7B models, in the case of only fine-tuning LoRA adapters in the text decoder. We speculate that full training could potentially address this issue. However, we leave this as future work due to resource constraints.

Observation 2: Mainstream Open-source VLM/LLMs Lack Fundamental Graph Reasoning Abilities. The zero-shot results illustrate that prominent open-source LLMs or VLMs, including LLaMA2, Vicuna, and LLaVA, exhibit minimal graph reasoning capabilities on the GVLQA-BASE dataset. Specifically, these models produce random answers, i.e., randomly responding with either "Yes." or "No." for tasks involving Connect and Cycle, resulting in a performance close to 50%. Cur-

rent SOTA closed-source LLMs or VLMs, including GPT-4 Turbo and GPT-4V, demonstrate superior zero-shot performance compared with the aforementioned open-source models. This observation implies that current open-source LLMs and VLMs lack basic graph reasoning ability, which may be attributed to the insufficient availability of relevant training data. Such observation also enhances our motivation to propose the GVLQA dataset, with the aim of improving the graph reasoning capabilities of VLMs/LLMs.

Observation 3: Increasing Model Size Leads to Better Graph Reasoning Capabilities. The comparison of VLMs/LLMs with different parameter sizes, specifically 7B and 13B models, verify the benefits of increasing the model size for graph reasoning capabilities. In this regard, GITA-13B outperforms its counterpart with 7B parameters (GITA-7B) both on average and across four of the seven tasks. However, it is worth noting that GITA-13B does not outperform GITA-7B on the other three tasks. We hypothesize that this discrepancy may be attributed to insufficient modality alignment due to LoRA fine-tuning.

Observation 4: Vision and Text Modalities Proficient in Different Types of Graph Reasoning Tasks. We explore the individual capabilities of the visual and textual modalities within the GITA framework. The results indicate that the text and vision modalities can complement each other and contribute to better performance than individual ones, as removing either modality leads to performance drops in most cases (Vicuna & GITA (VO) and GPT-4 Turbo & GITA (VO) in Table 1). While the graph reasoning capability provided by the vision modality may not be as strong as that of the text modality in most cases, relying solely on vision still enables the model to possess basic graph reasoning abilities. Specifically, the model outperforms text-based LLMs in 2 of the 7 tasks (Cycle and BGM) when relying solely on vision. This consistent improvement across all comparison groups demonstrates the potential of the vision modality to excel in certain graph reasoning tasks, leveraging its ability to capture visual patterns like cycles and graph properties such as bipartition. In contrast, text exhibits a higher proficiency than vision modality in sequence-related graph reasoning problems, particularly on tasks such as TS, SP, and HP, which require constructing ordered node sequences.

5.2 Evaluation for the Visual Graph Augmentations

To assess the impact of the proposed visual graph augmentation strategies (including layout, node shape, node outline style, and edge thickness augmentations), we compare the performance of visiononly GITA-7B models trained on the four augmented subsets of GVLQA and on GVLQA-BASE

(without augmentation). The results are presented in Table 2. To fully utilize the visual information in visual graphs, we fine-tune the visual encoder of VLMs in addition to the vision-to-text projector and the LoRA adapters within the text decoder in this experiment.

As can be seen from the results, a significant enhancement in overall performance is observed with the introduction of layout augmentation (GVLQA-AUGLY). The average performance improves remarkably from 38.91% to 63.36%. Notably, significant improvements are observed on SP (5.72% to 76.55%), HP (1.11% to 70.74%), and Connect (59.97% to 87.18%). These findings highlight the critical role of layout augmentation in generating visual graphs. In other words, this observation suggests the potential for creating larger-scale datasets for vision-language-based graph reasoning, which could significantly contribute to advancing this field. Conversely, the other three augmentations do not yield such substantial performance improvements, further emphasizing the importance of layout augmentation in vision-language-based reasoning.

5.3 Evaluation on Real-World Datasets

In this section, we study the effectiveness of GITA on the ca-GrQC [42] and ca-HepTh [42] datasets for the link prediction task, and on the PolBlog [1], Cora [65] and CiteSeer [65] datasets for the node classification task. Table 8 in the appendix C presents the statistics of these datasets. The graph can have thousands of nodes/edges, making it infeasible to feed the entire graph into the model. Consequently, we employ k -hop subgraph sampling (with k = 2 ) discussed in Sec 3.2 to satisfy the token length restriction of LLMs and visual graph scope effectively.

The experimental results are presented in Table 3. It is evident that GITA consistently outperforms the LLM baselines, and its performance progressively improves with the addition of layout augmentation and the use of the GVLQA checkpoint. Notably, we emphasize the advantages of using GVLQABASE as a pretrained dataset by comparing it with GITA-7b. Performance improvements of 0.67% and 0.92% are observed in the zero-shot and fine-tuning settings, respectively. This highlights the potential application value of the proposed GVLQA dataset.

5.4 Comparison of GITA with Dedicated Graph Baselines

Though GITA is designed for language-based general graph reasoning settings, which are much more user-friendly (by user-readable natural language) and general (unique model architecture for various scenarios) than the typical application of dedicated GNNs, it remains essential to conduct a comprehensive comparison with specialized GNNs to elucidate the strengths and limitations of GITA's applicability and capabilities. To this end, we assess the graph reasoning abilities of GITA against dedicated GNNs, including GCN [38] and SAGE [24], using the GVLQA-Base dataset, as detailed in Table 4. In addition, we explore and compare the effects of k -hop subgraph sampling on the proposed GITA and GNN baselines. Using the ca-Hepth dataset, we analyze the impact of increasing the number of hops k on the reasoning time and performance of both GITA and GNNs. The results are in Table 5.

Overall Graph Reasoning Ability Comparison. As shown in Table 4, compared to the dedicated GNNs, the fine-tuned GITA-7B models have comparable average graph reasoning performance, with the larger GATA-13B model performs slightly better. In particular, compared to GNNs, the GITA model shows a stronger ability in recognizing local structures in the graphs (Connect and Cycle) and to accomplish tasks with obvious layout heuristics (BGM). We believe that this advantage comes from GITA's visual perception. For SP and MaxFlow, GITA's performance is inferior to GNNs. This may be because GNNs process edge weight more effectively through the message-passing mechanism.

Scalability and Performance Variation with Different Numbers of Hops k . The inference time results are shown in Table 5. As can be seen, GITA demonstrates inferior scalability compared to the GNN baselines. Its scalability remains stable as the sampled graph size (i.e., k ) increases.

From the accuracy results in Table 5, GITA, GCN, and SAGE achieve their peak performance at k = 2 , suggesting that a small sampled graph size suffices for optimal performance. Though the dedicated GNNs attain higher peak performance than GITA, they exhibit performance declines as k increases (e.g., 3 or 4), while GITA's performance is more stable w.r.t. k .

5.5 Case Study

In this section, we present examples of graph information provided in both visual and textual formats, which offer some intuitive interpretations for our experimental results. Figure 2 (a) shows an example where the GITA-7B (VO) model outperforms its LLMs-based counterpart, and Figure 2 (b) shows an opposite scenario.

The task depicted in Figure 2(a) is cycle detection and the correct answer is 'No'. This is predicted successfully by the vision-only GITA7B model, while the text-based Vicuna-7B fails. In this example, recognizing cycle patterns is much easier in visual graphs, whereas text-based

In an undirected graph,

(i,j) means that node i

and node j are

connected with an

undirected edge.

The nodes are

numbered from 0 to 9,

and the edges are: (9,4)

(1,3) (8,1) (4,0) (4,7)

(2,7) (5,7) (2,8) (6,9)

Is there a cycle in this

undirected graph?

Visual  Graph

Ground Truth: No.

Vicuna-7B: Yes.

(a)

In an undirected graph,

the nodes are numbered

from 0 to 6, and the

edges are:

an edge between node 0

and node 5 with weight 1,

an edge between node 2

and node 0 with weight 1

...

Q: Give the shortest path

from node 4 to node 0:

Visual Graph

Vicuna-7B: 4->6->0.

(b)

LLMs struggle with disordered textual descriptions of edges, which could inherently involve greater complexity and more challenges.

On the other hand, the fixed layout of visual graphs presented in GVLQA-BASE may impede the visual encoder in identifying the shortest path between two nodes, although we have verified layout augmentation can greatly improve the graph reasoning abilities of models, as shown in Sec. 5.2. This limitation might arise from the confusion caused by the visual distance within an image, without considering the weights between the nodes. For instance, in Figure 2(b), the correct answer is '4->6->0', which visually appears as a more convoluted path but numerically has a shorter path length of 3 = 1 + 2 . In contrast, the incorrect answer given by GITA-7B (vision-only) is '4->2->0', which has a higher path length cost of 4 = 1 + 3 but visually seems like a more direct shortcut. This observation further validates the effectiveness of employing layout augmentation to enhance performance in this task. Layout variations of visual graphs play a crucial role in mitigating the visual confusion caused by the spatial arrangement within a visual graph. However, it seems more effective for text-based LLMs to handle explicitly separated nodes and weights, as illustrated by the text (in red) in Figure 2(b).

6 Conclusion

In this paper, we propose an end-to-end framework called GITA for vision-language-based graph reasoning. Extensive experiments validate the superiority of incorporating visual information into instruction-based graph reasoning. Furthermore, we conduct comparative analysis of the four proposed visual graph augmentations and identify layout augmentation as the most effective approach for enhancing visual graphs. This finding offers valuable insights for the development of largerscale datasets aimed at facilitating vision-language-based graph reasoning. Lastly, we highlight the potential application value of the proposed GVLQA dataset as a pretrained dataset.

References

Lada A Adamic and Natalie Glance. The political blogosphere and the 2004 us election: divided they blog. In Proceedings of the 3rd international workshop on Link discovery , pages 36-43, 2005.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 39-48, 2016.

Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems , 29, 2016.

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems , 26, 2013.

Qi Cao, Huawei Shen, Jinhua Gao, Bingzheng Wei, and Xueqi Cheng. Popularity prediction on social platforms with coupled graph neural networks. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 70-78, 2020.

Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning , pages 3469-3489. PMLR, 2022.

Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.08170 , 2024.

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (LLMs) in learning on graphs. arXiv preprint arXiv:2307.03393 , 2023.

W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 2023.

George Dantzig, Ray Fulkerson, and Selmer Johnson. Solution of a large-scale travelingsalesman problem. Journal of the operations research society of America , 2(4):393-410, 1954.

Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik , 1959.

Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In Proceedings of International Conference on Learning Representations , 2024.

Lester Randolph Ford and Delbert R Fulkerson. Maximal flow through a network. Canadian Journal of Mathematics , 1956.

Shuai Fu, Xiequn Wang, Qiushi Huang, and Yu Zhang. Nemesis: Normalizing the soft-prompt vectors of vision-language models. In Proceedings of International Conference on Learning Representations , 2024.

Emden R Gansner and Stephen C North. An open graph visualization system and its applications to software engineering. Software: practice and experience , 30(11):1203-1233, 2000.

Anne-Claude Gavin, Patrick Aloy, Paola Grandi, Roland Krause, Markus Boesche, Martina Marzioch, Christina Rau, Lars Juhl Jensen, Sonja Bastuck, Birgit Dümpelfeld, et al. Proteome survey reveals modularity of the yeast cell machinery. Nature , 440(7084):631-636, 2006.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Ronald J Gould. Advances on the hamiltonian problem-a survey. Graphs and Combinatorics , 2003.

Jiayan Guo, Lun Du, and Hengyu Liu. GPT4Graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066 , 2023.

Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM(United States), 2008.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 639-648, 2020.

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web , pages 173-182, 2017.

Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph representation learning. In Proceedings of International Conference on Learning Representations , 2024.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021.

Chao Huang, Huance Xu, Yong Xu, Peng Dai, Lianghao Xia, Mengyin Lu, Liefeng Bo, Hao Xing, Xiaoping Lai, and Yanfang Ye. Knowledge-aware coupled graph neural network for social recommendation. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 4115-4122, 2021.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6700-6709, 2019.

Hawoong Jeong, Sean P Mason, A-L Barabási, and Zoltan N Oltvai. Lethality and centrality in protein networks. Nature , 411(6833):41-42, 2001.

Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James Kwok. Forward-backward reasoning in large language models for mathematical verification. In Findings of the Association for Computational Linguistics , 2024.

Weisen Jiang, Yu Zhang, and James Kwok. Effective structured-prompting by meta-learning and representitive verbalizer. In International Conference on Machine Learning , 2023.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2901-2910, 2017.

Arthur B Kahn. Topological sorting of large networks. Communications of ACM , 1962.

Richard M Karp, Umesh V Vazirani, and Vijay V Vazirani. An optimal algorithm for on-line bipartite matching. In Proceedings of the twenty-second annual ACM symposium on Theory of computing , 1990.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems , 34:21618-21629, 2021.

Jure Leskovec, Lars Backstrom, Ravi Kumar, and Andrew Tomkins. Microscopic evolution of social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 462-470, 2008.

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graph evolution: Densification and shrinking diameters. ACM transactions on Knowledge Discovery from Data (TKDD) , 1(1):2-es, 2007.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , 2023.

Chang Liu and Bo Wu. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308.11224 , 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems , 2023.

Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. Mask and reason: Pre-training knowledge graph transformers for complex logical queries. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1120-1130, 2022.

Mark EJ Newman. The structure and function of complex networks. SIAM review , 45(2):167256, 2003.

OpenAI. GPT-3.5. Technical report, 2022.

OpenAI. GPT-4 Turbo. Technical report, 2023.

Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.05862 , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Robert Sedgewick. Algorithms in C, part 5: graph algorithms . Pearson Education, 2001.

Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems , 26, 2013.

Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6418-6428, 2019.

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.13023 , 2023.

Sandro Tosi. Matplotlib for Python developers . Packt Publishing Ltd, 2009.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat , 1050(20):10-48550, 2017.

Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In NeurIPS , 2023.

Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua. Learning intents behind interactions with knowledge graph for recommendation. In Proceedings of the web conference 2021 , pages 878-887, 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Proceedings of Neural Information Processing Systems , 2022.

Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. Kicgpt: Large language model with knowledge in context for knowledge graph completion. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 8667-8683, 2023.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems , 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134 , 2023.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models. In Proceedings of International Conference on Learning Representations , 2024.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6720-6731, 2019.

Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5317-5327, 2019.

Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mmllms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 , 2024.

Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140 , 2020.

Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs. Advances in Neural Information Processing Systems , 34:19172-19183, 2021.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS (Datasets and Benchmarks Track) , 2023.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of International Conference on Learning Representations , 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

A Visual Modality Enhances Effectiveness by Uncovering Critical Substructures

In this section, we present a case study to highlight the complementary role of the visual modality in graph reasoning tasks. The visual modality excels at recognizing beneficial substructures or local patterns, which are crucial for effective graph reasoning. For instance, identifying the "hop number" is essential for shortest path calculations, recognizing "leaf nodes" is vital for topological sorting, and detecting "cycles" is necessary to avoid in Hamilton path construction. We extracted these substructures from the GVLQA-Base dataset and manually labeled them. By employing a frozen Vision Transformer (ViT) in the LLaVA framework with a trainable Multi-Layer Perceptron (MLP) decoder, we achieved identification accuracies of 89.92%, 95.16%, and 92.39% for hop number counting, leaf node identification, and cycle detection, respectively. In contrast, using a pre-trained BERT model with the same trainable MLP decoder resulted in significantly lower accuracies of 55.47%, 26.33%, and 60.32% for the same tasks. Therefore, the enhanced effectiveness of integrating visual and textual modalities can be attributed to the additional structural information provided by the visual modality, which facilitates the identification of these critical substructures.

B Advantages of GITA Over Traditional Graph Neural Networks

GITA offers several advantages over traditional Graph Neural Networks (GNNs) in terms of generalizability, flexibility, and user-friendliness:

Unlike GNNs, which require task-specific feature engineering and architecture adjustments, GITA employs a unified model architecture for all tasks, demonstrating its generalizability . By separating task specifications from graph structures, GITA can handle various graph reasoning tasks seamlessly. Additionally, it exhibits strong zero-shot capabilities, allowing it to perform well on tasks it has not been explicitly trained on, which is a feature not commonly found in traditional GNNs.

Besides, traditional GNNs often demand specialized knowledge in model architectures and coding to accommodate diverse tasks, posing a challenge for non-experts. In contrast, GITA overcomes this barrier by employing language-based templates for task adaptation, enhancing its flexibility . This flexibility enables GITA to effectively handle a broad spectrum of tasks, offering a framework that can be customized to specific requirements using daily language, without the necessity of profound expertise in graph neural networks.

Moreover, by leveraging existing VLMs, GITA can respond in natural language, allowing for intuitive graph reasoning with simple queries like "Is there a cycle in this graph?" This stands in contrast to the unreadable vector representations typically used in GNNs, significantly enhancing GITA's user-friendliness .

C Datasets Statistics

D Illustrations for Visualization Tools in Graph Visualizer

The GITA graph visualizer incorporates a variety of implementations for existing visualization tools such as Graphviz, Matplotlib with NetworkX, and igraph, each selected for their unique capabilities

in graph rendering. These tools are implemented in our code as interchangeable modules, enhancing flexibility based on the requirements of different projects.

5

(a)Graphviz

(b）Networkx+

(c）Igraph+Matplotlib

Figure 3 showcases some visual graphs produced by these different graph visualizer implementations.

E Graph-describing Templates

The graph describer relies on a set of unified structured templates designed to generate coherent and informative descriptions that emphasize the inherent characteristics of the graph itself, regardless of specific task requirements. These graph-describing templates cover various scenarios, including directed graphs, undirected graphs, graphs with node identities or features, and graphs with edge weights or capacities. Table 9 provides an illustration of these templates, where [P] denotes placeholders required to be filled by corresponding graph information.

F Examples of Manual-template-based and LLM-agent-bootstrapped Query Generation

Manual-template-based Query Generation. The queries Q T G can be generated by task-specific manual templates. These templates are manually crafted by human to supplement descriptions/instructions about 1) concrete meanings of nodes and edges, 2) task responsibilities and 3) input/output specifications into the task-agnostic graph description D G . Therefore, the precision and faith of generated task-specific queries Q T G are guaranteed by human calibrations. An example of manual-templatebased query generation for topological sorting is illustrated in Figure 4. In this example, placeholders [P] are used to represent information that scripts will automatically fill in.

Textual Description

In a directed graph,

(i,j) means that node i and

node j are connected with

an undirected edge.

The nodes are  numbered

from 0 to 6, and the

edges are:

(0,2)

(2,6)

...

Task-specific Query

In a directed graph with 7

nodes numbered from 0 to 6:

node 0 should be visited before node

2,

...

This task is to find a valid topological

sorting for this directed graph.

Please provide a possible

topological ordering path, for

example: 0 -> 1 -> 2 -> 3 -> 4. Q:

The topological order of the directed

graph is:

Manual-template-based Questioner

Manual Template:

In a directed graph with [P] nodes numbered from [P] to

[P]:

node [P] should be visited before node [P],

...

This task is to find a valid topological sorting for this

directed graph.

Please provide a possible topological ordering path, for

example: 0 -> 1 -> 2 -> 3 -> 4. Q: The topological order of

the directed graph is:

T ask-agnostic textual description

T ask Responsibility meanings of nodes/edges

Output Specification

Figure 4: Examples of the manual-template-based query generation, where [P] denotes the placeholders.

LLM-agent-bootstrapped Query Generation. Figure 5 presents an example of employing a bootstrapped LLM agent, such as ChatGPT[48], for monster-hunting gaming. By incorporating task-specific information into the prompt, including node/edge meanings and task responsibilities, the LLM agent automatically generates a response that serves as the desired task-specific query. Compared to using manual templates, bootstrapping LLM agents for automated synthesis is more flexible and economic as it can take advantage of LLM agents to automatically interpret context and generate appropriate queries for various scenarios and minimize manual effort with changing conditions. Such properties make it suitable for dynamic or bespoke tasks, such as robotic planning or complex gaming scenarios.

Manual-template-based Questioner

Graph description:

In a directed graph, (i,j) means that

node i and node j are connected with an undirected

edge. The nodes are numbered from 0 to 3, and the

edges are:

(0,1) (1,2) (2,3) ...

You are required to help refine the graph description with

1) concrete nodes/edges meaning:

Node -> Monsters

Edge-> Two monsters coupled and hunting one of them

will irritate another.

2) task responsibility:

hunting

monster each by each without irritating more than 1.

3) A

short sentence to rule input/output specification, for

example, a sentence like "The output should be only a

number" or "The output should be only Yes. or No.".

Directly give me the refined graph description:

Task-specific Query

In a directed graph, each node represents a monster, and an undirected

edge between two nodes signifies that the corresponding monsters are

coupled. If one monster is hunted, it will irritate the other monster it is

coupled with. The nodes are numbered from 0 to 3, and the edges are

defined as follows:

(0,1) - Monster 0 and Monster 1 are coupled.

(1,2) - Monster 1 and Monster 2 are coupled.

(2,3) - Monster 2 and Monster 3 are coupled.

...

The task responsibility is to hunt each monster one by one without

irritating more than one monster at a time.

Input/Output Specification: The output should be a sequence of numbers

representing the order in which monsters can be hunted without irritating

more than one monster at a time.

T ask Responsibility

meanings of nodes/edges

Output Specification

Prompt

T ask-agnostic textual description

Textual

Description

LLM Agent (ChatGPT)

Text-specific

information

G Experiment Settings

For all fine-tuning experiments, we use a batch size of 128 and adopt the AdamW optimizer (with a learning rate of 0.0002 and 0.00002 for the LoRA adapters within the text decoder and vision-to-text projector, respectively).

Detailed Settings for GVLQA Dataset During the evaluation, the temperature is set to 0 for all baselines. All fine-tuning experiments are conducted on an NVIDIA DGX station with 8×A100 GPUs. We split the GVLQA dataset in the ratio of 7:3 for training and testing, respectively. The accuracy (%) metrics are computed by comparing the prediction and ground truths with exact matching. We use the next-token-prediction loss to fine-tune the LoRA [28] adapters of LLMs and the vision-to-text projector. Visual graphs are encoded as visual embeddings by a visual encoder. Visual embeddings are concatenated with the embeddings of textual descriptions and instructions (i.e., questions), then fed to the text decoder to generate the answer.

Real-world Datasets Here we provide more details about the five real-world datasets used in Sec 5.3. The datasets ca-GrQC and ca-HepTh represent collaboration networks from the arXiv sections of General Relativity and Quantum Cosmology, and High Energy Physics - Theory, respectively, featuring nodes as authors and edges as co-authorships. They can be downloaded from Stanford Network Analysis Project (SNAP) website 6 . PolBlogs is a network of U.S. political blogs from February 2005, categorized by political alignment and linked by blog references. Cora and CiteSeer are both citation networks, where nodes correspond to scientific papers and edges to citations, utilized for tasks such as document classification and citation analysis, with papers categorized into various research fields. Statistics of the datasets are shown in Table 8. For each dataset, 80% / 10% / 10% of the edges are randomly used for training/validation/testing, respectively.

Detailed Settings for Real-world Benchmarks In the conventional semi-supervised node classification setting, class labels are available for some nodes, which is reflected in the visual graph by coloring the nodes with a unique random color for each class. To focus on evaluating the model's ability to capture structural information, GITA filters out the influence of node features in Cora and CiteSeer datasets. For link prediction tasks on ca-GrQC and ca-HepTh datasets, GITA treats the graphs as undirected. In the test split, both the original links and their reverse links do not appear in the train and valid splits. During training and evaluation, an equal number of negative sampled links are used alongside the positive links. These negative links are sampled at each training epoch

but remain fixed during evaluation. For the GVLQA pretrain checkpoint, GITA adopts the 7B cycle checkpoint finetuned on GVLQA-BASE, where the performance is nearly mature. Hyperparameter combinations for each model are determined through grid search, and the specific combinations can be found in the provided code.

H Illustrations of GVLQA subsets

In this section, we present the illustrations of the GVLQA subsets. Figure 6 provides an overview of GVLQA-BASE. Subsequently, from Figure 7 to Figure 10, we showcase the augmented visual graphs in GVLQA-AUGLY (augment layouts), GVLQA-AUGNS (augment node styles), GVLQA-AUGNO (augment node outline styles), and GVLQA-AUGET (augment edge thickness), respectively.

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 5, and the

edges are: (0,5) (0,1)

(0,3) (1,5) (1,3) (2,4)

(3,5).

Q

: Is there a path between node 1 and node 5

in this undirected graph?

A

:

Yes

.

Visual graph

Textual description

1. Connectivity

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 4, and the

edges are: (1,0) (1,4)

(3,1) (2,0).

Q

: Is there a cycle in this undirected graph?

A

:

No

.

Visual graph

Textual description

2. Cycle

This diagram depicts

a directed graph, in

which each directed

edge from node A to

node B signifies that,

according to the

topological order,

node A must precede

directed graph with 5

nodes numbered from

0 to 4: node 1 should

be visited before

Q

: The topological order of the directed graph is:

A

:

1,4,3,0,2

.

Visual graph

Textual description

3. Topology Sort

This graphic illustrates

an undirected graph,

with each edge's

distance or length

indicated by a

numerical label in close

proximity. In a

undirected graph, the

nodes are numbered

from 0 to 4, and the

edges are: an edge

between node 1 and

node 0 with weight 1,

an edge ...

Q

: Give the shortest path from node 3 to node 1:

A

:

3->2->0->1

.

Visual graph

Textual description

4. Shortest Path

This graphic illustrates a

directed graph, with each

edge's capacity indicated by a

numerical label in close

proximity. In this directed

graph, the nodes are

numbered from 0 to 5, and the

edges are: an edge from node

1 to node 3 with capacity 9, an

edge from node 2 to node 0

with capacity 9, an edge from

node 2 to node 3 with capacity

7, an edge from node 4 to node

3 with capacity 1, an edge from

node 4 to node 2 with ...

Q

: What is the maximum flow from node 4 to node 3:

A

:

14.

Visual graph

Textual description

5. Maximum Flow

There are 4 hosts

numbered from 0 to 3,

and 3 tasks numbered

from 0 to 2. Each host

has a set of tasks that

it is interested in: Host

2 is interested in task

2. Host 2 is interested

in task 1 ... However,

each host is capable of

solving only one

task, and similarly,

each task can be

resolved by just one

host.

Q

: What is the maximum number of hosts that can be

assigned a task they are interested in?

A

:

2

.

Visual graph

Textual description

6. Bipartite Graph Matching

In this undirected graph,

(i,j) means that node i and

node j are connected with

an undirected edge. The

nodes are numbered from

0 to 10, and the edges are:

(7,1) (10,3) (0,6) (5,10)

(5,3) (9,8) (7,3) (1,10) (7,4)

(1,5) (4,10) (5,9) (7,5) (6,1)

(5,2) (2,8) (8,5) (4,2) (0,2)

(5,6) (1,4).

Q

: Begin with node 0,  Is there a path in this graph

that visits every node exactly once? If yes, give the

path. Note that in a path, adjacent nodes must be

connected with edges.

A

:

0->6->1->7->3->10->5->9->8->2->4.

Visual graph

Textual description

0

2

1

4

3

5

3

2

1

4

1

4

0

3

2

1

3

3

3

1

4

2

0

10

4

1

2

9

3

0

0

5

9

2

10

8

3

host 3

host 1

task 0

host 2

task 2

task 1

host 0

I Limitation

The GITA framework proposed in the paper, along with its experimental results, exhibits certain limitations that should be acknowledged. Firstly, when dealing with large-scale graphs, the conventional subgraph sampling strategy employed by GITA may result in imbalanced and insufficient sampling, leading to the loss of critical graph structural information. This compromise is necessary to accommodate the limited contextual length of the text-based LLM and the restricted scope of the visual graph. Secondly, due to computational constraints, the fine-tuning procedures in the paper were restricted to the LoRA framework. While this approach offers advantages, a more comprehensive fine-tuning process that considers both visual and text modalities is expected to better align the two and potentially enhance performance. Addressing these limitations should be considered as a future research direction in this field.

In this undirected

graph, (i,j) means

that node i and

node j are

connected with an

undirected edge.

The nodes are

numbered from 0

to 5, and the edges

are: (0,5) (0,1)

(0,3) (1,5) (1,3)

(2,4) (3,5).

Q

: Is there a path between node 1 and node 5

in this undirected graph?

A

:

Yes

.

Visual graph

Textual description

1. Connectivity

Q

: Is there a cycle in this undirected graph?

A

:

No

.

Visual graph

Textual description

2. Cycle

This diagram depicts

a directed graph, in

which each directed

edge from node A to

node B signifies that,

according to the

topological order,

node A must precede

directed graph with 5

nodes numbered from

0 to 4: node 1 should

be visited before

Q

: The topological order of the directed graph is:

A

:

1,4,3,0,2

.

Visual graph

Textual description

3. Topology Sort

This graphic illustrates

an undirected graph,

with each edge's

distance or length

indicated by a

numerical label in close

proximity. In a

undirected graph, the

nodes are numbered

from 0 to 4, and the

edges are: an edge

between node 1 and

node 0 with weight 1,

an edge ...

Q

: Give the shortest path from node 3 to node 1:

A

:

3->2->0->1

.

Visual graph

Textual description

4. Shortest Path

This graphic illustrates a

directed graph, with each

edge's capacity indicated by a

numerical label in close

proximity. In this directed

graph, the nodes are

numbered from 0 to 5, and the

edges are: an edge from node

1 to node 3 with capacity 9, an

edge from node 2 to node 0

with capacity 9, an edge from

node 2 to node 3 with capacity

7, an edge from node 4 to node

3 with capacity 1, an edge from

node 4 to node 2 with ...

Q

: What is the maximum flow from node 4 to node 3:

A

:

14.

Visual graph

Textual description

5. Maximum Flow

There are 4 hosts numbered

from 0 to 3, and 3 tasks

numbered from 0 to 2. Each

host has a set of tasks that it

is interested in: Host 2 is

interested in task 2. Host 2

is interested in task 1 ...

However, each host is

capable of solving only one

task, and similarly, each task

can be resolved by just one

host.

Q

: What is the maximum number of hosts that can be

assigned a task they are interested in?

A

:

2

.

Visual graph

Textual description

6. Bipartite Graph Matching

In this undirected graph,

(i,j) means that node i and

node j are connected with

an undirected edge. The

nodes are numbered from

0 to 10, and the edges are:

(7,1) (10,3) (0,6) (5,10)

(5,3) (9,8) (7,3) (1,10) (7,4)

(1,5) (4,10) (5,9) (7,5) (6,1)

(5,2) (2,8) (8,5) (4,2) (0,2)

(5,6) (1,4).

Q

: Begin with node 0,  Is there a path in this graph

that visits every node exactly once? If yes, give the

path. Note that in a path, adjacent nodes must be

connected with edges.

A

:

0->6->1->7->3->10->5->9->8->2->4.

Visual graph

Textual description

W

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 4, and the

edges are: (1,0) (1,4)

(3,1) (2,0).

N

3

7

3

2

4

1

0

3

2

1

0

3

1

1

4

task 1

host0

task 2

host 2

task O

host 3

host1

6

7

6

10

5

3

9

8

In this undirected

graph, (i,j) means

that node i and

node j are

connected with an

undirected edge.

The nodes are

numbered from 0

to 5, and the edges

are: (0,5) (0,1)

(0,3) (1,5) (1,3)

(2,4) (3,5).

Q

: Is there a path between node 1 and node 5

in this undirected graph?

A

:

Yes

.

Visual graph

Textual description

1. Connectivity

Q

: Is there a cycle in this undirected graph?

A

:

No

.

Visual graph

Textual description

2. Cycle

This diagram depicts

a directed graph, in

which each directed

edge from node A to

node B signifies that,

according to the

topological order,

node A must precede

directed graph with 5

nodes numbered from

0 to 4: node 1 should

be visited before

Q

: The topological order of the directed graph is:

A

:

1,4,3,0,2

.

Visual graph

Textual description

3. Topology Sort

This graphic illustrates

an undirected graph,

with each edge's

distance or length

indicated by a

numerical label in close

proximity. In a

undirected graph, the

nodes are numbered

from 0 to 4, and the

edges are: an edge

between node 1 and

node 0 with weight 1,

an edge ...

Q

: Give the shortest path from node 3 to node 1:

A

:

3->2->0->1

.

Visual graph

Textual description

4. Shortest Path

Q

: What is the maximum flow from node 4 to node 3:

A

:

14.

Visual graph

Textual description

5. Maximum Flow

There are 4 hosts

numbered from 0 to 3,

and 3 tasks numbered

from 0 to 2. Each host

has a set of tasks that

it is interested in: Host

2 is interested in task

2. Host 2 is interested

in task 1 ... However,

each host is capable of

solving only one

task, and similarly,

each task can be

resolved by just one

host.

Q

: What is the maximum number of hosts that can be

assigned a task they are interested in?

A

:

2

.

Visual graph

Textual description

6. Bipartite Graph Matching

In this undirected graph,

(i,j) means that node i and

node j are connected with

an undirected edge. The

nodes are numbered from

0 to 10, and the edges are:

(7,1) (10,3) (0,6) (5,10)

(5,3) (9,8) (7,3) (1,10) (7,4)

(1,5) (4,10) (5,9) (7,5) (6,1)

(5,2) (2,8) (8,5) (4,2) (0,2)

(5,6) (1,4).

Q

: Begin with node 0,  Is there a path in this graph

that visits every node exactly once? If yes, give the

path. Note that in a path, adjacent nodes must be

connected with edges.

A

:

0->6->1->7->3->10->5->9->8->2->4.

Visual graph

Textual description

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 4, and the

edges are: (1,0) (1,4)

(3,1) (2,0).

This graphic illustrates a

directed graph, with each

edge's capacity indicated by a

numerical label in close

proximity. In this directed

graph, the nodes are

numbered from 0 to 5, and the

edges are: an edge from node

1 to node 3 with capacity 9, an

edge from node 2 to node 0

with capacity 9, an edge from

node 2 to node 3 with capacity

7, an edge from node 4 to

node 3 with capacity 1, an

edge from node 4 to node 2

with ...

0

2

1

3

3

2

1

1

4

0

3

2

1

3

2

0

0

5

9

2

10

8

4

10

1

2

3

0

host

task 0

task 2

host

task 1

host0

In this undirected

graph, (i,j) means

that node i and

node j are

connected with an

undirected edge.

The nodes are

numbered from 0

to 5, and the edges

are: (0,5) (0,1)

(0,3) (1,5) (1,3)

(2,4) (3,5).

Q

: Is there a path between node 1 and node 5

in this undirected graph?

A

:

Yes

.

Visual graph

Textual description

1. Connectivity

Q

: Is there a cycle in this undirected graph?

A

:

No

.

Visual graph

Textual description

2. Cycle

This diagram depicts

a directed graph, in

which each directed

edge from node A to

node B signifies that,

according to the

topological order,

node A must precede

directed graph with 5

nodes numbered from

0 to 4: node 1 should

be visited before

Q

: The topological order of the directed graph is:

A

:

1,4,3,0,2

.

Visual graph

Textual description

3. Topology Sort

This graphic illustrates

an undirected graph,

with each edge's

distance or length

indicated by a

numerical label in close

proximity. In a

undirected graph, the

nodes are numbered

from 0 to 4, and the

edges are: an edge

between node 1 and

node 0 with weight 1,

an edge ...

Q

: Give the shortest path from node 3 to node 1:

A

:

3->2->0->1

.

Visual graph

Textual description

4. Shortest Path

Q

: What is the maximum flow from node 4 to node 3:

A

:

14.

Visual graph

Textual description

5. Maximum Flow

There are 4 hosts

numbered from 0 to 3,

and 3 tasks numbered

from 0 to 2. Each host

has a set of tasks that

it is interested in: Host

2 is interested in task

2. Host 2 is interested

in task 1 ... However,

each host is capable of

solving only one

task, and similarly,

each task can be

resolved by just one

host.

Q

: What is the maximum number of hosts that can be

assigned a task they are interested in?

A

:

2

.

Visual graph

Textual description

6. Bipartite Graph Matching

In this undirected graph,

(i,j) means that node i and

node j are connected with

an undirected edge. The

nodes are numbered from

0 to 10, and the edges are:

(7,1) (10,3) (0,6) (5,10)

(5,3) (9,8) (7,3) (1,10) (7,4)

(1,5) (4,10) (5,9) (7,5) (6,1)

(5,2) (2,8) (8,5) (4,2) (0,2)

(5,6) (1,4).

Q

: Begin with node 0,  Is there a path in this graph

that visits every node exactly once? If yes, give the

path. Note that in a path, adjacent nodes must be

connected with edges.

A

:

0->6->1->7->3->10->5->9->8->2->4.

Visual graph

Textual description

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 4, and the

edges are: (1,0) (1,4)

(3,1) (2,0).

This graphic illustrates a

directed graph, with each

edge's capacity indicated by a

numerical label in close

proximity. In this directed

graph, the nodes are

numbered from 0 to 5, and the

edges are: an edge from node

1 to node 3 with capacity 9, an

edge from node 2 to node 0

with capacity 9, an edge from

node 2 to node 3 with capacity

7, an edge from node 4 to

node 3 with capacity 1, an

edge from node 4 to node 2

with ...

0

2

3

5

3

2

1

0

4

1

3

1

3

3

1

4

2

0

1

4

3

2

10

10

1

2

6

9

3

0

host 3

host 1

task 0

host 2

tesk 2

task 1

host 0

In this undirected

graph, (i,j) means

that node i and

node j are

connected with an

undirected edge.

The nodes are

numbered from 0

to 5, and the edges

are: (0,5) (0,1)

(0,3) (1,5) (1,3)

(2,4) (3,5).

Q

: Is there a path between node 1 and node 5

in this undirected graph?

A

:

Yes

.

Visual graph

Textual description

1. Connectivity

Q

: Is there a cycle in this undirected graph?

A

:

No

.

Visual graph

Textual description

2. Cycle

This diagram depicts

a directed graph, in

which each directed

edge from node A to

node B signifies that,

according to the

topological order,

node A must precede

directed graph with 5

nodes numbered from

0 to 4: node 1 should

be visited before

Q

: The topological order of the directed graph is:

A

:

1,4,3,0,2

.

Visual graph

Textual description

3. Topology Sort

This graphic illustrates

an undirected graph,

with each edge's

distance or length

indicated by a

numerical label in close

proximity. In a

undirected graph, the

nodes are numbered

from 0 to 4, and the

edges are: an edge

between node 1 and

node 0 with weight 1,

an edge ...

Q

: Give the shortest path from node 3 to node 1:

A

:

3->2->0->1

.

Visual graph

Textual description

4. Shortest Path

Q

: What is the maximum flow from node 4 to node 3:

A

:

14.

Visual graph

Textual description

5. Maximum Flow

There are 4 hosts

numbered from 0 to 3,

and 3 tasks numbered

from 0 to 2. Each host

has a set of tasks that

it is interested in: Host

2 is interested in task

2. Host 2 is interested

in task 1 ... However,

each host is capable of

solving only one

task, and similarly,

each task can be

resolved by just one

host.

Q

: What is the maximum number of hosts that can be

assigned a task they are interested in?

A

:

2

.

Visual graph

Textual description

6. Bipartite Graph Matching

In this undirected graph,

(i,j) means that node i and

node j are connected with

an undirected edge. The

nodes are numbered from

0 to 10, and the edges are:

(7,1) (10,3) (0,6) (5,10)

(5,3) (9,8) (7,3) (1,10) (7,4)

(1,5) (4,10) (5,9) (7,5) (6,1)

(5,2) (2,8) (8,5) (4,2) (0,2)

(5,6) (1,4).

Q

: Begin with node 0,  Is there a path in this graph

that visits every node exactly once? If yes, give the

path. Note that in a path, adjacent nodes must be

connected with edges.

A

:

0->6->1->7->3->10->5->9->8->2->4.

Visual graph

Textual description

In this undirected graph,

(i,j) means that node i

and node j are connected

with an undirected edge.

The nodes are numbered

from 0 to 4, and the

edges are: (1,0) (1,4)

(3,1) (2,0).

This graphic illustrates a

directed graph, with each

edge's capacity indicated by a

numerical label in close

proximity. In this directed

graph, the nodes are

numbered from 0 to 5, and the

edges are: an edge from node

1 to node 3 with capacity 9, an

edge from node 2 to node 0

with capacity 9, an edge from

node 2 to node 3 with capacity

7, an edge from node 4 to

node 3 with capacity 1, an

edge from node 4 to node 2

with ...

3

2

1

0

4

1

3

1

3

3

1

4

2

0

0

2

3

5

1

3

2

0

10

host 3

task 0

task 2

host 1

host 2

task 1

host 0

0

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to the abstract part and the contribution enumeration at the tail of the introduction part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Appendix I

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not involve any theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show fundamental experiment settings in Section 5, and more details for experiments settings in Appendix G. Besides, we provide the complete codes as supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The complete codes are included, and the proposed GVLQA dataset is released with common access.

Guidelines:

The answer NA means that the paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment details are given in both Section 5 and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the machine (type and storage) requirements in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research does not have concerns about societal impacts because it is designed for general-purpose graph reasoning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The paper includes using an graph visualizer to generate abstract graph images, however, these images are focus on graph structure, without any sensitive information.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited necessary assets and conduct CC-BY 4.0 for our codes and datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and other supplementary materials are followed with readme and instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the GITA framework, which innovatively integrates visual graphs into general graph reasoning tasks for LLMs. By combining visual and textual information of a graph, GITA improves the comprehensibility and flexibility of graph reasoning, outperforming other LLM approaches. The authors also develop the Graph-based Vision-Language Question Answering (GVLQA) dataset, the first vision-language dataset for graph reasoning. The study also highlights the benefits of layout augmentation in visual graphs and pretraining on the GVLQA dataset.\n\nSTRENGTHS:\n1. Integrating visual graphs into graph reasoning tasks for LLMs, is a novel approach. This represents a creative combination of visual and textual modalities to improve graph comprehension and reasoning. The creation of the GVLQA dataset is original and fills a gap in the current datasets available for graph reasoning by incorporating both visual and textual elements.\n\n2.  Extensive experiments on both the GVLQA and five real-world datasets validate the effectiveness of the proposed framework, providing strong empirical evidence for the paper's claims.\n\n3. The paper is clearly written, with well-defined sections and logical flow. The detailed explanations of the GITA framework's components and the dataset construction process enhance understanding.\n\n4.  By successfully integrating visual information into graph reasoning tasks, the paper addresses a significant limitation of existing LLMs and VLMs. This has the potential to substantially advance the field of graph reasoning.\n\nWEAKNESSES:\n1. The motivation of the framework is challenged. As for the commonly used graph reasoning methods such as GNNs, the authors state that \"These methods often lack generalizability, flexibility, and user-friendliness.\" However, the advantages of GITA in these areas are not directly demonstrated. The zero-shot performance of LLM-based methods is poor for many questions (such as MaxFlow, SP, TS). Considering this, users need to fine-tune when addressing questions on a new dataset. Additionally, GITA requires data augmentation and manual template-based construction for task-specific queries. Therefore, it is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified.\n\n2. While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages.\n\n3. The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios. \n\n4. The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed.\n\n5. Some places need double-check. For example, the meaning of GITA needs to be unified: the authors mention GITA as \"Graph to vIsual and Textual Integration\" in line 44, while they refer to it as \"Graph to Image-Txt Assistant\" in line 117. Another example is in line 120, which needs to be changed to \"Firstly, V and D are designed to produce visual depictions and textual descriptions.\"\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\nThe authors have acknowledged several key limitations of their work but could benefit from providing more comprehensive solutions and experimental evidence.  As for my suggestion for improvement, please see the “weaknessess” part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

comments: SUMMARY:
To fill the gap that LLM overlook the rich vision modality with graph structure, this paper proposes GITA to incorporates visual graphs into general graph reasoning. A large graph vision-language dataset called GVLQA is designed to boost the general graph reasoning capabilities.

STRENGTHS:
1. This paper may involve a relatively large workload, proposing a large vision-language dataset, which can greatly promote the development of VLM.
2. Integrating visual graphs into VLM holds significant value.

WEAKNESSES:
1. This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology.
2. This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance.
3. In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.

QUESTIONS:
The dataset will be released in the future? If so, I appreciate it.

LIMITATIONS:
See the weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an end-to-end framework called Graph to Visual and Textual Integration (GITA) to visualize graphs in order to improve LLMs’ reasoning capabilities on graph tasks. GITA consists of three main components: a Graph Describer to translate a graph into a natural language description, a Graph Visualizer to visualize the graph as an image, and a Questioner to generate task-specific queries conditioned on the task information. GITA shows improvements compared to vanilla LLMs and VLMs on graph reasoning tasks and some real-world graph datasets.\n\nSTRENGTHS:\n1. The motivation is convincing. Just as humans reason over structured data, it is natural to visualize the structure first. The additional visual modality can provide rich information that can assist in reasoning over structured data. Therefore, generalizing this human behavior to LLMs/VLMs makes a lot of sense.\n\n2. The method is simple yet effective for some graph reasoning tasks on small graphs. The idea of visualization is a form of data augmentation.\n\nWEAKNESSES:\n1. The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value.\n2. The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets.\n3. Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken [1-3], etc.\n\n[1] Graph Chain-of-Thought, https://arxiv.org/abs/2404.07103\n\n[2] GraphLLM, https://arxiv.org/abs/2310.05845\n\n[3] GraphToken, https://arxiv.org/abs/2402.05862\n\nQUESTIONS:\n1. Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset?\n2. In Table 1, is GITA-7B (VO) equivalent to Llava-7B?\n3. In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?\n\nLIMITATIONS:\nSee weekness above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Graph to Visual and Textual Integration (GITA), a novel framework that enhances graph reasoning by integrating visual representations with traditional text-based processing. GITA's innovation lies in using both visual and textual inputs to address graph-structured data, a significant deviation from the typical text-only approaches in Large Language Models (LLMs). Key contributions include the development of GITA, the creation of the Graph-based Vision-Language Question Answering (GVLQA) dataset—the first for visual-textual graph reasoning—and extensive evaluations showing GITA's superior performance over existing LLMs across various datasets, highlighting the advantage of incorporating visual information into graph reasoning tasks.\n\nSTRENGTHS:\n1. The paper introduces the novel idea of integrating visual graphs with textual descriptions for enhancing graph reasoning tasks. \n2. The creation of the GVLQA dataset is a significant contribution, as it is the first vision-language dataset designed specifically for general graph reasoning.\n\nWEAKNESSES:\n1. While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with the visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well.\n2. I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else.\n3. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.\n\nQUESTIONS:\nplease refer to the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}