GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Authors: yanbin  wei
shuai  fu
weisen  jiang
zejian  zhang
zhixiong  zeng
qi  wu
james t kwok
yu  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:36:52

Abstract: Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish  $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

Main paper: 
 introduction: 
 Graph reasoning tasks are pivotal in domains such as recommendation systems [25, 60] , social network analysis [7, 29] , and knowledge graph reasoning [72, 46, 62] . Various architectures have been developed, from shallow embedding methods [6, 53] to advanced Graph Neural Networks (GNNs) [37, 64] and graph Transformers [71, 40, 8] . While these models excel in graph reasoning tasks, they often lack generalizability, flexibility, and user-friendliness. Achieving good performance with these models typically requires domain-specific tuning, which limits their abilities to generalize across different domains. Additionally, these models struggle to handle diverse tasks with the same architecture. Each task often requires a specialized design, including task-specific data processing and decoder, leading to limited flexibility. Lastly, unlike the Large Language Models (LLMs) that can engage in conversations with users, these models are less explainable and user-friendly.In contrast, LLMs have shown great generalization capabilities across a wide variety of reasoning tasks [61, 67, 74, 32, 33] by encapsulating task-specific demands within a cohesive and interpretable mechanism -text prompts, under a unified architecture with minimal domain-specific adjustments. These advantages have sparked investigations into the potential of LLMs for graph reasoning. Recent developments lend credence to the notion that LLMs can indeed interpret and manipulate graph-structured data through textual representations. For example, InstructGLM [66] , GPT4Graph [21] , and LLMtoGraph [44] convert graphs into textual descriptions and then use these descriptions paired with queries to enable LLMs to generate accurate responses for graph reasoning tasks. Furthermore, the introduction of benchmarks such as GraphQA [14] and NLGraph [59] is a testament to the growing interest in evaluating LLMs' effectiveness on graph reasoning tasks framed in natural languages.Despite the development of numerous methods and benchmarks for graph reasoning on LLMs, they often overlook the valuable vision modality, which is a natural means for humans to comprehend structural information and has demonstrated its success in various visual reasoning scenarios [30, 68, 69, 34, 3, 54] . Consequently, the following questions arise: (1) Can incorporating visual information be beneficial for general graph reasoning scenarios? (2) If so, how can we effectively integrate the vision modality into graph reasoning? To the best of our knowledge, these questions remain unexplored.To answer them, we first propose an end-to-end framework called Graph to vIsual and Textual IntegrAtion (GITA) 34 that systematically integrates visual information into instruction-based graph reasoning, by rendering graph structures to customized visual images which are called visual graph. Specifically, the GITA framework has four components: a graph visualizer for generating visual graphs, a graph describer for producing textual descriptions of the graph structure, a task-based questioner that organizes the description and requirements of the current task into prompt instruction, and a Vision-Language Model (VLM) to perform vision-language graph reasoning. In the proposed GITA framework, visual information can be incorporated into many tasks with explicit or implicit graph structures, without sacrificing its versatility, flexibility, or user-friendliness. Besides, since there is no dataset for vision-supported general graph reasoning capabilities, we construct the first vision-language dataset for general graph reasoning purposes called Graph-based Vision-Language Question Answering (GVLQA) 5 based on the proposed GITA framework. The GVLQA dataset consists of 526K instances covering seven representative graph reasoning tasks, aiming to thoroughly evaluate the structure-based graph reasoning abilities of VLMs and LLMs. Extensive experiments on the GVLQA dataset and five real-world datasets demonstrate the effectiveness of the proposed GITA model. Furthermore, we delve into the effects of visual graph augmentation strategies and find that layout augmentation can dramatically boost vision-based graph reasoning performance.Our main contributions are summarized as follows.• We introduce an end-to-end GITA framework, innovatively integrating vision modality to boost the graph reasoning abilities of language models.• We establish GVLQA, the first vision-language question-answering dataset for general graph reasoning purposes. It can be used to thoroughly evaluate the structure-based graph reasoning abilities of LLMs/VLMs and can also be used as pretraining data to boost the performance of downstream tasks.• Extensive experiments on benchmark datasets across various graph reasoning tasks demonstrate the effectiveness of the proposed GITA framework and the benefits of layout augmentation on visual graphs.In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge.The nodes are numbered from 0 to 4, and the edges are: (1,0) (1,4) (3,1) (2,0). 
related work: 
 Graph Reasoning. Graph reasoning [5, 63] aims to answer questions based on graphs, which involves utilizing graph structures to guide the reasoning process to generate answers. Graph reasoning has a wide variety of applications in social network analysis [47, 41] , bioinformatics [31, 18] , chemistry [19] , physics [4] , knowledge graph reasoning [6] , and recommendation systems [39, 26] . Many graph reasoning methods have been proposed. Early attempts [6, 53] learn node and edge representations through shallow modules, which may have only limited expressive power. Graph Neural Networks (GNNs) such as GCN [37] , GAT [58] , GraphSAGE [23] , MPNN [19] , and GIN [64] use messagepassing paradigm [19] to model graph dependencies and update node features. Transformer-based graph models [71, 40, 8] further propose to use self-attention to increase the expressiveness and longrange dependency. However, as discussed in Sec 1, these models may exhibit limited generalizability, flexibility, and user-friendliness.LLMs on Graph Reasoning. There have been many attempts to use LLMs in graph reasoning. Depending on how they align the input spaces of graphs and LLMs, we categorize them into two types: Graph-to-text and Graph-to-token. Graph-to-text methods transform a graph into textual descriptions, which are concatenated with the instructions and fed to the LLM for querying. For example, InstructGLM [66] uses natural language to describe the graph and proposes instruction prompts to fine-tune the LLM. He et.al [27] applies LLMs to explain graphs for training GNNs, while Chen et.al [10] treat LLMs as enhancers to exploit text attributes or as predictors for node classification on text-attributed graphs. GPT4Graph [21] and LLMtoGraph [44] convert graphs into specific code or natural language formats by the powerful ChatGPT [48, 49] . On the other hand, Graph-to-token methods include GraphGPT [55] , GraphToken [50] and LLaGA [9] . For these methods, the graph is represented as a specially designed token sequence, which is projected or merged into the LLM's token space for text-based reasoning. However, none of the aforementioned methods represent the graph structure information as images, highlighting the uniqueness of the proposed GITA framework and GVLQA dataset.Large Vision-Language Models. Large VLMs have significantly expanded the cognitive abilities of LLMs by integrating the vision modality to address vision-language tasks. Many methods have been proposed. Some early explorations like Flamingo [2] , CLIP [51] , and BLIP-2 [43] use a visual encoder for processing images and align the visual and textual embeddings. Subsequent models like LLaVA [45] and MiniGPT-4 [75] combine visual and textual inputs in a single LLM for solving multimodal tasks. InstructBlip [11] proposes an instruction-aware query transformer and trains a vision-language model by instruction tuning. However, despite progress in a wide range of visionlanguage tasks [70, 16] , using visual information in graph reasoning remains overlooked. We take the first step in this field, pushing the boundaries of VLMs in graph reasoning. (ii) the task requirement T , encompassing specific operations or questions pertaining to the graph.Based on the information provided in G and a specific task requirement T , models are expected to output a reasonable answer A. On the other hand, in the context of instruction-based graph reasoning methods, it is necessary to convert these inputs into textual form. This transformation facilities graph reasoning within natural language, allowing for improved interpretation and harnessing the formidable reasoning capabilities of large language models. 
architecture: 
 Overview. Different from the above graph reasoning methods, we propose a Graph to Image-Txt Assistant (GITA), which is the first attempt to perform graph reasoning in a vision-text-based manner. GITA comprises four pivotal components: a task-agnostic graph visualizer V , a graph describer D, a task-specific questioner Q, and a VLM reasoner R ϕ , as illustrated in Figure 1 . Firstly, V and G are designed to produce visual depictions (i.e., visual graphs) and textual descriptions of the graph structure inputs, respectively. Then, given the task requirement T and the textual description produced by D, Q is designed to form a task-specific query. Finally, R ϕ receives the visual input I G from V based on the visual graph and the textual input Q T G from Q, then generates answers A in natural language. In the following, we introduce the four components in detail.Graph Visualizer. The role of the graph visualizer is to generate visual graphs from structural graphs. The image representation of a structural graph is not unique, as there can be variations in many aspects, such as backdrop colors, layouts, and node shapes. These variations may enhance the robustness of models through effective training but simultaneously increase the learning difficulty for models. Therefore, balancing consistency and variety is necessary during the graph visualization process. This trade-off is reflected in our design of graph visualizer, by maintaining consistency inGraph-related Image StylesIn a directed graph, (i,j) means that node i and node j are connected with an undirected edge.The nodes are numbered from 0 to 6, and the edges are: (0,2) (2,6) ...Visual Graph Large Language Model Instructions Q: Is there a cycle in this graph?Q: Is there a path between node 0 and node 4?...Yes....Task InfoIn a directed graph with 7 nodes numbered from 0 to 6: node 0 should be visited before node 2, ... This task is to find a valid topological sorting for this directed graph. Please provide a possible topological ordering path, for example: 0 -> 1 -> 2 -> 3 -> 4. Q: The topological order of the directed graph is:Response: 0 -> 2 -> 6 -> 3 -> 1 -> 4 -> 5.Task ResponsibilityRefined descriptions (w/ the meanings of nodes/edges) basic image styles common to general images (i.e., size, resolution, backdrop) and only introducing customizable variations in four graph-related image styles unique to visual graphs (i.e., layout, node shapes, node outline styles, and edge thickness). Graph visualization in V can be formulated by the following equation:EQUATIONwhere I G denotes the visual graph derived from graph G, while Γ and ∆ are the fixed basic image styles and customizable graph-related image styles, respectively.Visualizing the entire graph can be challenging when the number of nodes or edges is very large, affecting the clarity of the images. To address this, our graph visualizer adopts the standard strategy of k-hop subgraph sampling. Specifically, k-hop subgraph sampling for a node u in the set of vertices C involves selecting a subgraph G u = {C u ⊆ N k (u), E u ⊆ E}, where N k (u) includes nodes within k steps from u and each edge (i, j) in E u connects nodes within C u . To generate the visual graph of the k-hop subgraph G u centered on u, the nodes within G u are relabeled from 0 to |C u | -1 to facilitate the generalization of visual graphs. Subsequently, this relabeled subgraph G u is fed to the graph visualizer to generate its visual graph I Gu by Eq. (1).In practice, the graph visualizer can be implemented by a variety of graphic visualization tools, such as Graphviz [17] , Matplotlib [56] , and NetworkX [22] . Among them, Graphviz can automatically design the layouts of visual graphs, and is especially suitable for building large-scale datasets. Matplotlib is excellent for customizable plots with fine-grained control, and NetworkX excels in complex network analysis. We have implemented various graph visualizers using modular, plug-in architecture in GITA. Specific examples of the visual graphs generated with these tools can be found in Appendix D.Graph Describer. The graph describer D is tasked with generating task-agnostic textual descriptions of a given graph G. To ensure clarity and fidelity of these descriptions, we meticulously craft a curated set of graph-describing templates. The graph description templates outlined in Appendix E are designed to cover a broad spectrum of scenarios, accommodating various graph configurations including directed or undirected graphs and those with or without node or edge weights. To generate the description for a given graph, the graph describer initially selects an appropriate template based on the graph's characteristics, such as its directionality and whether it includes node attributes or edge weights. Subsequently, this template is used by replacing placeholders with actual data, such as the number of nodes, the number of edges, and the endpoints of each edge, to craft detailed descriptions tailored to the specific graph in question. The process for D to generate textual descriptions can be formulated as follows:EQUATIONwhere D G denotes the textual description generated by graph describer, and P is the graph-describing template of the graph G.By introducing these unified and structured graph-describing templates, the graph describer is empowered to generate coherent and informative descriptions that focus on the inherent characteristics of the graph itself, independent of specific task requirements.Questioner. The questioner Q is tailored to capture the intricate requirements of specific tasks and reflect them in its output task-specific query. In detail, Q receives the task-agnostic textual descriptions from the graph describer and refines them to align with the task context by elucidating the concrete meanings of nodes and edges. These refined descriptions are then enriched with task responsibilities and input/output specifications to form task-specific queries. The formulation of the questioner to generate the task-specific queries can be represented as follows:EQUATIONwhere Q T G represents the task-specific query generated by the questioner with given the task requirement T and the textual description D G . The construction of task-specific queries can be approached in two main ways: manual template-based construction and bootstrapping LLM agents. Manual template-based construction enriches D G with task-specific manual templates, which is preferred for tasks with precise requirements, such as the Traveling Salesman Problem (TSP) [12] , where accuracy is critical and the task definitions are well-understood. This is because it can ensure clarity and reduce the risk of errors due to its meticulous attention to details. On the other hand, bootstrapping LLM agents for automated synthesis is more economical and suitable for dynamic or bespoke tasks, such as robotic planning or complex gaming scenarios, as it can take advantage of the speed and adaptability of LLM agents to interpret context and generate appropriate queries, minimizing manual effort and enhancing responsiveness to changing conditions. Both methods are illustrated with examples in Appendix F, showcasing their applications and benefits in different scenarios.VLM Reasoner. The VLM reasoner R ϕ performs final graph reasoning with visual inputs I G from V and textual inputs Q T G from Q, and outputs responses in natural language. This reasoning process can be represented as the following:EQUATIONwhere A is the answer generated by the vision-language model R. In this work, we adopt GPT-4V and LLaVA-7B/13B as VLM reasoners. These models are regarded as representatives in the realm of closed-source and open-source VLMs, respectively. In summary, GITA systematically incorporates the vision modality into instruction-based graph reasoning. In Appendix B, we discuss the characteristics of GITA, in aspects of generalizability, flexibility and user-friendliness.Visual graphs generated for the same graph G can be considered as an unique data augmentation technique. Building on the four graph-related image styles introduced in the graph visualizer part of Sec 3.2, we propose the following augmentation strategies: layout augmentation, node shape augmentation, node outline style augmentation, and edge thickness augmentation. Specifically, layout augmentation involves altering the layout styles while keeping all the other settings constant. Similarly, by changing only the respective attributes, we can implement node shape augmentation, node outline style augmentation, and edge thickness augmentation. These four proposed augmentation strategies facilitate studies on the importance of each in enhancing the graph reasoning abilities of VLM reasoners.Given a visual graph I G and a text-specific query Q T G , along with the target answer A t , the VLM reasoner of GITA is trained to generate answers A. Specifically, I G is input into the vision encoder of the VLM reasoner, resulting in a set of visual features F . If there is a dimension difference between F v and the pretrained word embeddings, these F will be aligned with the pretrained word embedding space of the text decoder by a vision-to-text projector. Finally, the aligned visual features F aligned and Q T G are concatenated as input sequences of the text decoder. Formally, given I G , Q T G , and A t , the VLM reasoner is trained by minimizing the following negative log-likelihood:EQUATIONwhere ϕ is the trainable parameter and A i denotes the prediction token at the i-th position. Besides, A <i represents the first i -1 predicted tokens. During the inference process, GITA is capable of accepting structure graphs as inputs and performing graph reasoning in an end-to-end manner.This graphic illustrates an undirected graph, with each edge's distance or length indicated by a numerical label in close proximity. In a undirected graph, the nodes are numbered from 0 to 4, and the edges are: an edge between node 1 and node 0 with weight 1, an edge ... 
gvlqa dataset: 
 In this section, we introduce the GVLQA dataset to fill the absence of a vision-language-based general graph reasoning dataset. It is designed to: 1) evaluate the graph reasoning capabilities of VLMs or LLMs; 2) help models acquire fundamental graph comprehension and reasoning abilities as a pretraining dataset.The GVLQA dataset is created by utilizing the graph visualizerthe graph describer, and questioner in GITA to generate vision-language-based question-answer pairs for graph reasoning on an open-source graph dataset. Specifically, we first extract both the original graph structures and the ground-truth outputs from the NLGraph-full dataset [59] . Then the graph visualizer (detailed in Sec 3.2) and the graph describer (outlined in Sec 3.2) are used to generate visual graphs and textual descriptions for these original graph structures, respectively. Afterwards, the questioner (described in Sec 3.2) further improves and enriches the textual descriptions by converting them into textual queries. At the same time, it transforms the ground-truth output into text-based answers, following specific output requirements. By combining these visual graphs, textual queries, and text-based answers, we obtain the Graph-based Vision-Language Question Answering (GVLQA) dataset.In the process of establishing GVLQA, we employed graphviz [17] to instantiate the graph visualizer. This choice is made due to its multitude of pre-defined layout algorithms, which enable convenient adjustment of visual graph layouts. Additionally, manual template-based constructed queries are utilized as the questioner because these tasks are famous with well-defined requirements.The GVLQA dataset comprises 526K samples, each consisting of a visual graph, a textual query, and its corresponding answer. It is divided into five subsets: GVLQA-BASE, and four augmentation subsets GVLQA-AUGLY, GVLQA-AUGNS, GVLQA-AUGNO, and GVLQA-AUGET. In GVLQA-BASE, the visual graphs are uniformly styled. The remaining four augmentation subsets are derived from GVLQA-BASE through the four visual graph augmentations (Sec 3.3), varying in six different layouts, three node shapes, four node outline styles, and four degrees of edge thickness, respectively. Detailed statistics of the four subsets are shown in Table 6 of Appendix C.Each GVLQA subset undergoes evaluation across seven graph reasoning tasks, outlined as follows.• Connectivity [52] (denoted Connect): Determine whether two randomly selected nodes u and v in an undirected graph are connected.• Cycle [52] : Identify whether a cycle exists in an undirected graph.• Topological Sort [35] (denoted TS): Find a valid topological sort for a directed acyclic graph. Here, topological sort outputs a linear ordering of the nodes such that for every directed edge u ← v, node u comes before v in the ordering.• Shortest Path [13] (denoted SP): Find the shortest path between two nodes in a weighted undirected graph. The shortest path between two nodes is the path connecting the two nodes with the minimum sum of edge weights along the path.• Maximum Flow [15] (denoted MaxFlow): Calculate the maximum flow from a source node to a sink node in a network graph.• Bipartite Graph Matching [36] (denoted BGM): Find a matching set in a bipartite graph with the largest number of edges. A matching set is a collection of edges in which no two edges share any common node.• Hamilton Path [20] (denoted HP): Find a valid Hamilton path in an undirected graph. A Hamiltonian path is a path that traverses each node in a graph exactly once.Figure 6 offers illustrations for these tasks in the GVLQA-BASE dataset. Illustrations of all the GVLQA subsets are provided in Appendix H. 
experiments: 
 In this section, we extensively evaluate the performance of LLM baselines and the proposed GITA on the GVLQA-BASE and five real-world datasets. To better clarify the reasoning capabilities of solely visual graphs, we also test GITA without the textual descriptions of graphs, which can be considered as a variant of GITA and denoted as vision-only (VO). In this case, the visual graph is the only information source for graph reasoning. Additionally, we investigate the importance of visual graph augmentation (Sec 3.3) strategies, by comparison GITA-7B trained on GVLQA-BASE and on the other augmentation subsets of GVLQA (Sec 4.2). Lastly, we investigate the effectiveness of using GVLQA as the pretrained dataset on real-world datasets. The evaluation metrics for all experiments are accuracy by exact matching. For the fine-tuning setting, we fine-tune the LoRA adapters [28] for all weight matrices in the text decoder of the VLM reasoner, while keeping the vision encoder in the VLM reasoner frozen. More detailed experimental settings are in Appendix G.In this subsection, we perform experiments on the GVLQA-BASE dataset to compare GITA with popular LLMs including GPT-4 Turbo [49] , LLaMA2-7B/13B [57] , and Vicuna-7B/13B [73] , under both zero-shot and fine-tuning settings. The experimental results are shown in Table 1 . Based on these results, we can obtain the following observations.Observation 1: GITA Outperforms LLM Baselines. As can be seen in Table 1 , GITA consistently outperforms the LLM baselines under the same setting. This underscores its SOTA effectiveness in instruction-based graph reasoning tasks, showing robust capabilities across different parameter scales under both fine-tuning and zero-shot settings. Moreover, under the fine-tuning setting, incorporating the vision modality consistently benefits 7B models. But for the 13B models, the performance of some tasks may degrade. This could be attributed to the greater challenge of aligning representations of the visual and textual modalities in the larger 13B models compared to the 7B models, in the case of only fine-tuning LoRA adapters in the text decoder. We speculate that full training could potentially address this issue. However, we leave this as future work due to resource constraints. 1 ). While the graph reasoning capability provided by the vision modality may not be as strong as that of the text modality in most cases, relying solely on vision still enables the model to possess basic graph reasoning abilities. Specifically, the model outperforms text-based LLMs in 2 of the 7 tasks (Cycle and BGM) when relying solely on vision. This consistent improvement across all comparison groups demonstrates the potential of the vision modality to excel in certain graph reasoning tasks, leveraging its ability to capture visual patterns like cycles and graph properties such as bipartition. In contrast, text exhibits a higher proficiency than vision modality in sequence-related graph reasoning problems, particularly on tasks such as TS, SP, and HP, which require constructing ordered node sequences.To assess the impact of the proposed visual graph augmentation strategies (including layout, node shape, node outline style, and edge thickness augmentations), we compare the performance of visiononly GITA-7B models trained on the four augmented subsets of GVLQA and on GVLQA-BASE (without augmentation). The results are presented in Table 2 . To fully utilize the visual information in visual graphs, we fine-tune the visual encoder of VLMs in addition to the vision-to-text projector and the LoRA adapters within the text decoder in this experiment.As can be seen from the results, a significant enhancement in overall performance is observed with the introduction of layout augmentation (GVLQA-AUGLY). The average performance improves remarkably from 38.91% to 63.36%. Notably, significant improvements are observed on SP (5.72% to 76.55%), HP (1.11% to 70.74%), and Connect (59.97% to 87.18%). These findings highlight the critical role of layout augmentation in generating visual graphs. In other words, this observation suggests the potential for creating larger-scale datasets for vision-language-based graph reasoning, which could significantly contribute to advancing this field. Conversely, the other three augmentations do not yield such substantial performance improvements, further emphasizing the importance of layout augmentation in vision-language-based reasoning.In this section, we study the effectiveness of GITA on the ca-GrQC [42] and ca-HepTh [42] datasets for the link prediction task, and on the PolBlog [1] , Cora [65] and CiteSeer [65] datasets for the node classification task. The experimental results are presented in Table 3 . It is evident that GITA consistently outperforms the LLM baselines, and its performance progressively improves with the addition of layout augmentation and the use of the GVLQA checkpoint. Notably, we emphasize the advantages of using GVLQA-BASE as a pretrained dataset by comparing it with GITA-7b. Performance improvements of 0.67% and 0.92% are observed in the zero-shot and fine-tuning settings, respectively. This highlights the potential application value of the proposed GVLQA dataset. Though GITA is designed for language-based general graph reasoning settings, which are much more user-friendly (by user-readable natural language) and general (unique model architecture for various scenarios) than the typical application of dedicated GNNs, it remains essential to conduct a comprehensive comparison with specialized GNNs to elucidate the strengths and limitations of GITA's applicability and capabilities. To this end, we assess the graph reasoning abilities of GITA against dedicated GNNs, including GCN [38] and SAGE [24] , using the GVLQA-Base dataset, as detailed in Table 4 . In addition, we explore and compare the effects of k-hop subgraph sampling on the proposed GITA and GNN baselines. Using the ca-Hepth dataset, we analyze the impact of increasing the number of hops k on the reasoning time and performance of both GITA and GNNs.The results are in Table 5 .Overall Graph Reasoning Ability Comparison. As shown in Table 4 , compared to the dedicated GNNs, the fine-tuned GITA-7B models have comparable average graph reasoning performance, with the larger GATA-13B model performs slightly better. In particular, compared to GNNs, the GITA model shows a stronger ability in recognizing local structures in the graphs (Connect and Cycle) and to accomplish tasks with obvious layout heuristics (BGM). We believe that this advantage comes from GITA's visual perception. For SP and MaxFlow, GITA's performance is inferior to GNNs. This may be because GNNs process edge weight more effectively through the message-passing mechanism.Scalability and Performance Variation with Different Numbers of Hops k. The inference time results are shown in Table 5 . As can be seen, GITA demonstrates inferior scalability compared to the GNN baselines. Its scalability remains stable as the sampled graph size (i.e., k) increases. From the accuracy results in Table 5 , GITA, GCN, and SAGE achieve their peak performance at k = 2, suggesting that a small sampled graph size suffices for optimal performance. Though the dedicated GNNs attain higher peak performance than GITA, they exhibit performance declines as k increases (e.g., 3 or 4), while GITA's performance is more stable w.r.t. k.Textual QueryIn an undirected graph, (i,j) means that node i and node j are connected with an undirected edge.The nodes are numbered from 0 to 9, and the edges are: Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The complete codes are included, and the proposed GVLQA dataset is released with common access.Guidelines:• The answer NA means that the paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we propose an end-to-end framework called GITA for vision-language-based graph reasoning. Extensive experiments validate the superiority of incorporating visual information into instruction-based graph reasoning. Furthermore, we conduct comparative analysis of the four proposed visual graph augmentations and identify layout augmentation as the most effective approach for enhancing visual graphs. This finding offers valuable insights for the development of largerscale datasets aimed at facilitating vision-language-based graph reasoning. Lastly, we highlight the potential application value of the proposed GVLQA dataset as a pretrained dataset.In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 10, and the edges are: (Q: Begin with node 0, Is there a path in this graph that visits every node exactly once? If yes, give the path. Note that in a path, adjacent nodes must be connected with edges.A: 0->6->1->7->3->10->5->9->8->2->4. In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 10, and the edges are: In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 5, and the edges are: (0,5) (0,1) (0,3) (1,5) (1,3) (2,4) (3, 5) .Q: Is there a path between node 1 and node 5 in this undirected graph? A: Yes. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Experiment details are given in both Section 5 and Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not include error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We report the machine (type and storage) requirements in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We make sure the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The research does not have concerns about societal impacts because it is designed for general-purpose graph reasoning.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we present a case study to highlight the complementary role of the visual modality in graph reasoning tasks. The visual modality excels at recognizing beneficial substructures or local patterns, which are crucial for effective graph reasoning. For instance, identifying the "hop number" is essential for shortest path calculations, recognizing "leaf nodes" is vital for topological sorting, and detecting "cycles" is necessary to avoid in Hamilton path construction. We extracted these substructures from the GVLQA-Base dataset and manually labeled them. By employing a frozen Vision Transformer (ViT) in the LLaVA framework with a trainable Multi-Layer Perceptron (MLP) decoder, we achieved identification accuracies of 89.92%, 95.16%, and 92.39% for hop number counting, leaf node identification, and cycle detection, respectively. In contrast, using a pre-trained BERT model with the same trainable MLP decoder resulted in significantly lower accuracies of 55.47%, 26.33%, and 60.32% for the same tasks. Therefore, the enhanced effectiveness of integrating visual and textual modalities can be attributed to the additional structural information provided by the visual modality, which facilitates the identification of these critical substructures.GITA offers several advantages over traditional Graph Neural Networks (GNNs) in terms of generalizability, flexibility, and user-friendliness:Unlike GNNs, which require task-specific feature engineering and architecture adjustments, GITA employs a unified model architecture for all tasks, demonstrating its generalizability. By separating task specifications from graph structures, GITA can handle various graph reasoning tasks seamlessly. Additionally, it exhibits strong zero-shot capabilities, allowing it to perform well on tasks it has not been explicitly trained on, which is a feature not commonly found in traditional GNNs.Besides, traditional GNNs often demand specialized knowledge in model architectures and coding to accommodate diverse tasks, posing a challenge for non-experts. In contrast, GITA overcomes this barrier by employing language-based templates for task adaptation, enhancing its flexibility. This flexibility enables GITA to effectively handle a broad spectrum of tasks, offering a framework that can be customized to specific requirements using daily language, without the necessity of profound expertise in graph neural networks.Moreover, by leveraging existing VLMs, GITA can respond in natural language, allowing for intuitive graph reasoning with simple queries like "Is there a cycle in this graph?" This stands in contrast to the unreadable vector representations typically used in GNNs, significantly enhancing GITA's user-friendliness. The GITA graph visualizer incorporates a variety of implementations for existing visualization tools such as Graphviz, Matplotlib with NetworkX, and igraph, each selected for their unique capabilities Figure 3 showcases some visual graphs produced by these different graph visualizer implementations.The graph describer relies on a set of unified structured templates designed to generate coherent and informative descriptions that emphasize the inherent characteristics of the graph itself, regardless of specific task requirements. These graph-describing templates cover various scenarios, including directed graphs, undirected graphs, graphs with node identities or features, and graphs with edge weights or capacities. Table 9 provides an illustration of these templates, where [P] denotes placeholders required to be filled by corresponding graph information.Manual-template-based Query Generation. The queries Q T G can be generated by task-specific manual templates. These templates are manually crafted by human to supplement descriptions/instructions about 1) concrete meanings of nodes and edges, 2) task responsibilities and 3) input/output specifications into the task-agnostic graph description D G . Therefore, the precision and faith of generated task-specific queries Q T G are guaranteed by human calibrations. An example of manual-templatebased query generation for topological sorting is illustrated in Figure 4 . In this example, placeholders [P] are used to represent information that scripts will automatically fill in.In an undirected graph, (i,j) means that node i and node j are connected with an undirected edge. In a directed graph, (i,j) means that node i and node j are connected with an undirected edge.The nodes are numbered from 0 to 6, and the edges are: (0,2) (2,6) ...In a directed graph with 7 nodes numbered from 0 to 6: node 0 should be visited before node 2, ... This task is to find a valid topological sorting for this directed graph. Please provide a possible topological ordering path, for example: 0 -> 1 -> 2 -> 3 -> 4. Q:The topological order of the directed graph is: LLM-agent-bootstrapped Query Generation. Figure 5 presents an example of employing a bootstrapped LLM agent, such as ChatGPT [48] , for monster-hunting gaming. By incorporating task-specific information into the prompt, including node/edge meanings and task responsibilities, the LLM agent automatically generates a response that serves as the desired task-specific query. Compared to using manual templates, bootstrapping LLM agents for automated synthesis is more flexible and economic as it can take advantage of LLM agents to automatically interpret context and generate appropriate queries for various scenarios and minimize manual effort with changing conditions. Such properties make it suitable for dynamic or bespoke tasks, such as robotic planning or complex gaming scenarios.Graph description: In a directed (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 3, and the edges are: (0,1) (1,2) (2, 3) ... You are required to help refine the graph description with 1) concrete nodes/edges meaning: Node -> Monsters Edge-> Two monsters coupled and hunting one of them will irritate another. 2) task responsibility: hunting monster each by each without irritating more than 1. 3) A short sentence to rule input/output specification, for example, a sentence like "The output should be only a number" or "The output should be only Yes. or No.". Directly give me the refined graph description:In a directed graph, each node represents a monster, and an undirected edge between two nodes signifies that the corresponding monsters are coupled. If one monster is hunted, it will irritate the other monster it is coupled with. The nodes are numbered from 0 to 3, and the edges are defined as follows: (0,1) -Monster 0 and Monster 1 are coupled.(1,2) -Monster 1 and Monster 2 are coupled.(2,3) -Monster 2 and Monster 3 are coupled. ... The task responsibility is to hunt each monster one by one without irritating more than one monster at a time.Input/Output Specification: The output should be a sequence of numbers representing the order in which monsters can be hunted without irritating more than one monster at a time.Output Specification For all fine-tuning experiments, we use a batch size of 128 and adopt the AdamW optimizer (with a learning rate of 0.0002 and 0.00002 for the LoRA adapters within the text decoder and vision-to-text projector, respectively).Detailed Settings for GVLQA Dataset During the evaluation, the temperature is set to 0 for all baselines. All fine-tuning experiments are conducted on an NVIDIA DGX station with 8×A100 GPUs. We split the GVLQA dataset in the ratio of 7:3 for training and testing, respectively. The accuracy (%) metrics are computed by comparing the prediction and ground truths with exact matching. We use the next-token-prediction loss to fine-tune the LoRA [28] adapters of LLMs and the vision-to-text projector. Visual graphs are encoded as visual embeddings by a visual encoder. Visual embeddings are concatenated with the embeddings of textual descriptions and instructions (i.e., questions), then fed to the text decoder to generate the answer.Real-world Datasets Here we provide more details about the five real-world datasets used in Sec 5.3. The datasets ca-GrQC and ca-HepTh represent collaboration networks from the arXiv sections of General Relativity and Quantum Cosmology, and High Energy Physics -Theory, respectively, featuring nodes as authors and edges as co-authorships. They can be downloaded from Stanford Network Analysis Project (SNAP) website 6 . PolBlogs is a network of U.S. political blogs from February 2005, categorized by political alignment and linked by blog references. Cora and CiteSeer are both citation networks, where nodes correspond to scientific papers and edges to citations, utilized for tasks such as document classification and citation analysis, with papers categorized into various research fields. Statistics of the datasets are shown in Table 8 . For each dataset, 80%/10%/10% of the edges are randomly used for training/validation/testing, respectively.Detailed Settings for Real-world Benchmarks In the conventional semi-supervised node classification setting, class labels are available for some nodes, which is reflected in the visual graph by coloring the nodes with a unique random color for each class. To focus on evaluating the model's ability to capture structural information, GITA filters out the influence of node features in Cora and CiteSeer datasets. For link prediction tasks on ca-GrQC and ca-HepTh datasets, GITA treats the graphs as undirected. In the test split, both the original links and their reverse links do not appear in the train and valid splits. During training and evaluation, an equal number of negative sampled links are used alongside the positive links. These negative links are sampled at each training epoch but remain fixed during evaluation. For the GVLQA pretrain checkpoint, GITA adopts the 7B cycle checkpoint finetuned on GVLQA-BASE, where the performance is nearly mature. Hyperparameter combinations for each model are determined through grid search, and the specific combinations can be found in the provided code.In this section, we present the illustrations of the GVLQA subsets. Figure 6 provides an overview of GVLQA-BASE. Subsequently, from Figure 7 to Figure 10 , we showcase the augmented visual graphs in GVLQA-AUGLY (augment layouts), GVLQA-AUGNS (augment node styles), GVLQA-AUGNO (augment node outline styles), and GVLQA-AUGET (augment edge thickness), respectively.In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 5, and the edges are: (0,5) (0,1) (0,3) (1,5) (1,3) (2,4) (3,5).Q: Is there a path between node 1 and node 5 in this undirected graph? A: Yes.Textual descriptionIs there a cycle in this undirected graph? A: No.This diagram depicts a directed graph, in which each directed edge from node A to node B signifies that, according to the topological order, node A must precede node B. In this directed graph with 5 nodes numbered from 0 to node 1 should be visited before node 0 ...The topological order of the directed graph is: A: 1,4,3,0,2.Give the shortest path from node 3 to node 1:A: 3->2->0->1.This graphic illustrates a directed graph, with each edge's capacity indicated by a numerical label in close proximity. In this directed graph, the nodes are numbered from 0 to 5, and the edges are: an edge from node 1 to node 3 with capacity 9, an edge from node 2 to node 0 with capacity 9, an edge from node 2 to node 3 with capacity 7, an edge from node 4 to node 3 with capacity 1, an edge from node 4 to node 2 with ...What is the maximum flow from node 4 to node 3: A: 14.There are 4 hosts numbered from 0 to 3, and 3 tasks numbered from 0 to 2. Each host has a set of tasks that it is interested in: Host 2 is interested in task 2. Host 2 is interested in task 1 ... However, each host is capable of solving only one task, and similarly, each task can be resolved by just one host.Q: What is the maximum number of hosts that can be assigned a task they are interested in? A: 2.Textual descriptionThe GITA framework proposed in the paper, along with its experimental results, exhibits certain limitations that should be acknowledged. Firstly, when dealing with large-scale graphs, the conventional subgraph sampling strategy employed by GITA may result in imbalanced and insufficient sampling, leading to the loss of critical graph structural information. This compromise is necessary to accommodate the limited contextual length of the text-based LLM and the restricted scope of the visual graph. Secondly, due to computational constraints, the fine-tuning procedures in the paper were restricted to the LoRA framework. While this approach offers advantages, a more comprehensive fine-tuning process that considers both visual and text modalities is expected to better align the two and potentially enhance performance. Addressing these limitations should be considered as a future research direction in this field.In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge.The nodes are numbered from 0 to 5, and the edges are: (0,5) (0,1) (0,3) (1,5) (1,3) (2,4) (3, 5) .Q: Is there a path between node 1 and node 5 in this undirected graph? A: Yes. There are 4 hosts numbered from 0 to 3, and 3 tasks numbered from 0 to 2. Each host has a set of tasks that it is interested in: Host 2 is interested in task 2. Host 2 is interested in task 1 ... However, each host is capable of solving only one task, and similarly, each task can be resolved by just one host.Q: What is the maximum number of hosts that can be assigned a task they are interested in? A: 2.Textual descriptionThere are 4 hosts numbered from 0 to 3, and 3 tasks numbered from 0 to 2. Each host has a set of tasks that it is interested in: Host 2 is interested in task 2. Host 2 is interested in task 1 ... However, each host is capable of solving only one task, and similarly, each task can be resolved by just one host.Q: What is the maximum number of hosts that can be assigned a task they are interested in? A: 2.In this undirected graph, (i,j) means that node i and node j are connected with an undirected edge. The nodes are numbered from 0 to 10, and the edges are: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Project Homepage: v-graph.github.io.Code Repository: https://github.com/WEIYanbin1999/GITA/.Dataset: https://huggingface.co/collections/Yanbin99/.https://snap.stanford.edu/index.html 


Decision: The paper presents GITA, a framework that integrates visual graphs generated from text inputs to enhance graph reasoning tasks in LLMs. The creation of the GVLQA dataset, specifically designed for vision-language graph reasoning, is a valuable contribution that fills a gap in the current landscape of graph reasoning datasets. The paper demonstrates promising results across multiple datasets, showcasing GITA’s effectiveness, particularly in local structure recognition and layout-sensitive tasks. Although some concerns were raised regarding scalability and comparisons to GNNs, the authors effectively addressed these by providing additional experiments and detailed explanations in their rebuttal. 

All the reviewers comments have been addressed. All the scores recommend acceptance. Congrats to the authors.

Reviews: review: paper_summary: paper_summary: This paper introduces Graph to Visual and Textual Integration (GITA), a novel framework that enhances graph reasoning by integrating visual representations with traditional text-based processing. GITA's innovation lies in using both visual and textual inputs to address graph-structured data, a significant deviation from the typical text-only approaches in Large Language Models (LLMs). Key contributions include the development of GITA, the creation of the Graph-based Vision-Language Question Answering (GVLQA) dataset—the first for visual-textual graph reasoning—and extensive evaluations showing GITA's superior performance over existing LLMs across various datasets, highlighting the advantage of incorporating visual information into graph reasoning tasks.
strength_weakness: strength_weakness: 1. While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with the visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well.
2. I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else.
3. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.
questions: questions: please refer to the weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces an end-to-end framework called Graph to Visual and Textual Integration (GITA) to visualize graphs in order to improve LLMs’ reasoning capabilities on graph tasks. GITA consists of three main components: a Graph Describer to translate a graph into a natural language description, a Graph Visualizer to visualize the graph as an image, and a Questioner to generate task-specific queries conditioned on the task information. GITA shows improvements compared to vanilla LLMs and VLMs on graph reasoning tasks and some real-world graph datasets.
strength_weakness: strength_weakness: 1. The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value.
2. The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets.
3. Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken [1-3], etc.

[1] Graph Chain-of-Thought, https://arxiv.org/abs/2404.07103

[2] GraphLLM, https://arxiv.org/abs/2310.05845

[3] GraphToken, https://arxiv.org/abs/2402.05862
questions: questions: 1. Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset?
2. In Table 1, is GITA-7B (VO) equivalent to Llava-7B?
3. In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?
limitations: limitations: See weekness above
paper_summary: paper_summary: To fill the gap that LLM overlook the rich vision modality with graph structure, this paper proposes GITA to incorporates visual graphs into general graph reasoning. A large graph vision-language dataset called GVLQA is designed to boost the general graph reasoning capabilities.
strength_weakness: strength_weakness: 1. This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology.
2. This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance.
3. In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.
questions: questions: The dataset will be released in the future? If so, I appreciate it.
limitations: limitations: See the weakness
paper_summary: paper_summary: The paper introduces the GITA framework, which innovatively integrates visual graphs into general graph reasoning tasks for LLMs. By combining visual and textual information of a graph, GITA improves the comprehensibility and flexibility of graph reasoning, outperforming other LLM approaches. The authors also develop the Graph-based Vision-Language Question Answering (GVLQA) dataset, the first vision-language dataset for graph reasoning. The study also highlights the benefits of layout augmentation in visual graphs and pretraining on the GVLQA dataset.
strength_weakness: strength_weakness: 1. The motivation of the framework is challenged. As for the commonly used graph reasoning methods such as GNNs, the authors state that "These methods often lack generalizability, flexibility, and user-friendliness." However, the advantages of GITA in these areas are not directly demonstrated. The zero-shot performance of LLM-based methods is poor for many questions (such as MaxFlow, SP, TS). Considering this, users need to fine-tune when addressing questions on a new dataset. Additionally, GITA requires data augmentation and manual template-based construction for task-specific queries. Therefore, it is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified.

2. While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages.

3. The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios. 

4. The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed.

5. Some places need double-check. For example, the meaning of GITA needs to be unified: the authors mention GITA as "Graph to vIsual and Textual Integration" in line 44, while they refer to it as "Graph to Image-Txt Assistant" in line 117. Another example is in line 120, which needs to be changed to "Firstly, V and D are designed to produce visual depictions and textual descriptions."
questions: questions: Please see the above weaknesses.
limitations: limitations: The authors have acknowledged several key limitations of their work but could benefit from providing more comprehensive solutions and experimental evidence.  As for my suggestion for improvement, please see the “weaknessess” part.

Comments: title: Response
Response
comment: Dear Reviewers,

We extend our heartfelt gratitude for your dedicated time and effort in reviewing our paper. Your constructive questions and valuable feedback have been immensely beneficial to our manuscript, and we will incorporate them into our revisions.

Thank you once again for your invaluable contributions to the review process.

Best regards,

Authors
Thank you to the authors for the detailed response. I have no further questions and will raise my score to 5.
> **Q2**. For W3, the average performance of GraphLLM and GITA is similar. I am conservative about the contribution of GITA over existing methods.

We would like to highlight that the contributions of GITA are unique compared with existing methods such as GraphLLM.

**1. Pioneering Use of Vision in Language-Based Graph Reasoning**: GITA is the first to explore the effectiveness of vision in language-based graph reasoning. We believe that this vision benefit (our contribution) is not covered by existing methods like GraphLLM. As illustrated, our experiments show that GITA and GraphLLM excel in different types of graph understanding tasks. Therefore, the methods proposed by us (vision modality) and by them (graph modality) are expected to be combined and complement each other in our future work. 

**2. Superior Performance**: Though GraphLLM and GITA-7B have similar average performance, when apply our proposed vision augmentation, GITA-VO(AUGLY) can be more powerful than GraphLLM in overall performance (i.e., 63.36\% in Table 2 vs. 60.36\%). Besides, zero-shot GITA can also achieve comparable average performance (59.73\% in rebuttal supplement Table 3) with GraphLLM, where the former is training-free. 


------------------------------

> **Q3.** For the new experiments during rebuttal, for table 5-7, the comparison with GNNs is an important baseline and should be displayed in the main text.

According to your suggestion, we will add them accordingly in the main text of the revision. 

> **Q4.** I am skeptical about the necessity of using LLM with visual information to address these tasks given simple GNNs have already perform good with great efficiency.

GITA and GNNs highlight the distinction solutions between general-purpose solutions and specialized solutions.  As discussed in our motivation and in our response to W1 of Reviewer oE8d, GNNs are not sufficiently flexible, general, or user-friendly for addressing general graph reasoning tasks, with the following points:

**1. Ease of Adaptation (Flexibility)**:  GNNs often need to be combined with other models to meet specific task requirements. For instance, sequence output tasks (such as SP, HP, TS) require integration with LSTM, transformers, etc. In contrast, GITA can achieve this with a unified model architecture. Besides, adapting GNNs to specific tasks requires modifications within the model structure, necessitating a background in deep learning and coding skills. GITA, on the other hand, only requires language skills that all humans possess to fit task variations.

**2. Zero-Shot Capability (Generalizability)**: GNNs lack zero-shot capabilities, whereas GITA has promising zero-shot capabilities (as demonstrated in rebuttal Table 3).

**3. Human-Readable Operations and serve as agent (User-friendliness)**: GNNs operate on unreadable vectors, while GITA operates on human-understandable language and images. Therefore, though it is not as efficient as GNNs, it can serve as an agent for answering graph questions with natural language in acceptable time, whereas GNNs show inabilities because they can not handle language.

Therefore, GITA is a more flexible, general and user-friendly framework for general-purpose graph reasoning than GNNs. 

------------------------------------

**To sum up**, this paper is the first work to conduct general-purpose vision-language graph reasoning, and has illustrated that vision could bring overall advances on LLM-based graph reasoning. As an initial direction, it is not as mature as GNNs with years explored. However, it has shown comparable overall performance and special superiorities (flexibility, generalizability, user-friendliness, and promising zero-shot capabilities), demonstrating it is a promising direction to explore as a general solution for graph reasoning.
Thanks for your detailed, patient and insightful discussion. We respond to these questions as follows.

> **Q1**. Regarding W1, my concern about the limited applications remains. I acknowledge that K-hop sampling is a widely adopted approach and may not lead to performance degradation in some cases. However, for datasets like Cora and CiteSeer, the reported accuracy is much lower compared to pure GNN methods (Table 3). I am uncertain whether this is due to the k-hop sampling or because GITA only provides graph structures without node/edge attributes. If it is the latter, GITA may not be effective enough for handling graphs with attributes, whether in numerical or text format.


We think the reported inferior performance in Cora and Citeseer to the leading models is mostly because we do not use node attributes, but not due to the k-hop sampling. That is because many methods (e.g., Graph Transformer like UGT, and most GNN variants where $k$ is equivalent to the number of GNN layers) in your listed leaderboard also use k-hop sampling to achieve leading performance.

However, we want to point out that though using extra information beyond graph structures like node attributes is helpful (in Cora and Citeseer, they provide extra 0-1 binary word occurrences info), it requires specific models to handle them, thus hurts the generalizability, conflicting with our motivation, i.e., a "general-purpose" graph reasoning framework. Note that "generality" is a significant reason for why research works like GraphLLM and our GITA are interested in using LLMs for graph reasoning, where GraphLLM also does not consider the node attributes. 

The hurt for the node attributes to generalizability can be reflected in several aspects: First, handling node attributes typically requires designing specific model tailored to their shapes (e.g., vector dimensions, length, matrix size), which hinders a general and consistent solution for handling diverse tasks. Second, the meanings (e.g., word vectors of titles, embeddings of degree or index, one-hot representation of node classes) of node attributes vary in data, thus the model can overfit in a specific task and hurt the zero-shot abilities. Finally, these diverse models to handle the node attributes also increase the complexity of the model.

As the major concern of GITA is how well vision+LLM performs on general graph scenarios, where node attributes are not necessary provided by default, we only concentrate on pure graph structures and do not incorporate task-specific models for handling extra node attributes. 

Moreover, we can provide several potential solutions for combining GITA with node attributes, and leave them for future work.
1. We can include the text node attributes with the explanation of their concrete meanings inside the text prompt. However, such an approach may not perform well for some types of attributes such as occurrence data, because they are too abstract.
2. Similar to many existing works, we can use a specific module to encode the attributes and another fusion module to combine them with the backbone model (i.e., MLLMs used in GITA). However, such an approach needs extra designs for these additional modules and needs task-specific adjustments.

In the following table, We also provide an additional comparison between GITA and dedicated GNNs under non-attribute setting on Cora and Citeseer, where non-attribute setting is commonly used to evaluate how well models can understand graph structure. The experimental results show that GITA is much more effective in non-attributed Cora and Citeseer than dedicated GNNs, showcasing a more powerful awareness of their pure graph structure.
| Model | Cora  | Citeseer |
|-------|-------|----------|
| GITA  | 85.24 | 75.07    |
| GCN   | 73.35 | 68.71    |
| SAGE  | 69.19 | 64.69    |
My concerns are addressed and I hope that the final version of the paper will be updated.
Thank you to the authors for the rebuttal. I have carefully reviewed it, as well as the responses to the other reviewers.

Regarding W1, my concern about the limited applications remains. I acknowledge that k-hop sampling is a widely adopted approach and may not lead to performance degradation in some cases. However, for datasets like Cora and CiteSeer, the reported accuracy is much lower compared to pure GNN methods[1] (Table 3). I am uncertain whether this is due to the k-hop sampling or because GITA only provides graph structures without node/edge attributes. If it is the latter, then GITA may not be effective enough for handling graphs with attributes, whether in numerical or text format. 

For W3, the average performance of GraphLLM and GITA is similar. I am conservative about the contribution of GITA over existing methods.

For the new experiments during rebuttal, for table 5-7, the comparision with GNNs is an important baseline and should be displayed in the main text. Besides, for small datasets (Table 5), simple GNN method has very close performance with GITA. For large datasets (Table 6,7), pure GNN method show better performance and significantly better efficiency. I am skeptical about the necessity of using LLM with visual information to address these tasks given simple GNNs have already perform good with great efficiency.


For W2 and Q1-Q3, my questions are well addressed.

[1] cora leaderboad, https://paperswithcode.com/sota/node-classification-on-cora
Thanks for the rebuttal. My concerns are addressed, therefore I maintain my score.
I thank the author for the rebuttal. It addressed most of my questions and I am increasing the score to 6.