GraphVis: Boosting LLMs with Visual Knowledge Graph Integration

Authors: yihe  deng
chenchen  ye
zijie  huang
mingyu derek ma
yiwen  kou
wei  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:23:52

Abstract: The rapid evolution of large language models (LLMs) has expanded their capabilities across various data modalities, extending from well-established image data to increasingly popular graph data. Given the limitation of LLMs in hallucinations and inaccuracies in recalling factual knowledge, Knowledge Graph (KG) has emerged as a crucial data modality to support more accurate reasoning by LLMs. However, integrating structured knowledge from KGs into LLMs remains challenging, as most current KG-enhanced LLM methods directly convert the KG into linearized text triples, which is not as expressive as the original structured data. To address this, we introduce GraphVis, which conserves the intricate graph structure through the visual modality to enhance the comprehension of KGs with the aid of Large Vision Language Models (LVLMs). Our approach incorporates a unique curriculum fine-tuning scheme which first instructs LVLMs to recognize basic graphical features from the images, and subsequently incorporates reasoning on QA tasks with the visual graphs. This cross-modal methodology not only markedly enhances performance on standard textual QA  but also shows improved zero-shot VQA performance by utilizing synthetic graph images to augment the data for VQA tasks. We present comprehensive evaluations across commonsense reasoning QA benchmarks, where GraphVis provides an average improvement of 11.1% over its base model and outperforms existing KG-enhanced LLM approaches. Across VQA benchmarks such as ScienceQA that share similar scientific diagram images, GraphVis provides a notable gain of 4.32%.

Main paper: 
 introduction: 
 The rapid evolution of large language models (LLMs) (Zhang et al., 2019; Brown et al., 2020; Touvron et al., 2023a; Chung et al., 2024) has unlocked new opportunities for interacting with multimodal data sources. Approaches that enable input from multi-modal data can expands the information that LLMs can take in for various downstream reasoning tasks across domains. The modalities in existing LLM-based models span across images (Zhu et al., 2023; Liu et al., 2023b ), videos (Maaz et al., 2023; Li et al., 2023c) , and audio (Zhang et al., 2023; Rubenstein et al., 2023) . Most recently, researchers have also begun to build an unified architecture to encode diverse modalities jointly (Wu et al., 2023) . Such unification holds considerable promise, and poses an interesting question on whether data from one modality could enhance the model performance in another. Beyond the frequently explored modalities such as vision and audio, knowledge graphs (KGs) are also gaining attention. Given LLMs' limitations such as hallucinations (Li et al., 2023a) , inaccuracies in recalling factual knowledge (Yang et al., 2023) , and the costly updates of knowledge via training (Ding et al., 2023) , researchers are exploring KGs as a robust source of structured and easy-to-update facts (Pan et al., 2023; Jin et al., 2023; Agrawal et al., 2023; Huang et al., 2023) . The use of KGs to enhance language models began with smaller models like BERT (Kenton and Toutanova, 2019; Huang et al., 2022) , incorporating KGs into the pre-training objectives (Zhang et al., 2019; Rosset et al., 2020; Wang et al., 2021) or integrating them through architectural modifications (Yasunaga et al., 2021; Zhang et al., 2022) . However, the recent development of larger and more complex LLMs poses challenges in adapting these earlier methods. Current strategies for integrating KGs into LLMs fall into two categories: (1) verbalizing relevant KG triples and directly appending them to prompts as "(node a, edge, node b)" (Guo et al., 2023; Feng et al., 2023; Baek et al., 2023) or (2) employing a graph neural network (GNN) to generate embeddings for relevant KG subgraphs and projecting these into the LLM's token embedding space (Chai et al., 2023; Tian et al., 2024) . Nonetheless, these approaches often yield results that are either weaker than or comparable to methods that fully fine-tune smaller LMs with integrated KG information, revealing an underutilization of the graph structure and rich relational context. Thus, effectively integrating the KG modality into LLMs and enabling them to comprehend graph concepts remains an unsolved challenge. With the rapid advancement of LLMs across various modalities, a question arises: can multimodal LLMs, trained in domains other than KGs, facilitate the understanding of graph structures? Large Vision Language Models (LVLMs) (Liu et al., 2023b) , pre-trained on an extensive corpus of image-text pairs, demonstrate exceptional abilities in processing image inputs. In response to this potential, we introduce a novel methodology, GraphVis, that enhances graph comprehension by visualizing subgraphs and leveraging LVLMs for KG-enhanced question answering. This approach involves translating retrieved subgraphs into images, which are then processed by an LVLM to aid in answering questions. Recognizing that LVLMs typically lack proficiency with visual graphs, we design a unique curriculum fine-tuning scheme. Initially, the model is trained to interpret simple graphical features, such as node count, edge count, and node degree, through self-supervised learning. It then progresses to handling more complex queries by integrating textual question-answer data with relevant visualized subgraphs, thereby fine-tuning the LVLM to respond accurately to KG-based questions using these images. Our experiments demonstrate that this approach effectively improves the model's performance on downstream QA tasks, surpassing both current KG-enhanced LLMs and traditional fully fine-tuned KG language models. While the vision modality significantly enhances the integration of KGs and improves the performance on textual QA tasks, our study extends this exploration to the benefits of KGs and textual QA data for LVLMs in zero-shot visual question-answering (VQA) tasks. Notably, images resembling graphs are prevalent in current VQA tasks (Lu et al., 2022; Yu et al., 2023a; Lu et al., 2024 ), yet similar training datasets are scarce. Our research demonstrates that the availability of extensive textual QA data and relevant KGs facilitates the generation of large synthetic datasets that feature graph images, effectively addressing this scarcity and supporting the training of LVLMs on such data. Evaluations across multiple VQA benchmarks reveal that our LVLM, fine-tuned with the GraphVis approach, also shows remarkable improvements in VQA performance. Our contributions are summarized as follows:• We introduce a novel method, GraphVis, that employs visual modality to enhance the understanding of KGs in LLMs, leveraging graph visualization to bridge the gap between structured KG data and multimodal LLM processing capabilities. • We present a unique curriculum fine-tuning scheme tailored for LVLMs that sequentially trains on graph-derived images first to visual graph comprehension and then to apply this understanding in more complex QA contexts. • We offer a new perspective on gathering fine-tuning data to enhance LVLMs. Specifically, we propose that pure textual data can be combined with relevant synthetic graph images derived from KGs to improve the LVLM's capability in image comprehension and reasoning.2 Related Work KG-Enhanced LLMs. Initial studies on KG-enhanced language models have shown that integrating KGs into the pre-training objectives can enrich the foundational knowledge of language models. This approach has been largely applied to encoder-only language models such as BERT (Kenton and Toutanova, 2019) with training objectives specifically tailored for these models such as masked word prediction (Zhang et al., 2019; Shen et al., 2020; Zhang et al., 2020; Rosset et al., 2020; Wang et al., 2021; Li et al., 2022; Kang et al., 2022; Baek et al., 2023) . Another line of work also relies on the encoder architecture of language models and performs full-parameter fine-tuning on a KG encoder for the fusion of knowledge (Sun et al., 2021; Yasunaga et al., 2021; Zhang et al., 2022; Yujie et al., 2023) . However, the advent of recent decoder-only LLM pre-trained on a significantly larger scale (e.g., the GPT series (Radford et al., 2019; Brown et al., 2020; OpenAI, 2023) , LLaMA series (Touvron et al., 2023a,b) and Mistral (Jiang et al., 2023) ), increases the difficulty and cost of adapting these KG-based pre-training and fine-tuning methods to current LLMs. Consequently, with KGs as a distinct modality, researchers have been exploring various methods for integrating the information into LLMs. One straightforward and most commonly used approach involves verbalizing relevant knowledge graphs and appending them to the prompts (Guo et al., 2023; Feng et al., 2023; Fatemi et al., 2023; Sun et al., 2023; Luo et al., 2023 ). For a notable example, KAPING (Baek et al., 2023) retrieves the top k most relevant knowledge triples to the prompt and appends it in the form of textual triples to the original prompt. Meanwhile, such an approach linearizes the originally structured information and does not maintain a natural language form. Another research direction therefore employs GNNs to generate embeddings for retrieved subgraphs, subsequently projecting these into the LLM's token embedding space as soft prompts to preserve structured graph information (Yasunaga et al., 2021; Hu et al., 2022; Zhang et al., 2022; Chai et al., 2023; Tian et al., 2024) . Nevertheless, GNN-based approaches require task-specific fine-tuning and may struggle with generalization across new tasks. Multimodal LLMs. Other than incorporating knowledge graphs, popular investigations on multimodal inputs to LLMs include image (Zhu et al., 2023; Liu et al., 2023b ), video (Maaz et al., 2023; Li et al., 2023c) , audio (Zhang et al., 2023; Rubenstein et al., 2023) and temporal data (Yu et al., 2023b; Chang et al., 2023) . Significantly, advances in pre-trained vision-language models (Radford et al., 2021; Jia et al., 2021; Alayrac et al., 2022) , which align the visual and textual embedding spaces on web-scale image-caption data, have facilitated substantial progress in the development of Large Vision Language Models (LVLMs) (Liu et al., 2023a,b; Zhu et al., 2023; Chen et al., 2023; Ye et al., 2023; Dai et al., 2023; Gao et al., 2023; Bai et al., 2023; Peng et al., 2023) . These models, with vision encoders trained on extensive collections of web images, exhibit robust visual reasoning capabilities across a range of tasks (Gao et al., 2015; Lu et al., 2022; Xu et al., 2023; Lu et al., 2024) . However, the incorporation of image data with graph structures into both pre-training and benchmark datasets remains limited, primarily appearing as scientific diagrams within visual question answering datasets for mathematics and science (Lu et al., 2022 (Lu et al., , 2024)) . This paper also sheds light on an interesting potential to acquire a large volume of graph images through text-based QA datasets to enhance the capabilities of LVLMs.3 Problem Setting and PreliminariesNotation. We use lower case letters to denote scalars and lower case bold face letters to denote vectors. We denote an input sequence, or prompt, as x = [x 1 , . L(θ) = E (x,y)∼S -log p θ y|x . (3.1)Given an LLM, an LVLM additionally contains two more components, including a vision encoder f v (•) and a projection network f p (•). The model processes an additional image input e, which is converted into visual tokens within the language token space by the vision encoder and the projection network, producingv = [v 1 , . . . , v k ] = f v •f p (e).The conditional probability distribution p θ (y|v, x) is thus decomposed asp θ (y|v, x) = m j=1p θ (y j |v, x, y <j ).(3.2) KG-enhanced LLMs. A knowledge graph, denoted as G = {V, E}, consists of a set of vertices V and their connections, or edges, E. Considering an input question x = [x 1 , . . . , x n ] with its corresponding ground truth answer y * , we define V x = {v i } i∈Ix ⊆ V as the vertices mentioned in x, where I x is the index set of vertices associated with the tokens in the question. The objective of KG-enhanced LLM can be decomposed into two steps: (1) relevant subgraph retrieval and (2) effective knowledge projection to the language embedding space. Subgraph retrieval involves designing a function f that generates a subgraph most relevant to the input prompt and containing the mentioned vertices V x and connected via the edges E x :f (x, V x , G) = {V x , E x } = G x ⊂ G.The function f could be pre-defined or trained. In this work, we consider the same approach as previous works (Feng et al., 2020) , where E x is obtained from all k-hop paths connecting two nodes in V x . Given a relevant subgraph G x , the target of effectively leveraging the information is to construct a function g that generates informative tokens such thatp θ (y * |x g ) = max x p θ (y * |x),where x g = [g(G x ), x] is the KG-augmented prompt. In essence, the function g finds a way of leveraging the knowledge graph to enhance the language model's capacity for answering questions.The current methods therefore fall into the framework as• Linearize. The linearization process is to represent the KG as a list of triples:g(G x ) = [(v 1 , e 1 , u 1 ), (v 2 , e 2 , u 2 ), • • • ]where edge e i ∈ E x and v i , u i are two endpoints of e i .• GNN-based. The GNN-based methods leverage a GNN model for the additional information:x g = g GNN .(G x ), x . 
method: 
 In this section, we formally introduce GraphVis, a technique that employs LVLMs to enhance the integration of KG information, thereby improving performance in downstream textual QA tasks. Reversely, GraphVis also enhances the performance of LVLMs in visual QA tasks by utilizing extensive data from both textual and KG modalities. The methodology of GraphVis is outlined in Algorithm 1 and demonstrated in Figure 2 . We further elaborate the details of the method below. GraphVis consists of two major components: (1) a novel integration of the retrieved subgraph for KG-enhanced QA via visualization of the graph, and (2) a progressive fine-tuning approach that starts by understanding graphical features and subsequently leverages them for reasoning. The primary objective of GraphVis is to improve the incorporation of KG information into LVLMs rather than enhancing retrieval techniques. Therefore, we adopt the same subgraph retrieval approach as previous studies (Lin et al., 2019; Feng et al., 2020; Yasunaga et al., 2021) , which involves retrieving k-hop paths between entities mentioned in the input prompts from the entire KG. For visualization, we utilize the Graphviz tool (Gansner and North, 2000) to generate visual representations for each retrieved KG subgraph. Most importantly, GraphVis employs a unique curriculum fine-tuning approach specifically designed for visual graph comprehension. While current LVLMs are fine-tuned on human-labeled visionlanguage instruction data, images of complex graph structures are much more scarce compared to the many natural images. The reasoning tasks designed for complex graph images are also very limited. GraphVis highlights the potential to leverage textual data and KG images to improve the LVLM's capability in reasoning with graph images. To address this, we initiate the fine-tuning process with simple, self-constructed questions about the structural and relational information in the graph, paired with automatically derived answers, training the model to thoroughly understand visual graphs before progressing to more complex reasoning tasks. The loss objective remains the same as SFT objective (3.1). These questions include,• Node description: name all nodes appeared in the image.• Node degree detection: answer with the degree of a named node in the image.• Highest node degree detection: answer with name(s) of the node(s) that has the highest degree in the image. • Node number detection: answer with the total number of nodes appeared in the image.• Edge number detection: answer with the total number of edges appeared in the image.• Triple listing: describe the image by listing all triples that appeared in the image. For each of the question types, we draw a prompt from a pool of five pre-defined prompts of the task to add variance to the data. After the model fully understands the features of a visual graph, we proceed to further fine-tune its ability to reason with the visual graph, enhancing its capability to respond to related queries. The original question from the textual QA training dataset is then augmented with the visual subgraph as the following, <visual subgraph> The image represents a knowledge graph relevant to the question, which may or may not be useful. Question: <original question>The ground truth answers remain unchanged from the textual QA training data. This KG-enhanced QA fine-tuning subsequently starts from the model weights learned in the previous visual graph comprehension fine-tuning phase. 
experiments: 
 In this section, we present experiment results of GraphVis on enhancing commonsense reasoning tasks with retrieved KG subgraphs from ConceptNet, as well as improving the zero-shot VQA capability of the LVLM by leveraging the data from the textual and KG modality. Across several benchmark datasets, we demonstrate the effectiveness of GraphVis.Model and Datasets. In experiments, we consider llava-v1.6-mistral-7b (Liu et al., 2023a) as our base VLM model. We consider ConceptNet (Speer et al., 2017) , a commonsense knowledge graph, as the KG used in our experiments. There are 799,273 nodes and 2,487,810 edges in total existing in the KG, and there are 42 specific different types of relations, merged into 17 relations (Feng et al., 2020) . In both fine-tuning stage and inference stage, we consider retrieving 2-hop subgraphs for the conciseness of the images while preserving important information. We then Detail all the vertices from the graph depicted in the image. The image depicts the following nodes: store, box, choker, boutique, jewelry store, jewelry box, jewlery box, jewelry.Identify one of the node that has the most connections in the graph and specify its degree. One node with the highest degree is "jewelry" with a degree of 5.What is the degree of the node with the name "jewlery box"? The degree of the node "jewlery box" is 1.How many nodes are there in the graph? There are 8 nodes in the graph. How many connections are there in the graph? There are 12 edges in the graph.Enumerate all the triples in the graph. The triples in the graph are listed as: (choker, at location, jewelry box), (jewelry, at location, box), (boutique, is a, store), (choker, at location, boutique), (box, used for, store), (choker, is a, jewelry), (choker, at location, jewlery box), ... G i = f (x (i) , V x (i) , G).Plot the KG subgraph to obtain the image for visualized KG v (i) . for j = 1, . . . M do Given p (j) and G i , automatically get answer a (j) . Add v (i) , p (j) , a (j) to D g . end for Add v (i) , x (i) , y (i) to D v . end for Graph understanding fine-tuning: update θ = argmin θ∈Θ (v,x,y)∈Dg -log p θ y|v, x . KG-enhanced QA fine-tuning: update θ = argmin θ∈Θ (v,x,y)∈Dv -log p θ y|v, x .Output: θ.consider Commonsense QA (CSQA) (Talmor et al., 2019) and OpenBook QA (OBQA) (Mihaylov et al., 2018) as the commonsense reasoning tasks that can be improved via relevant subgraphs in ConceptNet. For the zero-shot VQA tasks, we consider ScienceQA (Lu et al., 2022) , MMBench (Liu et al., 2023c) and POPE (Li et al., 2023b ) that share similar images or tasks as our synthetic data from textual QA with visual KG subgraphs. Specifically, ScienceQA focuses on scientific question answering and contains scientific diagrams. MMBench is a recent multi-modal benchmark that comprehensively evaluates a model's capabilities in a wide range of tasks and evaluation criteria. POPE evaluates the extent of object hallucinations for LVLMs, formulating a binary classification task by prompting the model with questions such as "Is there an <object> in this image?". For VQA benchmarks, we use the evaluation scripts provided by LLaVA (Liu et al., 2023a) to obtain the results for both our base model and after using GraphVis to ensure a fair comparison. In Figure 4 and 5 , we demonstrate the statistics of the synthetic visual knowledge graphs in CSQA. Baselines. We consider the previous KG-enhanced methods that fine-tune language models on the training data with ConceptNet as one category of the baselines, including the popular QA-GNN (Yasunaga et al., 2021) and GreaseLM (Zhang et al., 2022) . We further include the performance of current LLMs without KG or fine-tuning, including FLAN-T5-xxlarge (11B) (Chung et al., 2024) , which is the base LLM used for many KG-enhanced methods, and GPT-4. Lastly, we include the reported values of methods on KG-enhanced LLMs including KAPING (Baek et al., 2023) , KSL (Feng et al., 2023) and Graph Neural Promping (GNP) (Tian et al., 2024) , which all share the same setting of using ConceptNet for enhancement on commonsense reasoning tasks. In particular, KSL and GNP are fine-tuning approaches and we report their best performances (e.g. for GNP, we consider the results from both fine-tuning GNN and projection network and Low-Rank Adaptation (LoRA) (Hu et al., 2021) fine-tuning on LLM). Lastly, we note that these methods have not open-sourced their codes and models, and therefore we consider our re-implementation of KAPING based on the same VLM as a reference. In Table 1 , we present the main results of GraphVis on KG-enhanced question answering. GraphVis demonstrates a significant improvement in accuracy over the base model, with an increase of 12.3% on CSQA and 9.9% on OBQA. We include the full fine-tuning methods of KG-enhanced LMs as strong baselines, which include well-designed model architectures based on small-scale language models to better integrate KG information. Although the current methods proposed for KG-enhanced LLMs are not open-sourced at the time of this manuscript, we incorporate the reported values of the baselines, including KSL, KAPING, and GNP. We observe that, due to the strong performance of the base LLMs, prompting methods like KAPING can actually harm performance by causing notably longer contexts with information not in natural language form. Conversely, fine-tuning methods like KSL and GNP offer greater improvements, even though mostly under-performing or matching the performance of full-parameter fine-tuned LMs that have intrinsic architectural changes to adapt the KG information. Meanwhile, the scale of LLMs is unprecedented, causing difficulty in both modifying the architecture or fully fine-tuning all parameters. While GraphVis similarly employs LoRA fine-tuning to only update a small amount of parameters similar to KSL and GNP, we observe a much more significant improvement that suggests a better incorporation of the information. On CSQA, GraphVis with a 7B LLM surpasses the second-best result, KSL with GPT-3.5 (>100B), by a substantial margin of 3.2%. On OBQA, GraphVis remains the top-performing method, outperforming fine-tuning methods like GNP with an 11B LLM by 5.7%.Table 1 : Performance of GraphVis compared with the original VLM model across benchmarks and VQA tasks. As current baselines on LLMs are not open-sourced yet, we include the results directly reported from their papers (Zhang et al., 2022; Feng et al., 2023; Tian et al., 2024) 5.2.1 Leveraging KG and Textual Data to Enhance LVLM Furthermore, we investigate the benefit of GraphVis in the reverse direction, by leveraging textual QA dataset and KG to improve the LVLM's zero-shot performance on VQA tasks. We begin with the observation that many prevalent VQA benchmarks, such as ScienceQA Lu et al. ( 2022), feature images structured as directed graphs. For instance, ScienceQA contains a category of image exists as the food web images with questions to identify the decomposers or the producers in the web, as the example shown in Figure 6a . Similarly, MMBench (Liu et al., 2023c) includes a notable portion of images that comprise charts and diagrams, illustrated in Figure 6b . While current LVLMs are pre-trained and fine-tuned on large corpus of vision-language instruction data, images of graph structures are much more scarce compared to the many natural images, in addition to the scarcity of reasoning tasks designed specifically for graphs. The presence of structured graphical images within VQA benchmarks highlights the potential of GraphVis to leverage textual data and KG images to improve the LVLM's capability in reasoning with such type of images. In Table 2 , we present the performance of our base LVLM (llava-v1.6-mistral-7b) and its comparison after applying GraphVis. Notably, GraphVis employs synthetic images with textual QAs and does not require human-curated VQA training data, yet it robustly generalizes the visual graph comprehension capabilities to diagrams in current VQA benchmarks. We can observe a remarkable improvement of 4.32% on ScienceQA and 2.66% on MMBench. Furthermore, by leveraging the node description and number detection tasks in our graph comprehension fine-tuning, we explore the impact of GraphVis on object hallucinations. Through evaluations using POPE across its three scenarios (random, popular, and adversarial) we find that GraphVis effectively reduces object hallucinations in the LVLM, enhancing both the accuracy and the F1 score in determining whether an object is present in an image. This results in an average improvement of 1.09%. Additionally, we note that differences exist between our visualized knowledge graphs and the graph images in these VQA benchmarks in terms of visual clarity, graph layout, information density, and domain knowledge. Despite these disparities, the model consistently shows improvements across various distinct benchmarks and demonstrates robust generalization capabilities, transitioning effectively from abstract graph structures to real-world images. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Codes and scripts are provided in the supplemental materal.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ablation study: 
 In this section, we conduct further ablation studies to explore the different variants of GraphVis to illustrate the significance of the components within our method design.Curriculum fine-tuning. GraphVis emphasizes a curriculum fine-tuning scheme, initially training the model on fundamental visual graph concepts, such as node number and node degrees. Only after mastering these basic comprehension tasks does the model advance to train on the more complex reasoning tasks which require leveraging visual graphs to answer relevant questions. Here, we evaluate the impact of task sequencing in fine-tuning by comparing the standard GraphVis with a variant that jointly fine-tunes across the mixed data, encompassing both image comprehension and reasoning tasks. Additionally, we categorize image comprehension tasks into two distinct groups for more fine-grained curriculum fine-tuning:• OCR tasks, including node description and triple listing.• Graph tasks, including node degree detection, highest node degree detection, and node/edge number detection. Figure 7 presents the CSQA performance results for each fine-tuning strategy of GraphVis. Although GraphVis generally enhances performance across the different schemes, improvements are notably less significant when tasks are jointly trained. The curriculum-based approach yields an additional gain of 4.51% over joint fine-tuning. However, the benefits of more detailed fine-tuning appear minimal. Initiating fine-tuning with OCR tasks, followed by graph tasks and subsequent QA reasoning, leads to a marginal increase of 0.24%. Conversely, reversing the order of these detailed tasks results in a performance decline of 1.56%. These findings indicate that optimal fine-tuning involves separating initial image comprehension stages from subsequent reasoning tasks. Combination with prompting. As GraphVis is intrinsically compatible with prompting methods like KAPING, we explore the integration of the two methods, as demonstrated in Table 3 . Firstly, due to the strong performance of current LLMs, incorporating KAPING into the base model results in a performance decline of 2.87%, a trend consistent with results reported in the previous study (Tian et al., 2024) . Such degradation can be attributed to both the prolonged context and the fact that the original model was not adept at understanding graph structures. Meanwhile, for GraphVis with joint fine-tuning that understands the graph structure as well as the triplet format through the task of triple listing, an improvement of 0.66% is observed. However, for GraphVis with curriculum fine-tuning that has more effectively learned the visual graph, the addition of KAPING prompts appears to be redundant and causes a minor degradation of 0.82%. Performance on graph comprehension task. In Table 4 , we further evaluate the LVLM on the graph comprehension tasks we defined, both before and after training on the synthetic tasks. To ensure a fair comparison, we utilized synthetic images from the test data of CSQA to construct a test set. The accuracy for each individual task is reported. We implement exact matching in determining answer accuracy, which, while strict, provides insight into performance gains and error sources. We observed that graph comprehension tasks are essentially difficult for the LVLM, as such images and tasks are scarce in its pre-training and fine-tuning data. On tasks such as triple listing, it almost cannot fulfill the task. For an output example: "Based on the image provided, the graph appears to represent a network or a system with nodes (blue circles) and edges (black lines) connecting them. To list all the triples in the graph, I'll describe each triple as a sequence of three nodes in the graph, which are connected by edges. Here are the triples in the graph: 1. (node1, node2, node3) 2. (node2, node3, node4)..." Since these preliminary tasks were considered a warm start for the model to learn grounding its reasoning on graph images, we only fine-tuned on these tasks for one epoch. Nevertheless, we observed a notable gain across all tasks after just one epoch of fine-tuning. Qualitative example. In Figure 8 , we provide a specific example of the model generations for the VQA task ScienceQA. The displayed question fundamentally requires the model to traverse through the food web from a starting point, following the directed arrows, and match the target node names with the provided options. The original model failed to complete this task successfully. However, with GraphVis, the model's ability to handle such image data improved significantly, resulting in a correct answer. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Fine-tuning and evaluation details of our experiments are explained in Appendix A. The supplemental material also contains the codes for our work. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In conclusion, we proposed GraphVis, a new approach to integrate structured knowledge from KGs with LLMs through the visual modality. By preserving the intricate graph structure and employing a curriculum fine-tuning scheme, our method not only enhanced LLMs' ability to comprehend and reason over KG data to enhance its response to textual QAs but also significantly improves performance across several VQA benchmarks. GraphVis leveraged the strengths of both textual, visual and KG data, reducing factual inaccuracies and hallucinations typical in LLM outputs. The promising results achieved on multiple benchmarks underscore the potential of GraphVis to set a new approach of utilizing data from the KG modality and enhancing a model's performance in the cross-modal fashion.Limitations and future work. Firstly, we acknowledge the limitation induced by compute resources that our experiments are done on 7B models with LoRA fine-tuning. If compute resource permits, it is interesting to scale up the experiments with larger models and full fine-tuning. Another limitation is the size of the retrieved subgraph, for which we considerd a 2-hop subgraph to ensure that the visualization is not too complicated for the vision model to recognize. Extending from our current method, interesting future work includes exploring how different visualizations may influence the effectiveness of GraphVis. Additionally, instead of following the previous retrieval methods, it would be valuable to investigate better subgraph retrieval techniques and integrate them into the learning process. Lastly, while we used Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We used greedy-decoding during evaluation, which makes the process deterministic and affected by randomness in evaluation. Meanwhile, the fine-tuning process of a large vision language model is computationally expensive, and running the fine-tuning process for multiple times is prohibitively expensive. We admit the limitation caused by computational cost, and ensure that our experiment results are robust and reproducible. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Appendix B, we discussed the potential impacts of this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This work did not conduct any crowdsourcing experiments. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : Left: Accuracy improvement of GraphVis compared to the base model's performance on commonsense reasoning tasks. Right: Improvement by GraphVis on multiple VQA benchmarks over its base LVLM model LLaVA-v1.6 (Liu et al., 2024) . 
 
 Appendix: Visual graph comprehension data. We created a pool of five prompts for each of the task in visual graph comprehension, where the answers can be automatically extracted. For node description, we have• "List all nodes of the graph shown in the image."• "Provide the names of all nodes displayed in the graph image."• "Can you name all the nodes shown in the graph image?"• "Identify all the vertices in the diagram of the graph provided."• "Detail all the vertices from the graph depicted in the image."For highest node degree detection, we have• "Name one of the node with the highest degree in the graph. And what is its degree?"• "Identify one of the node that has the most connections in the graph and specify its degree."• "Can you tell me which node (name one) has the highest degree in this graph and what that degree is?"• "Provide the name and degree of the node with the most connections in the graph."• "Which node in the graph has the greatest number of connections, and what is that total?" For node degree detection, we have• "What is the degree of the node with the name "node"?",• "What is the degree of the node labeled "node"?",• "Can you tell me the degree of the node named "node"?"• "What is the total number of connections that the node "node" has?"• "How many connections does the node "node" have?"For node number detection, we have• "How many nodes are there in the graph?"• "What is the total number of nodes in the graph?"• "Can you tell me how many nodes are in the graph?"• "What is the total number of vertices in the graph?"• "How many vertices are there in the graph?"For edge number detection, we have• "How many edges are there in the graph?"• "What is the total number of edges in the graph?"• "Can you tell me how many edges are in the graph?"• "What is the total number of connections in the graph?"• "How many connections are there in the graph?"For triple listing, we have• "List all the triples in the graph."• "Provide all the triples in the graph."• "Can you list all the triples in the graph?"• "Detail all the triples in the graph."• "Enumerate all the triples in the graph."Fine-tuning. We train 1 epoch for both part of the fine-tuning process. We present the fine-tuning hyperparameters of GraphVis in Table 5 . Evaluation. We use the same evaluation scripts provided by LLaVA (Liu et al., 2023a) for all evaluations performed in this paper. We note that the new evaluation scripts (prompts) used to report the newest results of LLaVA-v1.6 are not released yet, which may cause minor differences in evaluation results of the original model compared to their reported values. Nevertheless, we use the same evaluation scripts throughout the paper to ensure fairness in comparison.Compute resources. Experiments of this paper were all conducted on NVIDIA RTX A6000 GPU clusters. The fine-tuning of LLaVA v1.5 (7B) on the visualized subgraphs takes approximately 3 hours on 4 GPUs. The time span for evaluations on the different benchmarks range from 0.5 to 8 hours using 1 GPU, depending on the varying size of the dataset.In Table 6 , we include the additional results on ScienceQA as one of the VQA tasks from either doing a curriculum fine-tuning or simply joint fine-tuning on the curated synthetic data. As indicated by the results, curriculum learning transfers to these VQA tasks as well.In Figure 9 , we investigate the influence of image quality for the synthetic visual graphs used for training. It is generally observed in VQA tasks that images with lower resolution can lead to degraded performance, as these images are considered as "corrupted" and often leads to object hallucinations.For the QA tasks that we considered in our evaluation, we conducted additional experiments using graph images with smaller sizes and consequently lower resolutions (50x50).Base LVLM GraphVis (small) GraphVis 0 By leveraging factual information from KGs, GraphVis aims to mitigate inaccuracies in the reasoning process and effectively reduced hallucinations in model outputs. This approach aims for a more accurate and reliable model, contributing positively in social impact by providing a more trustworthy and accountable AI model. The improved accuracy and reliability of GraphVis can enhance user trust in AI applications, especially in critical areas such as healthcare, education, and legal advice. Meanwhile, there are potential negative societal impacts of enhanced LVLMs capabilities. As GraphVis increases the effectiveness of these models, there is a risk of misuse in ways that could privacy and fairness. For instance, more advanced LVLMs could be exploited to generate misleading or deceptive content or amplify biases present in the underlying data, leading to unfair outcomes. To address these concerns, it is crucial to ensure transparency in how the models are trained and used, incorporating bias detection and mitigation strategies.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper introduces GraphVis, a novel method that integrates visual knowledge graphs with large vision language models (LVLMs) to enhance their reasoning capabilities on both textual and visual QA tasks. The reviewers raised concerns about the fairness of comparisons, the generalizability to other knowledge graphs, and the impact of visual graph comprehension (since this paper focus on fine-tuning open-source LLMs, comparison with closed-source LLMs are not necessary). 
However, the rebuttal and discussion addressed these concerns well, with additional experiments, ablations, and clarifications provided, demonstrating the robustness and effectiveness of GraphVis. The reviewers unanimously provided positive scores following the discussion. The AC also believes the paper makes a unique and solid contribution to integrating knowledge graph into LLMs, which may inspire further large-scale training of LLMs/VLMs. The authors are encouraged to incorporate the additional results and clarifications and further address the comments in the reviews in camera ready.

Reviews: review: paper_summary: paper_summary: The paper introduces GraphVis, a novel approach that enables Large Vision Language Models (LVLMs) to reason about visual knowledge graphs for QA tasks.  Unlike previous methods that either input knowledge graph (KG) triplets directly to LLMs or use graphical neural networks to capture structured representations, GraphVis instead represents the graph visually as images of nodes and edges. This inherently introduces an auxiliary OCR task to parse the image of visual knowledge graph, and then the model needs to take the graphical image as input to answer commonsense questions. The knowledge graphs are derived from ConcetNet and relevant nodes and edges are extracted based on the question and answer choices by using off the shelf parser. The authors demonstrate that finetuning LLaVA 1.6 on textual commonsense questions without paired image but instead with the retrieved visual knowledge graphs, leads to improvements in CSQA, as well as zero-shot performance in VQA tasks such as MMBench and ScienceQA.
strength_weakness: strength_weakness: - Missing details and analysis of the synthetic visual graphs. Authors should show the statistics of retrieved subgraphs per question, including average number of nodes, degrees of freedom, and etc. Evaluation of the visual graph comprehension tasks should be included to measure how effectively LVLMs understand the graph structure accurately, and their main sources of error in graph comprehension.  
- Unfair Comparison of GraphVis to zero shot approaches. KAPING is a zero-shot approach that involves no model training and only augments the knowledge directly in the input of LLM. GraphVis instead finetunes the model to understand the visual graph as context. A more fair comparison would be to either follow KAPING with finetuning by training **LVLMs with knowledge graph triplets**, or evaluate zero-shot performance of LVLMs with visual graph as input but no finetuning.
- Not enough evidence if the improvement comes from integrating the visual knowledge graph, or finetuning on the QA data.  The authors mostly compare GraphVis to the baseline LLaVA model trained on visual instruction tuning data. Since GraphVis additionally finetunes the base model with QA data, it is no surprise that the model outperforms the base model across the QA tasks. More appropriate baseline candidates are finetuning Mistral-7B LLM or LLaVA LVLMs with the QA data. 
  - The authors should also include all ablation studies for VQA tasks, and not only for CSQA.
- Missing qualitative results of GraphVis vs baseline to show the benefits of visual graphs for VQA tasks.
questions: questions: Questions and suggestions are derived from the weaknesses section. 
1. Disentangle the benefits of finetuning with visual graphs vs on QA data. From the ablation studies, it is not clear if the model improves by comprehending and integrating the visual graphs, or by simply finetuning more on QA data. 
2. Introduce more fair comparison of prior work. Authors should follow the finetuning adaptation of KAPING by train models with KG triplets, instead of visual graphs.
3. What are the main sources of error for graph comprehension tasks? It would be helpful  to identify what might be the bottleneck for understanding the graphical structure when KGs are presented as image.
4. Authors should present convincing qualitative results of their GraphVis model, not only on how the dataset is constructed.
5. What are the results if visual graph comprehension stage is omitted, and directly proceeds to KG-Enhanced QA Fine-tuning?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: GraphVis introduces a novel method for integrating knowledge graphs (KGs) into large language models (LLMs) by preserving the graph structure through the visual modality. Utilizing Large Vision Language Models (LVLMs) and a curriculum fine-tuning scheme, GraphVis enhances both textual QA and VQA performance, demonstrating significant improvements over existing KG-enhanced LLM methods.
strength_weakness: strength_weakness: 1. The paper does not discuss how the properties of the generated graph images, such as size and resolution, affect the model's performance. Understanding these factors is crucial for replicating and optimizing the method.
2. While the paper claims performance improvements, it is important to confirm whether this cross-modal methodology is a novel approach. A comparison with existing methods and a discussion on how GraphVis advances the current state-of-the-art would be beneficial.
3. The proposed method seems to lack significant technical contributions. The approach primarily leverages existing techniques (e.g., curriculum fine-tuning and visual graph generation) without introducing substantial innovations.
questions: questions: As mentioned in Weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a method to improve large vision language models (LVLMs) by integrating knowledge graphs (KGs) visually. The approach, GraphVis, uses LVLMs to understand KGs through image visualizations, enhancing comprehension and reasoning. It employs a curriculum fine-tuning strategy, starting with simple graph features and moving to complex QA tasks. Experiments show significant performance gains in both textual QA and zero-shot VQA, outperforming existing methods with efficient parameter training. The paper also addresses potential societal impacts and future work.
strength_weakness: strength_weakness: 1. While the paper demonstrates the effectiveness of GraphVis using ConceptNet, it may not be clear how well the approach generalizes to other knowledge graphs with different domains. Further research would be needed to confirm its effectiveness across various KGs.
2. What is the diversity of question-answer pairs used for visual understanding finetuning? Could training on overly monotonous datasets lead to overfitting, and would this affect downstream tasks?
3. Even if two graphs have identical structures, they can present different visualization results through changes in the positions of nodes, edges, and other elements. Would this affect the training of the model?
4. In Table 1, the comparison seems unfair for baseline methods. It could be better if authors provide the zero-shot performance on CSQA and OBQA by training on other available datasets.
questions: questions: Please refer to weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: - This paper proposes an instruction tuning method with visual knowledge graph to enhance the large vision language models with external knowledge and perform better on QA tasks.
strength_weakness: strength_weakness: - Only LLaVA-v1.6-Mistral is employed as a LVLM backbone in the experiments. As there are also many other LVLMs, I think more experiments should be conducted on diverse LVLMs to demonstrate the effectiveness of GraphVis.
- The visual graph understanding ability is not evaluated seperately. Only the final performance of VQA tasks are reported in the experiments. As a two-stage training framework, I think the Visual Graph Comprehension Fine-tuning should be also evaluated with suitable metrics.
- How can it be proved with some examples that the Visual Graph Comprehension Fine-tuning really works?
questions: questions: I hope authors could consider my comments and give me a response.
limitations: limitations: There is a limitation section in the paper.

Comments: title: Thank you
Thank you for the response.
Inquiry for discussion
Thank you
Reply to authors
Thank you
Rating Change
comment: Thank you for your reply and positive feedback! We appreciate your suggestions and will incorporate the additional experiment results into our revision. We will expand the results on the synthetic tasks with different model performances for better understanding of the tasks.
Thank you for the response. I have raised the score.
Please make sure to incorporate the results in the table of the attached pdf.

For Table 1 on synthetic graph comprehension task, it would be good to see the performance of proprietary models (GPT4-o, Claude) or other current open-sourced models (InternVL, LLaVA-next) to make the table more complete.
Dear reviewer GUVR,

Thank you again for taking the time to review our paper. We appreciate your detailed feedback and acknowledgement of the originality and significance of our work. 

In response to the feedback received, we have conducted additional experiments and provided detailed clarifications in our rebuttal. While most reviewers have responded positively to our revisions, we hope our detailed responses adequately address your concerns as well. We would appreciate your attention to our rebuttal and any further feedback you may have, as it will give us the opportunity to provide more details before the author-reviewer discussion session ends and help us continue to improve our work. Thank you for your valuable insights!
Thank you for getting back to us and for the positive feedback on our rebuttal!
The responses addressed my concerns. I have changed my rating.
Dear reviewer omgG,

Thank you again for your support and valuable feedback. We appreciate your insights and hope that we have adequately addressed your questions. Specifically, 

1. Exploring various KGs: We agree this is an important area for future work and will include a discussion in our revision.
2. Additional experiments: We've provided new results on:
    - The effect of diverse question formats (Table 4 in the [attached PDF](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf))
    - Different visualizations for synthetic graph images (Table 5 in the [attached PDF](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf))
3. Generalizability: We included additional baselines that were similarly fine-tuned and tested on the same data (Table 2 in the [attached PDF](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf)). Specifically regarding generalization, we clarified that our VQA tasks are performed in a zero-shot setting. Further explanation of our contribution regarding VQA is presented in [Global A3](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg).

We hope these responses and clarifications have been helpful. If you have any further questions about our rebuttal, we're happy to provide additional information or clarification. We sincerely appreciate the time and effort you've invested in reviewing our work!
Dear reviewer GUVR, 

Thank you again for your constructive feedback and questions. We sincerely hope that our responses and clarifications have been helpful in addressing your questions and concerns. Specifically, 

1. We provided additional statistics of the synthetic visual graphs ([Global A1](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg)) as well as the model’s performance on each task ([Global A2](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg)).
2. As suggested, we added more fine-tuning baselines including fine-tuning on textual QA only and fine-tuning with KG triples (Table 2 of [our attached pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf))
3. We extended our ablation study to VQA tasks (Table 3 of [our attached pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf)) 
4. We provided further explanation on the zero-shot setting of our VQA tasks and our corresponding contribution ([Global A3](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg)).
5. We provided a specific example to show how GraphVis improves the LVLM on reasoning with graph images (Figure 4 of [our attached pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf)).

We would like to inquire if there are any questions about our rebuttal, for which we're happy to provide additional information and further clarifications. We truly appreciate the time and effort you’ve invested into reviewing our work!
Thank you for your timely and encouraging feedback on our rebuttal!
Thanks for your response. I have updated my score.
Thank you for your prompt response and positive feedback on our rebuttal!
Thank you for your response. I have change my rating from 5 to 6.