GraphVis: Boosting LLMs with Visual Knowledge Graph Integration
Authors: Yihe Deng
Chenchen Ye
Zijie Huang
Mingyu Derek Ma
Yiwen Kou
Wei Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Knowledge Graphs
Multi-modal learning
Primary area: natural_language_processing

Abstract: The rapid evolution of large language models (LLMs) has expanded their capabilities across various data modalities, extending from well-established image data to increasingly popular graph data. Given the limitation of LLMs in hallucinations and inaccuracies in recalling factual knowledge, Knowledge Graph (KG) has emerged as a crucial data modality to support more accurate reasoning by LLMs. However, integrating structured knowledge from KGs into LLMs remains challenging, as most current KG-enhanced LLM methods directly convert the KG into linearized text triples, which is not as expressive as the original structured data. To address this, we introduce GraphVis, which conserves the intricate graph structure through the visual modality to enhance the comprehension of KGs with the aid of Large Vision Language Models (LVLMs). Our approach incorporates a unique curriculum fine-tuning scheme which first instructs LVLMs to recognize basic graphical features from the images, and subsequently incorporates reasoning on QA tasks with the visual graphs. This cross-modal methodology not only markedly enhances performance on standard textual QA  but also shows improved zero-shot VQA performance by utilizing synthetic graph images to augment the data for VQA tasks. We present comprehensive evaluations across commonsense reasoning QA benchmarks, where GraphVis provides an average improvement of 11.1% over its base model and outperforms existing KG-enhanced LLM approaches. Across VQA benchmarks such as ScienceQA that share similar scientific diagram images, GraphVis provides a notable gain of 4.32%.

Graph Boosting LLMs with Visual Knowledge Graph Integration
Yihe Deng" Chenchen Ye Zijie Huang
Mingyu Derek Ma Yiwen Kou
Wei Wang
University of California, Los Angeles
Abstract
The rapid evolution large language models (LLMs) has expanded their capabil- ities across various data modalities, extending from well-established image data to increasingly popular graph data. Given the limitation of LLMs in hallucina- tions and inaccuracies in recalling factual knowledge, Knowledge Graph (KG) has emerged as a crucial data modality to support more accurate reasoning by LLMs. However, integrating structured knowledge from KGs into LLMs remains challenging, as most KG-enhanced LLM methods directly convert the KG into linearized text triples, which is not expressive the original structured data. To address this, we introduce GraphVis, which conserves the intricate graph structure through the visual modality to enhance the comprehension of KGs with the aid of Large Vision Language Models (LVLMss. Our approach incorporates unique curriculum fine-tuning scheme which first instructs LVLMs to recognize basic graphical features from the images, and subsequently incorporates reasoning on QA tasks with the visual graphs. This cross-modal methodology not only markedly enhances performance on standard textual QA but also shows improved zero-shot VQA performance by utilizing synthetic graph images to augment the data for VQA tasks. We present comprchensive evaluations across commonsense reasoning QA benchmarks, where GraphVis provides an average improvement of 11.1% over its base model and outperforms existing KG-enhanced LLM approaches. Across VQA benchmarks such as ScienceQA that share similar scientific diagram images, GraphVis provides a notable gain of 4.32%. Code is made available on GitHub.
MMBench
66.41
POPE-random
80
63.25
75
89.73
70
68.86
Science
QA
87.73
60
$3.73
$6.47
55
PoPElpopular
87.07
50
C5QA
OBQA
Catagortes
POPE-adversarial
Figure 1: Left: Accuracy improvement of GraphVis compared to the base model's performance on commonsense reasoning tasks. Right: Improvement by GraphVis multiple VQA benchmarks over its base LVLM model LLaVA-v1.6 (Liu et al., 2024).
"Conresponding to Yihe Deng syihedeng@cs.uckkae @cs.ucla.edu..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1 Introduction
The rapid evolution of large language models (LLMs) (Zhang et al., 2019; Brown al., 2020; Touvron et al., 2023a; Chung et al., 2024) has unlocked new opportunities for interacting with multimodal data sources. Approaches that enable input from multi-modal data can expands the information that LLMs can take in for various downstream reasoning tasks across domains. The modalities in existing LLM-based models span across images (Zhu et al., 2023; Liu et al., 2023b), videos (Maaz et al., 2023; Li et al., 2023c), and audio (Zhang et al., 2023; Rubenstein et al., 2023). Most recently, researchers have also begun to build an unified architecture to encode diverse modalities jointly (Wu et al., 2023). Such unification holds considerable promise, and poses an interesting question on whether data from one modality could enhance the model performance in another.
Beyond the frequently explored modalities such as vision and audio, knowledge graphs (KGs) are also gaining attention. Given LLMs' limitations such as hallucinations (Li et al., 2023a), inaccuracies in recalling factual knowledge (Yang et al., 2023), and the costly updates of knowledge via training (Ding robust source of structured and easy-to- sas-tto-uddate
et al., 2023), researchers are exploring KGs facts (Pan et al., 2023; Jin et al., 2023; Agrawal et al., 2023; Huang et al., 2023). The use of KGs to enhance language models began with smaller models like BERT (Kenton and Toutanova, 2019; Huang et al., 2022), incorporating KGs into the pre-training objectives (Zhang et al., 2019; Rosset et al., 2020; Wang al., 2021) or integrating them through architectural modifications (Yasunaga et al., 2021; Zhang et al., 2022). However, the recent development of larger and more complex LLMs poses challenges adapting these earlier methods. Current strategies for integrating KGs into LLMs fall into two categories: (1) verbalizing relevant KG triples and directly appending them to prompts as "(node a, edge, node b)" (Guo et al., 2023; Feng et al., 2023; Back et al., 2023) or (2) employing graph neural network (GNN) to generate embeddings for relevant KG subgraphs and projecting these into the LLM's token embedding space (Chai et al., 2023; Tian et al., 2024). Nonetheless, these approaches often yield results that are either weaker than or comparable to methods that fully fine-tune smaller LMs with integrated KG information, revealing an underutilization of the graph structure and rich relational context. Thus, effectively integrating the KG modality into LLMs and enabling them to comprehend graph concepts remains an unsolved challenge.
With the rapid advancement of LLMs across various modalities, a question arises: can multi- modal L.Mss, trained in domains other than KGs, facilitate the understanding of graph structures? Large Vision Language Models (LVLMs) (Liu ct al., 2023b), pre-trained on an extensive corpus of image-text pairs, demonstrate exceptional abilities in processing image inputs. In response this potential, we introduce a novel methodology, GraphVis, that enhances graph comprehension by visualizing subgraphs and leveraging LVLMs for KG-enhanced question answering. This approach involves translating retrieved subgraphs into images, which are then processed by an LVLM to aid in answering questions. Recognizing that LVLMs typically lack proficiency with visual graphs, we design a unique curriculum fine-tuning scheme. Initially, the model is trained to interpret simple graphical features, such as node count, edge count, and node degree, through supe-sueeeeiee learning. It then progresses to handling more complex queries by integrating textual question-answe annweer data with relevant visualized subgraphs, thereby fine-tuning the L.L.M to respond accurately to KG-based questions using these images. Our experiments demonstrate that this approach effectively improves the model's performance on downstream QA tasks, surpassing both current KG-enhanced LLMs and traditional fully fine-tuned KG language models.
While the vision modality sigmificantly enhances the integration of KGs and improves the performance on textual QA tasks, our study extends this exploration to the benefits of KGs and textual QA data for (VQA) tasks. Notably, images resembling graphs
LVLMs in zero-shot visual gresstoon-anweering are prevalent in current VQA tasks (Lu et al., 2022; Yu et al., 2023a; Lu et al., 2024), yet similar training datasets are scarce. Our research demonstrates that the availability of extensive textual QA data and relevant KGs facilitates the generation of large synthetic datasets that feature graph images, effectively addressing this scarcity and supporting the training of LVLMs on such data. Evaluations across multiple VQA benchmarks reveal that our LVLM, fine-tuned with the GraphVis approach, also shows remarkable improvements in VQA performance.
Our contributions are summarized as follows:
* We introduce novel method, GraphVis, that employs visual modality to enhance the understanding of KGs in I.LMs, leveraging graph visualization to bridge the gap between structured KG data and multimodal LLM processing capabilities.

---

A revolving convenient for direction travel, but also serves as socurity mensure at what?
retrieval and visualization
B. library
C. department store
KG (ConceptNet)
E. new york
Answer option's letter the given choices directly.
Jalt
Ground truthss
Graphiviz
Visual Graph Comprehenson
Visual Graph
(2) KGG-Enhanced QA Fine-tuning
Compreheasion Flae-tuning
the vetticns goaph depicted image
IVLM
IVLM
Hov maa contectonn dees the uode revebving
What the and sumber ofandd in the graphe?
Curriculum fine-tuning with visual graphs
Figure 2: Overview of GraphVis. Given an input question and answer pair in the training data, we retrieve and visualize the relevant subgraph. With pre-defined questions on the basic features of the visual graphs such as numbers of nodes and node degree, we first construct data for visual graph comprehension fine-tuning. Subsequently, we incorporate the QA pair with the visual graph for KG-enhanced QA fine-tuning.
We present a unique curriculum fine-tuning scheme tailored for LVLMs that sequentially trains on graph-derived images first to visual graph comprehension and then to apply this understanding more complex QA contexts.
We offer a new perspective on gathering fine-tuning data to enhance LVLMs. Specifically, we propose that pure textual data can be combined with relevant synthetic graph images derived from KGs to improve the LVLM's capability in image comprchension and reasoning.
2 Related Work
KG-Enhanced LLMs. Initial studies on KG-enhanced language models have shown that integrating KGs into the pre-training objectives can enrich the foundational knowledge of language models. This approach has been largely applied to encoder-only language models such as BERT (Kenton and Toutanova, 2019) with training objectives specifically tailored for these models such masked word prediction (Zhang et al., 2019; Shen et al., 2020; Zhang et al., 2020; Rosset et al., 2020; Wang et al., 2021; Li et al., 2022; Kang et al., 2022; Baek et al., 2023). Another line of work also relies the encoder architecture of language models and performs full full-parameter fine-tuning on a KG encoder for the fusion of knowledge (Sun al., 2021: Yasunaga al., 2021; Zhang et al., 2022; Yujie al., 2023). However, the advent of recent decoder-only LLM pre-trained on a significantly larger scale (e.g.. the GPT series (Radford et al., 2019 Brown et al., 2020; OpenAL, 2023), LLaMA series (Touvron et al., 2023a,b) and Mistral (Jiang et al., 2023)), increases the difficulty and cost of adapting these KG-based pre-training and fine-tuning methods current LLMs. Consequently, with KGs as distinct modality, researchers have been exploring various methods for integrating the information into L.Mss. One straightforward and most commonly used approach involves verbalizing relevant knowledge graphs and appending them to the prompts (Guo et al., 2023; Feng et al., 2023; Fatemi et al., 2023; Sun al., 2023; Luo al., 2023). For a notable example, KAPING (Baek et al., 2023) retrieves the top k most relevant knowledge triples to the prompt and appends it in the form of textual triples to the original prompt. Meanwhile, such an approach linearizes the originally structured information and does not maintain a natural language form. Another research direction therefore employs GNNs to generate embeddings for retrieved subgraphs, subsequently projecting these into the LLM's token embedding space soft prompts to preserve structured graph information (Yasunaga et al., 2021; Hu et al., 2022; Zhang et al., 2022; Chai et al., 2023; Tian et al., 2024). Nevertheless, GNN- based approaches require task-specific fine-tuning and may struggle with generalization across new tasks. Multimodal LLMs. Other than incorporating knowledge graphs, popular investigations on multi- modal inputs to LLMs include image (Zhu et al., 2023; Liu et al., 2023b), video (Maaz et al., 2023; Li et al., 2023c), audio (Zhang et al., 2023; Rubenstein et al., 2023) and temporal data (Yu et al., 2023b; Chang et al., 2023). Significantly, advances in pre-trained vision-language models (Radford et al., 0021; Jia et al., 2021; Alayrac et al., 2022), which align the visual and textual embedding spaces on web-scale image-caption data, have facilitated substantial progress in the development of Large
3

---

Vision Language Models (LVLMs) (Liu et al., 2023a,b; Zhu et al., 2023: Chen et al., 2023; Ye et al., 2023; Dai et al., 2023; Gao al., 2023; Bai et al., 2023; Peng et al., 2023). These models, with vision encoders trained on extensive collections of web images, exhibit robust visual reasoning capabilities across a range of tasks (Gao et al., 2015; et al., 2022; Xu et al., 2023; Lu al., 2024). However, the incorporation of image data with graph structures into both pre-training and benchmark datasets remains limited, primarily appearing as scientific diagrams within visual question answering datasets for mathematics and science (Lu al., 2022, 2024). This paper also sheds light on an interesting potential to acquire a large volume of graph images through text-based QA datasets to enhance the capabilities of LVLMs.
Problem Setting and Preliminaries
Notation. We use lower case letters to denote scalars and lower case bold face letters to denote vectors. We denote an input sequence, or prompt, as x = [TI,...,TT]] where z; represents token in the LLM's vocabulary. Then, we use the symbol p(-|x) to represent the conditional probability of LLM's response given the prompt x. Lastly, we denote the sequence of tokens generated before the t-th token as < = .y...... Ut for & 1.
Generative Language Models. Let pe denotes LLM parameterized by O. We consider sequence X = 1 xn.] as the input prompt, for which each z, is a token from the LLM's vocabulary. The LLM then generates the response sequence y [y1. [y/1--... Yma by sampling from the conditional probability distribution po(-)x), where y/ denotes individual token for 1 S S m. The conditional distribution (y(x) can therefore be expressed
Markov process (y() = I Po(y y<t)
Given a supervised fine-tuning dataset, S
the training objective is therefore to
maximize the model's likelihood of generating y given x, resulting in the following loss function:
L( = (x,))) log (y)
(3.1)
Given an LLM, an LVLM additionally contains two more components, including a vision encoder f-(-) and a projection network fz(.). The model processes an additional image input e, which converted into visual tokens within the language token space by the vision encoder and the projection network, producing v = [ui...., = fo fpl).. The conditional probability distribution po (y(y x) thus decomposed as
Po v x) = II polu,
(3.2)
KG-enhanced LLMs. A knowledge graph, denoted as G EV.E), consists of set of vertices V and their connections, or edges, E. Considering an input question ground truth answer y", we define Vx = [vi]iez, C V/ as the vertices mentioned in x, where Ix is the index set of vertices associated with the tokens in the question. The objective of KG-enhanced LLM can be decomposed into two steps: (1) relevant subgraph retrieval and (2) effective knowledge projection to the language embedding space. Subgraph retrieval involves designing a function f that generates a subgraph most relevant to the input prompt and containing the mentioned vertices 1/x and connected via the edges Ex:
.21,.....T, Znn] with its corresponding
f(x, V) = (xx,Ex)= 9x C.
The function f could be pre-defined or trained. In this work, we consider the same approach as previous works (Feng et al., 2020), where& Ex is obtained from all k-hop paths connecting two nodes in Vx. Given a relevant subgraph Gx. the target of effectively leveraging the information to construct a function g that generates informative tokens such that
po(y" "x3) max po (y" |x),
where x = [g(9x), x] is the KG-augmented prompt. In essence, the function finds a way of leveraging the knowledge graph to enhance the language model's capacity for answering questions. The current methods therefore fall into the framework as
Linearize. The linearization process is represent the KG as list of triples: g(xx) [(V1, el u1), (V2, 02, u2)....) where edge e, E &x and v,, are two endpoints of e;-
GNN-hased. The GNN-based methods leverage GNN model for the additional information:
x (GxN

---

4 Method
In this section, we formally introduce GraphVis, a technique that employs LVLMs to enhance the integration of KG information, thereby improving performance in downstream textual QA tasks. Reversely, GraphVis also enhances the performance of LVLMS in visual QA tasks by utilizing extensive data from both textual and KG modalities. The methodology of GraphVis outlined in Algorithm and demonstrated in Figure 2. We further elaborate the details of the method below.
GraphVis consists of two major components: (1) novel integration of the retrieved subgraph for KG-enhanced QA via visualization of the graph, and (2) a progressive fine-tuning approach that starts by understanding graphical features and subsequently leverages them for reasoning. The primary objective of GraphVis is to improve the incorporation of KG information into LVLMs rather than enhancing retrieval techniques. Therefore, we adopt the same subgraph retrieval approach as previous studies (Lin et al., 2019; Feng et al., 2020; Yasunaga et al., 2021), which involves retrieving k-hop paths between entities mentioned in the input prompts from the entire KG. For visualization, we utilize the Graphviz tool (Gansner and North, 2000) to generate visual representations for each retrieved KG subgraph.
Most importantly, GraphVis employs a unique curriculum fine-tuning approach specifically designed for visual graph comprehension. While current LVLMs fine-tuned human-labeled vision- language instruction data, images of complex graph structures are much more scarce compared to the many natural images. The reasoning tasks designed for complex graph images also very limited. Graph Vis highlights the potential to leverage textual data and KG images to improve the LVLM's capability in reasoning with graph images. To address this, we initiate the fine-tuning process with simple, self-constructed questions about the structural and relational information in the graph, paired with automatically derived answers, training the model thoroughly understand visual graphs before progressing to more complex reasoning tasks. The loss objective remains the same as SFT objective (3.1). These questions include,
Node description: name all nodes appeared in the image.
Node degree detection: answer with the degree of a named node in the image.
Highest node degree detection:
answer with name(s) the node(s) that has the highest
degree in the image.
Node number detection: answer with the total number of nodes appeared in the image.
Edge number detection: answer with the total number of edges appeared in the image.
Triple listing: describe the image by listing all triples that appeared in the image.
For each of the question types, we draw prompt from pool of five pre-defined prompts of the task to add variance to the data. After the model fully understands the features of a visual graph, proceed to further fine-tune its ability to reason with the visual graph, enhancing its capability to respond to related queries. The original question from the textual QA training dataset then augmented with the visual subgraph as the following,
<visual subgraph>
The image represents knowledge graph relevant to the question, which may or may not be useful. Question: <original question>
The ground truth answers remain unchanged from the textual QA training data. This KG-enhanced QA fine-tuning subsequently starts from the model weights learned the previous visual graph comprehension fine-tuning phase.
Experiments
In this section, we present experiment results of GraphVis on enhancing commonsense reasoning tasks with retrieved KG subgraphs from ConceptNet, as well improving the zero-shot VQA capability of the LVLM by leveraging the data from the textual and KG modality. Across several benchmark datasets, we demonstrate the effectiveness of GraphVis.
5.1 Experiment Setup
Model and Datasets. experiments, we consider llava-v1. 6 S-mmsttrll77 7b (Liu al., 2023a) as our base VLM model. We consider ConceptNet (Speer et al., 2017), a commonsense knowledge graph, as the KG used in our experiments. There are 799,273 nodes and 2,487,810 edges in total existing in the KG, and there are 42 specific different types of relations, merged into 17 relations (Feng et al., 2020). In both fine-tuning stage and inference stage, we consider retrieving 2- 2-hop subgraphs for the conciseness of the images while preserving important information. We then
5

---

Detail the vertices from the graph depicted in the image. The image depicts the following nodes; store, box, choleer, boutique, jewelry store, jewelry hox, jewlery box, jewelry.
Idemtify the node that the most connections gruph and specify its degree ce.
Inkited x)
One node with the lighhst dogrce "jowelsy" with degree of
juisday
What the degree of node with name "jowlery box"? The degres of the node "jewlery box is L.
Jocatzal
iveatiaa
How many nodes are there in the graph?
jewalay
There are 8 nodes the graph.
How many connections are there in the graph?
here are odges in the graph.
Fnumerate
triples
graph.
The triples in the graph are listed as: (choker, at location, jewelty box), (jewelty, location, box). (boutique. store). (choker, location, boutique), (box, used for, storc), (choker,, is a, jewelry), (choker, at location, jewkety box).
Figure 3: Example of the visual graph comprehension question and answer pairs.
Algorithm 1 GraphVis
hie[N]. LVLM parameterized by 0: Pg. The relevant KG: G = &V.E). Relevent subgraph retrieval method f. Self-supervised graph
Input:
Training data from the textual QA dataset: {x("),
question set P = {p hE[M-]- [E[a]]
Let graphical feature training dataset D, = { and the graph VQA dataset D. = -0-
for i I....N do
Retrieve the relevant KG subgraph G; = f(x(1), V
Plot the KG subgraph to obtain the image for visualized KG v(0)
for =...= M do
Given p(5) and Gj automatically get answer (
Add (v(0) (1)),,, aQ) to Dg- end for
Add (v (11 (()) to D end for
Graph understanding fine-tuning: update 0 argmingce E(vxxy)ED (- (yggoo
KG-enhanced QA fine-tuning: update 0 = argmingee E(v.x,y)D. (-logpg
Output: θ.
consider Commonsense QA (CSQA) (Talmor et al., 2019) and OpenBook QA (OBQA) (Mihaylov et al., 2018) as the commonsense reasoning tasks that can be improved via relevant subgraphs in ConceptNet. For the zero-shot VQA tasks, we consider ScienceQA (Lu et al., 2022), MMBench (Liu et al., 2023c) and POPE (Li et al., 2023b) that share similar images or tasks as our synthetic data from textual QA with visual KG subgraphs. Specifically, ScienceQA focuses scientific question answering and contains scientific diagrams. MMBench is recent multi-modal benchmark that comprehensively evaluates model's capabilities in wide range of tasks and evaluation criteria. POPE evaluates the extent of object hallucinations for LVLMs. formulating binary classification task by prompting the model with questions such as "Is there an <object> in this image?", For VQA benchmarks, we use the evaluation scripts provided by LLaVA (Liu et al., 2023a) to obtain the results for both our base model and after using GraphVis ensure fair comparison. In Figure and 5, we demonstrate the statistics of the synthetic visual knowledge graphs in CSQA.
Baselines. We consider the previous KG-enhanced methods that fine-tune language models on the training data with ConceptNet as one category of the baselines, including the popular Q4- GNV (Yasunaga et al., 2021) and Greasel.M (Zhang et al., 2022). We further include the performance of current LLMs without KG or fine-tuning, including FLAN-75-xxlarg (11B) (Chung et al., 2024), which is the base LLM used for many KG-enhanced methods, and GPT-4. Lastly, we include the reported values of methods on KG-enhanced LLMs including KAPING (Back et al., 2023), KSL (Feng al., 2023) and Graph Neural Promping (GNP) (Tian et al., 2024), which all share the same setting of using ConceptNet for enhancement on commonsense reasoning tasks. In particular, KSL and GNP are fine-tuning approaches and we report their best performances (e.g. for GNP, we consider the results from both fine-tuning GNN and projection network and Low-Rank Adaptation (LoRA) (Hu

---

et al., 2021) fine-tuning on LLM). Lastly, we note that these methods have not open-soureed their codes and models, and therefore we consider our re-implementation of KAPING based on the same VLM as reference.
2000
1150
2328
2000
150
3000
520
20
60
30
330
320
340
38
Figure 4: Distribution of node number in CSQA.
Figure 5: Distribution of edge number in CSQA.
5.2
Main Results
In Table 1, we present the main results GraphVis on KG-enhanced question answering. GraphVis demonstrates significant improvement in accuracy the base model, with increase of 12.3% on CSQA and 9.9% on OBQA. We include the full fine-tuning methods of KG-enhanced LMs as strong baselines, which include well-designed model architectures based small-scale language models to better integrate KG information. Although the current methods proposed for KG-enhanced LLMs are open-sourced the time of this manuscript, we incorporate the reported values of the baselines, including KSL, KAPING, and GNP. We observe that, due to the strong performance of the base LLMs, prompting methods like KAPING can actually harm performance by causing notably longer contexts with information not in natural language form. Conversely, fine-tuning methods like KSL and GNP offer greater improvements, even though mostly under-performn performing or matching the performance of full-parameter fine-tuned LMs that have intrinsic architectural changes to adapt the KG information. Meanwhile, the scale of LLMs is unprecedented, causing difficulty both modifying the architecture or fully fine-tuning all parameters. While GraphVis similarly employs LoRA fine-tuning to only update small amount of parameters similar to KSL and GNP, we observe a much more significant improvement that suggests better incorporation of the information. On CSQA, GraphVis with 7B LLM surpasses the second-best result, KSL with GPT-3.5 (> (>100)), by a substantial margin of 3.2%. On OBQA, GraphVis remains the top-performing method, outperforming fine-tuning methods like GNP with an 1 LLM by 5.7%.
Table I: Performance of GraphVis compared with the original VLM model across benchmarks and VQA tasks. As current baselines on L.MMs
open-sourced yet, include the results directly reported from their papers (Zhang et al., 2022: Feng et al., 2023; Tian et al., 2024). We use FT indicate if method involves fine-tuning. The bold numbers indicate the best results among all methods and underscored numbers represent the second best.
Category
Method
Base Model
FT
CSQA
OBQA
QA-GNN
AristoRoBERTa (355M)
76.1
82.8
LM
GreaseLM
AristoRoBERTa (355M)
78.5
84.8
Base LLM
GPT-3.5 (>100B)
X
72.9
74.8
KSL
GPT-3.5 (>100B)
X
79.6
81.6
Base LLM
LLaMA (7B)
38.0
29.8
LLM
KSL
LLaMA (7B)
47.4
45.8
Base LLM
FLAN-T5-xxlarge (11B)
76.8
XX
KAPING
FLAN-TS-xxlarge (1IB)
60.0
GNP
FLAN-T5-xxlead (1IB)
79.8
Base LVLM
LLaVA-v1 1.-M/-ss...MMstrall (7B)
X
70.5
75.6
LVLM
KAPING
LLaVA-vi 1. (7B)
XX
67.7
71.2
GraphVis
LLaVA-v1.6-Mistr (7B)
62.8(+12.3)
85.5
7

---

5.2.1 Leveraging KG and Textual Data to Enhance LVLM
Furthermore, we investigate the benefit of GraphVis in the reverse direction, by leveraging textual QA dataset and KG to improve the LVLM's zero-shot performance on VQA tasks. We begin with the observation that many prevalent VQA benchmarks, such as ScienceQA Lu et al. (2022), feature images structured as directed graphs. For instance, ScienceQA contains a category of image exists as the food web images with questions to identify the decomposers or the producers in the web, as the example shown in Figure 6a. Similarly, MMBench (Liu et al., 2023c) includes a notable portion of images that comprise charts and diagrams, illustrated in Figure 6b. While current L. are pre-trained and fine-tuned on large corpus of vision-language instruction data, images of graph structures are much more scarce compared to the many natural images, addition to the scarcity of reasoning tasks designed specifically for graphs. The presence of structured graphical images within VQA benchmarks highlights the potential of GraphVis Vis to leverage textual data and KG images to improve the LVLM's capability in reasoning with such type of images.
(a) Example image from ScienceQA (Lu et al., 2022). Question: Which of the following organ- isms is the decomposer in this food web?
(b) Example image from MM-Bench (Liu et al., 2023c). Question: who is at the center of all of this?
Figure 6: Example images from VQA tasks that share resemblance the visualized KG subgraphs.
In Table 2, we present the performance of our base LVLM (1lava-vi (1lava-v1.6-mistraa) 6-misstral77b) and its comparison after applying GraphVis. Notably, GraphVis employs synthetic images with textual QAs and does not require human-curated VQÅ training data, yet it robustly generalizes the visual graph comprehension capabilities to diagrams in current VQA benchmarks. We can observe a remarkable improvement of 4.32% on ScienceQA and 2.66% on MMBench. Furthermore, by leveraging the node description and number detection tasks in our graph comprehension fine-tuning, we explore the impact of GraphVis on object hallucinations. Through evaluations using POPE across its three scenarios (random, popular, and adversarial) we find that GraphVis effectively reduces object hallucinations in the LVLM, enhancing both the accuracy and the Fl score in determining whether an object is present in an image. This results in an average improvement of 1.09%. Additionally, we note that differences exist between our visualized knowledge graphs and the graph images in these VQA benchmarks in terms of visual clarity, graph layout, information density, and domain knowledge. Despite these disparities, the model consistently shows improvements across various distinct benchmarks and demonstrates robust generalization capabilities, transitioning effectively from abstract graph structures to real-world images.
Table 2: VQA performance of GraphVis based on 1lava-v1, 6-mmittra-777 -bb
Model
ScienceQA Img-Acc
MMBench Overall
POPE-ran Acc
POPE-pop
POPE-adv
F1
Acc
Fl
Acc
F1
Base LVLM
68.86
63.75 66.41 (42.66)
88.56
87.65
87.73
86.53
86.47
85.37
GraphVis
73.18
89.73
89.12
88.73
87.89
87.07
86.32
6 Ablation Study
In this section, we conduct further ablation studies to explore the different variants of GraphVis to illustrate the significance of the components within our method design.
Curriculum fine-tuning. GraphVis emphasizes curriculum fine-tuning scheme, initially training the model on fundamental visual graph concepts, such as node number and node degrees. Only after mastering these basic comprehension tasks does the model advance to train on the more complex
8

---

reasoning tasks which require leveraging visual graphs to answer relevant questions. Here, we evaluate the impact of task sequencing in fine-tuning by comparing the standard GraphVis with a variant that jointly fine-tunes across the mixed data, encompassing both image comprehension and reasoning tasks. Additionally, we categorize image comprehension tasks into two distinct groups for more fine- grained curriculum fine-tuning:
OCR tasks, including node description and triple listing.
Graph tasks, including node degree detection, highest node degree detection, and node/edge number detection.
Figure 7 presents the CSQA performance results for each fine-tuning strategy of GraphVis. Although GraphVis generally enhances performance across the different schemes, improvements are notably less significant when tasks are jointly trained. The curriculum-baa -based approach yields an additional gain of 4.51% over joint fine-tuning, However, the benefits of more detailed fine-tuning appear minimal. Initiating fine-tuning with OCR tasks, followed by graph tasks and subsequent QA reasoning. leads to marginal increase of 0. 0.24%. Conversely, reversing the order of these detailed tasks results in a performance decline of 1.56%. These findings indicate that optimal fine-tuning involves separating initial image comprehension stages from subsequent reasoning tasks.
85.0
52:36
82.5
810
CSQA Accuracy 5%6
80.0
78.05
77.5
75.0
72.5
70.0
67.5
65.0
Bas
loint
Curricuuum
Fine-gt sineed (graph fins)
Fine-graing grained (oc7 first)
Figure 7: Comparison between different fine-tuning schemes with GraphVis CSQA.
Combination with prompting. As GraphVis is intrinsically compatible with prompting methods like KAPING, we explore the integration of the two methods, as demonstrated in Table 3. Firstly. due to the strong performance of current L.Mss, incorporating KAPING into the base model results in a performance decline of 2.87%, trend consistent with results reported in the previous study (Tian et al., 2024). Such degradation can be attributed to both the prolonged context and the fact that the original model was not adept at understanding graph structures. Mcanwhile, for GraphVis with joint fine-tuning that understands the graph structure as well as the triplet format through the task of triple listing, improvement of 0.666 is observed. However, for GraphVis with curriculum fine-tuning that has more effectively learned the visual graph, the addition of KAPING prompts appears to be redundant and causes minor degradation of 0.82%
Table 3: Performance of GraphVis based llava-vi. 6 www.---mma-----. with without the prompt- ing from KAPING.
Original
w/ KAPING
Base LVLM
70.52
67. (22.87
GraphVis (Joint)
78.05
78.7L(+0.66)
GraphVis
82.56
(1.74(-0.82)
Performance on graph comprehension task. In Table we further evaluate the LVLM on the graph comprehension tasks we defined, both before and after training on the synthetic tasks. To ensure a fair comparison, we utilized synthetic images from the test data of CSQA to construct test set. The accuracy for each individual task is reported. We implement exact matching in determining answer accuracy, which, while strict, provides insight into performance gains and error sources. We observed that graph comprehension tasks are essentially difficult for the LVLM, as such images and tasks are scarce its pre-training and fine-tuning data. On tasks such as triple listing, it almost cannot fulfill the task. For an output example: "Bused on the image provided, the graph appears to represent a network or a system with nodes (blue circles) and edges (black lines) connecting them. To list all the triples in the graph, I'll describe each triple as a sequence of three nodes in the

---

graph, which are connected by edges. Here are the triples in the graph: 1. (nodel, node2, node3) 2. (node2. node3, node4).." Since these preliminary tasks were considered a warm start for the model to learn grounding its reasoning on graph images, we only fine-tuned on these tasks for one epoch. Nevertheless, we observed a notable gain across all tasks after just one epoch of fine-tuning.
Table 4: Performance of LLaVA-v before and after fine-tuning on each graph comprehension task. N. denotes node and E. denotes Edge. For node description and triple listing, we consider the average accuracy of each test example. We use exact matching to determine the accuracy, which may be a stricter evaluation.
Model
N. description
N. degree
Highest N. degree
N. number
E. number
Triple listing
Original
1.4
15.3 (7.0
3.3 (1.6
16.7
9.7
0.6 8. 8.2
After fine-tuning
12.8(+11.0)
(7.5
(6.2
Qualitative example. In Figure 8, we provide specific example of the model generations for the VQA task ScienceQA. The displayed question fundamentally requires the model to traverse through the food web from a starting point, following the directed arrows, and match the target node names with the provided options. The original model failed to complete this task successfully. However, with GraphVis, the model's ability to handle such image data improved significantly, resulting in correct answer.
Query: Context: Below is food web from tundra ecosystem in Nunavut, territory in Northern Canada. A food web modcls how the matter caten by organisms moves through an coosystem. The arrows in food web represent how matter moves between organ- isms in an ecosystem.
Which of these organisms contains matter that was once part of the lichen?
A. mushroom B. short-tailed weusel
C. brown lemming
D. rough-tegged hawk E. bilberry
Answer with the option's letter from the given choices directly.
DGP AIOL
Base (ALaVA-v1.6 7B): C
GruphVis (LLaVA-v1.6 7B): A
Figure& 8: Example of model output on ScienceQA (VQA task). Note that after fine-tuning the base LVLM with Graph Vis on CSQA with synthetic KG images, the model can successfully traverse the graph to locate the correct answer.
7 Conclusion
In conclusion, we proposed GraphVis, a new approach to integrate structured knowledge from KGs with LLMs through the visual modality. By preserving the intricate graph structure and employing a curriculum fine-tuning scheme, our method not only enhanced LLMs' ability to comprehend and reason over KG data to enhance its response to textual QAs but also significantly improves performance across several VQA benchmarks. GraphVis leveraged the strengths of both textual, visual and KG data, reducing factual inaccuracies and hallucinations typical in LLM outputs. The promising results achieved on multiple benchmarks underscore the potential of GraphVis to set new approach of utilizing data from the KG modality and enhancing a model's performance in the cross-modal fashion.
Linitations and future work. Firstly, we acknowledge the limitation induced by compute resources that our experiments are done on 7B models with LoRA fine-tuning, If compute resource permits, it interesting to scale up the experiments with larger models and full fine-tuning. Another limitation is the size of the retrieved subgraph, for which we considerd a 2-hop subgraph to ensure that the visualization is not too complicated for the vision model to recognize. Extending from our current method, interesting future work includes exploring how different visualizations may influence the effectiveness of GraphVis. Additionally, instead of following the previous retrieval methods, would be valuable to investigate better subgraph retrieval techniques and integrate them into the learning process. Lastly, while we used ConceptNet as an example KG to enhance commonsense reasoning, there are numerous other KGs available. It is crucial to explore the generalizability of GraphVis to adapt to new KGs. Furthermore, it is possible to obtain multiple relevant subgraphs for given question from different KG sources. An open problem remains on how to leverage multiple KG subgraphs for enhanced reasoning in LLMs.
10

---

Acknowledgments
We sincerely thank the anonymous reviewers for their helpful comments. The work is partially supported by DARPA HR0011-24-9-033
NSF 2200274, 2106859, 2312501, NIH U54HG012517, U24DK097771, and Optum AI. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
References
AGRAWAL, G., KUMARAGE, T., ALGHAMI, Z. and LIU, H. (2023). Can knowledge graphs reduce hallucinations in llms?: A survey. arXiv preprint arXiv:211..0914.
ALAYRAC, J.-B., DONAHUE, J., LUC, P., MIECH, A., BARR, I., HASSON, Y., LENC, K., MENSCII A., MILLICAN, K., REYNOLDS, M. ET AL. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35 1273366
BAEK, J., АлI, A. F. and SAFFARI, (2023). Knowledge-auge augmeeet language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04336
BAL J., BAL, S., YANG, S., WANG, S., TAN, S., WANG, P., LIN, J., ZHOU, C. and ZHOU, (2023). Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 122.6660...66....
BROWN, T., MANN, B., RYDER, N., SUBBIAH, M., KAPLAN, J. D., DHARIWAL, P., NEELAKAN- TAN, A., SIYAM, P. SASTRY, G., ASKELL, A. AL. (2020). Language models are few-shot learners. Advances in neural information processing systems 33 1877-1901.
CHAL Z., ZHANG, T., WU, L., HAN, K., HU, X., HUANG, X. and YANG, Y. (2023). Graphilm: Boosting graph reasoning ability of large language model. arXiv preprint arXiv22110.0885.
CHANG, C., PENG, W.-C. and CHEN, T.-F. (2023). Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained 1lms. arXiv preprint arXiv:2308.088 08460.
CHEN, J., ZHU, D.. SHEN, X., LI, X., LIU, Z., ZHANG, P., KRISHNAMOORTHI, R., CHANDRA, V., XIONG, and ELHOSEINY, M. (2023). Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478
CHUNG, H. W., HOU, L., LONGPRE, S., ZOPH, B., TAY, Y., FEDUS, W., LI, Y., WANG, X., DEHGHANI, M., BRAHMA, S. ET AL. (2024). Scaling instruction-finetueed language models. Journal of Machine Learning Research 25 1-53.
DAI, W., LI, J., LI, D., TIONG, A. M. H., ZHAo, WANG, W., LI, B., FUNG, P. and Hor, S. (2023). Instructblip: Towards general-purpost vision-language models with instruction tuning.
DING, N., QIN, Y., YANG, G., WEI, F., YANG, Z., Su, Y., HU, S., CHEN, Y., CHAN, C.-M., CHEN, W. ET AL. (2023). Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 220-235.
FATEMI, B., HALCROW, J. and PERO7ZI, B. (2023). Talk like graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560.
FENG, C., ZHANG, X. and FEI, Z. (2023). Knowledge solver: Teaching 1lms to search for domain knowledge from knowledge graphs. arXiv preprint arX1v:2309.03118.
FENG, Y., CHEN, X., LIN, B. Y., WANG, P., YAN, J. and REN, X. (2020). Scalable multi- hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
GANSNER, E. R. and NORTH, S. C. (2000). An open graph visualization system and its applications to software engineering. Software: practice and experience 30 1203-1233.
GAO, H., MAO, J., ZHOU, J., HUANG, Z., WANG, L. and XU, W. (2015). Are you talking to machine? dataset and methods for multilingual image question. Advances neural information processing systems 28.

---

GAO, P., HAN, J.. ZHANG, R., LIN. Z., GENG, S., ZHOU, A., ZHANG, W., LU, P., HE, C., YUE, X., LI, H. and QIAO, Y. (2023). Llama-adapter v2: Parameter-efficient visual instruction model.
GUO, J., DU, L. and LIU, H. (2023). Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305 15066
HU, E. J., WALLIS, P., ALLEN-ZHU, Z., LI, Y., WANG, S., WANG, L.., CHEN, W. ET AL. (2021). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
HU, Z., XU, Y., YU, W., WANG, S., YANG, Z., ZHU, C., CHANG, K.-W. and SUN, Y. (2022). Empowering language models with knowledge graph reasoning for question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
HUANG, Z., LI, Z., JIANG, H.,, CAO, T., LU, H., YIN, B., SUBBIAN, K., SUN, Y. and WANG, W. (2022). Multilingual knowledge graph completion with self-supervised adaptive graph alignment. In Annual Meeting of the Association for Computational Linguistics (ACL).
HUANG, Z., WANG, D., HUANG, B., ZHANG, C., SHANG, J., LIANG, Y., WANG, Z., LI, X., FALOUTSOS, C., SUN, Y. and WANG, W. (2023). Concept2Box: Joint geometric embeddings for learning two-view knowledge graphs. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada.
JIA, C.. YANG, Y., XIA, Y.. CHEN, Y.-T., PAREKH, Z., PHAM, H., LE, Q.. SUNG, Y.-H., LI, Z. and DUERIG, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning. PMLR.
JIANG, A. Q.. SABLAYROLLES, A., MENSCH, A., BAMFORD, C., CHAPLOT, D. S., CASAS, D. D. L., BRESSAND, F., LENGYEL, G., LAMPLE, G., SAULNIER, L. ET AL. (2023). Mistral 7b. arXiv preprint arXiv:2310.06825.
JIN, B., LIU, G., HAN, C.. JIANG, M., J1, H. and HAN, J. (2023), Large language models on graphs: A comprehensive survey, arXiv preprint arXiv:2312.02783
KANG, M., BAEK, and HWANG, S. J. (2022). Kala: mhowledge-augmente language model adap- tation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
KENTON, J. D. M.-W. C. and TOUTANOVA, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, vol. 1.
LI, J., CHENGG X., ZiıA,, W. X., NIE, J. and WEN, J.-R. (2023a). Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
LI, LI, X., SHIANG, L., SUN, C.-J., LIU, B., J1, Z., JIANG, X. and LIU, Q. (2022). Pre-training language models with deterministic factual knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
LI, Y., DU, Y., ZHOU, K., WANG, J., ZHAO, W. X. and WEN, J.-R. (2023b). Evaluating object hallucination large vision-language models. arXiv preprint M.Xi::2005.1335:
LI, Y., WANG, C. and JIA, J. (2023c). Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:231
LIN, B. Y., CHEN, X., CHEN, J. and REN, X. (2019). Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
LIU, H., LI. C., LI, Y. and LEE, Y. J. (2023a). Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744
LIU, H., LI, C., LI, Y., LI, B., ZHANG, Y., SHEN, S. and LEF, Y. J. (2024). Llava-next: Improved reasoning, ocr, and world knowledge.
12

---

LIU, H., LI, C., WU, Q. and LEE, Y. (2023b). Visual instruction tuning. In NeurlPS.
LIU, Y., DUAN, H., ZHANG, Y., LI, B., ZHANG, S., ZHAO,, W., YUAN, Y., WANG, J., HE, C., LIU, Z. ET AL. (2023c). Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.
LU. P., BANSAL, H., XIA, T. LIU, J., LI, C., HAJIHIRII, H., CHENG, H., CHANG, K.-W., GALLEY, M. and GAo, (2024). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
LU. P.. MISHRA, S., XIA, T., QIU, L... CHANG, K.-w., ZHU, S.-C., TAFJORD, O., CLARK, P. and KALYAN, A. (2022). Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 2507-2521.
LUO, L., LI, Y.-F., HAP, R. and PAN, S. (2023). Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representa- lions.
MAAZ, M., RASHEED, H., KHAN, S. and KHAN, F. S. (2023). Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0555 05424.
MIHAYLOV,, T., CLARK, P., KHOT, and SABHARWAL, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
OPENAI (2023). Gpt-4 technical report.
PAN, S., Luo, L., WANG, Y., CHEN, C., WANG, J. and WU, X. (2023). Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306.06 08302.
PENG, Z., WANG. W., DONG, L., HAO, Y., HUANG, S., MA, S. and WEI, F. (2023). Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824.
RADFORD, A., KIM, J. W., HALLACY, C., RAMESH, A., GOH, G., AGARWAL, S., SASTRY, G.,
ASKELL, A., MISHKIN, P., CLARK, J. ET AL. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR.
RADFORD, A., WU, J., CHILD, R., LUAN, D., AMODEL, D., SUTSKEVER, I. ET AL. (2019).
Language models are unsupervised multitask learners. OpenAl blog 19.
ROSSET, C., XIONG, C.,, PHAN, M., SONG, X., BENNETT, and TIWARY, S. (2020). Knowledge- aware language model pretraining. arXiv preprint arXiv:2007.00655.
RUBENSTEIN, P. K., ASAWAROENGCHAI, C., NGUYEN, D. D., BAPNA, A., BORSOS, Z., QUITRY, F. C., CHEN, P., BADAWY, D. E., HAN, W., KHARITONOV, ET AL.. (2023). Audiopalm: large language model that can speak and listen. arXiv preprint arXiv:2306.1199)
SHEN, T., MAO, Y., HE, P., LONG, G., TRISCHLER, A. and CHEN, W. (2020). Exploiting structured knowledge in text via on Empirical Methods in Natural Language Processing (EMNIP).
representation learning. In Proceedings of the 2020 Conference
SPEER, R., CHIN, J. and HAVASI, C. (2017). Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, vol. 31.
SUN, J., XU, C., TANG, L.., WANG, S., LIN, C., GONG, Y., SHUM, H.-Y. and GUo, J. (2023). Think-on-graphe Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.077 200.0677777
SUN, Y., SHI, Q., QI, L. and ZHANG, Y. (2021). Jointlk: Joint reasoning with language models and knowledge graphs for commonsense question answering. In North American Chapter of the Association for Computational Linguistics.
TALMOR, A., HERZIG, J., LOURIE, N. and BERANT, (2019). Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers).
13

---

TIAN, Y., SONG, H., WANG, Z., WANG, H., HU, Z., WANG, F., CHAWLA, N. V. and Xu, P. (2024). Graph neural prompting with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38.
TOUVRON, H., LAVRIL, T., IZACARD, G., MARTINET, X., LACHAUX, M.-A., LACROIX, T., ROZIÈRE, B., GOYAL, N., HAMBRO, E., AZHAR, F. ET AL. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971.
TOUVRON, H., MARTIN, L., STONE, K., ALBERT, P., ALMAHAIRI, A., BABAEI, Y., BASHLYKOV, N., BATRA, S., BHARGAVA, P., BHOSALE, S. ET AL. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 23X7:22307009288
WANG, X., GAO, T., ZHU, Z., ZHANG, Z., LIU, Z.., LI, J. and TANG, J. (2021). Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics 9 91fi/ess77711111111 176-194.
Wu, S., FEI, H., QU, L., JI, W. and CHUA, T.-S. (2023). Next-gpt: Any-to-any multimodal llm. arXiv preprint (rX1::2309005599
XU, P., SHAO, W., ZHANG, K., GAO, P., LIU, S.. LEI, M., MENG, F., HUANG, S., QIAO, Y. and LUO, P. (2023). Lvim-ehub: comprehensive evaluation benchmark for large vision-language models. arXiv preprint 2396055306
YANG, L., CHEN, H., LI, Z., DING, X. and WU, X. (2023). Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306. 1489.
YASUNAGA, M., REN, H.,, BOSSELUT, A., LIANG, P. and LESKOVEc, J. (2021). Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
YE, Q., XU, H., Xu, G., YE, J., YAN, M., ZHOU, Y., WANG, J., HU, A., SHI, P., SHI, Y. ET AL. (2023). mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.144
YU, W., YANG, Z., LI, L., WANG, J., LIN, K., LIU, Z., WANG, X. and WANG, L. (2023a). Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 02490.
YU, X., CHEN, Z., LING, Y., DONG, S., LIU, 7., and LU, (2023b). Temporal data meets Ilm llm-xxplainable financial time series forecasting. arXiv preprint 102iv:2306.11025.
YUJIE, W., HU, Z., JIYE, L. and RU, L. (2023). Dynamic heterogeneous- reasoning with language models and knowledge representation learning for commonsense question answering. In Association for Computational Linguistics (ACL.).
ZHANG, D.., LI, S., ZHANG, X.,, ZHAN, J., WANG, P., ZHOU, Y. and QIU, X. (2023). Speechgpt: Empowering large language models with intrinsic cross cross-moda modal conversational abilities. In The 2023 Conference on Empirical Methods in Natural Language Processing.
ZHANG, D., YUAN, Z., LIU, Y., ZHUANG, F., CHEN, H. and XIONG, H. (2020). E-bert: A phrase and product knowledge enhanced language model for e-commerce. arXiv preprint 2xi2:2009.02835.2
ZHANG, X., BOSSELUT, A., YASUNAGA, M., REN, H., LIANG, P., MANNING, C. and LESKOVEC, J. (2022). Greaselm: Graph reasoning enhanced language models for question answering. In International Conference on Representation Learning (ICLR).
ZHANG, Z.,, HAN, X., LIU, Z.,, JIANG, X., SUN, M. and LIU, Q. (2019). Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.077 1905.07129.
ZHU, D., CHEN, J., SHEN X., LI, X. and ELHOSSINY, M. (2023). Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprini arXiv:2304. 20.52900.00592 10592
14

---

A Experiment Details
Visual graph comprehension data. We created pool of five prompts for each of the task in visual graph comprehension, where the answers can automatically extracted. For node description, we have
"List all nodes the graph shown in the image."
Provide the names of all nodes displayed in the graph image."
Can you name all the nodes shown in the graph image?"
"Identify all the vertices in the diagram of the graph provided."
"Detail all the vertices from the graph depicted in the image."
For highest node degree detection, have
"Name one of the node with the highest degree in the graph. And what its degree?"
"Identify one of the node that has the most connections in the graph and specify its degree."
"Can you tell me which node (name one) has the highest degree this graph and what that degree is?"
"Provide the name and degree of the node with the most connections in the graph."
"Which node in the graph has the greatest number of connections, and what is that total?"
For node degree detection, we have
"What is the degree of the node with the name "node"?",
"What is the degree of the node labeled "node"?".
"Can you tell the degree of the node named "node"?"
"What is the total number of connections that the node "node" has?"
"How many connections does the node "node" have?"
For node number detection, we have
"How many nodes are there the graph?"
"What is the total number of nodes the graph?"
"Can you tell how many nodes are in the graph?"
"What is the total number of vertices in the graph?"
"How many vertices are there in the graph?"
For edge number detection, we have
"How many edges are there in the graph?"
"What is the total number of edges the graph?"
"Can you tell me how many edges are the graph?"
"What is the total number of connections the graph?"
"How many connections are there in the graph?"
For triple listing, we have
"List all the triples in the graph."
"Provide all the triples in the graph."
"Can you list all the triples in the graph?"
"Detail all the triples in the graph."
"Enumerate all the triples the graph."
Fine-tuning. We train epoch for both part of the fine-tuning process. We present the fine-tuning
hyperparameters of GraphVis in Table 5.
15

---

all conducted NVIDIA RTX A6000 GPU Table 5: Fine-tuning hyperparameters.
lora
128
lora_ alpha
256
lora target
all
Learning rate
le-7
Optimizer
AdamW
Global batch size
gradient_accumuilee acccmmlatiion steps
weight_ decay
0
warmup_
0.03
Ir jr_scheduler_type
cosine
image_aspaa srss__sseee_____e ratio
pad
group wwl_mooddllleettne
True
model max Jength mmm_projetto__ir mm_projector_tyye
2048
2e-5
mip2x_geluu
Evaluation. We use the same evaluation scripts provided by LLaVA (Liu et al., 2023a) for all evaluations performed in this paper. We note that the new evaluation scripts (prompts) used to report the newest results of LLaVA-v1.6 are not released yet, which may cause minor differences in evaluation results of the original model compared to their reported values. Nevertheless, we use the same evaluation scripts throughout the paper to ensure fairness in comparison.
Compute resources. Experiments of this paper clusters. The fine-tuning of LLaVA v1.5 (7B) on the visualized subgraphs takes approximately hours on GPUs. The time span for evaluations on the different benchmarks range from 0.5 hours using 1 GPU, depending on the varying size of the dataset.
Additional Experiment Results Table 6, we include the additional results on ScienceQA as one of the VQA tasks from either doing curriculum fine-tuning or simply joint fine-tuning on the curated synthetic data. As indicated by the results, curriculum learning transfers to these VQA tasks as well. In Figure 9, we investigate the influence of image quality for the synthetic visual graphs used for
Table Performance LLaVA-v1.6 on ScienceQA compared with Graph Vis and Graph Vis (joint fine-tuning).
ScienceQA (%)
Base LVLM
68.86
Graph Vis (Joint)
71.94
Graph Vis
73.18
training. It generally observed in VQA tasks that images with lower resolution can lead to degraded performance, these images are considered as "corrupted" and often leads object hallucinations. For the QA tasks that we considered in our evaluation, we conducted additional experiments using graph images with smaller sizes and consequently lower resolutions (50x50).
CSQA Accuracy
60
50
60
30
Graphvis (small)
GraphVis
Figure 9: Effect of lower resolution graph images.
16

---

B Broader Impact
By leveraging factual information from KGs, GraphVis aims to mitigate inaccuracies in the reasoning process and effectively reduced hallucinations in model outputs. This approach aims for a more accurate and reliable model, contributing positively in social impact by providing a more trustworthy and accountable Al model. The improved accuracy and reliability of GraphVis can enhance user trust in Al applications, especially in critical areas such as healthcare, education, and legal advice.
Meanwhile, there are potential negative societal impacts of enhanced LVLMs capabilities. As GraphVis increases the effectiveness of these models, there is risk of misuse in ways that could harm privacy and fairness. For instance, more advanced LVLMs could be exploited to generate misleading or deceptive content or amplify biases present in the underlying data, leading to unfair outcomes. To address these concerns, it is crucial to ensure transparency in how the models trained and used, incorporating bias detection and mitigation strategies.
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificctin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in abstract and introduction of effectively leveraging KG via the vision modality and reversely improving LVLM's VQA capability are well-supported with our experiment results in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our work are discussed the conclusion section of this paper (Section 7).
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper focuses on methodology and empirical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We thoroughly provide the algorithm and method pipeline in Section 4 and hyperparameters used in our experiments in Appendix A to ensure reproducibility. Further- more, code and model weights will be released and maintained.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Codes and scripts are provided in the supplemental materal.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /tippscc/pput
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Fine-tuning and evaluation details of our experiments are explained Ap- pendix A. The supplemental material also contains the codes for our work.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We used decooddn during evaluation, which makes the process deter- ministic and affected by randomness in evaluation. Meanwhile, the fine-tuning process of large vision language model is computationally expensive, and running the fine-tund fine-tuning process for multiple times prohibitively expensive. We admit the limitation caused by computational cost, and ensure that our experiment results are robust and reproducible. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the information of the required compute resources in Section A.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte:///neuussspeeeeeeee ittpp///errpppppppppiiii//////// csGuidelines
Answer:
[Yes]
Justification: Yes, the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
In Appendix B, we discussed the potential impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
22

---

Justification: Our method does not focus on releasing a specific data or model such as pretrained language models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets used in this work are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not aim to introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work did not conduct any crowdsoureing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The question not applicable as this work does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "FWNL0S54sB", "reviewer": "Reviewer_GUVR", "summary": "The paper introduces GraphVis, a novel approach that enables Large Vision Language Models (LVLMs) to reason about visual knowledge graphs for QA tasks.  Unlike previous methods that either input knowledge graph (KG) triplets directly to LLMs or use graphical neural networks to capture structured representations, GraphVis instead represents the graph visually as images of nodes and edges. This inherently introduces an auxiliary OCR task to parse the image of visual knowledge graph, and then the model needs to take the graphical image as input to answer commonsense questions. The knowledge graphs are derived from ConcetNet and relevant nodes and edges are extracted based on the question and answer choices by using off the shelf parser. The authors demonstrate that finetuning LLaVA 1.6 on textual commonsense questions without paired image but instead with the retrieved visual knowledge graphs, leads to improvements in CSQA, as well as zero-shot performance in VQA tasks such as MMBench and ScienceQA.", "questions": "Questions and suggestions are derived from the weaknesses section. \n1. Disentangle the benefits of finetuning with visual graphs vs on QA data. From the ablation studies, it is not clear if the model improves by comprehending and integrating the visual graphs, or by simply finetuning more on QA data. \n2. Introduce more fair comparison of prior work. Authors should follow the finetuning adaptation of KAPING by train models with KG triplets, instead of visual graphs.\n3. What are the main sources of error for graph comprehension tasks? It would be helpful  to identify what might be the bottleneck for understanding the graphical structure when KGs are presented as image.\n4. Authors should present convincing qualitative results of their GraphVis model, not only on how the dataset is constructed.\n5. What are the results if visual graph comprehension stage is omitted, and directly proceeds to KG-Enhanced QA Fine-tuning?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Originality and Significance:** \n\nWhile knowledge augmented LLMs have been explored in prior work, GraphVis is the first to leverage the multimodal capabilities of LVLMs to explicitly understand and interpret the structured relationships visually represented in knowledge graphs (KGs). Incorporating tables, graph figures, and other structured representation as visual context has been investigated, but not specifically for knowledge graphs. \n\nThe paper demonstrates that training on text only QAs with synthetically generated knowledge graphs improves the zero-shot performance of visual QAs over the base model. This is quite a significant finding, and opens up more interesting applications of aligning text data with synthetic multimodal context for improving multimodal tasks. \n\nAblation studies thoroughly examine different strategies of training stages and the order of prompts for graph reasoning questions, and show that separating initial image comprehension stages from subsequent reasoning tasks leads to the best result.\n\n**Clarity and Quality:**\n\t\nThe paper clearly shows how a visual graph is constructed and interpreted by LVLMs. It is easy for readers to follow their training stages. Example images of VQA tasks involving graphs help the readers to understand the transferability of comprehending synthetic KGs in a multimodal setup.", "weaknesses": "- Missing details and analysis of the synthetic visual graphs. Authors should show the statistics of retrieved subgraphs per question, including average number of nodes, degrees of freedom, and etc. Evaluation of the visual graph comprehension tasks should be included to measure how effectively LVLMs understand the graph structure accurately, and their main sources of error in graph comprehension.  \n- Unfair Comparison of GraphVis to zero shot approaches. KAPING is a zero-shot approach that involves no model training and only augments the knowledge directly in the input of LLM. GraphVis instead finetunes the model to understand the visual graph as context. A more fair comparison would be to either follow KAPING with finetuning by training **LVLMs with knowledge graph triplets**, or evaluate zero-shot performance of LVLMs with visual graph as input but no finetuning.\n- Not enough evidence if the improvement comes from integrating the visual knowledge graph, or finetuning on the QA data.  The authors mostly compare GraphVis to the baseline LLaVA model trained on visual instruction tuning data. Since GraphVis additionally finetunes the base model with QA data, it is no surprise that the model outperforms the base model across the QA tasks. More appropriate baseline candidates are finetuning Mistral-7B LLM or LLaVA LVLMs with the QA data. \n  - The authors should also include all ablation studies for VQA tasks, and not only for CSQA.\n- Missing qualitative results of GraphVis vs baseline to show the benefits of visual graphs for VQA tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate your recognition of the originality and significance of our work, and grateful for the positive feedback on the clarity and quality of our paper. Regarding the raised questions, please find our detailed response below with additional experiments and clarifications to potential misunderstanding. We organized the questions according to the order of the weaknesses section.\n\n---\n\n**Q1**. Could the authors provide more details and analysis of the synthetic visual graphs? What are the performance and the main sources of error for graph comprehension tasks? \n\n**A1**. Thank you for raising these important questions. Please find our detailed response to each question in [global rebuttal](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg) on Global A1 (with Figures 1 and 2 in [attached pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf)) and Global A2 (with Table 1 in attached pdf).\n\nConcisely, the retrieved subgraphs (avg. 17 nodes, 25 edges) contain substantial information, but some can be overly complex. Graph comprehension tasks is challenging to LVLMs due to limited graph data in pre-training and fine-tuning. However, fine-tuning for just one epoch showed notable improvements. \n\n---\n\n**Q2**. More discussions on the baselines. Adding KAPING with finetuning would be more comprehensive.\n\n**A2**. We aimed to provide a comprehensive overview and comparison by including the most recent baselines on KG-enhanced LLMs. While KAPING is a prompting method, we also included fine-tuning methods like KSL and GNP, as well as the larger models they selected (FLAN-T5 11B and GPT-3.5). In Table 1 of our paper, we also highlighted the differences between none fine-tuning and fine-tuning methods.\n\nIn response to your suggestion, we included the additional baseline of fine-tuning LVLMs with KG triplets, to more comprehensively compare with KAPING. Following the original zero-shot setting, we maintained the top 10 retrieved triples and appended them to the question as the training data. The accuracy results of this comparison are presented in Table 2 of our attached pdf. \n\nAs observed, KAPING with fine-tuning (KAPING w/ FT) shows an improvement over the base LVLM and zero-shot KAPING. Meanwhile, the vision-language model benefits more from a visual graph input than a linearized textual input. We also note that the computation of top-k embeddings among all retrieved triples was very time-consuming, while GraphVis does not require such computations. \n\n---\n\n**Q3**. Disentangle the benefits of finetuning with visual graphs vs on QA data. \n\n**A3**. From our experiments in our previous response A2,fine-tuning with the additional visual information results in the best performances, while linearizing the retrieved knowledge subgraph into texts maintains helpful information but falls short in proving the useful structured information. In Table 1 of our attached pdf, we additionally add the baseline of fine-tuning on the QA training data without any additional KG information. \n\nWe will include these two additional baselines in our revision. \n\n---\n\n**Q4**. The authors should also include all ablation studies for VQA tasks. \n\n**A4**. As we do not retrieve KG subgraphs for the VQA tasks, it is not applicable to investigate our second ablation study “comparison with prompting”. However, in Table 3 of our attached pdf, we do include the additional results on ScienceQA as one of the VQA tasks from either doing a curriculum fine-tuning or simply joint fine-tuning on the curated synthetic data. As indicated by the results, curriculum learning transfers to these VQA tasks as well. \n\nRegarding the inapplicability of the second ablation study, we would like to further clarify our setting for VQA. The VQA tasks were done in the zero-shot setting and aimed to show the interesting benefits of fine-tuning LVLMs on the synthetic graph images and leveraging the existing textual QA data. As we highlighted in our paper, while current LVLMs are fine-tuned on human-labeled vision-language instruction data, images of complex graph structures are much more scarce compared to the many natural images, in addition to the scarcity of reasoning tasks designed for graph images. GraphVis highlights the potential to leverage textual data and KG images to improve the LVLM’s capability in reasoning with graph images. Therefore, we do not retrieve KG subgraphs for VQA tasks, but we consider that GraphVis provides a new perspective of data source for LVLMs. Our [Global A3](https://openreview.net/forum?id=haVPmN8UGi&noteId=qU5a2KzdFg) provides a more detailed explanation. \n\n---\n\n**Q5**. Missing qualitative results of GraphVis vs baseline to show the benefits of visual graphs for VQA tasks. \n\n**A5**. Thank you for your suggestion. In Figure 4 of our attached one-page pdf, we provided a specific example in the mode generations for the VQA task ScienceQA. The question fundamentally requires the model to traverse through the food web from a starting point, following the directed arrows. It must identify the potential nodes connected to the starting point in a certain direction and match the node names with the provided options. The original model failed to complete this task successfully. However, with  GraphVis, the model's ability to handle such image data improved significantly, resulting in a correct answer.\n\n---\n\n**Q6**. What are the results if visual graph comprehension stage is omitted, and directly proceeds to KG-Enhanced QA Fine-tuning?\n\n**A6**. We appreciate the suggestion to evaluate the impact of omitting the visual graph comprehension stage. In response, we conducted additional ablation studies to investigate this scenario. For the CSQA dataset, omitting the visual graph comprehension stage resulted in an accuracy of 77.5%, which underperforms the curriculum training result of 82.8%. We will add the comprehensive ablation study in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9b9H3klHx", "reviewer": "Reviewer_2Am7", "summary": "GraphVis introduces a novel method for integrating knowledge graphs (KGs) into large language models (LLMs) by preserving the graph structure through the visual modality. Utilizing Large Vision Language Models (LVLMs) and a curriculum fine-tuning scheme, GraphVis enhances both textual QA and VQA performance, demonstrating significant improvements over existing KG-enhanced LLM methods.", "questions": "As mentioned in Weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The use of visual representations to preserve the intricate structure of KGs is a novel approach, addressing limitations of linearized text triples and improving the expressiveness of structured data integration.\n2. The two-phase curriculum fine-tuning, starting with graphical feature recognition and progressing to reasoning tasks, is a technically-sound strategy.\n3. The paper provides extensive evaluations across commonsense reasoning QA and VQA benchmarks, showcasing substantial performance gains.", "weaknesses": "1. The paper does not discuss how the properties of the generated graph images, such as size and resolution, affect the model's performance. Understanding these factors is crucial for replicating and optimizing the method.\n2. While the paper claims performance improvements, it is important to confirm whether this cross-modal methodology is a novel approach. A comparison with existing methods and a discussion on how GraphVis advances the current state-of-the-art would be beneficial.\n3. The proposed method seems to lack significant technical contributions. The approach primarily leverages existing techniques (e.g., curriculum fine-tuning and visual graph generation) without introducing substantial innovations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your support and suggestions, for which we have included additional experiments accordingly. We hope our explanations below answer your questions and provide more clarity.\n\n---\n\n**Q1**. How the properties of the generated graph images, such as size and resolution, affect the model’s performance?\n\n**A1**. Thank you for your suggestion. It is generally observed in VQA tasks that images with lower resolution can lead to degraded performance, as these images are considered as \"corrupted\" and often leads to object hallucinations. For the QA tasks that we considered in our evaluation, we conducted additional experiments using graph images with smaller sizes and consequently lower resolutions (50x50). The results are summarized in Figure 3 of our [attached one-page pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf).\n\nFrom the figure, we can observe that reducing the size and resolution of the graph images (GraphVis (small)) leads to a decrease in performance compared to the standard GraphVis setup. This indicates that higher resolution graph images are crucial for the model to accurately comprehend and utilize the visual information encoded in the graph images as well.\n\nLastly, we emphasize that image size and resolution can be considered as hyperparameter choices. This does not affect our main contribution, which is demonstrating that graph images are a more effective means of conveying useful graph information compared to verbalization. \n\n---\n\n**Q2**. While the paper claims performance improvements, it is important to confirm whether this cross-modal methodology is a novel approach. \n\n**A2**. Thank you for pointing out the importance of highlighting the novelty of GraphVis. The contribution of GraphVis is two-fold. \n1. **Novel use of visual modality for KG-Enhanced LLMs**: GraphVis is the first to employ visual modality for KG-enhanced LLMs, leveraging graph visualization to bridge the gap between structured KG data and multimodal LLM processing capabilities.\n2. **Utilization of KG and textual data in fine-tuning LVLMs**: GraphVis uniquely proposes the use of KG and textual data to fine-tune LVLMs by leveraging synthetic graph images and textual QA datasets.\n\nWe recognize there may be some misunderstanding regarding our second contribution. To clarify, the VQA tasks considered in our paper were performed in a zero-shot setting. Our primary contribution lies in demonstrating the potential of utilizing vast training data from the text-only domain and generating synthetic images with graph structures to enhance the LVLM’s understanding of images that have underlying graph structures.\n\nTo provide further clarity, the current LVLMs (e.g. LLaVA-v1.6) are pre-trained and fine-tuned on large corpus of vision-language instruction-following data. This corpus includes data curated from various VQA training datasets, human annotations, and GPT-4V generations. Obtaining such training data for LVLMs, however, is considerably expensive as it involves data from different modalities. For instance, generating 6k image descriptions with 1k tokens per output using GPT-4V would cost approximately $200. Therefore, researchers have been exploring ways of generating synthetic data to further improve these LVLMs [1-3].\n\nOur major contribution here is to offer a new perspective on gathering fine-tuning data to enhance LVLMs. Specifically, we propose that pure textual data can be combined with relevant synthetic graph images derived from KGs to improve the LVLM’s capability in image comprehension and reasoning. This approach is particularly beneficial for images with graph structures, which are relatively scarce.\n\nWe will include and highlight the above discussion in our revised manuscript to provide more clarity on our contributions. \n\n---\n\n**Q3**. Further discussion on technical contribution.  \n\n**A3**. Thank you for your interest in the technical contributions of our work. As we addressed in our response to Q2, there may have been some misunderstanding regarding our contributions. Here, we clarify and elaborate on the technical advancements introduced by GraphVis. \n\nOur method is novel in advancing cross-modal improvements, which has not been explored for the three modalities that we consider, through two key mechanisms:\n\nVisual modality for KG-Enhanced LLMs: GraphVis is the first approach to integrate visual representations of KGs within the processing framework of KG-enhanced LLMs.\nSynthetic graph images for fine-tuning LVLMs: By combining textual data with synthetic visual graphs, we enhance the LVLM's ability to process and reason about graph-structured information, leading to improved performance in tasks that involve images with underlying graph structures. This approach is particularly valuable given the scarcity of real-world images with graph structures.\n  \nWe emphasize that efficient and affordable data curation for the fine-tuning of LLMs and LVLMs is one of the most important technical directions for advancing their performance. A significant body of work focuses on synthetic data generation (unimodal [4-6] or multimodal [1-3]) rather than redesigning or modifying the architecture of the large models. Our work contributes to this direction by providing a new method for leveraging data from an unused domain and generating synthetic multimodal data that enhances LVLM capabilities. \n\n---\n\n[1] Aligning modalities in vision large language models via preference fine-tuning.\n\n[2] Enhancing large vision language models with self-training on image comprehension.\n\n[3] Understanding alignment in multimodal LLMs: a comprehensive study.\n\n[4] Self-rewarding language models.\n\n[5] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.\n\n[6] Scaling relationship on learning mathematical reasoning with large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXxVG1BV45", "reviewer": "Reviewer_omgG", "summary": "The paper presents a method to improve large vision language models (LVLMs) by integrating knowledge graphs (KGs) visually. The approach, GraphVis, uses LVLMs to understand KGs through image visualizations, enhancing comprehension and reasoning. It employs a curriculum fine-tuning strategy, starting with simple graph features and moving to complex QA tasks. Experiments show significant performance gains in both textual QA and zero-shot VQA, outperforming existing methods with efficient parameter training. The paper also addresses potential societal impacts and future work.", "questions": "Please refer to weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach, GraphVis, which effectively integrates structured knowledge from knowledge graphs into large visual language models using a visual modality.\n2. GraphVis employs a unique, sequential curriculum fine-tuning scheme that progressively trains the model on basic graph features before moving to more complex reasoning tasks.\n3. The paper demonstrates that GraphVis not only improves textual question-answering performance but also enhances zero-shot visual question-answering capabilities.", "weaknesses": "1. While the paper demonstrates the effectiveness of GraphVis using ConceptNet, it may not be clear how well the approach generalizes to other knowledge graphs with different domains. Further research would be needed to confirm its effectiveness across various KGs.\n2. What is the diversity of question-answer pairs used for visual understanding finetuning? Could training on overly monotonous datasets lead to overfitting, and would this affect downstream tasks?\n3. Even if two graphs have identical structures, they can present different visualization results through changes in the positions of nodes, edges, and other elements. Would this affect the training of the model?\n4. In Table 1, the comparison seems unfair for baseline methods. It could be better if authors provide the zero-shot performance on CSQA and OBQA by training on other available datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support and constructive feedback. Please find our detailed response below for the questions raised in the review. \n\n---\n\n**Q1**. Further research would be needed to confirm its effectiveness across various KGs. \n\n**A1**. Thanks for raising this important aspect. As we have also mentioned in our conclusion paragraph on limitations and future work, it is indeed crucial to explore the generalizability of GraphVis across different KGs. While these investigations are beyond the scope of our current study, we will further emphasize this future research direction and outline our plans for future investigations in our revised manuscript. \n\n---\n\n**Q2**. What is the diversity of question-answer pairs used for visual understanding finetuning? Could training on overly monotonous datasets lead to overfitting, and would this affect downstream tasks? \n\n**A2**. Thank you for pointing out this. The synthetic QA data may indeed result in an overly monotonous dataset, and therefore we employed different questions for each synthetic task to increase the diversity. Moreover, as we discussed in line 197-201, we incorporated real QA data for finetuning, which further mitigates this potential issue. \n\nTo evaluate the impact of question format diversity on model performance, we conducted experiments using only one of the question formats for each synthetic task, as compared to the 5 formats used in our paper. We note that the 1 format is contained in the 5 formats for a fair comparison. The results are summarized in Table 4 of [our attached pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf). \n\n---\n\n**Q3**. Visualization may change even when the structure of the graph is identical. Would this affect the training of the model? \n\n**A3**. Thank you for raising this question. We acknowledge that different visualizations can exist for the same graph structure. In our study, we generated random visualizations of the graphs to capture an average performance across these variations.\n\nTo address the potential effects of different visualizations, we conducted an additional experiment using another set of randomly generated images with different visualization colors and shapes for the CSQA dataset (as shown in Table 5 of our attached pdf). The results of this experiment will be included in our revised manuscript to show the robustness of our result. \n\nWe further recognize that leveraging the diversity in graph images for the same graph structure is an interesting future direction. Utilizing this diversity could further enhance the model's ability to generalize and improve its robustness to different visual representations. We will add this aspect in the discussion on future research.\n\n---\n\n**Q4**. It could be better if authors provide the zero-shot performance on CSQA and OBQA by training on other available datasets.\n\n**A4**. We first clarify that the fine-tuning baselines that we compare with (e.g. GNP) similarly use the training data from CSQA and OBQA. To provide stronger and more comprehensive baselines, we conduct experiments with fine-tuning on the same training data without any additional KG information, and fine-tuning with KAPING prompting. Results are shown in Table 2 of our attached pdf, which continue to demonstrate the effectiveness of our method. We will include these two additional baselines in our Table 1 in our revision. \n\nRegarding the generalizability of our results, we would like to point out that our setting for VQA is indeed zero-shot. With fine-tuning on these synthetic graph images and textual QAs, the LVLM exhibited notable improvement on zero-shot tasks such as ScienceQA and MMBench, which also contains data of graph structures. These results highlight the successful transfer and generalization capabilities of our approach across different datasets and tasks.\n\n---\n\nThank you again for your helpful comments. We hope that our clarifications and additional experiments address the raised concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3tAO3EnyU", "reviewer": "Reviewer_Tf8h", "summary": "- This paper proposes an instruction tuning method with visual knowledge graph to enhance the large vision language models with external knowledge and perform better on QA tasks.", "questions": "I hope authors could consider my comments and give me a response.", "limitations": "There is a limitation section in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is a new idea that organize the external knowledge as an image to enhance the LVLMs.", "weaknesses": "- Only LLaVA-v1.6-Mistral is employed as a LVLM backbone in the experiments. As there are also many other LVLMs, I think more experiments should be conducted on diverse LVLMs to demonstrate the effectiveness of GraphVis.\n- The visual graph understanding ability is not evaluated seperately. Only the final performance of VQA tasks are reported in the experiments. As a two-stage training framework, I think the Visual Graph Comprehension Fine-tuning should be also evaluated with suitable metrics.\n- How can it be proved with some examples that the Visual Graph Comprehension Fine-tuning really works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for your support and helpful feedback. Please find our response below for the questions raised in the review and additional experiments. \n\n---\n\n**Q1**. More experiments should be conducted on diverse LVLMs to demonstrate the effectiveness of GraphVis. \n\n**A1**. Thank you for your suggestion. In our limitations discussion, we highlighted that scaling up experiments with larger models would be an interesting direction if compute resources permit. In response to the raised point, we have added experiments on the CSQA task using LLaVA-v1.5 (Vicuna-7B). This model has a different LLM backbone and pre-training process compared to the one initially used in our study.\n\n| Method | CSQA | \n| :--------: | :--------: | \n| Base LVLM | 68.1  |\n| GraphVis |79.9  |\n\n---\n\n**Q2**. Evaluate the model’s visual graph understanding ability separately.\n\n**A2**. Thank you for your suggestion. In Table 1 of the attached PDF, we further evaluated LVLM on these graph comprehension tasks, both before and after training on the synthetic tasks. To ensure a fair comparison, we utilized synthetic images from the test data of CSQA to construct a test set. The accuracy for each individual task is reported. Due to time constraints, we implemented exact matching in determining answer accuracy, which, while strict, provides insight into performance gains and error sources.\n\nWe observed that graph comprehension tasks are essentially difficult for the LVLM, as such images and tasks are scarce in its pre-training and fine-tuning data. On tasks such as triple listing, it almost cannot fulfill the task. For an output example: \n\n\"Based on the image provided, the graph appears to represent a network or a system with nodes (blue circles) and edges (black lines) connecting them. To list all the triples in the graph, I'll describe each triple as a sequence of three nodes in the graph, which are connected by edges. Here are the triples in the graph: 1. (node1, node2, node3) 2. (node2, node3, node4)...\"\n\nSince these preliminary tasks were considered a warm start for the model to learn grounding its reasoning on graph images, we only fine-tuned on these tasks for one epoch. Nevertheless, we observed a notable gain across all tasks after just one epoch of fine-tuning.  \n\n---\n\n**Q3**. Examples to show the effectiveness of GraphVis.\n\n**A3**. Thank you for your suggestion. In Figure 4 of our [attached one-page pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf), we provided a specific example in the mode generations for the VQA task ScienceQA. The question fundamentally requires the model to traverse through the food web from a starting point, following the directed arrows. It must identify the potential nodes connected to the starting point in a certain direction and match the node names with the provided options. The original model failed to complete this task successfully. However, with  GraphVis, the model's ability to handle such image data improved significantly, resulting in a correct answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their insightful and encouraging feedback. We are grateful for the recognition of the novelty and significance of our work (Reviewer GUVR, 2Am7,omgG,Tf8h), extensive experiments and superior performance (Reviewer GUVR, 2Am7,omgG), clear writing flow (Reviewer GUVR), etc. 

In response to the comments, we have provided additional experiments in the [attached one-page pdf](https://openreview.net/attachment?id=qU5a2KzdFg&name=pdf), including 
1. **Distribution of Retrieved Subgraphs (Figures 1 and 2)**: With an average of 17 nodes and 25 edges, the retrieved 2-hop subgraphs contain substantial information. However, some subgraphs are overly complex, indicating room for improvement in pruning methods. 
2. **Performance of LVLM on Synthetic Graph Comprehension Tasks (Table 1)**: These tasks are inherently challenging for LVLMs due to the scarcity of similar data in pre-training and fine-tuning stages. With fine-tuning on only one epoch, we observed a notable improvement. 
3. **Additional Baselines (Table 2)**: Adding fine-tuned LVLMs with textual QA (Base w/FT) and textual KG triplets (KAPING w/FT) respectively. The visual graphs provided additional information and performance gains for vision-language models. 
4. **Extending Ablation Study to VQAs (Table 3)**: We observed a similar pattern as in our ablation studies on QAs, where curriculum learning outperformed joint fine-tuning. 
5. **Lower Resolution Images (Figure 3)**: By corrupting visual graph inputs (resizing), we observed a slight performance decay, consistent with universal observations that corrupted images lead to worse responses and more hallucinations.
6. **Number of Synthetic Question Formats (Table 4)**: Reduced diversity in synthetic question formats resulted in slight performance decay for GraphVis. 
7. **Robustness to Different Visualizations of the Same Graph Architecture (Table 5)**: GraphVis demonstrated robustness to different visualizations. 

For the most raised questions, please find our detailed response below. 

---

**Global A1: statistics of the retrieved subgraphs**

We appreciate reviewers’ suggestions to include more detailed statistics on the synthetic visual graphs. In Figures 1 and 2 of our attached pdf, we provide the distribution of the number of nodes and edges in the retrieved subgraphs in CSQA. Specifically, we have one retrieved subgraph for each question, and the statistics for the retrieved subgraphs are:
- Average node number: 17.36 
- Average edge number: 25.48 
- Average node max degree: 7.82 

---

**Global A2: evaluation on the synthetic graph comprehension tasks**

In Table 1 of the attached PDF, we further evaluated LVLM on these graph comprehension tasks, both before and after finetuning on the synthetic graphs' tasks. To ensure a fair comparison, we utilized synthetic images from the test data of CSQA to construct a test set. The accuracy for each individual task is reported. Due to time constraints, we report answer accuracy using exact matching , which, while strict, provides insight into performance gains and error sources.

We observed that graph comprehension tasks are essentially difficult for the LVLM, as such graph images and tasks are scarce in its pre-training and fine-tuning data. On tasks such as triple listing, it can hardly answer correctly. For an output example: 

"Based on the image provided, the graph appears to represent a network or a system with nodes (blue circles) and edges (black lines) connecting them. To list all the triples in the graph, I'll describe each triple as a sequence of three nodes in the graph, which are connected by edges. Here are the triples in the graph: 1. (node1, node2, node3) 2. (node2, node3, node4)..."

Since these preliminary tasks were considered a warm start for the model to learn grounding its reasoning on graph images, we only fine-tuned on these tasks for one epoch. Nevertheless, we observed a notable gain ranging from 7.6\% to 11.7\% across all tasks after just one epoch of fine-tuning. 

---

**Global A3: clarification on our contribution in VQA tasks**

The VQA tasks were done in the zero-shot setting and aimed to show the interesting benefits of fine-tuning LVLMs on the synthetic graph images and leveraging the existing textual QA data. As we highlighted in our paper, while current LVLMs are fine-tuned on human-labeled vision-language instruction data, images of complex graph structures are much more scarce compared to the many natural images. Also the reasoning tasks designed for complex graph images are very limited. GraphVis highlights the potential to leverage textual data and KG images to improve the LVLM’s capability in reasoning with graph images. 

To further clarify, existing pre-training and fine-tuning corpus includes data curated from various VQA training datasets, human annotations, and GPT-4V generations. Obtaining such training data for LVLMs, however, is considerably expensive as it involves data from different modalities. For instance, generating 6k image descriptions with 1k tokens per output using GPT-4V would cost approximately $200. Therefore, researchers have been exploring ways of generating synthetic data to further improve the LVLMs [1-3].

Our major contribution here is to offer a new perspective on gathering fine-tuning data to enhance LVLMs. Specifically, we propose that pure textual data can be combined with relevant synthetic graph images derived from KGs to improve the LVLM’s capability in image comprehension and reasoning. This approach is particularly beneficial for images with graph structures, which are relatively scarce.

[1] Aligning modalities in vision large language models via preference fine-tuning.

[2] Enhancing large vision language models with self-training on image comprehension

[3] Understanding Alignment in Multimodal LLMs: A Comprehensive Study

---

We have also addressed the comments in each individual rebuttal.

Meta-review: This paper introduces GraphVis, a novel method that integrates visual knowledge graphs with large vision language models (LVLMs) to enhance their reasoning capabilities on both textual and visual QA tasks. The reviewers raised concerns about the fairness of comparisons, the generalizability to other knowledge graphs, and the impact of visual graph comprehension (since this paper focus on fine-tuning open-source LLMs, comparison with closed-source LLMs are not necessary). 
However, the rebuttal and discussion addressed these concerns well, with additional experiments, ablations, and clarifications provided, demonstrating the robustness and effectiveness of GraphVis. The reviewers unanimously provided positive scores following the discussion. The AC also believes the paper makes a unique and solid contribution to integrating knowledge graph into LLMs, which may inspire further large-scale training of LLMs/VLMs. The authors are encouraged to incorporate the additional results and clarifications and further address the comments in the reviews in camera ready.
Decision: Accept (poster)