Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering (2024)

Abstract: Knowledge-based visual question answering (KVQA) has been extensively studied to answer visual questions with external knowledge, e.g., knowledge graphs (KGs). While several attempts have been proposed to leverage large language models (LLMs) as an implicit knowledge source, it remains challenging since LLMs may generate hallucinations. Moreover, multiple knowledge sources, e.g., images, KGs and LLMs, cannot be readily aligned for complex scenarios. To tackle these, we present a novel modality-aware integration with LLMs for KVQA (MAIL). It carefully leverages multimodal knowledge for both image understanding and knowledge reasoning. Specifically, (i) we propose a two-stage prompting strategy with LLMs to densely embody the image into a scene graph with detailed visual features; (ii) We construct a coupled concept graph by linking the mentioned entities with external facts. (iii) A tailored pseudo-siamese graph medium fusion is designed for sufficient multimodal fusion. We utilize the shared mentioned entities in two graphs as mediums to bridge a tight intermodal exchange, while maximally preserving insightful intra-modal learning by constraining the fusion within mediums. Extensive experiments show the superiority of MAIL.

heading: 1 Introduction
text: Knowledge-based visual question answering (KVQA) aims to provide appropriate answers for questions about images based on external knowledge (Wang et al., 2017), such as knowledge graphs (KGs) (Marino et al., 2019; Zhang et al., 2022). It has various applications, especially for assisting the visually impaired users (Gurari et al., 2018), yet still, remains a challenging task that requires complex reasoning capability across different data modalities (Yu et al., 2020, 2017).
*Corresponding Author †Equal Contribution.
Recently, several studies have explored using large language models (LLMs) as supplementary knowledge bases and reasoning tools for KVQA (Yang et al., 2022; Gui et al., 2022; Lin et al., 2022); according to how they fuse the knowledge, they can be broadly categorized into direct prompting and modality-agnostic approaches, shown in Figure 1 (a) and (b), respectively. The former directly prompts the question and the corresponding image caption to LLMs for answers (Yang et al., 2022). The latter leverages LLMs to generate candidate answers with supporting evidence and simply combines both question and the external knowledge embedding, e.g., Wikidata (Shengyuan et al., 2024), for reasoning at the final stage (Gui et al., 2022; Lin et al., 2022).
While the above methods have employed LLMs in various ways for KVQA, we argue that they have not fully leveraged the knowledge from LLMs and lack the cross-modal reasoning ability, potentially resulting in sub-optimal performance for complex VQA scenarios. (i) LLMs could incorrectly answer questions or provide unreliable evidence for reasoning. On the one hand, direct prompting to LLMs may struggle to identify the right answer for many complex or domain-specific questions, due to the lack of domain knowledge (Amaro et al., 2023; Chen et al., 2024). On the other hand, LLMs
2417
may be prone to generating hallucination (Gravel et al., 2023) and producing misleading evidence in support of candidate answers. (ii) Integrating multimodal knowledge in a modality-agnostic manner can be sub-optimal. Specifically, existing methods simply concatenate different modal representations, e.g., questions, captions, tags, and external knowledge, for reasoning. This design lacks the necessary cross-modal exchange to enrich the semantics of entities, limiting the final reasoning performance. For example, to correctly answer the question in Figure 1, the model is required to infer the season based on a cross-modal understanding of the inputs, such as the “keep warm” purpose of “coat” and the “spring blooming” feature of “sakura”.
In this work, we study the following research question: How can we effectively leverage the knowledge from LLMs to enhance the comprehensive understanding and reasoning of the images and questions in KVQA? Answering this question is nontrivial due to the following challenges. (i) It is hard to properly incorporate the knowledge from LLMs. LLMs may generate hallucinations when dealing with requests that are not covered in their training corpus. Simply prompting them may generate noisy and irrelevant responses. (ii) Semantic alignment of multiple knowledge sources is challenging. Given image captions, object/region features, external knowledge from KGs, and implicit knowledge from LLMs, appropriately aligning relevant semantic information in different modalities cannot be readily achieved.
To tackle these challenges, we present a novel modality-aware framework to effectively integrate LLMs for KVQA in Figure 1 (c), dubbed MAIL. Specifically, (i) we propose a two-stage prompting strategy to maximally leverage the knowledge from LLMs for image understanding. We initialize a dense caption by prompting a visual LLM, e.g., Visual ChatGPT (Wu et al., 2023) and MiniGPT4 (Zhu et al., 2023). To depict the detailed visual scenes in the caption, we construct a scene graph by defining twelve condensed relations and prompting the LLM to extract spatial and object features accordingly in the form of triples, e.g., (sakura, at_location, tree). (ii) We integrate the external knowledge from KGs to form a coupled concept graph, where the mentioned entities in scene graphs are linked with real-world assertions and facts to facilitate knowledgeable reasoning, such as (coat, used_for, keep warm) and (sakura, typle_of, spring blooming). (iii) A tailored pseudo-siamese
graph medium fusion is designed for effective multimodal graph fusion. Inspired by the success of pseudo-siamese network in measuring the similarity of two correlative inputs (Xia et al., 2021; Gupta et al., 2023), we extend it to graphs to process intra-modal information. It consists of two graph attention networks with the same architecture but different weights. In each sub-encoder, we concentrate on one modality and design a tailored context-aware propagation. This guides our model to attentively prioritize the most valuable entities subject to the particular question. Then we leverage the shared mentioned entities in both coupled graphs as mediums to bridge the cross-modal interaction. The model continuously exchanges their embeddings between two modalities, bringing sufficient complementary knowledge to the other modality respectively. It merely allows inter-modal exchanging by constraining it within the mediums. In general, MAIL effectively enhances a tight intermodal fusion while maximally preserving the insightful intra-modal information for each modality.
Our major contributions are summarized below:
• We formally define a novel learning paradigm, modality-aware integration with LLMs for knowledge-based visual question answering.
• The implicit knowledge in LLMs is carefully leveraged via an effective prompting strategy for coupled scene/concept graph construction.
• We further propose a tailored pseudo-siamese graph medium fusion to integrate multimodal knowledge sources. It balances both intra-modal processing and inter-modal exchange.
• Extensive experiments are conducted on two benchmark datasets. MAIL achieves superior performance over a variety of state-of-the-art baselines with 2 ∼ 4× faster inferential time.

heading: 2 Problem Statement
text: KVQA requires the model to provide answers to the question Q of the corresponding image I based on external knowledge G. In this paper, we propose a novel learning paradigm for leveraging LLMs f(·) for comprehensive knowledge-based VQA. Given an image I, a relevant question Q and external knowledge G, we aim to integrate a visual LLM f(·) and fuse {f(I),Q,G} for prediction. The overall performance is evaluated by the accuracy of returned answers with the ground truths.

heading: 3 Methodology
text: In this section, we introduce the detailed rationale of our proposed framework. An illustration of MAIL is shown in Figure 2. We first carefully leverage the knowledge from LLMs for coupled graph construction. Then, we formulate the pseudo-siamese graph medium fusion (PS-GMF). Through an effective integration of two tailored training objectives, we jointly optimize the model for accurate prediction.

heading: 3.1 Scene Graph Construction
text: Dense Caption Generation We carefully design a hard prompt that requires a visual LLM f(·) to depict the detailed appearance of all the objects in the image and the spatial relations between them. We obtain the generated caption through
D = f(I, P rompt). (1) We consider the identified visual entities in the image as key mentioned entities appearing in the caption, denoted as M = [m1,m2...mn] ∈ D. They significantly dominate the multimodal information of both visual features and external knowledge required to answer the questions. Prompt-enhanced Triple Extraction Given the extracted mentioned entities, we employ LLMs to extract triples. To fully leverage LLMs’ comprehension of image captions and prioritize the important visual features, we pre-define 12 relations R = [r1, r2, ...r12] from two aspects: (i) Spatial features. We constrain the description with at_location, next_to, in_front_of, surrounded_by, covered_by, includes and holds. (ii) Object features are preserved with not only visual outlooks, i.e., has_property, has_color, made_of and wears, but also the intentions of the object if he/she is a human, i.e., intends_to. We design a hard template to prompt LLMs for scene graph construction as,
GS = f(Prompt,D,M,R). (2) The prompt template is used as follows: ‘Describe the image with as many details as possible. Generally, identify the objects and their spatial relations with each other. Specifically, include the visual outlook of different objects, e.g., color, style as well as the appearance of human beings.’ We show the detailed statistics and distributions of all twelve condensed relations in both benchmarks OK-VQA (Marino et al., 2019) and FVQA (Wang et al., 2017) in Table 2 below.

heading: 3.2 Concept Graph Construction
text: In parallel, we incorporate ConceptNet (Speer et al., 2018) for external commonsense knowledge to construct a concept graph. It is one of the largest KGs that provides a myriad of structured triples and contains more than eight million real-world entities. We link each mentioned entity m and the topic entity in the question with ConceptNet, and denote the constructed graph as GC with sufficient textual descriptions, attributes, categories, and properties of M, that are not present in the image so as to facilitate a more knowledgeable reasoning background for various questions. We design the prompt as
‘Given the image caption, based on your comprehensive understanding, construct a high-quality scene graph with as many meaningful details of the mentioned entities as possible in the form of a triple (head entity, relation, tail entity). \n Strictly use the twelve predefined relations from: R, e.g., (woman, in_front_of, car), (car, has_color, blue), only return the triples with no other content. \n Caption: D \n Mentioned Entities: M.’

heading: 3.3 Pseudo-siamese Graph Medium Fusion
text: Typical pseudo-siamese networks (PSNs) could effectively measure the similarity between two in-
puts (Gupta et al., 2023; Xia et al., 2021). We extend it to graphs, which naturally fit the requirement of learning coupled graphs for intra-modal processing, leading to pseudo-siamese graph neural networks (PSGs). However, PSG is incapable of cross-modal fusion. Particularly equipped for PSG to enable inter-modal learning, we further design a graph medium fusion (GMF) algorithm. Pseudo-siamese Graph Neural Network Locating valuable entities in different modalities is essential for KVQA. Here, we instantiate PSG with a novel context-aware message propagation scheme to prioritize the most important knowledge in each modality subject to the question context. Definition. [Pseudo-siamese GNN] We refer to a pseudo-siamese graph neural network that consists of two identical graph neural networks for two relevant inputs. They share the same architecture, i.e., attention mechanism, aggregation function, combination function and activation function, but different weights.
As two sub-networks in PSG share the same architecture, we uniformly provide formulations for the intra-modality processing. For each head entity h, we aggregate all the messages from its neighbor tail entities Nh and t ∈ Nh. Since relations in multimodal graphs contain indispensable information for reasoning various real-world questions, we establish the message passing at the triple level (Zhang et al., 2023a), i.e., (h, r, t) to capture abundant semantics as follows.
mt∈Nh = W (eh, er, et), (3)
where (eh, er, et) is the triple embedding associated with (h, r, t), and W is a learnable matrix for linear transformation. We initialize the entity and relation embedding with a pre-trained language model RoBERTa-large (Liu et al., 2019). While multimodal graphs always contain desperate information with each other, uniformly training each subnetwork in PSG based on the final prediction
lacks awareness of the multimodal characteristics. To this end, we design tailored graph attention networks (Veličković et al., 2017; Dong et al., 2023a) that allocate a context-aware weight â to each message, only prioritizing the multimodal messages in both coupled graphs that are highly related to the question. The context-aware weight âmt for each message mt is correspondingly computed as:
â(h,r,t) = Φ (mt∥c) , (4) where Φ is the adopted activation function, i.e., LeakyReLU. We endow the attention mechanism to be context-aware by concatenating the question context embedding c, expressed as ∥. Notably, we fix the question context embedding c with RoBERTa and only allow it to participate during the attention allocation process.
By normalizing the attention scores obtained previously, we further assign normative values α to each message mt of (h, r, t):
αmt = â(h,r,t)∑
(h,r′,t′)∈Nh â(h,r′,t′) . (5)
To this end, with a weighted sum aggregation operator, we are able to acquire the aggregated representation for entity h in the current layer from its neighbors as eℓNh = ∑ (h,r,t)∈Nh α(h,r,t)) ×m ℓ t , where the layer number in PSG is denoted as ℓ. We summarize the major functions in Table 1. We finalize the overall architecture of PSG for both inputs from scene graph GS and concept graph GC .
e S(ℓ+1) h = J(
∑
(h,r,t)∈Nh
α(h,r,t)) ×mt) + eS(ℓ)h ,
e C(ℓ+1)
ĥ = J(
∑
(ĥ,r̂,t̂)∈N ĥ
α(ĥ,r̂,t̂) ×mt̂) + e C(ℓ) ĥ ,
(6)
where J is a multi-layer perception. The model effectively combines the learned neighbor information eℓNh and itself e ℓ h in current layer. We obtain final representations of all the entities when the layer number ℓ reaches the pre-defined target. Graph Medium Fusion In this subsection, we aim to fill the gaps of the aforementioned PSG on inter-modal learning. However, there is a challenging dilemma centered around striking the right balance between two crucial aspects. On one hand, we want to maximize the inter-modal fusion, where multimodal information could collaborate to yield a more insightful and nuanced understanding of the underlying knowledge subject to answering the question. On the other hand, we recognize the necessity of preserving the integrity of intra-modal processing. Considering excessive inter-modal fusion could introduce noise from each other, we aim to maintain the distinctive characteristics and valuable insights that each modality inherently holds.
Since the mentioned entities M = [m1,m2...mn] are shared by GS and GC , we consider these entities existing in both coupled graphs as mediums that possess similar embeddings since they represent the same real-world object though appearing in different modalities. Motivated by this, we design a novel graph medium fusion algorithm that leverages the medium to bridge two modalities. To get rid of the dilemma, we (i) exchange the representations of mediums em within their respective graphs. This allows the model to delicately introduce cross-modal information with their neighbor entities in the respective graphs, i.e., eNm ; (ii) We strictly impose restrictions on the cross-modal exchange to be within the mediums. This gently brings two modalities closer to each other, while maximally maintaining their individualities. The formulated graph medium fusion process between the coupled graphs is written below.
eSm = { eSm, if ℓ = 0, eCm, otherwise. eCm = { eCm, if ℓ = 0, eSm, otherwise. (7)
Specifically, we froze the medium embeddings in the first layer to ensure they have initially aggregated important 1-hop neighbor information. Afterward, the embeddings for the same medium are automatically exchanged after message-passing in the current layer. This sequential approach ensures a high-quality exchange of information between modalities, i.e., visual features and external knowl-
edge, while initially preserving the local context within each modality before they engage in crossmodal interactions during the following layers.

heading: 3.4 Training Objective
text: Answer-targeted Inferential Loss The primary target of our model is to accurately predict the final answer subject to the particular image and question context. We adopt the binary cross-entropy loss to optimize the inferential performance:
LInference = −log MLP (ea + c)∑ a′∈GC MLP (ea′ + c) , (8)
where a is the correct answer and a′ is one of all the candidate answers from GC . We employ MLP (ea + c) to compute the probability of all the candidate entities in GC and prioritize the highest one as the final answer. Maximum Mean Discrepancy loss Based on the assumption that one medium in two modalities should be similar to the maximum extent, we approximate their similarity by adopting an auxiliary loss, i.e., Maximum Mean Discrepancy (MMD) loss. The basic kernel function is formulated as follows:
K(eSm, eCm) = exp ( −||e
S m − eCm||2 2σ2
) , (9)
where K represents the kernel function and σ is a hyperparameter controlling the width of the kernel (Steinwart and Scovel, 2012). Given a valid kernel function where K(eSm, eCm) = (ϕ(eSm) − ϕ(eCm)), we denote the corresponding feature mapping function as ϕ. The final MMD loss for crossmodal alignment is demonstrated hereunder,
LMedium = || 1 n
∑
m∈M ϕ(eSm)−
1 n
∑
m∈M ϕ(eCm)||2. (10)
We aim to minimize this loss to encourage the learned representations for the same medium from two modalities to be similar in the shared PSG architecture. This effectively guides the process of graph medium fusion by constraining the similarity of mediums in different modalities with each other.

heading: 3.4.1 Joint Optimization
text: The overall framework is jointly optimized according to the aforementioned training objectives. Despite the effectiveness of LMedium, it may introduce inevitable noise by irrespectively forcing the mediums from two modalities to be exactly aligned, which ignores the nature of different modalities. To
alleviate this problem, we introduce a hyperparameter λ to control the contribution from LMedium. To this end, the final training loss is calculated below:
LJoint = LInference + λLMedium. (11)

heading: 4 Experiments
text: In this section, we conduct a variety of experiments to demonstrate the effectiveness of our proposed MAIL. We aim to answer four research questions:
• RQ1 (Main Results): How does MAIL perform compared with different types of SOTA models?
• RQ2 (Hyperparameter analysis): How do hyperparameters influence the performance?
• RQ3 (Ablation studies): Does each component eventually contribute to the overall performance?
• RQ4 (Case study): How effectively does MAIL work in real-world VQA tasks?

heading: 4.1 Experimental Setup
text: Implementation Details We generate dense image captions with MiniGPT-4 (7B) (Zhu et al., 2023), as well as InstructBLIP (Dai et al., 2024) for generalization, and adopt ConceptNet (Speer et al., 2018) for external knowledge. We apply MiniGPT4 and InstructBLIP with one Tesla V100. The entire processing of OK-VQA and the corresponding Microsoft COCO images (Lin et al., 2014) including image-to-text and data cleaning takes about 4 rounds. We adopt ℓ = 3 and λ = 1e−3 after hyperparameter tuning. The generated caption is stored for further multimodal learning. Our codes and processed graphs will be open-sourced and publicly available. For the results of baseline LLMs, since they could occasionally refuse to answer with responses like either ‘As a language model, I am not capable of understanding images’ or ‘Sorry, there is no related information in the provided caption.’, we report the average accuracy over 2 rounds. Datasets Following the previous work (Marino et al., 2021; Yang et al., 2022; Gui et al., 2022; Wu et al., 2022; Lin et al., 2022), we mainly conduct our experiments on OK-VQA (Marino et al., 2019), which is currently the largest and most challenging benchmark, consisting of 14,055 image-question pairs. To further demonstrate the generalization, we also experimentalize on FVQA dataset (Wang et al., 2017), which was the first exploration of KVQA. Baselines We adopt two pipelines of off-the-shelf methods
for performance comparison. (i) Traditional endto-end baselines that design various multimodal learning algorithms for final reasoning over the posed questions, i.e., a direct answering based on questions only (Q Only) (Marino et al., 2019), BAN (Kim et al., 2018), MUTAN (Ben-Younes et al., 2017), ConceptBERT (Gardères et al., 2020), KRISP (Marino et al., 2021), MAVEx (Wu et al., 2022), VLC-BERT (Ravi et al., 2023), RA-VQA (Lin and Byrne, 2022), TriG (Gao et al., 2022), HCNMN (Zhang et al., 2023b) and MCAN (Yu et al., 2019). Moreover, as BAN and MUTAN merely learn the uni-modal visual features, they are augmented with ArticleNet (AN) (Marino et al., 2019) that is trained to retrieve knowledge from Wikipedia for corresponding question-image pair to facilitate the reasoning with external knowledge, denoted as ‘BAN + AN’ and ‘MUTAN + AN’ (Marino et al., 2019). (ii) LLM-enhanced baselines that leverage LLMs, i.e., GPT-3, for direct answer prediction or relevant supporting evidence generation, i.e., PICa (Yang et al., 2022), KAT (Gui et al., 2022), TwO (Si et al., 2023), a simple baseline for KBVQA (Xenos et al., 2023) and REVIVE (Lin et al., 2022).

heading: 4.2 Main Results
text: To answer RQ1, in Table 3 & 4, we summarize the comparisons with all the important baselines. The performance is evaluated by the soft accuracy following previous research (Hu et al., 2023). MAIL outperforms all the traditional baselines regardless of their various knowledge sources and the advantages of leveraging a feature-level image representation. MAIL achieves 12.04% improvements over the best traditional baseline, i.e., MCAN, on OKVQA and 14.7% on FVQA. For LLM-enhanced baselines, it is worth mentioning that they have utilized the generative ability from (Lin et al., 2022; Brown et al., 2020), which makes them especially advantageous in answering subjective questions, for instance, ‘Can people travel on the freeway’ or ‘Is it illegal?’. Despite this, MAIL still outperforms the best LLM-enhanced baseline with 2.28% increases in general, let alone 13.39% over PICa.
Moreover, MAIL is resource-efficient, requiring the smallest number of parameters among all the LLM-enhanced baselines, shown in Table 7. We have used significantly far fewer parameters than any other LLM-enhanced models, i.e., 7.13 B, for answer prediction. As a result, the inferential time of MAIL for one test question is 0.661s (when batch
Method Model Inputs External Knowledge Fusion Strategy Acc. (%)
Q Only Question + Image - - 14.93
Traditional End-to-end Baselines
BAN Question + Image - - 25.17 BAN +AN Question + Image Wikipedia Modality-agnostic 25.61 MUTAN Question + Image - - 26.41 MUTAN +AN Question + Image Wikipedia Modality-agnostic 27.84 ConceptBERT Question + Image ConceptNet Modality-agnostic 33.66
HCNMN Question + Image WordNet Modality-agnostic 36.74 Krisp Question + Image Wikipedia + ConceptNet Modality-agnostic 38.90
MAVEx Question + Image Wikipedia + ConceptNet + Google Images Modality-agnostic 41.37 VLC-BERT Question + Image COMET + ConceptNet Modality-agnostic 43.14
MCAN Question + Image - - 44.65
Large Language Model-enhanced Baselines
PICa-Base Question + Caption + Object Tags Frozen GPT-3 (175B) - 43.30 Pica-Full Question + Caption + Object Tags Frozen GPT-3 (175B) - 48.00
KAT (Single) Question + Caption + Object Tags Frozen GPT-3 (175B) + Wikidata Modality-agnostic 53.09 KAT (Ensemble) Question + Caption + Object Tags Frozen GPT-3 (175B) + Wikidata Modality-agnostic 54.41
REVIVE Question + Caption + Region Tags Frozen GPT-3 (175B) + Wikidata Modality-agnostic 53.83 MAIL (ours) Question + Image Frozen MiniGPT-4 (7B)∗ + ConceptNet Modality-aware 56.69
Table 3: The overall performance comparison on benchmark dataset OK-VQA. We also elaborate on the detailed comparison with a variety of baselines on the knowledge sources that support their inference, i.e., model inputs, external knowledge, as well as how they fuse multiple modalities.
∗ We merely leverage it for caption and scene graph construction, with no extra information that is not present in the images.
ACC.% ℓ = 2 ℓ = 3 ℓ = 4 ℓ = 5 ℓ = 6
size = 1). Generally, existing LLM-enhanced baselines commonly require 2∼4 times of inferential time than MAIL.

heading: 4.3 Hyperparameter Analysis
text: Search of Graph layers The main architecture of PS-GMF naturally comprises the discussion of the impacts from graph layers ℓ. We empirically hypothesize that augmenting the depth of the ℓ could facilitate both a deeper understanding of single modalities (i.e., PSG) and a more profound exchange of information between modalities (i.e., GMF). However, it remains unclear about when to reach the plateau. Simply adding
ACC.% λ = 0 1e− 5 1e− 4 1e− 3 1e− 2 1e− 1 MAIL 53.34 54.18 55.31 56.69 54.30 55.82
more layers may over-fuse two modalities and lose the ability of intra-modal processing, while reducing layers may lead to an adverse situation with inadequate inter-modal fusion. To this end, we vary the layer number and show the performance changes in Table 5. The final prediction performance of MAIL is reported when ℓ = 3. Investigation on hyperparameter λ
While an excessively strict alignment of mediums may homogenize the intra-modal information, we aim to find a suitable λ that constrains the impacts of LMedium. This could significantly encourage harmonious inter-modal fusion from multiple modalities while retaining the richness and specificity inherent to each modality. The experimentation process involves a systematic adjustment of λ across a range of values, specifically within the interval [0, 1e− 5, 1e− 4, 1e− 3, 1e− 2, 1e− 1]. We showcase the results in Table 6. Upon careful examination of the performance trends, we employ λ = 1e− 3 for a balanced trade-off.

heading: 4.4 Ablation Studies
text: Empirical comparison with LLMs In this ablation study, we further demonstrate our tailored multimodal learning module PS-GMF, and delineate the specific contributions by comparing it against frozen LLMs. Specifically, we adopt both
pure LLMs, i.e., Llama (Touvron et al., 2023a) and Llama2 (Touvron et al., 2023b), as well as visual LLMs with Visual ChatGPT (Wu et al., 2023) and MiniGPT-4 (Zhu et al., 2023). We exclusively constrain the inputs in a zero-shot setting with only dense captions and questions for LLMs, while raw images and questions for frozen visual LLMs. The results are summarized in Table 9. MAIL outperforms the best LLM GPT-4 with 2.36% improvements, attributed to the effective graph medium fusion that integrates external knowledge. MAIL also significantly outperforms Visual ChatGPT and MiniGPT-4 with 17.99% and 5.43% higher accuracy. The results shed light on the cross-modal reasoning ability of MAIL. Reasoning with PSG Only In this subsection, we explore the importance of inter-modality interaction by removing the graph medium fusion and only relying on PSG for inference. We list the performance of ‘PSG w/o (GMF)’ in Table 8. The complete multimodal reasoning with PS-GMF outperforms the version with only intra-modal learning with 1.16% improvements. Under this PSG-only setting, we seek to grasp insights into the necessity of graph medium fusion for fostering effective inter-modality interaction. Understanding the performance impact of omitting this fusion mechanism supports the value of shared entities and medium exchange in bridging the cross-modal interaction and facilitates our proposed modality-aware integration with LLMs. Ablation with InstructBLIP To investigate the generalization ability of our proposed MAIL, we adopt InstructBLIP (Dai et al., 2024) as an alternative LLM to take the place of
MiniGPT-4. We reconstruct the scene graph and concept graph based on the newly generated image captions. The main results on OK-VQA are reported in Appendix Table 10.

heading: 4.5 Case Studies
text: In this section, we answer RQ4 with six real-world examples from OK-VQA in Figure 3 to shed light on our effectiveness. Single-hop questions can be directly inferred with easily accessible information from either the visual content or external knowledge sources, while multi-hop questions pose more challenges for accurately locating answers several hops away from mentioned entities.
These cases show the adeptness of MAIL in handling a spectrum of questions, requiring both straightforward inferences from explicit information and complex multi-hop reasoning ability by integrating implicit knowledge sources. For example, Figure 3 (a) can be answered based on the visual information captured by the scene graph without external knowledge, while the answer of (e) needs to be artfully inferred from two different angles, i.e., both the blossom season of sakura and the warmth of people’s clothes. These can be attributed to (i) the coupled graph construction that contains abundant modality-aware knowledge to ground the reasoning, as well as (ii) the effective design of our pseudo-siamese graph neural network. It benefits sufficient preservation of intra-modal information and adequate cross-modal fusion, resulting in a powerful multi-hop reasoning ability over both inherent visual features and external knowledge.

heading: 5 Related Work
text: KVQA with KGs. Early studies either dedicated to integrating different knowledge sources (Wang et al., 2017; Dong et al., 2023b) or proposed various fusion algorithms for multimodal information (Marino et al., 2021). ConceptBERT (Gardères et al., 2020) constrains the multimodal information with question embedding and fuses embeddings of each modality for prediction. MAVEx (Wu et al., 2022) aims to discern the corresponding knowledge source for each candidate answer to reduce noise. KRISP (Marino et al., 2021) captures both implicit information in both questions, images and KGs. KVQA with LLMs. Recently, large language models (LLMs) have surprised the community with their superior understanding of texts (Guo et al., 2023). PICa (Yang et al., 2022) first leverages GPT3 (Brown et al., 2020) as an implicit knowl-
edge source for reasoning by prompting the image captions and in-context examples. Another pipeline of studies employs LLMs to generate candidates or supporting evidence for particular captions, e.g., KAT (Gui et al., 2022) and REVIVE (Lin et al., 2022). While they do not fully leverage the multiple sources of knowledge, we break the limitation of complex reasoning by developing a tailored multimodal fusion algorithm that balances intra- and inter-modal learning.

heading: 6 Conclusions
text: We present MAIL, a modality-aware integration with large language models for knowledge-based visual question answering. We formally define a novel multimodal learning paradigm for comprehensive cross-modal reasoning among multiple knowledge sources. The knowledge from LLMs is effectively leveraged via a carefully designed prompting for coupled graph construction, i.e., scene graph and concept graph. Then we integrate various multimodal information with a tailored pseudo-siamese graph medium fusion. It effectively enhances a tight inter-modal interaction and maximally preserves insightful intra-modal processing. MAIL achieves superiority on two benchmark datasets while requiring 2∼ 4× faster inferential time than the existing state-of-the-art baselines.
Limitations Since the real-world knowledge graphs only contain concrete real-world entities, our KG-based VQA is naturally limited to being able to answer subjective questions, e.g., YES/NO and abstract opinions. We will include this valuable direction in future work to equip our model with generative components to handle the subjective questions.
Ethics Statement We confirm that we have fully complied with the ACL Ethics Policy in this study. We conduct experiments with widely adopted publicly available datasets. The generated image captions, processed scene graphs and concept graphs will be opensourced for other researchers’ fair reproduction and further study in the active KVQA community.

heading: Acknowledgements
text: The work described in this paper was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU 25208322).

heading: A Appendix
text: A.1 Generalization study with InstructBLIP To demonstrate the generalization ability of our proposed MAIL, we adopt a more advanced multimodal LLM, i.e., InstructBLIP, to replace MiniGPT-4. Though we lack the subjective ability to answer YES/NO questions, we have achieved comparable improvements that significantly demonstrate ourselves with the nonnegligible merits for KBQA.
A.2 Baselines used on FVQA Dataset To further demonstrate the generalization ability of our proposed MAIL, we compare it with the widely adopted baselines on the first KVQA dataset FVQA, i.e., XNM (Shi et al., 2019), KINet (Zhang et al., 2021), UnifER (Guo et al., 2022), MCAN (Yu et al., 2019) and HCNMN (Zhang et al., 2023b). For external knowledge, KI-Net uses ConceptNet and Wordnet; UnifER uses VisualBert, LXMERT and ViLT; HCNMN uses WordNet, WikiText, ConceptNet and Visual Genome.
A.3 Additional Case Studies Case studies on MiniGPT-4. For fair comparisons, we further show the prediction results of MiniGPT4 over the six examples in Table 11. It can only answer 2 out of 6 questions, which effectively indicates the superiority of both our task setting with candidate answers from KGs and the reasonable design of PS-GMF. From the results, we see that multimodal LLMs like MiniGPT-4 indeed require a careful design to be properly utilized so as to maximize their ability while solely using it for VQA could be unreliable. Case Studies of MAIL w/o GMF. We showcase the predictions on both single-hop and multi-hop questions in this subsection without graph medium fusion in Table 11 line 3. The observation suggests the necessary cross-modal reasoning ability brought by GMF, where we can answer 3 out of 6 questions while failing the majority of complex multi-hop questions.

heading: Future Work
text: We will include this valuable direction in future work to equip our model with generative components to handle the subjective questions