MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
Authors: Shuo Yang, Caren Han, Siwen Luo, Eduard Hovy
Accepted at: acl25

MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering

Shuo Yang1, Soyeon Caren Han1*, Siwen Luo2, Eduard Hovy1
1The University of Melbourne, 2The University of Western Australia

Abstract
Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios. To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs. MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning. While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs. MAGIC-VQA bridges a key gap by unifying commonsense knowledge with LVLM-driven reasoning, eliminating the need for extensive pre-training or complex prompt tuning. Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA. Our implementation is open-sourced on GitHub1

1 Introduction
Visual Question Answering (VQA) (Antol et al., 2015; Goyal et al., 2017; Yue et al., 2024) is a complex task requiring models to understand the interaction between visual inputs and textual queries. In recent years, Large Vision-Language Models (LVLMs) (Dai et al., 2023; Wang et al., 2024b; Liu et al., 2024b; Li et al., 2024a; OpenAI, 2024b) have made substantial progress in VQA through extensive pre-training on massive image-text datasets and instruction tuning. These models excel at object-level visual recognition and semantic understanding, capturing attributes such as spatial relationships and contextual details.

Nevertheless, LVLMs often face challenges on questions requiring commonsense reasoning, particularly those hinging on implicit contextual cues or everyday world knowledge (Zhou et al., 2023; Ye et al., 2022; Li et al., 2023b)2. To overcome this limitation and improve the performance on , different methods have been explored. For example, Multimodal retrieval-augmented generation leverages dense retrieval to inject external multimodal information into the generation process, thereby enhancing the factual grounding of LVLMs (Lin and Byrne, 2022; Hu et al., 2023). Multimodal prompt tuning harnesses the model’s innate commonsense knowledge by carefully crafting prompts that combine visual and textual cues from representative samples, guiding LVLMs to leverage their internal reasoning for context-rich answers (Wei et al., 2022; Zhang et al., 2023). However, static prompt design usually lacks the dynamic adaptability required for novel scenarios, resulting in limited generalization to unseen or diverse inputs. Additionally, graph-based approaches utilize Graph Neural Networks (GNNs) to incorporate structured commonsense knowledge (Ravi et al., 2023; Wang et al., 2022), which surpasses the limitations of purely parametric LVLMs, enabling models to capture explicit and implicit knowledge connections via structured graphs.

However, a key missing component in existing works is the effective integration of commonsense knowledge with LVLMs while addressing their inherent shortcomings. Prior approaches either rely on static retrieval that indiscriminately injects input-unaware noisy knowledge or graph-based augmentation that overlooks the dynamic interplay between external and innate knowledge. Our work seeks to fill this gap by proposing a unified framework that systematically combines dynamic, contextu-

1*Corresponding author. Caren.Han@unimelb.edu.au
1https://github.com/adlnlp/magic_vqa
2The sample illustrations can be found in Section 6 and Appendix B.
ally aligned commonsense integration with structured graph-based reasoning to robustly filter and incorporate relevant commonsense knowledge.

In this paper, we introduce MAGIC-VQA, a novel framework designed to enhance VQA models by effectively integrating commonsense knowledge with LVLMs. MAGIC-VQA is built upon a three-stage process that not only improves reasoning capabilities but also mitigates the complexity associated with large-scale pre-training and inefficient prompt-based approaches. First, explicit Commonsense Knowledge Integration extracts relevant knowledge triples from external sources, establishing a reliable reasoning foundation. Secondly, by-Type Commonsense Knowledge Post-processing refines these triples based on input-specific needs, ensuring contextual relevance. Finally, implicit Commonsense Knowledge Augmentation constructs a heterogeneous multimodal graph processed by a GNN to capture intricate relationships, providing structured reasoning beyond what LVLMs alone can infer. By integrating explicit and implicit commonsense knowledge on top of LVLMs, MAGIC-VQA addresses both the limitations of previous approaches and the missing component in existing works. Our main contributions are as follows:

1. We propose MAGIC-VQA, a novel end-to-end framework that systematically integrates both explicit and implicit commonsense knowledge into VQA through, without extensive pre-training or intricate prompt tuning.

2. MAGIC-VQA employs a three-stage pipeline—explicit commonsense integration, by-type post-processing, and graph-based implicit augmentation—that dynamically extracts and filters commonsense knowledge in an input-aware manner, and leverages a GNN-based structured reasoning mechanism.

3. We conduct extensive evaluations across multiple VQA benchmarks, demonstrating robust improvement in commonsense understanding reasoning for VQA, surpassing existing models in both knowledge grounding and inference accuracy.

2 Related Work

2.1 VLPM and LVLMs on VQA

Vision-Language Pretrained Models (VLPMs) like ViLBERT (Su et al., 2019), ALBEF (Li et al., 2021) and ViLT (Kim et al., 2021) have advanced Visual Question Answering (VQA) by improving the alignment between visual and textual modalities in the last few years(Long et al., 2022a). Recently, Large Vision-Language Models (LVLMs) like InstructBLIP (Dai et al., 2023), LLaVA (Liu et al., 2024b), GPT4o (OpenAI, 2024b) and Gemini1.5 (Gemini Team, 2024) further push the boundary of VQA with strong in-context learning capability through extensive pre-training and instruction-tuning on large-scale image-text datasets. While these models have demonstrate their effectiveness in domains such as math (Jiang et al., 2024; Wang et al., 2025a), robotics (Zhou et al., 2025a), logical reasoning (Bi et al., 2025b), medical (Wang et al., 2025e,d) and science (Lu et al., 2022), these models still face challenges with questions requiring commonsense knowledge that is intuitive and straightforward for humans, such as reasoning based on implicit contextual cues or general world knowledge (Ye et al., 2023; Jiang et al., 2023; Chen et al., 2024a; Yang et al., 2025; Zhou et al., 2025b; Bi et al., 2025a; Jiang et al., 2025). The resource-intensive nature of these models further makes it infeasible to train a model from scratch for enhanced commonsense understanding (Bi et al., 2024).

2.2 Commonsense Knowledge Integration for Visual Question Answering

Several studies have highlighted the critical role of commonsense knowledge integration in enhancing the performance of VLPMs and LVLMs on VQA tasks (Wu et al., 2022; Zhang et al., 2022; Ding et al., 2022; Wang et al., 2024c; Li et al., 2025a; Wang et al., 2025b). These methods can be classified into two approaches: explicit commonsense knowledge integration and implicit commonsense knowledge integration.

1) Explicit commonsense knowledge integration directly incorporates external commonsense knowledge into model training through instruction tuning, prompt tuning, or reinforcement learning. For example, VLC-BERT (Ravi et al., 2023) encodes the contextualized commonsense knowledge of the question phrases as additional textual features and integrates with object visual features to fine-tune the VL-BERT (Su et al., 2019). MM-CoT (Zhang et al., 2023), T-SciQ (Wang et al., 2024a) and KAM-CoT (Mondal et al., 2024) fine-tune models on commonsense-augmented Chain-of-Thought (CoT) data to enhance their reasoning processes. Prism (Bi et al., 2025a) incorporate the
<table>
  <tr>
    <th>KG</th>
    <th>Size</th>
    <th>Main Coverage</th>
    <th>Key Relations</th>
  </tr>
  <tr>
    <td>ConceptNet</td>
    <td>8M</td>
    <td>PE</td>
    <td><i>IsA, UsedFor</i></td>
  </tr>
  <tr>
    <td>ATOMIC</td>
    <td>877K</td>
    <td>EC, SI</td>
    <td>xWant, oEffect</td>
  </tr>
  <tr>
    <td>ATOMIC2020</td>
    <td>1.33M</td>
    <td>PE, EC, SI</td>
    <td>23 relation types</td>
  </tr>
</table>

Table 1: Comparison of three commonsense knowledge graphs. 'PE' refers to physical entity-related commonsense, 'EC' to event-centered related commonsense, and 'SI' to social interaction-related commonsense.

filtered commonsense knowledge into model via dynamic data selection. VI-Rethinker (Wang et al., 2025b) and Pixel Reasoner (Su et al., 2025) inject commonsense knowledge into vision-language model via reinforcement learning. However, these methods suffer from static commonsense integration without dynamic filtering to adjust to varying input contexts, resulting in potential noise that impedes nuanced reasoning (Li et al., 2024b).

2) Implicit Commonsense Knowledge Integration focuses on transferring knowledge from a teacher to a student model without directly incorporating external datasets. For example, (Dai et al., 2022) distill knowledge from the dual-stream CLIP (Radford et al., 2021) into BART (Lewis, 2019), achieving strong zero-shot performance. Park et al. (2024) proposed a novel method to distill knowledge from LLMs on specific image regions, then guiding the LLM to infer commonsense knowledge about those areas. However, these methods often overlook the structured interplay among visual, textual, and commonsense cues, limiting their ability to perform nuanced reasoning (Wu et al., 2025).

3 MAGIC-VQA

MAGIC-VQA employs a three-stage process to integrate commonsense knowledge into LVLMs, as in Figure 1. (1) Explicit Commonsense Knowledge Retrieval extracts relevant triples from an external knowledge graph. (2) By-Type Commonsense Knowledge Post-processing refines these triples, aligning them with dataset-specific distributions and assigning relevance levels. (3) Implicit Commonsense Knowledge Augmentation constructs a multimodal graph processed by a GNN to generate confidence scores. These scores, along with the refined triples with relevance level, image, and question, form a comprehensive input to the LVLMs for robust commonsense-grounded inference.

3.1 Explicit Commonsense Knowledge Integration

We begin by integrating explicit commonsense knowledge into LVLM for each input modality. Given a dataset sample consisting of an image \( I \) and an associated question \( Q \), we first generate an image caption \( C \) using BLIP2 (Li et al., 2023a) as additional contextual information. Next, we encode the inputs \( \{I, Q, C\} \) into a shared embedding space, obtaining representations \( f_I, f_Q, \) and \( f_C \) using the same model.

We adopt ATOMIC2020 (Hwang et al., 2021) as our external knowledge source because of its broad coverage of physical-entity (PE), event-centered (EC), and social-interaction (SI) relations, as shown in Table 1. Spanning 1.33 million triplets and 23 relation types, it offers a more balanced scope than either ConceptNet (Speer et al., 2017) or the earlier ATOMIC (Sap et al., 2019), making it especially relevant for everyday objects, actions, and social contexts encountered in VQA. These 23 relations fall into three groups: (1) Physical Entity (PE): object properties and functions like “paper is made of cellulose”. (2) Event-Centered (EC): situational sequences or events, such as “X eats breakfast” typically happening before “X goes to work.” (3) Social Interaction (SI): human interactions, intentions and emotions, such as “PersonX gives a gift,” leading to “PersonY feels appreciated.” The complete list of relations within each group is covered in Appendix C.

To retrieve relevant commonsense knowledge, we encode the head and tail entities of all ATOMIC2020 candidates using the same BLIP2 model, then compute cosine similarities between these entity embeddings and input embeddings \( f \in \{f_I, f_Q, f_C\} \). We select the top \( K \) triplets with the highest cosine similarity scores per input embedding \( f \). This ensures only the contextually pertinent commonsense knowledge is retained, providing a solid foundation for the subsequent refinement and integration stages.

3.2 By-type Commonsense Knowledge Post-Processing

After acquiring an initial pool of commonsense triplets, we further refine them through a by-type post-processing stage, ensuring each of them is both tailored to each dataset’s specific needs and contextually aligned. This stage involves two main steps: (1) By-type Commonsense Knowledge Fil-
Figure 1: The proposed MAGIC-VQA Framework Architecture, which includes diverse approaches to integrate commonsense knowledge to Visual Question Answering. The detailed description of each step - 1) Explicit Commonsense Knowledge Retrieval, 2) By-Type Post-processing, 3) Implicit Commonsense Augmentation - is aligned with the subsection titles under Section 3.

tering, and (2) Relevance Level Assignment.

By-type Commonsense Knowledge Filtering customizes the selection of retrieved triplets by matching the desired commonsense type distribution for each dataset. As discovered in Figure 3, each dataset benefits from a distinct mix of commonsense types. We first discard triplets with similarity scores below a threshold \( \tau \). Let \( T = \{CS\text{-}PE, CS\text{-}EC, CS\text{-}SI\} \) represent the commonsense types, with each type \( t \) allocated a target proportion \( p_t \). We then select \( k_t = \lfloor p_t \times k \rfloor \) triplets from each type \( t \) with the highest similarity scores, ensuring the final set reflects the dataset’s recommended distribution of commonsense knowledge. Details on these ratios are in Section 4.2.

Relevance Level Assignment further assign a qualitative relevance level to each filtered triplet based on its cosine similarity score with the input sample, assisting the model in prioritizing most meaningful knowledge during reasoning. For each input source \( f \in \{f_I, f_Q, f_C\} \), we first aggregate all cosine similarity scores \( S_f = \{s_j^{(f)}\} \) of the selected triplets. We compute the mean \( \mu_f \) and standard deviation \( \sigma_f \) of these scores for each dataset:

\[
\mu_f = \frac{1}{N_f} \sum_{j=1}^{N_f} s_j^{(f)}
\] (1)

\[
\sigma_f = \sqrt{\frac{1}{N_f} \sum_{j=1}^{N_f} \left( s_j^{(f)} - \mu_f \right)^2}
\] (2)

where \( N_f \) represents the total number of selected triplets for that input source \( f \). As the scores have a roughly normal distribution, we apply dynamic thresholding that uses mean \( \mu_f \) and standard deviation \( \sigma_f \) to assign each triplet a relevance level:

\[
L(s_j^{(f)}) =
\begin{cases}
High & \text{if } s_j^{(f)} \geq \mu_f + \frac{\sigma_f}{2} \\
Medium & \text{if } \mu_f - \frac{\sigma_f}{2} \leq s_j^{(f)}, \\
& \text{and } s_j^{(f)} < \mu_f + \frac{\sigma_f}{2} \\
Low & \text{if } s_j^{(f)} < \mu_f - \frac{\sigma_f}{2}
\end{cases}
\] (3)

Detailed distribution of the similarity score for each dataset is provided in Appendix 13.

3.3 Implicit Commonsense Knowledge Augmentation

While explicit retrieval yields relevant commonsense triplets, an implicit augmentation step allows these triplets to be more deeply integrated into the reasoning process. We construct a heterogeneous graph \( G_n = \{V, E\} \) where each input node (image \( I \), question \( Q \), and caption \( C \)) is interconnected and also linked to \( k \) additional commonsense nodes.
Figure 2: The comparison among VQA datasets. We selected one dataset from each of three groups. We modified the figure from (Yue et al., 2024).

These commonsense nodes are derived by flattening filtered commonsense triplets from Section 3.2, thereby converting each triplet into a short natural-language sentence for more straightforward integration3. Edges between nodes are constructed based on cosine similarity scores between their embeddings, highlighting the semantic relevance between each pair of nodes. The graph is then processed using a two-layer Graph Convolutional Network (GCN) to iteratively update node embeddings:

\[
H^{(l+1)} = \rho \left( \tilde{A} H^{(l)} W_l \right)
\]

where \( \rho \) is a nonlinear activation function and \( \tilde{A} \) is the normalized adjacency matrix. The node embeddings \( H^{(2)} \) are pooled to form a unified graph representation for each sample, which is then passed through a Multi-Layer Perceptron (MLP) to produce a confidence score over candidate answers. These confidence scores provide a commonsense-augmented signal to the LVLM, enabling it to prioritize answers grounded in relevant knowledge and improving inference reliability.

3.4 Commonsense Grounded Inference

In the final inference stage, we combine all processed elements—original inputs (\( I, Q, C \)), refined commonsense triplets (with assigned relevance levels) and GNN-generated confidence scores—into a unified input structure for inference with LVLMs. A complete input example is provided in Appendix F. By fusing explicit and implicit commonsense knowledge with visual and textual signals, the LVLMs can reason effectively about nuanced relationships, delivering answers better aligned with real-world understanding.

Figure 3: The distribution of categories of commonsense. CS-PE refers to physical entity-related commonsense, CS-EC to event-centered related commonsense, and CS-SI to social interaction-related commonsense.

4 Experiment

4.1 Dataset

We evaluated MAGIC-VQA on three representative VQA benchmarks of diverse complexity and depth as highlighted in Figure 2.

ScienceQA (Lu et al., 2022) comprises over 21,000 multiple-choice questions from elementary and middle school curricula in natural, social, and language science. It tests factual and procedural understanding, requiring integration of commonsense about the physical world and scientific phenomena. We select only samples with image contexts.

TextVQA (Singh et al., 2019) contains over 45,000 questions grounded in 28,000 real-world images with embedded text like signs and labels. It demands OCR to extract textual elements and integrate them with everyday commonsense provided in the context to interpret them within the visual scene. We use its validation set for our evaluation.

MMMU (Yue et al., 2024) consists of 11,550 college-level questions spanning diverse disciplines. It features challenging image types such as medical diagnosis, music sheets and so on, which goes beyond the everyday commonsense understanding emphasized in ScienceQA and TextVQA. We choose its validation set to evaluate our model.

4.2 Commonsense Knowledge Distribution

To tailor the commonsense knowledge to each dataset’s specific reasoning requirements, we ana-

3We apply a rule-based triplets flatten mechanism covered in Appendix C
Table 2: Performance comparison under four configurations: (1) None: Inputs with no additional commonsense knowledge. (2) CS Sources: Inputs enriched with commonsense knowledge from different sources, including question (CS-Q), image (CS-I), and image caption (CS-C). (3) CS Categories: Inputs enriched with commonsense knowledge from different categories, including Physical Entities (CS-PE), Event-Centered (CS-EC), and Social Interaction (CS-SI). (4) All CS: Inputs enriched with all source-based and category-based commonsense.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">None</th>
    <th colspan="3">CS Sources</th>
    <th colspan="3">CS Categories</th>
    <th rowspan="2">All CS</th>
  </tr>
  <tr>
    <th>CS-Q</th>
    <th>CS-I</th>
    <th>CS-C</th>
    <th>CS-PE</th>
    <th>CS-EC</th>
    <th>CS-SI</th>
  </tr>
  <tr>
    <td colspan="9"><b>ScienceQA<sub>IMG</sub></b></td>
  </tr>
  <tr>
    <td>LLaVA1.6</td>
    <td>67.50</td>
    <td>68.83</td>
    <td>71.56</td>
    <td>70.35</td>
    <td>71.12</td>
    <td>69.01</td>
    <td>70.83</td>
    <td><b>72.30</b></td>
  </tr>
  <tr>
    <td>BLIP3</td>
    <td>70.00</td>
    <td>71.56</td>
    <td>73.88</td>
    <td>72.97</td>
    <td>74.03</td>
    <td>71.57</td>
    <td>71.05</td>
    <td><b>74.30</b></td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>71.99</td>
    <td>72.58</td>
    <td>74.37</td>
    <td>73.91</td>
    <td>74.56</td>
    <td>73.09</td>
    <td>73.21</td>
    <td><b>74.62</b></td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>71.39</td>
    <td>72.21</td>
    <td>74.83</td>
    <td>71.86</td>
    <td>74.22</td>
    <td>72.03</td>
    <td>72.57</td>
    <td><b>75.95</b></td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>76.45</td>
    <td>77.34</td>
    <td>79.83</td>
    <td>77.17</td>
    <td>79.63</td>
    <td>77.52</td>
    <td>78.87</td>
    <td><b>81.22</b></td>
  </tr>
  <tr>
    <td colspan="9"><b>TextVQA<sub>val</sub></b></td>
  </tr>
  <tr>
    <td>LLaVA1.6</td>
    <td>62.30</td>
    <td>63.55</td>
    <td>64.82</td>
    <td>64.23</td>
    <td>64.77</td>
    <td>65.05</td>
    <td>64.89</td>
    <td><b>65.20</b></td>
  </tr>
  <tr>
    <td>BLIP3</td>
    <td>67.80</td>
    <td>68.49</td>
    <td>69.64</td>
    <td>68.29</td>
    <td>69.12</td>
    <td>69.64</td>
    <td>69.24</td>
    <td><b>69.80</b></td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>73.21</td>
    <td>74.06</td>
    <td>75.19</td>
    <td>74.81</td>
    <td>74.60</td>
    <td>75.01</td>
    <td>74.82</td>
    <td><b>75.30</b></td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>75.30</td>
    <td>76.07</td>
    <td>77.63</td>
    <td>77.05</td>
    <td>76.57</td>
    <td>78.02</td>
    <td>76.85</td>
    <td><b>78.90</b></td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>78.98</td>
    <td>79.34</td>
    <td>81.25</td>
    <td>80.63</td>
    <td>80.93</td>
    <td>81.51</td>
    <td>81.22</td>
    <td><b>82.13</b></td>
  </tr>
  <tr>
    <td colspan="9"><b>MMMU<sub>val</sub></b></td>
  </tr>
  <tr>
    <td>LLaVA1.6</td>
    <td>48.38</td>
    <td>49.27</td>
    <td>53.52</td>
    <td>49.85</td>
    <td>52.03</td>
    <td>52.57</td>
    <td>53.10</td>
    <td><b>54.30</b></td>
  </tr>
  <tr>
    <td>BLIP3</td>
    <td>41.31</td>
    <td>42.54</td>
    <td>45.89</td>
    <td>42.19</td>
    <td>44.12</td>
    <td>46.03</td>
    <td>45.89</td>
    <td><b>47.60</b></td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>51.00</td>
    <td>52.17</td>
    <td>55.48</td>
    <td>54.21</td>
    <td>54.23</td>
    <td>52.67</td>
    <td>53.50</td>
    <td><b>55.80</b></td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>51.10</td>
    <td>52.69</td>
    <td>55.89</td>
    <td>54.83</td>
    <td>53.60</td>
    <td>54.57</td>
    <td>54.10</td>
    <td><b>57.42</b></td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>55.89</td>
    <td>56.53</td>
    <td>58.79</td>
    <td>56.21</td>
    <td>58.12</td>
    <td>57.57</td>
    <td>57.89</td>
    <td><b>60.87</b></td>
  </tr>
</table>

lyze the distribution of commonsense types across each dataset using GPT4 (OpenAI, 2024a)4. As Figure 3 suggests, ScienceQA requires more Physical Entity (CS-PE) knowledge, possibly due to its focus on factual scientific concepts. Meanwhile, TextVQA, which often involves contextual understanding in images, benefits more from Event-Centered (CS-EC) knowledge. MMMU, however, requires a balanced mix of Physical Entity, Event-Centered, and Social Interaction (CS-SI) commonsense due to its multi-disciplinary nature. As a result, we set the by-type filtering ratio of {CS-PE:CS-EC:CS-SI} mentioned in Section 3.2 as {0.7:0.15:0.15} in ScienceQA, {0.2:0.6:0.2} in TextVQA, {0.33:0.33:0.33} in MMMU dataset.

4.3 Baselines, Metric, and Implementations

The selected baselines are four open source state-of-the-art LVLMs: LLaVA-1.6 (Liu et al., 2024a), XGen-MM (BLIP-3) (Xue et al., 2024), InternVL2 (Chen et al., 2024b), Qwen2VL (Wang et al., 2024b), and one proprietary model, GPT4o-mini (OpenAI, 2024b). These LVLMs are selected for their outstanding zero-shot performance in VQA tasks. Details of each baseline model are in Appendix D. We adopt accuracy as the evaluation metric following prior works (Singh et al., 2019; Lu et al., 2022; Yue et al., 2024). All experiments are conducted with and without the proposed MAGIC-VQA under a zero-shot setup. More implementation details are in Appendix E.

5 Results

5.1 Explicit Commonsense Knowledge

We evaluated the explicit integration of commonsense knowledge triplets by systematically testing four configurations of: (1) None: Inputs with no additional commonsense; (2) CS Sources: Inputs augmented with commonsense from questions, images, or captions; (3) CS Categories: Inputs augmented with commonsense grouped by category (Physical Entities, Event-Centered, Social Interaction); and (4) All CS: Inputs augmented with all retrieved commonsense5. As in Table 2, integrating explicit commonsense consistently improves performance across all baselines and three datasets. For instance, on ScienceQA, GPT-4O’s accuracy rises from 76.45% (None) to 81.22% (All CS), and

4The prompt template is in Appendix G

5Each experiment is tested with a fixed number of k = 6 to maintain a fair comparison. We further investigated the impact of incorporating different numbers of commonsense knowledge triplets in Appendix A.1.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Ex-CS Im-CS</th>
    <th colspan="3">Performance</th>
  </tr>
  <tr>
    <th>CS Rel</th>
    <th>Conf</th>
    <th>SQA</th>
    <th>MMMU</th>
    <th>TVQA</th>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✗</td>
    <td>✗</td>
    <td>71.39</td>
    <td>51.10</td>
    <td>75.30</td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✓</td>
    <td>✗</td>
    <td>75.11</td>
    <td>56.00</td>
    <td>78.50</td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✗</td>
    <td>✓</td>
    <td>72.88</td>
    <td>53.41</td>
    <td>76.42</td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✓</td>
    <td>✓</td>
    <td>75.95</td>
    <td>57.42</td>
    <td>78.90</td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✓</td>
    <td>✓</td>
    <td>76.42</td>
    <td>57.21</td>
    <td>79.10</td>
  </tr>
  <tr>
    <td>Qwen2VL</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>77.12</b></td>
    <td><b>58.72</b></td>
    <td><b>79.80</b></td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>✗</td>
    <td>✗</td>
    <td>76.45</td>
    <td>55.89</td>
    <td>78.98</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>✓</td>
    <td>✗</td>
    <td>80.07</td>
    <td>59.30</td>
    <td>81.73</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>✗</td>
    <td>✓</td>
    <td>77.02</td>
    <td>57.64</td>
    <td>79.55</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>✓</td>
    <td>✓</td>
    <td>81.22</td>
    <td>60.87</td>
    <td>82.13</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>✓</td>
    <td>✓</td>
    <td>80.94</td>
    <td>60.25</td>
    <td>82.50</td>
  </tr>
  <tr>
    <td><b>GPT4o-mini</b></td>
    <td>✓</td>
    <td>✓</td>
    <td><b>82.50</b></td>
    <td><b>61.03</b></td>
    <td><b>83.37</b></td>
  </tr>
</table>

Table 3: Quantitative analysis on the effect of each component of MAGIC-VQA. The "Ex-CS" (CS and Rel) denotes explicit commonsense knowledge inclusion (All-CS in Section 5.1), while "Im-CS" (Conf) denote implicit commonsense inclusion. (√) and (✗) denotes the inclusion and exclusion of a component.

Qwen2VL improves from 51.10% to 57.42% on MMMU under the same setup. Examining the effect of source-based commonsense reveals that image-driven knowledge (CS-I) typically provides the largest gains. For example, LLaVA1.6 on MMMU jumps from 48.38% to 53.52% with CS-I, surpassing the minor improvements from CS-Q or CS-C. This suggests that leveraging image-aligned commonsense offers more grounded cues for inference. However, category-based commonsense (CS-PE, CS-EC, and CS-SI) exhibits dataset-dependent effectiveness. On ScienceQA, CS-PE is most beneficial, while on TextVQA, CS-EC dominates, and MMMU shows a more balanced pattern. These results align with our earlier commonsense distribution analysis in Section 4.2, highlighting the importance of tailoring knowledge retrieval to the dataset’s unique characteristics.

5.2 Implicit Commonsense Knowledge

We next examined the effect of implicit commonsense knowledge augmentation as outlined in Section 3.3 using two representative LVLMs, Qwen2VL and GPT4o-mini. We also include results from explicit commonsense knowledge integration (All-CS in Section 5.1) as a reference. As Table 3 suggests, while the implicit commonsense knowledge Conf does not contribute as significantly as explicit commonsense knowledge (Ex-CS), it nonetheless provides complementary information that enhances overall performance. Incorporating only Conf with Qwen2VL improves the MMMU accuracy from 51.10% to 53.41%. We also observe that adding Rel notably improves results across all three datasets. Furthermore, combining implicit and explicit commonsense yields the highest overall performance, indicating that implicit augmentation complements explicit knowledge by capturing additional context that explicit methods alone may miss. Further analysis is provided in Section 6 and Appendix B.

<table>
  <tr>
    <th>Model</th>
    <th>Knowledge</th>
    <th>A-OKVQA</th>
    <th>VCR</th>
  </tr>
  <tr>
    <td>VLCBERT</td>
    <td>COMET</td>
    <td>38.05</td>
    <td>79.24</td>
  </tr>
  <tr>
    <td>KAT</td>
    <td>Wikidata + GPT3</td>
    <td>49.74</td>
    <td>83.18</td>
  </tr>
  <tr>
    <td>KRISP</td>
    <td>Wikidata + CNet</td>
    <td>27.10</td>
    <td>65.12</td>
  </tr>
  <tr>
    <td>Ours (LLaVA1.6)</td>
    <td>ATOMIC2020</td>
    <td>73.33</td>
    <td>89.91</td>
  </tr>
  <tr>
    <td>Ours (Qwen2VL)</td>
    <td>ATOMIC2020</td>
    <td>75.95</td>
    <td>91.24</td>
  </tr>
  <tr>
    <td>Ours (GPT4o-mini)</td>
    <td>ATOMIC2020</td>
    <td><b>76.55</b></td>
    <td><b>93.42</b></td>
  </tr>
</table>

Table 4: Comparison of knowledge-augmented VQA models on two benchmarks. For A-OKVQA, we select only its multiple-choice question as the subset. CNet stands for the Conceptnet dataset

To further assess the effectiveness of MAGIC-VQA’s knowledge injection, we compare it with three representative knowledge-augmented baselines including KAT (Gui et al., 2021), VLC-BERT (Ravi et al., 2023) and KRISP (Marino et al., 2021) on two knowledge-intensive VQA benchmarks: VCR (Zellers et al., 2019) and the multiple-choice questions subset of A-OKVQA (Schwenk et al., 2022). As shown in Table 4, MAGIC-VQA consistently outperforms all competitors on both datasets. In particular, the GPT-4o-mini variant achieves the highest accuracy of 76.55% on A-OKVQA and 93.42% on VCR, exceeding KAT by more than 26% and 10%, respectively. This improvement highlight the strength of our commonsense-infused visual reasoning. Moreover, unlike prior knowledge-augmented models that require specialized pre-training, MAGIC-VQA is a plug-and-play framework that can be seamlessly integrated with different backbones, making it easily extensible to future models and knowledge bases.
Figure 4: Effect of the ratio of commonsense knowledge categories on Qwen2VL. The three categories of explicit commonsense knowledge include social interactions (CS-SI), physical entities (CS-PE), and event-based relations (CS-EC).

5.4 Ratio of Commonsense Knowledge Type
To determine the optimal ratio of commonsense knowledge types, we conduct an experiment to analyze how varying distributions of CS-PE, CS-EC, and CS-SI knowledge triplets impact the performance of Qwen2-VL on the ScienceQA dataset. Figure 4 indicates that the model achieves its highest accuracy of 75.11% with a 4:1:1 ratio, which places greater emphasis on CS-PE knowledge. Additionally, distributions favoring CS-PE triplets consistently result in improved performance. For example, a 3:1:2 ratio achieves an accuracy of 74.73%. In contrast, ratios prioritizing CS-EC or CS-SI, such as 1:2:3 or 1:3:2, yield lower accuracies of 72.56% and 72.41%, respectively. These results suggest that CS-PE is the most essential commonsense knowledge type for the ScienceQA, aligning with its focus on physical concepts and entities as discussed in Section 5.1. Notably, the 4:1:1 ratio closely mirrors the inherent distribution of commonsense knowledge in ScienceQA in Figure 3. This alignment suggests tailoring the balance of commonsense knowledge to match the dataset’s inherent characteristics, as demonstrated by our approach, leads to the most significant performance improvements. We further attach the effect of diverse similarity metrics by comparing Manhattan, Cosine, and Euclidean Distance when applied to the explicit knowledge retrieval in Appendix A.2.

5.5 Break Down Results
We provide break down result on subcategories within each selected dataset using Qwen2VL and GPT4o-mini as illustrated in Figure 5. Across all datasets and subcategories, incorporating commonsense significantly improves the accuracy. Each dataset features distinct subcategories that would benefit from varying aspects of commonsense reasoning. First, as illustration of ScienceQA in Figure 5a, commonsense augmentation yields notable improvements in language-related subcategories, reflecting the value of context-sensitive reasoning. Next shown in Figure 5b for TextVQA, categories involving concrete objects such as ‘uniform’ and ‘books’, benefit more from commonsense augmentation compared to abstract categories like ‘persons’, indicating concrete objects allow for more precise retrieval of relevant knowledge. MMMU in Figure 5c, commonsense augmentation benefits easy-level questions closely tied to everyday knowledge, while struggles with hard-levels that demand complex reasoning beyond commonsense.

![Bar chart showing the effect of the ratio of commonsense knowledge categories on Qwen2VL](page_98_120_627_246.png)

(a) ScienceQA

(b) TextVQA

(c) MMMU

![Three bar charts showing subcategory-level accuracy for ScienceQA, TextVQA, and MMMU under three conditions: Without CS, With EX-CS and With All-CS](page_728_120_627_1012.png)
<table>
  <tr>
    <th>Category / Model</th>
    <th>Year</th>
    <th>Size</th>
  </tr>
  <tr>
    <th colspan="3">Vision-Language Transformers</th>
  </tr>
  <tr>
    <td>VL-BERT (Su et al., 2019)</td>
    <td>2019</td>
    <td>111M</td>
  </tr>
  <tr>
    <td>VILT (Kim et al., 2021)</td>
    <td>2021</td>
    <td>112M</td>
  </tr>
  <tr>
    <td>UnifiedQA (Khashabi et al., 2020)</td>
    <td>2020</td>
    <td>223M</td>
  </tr>
  <tr>
    <th colspan="3">Knowledge Augmented Transformers</th>
  </tr>
  <tr>
    <td>KRISP (Marino et al., 2021)</td>
    <td>2020</td>
    <td>116M</td>
  </tr>
  <tr>
    <td>KAT (Gao et al., 2021)</td>
    <td>2021</td>
    <td>175B</td>
  </tr>
  <tr>
    <td>ViCBERT (Ravi et al., 2023)</td>
    <td>2022</td>
    <td>118M</td>
  </tr>
  <tr>
    <th colspan="3">Multimodal CoT Models</th>
  </tr>
  <tr>
    <td>MM-COT (Zhang et al., 2023)</td>
    <td>2023</td>
    <td>223M</td>
  </tr>
  <tr>
    <td>KAM-COT (Mondal et al., 2024)</td>
    <td>2024</td>
    <td>250M</td>
  </tr>
  <tr>
    <th colspan="3">Large Vision-Language Models</th>
  </tr>
  <tr>
    <td>LLaVA (Liu et al., 2023)</td>
    <td>2023</td>
    <td>7B</td>
  </tr>
  <tr>
    <td>GPT4 (OpenAI, 2024a)</td>
    <td>2024</td>
    <td>>175B</td>
  </tr>
  <tr>
    <td>Qwen2-VL (Wang et al., 2024b)</td>
    <td>2024</td>
    <td>7B</td>
  </tr>
  <tr>
    <th>Ours</th>
    <th></th>
    <th>0.33M</th>
  </tr>
</table>

Table 5: Comparison with notable previous methods from different categories with respect to parameter size.

5.6 Computation Resource Comparison

We provide a computational resources analysis to demonstrate MAGIC-VQA’s exceptional efficiency and scalability. As highlighted in Table 5, our entire framework spans only 0.33 M trainable parameters, vastly leaner than conventional vision–language transformers, knowledge-augmented transformers, multimodal chain-of-thought models, and today’s large VLMs. This dramatic reduction stems from MAGIC-VQA’s plug-and-play architecture: explicit knowledge retrieval and rule-based filtering are parameter-free, while the implicit commonsense module trains only a lightweight GNN. Unlike previous work that must pre-train or fine-tune hefty backbones to absorb external knowledge, our design cleanly decouples knowledge acquisition from model capacity, enabling rapid adaptation to new vision–language encoders with negligible computational overhead.

6 Qualitative Analysis

Figure 6 compares MAGIC-VQA with GPT4o and Qwen2VL, demonstrating how our framework effectively integrates both explicit and implicit commonsense knowledge for enhanced visual question answering. As illustrated in Figure 6a, while GPT4o struggles to deduce the complete answer (big buff ale) to input query, MAGIC-VQA successfully incorporate contextual knowledge, such as “Person X owns the tap sells beer” and “beverage dispenser used as a beer tap in a bar,” linking beer consumption, tap functionality, and beverage machines with the input question to arrive at the correct answer. In Figure 6b, while Qwen2VL incorrectly identifies the colony as North Carolina, our MAGIC-VQA addresses this limitation by integrating explicit image-based commonsense knowledge about Virginia’s location, historical turnpikes, and wildlife, correctly concluding that the answer is Virginia. Confidence scores derived from implicit commonsense knowledge further reinforce the evidence for the final accurate prediction. Additional case studies demonstrating the characteristics of each dataset and the benefit from each commonsense knowledge injection component are shown in Appendix B.

![TextVQA example](page_1012_186_324_246.png)
(a) TextVQA example

![ScienceQA example](page_1346_186_324_246.png)
(b) ScienceQA example

Figure 6: Comparison of results of commonsense knowledge-injected MAGIC-VQA (ours) and original GPT4o-mini and Qwen2VL across different datasets, including TextVQA and ScienceQA. Each example highlights the question-based and image-based explicit commonsense knowledge. Example in ScienceQA is also injected with implicit commonsense knowledge.

7 Conclusion

This paper introduced MAGIC-VQA, a novel framework integrating commonsense knowledge into VQA to address the limitations of existing LVLMs. MAGIC-VQA’s three-stage process—knowledge retrieval, commonsense post-processing, and GNN-based augmentation—enables nuanced reasoning without extensive pre-training or complex prompt tuning. Evaluations on ScienceQA, TextVQA, and MMMU demonstrate significant improvements in tasks requiring advanced reasoning. This framework establishes a robust approach for bridging raw visual inputs with high-level reasoning, offering scalable enhancements for VQA. We hope this work inspires further research into structured commonsense reasoning for complex multimodal challenges.
Limitation

While the MAGIC-VQA framework demonstrates significant improvement, it currently relies on external knowledge graphs, such as ATOMIC2020 and predefined commonsense categories, which may limit its adaptability to diverse and unforeseen domains. Additionally, real-world VQA scenarios often involve noisy or ambiguous inputs that may not always align with the structured assumption of the commonsense knowledge graph. To address these limitations, we plan to extend our approach by developing and incorporating a more diverse and extensive range of multimodal commonsense knowledge sources. Expanding the scope of knowledge representation will enhance multimodal understanding and learning ability and help us handle more multimodal reasoning tasks.

Acknowledgment

This study was supported by funding from the Google Award for Inclusion Research Program (G222897).

References

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433.

Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025a. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv preprint arXiv:2502.12119.

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2024. Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. arXiv preprint arXiv:2412.12359.

Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. 2025b. Cot-kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408.

Annie S Chen, Alec M Lessing, Andy Tang, Govind Chada, Laura Smith, Sergey Levine, and Chelsea Finn. 2024a. Commonsense reasoning for legged robot adaptation with vision-language models. arXiv preprint arXiv:2407.02666.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024b. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Enabling multimodal generation on clip via vision-language knowledge distillation. Preprint, arXiv:2203.06386.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructionlip: Towards general-purpose vision-language models with instruction tuning. Preprint, arXiv:2305.06500.

Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. 2022. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5089–5098.

Google Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2021. Kat: A knowledge augmented transformer for vision-and-language. arXiv preprint arXiv:2112.08614.

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23369–23379.

Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. In AAAI.

Yifan Jiang, Filip Ilievski, Kaixin Ma, and Zhivar Sourati. 2023. BRAINTEASER: Lateral thinking puzzles for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14317–14322, Singapore. Association for Computational Linguistics.

Yifan Jiang, Kexuan Sun, Zhivar Sourati, Kian Aharian, Kaixin Ma, Filip Ilievski, Jay Pujara, et al. 2024. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. Advances in Neural Information Processing Systems, 37:46567–46592.
Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiayang Yu. 2025. Hiddendetec: Detecting jailbreak attacks against large vision-language models via monitoring hidden states. arXiv preprint arXiv:2502.14744.

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700.

Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. Preprint, arXiv:2102.03334.

Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. Preprint, arXiv:1609.02907.

M Lewis. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.

Yijiang Li, Qingying Gao, Haoran Sun, Haiyun Lyu, Dezhi Luo, and Hokin Deng. 2024b. Cogdevelop2k: Reversed cognitive development in multimodal large language models. arXiv preprint arXiv:2410.10855.

Yijiang Li, Qingying Gao, Tianwei Zhao, Haiyun Lyu, Bingyang Wang, Haoran Sun, Dezhi Luo, and Hokin Deng. 2025a. Core knowledge deficits in multi-modal language models. arXiv preprint arXiv:2410.10855.

Yijiang Li, Genpei Zhang, Li Yi, Xiaojun Shan, Dashan Gao, Jiancheng Lyu, Yuan Li, Ning Bi, Jiacheng Cheng, and Vasconcelos Nuno. 2025b. Egoprivacy: What your first-person camera says about you?

Zhenyang Li, Yangyang Guo, Kejie Wang, Xiaolin Chen, Liqiang Nie, and Mohan Kankanhalli. 2023b. Do vision-language transformers exhibit visual commonsense? an empirical study of vcr. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5634–5644.

Weizhe Lin and Bill Byrne. 2022. Retrieval augmented visual question answering with outside knowledge. arXiv preprint arXiv:2210.03809.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.

Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang. 2022a. Vision-and-language pretrained models: A survey. In 31st International Joint Conference on Artificial Intelligence, IJCAI 2022, pages 5530–5537. International Joint Conferences on Artificial Intelligence.

Siqu Long, Soyeon Caren Han, Xiaojun Wan, and Josiah Poon. 2022b. Gradual: Graph-based dual-modal representation for image-text matching. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3459–3468.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521.

Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14111–14121.

Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. 2024. Kamcot: Knowledge augmented multimodal chain-of-thoughts reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18798–18806.

OpenAI. 2024a. Gpt-4 technical report. Preprint, arXiv:2303.08774.

OpenAI. 2024b. Gpt-4o system card. Preprint, arXiv:2410.21276.

Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, et al. 2024. Localized symbolic knowledge distillation for visual commonsense models. Advances in Neural Information Processing Systems, 36.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023. Vlc-bert: Visual question answering with contextualized commonsense knowledge. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1155–1165.

Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–3035.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31.

Alex Su, Haozhe Wang, Weimin Ren, Fangzhen Lin, and Wenhui Chen. 2025. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505.15966.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vi-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530.

Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, and Fangzhen Lin. 2025a. Learning autonomous code integration for math language models. arXiv preprint arXiv:2502.00691.

Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhui Chen. 2025b. Vi-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837.

Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. 2024a. T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19162–19170.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Qunbo Wang, Ruyi Ji, Tianhao Peng, Wenjun Wu, Zechao Li, and Jing Liu. 2024c. Soft knowledge prompt: Help external knowledge become a better teacher to instruct llm in knowledge-based vqa. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6132–6143.

Xiaofeng Wang, Zhixin Zhang, Jinguang Zheng, Yiming Ai, and Rui Wang. 2025c. Debt collection negotiations with large language models: An evaluation system and optimizing decision making with multi-agent. arXiv preprint arXiv:2502.18228.

Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec. 2022. Vqa-gnn: Reasoning with multimodal semantic graph for visual question answering. arXiv preprint arXiv:2205.11501.

Yiting Wang, Tianya Zhao, and Xuyu Wang. 2025d. Fine-grained heartbeat waveform monitoring with rfid: A latent diffusion model. In Proceedings of the 3rd International Workshop on Human-Centered Sensing, Modeling, and Intelligent Systems, pages 86–91.

Yiting Wang, Jiachen Zhong, and Rohan Kumar. 2025e. A systematic review of machine learning applications in infectious disease prediction, diagnosis, and outbreak forecasting.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022. Multi-modal answer validation for knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2712–2721.

Wangyu Wu, Tianhong Dai, Zhenhong Chen, Xiaowei Huang, Fei Ma, and Jimin Xiao. 2025. Generative prompt controlled diffusion for weakly supervised semantic segmentation. Neurocomputing, page 130103.

Wangyu Wu, Tianhong Dai, Xiaowei Huang, Fei Ma, and Jimin Xiao. 2024. Image augmentation with controlled diffusion for weakly-supervised semantic segmentation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6175–6179. IEEE.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. 2024. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872.

Shuo Yang, Siwen Luo, and Soyeon Caren Han. 2025. Multimodal commonsense knowledge distillation for visual question answering. Proceedings of the AAAI conference on artificial intelligence.

S Ye, Y Xie, D Chen, Y Xu, L Yuan, C Zhu, and J Liao. 2022. Improving commonsense in vision-language models via knowledge graph riddles (2022). URL https://arxiv.org/abs/2211.16504. doi, 10.

Shuquan Ye, Yuqia Xie, Dongdong Chen, Yichong Xu, Lu Yuan, Chenguang Zhu, and Jing Liao. 2023. Improving commonsense in vision-language models via knowledge graph riddles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2634–2645.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. Preprint, arXiv:1811.10830.

Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel-Eskin. 2022. Visual commonsense in pretrained unimodal and multimodal models. Preprint, arXiv:2205.01850.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multi-modal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923.

Changshi Zhou, Rong Jiang, Feng Luan, Shaoqiang Meng, Zhipeng Wang, Yanchao Dong, Yanmin Zhou, and Bin He. 2025a. Dual-arm robotic fabric manipulation with quasi-static and dynamic primitives for rapid garment flattening. IEEE/ASME Transactions on Mechatronics, pages 1–11.

Changshi Zhou, Haichuan Xu, Jiarui Hu, Feng Luan, Zhipeng Wang, Yanchao Dong, Yanmin Zhou, and Bin He. 2025b. Ssfold: Learning to fold arbitrary crumpled cloth using graph dynamics from human demonstration. IEEE Transactions on Automation Science and Engineering.

Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang. 2023. Rome: Evaluating pre-trained vision-language models on reasoning beyond visual common sense. arXiv preprint arXiv:2310.19301.
A Additional Experiment Results

A.1 Number of Knowledge Triplets

![Line graphs showing effect of number of knowledge triplets k: Comparison between Qwen2VL and GPT4o-mini.](page_186_153_579_246.png)

Figure 7: Effect of number of knowledge triplets \( k \): Comparison between Qwen2VL and GPT4o-mini.

Inspired by (Wu et al., 2024), to optimize the integration of explicit commonsense knowledge into our model while minimizing the risk of introducing excessive noise, we investigate how varying the number of retrieved commonsense triplets \( k \) impacts performance. We vary \( k \) from 0 to 10, incrementally increasing the number of triplets provided to Qwen2VL and GPT4o-mini and measuring the corresponding accuracy. In Figure 7, the models achieve optimal performance when \( k = 6 \) triplets are incorporated. Utilizing fewer than \( k = 6 \) appears insufficient to provide the contextual information for efficient reasoning, while exceeding \( k = 6 \) triplets includes irrelevant or redundant information, diminishing performance. Therefore, using \( k = 6 \) represents the optimal balance/sweet spot for enriching the model with necessary knowledge while focusing on pertinent information.

A.2 Effect of Similarity Metric

We further conduct an ablation study to evaluate the effect of different retrieval metrics on explicit commonsense knowledge retrieval across different datasets and input sources using GPT4-O model. We explore the performance of three retrieval metrics: Cosine Distance, Manhattan Distance, and Euclidean Distance to understand their influence on retrieval efficacy across these datasets and input types.

As suggested on the right Figure 8, the performance trends reveal notable differences in retrieval effectiveness depending on the metric and dataset. For ScienceQA, Manhattan Distance achieves the highest scores. Meanwhile, the MMMU dataset shows relatively low and uniform scores across all metrics, suggesting that this dataset’s retrieval performance is less sensitive to the choice of metric, potentially due to the diversity and complexity of MMMU’s multimodal inputs.

When comparing retrieval metrics across different input sources, we observe further variations. For question-based retrieval, Manhattan Distance consistently yields higher performance scores, indicating that the absolute differences in feature spaces may be more informative for question-centric retrieval. In contrast, image-based and caption-based retrieval achieves the highest scores with Cosine Distance, suggesting that angle-based similarity is more effective for capturing visual context in the knowledge graph.

![Bar charts comparing accuracy across three datasets (ScienceQA, TextVQA, MMMU) using different distance metrics (Cosine, Manhattan, Euclidean) for explicit commonsense retrieval (Left). The right panel illustrates the performance of these metrics across different input sources (Question, Image, Caption) within the ScienceQA dataset.](page_820_153_579_246.png)

Figure 8: Comparison of accuracy across three datasets (ScienceQA, TextVQA, MMMU) using different distance metrics (Cosine, Manhattan, Euclidean) for explicit commonsense retrieval (Left). The right panel illustrates the performance of these metrics across different input sources (Question, Image, Caption) within the ScienceQA dataset.

A.3 VLPM-Style Fine-Tuning Results

We further perform VLPM-style fine-tuning (Long et al., 2022b; Li et al., 2025b; Wang et al., 2025c) to investigate the effectiveness of implicit multimodal commonsense in MAGIC-VQA framework. Table 6 presents a quantitative analysis of the impact of different input nodes—image (I), question (Q), and generated caption text (C)—on the performance of two baseline models (VILT and ALBEF) in VLPM-style fine-tuning. Each node type corresponds to a specific input modality, and when removed, the original embeddings from the pre-trained models are used instead of GCN-trained node embeddings. It is found that including all nodes consistently yields the best outcomes, underscoring the complementary contributions of visual, textual and caption inputs multimodal reasoning.

Notably, the question node proves to be the most crucial across all nodes. For example, ALBEF’s accuracy on ScienceQA declines from 68.33% to
<table>
  <tr>
    <th>Model</th>
    <th>I</th>
    <th>Q</th>
    <th>C</th>
    <th>SQA</th>
    <th>MMMU</th>
    <th>TextVQA</th>
  </tr>
  <tr>
    <td>ViLT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>56.14</td>
    <td>23.04</td>
    <td>41.49</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ViLT)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>60.53</td>
    <td>20.12</td>
    <td>40.13</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ViLT)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>53.32</td>
    <td>14.28</td>
    <td>32.24</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ViLT)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>63.45</td>
    <td>22.37</td>
    <td>43.98</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ViLT)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>65.41</b></td>
    <td><b>23.35</b></td>
    <td><b>44.12</b></td>
  </tr>
  <tr>
    <td>ALBEF</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>59.12</td>
    <td>25.38</td>
    <td>39.27</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ALBEF)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>61.24</td>
    <td>24.43</td>
    <td>37.28</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ALBEF)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>57.42</td>
    <td>17.21</td>
    <td>28.42</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ALBEF)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>66.79</td>
    <td>26.91</td>
    <td>42.88</td>
  </tr>
  <tr>
    <td>MAGIC-VQA<sub>(ALBEF)</sub></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>68.33</b></td>
    <td><b>27.32</b></td>
    <td><b>43.25</b></td>
  </tr>
</table>

Table 6: Combined Results with Multimodal Contributions. The green checkmarks (✓) denote the inclusion of a component, while the red crosses (✗) denote its exclusion.

![Bar charts comparing Model Performance on VCR and A-OKVQA](page_184_682_495_312.png)

Figure 9: Further Experiment results on VCR and A-OKVQA

57.42% when the question node is excluded, highlighting its essential role in guiding the model’s attention toward relevant aspects of the image and improving reasoning. On the other hand, the image node also plays a significant role in performance. Visual inputs provide critical scene-level information, enabling models to capture object attributes and spatial relationships, which cannot be fully compensated by text-based inputs alone.

A.4 Additional Results on Knowledge Intensive Benchmarks

We further attach the result of MAGIC-VQA on two knowledge-intensive tasks including VCR and A-OKVQA as a further complementary analysis.

B Additional Case Studies

We summarize different qualitative analysis case studies in three types: 1) The cases when the image-based explicit commonsense knowledge plays an essential role (Figure 10), 2) The cases when implicit commonsense-based confidence plays an essential role (Figure 11), and 3) the cases when by-type commonsense knowledge post-processing plays an important role (Figure 12).

![Visualisations of MAGIC-VQA results on MMMU and ScienceQA datasets](page_1012_682_495_312.png)

Figure 10: Visualisations of MAGIC-VQA results on the MMMU and ScienceQA datasets, showcasing the role of image-based commonsense knowledge in deriving correct answers. This highlights the cases when image-based commonsense knowledge is more influential in finding the answer.

![Visualisation of MAGIC-VQA results on MMMU datasets](page_1012_1012_495_312.png)

Figure 11: Visualisation of MAGIC-VQA results on MMMU datasets. This highlights the cases when implicit commonsense-based confidence plays an essential role.

C Commonsense Relation Transformation Table

We use "Someone", "Someone's" to replace the "PersonX", "PersonX's" and "Another", "Another one's" to replace "PersonY", "PersonY's" sepa-
(a) The ScienceQA case when CS-PE is influential

(b) The ScienceQA case when CS-SI is influential

(c) The TextVQA case when CS-PE is influential

Figure 12: Visualisation of MAGIC-VQA results on ScienceQA and TextVQA datasets. This highlights the cases when by-type commonsense knowledge post-processing plays an important role

rately, in the heads and tails of the Atomic2020 triplets to enhance clarity and coherence in commonsense-grounded inference.

D Baselines

• LLaVA-1.6 (LLaVA-Next) (Li et al., 2024a): is an open-source Large Multimodal Model (LMM) designed for enhanced visual and conversational understanding built upon LLaVA (Liu et al., 2024b). It supports higher input resolutions (up to 672x672 pixels) for finer visual detail recognition and incorporates improved visual instruction tuning for better reasoning and OCR capabilities. LLaVA 1.6 is highly efficient, using fewer than 1 million visual instruction tuning samples and a streamlined training process. Its versatility enables it to handle a wide range of applications, from image and text-based tasks to complex multimodal interactions, all while maintaining a minimalist and data-efficient design. We use llava-v1.6-mistral-7b-hf checkpoints for zero-shot testing.

<table>
  <tr>
    <th>Relation</th>
    <th>Transformed Format</th>
  </tr>
  <tr>
    <td colspan="2"><b>Physical-Entity</b></td>
  </tr>
  <tr>
    <td>ObjectUse</td>
    <td>is used for</td>
  </tr>
  <tr>
    <td>AtLocation</td>
    <td>is at</td>
  </tr>
  <tr>
    <td>MadeUpOf</td>
    <td>is made up of</td>
  </tr>
  <tr>
    <td>HasProperty</td>
    <td>can be</td>
  </tr>
  <tr>
    <td>CapableOf</td>
    <td>is capable of</td>
  </tr>
  <tr>
    <td>Desires</td>
    <td>desires</td>
  </tr>
  <tr>
    <td>NotDesires</td>
    <td>does not desire</td>
  </tr>
  <tr>
    <td colspan="2"><b>Event-Centered</b></td>
  </tr>
  <tr>
    <td>IsAfter</td>
    <td>occurs after</td>
  </tr>
  <tr>
    <td>HasSubEvent</td>
    <td>has sub-event</td>
  </tr>
  <tr>
    <td>IsBefore</td>
    <td>occurs before</td>
  </tr>
  <tr>
    <td>HinderedBy</td>
    <td>is hindered by</td>
  </tr>
  <tr>
    <td>Causes</td>
    <td>causes</td>
  </tr>
  <tr>
    <td>xReason</td>
    <td>is because someone</td>
  </tr>
  <tr>
    <td>isFilledBy</td>
    <td>is filled by</td>
  </tr>
  <tr>
    <td colspan="2"><b>Social-Interaction</b></td>
  </tr>
  <tr>
    <td>xNeed</td>
    <td>then someone needs</td>
  </tr>
  <tr>
    <td>xAttr</td>
    <td>then someone has attributes</td>
  </tr>
  <tr>
    <td>xEffect</td>
    <td>then someone has the effect</td>
  </tr>
  <tr>
    <td>xReact</td>
    <td>then someone reacts with</td>
  </tr>
  <tr>
    <td>xWant</td>
    <td>then someone wants</td>
  </tr>
  <tr>
    <td>xIntent</td>
    <td>then someone intends</td>
  </tr>
  <tr>
    <td>oEffect</td>
    <td>then the effect on another is</td>
  </tr>
  <tr>
    <td>oReact</td>
    <td>then another reacts with</td>
  </tr>
  <tr>
    <td>oWant</td>
    <td>then another one wants</td>
  </tr>
</table>

Table 7: Transformation template of commonsense relations into natural language phrases

• InternVL2 (Chen et al., 2024b): is a state-of-the-art multimodal large model developed by OpenGVLab. It integrates image, video, text, speech, and 3D data, supporting over 100 tasks with exceptional performance across benchmarks. InternVL2 leverages a progressive alignment training strategy and have achieved outstanding results in complex multimodal understanding tasks, rivaling leading commercial closed-source models like GPT-4V (OpenAI, 2024a). It introduces innovations like vector linking for diverse outputs. It also has parameter sizes ranging from 1B to 76B optimized for efficiency, which delivers high performance even on limited resources. We use InternVL2-8B-hf checkpoints for zero-shot testing.

• xGen-MM(BLIP-3) (Xue et al., 2024): is a cutting-edge framework for Large Multimodal Models (LMMs) developed by Salesforce AI Research. It features a modular architecture with a scalable vision token sampler and a pre-trained language model, optimized for diverse multimodal tasks such as image captioning, visual question answering, and OCR.
It simplifies training objectives with a unified auto-regressive loss and incorporates post-training techniques like Direct Preference Optimization (DPO) and safety fine-tuning to improve truthfulness and mitigate harmful behaviors. We use xgen-mm-phi3-mini-instruct-r-v1 checkpoints for zero-shot testing.

• Qwen2-VL (Wang et al., 2024b): is a cutting-edge vision-language model designed with a robust technical architecture to process multi-modal inputs efficiently. It integrates a 675M-parameter Vision Transformer (ViT) enhanced with a Naive Dynamic Resolution mechanism, enabling adaptive encoding of images and videos into variable-length visual tokens to capture detail at multiple scales. To align spatial and temporal information, the model employs Multimodal Rotary Position Embedding (M-RoPE), decomposing positional information into temporal, height, and width dimensions. It also leverages dynamic sequence lengths and efficient parallelism techniques, allowing for deployment in sizes of 2B, 7B, and 72B parameters. We use Qwen2-VL-7B-Instruct checkpoints for zero-shot testing.

• GPT-4o (OpenAI, 2024b): is an advanced autoregressive model that processes and generates multimodal content, including text, images, audio, and video, using a unified neural network architecture. It offers significant enhancements in vision and audio understanding, multilingual text generation, and operational efficiency. The model’s training incorporates diverse public and proprietary datasets across modalities, with rigorous post-training alignment to ensure safety and mitigate risks such as bias, misinformation, and unauthorized content generation. We use gpt-4o-2024-08-06 checkpoints for zero-shot testing.

E Implementation Details

We set \( K = 30 \) to retrieve explicit commonsense knowledge. For by-type commonsense knowledge processing, we configure \( \varepsilon = 0.1 \) and \( k = 6 \) to effectively integrate rich commonsense knowledge while minimizing the introduction of excessive noise. For implicit commonsense confidence augmentation, we follow the default setup in Kipf and Welling (2017) to explore a standard two-layer GCN. The dimension of the hidden size is set to be 256 and 512, each followed by a dropout layer with the rate to be 0.4. To train the teacher model, we explore batch size to be 64, learning rate to be 1e-5 and epoch to be 30 with early stopping for all models and datasets. There is a global average pooling layer and a output layer using the softmax function after the last GCN layer.

All experiments are conducted on a workstation equipped with one A100 GPU with 40 GB of VRAM. We utilize PyTorch 1.10.0 for model training and the HuggingFace Transformers library for accessing pre-trained models. Our code was written in Python 3.8, and CUDA 11.2 was used for GPU acceleration.

F Concrete Example of Input Prompt

We further include a concrete prompt example of our MAGIC-VQA framework. as demonstrated in in Table 8, the input sample is augmented with both explicit and implicit commonsense knowledge providing the background information. We ask the model to first generate the rational then answer the question.

G Commonsense Category Analysis Prompt Format

To analyze the commonsense knowledge distribution within each selected dataset, we provide the following prompt template to classify each sample to their most relevant commonsense knowledge (CS-PE, CS-EC, CS-SI) covered in Table 9.

H Retrieved Commonsense Triplet Cosine Similarity Distribution

Figure 13 depicts the cosine similarity distribution of the retrieved triplets of each input source across all three datasets.
Background
You are an advanced Vision-Language Model assistant designed to answer multiple-choice questions based on a given image. Your task is to select the most appropriate option from the provided answer choices. You are given an input image, a question related to the image, the image caption, multiple-choice answer options, and both explicit and implicit commonsense knowledge.

Explicit commonsense knowledge consists of statements related to the input, categorized as image-related commonsense, question-related commonsense, caption-related commonsense. Each statements is related to a relevance level including highly relevant, relevant, less relevant. Implicit commonsense knowledge is the confidence score of each candidate option, higher values indicate a greater likelihood of being correct.

Your objective is to integrate the explicit and implicit commonsense knowledge with the provided information to generate a step-by-step reasoning. Based on this rationale, you will select the most appropriate answer from the given options.

Input Information

Image:

![A world map with labeled continents and oceans](page_324_563_377_181.png)

Question: Which of these oceans does the prime meridian intersect?
Caption: An image of a world map with labeled continents and oceans.
Options:
  A. "the Atlantic Ocean"
  B. "the Indian Ocean"
  C. "the Pacific Ocean"

Explicit Commonsense Knowledge

Image-Related Commonsense:
  - The Atlantic Ocean is at the western hemisphere. (Highly Relevant)
  - A world traveler is capable of crossing many time zones. (Relevant)
Question-Related Commonsense:
  - A traveler is capable of crossing geographical borders. (Highly Relevant)
  - Someone who is far from home might want to measure the distance. (Less Relevant)
Caption-Related Commonsense:
  - The Atlantic Ocean is used for separating continents. (Highly Relevant)
  - If someone sees the ocean, they might think of traveling to it. (Relevant)

Implicit Commonsense Knowledge (Confidence for Each Option)
  A: 0.6
  B: 0.05
  C: 0.35

Rationale:

Answer:
<table>
  <tr>
    <th>Prompt Template for Commonsense Category Classification</th>
  </tr>
  <tr>
    <td><b>Instructions:</b><br>You are an expert in commonsense reasoning and knowledge representation. Your task is to classify each sample into one of three commonsense categories:<br>1. <b>Physical-Entity Commonsense (CS-PE):</b> Knowledge about physical objects, their properties, uses, locations, and physical attributes. This includes understanding what things are made of, typical or atypical uses, and physical characteristics.<br>2. <b>Event-Centered Commonsense (CS-EC):</b> Knowledge about events, including their causes, effects, prerequisites, sequences, and hindrances. This encompasses understanding how events are related in time and causality.<br>3. <b>Social-Interaction Commonsense (CS-SI):</b> Knowledge about social behaviors, mental states, interactions, and interpersonal dynamics. This involves understanding intentions, emotional reactions, and attributes in social contexts.<br><b>Sample:</b><ul><li><b>Image:</b> <i>&lt; ImageCaption &gt;</i></li><li><b>Question:</b> <i>&lt; Question &gt;</i></li><li><b>Choices:</b> <i>&lt; Options &gt;</i></li><li><b>Answer:</b> <i>&lt; Answer &gt;</i></li></ul><b>Reasoning Steps:</b><br>Please first examine the question and answer choices, along with the image caption, to identify the main focus of the sample. Then provide a step-by-step reasoning on how specific elements of the sample align with the potential commonsense category. Then assign the appropriate commonsense category (CS-PE, CS-EC, or CS-SI) based on the provided rationale.<br><b>Classification:</b></td>
  </tr>
</table>

Table 9: Prompt Template for Classifying Samples into Commonsense Categories
(a) ScienceQA

(b) TextVQA

(c) MMMU

Figure 13: Overall cosine similarity distributions for three input sources within each dataset. The first column represents the cosine similarity distribution of retrieved triplets for input question. The second column represents the cosine similarity distribution of retrieved triplets for input image. The third column represents the cosine similarity distribution of retrieved triplets for input caption.

Reviewer paper summaries: This paper proposes MAGIC-VQA, a framework to improve VQA performance of VLMs by integrating commonsense knowledge with VLMS.
The framework uses a 3 stage pipeline: explicit knowledge integration, contextual refinement, and implicit knowledge augmentation using graph neural networks. Evaluations on multiple VQA benchmarks shows that MAGIC-VQA significantly improves commonsense reasoning and achieves improvement in performance.

The paper proposes MAGIC-VQA, a novel framework for enhancing Visual Question Answering (VQA) by integrating commonsense knowledge with Large Vision-Language Models (LVLMs). The central hypothesis is that although LVLMs perform well in object recognition and basic semantic understanding, they often fall short in reasoning tasks that require implicit or contextual commonsense understanding.

To address this, MAGIC-VQA introduces a three-stage pipeline:

Explicit Commonsense Knowledge Integration: External commonsense knowledge is retrieved from the ATOMIC2020 knowledge graph based on contextual similarity to the image, question, and generated caption. Relevant triplets are selected to provide external reasoning support.

By-Type Post-Processing: The retrieved knowledge is filtered and balanced across three categories—Physical Entity (CS-PE), Event-Centered (CS-EC), and Social Interaction (CS-SI)—based on the dataset's needs. Relevance levels (High, Medium, Low) are also assigned to each triplet using statistical thresholds to guide reasoning.

Implicit Knowledge Augmentation via Graph Neural Networks (GNNs): A multimodal graph is constructed by linking image, question, caption, and commonsense nodes. A GNN then refines this structure to compute confidence scores, allowing deeper structured reasoning.

The authors evaluate MAGIC-VQA across three diverse VQA datasets—ScienceQA, TextVQA, and MMMU—each requiring different types of reasoning. Results show that MAGIC-VQA consistently improves performance across all datasets and baselines, including both open-source (e.g., LLaVA, BLIP3, InternVL2, Qwen2VL) and proprietary models (GPT-4o-mini). Notably, the integration of both explicit and implicit commonsense knowledge leads to state-of-the-art results, particularly in complex scenarios requiring grounded reasoning.

Extensive ablation studies demonstrate the importance of:

Tailoring commonsense knowledge to dataset-specific reasoning needs.

Balancing the number and type of knowledge triplets.

The synergistic benefit of combining both explicit and implicit knowledge over using either alone.

The paper emphasizes that MAGIC-VQA avoids heavy model re-training or complex prompt engineering, offering a scalable, plug-in approach to improve VQA performance by bridging perceptual inputs and abstract reasoning.

This work contributes meaningfully to the VQA and multimodal reasoning community by addressing a key limitation of current LVLMs—their lack of integrated commonsense reasoning capabilities—and presents a practical, interpretable, and modular solution.
Summary of strengths: - Results indicate that incorporating commonsense knowledge leads to a substantial improvement over the base model.
- Training a GNN-based component for confidence score estimation is an interesting and promising approach.
- The qualitative results highlight MAGICVQA's ability to leverage contextual knowledge effectively, enabling it to produce correct answers in cases where standalone LVLMs fall short.

1. Novel and Modular Framework for Commonsense-Augmented VQA
The paper introduces MAGIC-VQA, a well-motivated and technically innovative framework that unifies both explicit and implicit commonsense knowledge integration within a VQA pipeline. Unlike prior work that either uses static prompt tuning or simplistic retrieval-based methods, MAGIC-VQA proposes a three-stage pipeline—explicit retrieval, by-type filtering, and graph-based reasoning—offering a scalable and modular approach that can be adopted by a wide range of LVLMs.

2. Strong Empirical Performance Across Diverse Datasets
The proposed approach demonstrates state-of-the-art results on multiple challenging VQA benchmarks—ScienceQA, TextVQA, and MMMU—which are diverse in domain and reasoning complexity. The framework significantly outperforms strong baselines (LLaVA, BLIP3, Qwen2VL, GPT-4o-mini) under zero-shot settings, showing consistent gains from the integration of both explicit and implicit commonsense knowledge.

3. Insightful Commonsense Type Analysis and Dataset Alignment
The paper goes beyond accuracy reporting by conducting a fine-grained analysis of the types of commonsense knowledge (Physical Entity, Event-Centered, Social Interaction) required by each dataset. This thoughtful alignment between dataset characteristics and knowledge filtering strategies adds substantive insight into how VQA models can be tailored for specific reasoning needs.
Summary of weaknesses: - The approach follows a common pipeline of generate commonsense → filter → integrate dynamically with implicit knowledge, which has been explored in prior works (VLC-BERT, KAT and many other works in NLP literature). The framework appears quite similar to existing methods like VLC-BERT, with the key difference being the replacement of **VL-BERT** with **LVLMs**. This makes the GNN-based selection the primary novel component, but its necessity is unclear—alternative approaches, such as using an LLM for triplet ranking or other selection methods are not compared.
- Comparison with alternative knowledge integration methods is **missing**. Specifically, multiple baselines from Section 2.2 on external knowledge integration are missing, which would provide a clearer perspective on whether this particular pipeline adds more value.
- The use of ATOMIC as a knowledge source is a reasonable choice, but it is a static knowledge base. Prior work in NLP and vision has explored dynamic alternatives like COMET or a larger LLM, which could offer a more adaptive integration of commonsense knowledge.
- The selected benchmarks are not optimal for evaluating commonsense reasoning. For instance, TextVQA is not particularly commonsense-heavy compared to datasets like VCR or A-OKVQA, which could provide a more rigorous assessment of the model’s ability to leverage commonsense knowledge. Furthermore, on what basis, the benchmarks were picked from the 3 selected categories is unclear.

While the qualitative results are strong, the paper lacks a deep analysis of failure cases or conditions under which MAGIC-VQA may struggle (e.g., noisy OCR, ambiguous images, or questions requiring complex symbolic reasoning beyond commonsense). A more critical evaluation of model failure would help clarify the method's boundaries.
Comments, suggestions and typos: - Why was atomic used instead of COMET?
- Though the performance improvement from adding commonsense seems significant, I am not convinced that this particular selection and integration strategy caused this improvement. How would this fair against a simple reimplementation of methods like VLC-BERT which uses COMET or KAT which uses GPT3 as knowledge source? I think those methods simply differ in the fact that they are integrated with VL-transformer architectures instead of VLMs. 
- Writing comments: Consider including the benchmarks and average improvements achieved in the intro/abstract. Consider adding a motivating example for why commonsense knowledge is lacking in the intro.

While the by-type post-processing is helpful, future work could explore learned filtering mechanisms rather than rule-based thresholds. A discussion of this potential direction could enhance the completeness of your analysis.
Overall assessment scores: 3

3.5
Metareview score: 3
Metareview: This paper presents MAGIC-VQA, a VQA framework that integrates common-sense knowledge with LVLMs. MAGIC-VQA integrates explicit knowledge from external sources, conducts post-processing with contextual refinement, and conducts knowledge augmentation via a GNN. MAGIC-VQA achieves SOTA on ScienceQA, TextVQA and MMMU (and, during the rebuttal phase, on OK-VQA and A-OKVQA as well).
Summary of reasons to publish: Reviewers noted that MAGIC-VQA is well-motivated, innovative, and offers a principled, scalable and modular approach to improve common-sense reasoning without retraining large models. The evaluation and empirical results are comprehensive and convincing, and the authors conduct a fine-grained analysis of results based on common-sense knowledge types. The modular design also introduces an element of transparency and reproducibility, and the by-type post-processing and GNN approach allows adaptability across various VQA tasks and datasets.

Many reviewers also adjusted their scores upward during the discussion period, indicating the authors' responses and additional clarifications were convincing.