SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Authors: Xin Su
Man Luo
Kris W Pan
Tien Pei Chou
Vasudev Lal
Phillip Howard
Venue: ICML 2025 oral (2025)
Keywords: Multimodal
retrieval augmented generation
data generation
Primary area: applications->everything_else
TL;DR: We introduce SKVQA, a large-scale synthetic multimodal dataset, for improving context-aware generation capability of multimodal.

Abstract: Multimodal retrieval-augmented generation (RAG) plays a crucial role in domains such as knowledge-based visual question answering (KB-VQA), where models should effectively integrate additional knowledge to generate a response. However, existing vision and language models (VLMs) are not inherently designed for context-augmented generation, limiting their effectiveness in such tasks. While synthetic data generation has recently gained attention for training large VLMs, its application for context-augmented generation remains underexplored. To address this gap, we introduce SKVQA, a large-scale synthetic multimodal dataset containing over 2 million visual question-answer pairs, each associated with external knowledge sources to determine the final answer. Compared to previous datasets, SKVQA exhibits 11× more unique questions, greater domain diversity, and a broader spectrum of image sources. Through human evaluations, we confirm the high quality of the generated question-answer pairs and their contextual relevance. Extensive experiments show that SKVQA serves both as a challenging benchmark for knowledge-based VQA and as an effective training resource for adapting generative multimodal models to context-augmented generation. Our results further indicate that models trained on SKVQA demonstrate enhanced generalization in both context-aware VQA and multimodal RAG settings.

SK-VQA:
Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Xin Su Man Luo"
Kris W Pan
Tien Pei Chou
Vasudev Lal Phillip Howard
Abstract
Multimodal retrieval augmented generation (RAG) plays a crucial role in domains such as knowledge-based visual question answering (KB- VQA), where external knowledge is needed question. However, existing multimodal LLMs (MLLMs) are not designed for context- augmented generation, limiting their effective- ness in such tasks. While synthetic data gener- ation has recently gained attention for training MLLMs, its application for context-augmented generation remains underexplored. To address this gap, we introduce SK- SK-VAA, a large-scale scale syn- thetic multimodal dataset containing over 2 mil- lion visual question-answer pairs, each associated with context documents containing information necessary to determine the final answer. Com- pared to previous datasets, SK-VQA VQA contains 11x more unique questions, exhibits greater domain diversity, and covers a broader spectrum of image sources. Through human evaluations, we confirm the high quality of the generated question-answer pairs and their contextual relevance. Extensive experiments show that SK-VQA serves both as a challenging XB-VQA benchmark and as an ef- fective training resource for adapting MLLMs to context-auge augmeetd generation. Our results further indicate that models trained on SK-VQA demon- strate enhanced generalization in both context- aware VQA and multimodal RAG settings. SK- VQA is publicly available via Hugging Face Hub.
Conteat: The Beverly Hilton s kaurry hotel located in Beverly Hills, Califoria, Opened in 1955, the hotel was designed architect Welton Becket and owned the West
prestigious hosted Hilton features more than guest rooms, Including suites, and variety
history,
annual
Beverly
building
dwns building in image?
Dasi West Realty, Ltd
What type of events the location the image known for hosting annually?
Golden dlobe Awards
available building shown image?
Figure f. Example from SK-VQA. Each image is paired with synthetically-genentt context document & QA pairs.
domain, enabling advanced reasoning and chat capabilitiess multimodal input queries consisting of both text and im- ages (Achiam et al., 2023; Liu et al., 2024b). While MLLMs have demonstrated promising results, they suffer from the same hallucination and reliability issues as LLMs (Li et al., 2023; Zhou et al., 2023; Liu et al., 2024c). This motivates the need to incorporate MLLMs into retrieval- -uggmente generation (RAG) systems. where retrieved documents can ground answer generation in factually-corres information via context augmentation (Lewis al., 2020; Ram al., 2023). However, context augmentation for MLLMs presents unique challenges. Generated answers must be conditioned on both multimodal input queries as well as retrieved con- texts which potentially span multiple modalities. Existing MLLMs have not been trained with context- auggeete gen- eration, which makes them incompatible for use in RAG system. Adapting MILMs for in RAG system requires extensive datasets which can support the training of models with multimodal queries and relevant context documents. Unfortunately,
1. Introduction
Recent advances in Multimodal LLMs (MLLMs) have tended the impressive capabilities of LLMs to the vision
Equal contribution 'Intel Labs AAmazon Thoughtworks.. Correspondence Xin Su Exin.su@intel.com. Man Luo <man.luo@ cotel.com>.
data of this kind rel- atively scarce; unlike other common types of internet data (e.g., text, image-text pairs), input queries consisting of both
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Context & QA Sources
Image Sources
Context: Victorian fashion refers With the rise of fashion magazines, the spread of trendy styles and the latest fashion news became more widespread, significantly influencing public taste and clothing choices.
LAION 9
What type of publication most likely influenced the outfit in image?
LATON
A: Cashion mapazines
Context: Handball popular and widely played sport Brazil... The women's notably the World Championship in 2013, which marked ....
GPT-4
What specific achievement has the women's team of the country the accomplished?
WiMpodla
A: World Championshhi
Context: The Labrador Retriever is one of the most popular and versatile dog breeds in the world. dense, water recistant coat that helps protect them cold water. Labradors medium to large-sized dogs,..
COCO-
". They
Counter
factuals
Q: What characteristic helps this breed
cold water?
A: Weteerresistaat coot
Wikipedia
Stupino (Russian: Crýmeeо) is town and the administrative center of Stupinsky District in Moscow Oblast, Russia,
Context:
the River, 99 kilometers [62 south Moscow. Population: 66,816 (2010 Census);
Wikkpedia
6
On which river place located?
A: Oka River
Figure 2. Examples of QA pairs in our dataset created for images and contexts collected from different sources. Unlike existing resources which are limited to images that can be linked to Wikipedia passages, our dataset contain images from arbitrary sources by leveraging muuthetically-generaae context documents in addition to context documents sourced from Wikipedia.
images and text with associated context documents are not readily available at the scale needed for training MLLMs. This motivates us to develop a synthetic generation pipeline to create a large-scale, diverse dataset that facilitates the advancement of context-augmented MLLMs.
which utilizes strong foundation model (GPT-4) to produce relevant context documents and multiple QA pairs for a given image (Figure 1). This enables the acquisition of data which spans diverse sources of images (Figure 2), even allowing for the generation of fully-synthetic data which includes synthetic images, contexts, and QA pairs.
Synthetically generated data has recently grown in popu- larity as a solution for cases where sources of naturally- occurring data are scarce or have been exhausted in exist- ing training datasets. In the context of training MLLMs, synthetic data has played an important role in the visual instruction tuning required to develop such models (Liu et al., 2024b). A limited number of knowledge-baadd based visual question answering (KB-VQA) datasets suitable for training MLLMs in context-augmentee setting have been con- structed by synthetically producing question-answer (QA) pairs for real images and related text documents (Chen et al., 2023; Lerner et al., 2022; Mensink et al., 2023). However, these existing resources are mostly limited to images which can be linked to context documents sourced from Wikipedia, focus entity-specific knowledge, and lack question diver- sity due to their reliance on templates for constructing QA pairs. As we will demonstrate in the experiments ($5), train- ing a model on such datasets leads to poor generalization.
We show that SK-VQA is much more diverse than other KB- VQA datasets such as ViQuAE (Lerner et al., 2022), InfoS- eek (Chen et al., 2023), and Encyclopedic- -VQA (Mensink et al., 2023). Unlike these existing resources, SK-VQA uti- lizes images from a larger variety of sources because our generation methodology does not rely on linking images to Wikipedia pages in order obtain context documents. To demonstrate the utility of SK-VQA we first conduct zero- shot experiments on six state-of of-the-rrtttttto MLLMs, showing that it is a challenging benchmark for these powerful models. We then fine-tune MLLMS on our dataset and compare their performance to models fine-tuned on existing KB-VQA datasets. Our experiments show that SK-VQA enhances the generalization capabilities of MILMs, whereas other datasets result in poor generalization performance. We at- tribute this improved generalization capacity to the diversity of our dataset. To summarize, our contribution are threefold:
To address these deficiencies, we construct SK-V BK-VQA the largest KB-VQA dataset to-date, containing over 2 million QA pairs associated with synthetic context knowledge and images sourced from LAION (Schuhmann et al., 2021), WIT (Wikipedia images) (Srinivasan et al., 2021), and the synthetic COCO-Counterfactii
We create the largest and most diverse multimodal dataset for KB-VQA to-date. Unlike existing datasets which are limited to images that can be linked to Wikipedia pas- sages, our dataset encompasses broader range of images, features a more diverse set of question types, and pos- sesses richer linguistic style. Additionally, human evalua- tions affirm the correctness of the generated QA pairs and the factuality of the generated context documents. Our
dataset (I.e et al., 2024). Rather than relying on template-based methods to construct QA pairs for real data, we construct SK-VQA using a fully automated synthetic multimodal data generation approach
2

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
'ataset' and its generation code² are publicly available. We perform zero-shot evaluations and fine-tuning of sev- eral state-of-the-art MLLMs on both our dataset and ex- isting datasets. The zero-shot results indicate that our dataset is more challenging compared to others, demon- strating its complexity despite being synthetically pro- duced. In addition, the fine-tuning results demonstrate that our dataset consistently improves the out-of-domain performance across model sizes on multiple datasets. We conduct detailed ablation experiments to evaluate the performance of fine-tuned models across different image types and test their performance within a real-world RAG environment. The experimental results indicate that our dataset effectively enhances the model's out-of-domain performance.
Knowledge-Basse Visual Question Answering Datasets Marino et al. (2019) introduced OK-VQA, KB-VQA dataset of 14k crowdsourced questions for COCO images which are designed to require external knowledge to an- swer, but are not associated with ground truth context docu- ments. Lerner et al. (2022) introduced the ViQuAE dataset, which consists of 3.7k questions about named entities paired with images and text articles from Wikipedia. Chen et al. (2023) found that many questions in OK-VQA and ViQuAE can be answered without external knowledge; motivated by this finding, they introduced the InfoSeek dataset contain- ing over 1.3 million information-sscking questions paired with images from existing image classification and retrieval datasets which have been grounded to Wikipedia articles. Although they curate a smaller set of 8.9k wwrtten questions for testing, the vast majority of InfoSeek is auto- matically constructed by populating human- authored tem- plates from Wikidata triples.
2. Related Work
Synthetic Data Generation Synthetic data has grown in popularity lately as an effective strategy for data augmenta- tion, particularly in the multimodal domain where data is often more scarce. Advances in text-to-image diffusion mod- els (Nichol al., 2021; Rombach et al., 2021; Saharia et al., 2022; Ramesh et al., 2022) have enabled the generation of synthetic data for variety of use cases such as image clas- sification (He et al., 2022; Trabucco et al., 2023; Vendrow et al., 2023) and image-text counterfactuals (Le et al., 2024; Howard et al., 2024). In the domain of NLP, augmenting prompts with LLM-generated context documents has been demonstrated be competitive with retrieving real text doc- uments for context augmentation in RAG systems (Yu et al., 2022). Synthetic data has also been shown to be useful for training text embedding models for retrieval (Wang et al., 2023). To the best of our knowledge, our work is the first to explore the use of fully synthetic datasets to adapt MLLMS for context-augmented generation.
Encyclopedic VQA (Mensink et al., 2023) is another excently-proposee KB- KB-VQA dataset consisting of 221k unique QA pairs which are each associated with up to 5 im- ages from the iNaturalist (Van Horn al., 2021) and Google Landmarks (Weyand al., 2020) datasets. They utilize the WIT dataset (Srinivasan et al., 2021) to link images with Wikipedia text documents and employ templates along with a question generation model to automatically construct I million question- -nnwwer pairs. SnapNTell (Qiu et al., 2024) also contains XBA questions requiring entity-specific external knowledge to answer, but contains fewer QA pairs (75.6k (6).k) and was not publicly available at the time of writing. Other knowledge-intensiv VQA datasets have been pro- posed for more specific domains of multimodal documents, including technical engineering requirements (Doris et al., 2024) and scientific journal articles (Ding et al., 2024).
Multimodal RAG systems In the domain of KB- KB-VQA, augmenting transformer-based generators with retrieved multimodal documents has been shown to be effective in architectures such as RA-CM3 (Yasunaga et al., 2022), MuRAG (Chen et al., 2022), and REVEAL (Hu et al., 2023). More recently, LLMs augmented with vision encoders such as LLaVA (Liu et al., 2024b) and GPT-4 (Achiam et al., 2023) have demonstrated state-of-the-a performance on image-to-text generation tasks, motivating the investiga- tion of retrieval-based based context augmentation for such mod- els. Re-V (Yang et al., 2023) augments Flamingo with retrieved documents, while Wiki-LLaVA (Caffagni et al., 2024) augments LLaVA model with Wikipedia documents.
Despite its demonstrated benefits, several risks have been noted in utilizing synthetic data for training. Shumailov et al. (2023) showed that training language models on data that is contaminated with increasing amounts of model-generated content leads to model collapse, while Gerstgrasser et al. (2024) found that accumulating -ener-eene content without replacing original content can avoid this. the context of training vasion-language models, synthetic image data has been shown to scale similarly in effectiveness of CLIP (Radford et al.. 2021) training as real images, while significantly under performing real data in training super- vised image classifiers (Fan et al., 2023). Improving out-of- domain generalization by training on synthetic data has also been shown to be sensitive to the ratio of real and synthetic data (Howard et al., 2022; Le et al., 2024).
Wei et al. (2023) proposed UniTR for multimodal retrieval, utilizing score-level and fcature-level fusion approaches with pre-trained CLIP (Radford et al., 2021) and BLIP (Li et al., 2022) models. Sharifymoghaddam et al. (2024) showed that UnilR can improve the performance of large
Our dataset is available via Hugging Face Hub Our code is available via GitHub
3

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
multimodal language models on image captioning and im- age generation tasks. UniMur (Wang et al., 2024b) embeds multimodal inputs and retrieves multimodal outputs via frozen LLMs. Our work differs from these studies in that we focus on how to adapt MLLMs for context-augmented generation in a RAG system rather than training the retriever.
Write a Wikipedia article related to this image without directly referring to the image. Then write question answer pairs. The question answer pairs should satisfy the following criteria.
1: The question should refer to the image.
2: The question should avoid mentioning the name of the object in the image.
3. Methodology
3: The question should be answered by reasoning over the Wikipedia article.
3.1. Dataset generation
4: The question should sound natural and concise.
Motivated by recent advances in MLLMs,, we use fully au- tomated approach to generate synthetic context documents and question-ansee answer (QA) pairs for a given image with GPT-4. This provides several distinct advantages. First, the powerful language abilities of GPT-4 allow us to acquire more natural and diverse questions than previous datasets which rely on templated construction of question-ann ansser pairs. Second, generating context documents enables the use of much broader range of images than previous datasets, where images are typically restricted only to those that can be linked to Wikipedia passages.
5: The answer should be extracted from the wikipedia article.
6: The answer should not be any objects in the image.
7: The answer should be single word phrase and list all correct answers separated by commas.
8: The answer should not contain 'and', 'or', , rather you can split them into multiple answers.
Figure 3. Our prompt for generating synthetic data.
Given an input image, we prompt GPT-4 to generate a con- text document related to the image and QA pairs which require reasoning over both the image and the context doc- ument. The complete prompt we for this purpose is provided in Figure 3 (see Figure 7 of Appendix H for addi- tional discussion). Importantly, we generate both the context document and QA pairs in a single inference step. In do- ing so, the generation of the context is conditioned on the task of producing questions that require both the image and the context. This helps ensure that the context associated with each image is suitable for the creation of the style of QA pairs we seek, which is not necessarily the case when context documents are acquired automatically from existing sources such as Wikipedia. Following generation, we parse the output of GPT-4 to extract the context document and QA pairs (see Appendix H for details). We then apply two stages of filtering to create separate filtered subsets.
of image-specific context documents. We therefore create a filtered subset of our dataset which excludes these cases by identifying the presence of the words pi cture, photoo, image, paint ing in the generated context document. We refer to this subset as SK-VQAIR-
3.3. Context Answer Presence (CAP) filtering
In existing datasets for KB-VQA, it is common for the answer to be explicitly stated in the associated context docu- ment. This is not necessarily required in order for a QA pair to be valid since the answer could sometimes be inferred indirectly rather than being explicitly stated in the context. Nevertheless, the presence of the answer the context doc- ument provides an indication that the question can indeed be answered from information contained in it. Therefore, we create an additional filtered subset of our dataset which only contains QA pairs where (1) at least one of the answer candidates is contained in the context document, and (2) the context does not directly reference the image (as described previously). We refer to this subset as SK-V SK-VQAIR-CAP
3.2. Image Reference (IR) filtering
In the manual evaluation of generated context documents, we found that GPT-4 sometimes directly references the input image that was provided. For example, the context docu- ments may include references such "In the image, ..." or "As shown in the picture, ...". In such cases, the information contained in the context document is more similar to an extended caption or image description than a knowledge- intensive document. While this may not necessarily be detrimental to the training of multimodal RAG systems, it is unlikely in practice for RAG systems require the retrieval
4. Dataset Analysis
4.1. Dataset Composition
In order to acquire synthetic data which spans a broad range of different domains, we utilize images from mul- tiple sources during generation. These sources include LAION-400m, Wikipedia images contained in the WIT dataset, and synthetically generated images from COCO-
"See Appendix O for a discussion of hallucination impact
4

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Topic Distribution
Table Total numher of QA pairs in dataset by image and context source, computed separately for each filtered subset.
Architecture
Culinay
Evants
Image source
Context source
SK-VQA
SK-VQAIR SK-VQAI--CAP
Structures
Space
4/7%
4.9%
Coestal
Music
2.%
3/2
Regions
2.55
Nature
LAION
GPT-4
908,116
584,126
371,936
Gaming
Wikipedia
GPT-4
702,332
585,768
354,244
Science
1.%
X3%
Wikipedia COCO-CFs
Wikipedia GPT-4
181,554
167,352
137,160
other
214,487
193,226
121,284
2,006,489
1,530,472
984,624
Sports
Table Comparison of question (Q) diversity in KB- -B-VAA datasets
(* values are previously reported by Lerner et al. (2024)).
28.7%
Art
Dataset
Total Qs
Unique Q+
Unique POS
Vucab Size
Length
ViQuAE
3,700
3,562
2,759
4,700
12.4
InfoSock*
1,356,000
1,498
267
Enc-VQA*
1,036,000
175,000
91,945
40.787
11.6
SK-VQA
2,006.489
1,928,336
926,817
138,372
12.7
Figure 4. Topic distribution for SK-VQA context documents.
Counterfactuals (COCO-CFs). We use the entirety of COCO-Counterfactuall along with sub-sample of images from LAION and Wikipedia to generate context documents with QA pairs using our prompt. We also generate only QA pairs for a sub-sample of Wikipedia images paired with Wikipedia context documents, which enables us to compare the effect of using real context documents to synthetically generated contexts (see Appendix H for details).
to having nearly 50% more questions then the next-largest dataset, our dataset also exhibits significantly greater ques- tion diversity. The only two existing datasets of comparable size, InfoSeek and Encyclopedic-VOU (Ene-VQA) con- tain significantly fewer unique questions; less than 1% of questions in InfoSeek are unique, while fewer than 17% are unique in Enc- VQA. contrast, over 96% of the questions in SK-VQA are unique, which corresponds to 11x more unique questions than Enc-VQA. Questions in SK-VQA also exhibit greater number of unique POS sequences, total vocabulary size, and mean word length. This points to the value of leveraging powerful MLLMs for synthetic data generation over simpler techniques (e.g., templates).
Table provides breakdown of the total number of QA pairs in our dataset by image and context source. Our full SK-VQA dataset contains over 2 million QA pairs, making it the largest KB-Ve KB-VQA dataset created to-date. Of these 2 million QA pairs, 45% are associated with images sourced from LAION, 44% are associated with Wikipedia images, and the remainder are paired with synthetic images from COCCO-ouuterffattual... The SK-V SK-VQAIR subset contains 24% less QA pairs than the full SK-VQA dataset, while the SK- SK-VQA -QAR-CPP subset contains approximately half the number of QA pairs as the full SK-V VQA dataset.
4.3. Knowledge Classification
We apply an unsupervised topic modeling technique to cate- gorize the knowledge contained in our dataset's 's context doc- uments (see Appendix J for details). Figure depicts the dis- tribution of major topic categories identified in this analysis. Whereas previous KB-VQA datasets have focused primarily entity-specific knowledge, our dataset spans a broader range of topics such as art, fashion, sports, events, and mu- sic. This demonstrates the diversity of external knowledge required by questions in our dataset and suggests that it can serve as a complementary resource to existing datasets which focus on entity-specific knowledge.
Our full dataset contains 290,266 unique image-context pairs, which each have 7 QA pairs on average. GPT-4 generated context documents are associated with 7.1 QA pairs on average, whereas Wiki context documents are only associated with 5.7 QA pairs. This indicates that having GPT-4 generate both the context document and QA pairs simultancously enables the acquisition of more QA pairs, which could be attributable to how the generation of the context document conditioned on the subsequent task of producing QA pairs.
4.4. Human Evaluation
4.4.1. 1. QUESTION ANSWER PAIRS QUALITY
4.2. Question Diversity
We randomly sample 100 QA pairs from our dataset for human labeling by three of the authors of this work, ensuring that the 100 QA pairs are equally distributed across the four
Table 2 provides statistics on the diversity of questions in our dataset and other existing KB-VQL VQA datasets. In addition
5

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
50 QA pairs and supporting evidence in context documents using online sources. 86% are verified as factual, 4% are non-factual, 2% are partially factual, and 8% can not be determined due to a lack of available information.
Table Human performance on different subsets of 100 sampled QA pairs from SK- SKA calculated using semantic evaluation.
Mean
Standard Deviation
VQA
0.77
0,02
SK- EK-VQAIR
0.77
0.02
4.5. Automatic Dataset Quality Evaluations
SK- OK-RQAR+
0.87
0.03
We conduct comprehensive automatic quality assessments of our generated dataset through multiple methods.
image & context source types shown in Table For each QA pair, the annotators were presented only with the image, context document, and question. They were then instructed to write an answer to the question, and optionally note any deficiencies that were evident.
1.5.1. I. RULE-BASED AND NLP TOOL EVALUATION
We assess the quality of our generated context documents through grammatical evaluation. Specifically, we use the widely-used LanguageTool on random sample of 10K context documents. Out of 436k characters, only 6.69 % are flagged for grammatical issues. A manual review of 50 flagged sentences showed that 80% are minor issues related to spelling, style, or punctuation. This reinforces our con- fidence that the generated text maintains high grammatical accuracy.
Table provides the mean and standard deviation of an- notator accuracy, calculated using the Enc- VQA semantic evaluation method (see Section 5.2 for details). The overall mean accuracy of the three annotators was 77% for the 100 QA pairs sampled from SK-VQA and the subset of those which belong to SK-V For the subset of annotated QA pairs which belong to SK-VQA the mean human ac- curacy increases to 87%, which is consistent with previously reported human accuracy for other VQA datasets (Hudson & Manning, 2019; Sheng et al., 2021). The relatively low standard deviation indicates annotators performed similarly.
Another potential concern with synthetically generated data is the presence of biases, which can be reflective of those possessed by the models used for generation. We use state- of-the-an the-art bias and toxicity detection models to examine all generated documents for potential bias or toxic content. The detection results show no bias or toxicity in our dataset.
To understand potential failure cases in our dataset, we categorized common annotator comments by identifying those for which at least two annotators recorded the same category of issue for a question. The most common issue re- ported by annotators were cases which the question could be answered solely using the context document, assuming that the context document was provided at inference time. While this concern was noted for 9% of evaluated QA pairs, these examples may still require multimodal reasoning in a broader RAG system in which the question and image are necessary to retrieve the relevant context document. A small number of examples (5%) were identified as being answerable solely by looking at the image, while I question was noted having insufficient information in the context and image to answer (see Figure 12 for examples).
4.5.2. LLM-AS-JUDGE EVALUATION
To evaluate the quality of our dataset at scale, we employ an LLM-as- LLM-as-judge methodology using GPT-4o (OpenAI, 2024), different model from the one used for data synthesis. This automated evaluation targets four key dimensions of QA quality: (1) factuality of the image description, (2) relevance of the question to the image, (3) answerability of the ques- tion given the context, and (4) factual correctness of the answer.
For umage-descripee factuality, we prompt GPT-4o to as- sess how accurately the textual description reflects the image content on a scale from 0 (completely inaccurate) to 5 (fully accurate). For QA pair quality, we evaluate three aspects: relevance of the question to the description (0-5 scale), an- swerability based on the description (Yes/No), and factual correctness of the answer with respect the description (Yes/No). The detailed prompts are provided in Tables 12 and 13.
4.4.2. FACTUALITY EVALUATION
Factual accuracy is not primary concern for our synthetic data, as its main purpose is to train MLLMs to effectively uti- lize long contexts for VQA. Nevertheless, we conduct anal- yses to validate factual accuracy. Initially, we implement the automated fact-checking method SAFE (Wei et al., 2024). However, we find it ineffective for our dataset because it misclassified
Using this automated framework, we conduct a large-scale evaluation of our synthetic dataset. The results demonstrate high quality across all dimensions:
sentences unsupported due to its inability to process multimodal inputs.
'LanguageTool via jxmorris12 12
'NLP Bias Detector via vector institute
Therefore, we conduct a human evaluation to assess factu- ality. Specifically, we ask a native speaker to fact-check
RRoBERTa Detector via Facebook
6

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Factuality of Descriptions: Average score of 4.6/5, with 87.5% receiving perfect scores, indicating strong alignment between generated descriptions and visual content.
et al. (2023), where only external textual knowledge is re- quired for the given questions and images (denoted as task 6 by Wei et al. (2023)). We use the original Enc-VQA training set, but since each question can be paired with mul- tiple images, we select only the first image from the original annotations for the training set, which results in approxi- mately 220K training samples. For a fair comparison, we down-sample our dataset subsets to 200K samples each. For the real Wikipedia context from the WIT dataset, we use the paragraph-Ievel context associated with each image. See Appendix F and G for additional details.
Question Relevance: Average score of 4.9/5, with 92.0% rated at the highest level, demonstrating well- aligned questions with image context.
Answerability: 99.6% of questions are clearly answer- able based on the provided context, suggesting high internal coherence.
Answer Correctness: 90.7% of answers are factu- ally correct with respect to the description, indicating reliable generated answers.
5.2. Evaluation Datasets and Metrics
We use three existing KB-VQA datasets for evaluation: InfoSeck, Enc-VQA and ViQuAE. Additionally, we use 10,744 samples from SK- SK-VQAIR associated with images from LAION for model evaluation. For InfoSeek, we use a subset of its validation set processed by Wei et al. (2023), which includes 11.323 samples where only external textual knowledge is required for the given questions and images. For Enc-VQA, use its official test set, which contains 5,750 samples. Due to the small size of the ViQuAE test set, we combine the train, validation, and test sets to create a larger testing set of 3,625 samples. We use the official se- mantic evaluation method for Ann, BEM (Bulian et al., 2022). For other datasets, we use exact string matching.
Together with our human evaluation, these automated as- sessments provide scalable and interpretable validation of our dataset quality across multiple dimensions, confirming the high quality of SK-VQA for training and evaluating MLLMs on context-augmented VQA tasks.
4.6. Comparison with Existing Datasets
We conduct an analysis to compare our dataset to two types of multimodal datasets w.r.t. comprchensiveness and data quality: VQA datasets and those generated by LLMs. A detailed summary of our findings is provided in Table 11 of the Appendix. Existing VQA datasets are typically created using pre-defined templates or generated manually, which limits their diversity and scalability. On the other hand, previous datasets generated by L.LMs are often not verified by humans. In contrast, SK- VKAQ is both comprehensive in size and has been validated to possess high quality through human evaluations which exceed those employed for other L.L--generteed datasets.
5.3. Fine-tuning Results
We evaluate fine-tuned MLLMs on their ability to gener- alize to other datasets (i.e., out-of-domain performance). Figure 5 shows that for LLa fine-tuning with the InfoSeck dataset enhances performance the SK-VQA test set but does not yield improvements on Enc-VQA or ViQuAE. Similarly, fine-tuming with Enc-VQA fails to sur- pass baseline performance across all datasets. In contrast, models trained on our SK-VQA dataset achieve significant zero-shot improvements on both InfoSeek and Enc-VQA while outperforming the models trained on the other two datasets ViQuAE. We also trained LLaVA-13B models, resulting in similar trends (see Appendix A for details).
5. Experiments
5.1. Experimental Setup
We conduct zero-shot and fine-tuning experiments on MLLMs using both our dataset and existing KB-VQA datasets. For the zero-shot experiments, we test the fol- lowing popular MLLMs: PaLIGemma-3B (Beyer et al., 2024), LLaVA-v1.55-2 1-5--- (Liu et al., 2024b), LLaVA-1.66 7B/34B (Liu et al., 2024a), Qwen-VL-78 (Bai et al., 2023), and Idefics2-8B (Laurençon al., 2024). For the fine-tuning experiments, we utilize LLaVA-v1.5-7B and PaLI-Gemma- 3B. To demonstrate the effectiveness of our synthetic data, we train models using various subsets of our dataset gen- erated from different sources, as described previously in 3.1. Additionally, we train two baseline models on existing KB-VQA datasets: InfoSeck and NQA. For InfoSeck, we use 1 140K subset of the training data processed by Wei
Fine-tuning PaliGemma-3B using InfoSeek results in per- formance degradation in 2 out of 3 settings compared to the zero-shot baseline. However, fine-tuning with Enc- VQA consistently improves performance. Again, models trained on our dataset show significant performance im- provements in all 9 cases and achieve the best oft-of-dommin performance. Overall, these results indicate that fine-tuning MLLMs with SK- VQA effectively improves oft-of-dommin performance in most cases. Even when there is no improve- ment, SK-VQA does not result in performance degradation as with fine-tuning on other datasets. The in-domain perfor- mance, as expected, shows that the model performs best on
7

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
ExpAGGA
Sk-Aij
SK-VIAA
MAVA-1B
PellGemmnn-13
70-
70-
60
60-
40
40
30+
10
10
Eae-AQA Tisting Dataset
VIQUAE
SK-VQA
In6-Seck
Ens-VQA
VIQUAE
SK-VQQ
Testing Darases.
Figure 5, Performance of models trained on different KB-VQU -QQ datasets (indicated by har colors) and tested on various datasets (x-axis labels). Red dashed lines indicate baseline performance of MLLM without training.
Bandrw
Isk.Bck
En-Vija
SEAVOAA
Table Zeroshot evaluation of SOTA MLLMs on existing KB-
PalEGenana 3B W/ Reliteven: Coattest
VQA and datasets.
Model
Infoseek
Enc-VQA
ViQuAE
SK-VQA
PaliGemma-3B
25.66
32.89
47.72
25.51
L.aVA-vI.5-7B
42.82
$3.69
78.41
40.99
LLaVa-v1.6-7B
41.94
57.92
72.00
46.68
Qwen-VL-7B
39.48
$3.67
69.46
42.55
Idefies2-8B
44.33
67.92
82.43
38.08
LLaVa-v1.5-13B
$7.82
LLaVa-v1.6-13BB
46.32
61.39
83.75
45.57
LLLVa-v1.6634B
38.81
77.73
79.17
50.02
InfoSeck
Eac-VQA Testing Dalasel
VIQuAE
SK-VQA
the data it fine-tuned on (see Appendix Table 7).
Figure 6. Generalization Performance of fine-tuned PaliGemma in RAG setup. Our models achieve the best generalization.
5.4. Zero-shot Evaluation Results
Table 4 provides the results for zero-shot evaluations. All tested state-of-the-art MLLMs perform better on Enc-VQA and ViQuAE compared to InfoSeek and our SK-VQA dataset. This suggests that SK-VQA and InfoSeek present significant challenges to these models. Unlike with Enc- VQA and ViQuAE, larger models do not always yield better performance on InfoSeck and SK-VQA. This indicates that simply relying on model size may not be sufficient to ad- dress the reasoning challenges presented by these datasets. Additional evaluation results on recent state-of-the -ftth-r--he-art mod- els can be found in Appendix D.
is more challenging because the model needs distinguish relevant parts of the context from irrelevant information. In this experiment, we focus on using the Paligemma- model (see Appendix A for the construction external knowledge bases). For each question, we retrieve the top most relevant passages. During inference, we combine each of these 10 retrieved passages with the question and perform inference. The final answer is determined by se- lecting the most frequently occurring answer among these 10 inferences. Results by model are presented in Figure 6. The results show that even when using retrieved contexts, the model trained on our dataset performs strongly both in- domain and out-of-domain. Notably, in out-of-domain per- formance, it surpasses the baseline zero-shot performance and models trained on the other two datasets in all 9 cases.
5.5. A Alltioo Studies
Retrieval Augmented Generation (RAG) Results In ad- dition to utilizing the gobd-label context documents (Tables 4 & 7), we use the CLIP Score Fusion model from Wei et al. (2023) to retrieve knowledge from external text knowledge bases as context to simulate real RAG setup. This setting
Impact of Generation Source We explore the perfor- mance of models trained on data generated from different
8

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
Table Performance comparison of LLaVa-v1.5-7B trained
Table 6. Zeroshot evaluation of SOTA MI.LMs on existing KB- VQA and our datasets.
synthetic data derived from different sources (without filtering).
Image Source
Context Source
Infoseek
Ene-VQA
VIQuAE
Avg.
Model
LAION
WIT(GPT-4)
WiT(Wiki)
Coco-CF
LLaVA-v1.5--7
40.99
44.35
50.45
41.4
LAION
GPT4
44.32
65.44
79.22
62.99
LLaVa-v1.6-78
46.68
48.9
54.8
46.85
Wiki
GPT4
47.00
53.98
78.58
59.85
Qwen-VL.-7B
42.55
42.45
47.6
41.6
Wiki
Wiki
47.75
66.67
77.95
64.12
LLVa-v1.5-138
40,42
41.5
50.85
39.4
COCO-CFs
GPT4
48.00
65.42
79.23
64.22
LLaVa-v1.6-13B
45.57
46.5
56.25
43.5
that our dataset can serve as a more challenging benchmark than existing resources. Additionally, training MLLMs on our dataset leads to greater improvements in out-of-domain generalization than other datasets. These results point to not only the utility of SK-VQA, but also the effectiveness of our approach for acquiring synthetic multimodal data at scale. Opportunities for future work in this direction include lever- aging larger amounts of synthetic image data to produce fully-synthetic datasets for other domains and leveraging SK-VQA for training multimodal retrieval models to aid in the development of RAG systems.
images and context sources to understand how each im- age source contributes to the dataset's 's effectiveness. Ta- ble 5 shows that the best combination involves using images from COCO-CFs and context documents from GPT-4. No- tably, this combination even surpasses using images from Wikipedia with their real context, demonstrating that syn- thetic images paired with generated contexts can be effec- tive as-or even more effective than-real image-context pairs. This indicates that SK-VQA can offer advantages for fine-tuning MLLMs compared to real data. Additionally, when using GPT-4 generated context documents, we observe that models fine-tuned on data derived from LAION images perform better on VQQ and ViQuAE, whereas models fine-tuned data derived from Wiki images perform better on InfoSeek. This suggests that different image sources contribute distinct generalization capabilities, and that com- bining images from diverse be necessary to achieve better generalization across all external datasets, as shown in $5.3.
Impact Statement
This paper presents work whose goal is to advance the field of multimodal machine learning through the creation of a large-scale synthetic dataset for knowledge-base based visual question answering. While our primary contribution is to improve the training and evaluation of multimodal language models, we acknowledge several broader implications of our work.
iictiess-iicuiinngg synthetic
Exploration of Using LLaVA-34B for Generation We also explored the use of the state-of-the-art open-source model VA-34B ing synthetic data. We follow the same pipeline mentioned earlier and manually evaluate the generated data. However, the annotators report that 76% of the questions generated by LLaVA-34B are invalid due to one of the three reasons discussed in $4.1. Most of these questions can be answered solely by the context, without requiring the image, such as "What the purpose of the Great Wall?" or "What is the main purpose of dining set?". Additionally, the questions generated by LLa are much simpler compared to those generated by GPT-4, often not requiring complex reasoning to answer. These findings suggest that maintaining the quality of generated datasets requires using the most advanced MLLM, even if it proprietary.
The use of synthetic data generation at scale presents both opportunities and risks. On the positive side, our approach democratizes access to high-quality training data, poten- tially enabling researchers with limited resources to develop competitive multimodal systems. However, as discussed in our limitations (Appendix O), the synthetic nature of our dataset means it may contain inaccuracies reflect biases present in GPT-4, despite content filtering measures. We en- courage users of our dataset to validate model performance thoroughly before deployment.
replacement for GPT-4 in generat-
find that
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
6. Conclusion
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: frontier large vision- language model with versatile abilities. arXiv preprint arXi: 2023.
We introduced SK-VQA: large dataset of 2 million QA pairs over images from multiple different sources. SK-VQA is the largest and most diverse resource of its kind, pos- sessing 11x more unique questions than similar datasets for KB-VQL VQA. Our evaluations of popular MILMSs showed
Beyer, L., Steiner, A., Pinto, S., Kolesnikov, A., Wang,

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
X., Salz, D., Neumann, M., Alabduulmosinn, I., Tschan- nen, M., Bugliarello, E., Unterthiner, T., Keysers, D., Koppula, S., Liu, F., Gryener, A., Gritsenko, A., Houlsby, N., Kumar, M., Rong, K., Eisenschlos, J., Kabra, R., Bauer, M., Bošnjak, M., Chen, X., Minderer, M., Voigt- laender, P., Bica, I., Balazevic, I., Puigcerver, J., Papalam- pidi, P., Henaff, O., Xiong, X., Soricut, R., Harmsen, J., and Zhai, X. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:240..07726 2024.
Grootendorst, M. Bertopic: Neural topic modeling with class-based tf-idf procedure, 2022.
He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and Qi, X. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574. 2022.
Howard, P., Singer, G., Lal, V., Choi, Y., and Swayamdipta, Neurocounterfactuals Beyond minimal-edit coun- terfactuals for richer data augmentation. arXiv preprint arXiv:22 20.1::22101233660 /2365, 2022.
Bulian, J., Buck, C., Gajewski, W., Börschinger, B., and Schuster, T. Tomayto, tomahto. beyond token-level an- swer equivalence for question answering evaluation. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pp. 291-305, 2022.
Howard, P., Madasu, A., Le, T.,, Moreno, G. L., Bhiwandi- walla, A., and Lal, V. Socialcounterfactuall:: Probing and mitigating intersectional social biases in vision-language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 11975-11985, 2024.
Caffagni, D., Cocchi, F., Moratelli, N., Sarto, S., Cornia, M., Baraldi, L., and Cucchiara, R. Wiki-llava: Hierarchi- cal retrieval-auue auggente generation for multimodal Ilms. arXiv preprint arXiv:2404. 15466, 2024.
Hu, Z., Iscen, A., Sun, C., Wang, Z., Chang, K.-W., Sun, Y., Schmid, C., Ross, D. A., and Fathi, A. Reveal: Retrieval-augmenter visual-language pre-training with multi-source multimodal knowledge memory. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23369-23379, 2023.
Chen, W. HU, H., Chen, X., Verga, P., and Cohen, W. W. Murag: Multimodal strrival-augmentt generator for open question answering over images and text. arXiv preprint wxxv:2210.00928. 2022.
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Rit- ter, A., and Chang, M.-W. Can pre-trained vision and language models answer visual information-ssekingg ques- tions? arXiv preprint arXiv: 11713, 2023.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
Ding. Y., Ren, K., Huang, J., Luo, S., and Han, S. C. Mvqa: A dataset for multimodal information retrieval in pdf-based visual question answering. arXiv preprint arXiv:2404. 12720, 2024.
Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision- lisiun-language models?, 2024.
Le, T., Lal, V., and Howard, P. Coco-counterfactuaal:: Auto- matically constructed counterfactual examples for image- text pairs. Advances in Neural Information Processing Systems, 36, 2024.
Doris, A. C., Grandi, D., Tomich, R., Alam, M. F., Cheong, H., and Ahmed, Designqa: A multimodal benchmark for evaluating large language models' under- standing of engineering documentation. arXiv preprint arXiv:2404.07917, 2024.
Lerner, P., Ferret, O.. Guinaudeau, C., Le Borgne, H., Besançon, R., Moreno, J. G., and Lovón Melgarejo, J. Viquae, a dataset for knowledge-based visual question an- swering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. -120-,,,,, 2022.
Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model training... for now. arXiv preprint ar21v:2312.04567, 2023.
Gao, .. Pi, R., Zhang. J.. Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al. G-llava: Solving geo- metric problem with multi-modal large language model. arXiv preprint arXiv:2312.1 11370, 2023.
Lerner, P., Ferret, O., and Guinaudeau, C. Cross-modal retrieval for knowledge-based visual question answering. In European Conference Information Retrieval, pp. 421-438. Springer, 2024.
Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, arXiv preprint arXiv:2404.0 0/413, 2024.
Lewis, P., Perez, E.,, Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,, Lewis, M., Yih, W.-t,, Rocktäschel, T., et al. serrivaal-augmener generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-9944 2020.
10

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning. pp. 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
PMLR,
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355, 2023.
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton- K., and Shoham, Y. In-context retrieval retrieval-augmen augmented language models. Transactions of the Association for Computational Linguistics, 11:1316- 1331, 2023.
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26296-26306, 2024a.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Reimers, N. and Gurevych, I. Sentence-bert. Sentence beddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computational Linguistics, 11 2019. URL ptatps://arxi.... abs/ 1008/1908.0 10084.
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L.. Li, R., and Peng. W. A survey on halluci- nation in large vision-languan models. arXiv preprint arXiv:2402.00055, 2024c.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Lu, S., Li, Y., Chen, Q-G., Xu, Z., Luo, W., Zhang, K., and Ye, H.JJ Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.22 207977 2024.
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479-36494, 2022.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:21 1..02:211.022114, 2021.
Mensink, T., Uijlings, J., Castrejon, L.., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., and Ferrari, V. Encyclo- pedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3113- 3124, 2023.
Sharifymoghaddam, S., Upadhyay. S., Chen, W., and Lin, J. Unirag: Universal retrieval augmentation for multi-modal large language models. arXiv preprint arXiv:2405.103: 2024.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I, and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 2021.
Sheng, S., Singh, A., Goswami, V., Magana, J., Thrush, T., Galuba, W., Parikh, D., and Kiela, D. Human-adversee visual question answering. Advances in Neural Informa- tion Processing Systems, 34:20346 20006,2203992222 2021.
OpenAI. Hello got-4o, May 2024. URL https:// 4o/.
openai com
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training generated data makes models forget. arXiv preprint arXiv:2305. 2023.
Qiu, J., Madotto, A., Lin, Z.,, Crook, P. A., Xu, Y. E., Dong, X. L., Faloutsos, C., Li, L.. Damavandi, B., and Moon. S. Snapntell: Enhancing entity entity-centric visual question an- swering with retrieval augmented multimodal Ilm. arXiv preprint 2xx::24433.0477 2024.
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wir: Wikipedia- based image text dataset for

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
multimodal multilingual machine learning. In Proceed- ings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2443-244 2449, 2021.
Yang. Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., al. visual language model for zero and few-shot image captioning, arXiv preprint arXiv:2302.04830 2023.
Re-vilm: Retrieval-augmentt
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https://qwwnlli github..hh......... io/ blog/qwent ht/p///wn/......
Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., and Yih, W.-t. annmeete multimodal language modeling. arXiv preprint arXiv:2211 1.::611,, 2022.
Trabucco, B., Doherty, K., Gurinas, M., and Salakhutdinov, R. Effective data augmentation with diffusion models. arXiv preprint ar arXiv:302..07944 2023.
Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal, S., Zhu, C., Zeng, M., and Jiang, M. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209. 1000: 2022.
Van Horn, G., Cole, E., Beery, S., Wilber, K., Belongie, S., and Mac Aodha, O. Benchmarking representation learning for natural world image collections. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12884-12893, 2021.
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.0075: 2023.
Vendrow, J., Jain, S., Engstrom, L., and Madry, A. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.07865, 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing iision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:: hatp::://pperreeeee wpw.reviewweeeeeeeoepee net forum?
Wang. I.,, Yang, N., Huang, X., Yang, I.., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.0033 00366, 2023.
id=1t2bq88f27.
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving text embeddings with large lan- guage models. In Ku, I..W.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11897-11916, Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. 10.18655//1/20022allllooggg 642. URL https: //acl antho wtg. acl - logg 642/.
Wang, Z., Elfardy, H., Dreyer, M., Small, K., and Bansal, M. Unified embeddings for multimodal retrieval via frozen lims. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 1537-1547, 2024b.
Wei, C., Chen, Y., Chen, H., Hu, H., Zhang, G., Fu, J., Ritter, A., and Chen, W. Uniir: Training and benchmarking uni- versal multimodal information retrievers, arXiv preprint arXiv:2311.17156 2023.
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Tran, D., Peng, D., Liu, R., Huang, D., Du, C., et al. Long- form factuality in large language models. arXiv preprint arXiv:2403. 18802, 2024.
Weyand, T., Araujo, A., Cao, B., and Sim, J. Google land- marks dataset v2-a large-scale benchmark for instance- level recognition and retrieval. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020.
12

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Table Performance of Generator fine-tuned 200K from different datasets. on semantic metric, others are Exact Matching scores.
denotes the in-domain results. Encyclopedia evaluated
Training Data
Infoseek
Ene-VQu
ViQuAE SK-VQAIR
25.66
32.89
47.72
25.51
PaliGemma-33
Infoseek
66.63*
29.58
64.22
15.27
Enc-VQA
35.78
83.30*
65.19
35.29
SK-VQA
42.50
55.53
71.12
65.02*
SK-VQA
42.69
55.67
71.48
65.26*
IK-VQAIR-CAP
43.72
53.72
71.97
64.69*
42.82
53.69
78.41
40.99
LLaVA-1.6-338 LLaVA-7B
Infoseek
68.68*
38.68
69.90
43.69
Enc-VQA
31.39
88.75*
55.59
21.60
SK-VQA
47.35
63.89
77.60
68.30*
SK-VQA
48.11
70.99
78.04
68.35*
SK- SK-VQA SK-VQAIR-CAP
47.55
72.33
78.95
68.61
46.32
61.39
83.75
45.57
Infoseek
74.48*
54.59
79.56
52.98
Enc-VQA
40.36
90.59*
72.88
29.71
SK-VQA
47.58
66.84
82.04
70.27*
SK-VQAIR
50.28
66.58
81.02
70.23
SK-VQAIR-CAP
48.77
67.35
81.79
70.79*
A. Additional Fine-tuning Experimental Results
LLaVA-1.6-13B Fine-tuned Results Table 7 showcases the performance of fine-tuning -LaVA-1.6-132) on different datasets. The models trained on our datasets achieve the best generalization performance compared to the models trained on other two existing datasets.
In-domain performance Table provides additional in-domain performance results of fine-tuned models from the experi- ments discussed in $5.3. The results indicate that models trained specific datasets perform best their corresponding test sets. However, as noted in $5.3, good in-domain performance for models trained on InfoSeek and Enc-VQA does not guarantee good out-of-domait performance, In contrast, our models, trained on our dataset, achieve strong performance both in-domain and out-of-domain.
Retrieval Augmented Generation External Knowledge Bases Construction For constructing external knowledge bases, we use the InfoSeek dataset knowledge base processed by Wei et al. (2023), which includes 611,651 passages. For the other three datasets, Enc- VQA, InfoSeek, and ViQuAE, we create synthetic external knowledge bases by merging the Enc-VQA,
corresponding gold passages for each test question. The sizes of the knowledge bases for Enc-VQA, ViQuAE, and synthetic dataset are 3,859, 71,985, and 1,514 passages, respectively.
B. Ablation Study on Impact of Filtering Techniques
To further explore the impact of different filtering methods and their interaction with image sources, we fix the image source for data generation and compare the performance models trained on data filtered by various methods across three out-of-domain datasets, shown Table 8. We sequentially apply IR and CAP filtering on SK-VQA. For data from LAION, SK-VQA, retains 64% of the original data, while SK-VQA retains 40%. For data from Wikipedia, SK-VQA gR retains 83%, and SK-VQAIR-CAP retains 50%.
The results reveal how filtering methods interact differently with various image sources. With LAION as the image source, SK-VQAIRACAP achieves the best average performance across the three datasets, though SK-VQA outperforms SK-VQAR- For data from Wikipedia, SK- SK-VQAIR outperforms both SK-V VQAIIAAI and KK-VQA overall, while certain datasets (InfoSeek) benefit most from the full SK-VQA dataset. This demonstrates that LAION and Wiki sources exhibit
13

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
Table Impact of filtering techniques by image source.
All results utilize context documents from GPT-4.
Training Data
Image Source
Infoseek
Ene-VQA
ViQuAE
SK-VQA
LAION
44.32
65.44
79.22
62.99
SK-VQAIR
LAION
44,43
63.08
75.50
61,00
SK-VQAB-CAP
LAION
45.85
69.88
78.18
64.64
SK-VO SK-VQA
Wiki
47.00
53.98
78.58
59.85
SK-VQAIR
Wiki
45,99
67.36
79.37
64.24
3K-VQAB-CCA
Wiki
46.48
64.55
79.83
63.62
distinct generalization patterns when combined with different filtering strategies, suggesting that specific filtering methods may improve performance for certain domains or datasets, while the full unfiltered SK-VQA dataset might be more valuable for others. These findings highlight the versatility of our multi-source approach and filtering techniques. The various filtered subsets of our dataset can be treated as hyperparameter to find the best performance for specific tasks. We also note that significant benefit of filtering the ability to achieve similar better performance with significantly fewer samples, which holds true for both filtering methods across all datasets.
C. Difficulty Analysis Across Different Image Sources
To better understand the role of different image sources our dataset, we evaluate the performance of state-of-the-ane multimodal LLMs on different subsets of SK- "K-VQA, divided based the source and type of image content. Specifically, test LLa VA-v1.5 (7B and 13B), LLaVA-v 1.6 (7B and 13B), and Qwen-VL (7B) (Bai al., 2023) across four distinct subsets: WIT (Wikipedia context), WIT (GPT-4 generated context), LAION, and COCO-CF.
Table Model performance across different image source subsets SK-VQA. Results show consistent difficulty ordering across all models.
Model
LAION
WIT (GPT-4)
WiT (Wiki)
COCO-CF
LLaVA-vI.5-77
40.99
44.35
50.45
41,40
LLaVA-v1.6-7B
46.68
48.90
54.80
46.85
Owen-VL-7B
42.55
42.45
47.60
41.60
LLaVA-vI.5-13B
40.42
41.50
50.85
39.40
ILaVA-v1.6-13B
45.57
46.50
56.25
43.50
Average
43.24
44.74
51.99
42.55
The results in Table 9 demonstrate that each subset presents a distinct level of difficulty. We observe a consistent pattern across all models, with difficulty increasing in the following order: WIT (Wikipedia context), WIT (GPT-4 generated context), LAION, and COCO-CF.
We hypothesize that the WiT (Wiki) subset achieves the highest performance because LLMS likely trained substantial amounts of Wikipedia content, making this subset more familiar and casier to process. In contrast, the COCO-CF subset, which includes counterfactual images paired with GPT-4 generated content, presents the highest degree of difficulty. These counterfactual image-conten pairs are largely -iitr--dittiitttion relative to the training data of existing models, resulting in consistently lower performance.
These findings highlight the diversity of our dataset and underscore importance incorporating varied content sources in the construction of SK- KQA. While many existing knowledge-ban VQA datasets predominantly rely on Wikipedia-basee images, our results demonstrate that including diverse sources parttcuuaallllllyyy those beyond creattss a more challenging and comprehensive benchmark for evaluating multimodal LLMs.
D. Evaluation on Additional State-of-the-Art Multimodal LLMs
To explore the generalization our benchmark across a broader range of architectures and model scales, we evaluate additional state-of-the-ar multimodal LLMSs on SK- SK-VQA Specifically, we test GPT-4o (OpenAI, 2024), two recently
14

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
released model families: Qwen-2.5-VL (3B, 7B, 32B, and 72B parameters) (Team, 2024) and Ovis (IB 34B parameters) (Lu et al., 2024), which represent of the most advanced multimodal models currently available. We use the same test set as described in Section 5 to ensure fair comparison with previously reported results.
Table 10. Performance of additional ofate-of-the-ee art multimodal LLMs SK-VQA compared to ViQuAE. Results demonstrate that SK-V -Q- provides consistent and challenging benchmark across diverse model architectures and scales.
Model
SK-VQA ViQuAE
GPT-4o
58.90
Qwen-2.5-VL-3B
53.74
Qwen-2.5-VL-7B
49.26
52.08
Qwen-2.5-V1-772.
49.09
Ovis- IB
32.25
39.50
Ovis-2B
44.54
67.09
Ovis-4B
50.55
49.38
Ovis-8B
50.36
57.96
Ovis-16B
52.36
72.77
Ovis-34B
55.20
67.03
The results in Table 10 reveal several important findings. First, even the most advanced models achieve moderate performance on SK-VQA, with GPT-4o reaching 58. 80%900% accuracy and the hest-performing open model (Ovis-34B) achieving 55. 55.20%. This indicates that SK-VQA captures complex multimodal reasoning challenges that remain difficult for current state-of-the- art models.
Second, comparing the Ovis models' performance on SK-VQA versus ViQuAE demonstrates that our dataset provides more consistent evaluation across model scales. While Ovis models show highly variable performance ViQuAE (ranging from 39.50% to 72.77%), their performance SK-VQA remains within narrower range (32.25% to 55.20%). This consistency suggests that SK-VQA avoids the evaluation instabilities observed in existing benchmarks.
E. LLM Evaluation
Previous studies have shown that a significant proportion of questions in OK-VQA and ViQuAE can be answered by an LLM when prompted with only the question (Chen al., 2023). To investigate whether this the case for our dataset, we generate answers from I.LaM-3--0b-Inssmnn for all million QA pairs in dataset using the following prompt:
Write a single word or phrase which answers the question. Question: [QUESTION]
where [QUESTION] is populated with questions from our dataset query time. We found that the exact match accuracy of LLaMA-3-70b is only 9.92% for our dataset, indicating that the vast majority of questions do indeed require reaosning over the associated images and context documents.
F. MILMs Zero-shot Prompts
For all MILMs in 4, we use the following text prompt when conducting zero-shot prompting, addition to each model's specific image token:
Context (context) Based on the context, (question) answer the question using single word or phrase.
15

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
Krite Wikipedia article related to this image without directly referring to the image Then write question answer pairs. The question answer pairs should satisty the following criteria.
3 1: Guide the model to generate questions using image information.
2: Avoid questions that can answered without looking at the image.
5
3: Guide the model to generate questions using external context rather than simple visual information from the image.
6
4: Since GPT-4 tends to generate unnecessarily lengthy questions that do not sound natural, this condition helps to prevent such questions.
5: Guide the model to utilize the context and also make answer evaluation streightforward.
8
6: GPT-4 tends to ask questions where the answer is an object in the image. For such questions, context is not needed, which is not of our interest.
7: we can split multiple correct answera into a 1ist to make the evaluation easier.
10
8: This condition is also for making the evaluation easier.
Figure 7. Explanation for each prompt condition. The numbered explanations correspond to the numbered conditions in our prompt (Figure 3)
G. MLLMs Fine-tuning Hyperparameters
We use the official codebase? from VA-1.5 to fine-tune the Ilava-v1.5-7b model and the Trainer from Huggingface Transformers library" to fine-tune the Saligemma--b-mix--o
model 10, For the Ilava-v1,5-7b model, we use a batch size of 16 and learning rate of 2e-5, training the model for one epoch using 1. Similarly, for the paligemma-Sb-mng model, we use a batch size of 64 and a learning rate of 2c-5, also training for one epoch using bíloat16. These are default hyperparamter values which were not tuned as part our expierments. The inputs to the models are combination the question, image, and context, and the outputs the answers the questions.
H. Additional details of GPT-4 generation
Prompts Figure 7 provides an explanation of the motivation for each condition which we include in our prompt. The numbered explanations correspond to the numbered conditions in our prompt which shown in Figure 3. The instruction and conditions in our prompt were derived through manual prompt engineering, where different prompts were tested and iteratively updated in response to issues that were identified in the synthetic data produced by GPT-4.
As discussed in Section 4.1, we also generated only QA pairs for a sub-sample of Wikipedia image-context pairs sourced from the WIT dataset. Figure provides the prompt that we used with GPT-4 for this generation setting. In this prompt, [CONTEXT] is placeholder where the actual Wikipedia context document is inserted at inference time. Other conditions in this prompt are identical to those in main prompt specified in Figure 3.
Output parsing Here we describe the process of extracting context, question, and answer pairs from the text output generated by GPT-4. We first segment the entire output into two parts: the Wikipedia article and Question Answering pairs, identified by the line containing "question", "answer", and "pair". For both chunks, remove extra symbols like hashes, stars, and consecutive spaces. In the Wikipedia article, we also remove the words "Wikipedia article" at the beginning. For the Question Answering pair chunk, we segment it by line and extract the question and answer by splitting each line using the symbol ".", retaining only the sentences after the "..
API We accessed GPT-4 via the Azure OpenAl API and collected our entire dataset between the dates of May 24, 2024 and June 5, 2024. We used the 2ot-44-2024405--2 version of GPT-4 for all of our synthetic data generation.
"https://ght :ttp://ggttt/////// hwt.g/aa....aaatttiiaaiiiaaaaaa.
-eeirmaa-bbmiee 22-4
16

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Contex--Angmeteed Multimodal LLMs
Here is Wikipedia article related to this
inage:
3
4
[CONTEXT]
write question answer pairs which require both the image and the Wikipedia article. The question
7
answer pairs should satisfy the following criteria.
8
1: The question should refer the image.
10
2: The question should avoid mentioning the name of the object the image.
11
3: The question should be answered by reasoning over the Wikipedia article.
12
4: The question should sound natural and concise
13
5: The answer should be extracted from the Wikipedia article.
14
6: The answer should not be any objects in the image.
7: The answer should be a single word or phrase and 1ist all correct answers separated
by commas.
16 8: The answer should not contain 'and', 'or', rather you can split them into multiple
answers.
Figure 8. Prompt used to generate only QA pairs for an existing image-context pair using GPT-4.
I. Details of compute infrastructure used in experiments
We utilized 24 Intel Gaudi2 Al Accelerators to obtain LLaMA-3-70b predictions for our dataset, which were used to create the 'hard' version of our test dataset (Appendix A) and perform LLM evaluation using only questions from our dataset (Appendix E).
For our zero-shot MILMM evaluation and MILMM training experiments, we used internal linux slurm cluster with Nvidia RTX 3090, Nvidia A6000, and Nvidia A GPUs. We used up to 48 GPUs parallelize various experiments on this cluster. Each parallelized worker was allocated 14 Intel(R) Xeon(R) Platinum 8280 CPUs, 124 GB of RAM, and 1 GPU, The total comptue time for job varied between 6-48 hours depending upon the model, dataset, and evaluation setting.
J. Topic model details
We removed stop words from context and applied BERTopic (Grootendorst, 2022) to apply categorical TF-IDF on context embeddings created with All-MiniLMM-L-- sentence transformer model(Reimers Gurevych, 2019). We initially reduced the number of topies 40 using agglomerative clustering. Subsequently, we manually merged semantically related clusters, resulting in the following 25 topics, listed from most to least frequent: general, design, fashion, sports, wildlife, nature, coastal, events, culinary, architecture, transport, pets, structures, space, music, regions, gaming, science, politics, biology, military, postal, entertainment, economics, and religion. The "general" category could not be easily interpreted because it contained a mixture of many different unrelated topics; to improve visual clarity of the figure, was therefore excluded, and categories representing less than 1% the dataset were grouped under 'Other' category.
K. License information
We abide by the licenses and intended uses of all models and datasets which were employed in this study. License information for models and datasets are provided below.
Models used in our study The PaliGemma-3BB model is available under the Gemma license. The L.L.VA-1.5--7 is available under the Llama Community License. The LLaVa-v1.6-77 under the Apache License, Version 2.0. Qwen-VL.e VL-7B is available under the Tongyi Qianwen License.
1.6-34B, and idefics2-8b are available
Existing datasets used in our study The WIT dataset is available under the Creative Commons Attribution Siriiion-SharAAiiie 3.0 Unported license. The ViQuAE datset is available under the MIT license. The ConC--ouuterfacttuuls dataset available under the CC BY 4.0 license. The InfoSeek dataset available under the Apache 2.0 license.
17

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Contex--Augmented Multimodal LLMs
Context source: GPT-4
Context source: Wikipedia
the sport association football, teams are distinguished by the unique colors and designs of their kits. Kits usually consist of a shirt, shorts, and sacks, and teams have both home and an away version. The home kit is typically in the team's 's primary colors, while the away kit clashes with other teams. Stripes are common feature football designs, with many teams opting for vertical or horizontal stripes two or more colors. Red and white striped kits are popular among several football clubs around the warld, a design often associated with clubs like Sunderland AFC and Atlético Madrid. Similarly, green and white horizontally striped kits
Context:
Context: Alice Springs Celtic an Australian soccer club based in Alice Springs, the Northern Territory. The club
of the first football clubs to be founded in the FFNT Southern Zone Premier League. The club is part of the Football Federation Northern Territory. The club has seen great success in recent years, becoming Premiers in 2015 and Champions in 2016, the club has competed the annual FFA since 2015 where reached the Seventh Qualifying Round, eventually losing 5-0 to Darwin Olympic. The club was knocked out in the Alice Springs final in the 2016 edition of the FFA Cup losing 1-0 to Gillen Scorpions and narrowly lost 2-0 in the 2017 edition the Sports Ministers Cup to Darwin Rovers.
color that contrasts with their home kit to avoid
traditional for teams such as Celtic FC. Kits aften feature the team logo, the manufacturer's..
beginning the 2017 season prolific striker Jaseph Worrall transferred to MPH Vikings
Figure 9. Comparison of context documents sourced from GPT-4 and Wikipedia for the same Wikipedia image. The context document from Wikipedia contains more entity-spocific knowledge, whereas the GPT-4 context document contains more general knowledge about what is depicted in the image.
Our dataset We make our dataset publicly available under the Intel Research & Development License. In addition to the terms of this license, use of our dataset should abide by the OpenAl terms of use.
L. Additional examples
Comparison of context documents sourced from GPT-4 and Wikipedia A subset of our dataset contains Wikipedia images for which we obtained context documents both from GPT-4 and from Wikipedia (via the image-tex associations provided the WIT dataset). Figures 9, 10, and 11 provide examples of the GPT-4 and Wikiieddia-soureed context documents for identical Wikipedia images. The example in Figure shows how context documents obtained from Wikipedia tend to have more entity-specific knowledge, whereas GPT-4 often generates more general knowledge which is related to what is depicted in the image. In Figure 10, the context document generated by GPT-4 is more specific to what is depicted in the image (a vineyard) than what is discussed in the Wikipedia context document (a specific type of wine). Finally, Figure 11 shows how the GPT-4 generated context documents can be longer and more detailed than the Wikipedia context documents which are linked to the image via the WIT dataset.
Examples of failure cases identified by human annotators Figure 12 provides examples of failure cases which were identified by human annotators. As described previously in Section 4.4, the three common categories of synthetic data generation failures identified by human annotators were cases where the question answerable without looking at the image (Figure 12a), cases where the question is answerable without the context document (Figure 12b), and a single case where insufficient information was provided to answer the question (Figure 12c).
In all three cases depicted in Figure 12, the generated context documents are highly relevant to he image and also contain the answer. However, the question for for example depicted in Figure 12a does not reference the image, and could potentially be answered solely by uni-modal retrieval of the context document based on the question. The question associated with Figure 12b could potentially be answered solely through visual perception capabilities. although the associated context document would increase the chance of generating the correct answer. Finally, the example depicted in Figure 12c for a
18

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Aatina
Context source: GPT-4
Context source: Wikipedia
Context: vineyard agricultural where grapevines grown for the purpose wine production. The history of viticulture dates back to ancient times, with early evidence of grape cultivation and wine-making discovered in regions such as Mesopotamia, Egypt, and the Mediterranean. Vineyards are typically located on slopes with good exposure well-draining soil, which are ideal conditions for growing grapes. Vineyards often characterized by rows of trellised vines that maximize sunlight exposure and air circulation, benefiting the health and productivity of the grapevines. The rows of vines are usually planted with companion plants like roses at the end each row, which serve as an early warning system for diseases affect the vines. Wine estates, known as wineries,
Context: New World wines those produced outside the traditional winegrowing
Europe and the Middle East, in particular from Argentina, Australla, Canada, Chile. Mexico, New Zealand, South Africa and the United States. The phrase connotes distinction between these "New World" wines and those wines produced in "Old wine production most notably, France, Italy, Germany, Spain and Portugal. In 1595 the Spanish Crown banned the establishment of new vineyards in the Americas, but this order was largely ignored. The ban sought to protect Iberian wine from competition by Peruvian wine and can considered example commodity mercantilism. Moreover, the Spanish Crown banned the export of Peruvian wine Panamá
countries with long established history
Figure 10. Comparison of context documents sourced from GPT-4 and Wikipedia for the same Wikipedia image. The context document sourced from Wikipedia contains knowledge that is less specific to the image than the GPT-4 generated context document.
Context source: GPT-4
Context source:
Wikipedia
Context: Common Evening Primrose, scientifically known "Genothera biennis*, a biennial plant commonly found in North America. Typically, it grows in less-cultivated areas such fields and roadsides. The plant is noted for its tall, erect stems that can reach up 1.5 meters height. The flowers of the Common Evening Primrose are usually yellow open These flowers known for their sweet fragrance, which attracts pollinators such as bees and moths. The blooming season extends from late spring to early fall, where the flowers are most vibrant. In traditional medicine, the Evening Primrose has been used for various purposes. Americans extracted from plant to wounds and inflammation. modern times
Context: Oenothera genus about 145 species of herbaceous flowering plants native to the Americas. is the type genus the family Onagraceae. Common names include evening primrose, suncups, and sundrops. They are not closely related to the true primroses.
evening, giving plant name.
Figure I... Comparison of context documents sourced from GPT-4 and Wikipedia for the Wikipedia image. The GPT-4 generated context document is significantly longer and more detailed than the context document sourced from Wikipedia (via the WIT dataset).
19

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
question which references "classical approach", which is not described in the associated context document.
M. Synthetic Data Quality Control
As discussed in Section 4.5, we thoroughly evaluate our synthetic dataset from various dimensions through manual assessments, complemented by an array of filtering techniques and large-scale automated methods (e.g., grammar checks, bias detection, toxicity screening). We compare our data quality control strategy with other existing approaches that also generate and use synthetic data for training, and present the results in Table 11. As shown in Table 11, we invest significantly more effort in ensuring high-quality data than these prior works.
Table 11. Comparisons of Quality Control Methods in Synthetic Datasets (QA: randomly sampled Q&A pairs for manual labeling,
grammar: manual grammar review, factual: manual fact-checking using online sources).
Dataset Name
Human Evaluation Size
Quality Validation Control Methods
SK-VQA (ours)
200 (100 QA, 50 grammar,
heuristic filtering, downstream task evaluation,
50 factual)
toxicity/rrammarrbias detection with existing model downstream retrieval tasks evaluation
E5 (Wang al.,
N/A
2024a)
LLAVA (Liu et al., N/A 2024b)
downstream task evaluation
MiniGPT-4 et al., 2024)
(Zhu
N/A
heuristic filtering, downstream task evaluation
Owen- VL
(Bai
N/A
downstream task evaluation
al., 2023)
G-LLAVA (Gao N/A
downstream task evaluation
al., 2023)
N. LLM-as-Judge Evaluation Method Prompts
We present prompts used in LLM-as-judge methods in Table 12 and Table 13.
Table 12. Factuality Evaluation Prompt
Evaluation Instructions
You are evaluating the factuality of a deseription of image.
1. Look the image.
2. Carefully read the following description of the image.
3. Score the factuality of the description from 0 to 5:
-0 = Completely inaccurate
5 = Fully accurate and matches the image
If the score is less than 5, also identify which
specific sentences are inaccurate or misleading and list them in field called "errors".
Description:
[Context comes here]
O. Limitations and Ethical Considerations
Limitations In this work, we explored the generation of synthetic data from GPT-4 due to its demonstrated state-of-the-art performance in broad range of multimodal reasoning tasks. While our data generation approach could be used with other MLLMs, we leave the investigation of such applications to future work. Our dataset limited to English language QA pairs and context documents. While the images in our dataset were collected from a diverse range of sources, they may be
20

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs
Image source: Synthetic CPs)
Image source: Wikloedis
Contest source: GPI-4
Context source: GPT-4
Contect: Aviary birts are divens group hird species aviaries for observation conservation purposes. These
Context: Classic Men's Hairstyles in the 20th Century
known for their vibrant plumage melodic songs, making them popular among bird asts Common species found in avitries include various finches, parrots, and doves, among others...
During the 20th century, men's hairstyles saw sign siccaiant transformations that refiected
social and cultural chonges. One particulorly iconic style from the md-20th century was the neatly combed hairstyle often associated with formality and professioraliom
Popular Specias
Mid-20n Cantury Haisstyle
popular spacies of aviary birds include:
bleek Men's was typically styled with side part, using gel or pomace
hairstyle papular
characterized
combed appearance. the hair
Gouldian Finches, known for their striking patterns.
Zebra Finches, recognized for their distinctive calls.
place and achieve a gessy lock. This style exuded a sense of refinement and order. aligning with the fashion and cultural values of the time. it was often complemented by wearing formal attire such
Lovebirds, apprecated ated for their offectionate behavior.
Cockatiels, valued their playful nature and ability to mimic sounds.
and
Importance
Fechion Gultural Influince
Aviories serve not only as sanctuaries for birds but also as educational resources for the public They offer a unique osportunity to observe and learn about bird behavior,
the neat and orderly hairstyle represented more than just a grooming thoice; was
reflection of the cultural norms and values of the mid-20th century. This era emphasited...
Q: Name a popular specias of aviary birds. char cneracterieeddy striking color pattems.
Q: How would you describe the overall appearance of the haintyle the image?
Gouldian Finches
(Seck, glossy
(a) Answerable without image
(h) Answerable without context
Image LAION
source: GP
Context: Sunset marine landscapes been prominent subject for centuries, capturing the serene beautly of the sea as the sun dips below the harizon. Thiss pieces frequently hightgght the interplay of light and water, creating reflective surfaces that often onhance the tranquility the oranges, pinks, and purples regresent the fading doylight, contrasted against cooler tones of the approaching night sky and temperate waters. Historical examples of surset marine landscapes date back to the Romantic era, where the subime and emotional espects nature emphasioed, Renowned painters Monet, used such settings explore nature. Medeem interpretations continue to show an appreciation nautic elements, otten depicting sallboats, piers, and harbors against the backdrop of setting sun. These scones often evoka sense lsisure and calm, reminiscent of quiet evanings spant by the water, minimal
Artists typically
time, indlding J.M.W. Tumee and soltudee peace,
power
disturb natural How does modern interoretation theme compare the appreach? Min mai human presence
(c) Insufficient information to answer
Figure 12. Examples synthetic data generation failures noted by human annotators.
representative of all images domains which might be relevant to users of our dataset.
Due to the scale of our automatically constructed dataset, we are unable to fully validate the accuracy all examples that contains. We believe that our empirical results provide strong evidence of its quality and usefulness for training MLLMs. While human annotators did not explicitly validate the accuracy of all information contained in evaluated context passages, no obvious cases of hallucination were identified during the annotation process. However, the synthetic nature of the data introduces the possibility that it contains fallacies. Since our primary aim is to train MLLMs to ground generated answers in context documents, we believe such errors pose relatively low risk to the intended use of our dataset. Nevertheless, caution should be exercised when utilizing our dataset, including validation of the performance of any models which are trained on it.
21

---

SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Angmented Multimodal LLMs
Table 13. Quality Evaluation Prompt.
Evaluation Instructions
You are evaluating a visual question answering dataset.
Context (description of the image): [Context]
Question: [Question]
Proposed Answer:
[Answer]
Please rate:
Relevance of the question to the description (0-5)
2. Is the question clearly answerable based on the description? (Yes/No)
3. Is the answer factually correct based on the description? (Yes/No)
Ethical Considerations Our dataset was generated from GPT-4 using the Azure OpenAl API, which includes content filter for multiple risk categories (e-g., hate speech, fairness, sexual language, violence). As this filter automatically removes potentially offensive content that generated by GPT-4, we believe that the likelihood of our dataset containing such content relatively low. However, content filtering models not infallible and we unable to manually inspect our entire dataset for the presence of offensive content due to its large scale. It is also possible that potentially harmful biases possessed by GPT-4 which do not trigger content filters are reflected in our dataset. Users should carefully consider these risks relative to the benefits of our synthetic dataset before deploying systems which are built using it.
22


Reviews:
[{"review_id": "7uqKeqXQ33", "reviewer": "Reviewer_jWqo", "summary": "This paper introduces SK-VQA, a dataset with over 2 million question-answer pairs associated with context documents for training multimodal language models in knowledge-based visual question answering. Using GPT-4, the authors generated context documents and diverse QA pairs for images from varied sources, creating a dataset with 11× more unique questions than existing resources. Their experiments show that SK-VQA serves as both a challenging benchmark and effective training resource, with models trained on it demonstrating superior generalization in context-augmented settings compared to models trained on other datasets. This addresses a critical limitation of current multimodal LLMs which aren't designed for context-augmented generation in knowledge-intensive tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the strengths of our work, including the dataset’s scale and diversity, the quality-controlled generation process, and the robustness of our experimental design. We have carefully addressed all your concerns as follows:\n\n> **However, there is one major limitation: the human evaluation covered only 100 QA ...**\n\nWe appreciate the reviewer’s concern regarding the sample size of our human evaluation. We would like to highlight that our initial evaluation of 100 QA pairs already exceeds the human evaluation effort in prior synthetic dataset works (see Appendix L for a detailed comparison).\n\nTo further address this concern, during the rebuttal period we conducted an additional human evaluation on 100 new QA pairs using the same methodology described in Section 4.4. The results remained consistent with the original analysis, with a mean accuracy of 77.0%, reinforcing the reliability and representativeness of our evaluation.\n\n- For factuality, we asked the model to score the factuality of the description from 0 to 5: 0 = Completely inaccurate, 5 = Fully accurate and matches the image. The result shows the average score is 4.6, with 87.5% cases scored of “5”. \n- For the question relevancy, we asked the model to score the relevance of the question to the description and image (0–5). The result shows the average score is 4.9, with 92.0% is a score of 5. \n- For the question answerable, we ask the model is the question clearly answerable based on the description? (Yes/No). The result shows 99.6% questions are answerable. \n- For the answer correctness, we ask “Is the answer factually correct based on the description? (Yes/No)”. The result shows 90.7% questions are answerable. \n\nThese results, combined with our original and extended human evaluations, automated grammar checks (LanguageTool), fact-checking (via manual validation), bias/toxicity screening, and strong downstream task performance across multiple benchmarks, collectively demonstrate the high quality and utility of our dataset.\n\n> **Also, when testing with LLaMA-3-70b ... what percentage of questions are answerable ...**\n\nThank you for raising this point. To clarify: we applied LLaMA-3-70B-Instruct to QA pairs from the SK-VQA testing set, providing only the context document and question, without access to the image. Among these 26.5% are answerable, and we use the rest 73.5% which are answered incorrectly by the model as the hard subset. We will add this percentage in the paper.\n\n> **... The reviewer recommends the authors release the datasets and codebase upon acceptance.**\n\nWe fully agree, and we confirm that we will release the full SK-VQA dataset, as well as the code, upon acceptance of the paper.\n\n> **Table 6 caption misspelled: \"semantic matric\".**\n\nThank you for pointing this out. We will correct the typo in Table 6 and change \"semantic matric\" to \"semantic metric\" in the final version.\n\n> **... why factual accuracy is not a primary concern ...**\n\nWe would like to clarify that the term “factual accuracy” in our paper refers to the alignment of the context document with real-world facts—not the correctness of the answer with respect to the context. And this alignment with real-world knowledge is not necessary for our task because we aim to teach models to effectively utilize long multimodal contexts for grounded reasoning, rather than memorizing the context.  \n\nThat said, we still conducted fact-checking via manual verification as described in the paper. In addition, during the rebuttal, we applied the LLM-as-judge approach to assess factuality, question relevance, and answer correctness (the results are mentioned in the previous answer), all of which further support the high quality and utility of our dataset.", "claims_and_evidence": "The paper's central claims about SK-VQA's size, diversity, and performance improvements are well-supported by quantitative evidence. The dataset metrics showing 11× more unique questions than comparable datasets are documented in Table 2, while the performance advantages of models trained on SK-VQA are consistently demonstrated across multiple experiments in Figure 5 and Table 4. \n\nHowever, there is one major limitation: the human evaluation covered only 100 QA pairs (0.005% of the dataset), raising questions about the representativeness of the evaluated samples and therefore overall quality of the dataset.", "methods_and_evaluation_criteria": "Yes. Using GPT-4 to generate synthetic QA pairs and context documents addresses the scarcity of suitable training data, while the filtering techniques (IR and CAP) help ensure data quality. The evaluation metrics, such as BEM and exact match, are standard. The evaluation framework is comprehensive, examining both zero-shot performance on existing benchmarks and fine-tuning outcomes across multiple models, including out-of-domain generalization which is particularly relevant for real-world applications. The RAG experiments simulate practical use cases where retrieved knowledge must be integrated with visual information. The comparison against existing datasets (InfoSeek, Enc-VQA, ViQuAE) provides meaningful context.", "theoretical_claims": "No theoretical claims are provided in the paper.", "experimental_designs_or_analyses": "The zero-shot and fine-tuning evaluations use appropriate metrics and multiple model sizes, strengthening validity. The comparative analysis against InfoSeek, Enc-VQA, and ViQuAE provides the necessary benchmarking context. \n\nHowever, the RAG experiments use only one model architecture (PaliGemma-3B), limiting generalizability claims across architectures. Also, when testing with LLaMA-3-70b to create a \"hard\" subset, the authors don't clearly establish what percentage of questions are answerable by looking only at context (only provided the final number of samples), making it difficult to assess the true multimodal reasoning requirements of the dataset. Is it the case that only $2,853$ out of 2 million can be answered just using the context?", "relation_to_broader_scientific_literature": "The prior KB-VQA datasets seem to already be included in the paper.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strength: the diversity of the dataset is much greater than prior work -- over 96% of the questions in SK-VQA are unique, an 11x improvement than Enc-VQA; the questions in this work also have a greater number of unique POS sequences, total vocabulary size, and mean word length.\n\nWeakness: No assets were provided in the submission. The reviewer recommends the authors release the datasets and codebase upon acceptance.", "other_comments_or_suggestions": "Table 6 caption misspelled: \"semantic matric\".", "questions_for_authors": "From Lines 327 (left) - 287 (right), \"Factual accuracy is not a primary concern...  as its main purpose is to train MLLMs to effectively\nutilize long contexts for VQA ... specifically, we ask a native speaker to fact-check 50 QA pairs and supporting evidence in context documents using online sources. 86% were verified as factual, 4% were non-factual, 2% were partially factual, and 8% could not be determined due to a lack of available information.\"\n\nCould the authors further explain why factual accuracy is not a primary concern as a QA dataset? Noisy labels may undermine the performance from training / fine-tuning the MLLMs, and the 86% factual accuracy may hinder the practical usage of this dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7CO57Py51p", "reviewer": "Reviewer_nbTf", "summary": "This paper presents a large-scale synthetic dataset containing over 2 million visual questions with answers that require information from associated context. The images used in this dataset are from a hybrid of synthetic images from COCO-CFs and real images from Wikipedia and LAION, while the context and answers are generated from GPT-4. In the evaluation, they illustrate that this dataset can be used as a challenging benchmark of KB-VQA models and can also be effectively used as training data for multimodal context-augmented generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive review. We're grateful for your recognition of the strengths of our approach — particularly the scale and diversity of the dataset, our use of varied image and context sources, and the overall soundness of our methodology. We have addressed your concerns in detail as follows:\n\n> **One concern I have is regarding the quality of the question-answer pairs...**\n\nWe appreciate the reviewer’s concern regarding the sample size of our human evaluation. We would like to highlight that our initial evaluation of 100 QA pairs already exceeds the human evaluation effort in prior synthetic dataset works (see Appendix L for a detailed comparison).\n\nTo further address this concern, during the rebuttal period we conducted an additional human evaluation on 100 new QA pairs using the same methodology described in Section 4.4. The results remained consistent with the original analysis, with a mean accuracy of 77.0%, reinforcing the reliability and representativeness of our evaluation.\n\n- For factuality, we asked the model to score the factuality of the description from 0 to 5: 0 = Completely inaccurate, 5 = Fully accurate and matches the image. The result shows the average score is 4.6, with 87.5% cases scored of “5”. \n- For the question relevancy, we asked the model to score the relevance of the question to the description and image (0–5). The result shows the average score is 4.9, with 92.0% is a score of 5. \n- For the question answerable, we ask the model is the question clearly answerable based on the description? (Yes/No). The result shows 99.6% questions are answerable. \n- For the answer correctness, we ask “Is the answer factually correct based on the description? (Yes/No)”. The result shows 90.7% questions are answerable. \n\nThese results, combined with our original and extended human evaluations, automated grammar checks (LanguageTool), fact-checking (via manual validation), bias/toxicity screening, and strong downstream task performance across multiple benchmarks, collectively demonstrate the high quality and utility of our dataset.\n\n> **Another evaluation that is currently missing is the ablation of using different image sources...**\n\nWe thank the reviewer for the suggestion. In fact, we have already included this analysis in Table 5 and Table 8 of our paper.\n\nIn particular, Table 5 shows that models trained on synthetic images (COCO-CFs) paired with GPT-4 context perform on par with or better than those trained on real images (e.g., from Wikipedia). Table 8 further explores how filtering methods interact with image sources, showing that certain sources (e.g., LAION vs. Wiki) generalize differently across downstream tasks.\n\nThese results suggest that diverse image sources — including synthetic ones — contribute positively to generalization, and that combining them can be more effective than relying on a single source. We will make this clearer in the final version.\n\n> **...it is not clear to me whether the advantage is coming from the quality of the synthetic context and QA pairs, or just that the number of the contexts is increased.**\n\nTo ensure a fair comparison, all results in Table 5 were obtained using equal-sized subsets (downsampled to 200K training samples per setting). We will make this clarification more explicit in the final version of the paper.\n\n> **The title of Section 4.4 is Human Evaluation...consider using other names to avoid confusion.**\n\nThank you for the helpful suggestion. In the final version, we will keep 4.4. As Human Evaluation, and split the 4.4.2 to a new section titled “Automatic Evaluation”. \n\n> **The context in WIT dataset...What kind of contexts are you using to compare with the synthetic contexts?**\n\nThank you for pointing this out. For Wikipedia-based contexts, we use the paragraph-level context associated with each image from the WIT dataset. We will make this more clear in our final version of the paper.", "claims_and_evidence": "1. One concern I have is regarding the quality of the question-answer pairs. It seems that the only quality control in this paper regarding the validity of question-answer pairs is the human evaluation. While it is conducted on a very small scale (100 QA pairs). Therefore, it may be questionable whether the dataset can be used as a reliable benchmark to evaluate the KB-VQA models.", "methods_and_evaluation_criteria": "The proposed method is, in general, technically sound. By synthetically generating context and QA pairs, it can potentially increase the diversity of data when training the KB-VQA models. The evaluation of using different data sources at a similar scale as the training set and test on other datasets is also legitimate.\n\nAnother evaluation that is currently missing is the ablation of using different image sources. It is not clear what roles different image sources play in either training or evaluation. Especially in this paper, there are also some synthetic images. It would be meaningful to see some analysis in this direction.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Most of the experimental designs and analyses are sound to me. One issue I found is related to the impact of using generation source and real source for Wiki-based images (Table 5). It is not clear to me whether it is a fair comparison because from Table 1, we can find that the number of synthetic contexts is much larger than the number of real contexts from Wikipedia. So it is not clear to me whether the advantage is coming from the quality of the synthetic context and QA pairs, or just that the number of the contexts is increased.", "relation_to_broader_scientific_literature": "This proposed synthetic data generation method can be potentially adopted by other KB-VQA work to enhance the training data or as a challenging benchmark for evaluation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper presents an interesting way that additionally focuses on generating the context and then the question answer pairs, as well as using synthetically generated images.", "other_comments_or_suggestions": "The title of Section 4.4 is *Human Evaluation*, while some part of Section 4.4.2 *Additional Dataset Quality Evaluations* are automatic evaluation using LanguageTool. The authors may consider using other names to avoid confusion.", "questions_for_authors": "The context in WIT dataset is organized at different levels, for example, caption, paragraph or the whole Wikipedia page. What kind of contexts are you using to compare with the synthetic contexts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLF2AavcJn", "reviewer": "Reviewer_GcyS", "summary": "This paper provides and analyzes a new dataset called SK-VQA, which is a large-scale dataset designed to train multimodal language models for knowledge-based visual question answering with context augmentation.\n\nThe authors’ motivation is that existing datasets for this specific task do not cover large and diverse enough topics and questions. They leverage GPT-4 to produce synthetic data, resulting in a dataset with greater question diversity and broader knowledge coverage compared to previous resources. Evaluation demonstrates the proposed dataset could serve as a challenging benchmark and an effective training tool.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed review and your recognition of the dataset’s scale, diversity, and its potential to benefit context-aware multimodal research. We have addressed your concerns as follows:\n\n\n> **it is not clear why the results of training and testing with SK-VQA are missing in the figure 5**\n\nFigure 5 focuses only on out-of-domain generalization, where models are trained on one dataset and evaluated on the test sets of other datasets. This is why we did not include results where both training and testing are done on SK-VQA (i.e., in-domain performance).\n\nThe in-domain performance—where training and testing are on SK-VQA—is provided in Table 6 (Appendix A) for completeness. We intentionally separated in-domain and out-of-domain results to better highlight the generalization ability of each dataset.\n\n> **The quality of the generated context documents is not well assessed in terms of correctness, diversity, and completeness.**\n\n> **..the correctness of the generated document is not shown...**\n\nDuring the rebuttal time, we have added a new experiment based on LLM-as-judge (GPT-4o) to analyze the quality of the dataset. Specifically, we asked the model to check the factuality, question relevancy, question answerable, and answer correctness. \n\n- Factuality (0 = Completely inaccurate to 5 = Fully accurate and matches the image): the average score is 4.6, with 87.5% cases scored of “5”. \n- Question relevancy (0 = not relevant to image and context to 5 = relevant to image and context): the average score is 4.9, with 92.0% is a score of 5. \n- Question Answerable(Yes/No): 99.6% questions are answerable. \n- Answer correctness (Yes/No): 90.7% questions are answerable. \n\nAdditionally, we performed an additional 100 human analysis as in section 4.4, the results are consistent with the analysis in the Table 3 of the paper, reinforcing the reliability and representativeness of our evaluation. These new experiments plus the analysis in our paper for correctness (Table 3 for Human analysis and Section 4.4.2 for grammar), diversity (Table 2), completeness (above new LLM-as-judge analysis), and Bias and Toxicity (Section 4.2.2), show that SK-VQA undergoes richer validation, both automated and human, than most existing benchmarks (see Appendix L for a detailed comparison).\n\n> **I wonder if the GPT-4 generated dataset is qualified to evaluate other models...lacking some recent models, like Qwen-VL2, Ovis, and Molmo...**\n\nDuring the rebuttal, we evaluated more recent and powerful VLMs, including Qwen-2.5-VL (3B/7B/32B/72B) and Ovis (1B/2B/4B/8B/16B/34B). On SK-VQA, these models achieved:\n\n- Qwen-2.5-VL: 53.74, 49.26, 52.08, 49.09\n- Ovis: 32.25, 44.54, 50.55, 50.36, 52.36, 55.2\n\nFor comparison, on ViQuAE, the same Ovis models achieved:\n\n- Ovis: 39.50, 67.09, 49.38, 57.96, 72.77, 67.03\n\nThese results show that SK-VQA is a more challenging dataset compared to the existing ones. \n\n> **lack diversity in both the knowledge and the writing styles ... consistancy can also be guaranteed if we first prepare the context document from web or other resouces ...**\n\nWe address the two points as follows:\nOn diversity of generated contexts: We designed our pipeline and prompts as open-ended to promote both linguistic and knowledge diversity. As shown in Table 2, SK-VQA exhibits significantly higher diversity in POS patterns, vocabulary, and question structure than prior datasets. Figure 4 further shows that our context documents span 25+ knowledge domains, based on unsupervised topic modeling. Additionally, the use of diverse image sources (e.g., LAION, Wikipedia, COCO-CFs) ensures a wide range of visual prompts for generation, resulting in stylistic and content variation. The strong zero-shot difficulty across models (Table 4) also supports the idea that the data is not overly templated or repetitive.\n\nOn generating QA and context together: We agree that it is possible to first retrieve real documents and then generate QA pairs. However, our method generates context and QA jointly in one step, which allows us to explicitly control key constraints — such as ensuring the answer is only in the context (not the image), that object names are avoided, and that reasoning is required. This level of alignment is difficult to achieve when using unstructured web data, where the context may not be tailored to support the desired QA types.\n\n> **...it is also important to discuss the knowledge coverage...**\n\n We applied topic modeling and identified 25 distinct domains (Figure 4), showing broad topical diversity beyond entity-specific facts.\n\n > **What is the release plan of the proposed dataset?**\n\n We will publicly release the full SK-VQA dataset, along with the code, upon acceptance.", "claims_and_evidence": "There are three major claims made in this paper.\n1. Introduction of SK-VQA as a large-scale and diverse synthetic multimodal dataset for context-augmented KB-VQA: This claim is supported by its size when compared to other datasets.\n2. SK-VQA exhibits greater question diversity compared to previous datasets: It is supported by Table 2. It has significantly more questions than other KB-VQA datasets like InfoSeek and Encyclopedic-VQA.\n3. SK-VQA presents challenges on existing models, especially for zero-shot: This result should be supported by Figure 5. However, it is not clear why the results of training and testing with SK-VQA are missing in the figure. The author could split the proposed dataset into two splits and provide the results.", "methods_and_evaluation_criteria": "1. The proposed data synthesis method is targeted at addressing the data scarcity problem. Although the motivation is reasonable, the generation of context documents is questionable. The quality of the generated context documents is not well assessed in terms of correctness, diversity, and completeness.\n\n2. Another question is since both the context documents and QAs are all generated with GPT-4, I am worried that the quality of the dataset might not keep up with the current advanced multimodal LLMs. As little human effort is included in the loop during data synthesis, I wonder if the GPT-4 generated dataset is qualified to evaluate other models that already achieve better performance than GPT-4 on most benchmarks.\n\n3. The evaluation results look comprehensive while lacking some recent models, like Qwen-VL2, Ovis, and Molmo, etc. It would be more informative to readers if these methods could be included.", "theoretical_claims": "The paper is about proposing a dataset. There is no theoretical claim.", "experimental_designs_or_analyses": "1. The finetuned results of different datasets are not clear. In table 5, I feel the author is using split training and test sets of InfoSeek during training and test stages while using the same proposed SK-VQA both for training and testing. In that case, the author should make it clear in the paper.", "relation_to_broader_scientific_literature": "The dataset would be a choice of evaluating context-aware VQA models and might benefit the multimodal LLM society.", "essential_references_not_discussed": "The paper might lack some recent models, like Qwen VL2, Ovis, InternVL, Molmo, etc.", "other_strengths_and_weaknesses": "Weakness:\n1. One major concern is about how the accompanied document being generated in the proposed dataset. As stated in Section 3.1, the context document is generated with QA pairs from GPT-4 at the same time. However, as the context documents are generated from a single model using the same prompt, they might lack diversity in both the knowledge and the writing styles. This might make the synthesized dataset 'not real' compared to other multimodal RAG datasets.\n\nSimilarly, I am not conviced by the argument between L194 - 196. From my point view, the consistancy can also be guaranteed if we first prepare the context document from web or other resouces and then ask GPT-4 to prepare QA pairs based on the document.\n\n2. The quality assessment of the generated document is not comprehensive. The author discuss 'no obvious cases of hallucination were identified' in L856 of their supplemenraty. However, the correctness of the generated document is not shown or hard to evaluate based on the propsoed dataset creation pipeline. Besides, it is also important to discuss the knowledge coverage of the generated document.", "other_comments_or_suggestions": "What is the release plan of the proposed dataset?", "questions_for_authors": "I have no other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YRN68Qkqvq", "reviewer": "Reviewer_eTjf", "summary": "The paper proposes a dataset for visual question answering with external knowledge or for evaluating MLLM + RAG systems. \n\nThe dataset is constructed by using images from multiple datasets as seed images, and then writing context for those images using GPT-4o or using paired Wikipedia context when available, and then writing questions based on those contexts. Human analysis is done on the generated question-answer pairs. A number of quality control steps are performed.\n\nThe experiments section describes a series of experiments where transfer is measured from the proposed dataset to other datasets. \n\nThe proposed dataset is substantially larger than existing datasets, and training on the proposed dataset transfers well to other datasets. The dataset is of comparable difficulty to existing datasets w.r.t to both human and model evaluation. Quality control checks suggest that the level of noise in the dataset are low.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review and your recognition of the dataset’s quality, thoughtful design, large scale, and strong transfer performance. We have carefully addressed your comments and concerns as follows:\n\n> **...did not see any analysis done on how many model errors are the result of possibly noisy questions vs a genuine error...**\n\n> **...I don't see any evaluations of frontier models like GPT-4o on this dataset...***\n\nDuring the rebuttal period, we evaluated GPT-4o on the SK-VQA test set, and it achieved a score of 58.9%. We also conducted a manual error analysis on randomly sampled 50 examples that were marked incorrect by automatic evaluation. Our findings show that:\n\n\n- 40% were genuine model errors\n- 30% were partially correct\n- The remaining 30% were actually correct.\n\nWe will add these additional GPT4o’s results in our final version. \n\n\n> **...you should prepare a smaller evaluation split of the dataset that could be evaluated on in a smaller amount of time.**\n\nWe fully agree with the reviewer’s suggestion. To improve accessibility and encourage broader adoption, we will release a “-lite” evaluation split alongside the full dataset. This smaller subset will be designed to run efficiently on limited compute while preserving task diversity.\n\n> **What was your motivation for this work? In particular, can you point out the problems with InfoSeek or EncylopedicVQA that you were trying to solve with this paper? Are there new applications that SKVQA enables that were nontrivial to do with EncyclopedicVQA or InfoSeek? Concrete examples would be helpful...**\n\nWe appreciate this opportunity to clarify our motivation and contributions.\n\nOur goal was to address key limitations of existing KB-VQA datasets such as InfoSeek and Encyclopedic-VQA, which include:\n\n- Narrow image coverage: These datasets mostly include images that can be linked to Wikipedia entities (e.g., landmarks, animals), excluding everyday or abstract visuals. In contrast, SK-VQA includes images from LAION, Wikipedia, and COCO-Counterfactuals, enabling coverage of open-domain, synthetic, and artistic images.\n- Low question diversity: InfoSeek uses templated QA generation (e.g., “What is the capital of X?”), leading to <1% unique questions. SK-VQA uses GPT-4 to generate both context and QA pairs together, resulting in ~96% unique questions with richer phrasing (Table 2).\n- Limited knowledge types: Prior datasets focus on entity-centric facts. SK-VQA includes a broader range of topics (25 identified via topic modeling; Fig. 4), including art, cultural events, sports, and fashion.\n\nSome concrete examples:\n\n- Figure 1 (main paper): Shows a question about the Golden Globe Awards hosted at the Beverly Hilton — a cultural-event-centric question that would be unlikely in InfoSeek or Encyclopedic-VQA due to lack of coverage.\n- Figure 2: Demonstrates questions such as “What characteristic helps this breed adapt to cold water?” (about Labrador Retrievers) — combining visual traits with world knowledge, beyond simple object labels.\n- Appendix Figure 10: Compares a GPT-4 generated context about vineyards to a Wikipedia context about New World wines. The synthetic version better aligns with the image and supports diverse, image-grounded questions.\n\nNew Applications Enabled by SK-VQA:\n\n- Multimodal RAG Training & Evaluation: SK-VQA includes paired image, context, and QA for over 2M examples — a scale not offered in existing datasets — enabling training of models that retrieve and reason over context.\n- Fully-synthetic training and counterfactual reasoning: The inclusion of COCO-CFs and GPT-4 generated knowledge allows SK-VQA to support training and testing on hypothetical, non-real-world scenarios.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no proofs or theoretical claims.", "experimental_designs_or_analyses": "I checked all of $\\S5$. I do not see any issues.", "relation_to_broader_scientific_literature": "External knowledge visual question answering datasets started off small, with datasets like OK-VQA and A-OKVQA. These datasets have been superseded by more recent, larger datasets that are harder, like InfoSeek and EncyclopedicVQA. The proposed dataset is larger and appears to be higher quality than InfoSeek and EncylopedicVQA. I'm not sure if it is conceptually any different than EncyclopedicVQA and InfoSeek.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "The main strength is that this is a high quality dataset with a well-thought out design that appears to be more effective as a source of training data than previous datasets. The main weakness is that conceptually, it seems to offer nothing new (other than being larger + higher quality) than existing datasets.\n\nAdditional strengths:\n- The dataset contains nearly 50% more questions than the next largest dataset.\n- In addition to being larger, it is substantially more diverse (11x unique questions vs EncylopedicVQA).\n- There is a human evaluation performed and all humans perform similarly on the dataset.\n\nAdditional weaknesses:\n- Open-ended VQA datasets are known to contain high levels of noise. Specifically, for some questions the annotated answer might not be the only correct answer, or some questions might be ill-posed. I did not see any analysis done on how many model errors are the result of possibly noisy questions vs a genuine error. This could be done by evaluating a model on a subset and looking at a few errors (let's say 50-100). \n- I don't see any evaluations of frontier models like GPT-4o on this dataset. If this dataset is generated by GPT-4o, does that mean it is already \"solved\" by GPT-4o and cannot be used to evaluate GPT-4o. This is likely not the case for something like EncVqa, for which AFAICT even frontier models (at the time of evaluation) perform poorly.", "other_comments_or_suggestions": "The authors should have a \"-lite\" split of their dataset. This dataset is large, and to make it accessible to the community, you should prepare a smaller evaluation split of the dataset that could be evaluated on in a smaller amount of time.", "questions_for_authors": "1. What was your motivation for this work? In particular, can you point out the problems with InfoSeek or EncylopedicVQA that you were trying to solve with this paper? Are there new applications that SK-VQA enables that were nontrivial to do with EncyclopedicVQA or InfoSeek? Concrete examples would be helpful. If answered convincingly, I will raise my rating.\n2. I don't see any evaluations of frontier models like GPT-4o on this dataset. If this dataset is generated by GPT-4o, does that mean it is already \"solved\" by GPT-4o and cannot be used to evaluate GPT-4o? This is likely not the case for something like EncVqa, for which AFAICT even frontier models (at the time of writing) perform poorly, though they only evaluate on GPT-3. Should frontier models only be evaluated on the WiT split of the dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the rebuttal, all reviewers have raised their ratings to Accept (4), and all concerns are addressed by the authors. It is a clear accept paper, and it would be better to incorporate all rebuttal discussions into the camera-ready version.
Decision: Accept (oral)