PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024)

Abstract: Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pretrained version of the recently developed Finegrained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in generalpurpose multi-modal retrievers. The code, demo, dataset, and pre-trained checkpoints are available at https://preflmr.github.io/.

heading: 1 Introduction
text: Knowledge-based Visual Question Answering (KBVQA) systems generate answers to queries consisting of questions about given images. Correctly answering these questions requires accessing relevant world knowledge as well as vision and language understanding. Despite their demonstrated abilities in vision and language, recent Large Multimodal Models (LMMs) (Chen et al., 2023a; Driess et al., 2023; Liu et al., 2023a; Zhu et al., 2023; OpenAI, 2023) have performed poorly in recent challenging KB-VQA tasks (Chen et al., 2023b; Mensink et al., 2023a). One promising approach to improve their KB-VQA performance is RetrievalAugmented Generation (RAG), in which answer
*Weizhe Lin, Jingbiao Mei, and Jinghong Chen equally contributed to this work.
generation by LMMs is grounded in relevant documents retrieved from a knowledge base.
The best-performing document retrieval approach for KB-VQA to date is Fine-grained Lateinteraction Multi-modal Retrieval (FLMR) (Lin et al., 2023b). FLMR uses multi-dimensional embedding matrices to represent documents and queries and then efficiently computes their relevance scores via late-interaction (Khattab and Zaharia, 2020), thus capturing fine-grained relevance at the token level rather than at the passage level, as in Dense Passage Retrieval (DPR) (Karpukhin et al., 2020). As a late-interaction retriever, FLMR substantially outperforms DPR on a range of KBVQA tasks, with only minor speed penalties. In all of these methods, model and data size are important considerations. There has been much work in scaling up Large Language Models (LLMs) (Kaplan et al., 2020; Alabdulmohsin et al., 2022; Chen et al., 2023a) and text-based retrieval (Ni et al., 2022), but the scaling properties of these vision and language retrieval systems have not been studied. We therefore investigate the following three aspects of FLMR in KB-VQA.
(1) Vision & Text Encoding: We investigate how KB-VQA performance is affected by scaling the size and complexity of vision and text encoders.
(2) Pre-training: As originally formulated, FLMR employs simple, lightly trained Multi-Layer Perceptrons (MLP). We investigate whether gains can be had through more extensive model pretraining.
(3) Task Diversity: We gather nine open-source vision-language datasets into a suite of benchmark tasks, M2KR, for assessing Multi-task Multi-modal Knowledge Retrieval. M2KR encompasses Imageto-Text, Question-to-Text, and Image&Questionto-Text retrieval tasks, and also includes prompting instructions that can be provided to an LLM for each of the component tasks. General purpose multi-modal retrieval models can be created by
5294
training on the entirety of the M2KR training data and these models can then be evaluated on any or all of the included tasks. Models can further be fine-tuned for specific M2KR tasks using the task-specific tuning data included in the collection.
We show that M2KR can be used in training an FLMR-based RAG LLM for multi-task multimodal retrieval. We refer to this model as PreFLRM (for Pre-trained FLMR). PreFLMR can be used directly in its pre-trained form for multi-task multi-modal retrieval. PreFLMR can also be finetuned for specific task-specific performance. In both uses we find that PreFLMR gives us substantial gains across the M2KR tasks.
Contributions of this paper are:
• The M2KR task suite encompassing nine datasets and three types of retrieval tasks for training and evaluating general-purpose visionlanguage retrievers. We create M2KR by repurposing various vision and language data sets that might not be originally created for knowledge-based visual question answering, thus ensuring a rich and diverse collection.
• PreFLMR, a strong multi-modal retriever pretrained on a vision-language corpus of over ten million items. We show that PreFLMR performs well across a range of knowledge retrieval tasks when given the appropriate instructions. We will release PreFLMR upon publication.
• A study of the scaling behaviour of FLMR in terms of its model parameters and training data. To our knowledge, this is the first systematic study of scaling in late-interaction based visionlanguage retrievers and should provide empirical guidance for future work.

heading: 2 Related Work
text: Document Retrieval. DPR has become a cornerstone in knowledge-intensive tasks (Chen et al., 2017; Izacard and Grave, 2021; Guu et al., 2020; Lee et al., 2019; Lewis et al., 2020) as well as in KB-VQA tasks due to its fast and precise retrieval capabilities (Karpukhin et al., 2020; Gui et al., 2021; Luo et al., 2021; Lin and Byrne, 2022; Wu and Mooney, 2022). Recent developments in retrieval methods, particularly Late Interaction models (Khattab and Zaharia, 2020; Santhanam et al., 2022b), have shown notable performance gains over DPR, albeit with some efficiency tradeoffs (Lin et al., 2023a,b). In multi-modal retrieval,
FILIP (Yao et al., 2022) used pre-trained late interaction models for single-modal image-text retrieval, while FLMR (Lin et al., 2023b) extended the approach to multi-modal retrieval for KB-VQA with finer-grained visual and text features. This paper further extends FLMR and explores its scaling properties in multi-modal retrieval. Similar to our M2KR benchmark, A concurrent work (Wei et al., 2023) introduces M-Beir, which combines several retrieval tasks and can be used to train and evaluate universal multi-modal retrievers.
Another line of relevant research is KB-VQA retrieval involving Named Entities, where retrieved documents must identify the person in the image. For example, on ViQuAE (Lerner et al., 2022), Lerner et al. (2023) trains the retriever with a multimodal inverse cloze task, while Lerner et al. (2024) shows that combining mono- and cross-modal retrieval improves performance. Both use a weighted sum of BERT (Devlin et al., 2019) and CLIP (Radford et al., 2021) embeddings, while our work trains a single multi-modal late-interaction retriever. Knowledge-based VQA Systems. Recent multimodal systems have significantly improved in complex tasks like OKVQA (Schwenk et al., 2022) that require external knowledge sources (Narasimhan et al., 2018; Garderes et al., 2020; Li et al., 2020; Wu et al., 2022; Marino et al., 2021; Chen et al., 2023d; Gao et al., 2022; Gui et al., 2021; Hu et al., 2023b; Rao et al., 2023). Systems like KAT (Gui et al., 2021) and REVIVE (Lin et al., 2022) used LLMs (e.g. GPT-3) for generating candidate answers. Challenges remain in answering more knowledge-intensive questions (Chen et al., 2023b; Mensink et al., 2023a), underscoring the need for robust document retrieval. Mensink et al. (2023a) showed that even state-of-the-art LLMs perform poorly on difficult KB-VQA questions, with an accuracy of under 20% when retrieval is not incorporated. RA-VQAv2 (Lin et al., 2023b) and prior work (Lin and Byrne, 2022; Luo et al., 2021; Qu et al., 2021; Gao et al., 2022; Hu et al., 2023b; Mensink et al., 2023a) demonstrated strong performance in KB-VQA by using external knowledge databases. Scaling Retrieval Systems. Previous work has explored scaling laws in language/vision systems (Kaplan et al., 2020; Alabdulmohsin et al., 2022), revealing correlations between model performance, computation, number of parameters, and dataset sizes. In retrieval, Ni et al. (2022) and Hu et al. (2023b) both observe improvements in DPR-like
models with one-dimensional embeddings by increasing the size of language/vision encoders. This paper reports similar scaling investigations in multimodal late-interaction retrieval.

heading: 3 The M2KR Benchmark Suite
text: Current multi-modal retrievers are typically trained and evaluated on a single dataset only. To properly study general-purpose multi-modal retrievers, we introduce the Multi-task Multi-modal Knowledge Retrieval (M2KR) benchmark suite. We convert nine diverse datasets, originally designed for vision and language tasks such as image recognition, image captioning, and conversational interactions, into a uniform retrieval format. Details of the preprocessing steps, data partition, and prompting instructions are provided in Appendix A, but we note here that re-purposing these datasets into a single consistent collection for knowledge-based visual question answering represents a non-trivial effort. M2KR will be released with our models.

heading: 3.1 Tasks and Datasets
text: Table 1 shows the composition of M2KR. We preprocess the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The M2KR benchmark contains three types of tasks:
Image to Text (I2T) retrieval. These tasks evaluate the ability of a retriever to find relevant documents associated with an input image. Component tasks are WIT (Srinivasan et al., 2021), IGLUEen (Bugliarello et al., 2022), KVQA (Shah et al., 2019), and CC3M (Sharma et al., 2018). CC3M is included in the M2KR training set to improve scene understanding but not in the validation/test set as the task concerns caption generation, not retrieval. The IGLUE test set, which is a subset of WIT and has an established benchmark, is included to enable comparison with the literature. The KVQA task, initially designed as a KB-VQA task, has been re-purposed into an I2T task for our modelling purposes (Appendix A.1.3).
Question to Text (Q2T) retrieval. This task is based on MSMARCO (Bajaj et al., 2018) and is included to assess whether multi-modal retrievers retain their ability in text-only retrieval after any retraining for images.
Image & Question to Text (IQ2T) retrieval. This is the most challenging task which requires
joint understanding of questions and images for accurate retrieval. It consists of these subtasks: OVEN (Hu et al., 2023a), LLaVA (Liu et al., 2023b), OKVQA (Schwenk et al., 2022), Infoseek (Chen et al., 2023c) and E-VQA (Mensink et al., 2023b). We note in particular that we convert LLaVA, a multi-modal conversation dataset, into a multi-modal retrieval task (Appendix A.3.1).
The training/validation/test examples are downsampled from the respective sets of the original datasets. We take test examples from the original validation sets for LLaVA and Infoseek since LLaVA has no test sets and the test set annotation of Infoseek has not been released. We limit the maximum test samples to 5,120 for each dataset to allow faster performance tests on all 9 datasets. Data preprocessing and partitioning details are in Appendix A. We further verified that there are no identical images between the training and test sets by checking the MD5 of the images, thereby preventing data contamination during training. We use the validation splits to select hyperparameters for the models which can be found in detail in Appendix B.2

heading: 3.2 Evaluation
text: We use Recall@K (R@K), which measures whether at least one of the target documents is in the topK retrieved entries, to evaluate retrieval performance. Additionally, for the datasets Infoseek, EVQA, and OKVQA, we mainly employ Pseudo Recall/PRecall@K (PR@K) for evaluation. This metric measures whether at least one of the top K documents includes the target answer.1
1In practice, PR@K more accurately reflects actual retrieval performance and exhibits a stronger correlation with
We use R@10 for WIT and MSMARCO, and R@1 for LLaVA and IGLUE. Other datasets are evaluated with R@5 or PR@5. As in Table 2, we also report the average rank (A.R.) of each model over all datasets to indicate multi-task retrieval performance relative to other models in comparison; lower is better.

heading: 3.3 Baselines
text: For each dataset, we show the best published results in recent literature as baselines, if available (Table 2). For datasets without previous results such as LLaVA and OVEN, we use our replication of CLIP (Radford et al., 2021) and FLMR as zero-shot baselines following Lin et al. (2022).

heading: 4 PreFLMR Architecture and Training
text: Our architecture generally follows that of FLMR (Lin et al., 2023b) as shown in Fig. 1. PreFLMR uses token embedding matrices Q and D to represent query and document, respectively. Given a query q̄ consisting of texts q and an image I , PreFLMR uses a language model FL to obtain embeddings of all tokens in q, a vision model FV to obtain embeddings of I , and a mapping structure FM to project image embeddings into the text embedding space. All token-level embeddings are concatenated to form the query representation Q. The document matrix D is obtained similarly with the language model FL but without visual features.
The relevance score r(q̄, d) is computed via lateinteraction (Khattab and Zaharia, 2020) between Q and D, aggregating the maximum dot products over all query tokens with respect to all document tokens (Eq. 9). lQ and lD denote the total number of tokens in query q̄ and document d, respectively.
r(q̄, d) =
lQ∑
i=1
lD max j=1 QiD ⊤ j (1)
PreFLMR improves over FLMR in the following aspects: (1) While FLMR only uses the [CLS] embedding from ViT as the image representation, in PreFLMR we additionally extract embeddings of image patches from ViT’s penultimate layer to obtain a detailed visual representation. (2) We introduce Transformer blocks with cross-attention into the mapping structure to obtain query-aware visual
the ultimate VQA performance. This is because document annotations are frequently incomplete, and alternative documents within the corpus can often provide answers to the questions.
representation. The Transformer blocks take the image patch embeddings as input, and use crossattention to integrate the features of the text encoder. This allows PreFLMR to attend to different aspects of the image under different queries. These Transformer blocks are placed in parallel with FLMR’s 2-layer MLP mapping structure. (3) We append task-specific instructions to the text query to distinguish between tasks. The list of instructions for each task can be found in Appendix A. For each query, the instruction is randomly sampled from the corresponding instruction list. Instruction tokens are masked in computing relevance score. For Q2T retrieval training, we feed a blank image as PreFLMR’s image input. For I2T retrieval training, we use instructions as text input to PreFLMR.
PreFLMR training and inference follow that of FLMR. When training on data consisting of several datasets, we randomly shuffle the entire training data and only use in-batch negative examples from the same corpus. Post-training, all documents are indexed through PLAID (Santhanam et al., 2022a) for efficient late-interaction retrieval. For detailed evaluation of retrieval efficiency, we refer readers to Lin et al. (2023b).
The detailed formal expression of the entire model can be found in Appendix B.4.

heading: 4.1 Training Procedures
text: PreFLMR’s pre-training involves four stages.
Stage 0: Text Encoder Pre-training. We train ColBERT following Khattab and Zaharia (2020) on the MSMARCO dataset to obtain the initial checkpoint for PreFLMR’s text encoder FL. This is a straightforward replication of ColBERT used as an initial text encoder as was done in FLMR, but also allowing for size variations.
Stage 1: Training the Mapping Structure. In this stage, we only train the mapping structure FM , keeping the language and vision models frozen. This approach is an extension of the FLMR methodology, incorporating a larger dataset and an additional cross-attention mapping layer. The training is performed on the IQ2T dataset (LLaVA, OVEN), I2T datasets (WIT, CC3M, KVQA), and Q2T dataset (MSMARCO). Our objective is to encompass all three task types in M2KR without the need to optimize the data mixing ratio or manually select datasets to achieve an effective mapping structure. This strategy is inspired by pre-
vious studies (Lin et al., 2023b; Zhu et al., 2023; Liu et al., 2023b), which utilized relatively simple multi-modal tasks to develop image-to-text mappings.
We mask the late-interaction token embeddings in query matrix Q that are produced by the language model (not the token embeddings at the input embedding layer). This encourages the Transformer cross-attention layer to integrate information from its textual inputs and enables PreFLMR to perform IQ2T, I2T, and Q2T retrieval when provided with the appropriate instructions for each task.
Stage 2: Intermediate KB-VQA Pre-training. We tune the text encoder FL and the mapping structure FM on the E-VQA dataset, a large and high quality KB-VQA dataset, to enhance PreFLMR’s retrieval performance. Including an intermediate pre-training stage to align the model with in-domain data has been well-explored in the literature (e.g., Google’s TAPAS (Eisenschlos et al., 2020)). We opt for a straightforward procedure to train on E-VQA in the intermediate stage because of its diversity, increased difficulty, and larger quantity compared to other KB-VQA datasets. Specifically, E-VQA requires recognition of less common entities such as spotted hyenas and relies on more specialized domain knowledge such as American landmarks, making it good for retrieval training. This design choice is well-supported by experimental results (Table 2 #8 vs #5, #3 vs #2) and we
provide detailed analysis in Sec. 5.6.
Stage 3: Full-scale Fine-tuning. We train on the entire M2KR corpora, including OKVQA and Infoseek. This stage is straightforward multi-task learning. We tune the entire model except the vision encoder FV . We adjust the dataset proportions to ensure balanced learning on these datasets of varying sizes (Appendix B.1). Additionally, we use separate text encoders to encode queries and documents; their parameters were shared in previous steps.

heading: 4.2 Training Configurations
text: We use the Adam optimizer (Kingma and Ba, 2015) with a fixed learning rate of 10−4 for the mapping structures and 10−5 for other parameters in all experiments. Training was run up to 300k, 220k, 12k, and 50k steps in the four stages, respectively. Full training configurations (including the hyperparameters for downstream VQA fine-tuning) can be found in Appendix B.2.

heading: 5 Experiments and Results
text: In this section we present results of scaling PreFLMR components (Sec. 5.2, 5.4), analyze the effect of each training stage (Sec. 5.3, 5.6), and evaluate on the downstream KB-VQA tasks (Sec. 5.5). We summarize our findings in Sec. 5.7. Multi-task performance refers to PreFLMR results, i.e. Stages 0, 1, 2, and 3, without any single-task fine-tuning.

heading: 5.1 Model Variants
text: We experiment with a range of model configurations. Model sizes range from BERT-Small (28.8M), BERT-Medium (41.1M), BERT-Base (110M) to BERT-large (340M). ColBERT text encoders are denoted as "[BERT size]-[pre-training scheme]". There are two ColBERT pre-training schemes: “v1” (Khattab and Zaharia, 2020) and “v2” (Santhanam et al., 2022b). “v2” yields a better performing model than “v1” as evaluated on MSMARCO. We compare models initialized from “v1” and “v2” checkpoints to investigate how the performance of the initial uni-modal text retriever affects the final multi-modal vision-language retriever. Except for “Base-v2”, all ColBERT variants are trained using our replication of ColBERT following the “v1” pre-training scheme.2 For the vision encoders, we use the ViT variants: ViT-B (88M), ViT-L (303M) (Radford et al., 2021), ViT-H (631M) and ViT-G (1.84B) (Cherti et al., 2023).

heading: 5.2 PreFLMR Performance
text: The best-performing PreFLMR model (ViT-G + Base-v2) outperforms other variants on most of M2KR benchmark (Table 2, #13). Without singletask fine-tuning, PreFLMR outperforms baseline models optimized for the individual tasks on 7 out of 9 datasets, showcasing its capability as a general visual-language retriever. We now analyze how each PreFLMR component affects performance.
Vision Encoder Scaling. Scaling ViT from ViTB (86M) to ViT-G (1.8B) while keeping the text encoder fixed brings about substantial performance gain across all tasks (Table 2 #2, #5, #12, #13), e.g. 48.8 to 59.6 on Infoseek and 67.9 to 73.1 on E-VQA. The gain is greater when upgrading ViT-B to ViT-L with recall improvements of ∼ 10% on WIT, KVQA, OVEN, and Infoseek, showing the benefit of using better vision encoders. In addition, Fig. 3 in the appendix illustrates performance gains in scaling the vision encoder with a radar plot. However, the performance plateaus when scaling ViT to H and G. This observation aligns with results reported in the literature. OpenCLIP (Cherti et al., 2023) and BLIP2 (Li et al., 2023) have reported marginal or no performance improvement when scaling beyond ViT-L across several datasets. A plausible explanation is that if the ViT model is not pre-trained on domain-specific data, it may struggle to make fine distinctions.
2The training code of “v2” has not been released officially.
Text Encoder Scaling. Scaling up the text encoder from BERT-Small-v1 to Medium-v1 to Basev1 (Table 2 #9, #10, #4) yields substantial performance gain (A.R. 8.3, 8.2, and 5.6). However, we find that further scaling to Large-v1 (#11) adversely impacts the performance (A.R. decreased to 6.6). We attribute this to overfitting and unstable training for large models given the available data (Appendix B.3). The results suggest that BERTBase (110M) is adequate for building a capable vision-language retriever.
Improving Text Encoder. Compared to PreFLMR models initialized from Base-v1, models initialized from Base-v2 have better multi-tasking performance indicated by better A.R. (Table 2 #1 vs #2 and #4 vs #5). The gain from improving the text encoder is more substantial when using the “ViT-L” vision model (-2.4 A.R.) compared to using “ViT-B” (-0.8 A.R.), indicating that the text encoder is relatively weak as the vision model improves.

heading: 5.3 Performance of Each PreFLMR Stage
text: In this section, we analyze intermediate performance in the earlier stages of pre-training to better understand the scaling behaviour of PreFLMR.
Text Encoder Pre-training. We train “ColBERTv1” at different sizes and evaluate on the MSMARCO dataset. Table 3 shows larger model sizes consistently yield better text retrieval performance. In contrast to the multi-modal case, scaling up to “Large-v1” does not destabilize training and leads to better performance compared to “Base-v1”.
Training the Mapping Structures. Table 4 details system performance after Stage 1 training, in which only the vision-language mapping structure is trained. Similar to Sec.5.2, scaling up the vision encoder improves performance across tasks. PreFLMR exhibits strong zero-shot KB-VQA performance at this preliminary stage (50.87 in Infoseek, 42.44 in E-VQA, and 52.14 in OKVQA). After Stage 1, PreFLMR with ViT-G performs worse than other variants on IGLUE, E-VQA and OKVQA. However, it attains the best performance on these datasets after Stage 3. This suggests that tuning the mapping structure alone is not enough to fully utilize larger vision models.
Intermediate Pre-training. Stage 1 improves performance on KB-VQA tasks (Table 2 #3 vs #2
and #8 vs #5). Although we are only training on EVQA, the score on other KB-VQA tasks (Infoseek, KVQA, OKVQA) increases by ∼1% or more. This shows that E-VQA is an appropriate corpus for training a general-purpose knowledge retriever. We analyze the gain from intermediate pre-training in more detail in Sec.5.6.

heading: 5.4 Ablation Studies
text: Instructions. Removing instructions (Table 2 #7) results in much worse overall performance, with the WIT recall rate reduced to 13.3. This shows that instructions are necessary for multi-task learning and that our instruction scheme works well (the full list of instructions is given in Appendix A).
Pre-training Datasets. As shown in Table 5, adding CC3M to training improves performance on all metrics, showing that learning to understand scene via captioning datasets is beneficial. Removing either LLaVA or MSMARCO harms zero-shot KB-VQA performance (−3.0 in Infoseek), noting
3The performance is not fully comparable due to differences in the construction of the test passage corpus and the proprietary nature of the data and pipeline used in Lens. The reported figures serve as a reference point.
that Infoseek is not used in this stage. Training on these datasets facilitates learning question-aware visual representations as the cross-attention in the mapping structure must attend to the text input to perform well on these tasks. Omitting knowledgeintensive datasets (OVEN and KVQA) negatively impacts the zero-shot performance on Infoseek, showing the importance of in-domain data.
Mapping Structure Scaling. Table 6 illustrates the impact of scaling up the mapping structure under two PreFLMR configurations. Increasing crossattention layers from 1 to 4 marginally improves LLaVA performance (+0.5, approx.), but adversely
impacts performance on WIT (-4, approx.) and Infoseek (-2, approx.). We adhere to the 1-layer design, noting that adding parameters to the mapping structure does not improve performance.

heading: 5.5 Retrieval Augmented Visual Question Answering with PreFLMR
text: We build on RA-VQAv2 (Lin et al., 2023b), a strong retrieval-augmented visual question answering system to tackle OKVQA, Infoseek, and E-VQA. We fine-tune the best-performing PreFLMR variant on the target retrieval task (ViT-G + Base-v2, Table 2 #14) and follow RA-VQAv2 to fine-tune a BLIP-2 answer generator on the target
450.7 for Unseen Entity and 56.4 for Unseen Question; no overall accuracy is reported.
M2KR KB-VQA task.5 Following previous literature (Schwenk et al., 2022; Chen et al., 2023c; Mensink et al., 2023b), we use VQA score, Accuracy, and BERT matching (BEM) (Bulian et al., 2022) to evaluate performance on OKVQA, Infoseek, and E-VQA, respectively.
A brief summary of the systems shown in Table 7: PaLM-E (Driess et al., 2023), PALI-X (Chen et al., 2022) and PaLM-B (Anil et al., 2023) are large multi-modal models with 562B, 55B, and 1T parameters, respectively. The E-VQA SOTA (Mensink et al., 2023b) uses Lens (Google), the Google API for image retrieval. AVIS (Hu et al., 2024) is a hybrid system with many components (such as PaLI, PaLM, and Google Lens&Web Search API) and planning stages powered by LLMs. We note that PreFLMR could be used as part of the AVIS pipeline to enhance its ability to fetch relevant documents given questions and images.
As shown in Table 7, compared to models without retrieval, PreFLMR improves performance by approximately 6% on OKVQA, 9% on Infoseek, and 34% on E-VQA. These results highlight the effectiveness of PreFLMR in document retrieval for KB-VQA tasks.
On OKVQA, the performances of RA-VQAv2 (PreFLMR) and RA-VQAv2 (FLMR) are similar. Table 2 #13 shows that PreFLMR attains similar Recall@5 as FLMR on OKVQA even though it has a much larger vision encoder. As a possible explanation, compared to E-VQA and Infoseek, the knowledge required to answer OKVQA question is less specialized and many OKVQA questions can be answered without document retrieval (Mensink et al., 2023b). See Appendix E for qualitative analysis.
5We note that this work was conducted during the early stage of the release of Infoseek and E-VQA. We prepared the data splits according to the need for retrieval training following Appendix A. The systems are trained and evaluated on the data splits provided in M2KR to show the improvement relative to systems without retrieval.
Another possibility is that, compared to E-VQA and Infoseek where the ground-truth document is provided for each question, the OKVQA training set does not provide ground-truth knowledge documents. The retriever uses pseudo-relevant documents in training that contain the target answer but these may not be truly useful for answering the question. This is evidence that data quality should be improved along with model scaling.

heading: 5.6 Analysis of Intermediate Pre-training
text: Sec. 5.3 shows that Stage 2 Intermediate Pretraining improves the performance as evaluated by task-specific metrics. In this section, we further quantify the gains from Stage 2 for each dataset and more clearly show that KB-VQA tasks benefit more from Stage 2 than other tasks. We use the difference in minimal validation loss6 achieved on each dataset starting from checkpoints before or after Stage 2 Intermediate Pre-training as a measure of benefit. This enables comparison of tasks with different performance metrics. Intuitively, a larger absolute difference in validation loss indicates that the dataset benefits more from the Intermediate Pre-training stage.
Figure 2 plots the difference in validation loss of every dataset when the starting checkpoints have undergone Ninter intermediate pre-training steps using either BERT-medium or BERT-base as the text encoder backbone. As expected, starting from E-VQA-pre-trained checkpoints yields lower validation loss in knowledge-intensive tasks such as OKVQA, KVQA, and OVEN after the same number (5,000) of fine-tuning steps. Performance on
6We find that the validation loss is predictive of the actual performance. A lower validation loss usually suggests a better performance in the tasks that we study.
these datasets indeed sees more gain from Stage 2 training (Table 2, #5 v.s. #8). Figure 2 also indicates the existence of an optimal Ninter, beyond which the model overfits to E-VQA, harming performance on other datasets. The larger PreFLMR model with BERT-base text encoder overfits faster than PreFLMR with BERT-medium (Ninter ≈ 15, 000 versus Ninter ≈ 10, 000). We use V-Entropy (Xu et al., 2020) to formalize our analysis as an empirical measure of mutual information between datasets in Appendix D.

heading: 5.7 Summary of Findings
text: We summarise the results of our investigations into scaling behaviour as follows:
• The text encoder size need not exceed that of BERT-base (110M) to achieve competitive multi-modal retrieval performance (Sec.5.2).
• Scaling up the vision encoder from ViT-B to ViT-G yields substantial gains (Sec.5.2).
• Scaling up the mapping structure does not improve performance (Sec.5.4).
• Intermediate pre-training on high-quality indomain data (E-VQA) effectively improves retrieval performance across KB-VQA tasks (Sec.5.3, 5.6).
• Strong knowledge retrievers boost performance on challenging KB-VQA tasks such as OKVQA, Infoseek, and E-VQA via RetrievalAugmented Generation (Sec.5.5).
• Ground-truth document labels are important to make full use of large models in training multimodal retrievers (Sec.5.5).

heading: 6 Conclusion
text: This work has studied the scaling behaviour of state of the art multi-modal document retrieval systems, with a focus on enhancing fine-grained lateinteraction retrieval for knowledge-based visual question answering. We contribute a comprehensive training and evaluation framework, M2KR, for general-purpose multi-modal knowledge retrieval. The PreFLMR system we train in the M2KR framework yields excellent retrieval performance across a range of tasks and can also serve as a base for further task-specific fine-tuning.
Limitations
Limited by available computational resources, we leave several further investigations as future work: (1) The CLIP-ViT models (Cherti et al., 2023) were not pre-trained on in-domain data of knowledgeintensive tasks. Further training may enhance the model’s ability to recognize a broader range of objects; (2) Advanced training approaches beyond contrastive learning, such as score distillation (Santhanam et al., 2022b), could be explored to further enhance retrieval performance; (3) Investigating a more optimal mix proportion of datasets with varying sizes also warrants further exploration.
Ethics Statement
Our proposed model retrieves documents without generating new content. We acknowledge the potential for the retrieved documents to include inappropriate information if the document database lacks adequate filtering. Consequently, extra care must be taken to ensure the sanitization of the document database, particularly when employing this model in applications involving direct interaction with real users.

heading: Acknowledgments
text: This work was supported in part by the AWS Cloud Credit for Research programme.
Weizhe Lin is supported by a Research Studentship funded by Toyota Motor Europe (RG92562(24020)) for the undertaking of the PhD in Engineering at the University of Cambridge.
Jingbiao Mei is supported by Cambridge Commonwealth, European and International Trust for the undertaking of the PhD in Engineering at the University of Cambridge.
Jinghong Chen is supported by the Warwick Postgraduate Studentship from Christ’s College and the Huawei Hisilicon Studentship for the undertaking of the PhD in Engineering at the University of Cambridge.
Prof. Bill Byrne holds concurrent appointments as a Professor of Information Engineering at Cambridge University and as an Amazon Scholar. This publication describes work performed at Cambridge University and is not associated with Amazon.
We would also like to thank all the reviewers for their knowledgeable reviews.

heading: A Datasets details
text: This section outlines the preprocessing methods used to convert various datasets into formats suitable for retrieval tasks. Table 8 provides examples from each dataset, demonstrating the transformation from their original to the adapted structure. Subsequent subsections detail the specific preprocessing steps for each dataset. The M2KR dataset is available at Huggingface Hub.
A.1 I2T Retrieval
A.1.1 WIT WIT (Srinivasan et al., 2021) is a corpus based on Wikipedia with image-text pairs, where the text is the Wikipedia passage associated with the image. To enhance data quality, we exclusively select image-text pairs where the images are the main/title images of their respective Wikipedia documents, and we limit our scope to English-language documents.
Our training set, comprising 2.8 million examples, is sourced from the original WIT training set. 20,102 and 5,120 examples from the original WIT validation set are selected to build the validation set and test set in our M2KR benchmark, respectively. The test corpus includes all documents from the original WIT validation and test sets. This setting ensures that there is no overlap between different sets.
Each image-document pair is paired with a randomly selected instruction from our set of templates. The task is to retrieve the correct document from the test corpus, given the image and instruction.
A.1.2 IGLUE The IGLUE English retrieval test set (Bugliarello et al., 2022), which is a subset of the WIT test set and has an established benchmark for imageto-text retrieval, is included to enable comparison with models in previous literature. Following Bugliarello et al. (2022), the test set contains 685 unique images and 1,000 Wikipedia passages. The task is similar to WIT: using the image and the instruction to retrieve the corresponding Wikipedia passage. Instruction templates for WIT and IGLUE:
• <Image> Identify the document that is connected to this image.
• <Image> Provide information about the document linked to this image.
• <Image> Please describe the document that corresponds to this image.
• <Image> What is the document that this image is related to?
• <Image> Could you elucidate the document associated with this image?
• <Image> Describe the document that accompanies this image.
• <Image> Please give information on the document that goes with this image.
• <Image> What document is represented by this image?
• <Image> Identify the document that this image pertains to.
A.1.3 KVQA KVQA (Shah et al., 2019) is a dataset containing a rich collection of entities representing famous individuals. The KVQA task, initially designed as a KB-VQA task, has been re-purposed into an I2T task for our modelling purposes. This adaptation is based on our findings that using images as queries alone suffices to retrieve the documents containing the correct identities. In our context, where the primary focus is on document retrieval, the original questions are unnecessary. Our reformulated task for KVQA is to retrieve the details of famous people like gender, nationality, birthplace, and employment history based solely on their images. The training set is downsampled from the KVQA original training set by removing repeated examples of the same famous individuals. We transformed the structured entities such as gender and nationality into passages. For example, “nationality: America; date of birth: dd/mm/yyyy; ...” is serialized as “nationality is America, date of birth is dd/mm/yyyy, ...”.
The training corpus is composed of all the documents that appear in the original KVQA training set. For the validation/test set, we selected a subset of 13,365/5,120 samples from the original KVQA validation set. Correspondingly, the test corpus encompasses all documents found in the original KVQA validation set.
The instruction we use for KVQA is: <Image> Provide a brief description of the image and the relevant details of the person in the image.
A.1.4 CC3M CC3M (Sharma et al., 2018) is a dataset consisting of a vast collection of image-caption pairs. Instead of utilizing the entire dataset comprising 3 million
pairs, we adopt the downsampling methodology as delineated in LLaVA’s work (Liu et al., 2023b), resulting in a reduced dataset of approximately 595K.
We reformulate the image-caption pairs into image-to-text retrieval tasks in our pre-training. To construct the training corpus, we treat each caption as an individual document linked to its corresponding image. The task then involves retrieving the most relevant caption for a given image, guided by a set of randomly selected instructions. Since CC3M is originally an image captioning task, we do not validate or test our retriever on CC3M. Instruction templates for CC3M
• <Image> Describe the image concisely. • <Image> Provide a brief description of the given image. • <Image> Offer a succinct explanation of the picture presented. • <Image> Summarize the visual content of the image. • <Image> Give a short and clear explanation of the subsequent image. • <Image> Share a concise interpretation of the image provided. • <Image> Present a compact description of the photo’s key features. • <Image> Relay a brief, clear account of the picture shown. • <Image> Render a clear and concise summary of the photo. • <Image> Write a terse but informative summary of the picture. • <Image> Create a compact narrative representing the image presented.
A.2 Q2T Retrieval
A.2.1 MSMARCO MSMARCO (Bajaj et al., 2018) stands for Microsoft Machine Reading Comprehension dataset. It is a text-only dataset with around 1 million questions and 8 million passages. At stage 0, we train according to ColBERT-v1 by Khattab and Zaharia (2020). For later stages, we downsample the dataset to 400K questions to balance between the multimodal tasks and unimodal tasks. For the training corpus, we still use the full 8 million passages. For testing, we select 6,980 and 5,120 samples from the original MSMARCO validation set and sample 400K passages to retrieve from and ensure the subset contains all ground-truth passages. Instruction templates for MSMARCO:
• <Blank image> Retrieve the document that answers this question. <Questions>
• <Blank image> Find the document that is most relevant to the question. <Questions>
• <Blank image> Obtain the document that resolves this query. <Questions>
• <Blank image> Acquire the document that elucidates this question. <Questions>
• <Blank image> Choose the document most relevant to the query. <Questions>
• <Blank image> Identify the document most applicable to the question. <Questions>
• <Blank image> Extract the document that answers this query. <Questions>
• <Blank image> Locate the document that addresses the query.<Questions>
A.3 IQ2T Retrieval A.3.1 LLaVA The LLaVA instruction following dataset contains GPT-3.5 generated high-quality conversation about an image between a human and an AI assistant. There are around 150K rounds of conversations. We took each conversation (each question from the human and the answer from the AI assistant) as a separate sample. This results in a total of 356K samples. Since there are no original validation or test sets associated with the LLaVA, we manually split the sample pool into 351K training examples and 5,120 test examples.
The task is reformulated to an Image&Question to Text retrieval task. The training corpus and test corpus each contain the associated answers as passages to be retrieved by the image and question pairs. We use two types of instruction templates depending on the preciseness of the question:
• <Image> Provide a brief description of the image along with the following question: <Question>
• <Image> Provide a concise explanation of the image along with the following question: <Question>
A.3.2 OVEN OVEN is a dataset targeting open-domain visual entity recognition. The dataset consists of two splits: entity set and query set. The entity set is derived from image classification datasets such as INaturalist2017 (Cui et al., 2018), Food-101 (Bossard et al., 2014), Cars196 (Krause et al., 2013) and Google Landmarks Dataset v2 (Weyand et al., 2020). The query set is derived from VQA datasets such as VQAv2 (Goyal et al., 2017) and OKVQA
(Schwenk et al., 2022). To avoid overlapping with our other KB-VQA datasets, we only use the entity set of OVEN. The entity set contains about 10K unique entities.
The original entity set contains about 5 million question-image pairs. However, the questions are highly duplicated in the original OVEN dataset. We downsample the dataset by removing repeated questions corresponding to the same entity. This reduces duplications while maximizing the diversity of the questions and coverage of entities. After the filtering, we keep 339K training samples. For validation and testing, we select 20,000 and 5,120 examples from the original OVEN Entity validation set. The original test set is not used in M2KR due to the lack of annotation.
The original task is to link the image to a specific Wikipedia Entity given a question. To formulate the task as a retrieval problem, for each entity, we use its associated Wikipedia passage as the document to retrieve. The query side of this retrieval task contains the image and its question with the inclusion of a randomly sampled instruction. Given this query, the task is to obtain the relevant Wikipedia passage. The training corpus contains about 10K passages, while the test corpus contains about 3.2K passages that cover all entities in OVEN’s original training set and validation set respectively.
A.3.3 E-VQA, Infoseek and OKVQA E-VQA, Infoseek, and OKVQA are Knowledgebased VQA (KB-VQA) datasets. For each given image and question (with instruction), the task is to retrieve the corresponding knowledge passage.
For E-VQA (Mensink et al., 2023a), the original training set contains around 1 million samples. However, it includes duplicated questions and answers referring to the same Wikipedia Entity with different query images. We filter duplicated questions that pertain to the same Wikipedia Entity. To align with the original evaluation setting of EVQA, we further excluded samples that necessitate multiple knowledge bases, reducing the count to 167K training samples. To be consistent with the original E-VQA paper, our validation and testing sets exclusively include questions that can be answered using single knowledge. These sets contain 9,852 and 3,750 samples, respectively. We use the WikiWeb2M (Burns et al., 2023) as the knowledge source. For the training and test passage corpus, we keep all the passages that appear in the original E-VQA to align with the official E-VQA’s setting
for retrieval. For OKVQA, we use the original training and test set. Following Lin et al. (2023b), we prepare a knowledge corpus with Wikipedia documents based on pseudo-relevance. The training and test passage corpus both contain all passages in the knowledge corpus.
For Infoseek, following the preprocessing steps described by Chen et al. 2023c, we use Wikipedia documents as knowledge sources and remove examples whose answers can not be found in the ground-truth documents. We randomly selected 100K examples from the training set for training and 4,708 examples from the validation set for testing (the annotation of the original test set has yet been released). The downsampling is motivated by our observation that many questions are repeated and the number of unique documents associated with the whole dataset is only about 40K. We downsampled the dataset such that the model won’t overfit severely to Infoseek passages.
Note that the aforementioned downsampling procedure for the test set is only used for constructing the M2KR benchmark. For downstream VQA evaluation, we use the same test set that existed in previous literature to ensure a fair comparison. Instruction templates for OVEN, Infoseek, EVQA, and OKVQA
• <Image> Using the provided image, obtain documents that address the subsequent question: <Question>
• <Image> Retrieve documents that provide an answer to the question alongside the image: <Question>
• <Image> Extract documents linked to the question provided in conjunction with the image: <Question>
• <Image> Utilizing the given image, obtain documents that respond to the following question: <Question>
• <Image> Using the given image, access documents that provide insights into the following question: <Question>
• <Image> Obtain documents that correspond to the inquiry alongside the provided image: <Question>
• <Image> With the provided image, gather documents that offer a solution to the question: <Question>
• <Image> Utilizing the given image, obtain documents that respond to the following question: <Question>
B Implementation Details
B.1 Breakdown of Data Used in Training
In Stage 3 Full-scale Fine-tuning, the different subtasks in the M2KR dataset are downsampled or duplicated to balance the dataset proportions during training. The detailed breakdown of the data used in different phases is presented in Table 9. We observed that without adjusting the data proportions during training, the model’s training losses on certain datasets like WIT, Infoseek, and OVEN decrease much faster than on others once all parameters become trainable. This goes against our goal of training a multi-tasking system. Adjusting the data proportions is crucial to ensure a more consistent learning process across different tasks.
B.2 Detailed Hyperparameters
We use the Adam optimizer (Kingma and Ba, 2015) with a fixed learning rate of 10−4 for the mapping structure and 10−5 for the rest parameters in all experiments in all training stages. 4 Nvidia A100 GPUs were used with data parallel in all experiments.
Stage 0: Training was run up to 300k steps. The batch size is 8 and the gradient accumulation step is 8. The number of negative examples is 1. The validation ran per 10k steps. The checkpoint was taken at the best Recall@50 on the original MSMARCO validation set, following Khattab and Zaharia (2020). The total training time is approximately 1.5 days per model.
Stage 1: Training was run up to 220k. The batch size is 8 and the gradient accumulation step is 8. The number of negative examples is 4. The validation interval is 10k steps. The checkpoint was taken at the best Recall@10 on the validation set of WIT in M2KR. The total training time is approximately 5 days per model.
Stage 2: The intermediate pre-training was run for 12k steps for all experiments. The batch size is 8 and the gradient accumulation step is 8. The number of negative examples is 4. The total training time is approximately 2 days per model.
Stage 3: Training was run for 50k for all experiments. Training was early-stopped if the performance on WIT or E-VQA decreases for 3 consecutive validation runs. Validation was run per 10k steps. The batch size is 8 and the gradient accumulation step is 8. The number of negative examples is 4. The total training time is approximately 2 days per model.
Single-task Downstream Fine-tuning: The batch size is 8 and the gradient accumulation step is 8. The number of negative examples is 4. For reference, in our experiments, the downstream finetuning took 20k, 5k, 1k, 15k, 2.5k steps to achieve the best performance for WIT, OVEN, Infoseek, E-VQA, OKVQA respectively (for the ViT-G + Base-v2 PreFLMR). The total training time is approximately 1 days per model per task.
VQA Fine-tuning: We used BLIP2-T5XL as the answer generator as in RA-VQAv2 (Lin et al., 2023b). The retriever was frozen during training and inference. The batch size is 1 and the gradient accumulation step is 16. For each question in a training batch, top-5 relevant documents were preextracted using the retriever, and 3 out of 5 were randomly selected. These 3 documents were concatenated to the question and sent to the answer generator for one forward pass individually. This setting is to enable training with top-5 documents given limited GPU memory. The total training time is approximately 2 days per model.
Every model reported in this paper was reproduced once to make sure the training is reproducible. The best result is reported since the model with the best result will be released to the community. There is not much difference in the two runs. The absolute difference is less than 0.2 Recall score in most datasets (except that PreFLMR_ViTB_Base-v2 has a -0.4 difference on Infoseek).
B.3 Large-v1 Training In our experiments, we found that training Large-v1 during Stage 2/3 was not steady. First, the loss decreased faster than in other systems, like Base-v1, even though Large-v1 had worse system performance. This happened because Large-v1’s bigger model capacity made it more prone to overfitting.
Next, the loss suddenly shot up, causing the train-
ing to collapse, despite using the same data and strategy as Base-v1. We tried different hyperparameters, like lowering the learning rate to 1e− 6, 3e− 6, but the model still collapsed.
Finally, when we used LoRA (Hu et al., 2022) with Large-v1 during training, it helped stabilize the process. The LoRA hyperparameters used were: r = 16, α = 32, and a dropout rate of 0.05.
B.4 Model Design in Detail Similar to FLMR, PreFLMR consists of three components: a vision model FV , a mapping structure FM , and a language model FL. Feature Extraction. The textual query q consists of an instruction and (optionally) a question (e.g., "Utilize the given image to procure documents addressing the following query: [Question]"). We use a language model with hidden size dL to obtain embeddings for all Nq tokens which are concatenated into matrix Qq:
Qq = FL(q) ∈ RNq×dL (2) Like FLMR, a vision model FV encodes the input image I , extracting the [CLS] token embeddings from the last layer. PreFLMR additionally uses the patch embeddings from the penultimate layer of ViT for more complete representation.
QI,[CLS] = FV (I) ∈ R1×dV (3)
QI,PATCH = FV,−2(I) ∈ RNV ×dV (4)
The mapping structure FM comprises two components: a 2-layer MLP FMLPM and a Transformer block FTRM .
Following the FLMR model, a 2-layer MultiLayer Perceptron (MLP) FMLPM is utilized to convert the initial token embeddings into visual token embeddings with a length of Nvt and a hidden size dh:7
QMLPI = FMLPM (QI,[CLS]) ∈ RNvt×dh (5)
Moreover, an additional Transformer module FTRM is introduced to manage all patch embeddings. It is a stack of NTR transformer layers with a hidden size dL, followed by a simple MLP layer at the
7Transformation sequence: RdV → RNvtdh/2 → RNvtdh , subsequently reshaped into RNvt×dh .
end. This module leverages cross-attention with the text query Qq, enabling query-aware image feature mapping.
QTRI = FTRM (Fv(QI,PATCH),Qq) ∈ RNV ×dh (6)
Here, Fv represents a 1-layer MLP that adapts the dimension from dV to dL, which is subsequently transformed to dh by the linear MLP layer of FTRM . The resultant features from these processes are concatenated to formulate the query embeddings:
Q = [ Qq|QMLPI |QTRI ] ∈ R(Nvt+NV +Nq)×dh
(7) Furthermore, the document representations in the knowledge base are denoted by D, derived from the document content d with length lD:
D = Fl(FL(d)) ∈ RlD×dh , (8) where Fl signifies a straightforward MLP layer tasked with mapping dL to dh, thereby aligning the dimensionality with the query embeddings.
Multi-Modal Late Interaction. The relevance score between a question-image pair q̄ = (q, I) and a document d is calculated using a lateinteraction paradigm:
r(q̄, d) = r((q, I), d) =
lQ∑
i=1
lD max j=1 QiD ⊤ j (9)
where lQ = Nvt + NV + Nq. For each token in the query, the system aggregates the maximum relevance score across all tokens in the document.
Training and Inference. For model training, documents d∗ corresponding to a query q are considered gold (positive) samples. We incorporate random negative sampling from the corpus.8 Additionally, we adopt in-batch negative sampling as suggested by Karpukhin et al. (2020), treating all non-corresponding documents in a batch as negatives for q, denoted as N (q). The model is trained using a contrastive loss across the dataset D:
L = − ∑
(q,d∗)∈D log
exp (r(q, d∗))
exp (r(q, d∗)) + ∑
z∈N (q) exp (r(q, z))
(10) 8In multi-dataset scenarios, negative samples are selected
from the same corpus as d∗.
Post-training, all documents are indexed through PLAID (Santhanam et al., 2022a) for efficient lateinteraction retrieval. For detailed evaluation of retrieval efficiency, we refer readers to Lin et al. (2023b).

heading: C Ablation Study on Pre-training Stages
text: We present the ablation study for the four pretraining stages in Table 10. To ensure consistent comparison, these ablated versions underwent the same number of training steps as PreFLMR_ViTB_Base-v2. The results clearly indicate that the removal of any stage deteriorates performance. Specifically, disabling Stage 0 (i.e. using untrained text encoder) leads to the most significant performance decline because the text encoder is not pre-trained on late-interaction, resulting in a diminished ability to capture fine-grained relevance within the same computational budget. Note that removing Stage 0 leads to collapsed performance on Stage 1, where the text encoder is frozen. Furthermore, removing Stage 2 notably affects the performance on E-VQA more than on other KBVQA datasets, highlighting the challenge posed by E-VQA and the necessity of intermediate pretraining.
D V-Entropy-based Analysis of Intermediate Pre-training
V-Entropy (Xu et al., 2020), HV(Y |X), is the minimal Negative Log-Likelihood (NLL) achievable by the probabilistic predictor f(Y |X) under the predicative family V . A predicative family can be viewed as the set of reachable models under a certain model architecture and training budgets.
We define Mutual Information IV[Nf ](D1 → D2) between datasets D1 and D2 in Eq.11. We define HV[Nf ](D2) as the minimal achieved NLL loss on the validation set of dataset D2 after Nf training steps on D2. V[Nf , D1, Nt] denotes the set of reachable models after Nf fine-tuning steps on D2 starting from a checkpoint that has been trained on dataset D1 for Nt steps. This is V-Entropy with additional predictive family specification.
IV[Nf ](D1 → D2) = HV[Nf ](D2) −HV[Nf ,D1,Nt](D2) (11)
Intuitively, D1 has high mutual information with D2 if models initialized from D1 checkpoints attain much lower NLL loss compared to models
initialized without training on D1. Nf and Nt set the computation constraints for training on D2 and D1, respectively. In our experiment, V is the PreFLMR architecture, D1 is the E-VQA dataset and D2 is the training set of M2KR. Nf corresponds to Ninter in Sec. 5.6, which is the intermediate training steps on the E-VQA dataset. In the analysis, we set Nf to 5,000 and sweep Nt from 0 to 25,000 in intervals of 5,000.
We refer readers to Xu et al. (2020) for detailed properties of V-Entropy and emphasize that IV[Nf ](D1 → D2) is an empirical value we define to estimate mutual information between datasets. It is different from the V-Information defined in Xu et al. (2020) which estimates the mutual information between model input and output.

heading: E Qualitative Analysis for OKVQA and E-VQA
text: In this section, we compare examples from the OKVQA and E-VQA datasets to highlight their differences. To avoid cherry-picking, we use examples from its official website9 for OKVQA. Similarly, we use the examples included in the paper for E-VQA. Table 11 presents three examples from each dataset.
The OKVQA examples typically require common sense knowledge, like ‘people attend church on Sundays’ or ‘firetrucks use fire hydrants.’ Stateof-the-art Large Language Models (LLMs) often have this common sense knowledge inherently built-in, making additional knowledge retrieval less impactful for OKVQA tasks.
In contrast, E-VQA examples demand more specialized, expert-level knowledge, necessitating an effective knowledge retrieval system. For instance, correctly answering a question about ’Acacia paradoxa’ requires first retrieving the relevant document providing specific information about this plant species. Enhancing the knowledge retrieval
9https://okvqa.allenai.org/
system to source accurate documents is crucial for improving performance on the E-VQA dataset.

heading: F Artifacts and License
text: We list the resources used and their License below: (1) huggingface-transformers (Apache License 2.0) provides pre-trained model checkpoints for BLIP 2, DPR, and their tokenizers: https://github.com/huggingface/transformers
(2) FAISS (Johnson et al., 2019) (MIT License) is used to index document embeddings for fast retrieval with DPR: https://github.com/facebookresearch/faiss
(3) huggingface-PEFT (Apache License 2.0) for parameter-efficient LoRA fine-tuning: https://github.com/huggingface/peft
(4) PLAID and ColBERTv2 (MIT License): https://github.com/stanford-futuredata/ColBERT
(5) RA-VQA-v2 official repository with training and testing codes (GNU General Public License v3.0): https://github.com/LinWeizheDragon/RetrievalAugmented-Visual-Question-Answering.
(6) Datasets used in building the M2KR benchmark:
• WIT (Creative Commons Attribution-ShareAlike 3.0 Unported https://github.com/google-researchdatasets/wit/blob/main/LICENSE);
• MSMARCO (noncommercial research purposes only https://microsoft.github.io/msmarco/);
• CC3M (Free for any purposes https://github.com/google-researchdatasets/conceptual-captions);
• LLaVA, the image of LLaVA is a subset of CC3M. It should inherit the license of CC3M. The conversation data follows policy of OpenAI: https://openai.com/policies/terms-of-use.
• IGLUE (MIT license https://github.com/ebug/iglue/blob/main/LICENSE);
• KVQA (No specific license is mentioned https://malllabiisc.github.io/resources/kvqa/);
• OVEN (Apache-2.0 license https://github.com/open-visionlanguage/oven/blob/main/LICENSE);
• E-VQA (no specific license mentioned https://github.com/google-research/googleresearch/tree/master/encyclopedic_vqa);
• Infoseek (Apache License 2.0 https://github.com/open-visionlanguage/infoseek/blob/main/LICENSE)
• OKVQA (Copyright (c) 2021, Chen Qu and Center for Intelligent Information Retrieval, University of Massachusetts, Amherst. https://github.com/prdwb/okvqarelease/blob/main/LICENSE)
In particular, we emphasize that no changes are made to the original data of all the datasets used in our work. Our released models and artifacts should only be used for non-commercial purposes. By using the pre-trained models, users agree to respect the terms and conditions of the datasets used in pre-training.

heading: G PreFLMR model performance radar chart on M2KR tasks
text: Fig. 3 demonstrates the performance of PreFLMR with a radar plot. The best and worst numbers of each task are annotated.

heading: H AI Assistance
text: Our coding work was assisted by Github Copilot.10 OpenAI ChatGPT11 was only used in proofreading and spell-checking. We claim that the content presented in this paper was fully original.
10https://github.com/features/copilot 11https://chat.openai.com/

heading: Future Work
text: com/stanford-futuredata/ColBERT
(5) RA-VQA-v2 official repository with training and testing codes (GNU General Public License v3. 0): https://github. com/LinWeizheDragon/RetrievalAugmented-Visual-Question-Answering. (6) Datasets used in building the M2KR benchmark:
• WIT (Creative Commons Attribution-ShareAlike 3. 0 Unported https://github. com/google-researchdatasets/wit/blob/main/LICENSE);
• MSMARCO (noncommercial research purposes only https://microsoft. github. io/msmarco/);
• CC3M (Free for any purposes https://github. com/google-researchdatasets/conceptual-captions);
• LLaVA, the image of LLaVA is a subset of CC3M. It should inherit the license of CC3M. The conversation data follows policy of OpenAI: https://openai. com/policies/terms-of-use. • IGLUE (MIT license https://github. com/ebug/iglue/blob/main/LICENSE);
• KVQA (No specific license is mentioned https://malllabiisc. github. io/resources/kvqa/);
• OVEN (Apache-2. 0 license https://github. com/open-visionlanguage/oven/blob/main/LICENSE);
• E-VQA (no specific license mentioned https://github. com/google-research/googleresearch/tree/master/encyclopedic_vqa);
• Infoseek (Apache License 2. 0 https://github. com/open-visionlanguage/infoseek/blob/main/LICENSE)
• OKVQA (Copyright (c) 2021, Chen Qu and Center for Intelligent Information Retrieval, University of Massachusetts, Amherst. https://github. com/prdwb/okvqarelease/blob/main/LICENSE)
In particular, we emphasize that no changes are made to the original data of all the datasets used in our work. Our released models and artifacts should only be used for non-commercial purposes. By using the pre-trained models, users agree to respect the terms and conditions of the datasets used in pre-training