SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(2) Exact Match (EM) awards point if any of the annotated answers is generated exactly:  $\mathrm{EM}(y, S) = \min(\#_{S}(y), 1)$ .
(3) Pseudo Relevance Recall (PRRecall@K): To evaluate the retriever, we adopt pseudo relevance following Luo et al. [2021] due to the absence of ground-truth knowledge documents for each query. A document is considered pseudo-relevant if it contains any human-annotated answers. PRRecall@K measures whether the retrieved  $K$  documents contain at least one pseudo-relevant document: PRRecall@K = min  $\left( \sum_{k=1}^{K} H(d_k, S), 1 \right)$ , where  $H(d_k, S)$  evaluates to 1 if the retrieved document  $d_k$  contains any answer in  $S$ , and 0 otherwise. The metric is averaged across the test set.
Baselines. To demonstrate the effectiveness of FLMR, we take a DPR retriever as a baseline. In later sections, FLMR w/o Late Interaction refers to the corresponding DPR baseline. We use the same training data and hyper-parameters to build a multi-modal retriever based on DPR. For fair comparison, we keep the product  $N_{vt} \times d_{L}$  identical for DPR and FLMR so that they have the same number of parameters in the mapping networks. Since DPR can only handle one-dimensional query and document embeddings, we sum the embeddings of the [CLS] token from  $\mathcal{F}_L(\cdot)$  and the visual tokens from  $\mathcal{F}_M(\mathcal{F}_V(\cdot))$  to reduce the dimension to  $1 \times d_{L}$  (details in Appendix D).
We also compare our VQA performance with the latest KB-VQA models. Amongst these models, ConceptBERT [Garderes et al., 2020], KRISP [Marino et al., 2021], VRR [Luo et al., 2021], MAVEx [Wu et al., 2022], KAT-T5 [Gui et al., 2021], TRiG-Ensemble [Gao et al., 2022], and RA-VQA are relatively small in model size (<1B), whereas PICa [Yang et al., 2022], KAT [Gui et al., 2021], Prophet [Shao et al., 2023], PromptCap [Hu et al., 2022a], REVIVE [Lin et al., 2022], PALI [Chen et al., 2022b], Flamingo [Alayrac et al., 2022], PaLM-E [Driess et al., 2023] use very large models such as GPT-3 (175B) and PaLM-E (562B).
5 Results and Key Findings
5.1 VQA Performance
As shown in Table 1, recent models leveraged LLM or Large Multi-modal Models (LMMs) to achieve excellent performance on OK-VQA. The best performance to date is by PaLM-E, achieving a VQA score of 66.1 with 562 billion pre-training parameters. The original RA-VQA formalism achieves a lower VQA Score of 54.48 but with only 800 million parameters.
We first compare RA-VQA-v2 (with FLMR retrieval) with RA-VQA (with DPR retrieval). Our replication of RA-VQA (T5-Large) (Table 1, Row 22) achieves similar PRRecall@5 and VQA Score as the published results of RA-VQA (Table 1, Row 8). Compared with RA-VQA (T5-large), RA-VQA-v2 (T5-large) improves the PRRecall@5 significantly from  $83.08\%$  to  $89.32\%$ , leading to a gain of 3.4 in VQA Score (51.45 to 54.85). This suggests that improvement in knowledge retrieval benefits answer generation via retrieval augmentation.
We also show the effectiveness of knowledge augmentation by comparing the underlying base models with their retrieval-augmented version. As shown, T5-large and BLIP 2 (fine-tuned with OK-VQA data) achieve 47.52 and 55.44 VQA Scores, respectively. Their retrieval-augmented version, RA-VQA-v2 (T5-large) and RA-VQA-v2 (BLIP 2) gain 7.33 and 6.64 in VQA Score, respectively. For readers' interest, we provide more thorough analyses on the performance that the underlying answer generator model attains and the gain brought by knowledge retrieval in Appendix I.
To confirm that text-based vision can aid LMMs such as BLIP 2 which already has its own image encoder in VQA tasks, we remove text-based vision from RA-VQA-v2 (BLIP 2) and BLIP 2 (finetuned). This results in a decrease in VQA performance from 62.03 to 60.37 and 55.44 to 54.10, respectively (Table 3), suggesting that text-based vision contains useful information not included in the visual features obtained by BLIP 2's own image encoders.
RA-VQA-v2 achieves comparable and even better performance when compared with systems that use very large ( $\geq 13\mathrm{B}$  parameters) LLMs and LMMs. With BLIP 2 ( $\approx 3\mathrm{B}$ ), RA-VQA-v2 outperforms Flamingo (80B) by 4.19 VQA Score. It also outperforms many recent systems that use GPT-3 (175B) as an answer generator or knowledge source, such as PromptCap, REVIVE, and KAT. It achieves similar performance to that of PALI (17B) (62.03 vs 64.5 VQA Score). With comparable parameter sizes, RA-VQA-v2 (BLIP 2, 3B) outperforms PALI (3B) by a large absolute margin (62.03 vs 52.40 VQA Score). We emphasize that RA-VQA-v2 can be used in conjunction with virtually any existing LLMs and LMMs to offer substantial improvements, as demonstrated by the T5-large and BLIP 2 experiments.
Table 1: Model Performance on OK-VQA. Knowledge Source abbreviations: C: ConceptNet; W: Wikipedia; GS: GoogleSearch; GI: Google Images. EM stands for Exact Match. VQA stands for VQA Score. R stands for PRRecall. The best performance in literature is underlined.
#	Model	Base Models	K	Knowl. Src.	R@5	EM	VQA
1	ConceptBERT	C	33.66
2	KRISP	C + W	38.35
3	VRR	100	GS	45.08
4	MAVEx	W + C + GI	39.40
5	KAT-T5	T5-large	40	W	44.25
6	TRiG-Ensemble	T5-large	100	W	54.73	50.50
7	RA-VQA (joint training)	T5-large	50	GS	82.84	59.41	54.48
8	RA-VQA	T5-large	5	GS	81.25	55.77	51.22
Systems based on large models (≥3B parameters)
9	PICa	GPT-3	GPT-3	48.00
10	KAT-Ensemble	T5-large, GPT-3	40	W + GPT-3	54.41
11	Prophet	GPT-3	GPT-3	61.11
12	PromptCap	GPT-3	GPT-3	60.40
13	REVIVE	GPT-3	W + GPT-3	58.00
14	PALI	PALI (3B)	PALI	52.40
15	PALI	PALI (15B)	PALI	56.50
16	PALI	PALI (17B)	PALI	64.50
17	Flamingo	Flamingo (80B)	Flamingo	57.80
18	PaLM-E	PaLM-E (562B)	PaLM-E	66.10
Baselines without knowledge retrieval
19	T5-large (fine-tuned) w/o knowledge	T5-large	51.38	47.52
20	BLIP 2 (fine-tuned) w/o knowledge	BLIP 2 (T5-XL)	59.49	55.44
Our proposed models (models w/o Late-interaction use DPR instead of FLMR)
21	RA-VQA-v2 (T5-large)	T5-large	5	GS	89.32	58.85	54.85
22	w/o ROI & VE & Late-interaction	T5-large	5	GS	83.08	55.89	51.45
23	RA-VQA-v2 (BLIP 2)	BLIP 2 (T5-XL)	5	GS	89.32	62.01	62.08
24	w/o ROI	BLIP 2 (T5-XL)	5	GS	87.02	61.63	60.75
25	w/o ROI & VE	BLIP 2 (T5-XL)	5	GS	85.99	59.95	60.41
26	w/o Late-interaction	BLIP 2 (T5-XL)	5	GS	82.90	59.00	58.20
27	w/o ROI & Late-interaction	BLIP 2 (T5-XL)	5	GS	83.43	60.18	59.21
28	w/o ROI & VE & Late-interaction	BLIP 2 (T5-XL)	5	GS	83.08	59.49	58.70
5.2 Retrieval Performance
Text-based/Feature-based Vision. As shown in Table 2, previous retrievers (RA-VQA, VRR) achieve  $\approx 82.84\%$  PRRecall@5 using only text-based vision (textual descriptions of visual scenes). We show that visual features obtained via aligned vision models (feature-based vision) are equally effective as text-based vision. Relying on questions only, FLMR has a baseline retrieval score of 74.81 PRRecall@5. Incorporating text-based vision and feature-based vision increase PRRecall@5 to 85.99 and 85.08, respectively. Furthermore, feature-based vision provides information complementary to test-based vision, as demonstrated by the better PRRecall@5 at 87.02 when the two are combined. The same trend is observed for DPR-based retrieval system, though less dramatically (from 83.08 to 83.43). We note that pre-training the mapping network for vision-language alignment is crucial for good performance. Without such pre-training, performance degrades to 85.71. These results confirm that incorporating aligned vision encoders in the retrieval process compensates for the information loss in image-to-text transforms.
Effects of Late Interaction and ROIs. Late Interaction enables FLMR to capture fine-grained relevance of token-level embeddings. As shown in Table 2, using the same query and document representations, upgrading DPR to FLMR leads to consistent improvement in retrieval performance by a large margin up to  $\sim 6\%$  (comparing Table 2 Row 8 & 13).
Table 2: Retrieval performance on Google Search (GS) and Wikipedia. Text-based vision refers to textual descriptions of images (such as OCR, caption, objects and attributes). Feature-based vision is obtained using a neural vision model directly (e.g. ViT). R@K refers to PRRecall@K.
#	Retriever	Text-based Vision	Feature-based Vision	GS	Wikipedia
R@5	R@10	R@5	R@10
1	VRR	✓	-	80.4	88.55
2	RA-VQA-FrDPR	✓	-	81.25	88.51
3	RA-VQA	✓	-	82.84	89.00
4	DPR	-	-	73.11	82.05	57.03	69.84
5	DPR	✓	-	83.08	89.77	66.04	75.94
6	DPR	-	✓	80.52	88.27	65.84	75.85
7	DPR	✓	✓	83.43	90.31	66.88	76.35
8	DPR	✓	✓+9ROIs	82.90	89.95	65.86	75.90
9	FLMR	-	-	74.81	83.10	57.20	70.11
10	FLMR	✓	-	85.99	92.79	66.50	76.80
11	FLMR	-	✓	85.08	91.80	66.90	77.05
12	FLMR	✓	✓	87.02	92.69	67.50	77.60
13	FLMR	✓	✓+9ROIs	89.32	94.02	68.10	78.01
14	w/o alignment pre-training	✓	✓+9ROIs	85.71	92.41	66.40	76.10
Table 3: Removing text-based vision from answer generation reduces the VQA performance, showing that text-based vision offers more complete image understanding.
Model	VQA Score
RA-VQA-v2 (BLIP 2)	62.03
w/o text-based vision	60.37
BLIP 2 (fine-tuned) w/o knowledge	55.44
w/o text-based vision	54.10
Table 4: Comparison of ROI selection methods. R stands for PRRecall.
R@5	R@10
4 Object-centric ROIs	88.01	93.62
4 Random ROIs	86.9	93.20
4 Evenly-split ROIs	86.96	93.16
Figure 2: PRRecall@5 versus the number of ROIs. Finer-grained ROIs cause performance degradation in DPR, while FLMR captures them to improve retrieval performance.
In addition to token-level relevance, FLMR can utilize fine-grained Region-of-Interest (ROI) features with Late Interaction whereas DPR can not. This can be demonstrated by Fig. 2: as the number of ROIs increases, DPR performance degrades. This may be because DPR's one-dimensional query and document embeddings are not expressive enough to encompass fine-grained details of the ROI visual cues. As shown in Table 2 and Table 1 Row 27-28, adding more ROIs effectively adds noise which adversely impacts the retrieval performance (83.43 to 82.9), and in turn worsen VQA scores (59.2 to 58.2).
Object-centric ROIs improve retrieval. We also conduct ablation studies to show that the performance improvements brought by ROIs come from the finer-grained information captured by them rather than from increases in the number of features. We compare FLMR with 4 object-centric ROIs (obtained by object detection) against 2 baseline ROI selection methods: (1) randomly crop 4 patches of size larger than  $100 \times 100$  from the image as ROIs; (2) evenly split the image to obtain the top-left, top-right, bottom-left, and bottom-right of the image as ROIs. As shown in Table 4, FLMR with 4 ROIs from VinVL object detection outperforms others, suggesting that it is the object-centric, fine-grained ROIs that improve the performance.
Retrieval performance on FVQA and Infoseek. As shown in Table 5, we observed similar improvements with FLMR. FLMR with both text- and feature-based vision improves DPR by  $2.3\%$
Table 5: Retrieval performance on FVQA [Wang et al., 2017] and Infoseek [Chen et al., 2023b]. Average recall on 5 splits is reported for FVQA. FLMR outperforms DPR trained with the same data with a clear margin.
FVQA Recall@5(Std.)	Infoseek Recall@5
DPR	68.58(0.01)	44.88
FLMR (Visual Encoder)	70.88(0.01)	46.42
FLMR (Visual Encoder & 10 ROIs)	72.37(0.01)	47.08
Question: How many teeth does this animal use to have?
Figure 3: Selected query tokens connected by document tokens that have the highest token-level relevance with them, as computed by FLMR. For example, amongst all document tokens, '26' and '30' have the highest relevance with the query token 'how' and 'many', respectively. This shows that FLMR can capture fine-grained document relevance. Zoom in for better visualization.
and  $1.54\%$  Recall@5 on FVQA and Infoseek, respectively. Incorporating ROI features further improves its performance to 72.37 on FVQA and 47.08 on Infoseek. This suggests that FLMR is generalizable to other KB-VQA retrieval tasks and can bring steady improvements relative to baselines.
Qualitative analysis of FLMR retrieval. Figure 3 shows FLMR retrieval in action. The orange lines connect the query token and the document token with the highest relevance score, which will be preserved after the MaxSim operation and will contribute to the final retrieval score. We can see that token-level interaction indeed captures fine-grained relevance between the query and the document. For example, the retriever recognizes that the numbers “26” and “30” in the document are highly relevant to “How many” in the query. We can also see that the image tokens are aligned with the text tokens: the image tokens corresponding to the cat (IMG, ROI3 and ROI4) point to the words “cats” and “cat” in the document. This demonstrates the effectiveness of vision-language alignment that gives rise to explainable cross-modality relevance. We provide more case studies in Appendix J.
In this paper, we proposed Fine-grained Late-interaction Multi-modal Retrieval (FLMR), the first of its kind to leverage fine-grained token-level relevance between queries and documents for VQA tasks. FLMR incorporates feature-based vision using an aligned vision model that complements text-based vision to enhance image understanding, improve retrieval performance and advance VQA performance. We achieve superior performance in OK-VQA, greatly surpassing previous systems with similar parameter size and closes the gap with those systems utilizing very large (>13B) models.
Weizhe Lin was supported by a Research Studentship funded by Toyota Motor Europe (RG92562(24020)). We thank our colleagues, Daniel Olmeda Reino (Toyota Motor Europe) and Jonas Ambeck (Toyota Motor Europe), who provided insight and expertise in this project.
Prof. Bill Byrne holds concurrent appointments as a Professor of Information Engineering at Cambridge University and as an Amazon Scholar. This publication describes work performed at Cambridge University and is not associated with Amazon.
We would also like to thank all the reviewers for their knowledgeable reviews.
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised visual-retriever-reader for knowledge-based question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6417-6431, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-main.517.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform-retrieve-produce: Natural language-centric outside-knowledge visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5067-5077, 2022.
Weizhe Lin and Bill Byrne. Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11238-11254, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.772.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.550.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE transactions on neural networks and learning systems, 29(12):5947-5959, 2018.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281-6290, 2019.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. (arXiv:2303.01903), Mar 2023. doi: 10.48550/arXiv.2303.01903. URL http://arxiv.org/abs/2303.01903. arXiv:2303.01903 [cs].
Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model. (arXiv:2209.06794), Sep 2022b. doi: 10.48550/arXiv.2209.06794. URL http://arxiv.org/abs/2209.06794. arXiv:2209.06794 [cs].
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv, 2022. doi: 10.48550/arxiv.2204.14198.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Thompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
We chose the Google Search corpus [Luo et al., 2021] for our question-answering system as it provides good coverage of the knowledge needed and is publicly available. However, as noted by the authors of RA-VQA, additional knowledge bases may be required to answer some questions correctly. Future work may address the issue by improving the quality and expanding the coverage of knowledge.
We do not perceive any immediate ethical concerns associated with the misuse of our proposed system. There is a possibility that the trained KB-VQA system might generate inappropriate or biased content as a result of the training data biases during LLM and LMM pre-training and fine-tuning. Therefore, it is advised to conduct an ethical review prior to deploying the system in live service.
C Data Statistics
Table 6 shows the data statistics of the OK-VQA dataset. Table 7 displays the number of passages in the document collections used for evaluating the retrieval systems. Note that the WIT corpus is introduced in Appendix H, which is used for investigating the retrieval of multi-modal documents.
Table 6: OK-VQA dataset statistics.
Category	Number
train questions	9,009
valid questions	5,046
images	14,055
Table 7: Data statistics of document collections used in retrieval.
Corpus	# of passages
GS for OK-VQA [Luo et al., 2021]	168,306
Wikipedia for OK-VQA	114,637
WIT for OK-VQA (Appendix H)	87,419
D Details of DPR baselines
We build a DPR retriever as a baseline for FLMR. We apply the same pre-training strategy, training data, and hyperparameters to construct a multi-modal retriever based on DPR. Particularly, we keep the product  $N_{vt} \times d_L$  and the number of parameters of the vision mapping networks identical for FLMR and DPR for a fair comparison. Since DPR can only handle one-dimensional query and document embeddings, we sum the embeddings of the [CLS] token from  $\mathcal{F}_L(\cdot)$  and the visual tokens from  $F_M(F_V(\cdot))$  to reduce the dimension to  $1 \times d_L$ . Formally, the query and document embeddings are:
$$
\mathbf {Q} _ {\mathbf {d p r}} = \left(\mathcal {F} _ {L, \mathrm {C L S}} (q) + \mathcal {F} _ {M} \left(\mathcal {F} _ {V} (g)\right) + \sum_ {i = 1, \dots , N _ {R O I}} \mathcal {F} _ {M} \left(\mathcal {F} _ {V} \left(r _ {i}\right)\right)\right) \in \mathcal {R} ^ {d _ {L}}, \tag {10}
$$
$$
\mathbf {D _ {d p r}} = \mathcal {F} _ {L, C L S} (d) + \mathcal {F} _ {M} (\mathcal {F} _ {V} (I _ {d})) \in \mathcal {R} ^ {d _ {L}}.
$$
where  $I_{d}$  is the image of the document if multi-modal document collection is used and otherwise omitted. The inner product search (supported by FAISS [Johnson et al., 2019]) is used to train and retrieve documents with DPR.
E Training and Hyperparameter Details
We use ColBERTv2 and openai/clip-vit-base-patch32 to initialize the text-based retriever and vision encoder. For the DPR baseline, we use facebook/dpr-single-nq-base to initialize the retriever. In answer generation, we use t5-large and Salesforce/blip2-flan-t5-x1.
With openai/clip-vit-base-patch32,  $d_V = 768$ . For FLMR, we use  $N_{vt} = 32$  visual tokens per image representation and  $d_L = 128$ . For DPR, we use  $N_{vt} = 6$  and  $d_L = 768$  so that the number of parameters of vision mapping network is similar to that of FLMR:  $N_{vt} \times d_L \sim 128 \times 32$ . The mapping network consists of two fully-connected layers with tanh activation. The output of last layer is reshaped into  $N_{vt} \times d_L$  visual tokens. Other model parameters are:  $l_q = 512$ ,  $l_d = 512$ ,  $N_{ROI} = 9$  unless otherwise specified.
We use 1 Nvidia A100 (80G) for all experiments. The optimizer is Adam [Kingma and Ba, 2015]. In training the retrievers, we use learning rate  $10^{-4}$ , batch size 30, gradient accumulation steps 2 for 10k steps (for both DPR and FLMR retrievers). When training RA-VQA-v2 (T5-large), we use learning rate  $6 \times 10^{-5}$ , batch size 2, gradient accumulation 16 for up to 20 epochs. We use a linearly-decaying scheduler to reduce learning rate from  $6 \times 10^{-5}$  to 0 after 20 epochs. We use LoRA [Hu et al., 2022b] to train RA-VQA-v2 (BLIP2) with learning rate  $10^{-4}$ , batch size 4, gradient accumulation steps 16 for up to 6k steps. LoRA is configured to use the default huggingface-PEFT setting:  $r = 8$ , lora_alpha=32, lora_dropout=0.1.
The vision model is frozen throughout all experiments. In pre-training the mapping network, only the mapping network is trainable. When training the answer generator, the retriever is frozen.
We report the required GPU hours on 1 Nvidia A100 (80G): for vision-language alignment of retrieval models, approximately 4 GPU hours are needed. Training the FLMR retriever requires around 12 GPU hours (10k steps) including the time of running testing after training is complete. Training RA-VQA-v2 (BLIP 2) with LoRA requires around 12 GPU hours (6k steps) including the time of running validation per 1k steps. Training the RA-VQA-v2 (T5-large) requires around 12 GPU hours (3k steps) including the time of running validation every 500 steps.