Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
Authors: Weizhe Lin
Jinghong Chen
Jingbiao Mei
Alexandru Coca
Bill Byrne
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge-based visual question answering
knowledge retrieval
multi-modality
vision-and-language
TL;DR: We proposed FLMR, Fine-grained Late-interaction Multi-modal Retrieval, which improves the performance of the RA-VQA framework significantly.

Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmented Visual Question Answering (RA-VQA), a strong framework to tackle KB-VQA, first retrieves related documents with Dense Passage Retrieval (DPR) and then uses them to answer questions. This paper proposes Fine-grained Late-interaction Multi-modal Retrieval (FLMR) which significantly improves knowledge retrieval in RA-VQA. FLMR addresses two major limitations in RA-VQA's retriever: (1) the image representations obtained via image-to-text transforms can be incomplete and inaccurate and (2) similarity scores between queries and documents are computed with one-dimensional embeddings, which can be insensitive to finer-grained similarities.
FLMR overcomes these limitations by obtaining image representations that complement those from the image-to-text transform using a vision model aligned with an existing text-based retriever through a simple alignment network. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained similarities between queries and documents. 
FLMR significantly improves the original RA-VQA retriever's PRRecall@5 by approximately 8\%. Finally, we equipped RA-VQA with two state-of-the-art large multi-modal/language models to achieve $\sim62$% VQA score in the OK-VQA dataset.

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering
Weizhe Lin, Jinghong Chen," Jinghiao Mei, Alexandru Coca, Bill Byrne
Department of Engineering University of Cambridge Cambridge, United Kingdom CB2 IPZ
{w1356, jc2124, jm2245, ac2123, xjb31)0cam. att.ecam.c..... uk
Abstract
Knowledge-basce Visual Question Answering (KB- -QA) requires VQA systems utilize knowledge from external knowledge bases answer risually-grounde questions. Rerrievll-Augmente Visual Question Answering (RA-VQA), a strong framework tackle KB-VQA, first retrieves related documents with Dense Passage Retrieval (DPR) and then uses them to answer questions. This paper proposes Fine-grained Late-interaction Multi-modal Retrieval (FLMR) which significantly improves knowledge retrieval in RA-VQA. FLMR addresses two major limitations in RA-VQA'S retriever: (1) the image representations obtained via image-to-text transforms can be incomplete and inaccurate and (2) relevance scores between queries and documents are computed with diee-iinniional embeddings, which can be insensitive to finer-gran finer-grained relevance. FLMR overcomes these limitations by obtaining image representations that complement those from the image-to- text transforms using vision model aligned with existing text-based retriever through simple alignment network. FLMM also encodes images and questions using multi-dimensional embeddings capture finer-grained relevance between queries and documents. FLMR significantly improves the original RA-VQA retriever's PRRecall @5 by approximately 8%. Finally, equipped RA-VQA with two stte-offthheart large multi mulliimodal/languaee models to achieve 61% VQA score in the OK-V -K-QA dataset.
Introduction
Knowledge-based Visual Question Answering (KB-VQA) is challenging problem that lies the intersection of Computer Vision, Natural Language Processing, and Information Retrieval, The objective of VQA is read an image and answer question related the image content. KB-VQA poses an additional challenge: in order to answer the question correctly, the system needs to draw on relevant information from external knowledge source, such knowledge graph database. Therefore, tackling KB-VQA tasks crucially depends on the ability to retrieve relevant information and to ground the answer generation process in the retrieved knowledge.
Retrieval Augmented Visual Question Answering (RA- (RA-VQA) is a framework designed to answer difficult KB-VQA questions (Luo al. 2021 Gao al. 2022 Lin and Byrne 2022] with the most recent version from Lin and Byrne [2022] achieving performance close to large models (such as GPT-3 [Brown et al. 2020) while using much simpler models. RA-VQA first retrieves K documents relevant to the image and the question from an external knowledge base, and then generates the answer using Large Language Model (LLM) grounded in the retrieved passages.
"Equally contributed as the first author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We focus on two major limitations in RA-VQA' retriever. (1) Incomplete image understanding: image representations are obtained via image-to-ter transforms such as captioning and object detection. While effective, this approach can result in incomplete image understanding, which hinders the retrieval of relevant knowledge. This is a common issue for retrieval-basse in the literature. (2) Lossy compression of visual scenes and questions to single embedding: the Dense Passage Retrieval (DPR) [Karpukhin et al. 2020) retriever, widely used in current retrieval- based QA systems, computes relevance scores between queries and documents with their respective, one-dimensional embeddings. However, compressing complex visual scenes and questions into a single embedding can be lossy. This is especially problematic in KB-V KB-VQA, where queries and visual elements are more diverse than in other Open-domale QA tasks. DPR could overlook finer-graine relevance, resulting in degraded retrieval performance.
KB-VQA systems
To address these two limitations we propose an enhanced knowledge retrieval approach called Fine- grained Late-interaction Multi-modal Retrieval (FLMR). FLMR incorporates finer-grained, token- level visual and textual features into multi multi-dimensional embeddings. When computing relevance scores, FLMR considers the interaction between every pair of token embeddings, including cross- modality interaction between texts and images, enabling finer-grained assessment of relevance. We also introduce large vision models such as ViT [Dosovitskiy et al. 202 to produce visual tokens that complement text-based image representations for more complete image understanding. To ensure that the interactions between visual and text tokens well-defined, we align the vision model with the text-based retriever with a simple yet effective aligmment training procedure. We also find that FLMR regions of interest, leading to better recall rate, whereas DPR's s
is able to make use of recall rate degrades when these finer-grained features are incorporated. Our FLMR retriever achieves significant increase of approximately 8% in PRRecall @5 for knowledge retrieval, and competitive VQA score of 61% %, surpassing the state-of-the-arr models with the same scale of parameters.
We summarize our contributions follows:
We introduce FLMR, the trst-of-its-kind to leverage Late Interactior and dimensoaila representations to capture fine- gre-ggrined, cross-modality relevance that significantly improve retrieval performance over existing state-of-the-art KB-V KB-VQA retrievers;
We show that introducing image representations from large vision model after simple yet effective alignment procedure can complement image representations obtained via image-to- text transforms, leading to more complete image understanding, better knowledge retrieval and higher VQA accuracy. This offers improvements to current VQA systems as many systems have only a single mode of image understanding that relies on either image-to-text transforms or vision models;
We achieve substantial improvement of approximately 8% in knowledge PRRecall @ over other state-of-the-art retrievers in the OK-VQA dataset, with an accuracy of 61 %% that surpasses other systems with similar parameter sizes.
2
Related Work
Visual Question Answering Systems. Recent work in VQA can be roughly divided into four
categories with respect to multi-modal modeling: (1) Visual and textual features can be fused via
cross-modality fusion Yu et al. 2018 Singh et al., 2019 Yu al. 2019 Jiang et al. 2020 Guo et al.
2021]; (2) Multi-modal models can be trained from scratch to jointly understand vision and language
before they are fine-tuned to perform VQA tasks [Tan and Bansal 2019 Chen et al. 2020 Gan et al.
2020 Li et al. 2020a] Wang et al.) 2022 Zhang et al. 2021 Li et al. 2021 | (3) Vision model and
language model that have been pre-trained on uni-modal corpora can be aligned to avoid expensive
multi-modal pre-training [Guo et al. [023 Dai et al. 20222] (iigh et al. 2022]. (4) Image-to-text transforms such as captioning can be used to transform images into texts to enable the use of text-only
reasoning pipelines [Lin and Byrne| 2022 Gui et al.) 2021 Lin al. 2022 Luo et al. 202 Yang ct al. 2022, Gao et al. 2022 Hu et al. 2022al. Building on these Vision-and-Languaee modeling techniques, our work shows that image-to-text transforms and aligned vision models can complement each other to provide more complete visual information, leading to improved performance in both knowledge retrieval and VQA.
'Dual encoder architecture where the queries and documents are first encoded into token-level embeddings and these embeddings then aggregated to compute final relevance scores

---

Crop Region of Interests
Documents in KB
Tmage-lo/Text Traneform
have? many does
looking laying
<ECC>
<BOV> white brown sitingg sat <SOV> pink
nose <SOV> orange pink brown ear
<EOV>
Vision Model
Question: How many teeth does this animal to have? Answer: 26
A1
Text Retnever
A2
Mapping Network
Text Retriever
A3
MaxSim
MaxSim
MaxSim
Knowledge
Taken-lavel embeddingg
Concatinatee
cueen typical eoult mouth have leeth, which be the secend and the last set in the mouth.
B1
B2
Max similarity operation
MaxSim
u... most cats have deciclous leeth and
Anewer
permanent teath.
Generator
Summation
.... dogs grow older, they will
permanent
Answer Generation
feeth while cats will have
Figure I: Overview of RA- -Q.-v2. The system consists of two steps: (A) Knowledge Retrieval and (B) Answer Generation. (A.1) text retriever used to obtain token-level embeddings of text-based vision (obtained by captioning and object detection) and text documents in the database. (A.2) Visual tokens are obtained from the image and the region-of-interss mapping network. (A.3) Relevance score between the query and the document is computed by aggregating the fine-grained relevance at token level with late interaction mechanism (Eq, 12). (B.1)
patches using vision model and
The answer generator takes the text query, the image, and the retrieved documents as input, generating one candidate answer per retrieved document. (B.2) The answer with the highest joint probability is selected.
Knowledge-based VQA Systems. Recent KB- VQA systems can access both structured data, such as ConceptNet and other KGs Narasimhan et al., 2018 Garderes et al. 2020 Li et al. 2020b Wu
al.
2022 Marino et al.
202 2021
al. 2023a , as well as unstructured data such as Wikipedia
passages [u et al. 2022 Gao et al. 2022 Gui et al. 2021 for knowledge retrieval. LLMs can
also be source of "implicit world knowledge": KAT Gui et al.) [021] and REVIVE [Lin et al.
2022] prompt GPT-3 to generate potential answer candidates. RA- RA-VQA [Lin and Byrne 2022 and
its prior works [Luo al., [02T Qu et al. 2021 Gao al., 2022] ground answer generation the retrieved knowledge from external KBs to achieve excellent VQA performance. Our work improves this retriever-reader pipeline with a novel knowledge retriever which significantly improves the recall rate of knowledge retrieval as well as the final VQA performance.
Knowledge Retrieval. Most knowledge retrievers in QA systems are based on DPR and its vari- ants (Karpukhin et al.) 2020 Gui et al.] 2021 Luo et al. 2021 Gui et al. [021 Lin and Bymn 2022 Wu and Mooney [022]] These mainly use one- dimensinni embeddings and contrastive learning for
training. Late Interaction models [Khattab and Zaharia 2020 Santhanam et al. 2022a have recently
achieved state-of-the -th- performance on QA knowledge retrieval. Our FLMR extends this paradigm to work with multi-modaa features and shows that incorporating finer-grained visual features, such as regions-of-inte of-interest, leads to superior retrieval performance. EnFoRe [Wu and Mooney [2022] retrieves a list of entities from the image, the query, and the answer candidates, and then explicitly learns scores to indicate the importance similar late-inted inceitteraction setting but it focuses single modal query (image-text retrieval), To the best of our knowledge, FLMR is also the first to introduce cross- modsaltd,llty... token-level late interactions to compute relevance scores for KB-VQA knowledge retrieval. We also propose a light-weight method that aligns a vision model with text-based retriever to incorporate more complete multi-modal information in retrieval queries. Compared previous approaches that rely expensive pre-training on multi-modal datasets [Chen et al. 2022a Yao et al. [022]. FLMR's vision-language alignment process is efficient and can be done in 4 hours with A-100 GPU.
each fine-grained entity. FILIP (Yao et al, 2022 has

---

3
Method
In this section, we introduce RA- VQA-22, which builds upon the original RA-VQA framework [Ln and Byrne 2022] but is equipped with Fine-grained Late-interaction Multi-modal Retriever (FLMR) enhance knowledge retrieval. As illustrated in Fig. [ the framework consists of two stages: Knowledge Retrieval (Sec. 3.1 and Answer Generation (Sec. 3.2
3.1 Knowledge Retrieval
The FLMR system consists of two encoders: vision model Fy and language model Fr. that encode image and textual features, respectively.
Visual Features. We utilize two types of visual representations: (1) text-based vision representations (textual description of visual elements) obtained by image-to-text transforms and (2) feature-based vision representations obtained by large vision models.
For text-based vision representations, to allow a direct comparison, we follow Lin and Byree [2022] to extract objects and their attributes using VLn [Zhang et al. [02T]] and generate image captions using Oscar [Li et al., 2020a]. For each image 7, we obtain a textual description that contains serialized object names, attributes, and descriptive captions (Lin and Byrne] [022]. The sequence is appended to the question q to form the query. For simplicity of notation, the question always includes text-based vision unless otherwise specified.
For feature-based vision representations, we use the vision model Fv to extract both global and regional image feature representations. For regional image feature representations, further the object detection results of VLn to locate NROS (Region-of-Intee bounding box proposals from VLn we use the predicted class name associated with each box to select objects explicitly mentioned in the question q, and then prioritize bounding boxes with larger areas. Using the vision model Fy, we then obtain global image representation g (T) R. from the image I and ROI-based regional representations {r; Fr((,)) E Rdv }i= Ii=I,...,Naa NRos from the image ROI patches { i I,., ] which contain finer-grained details.
bounding boxes. To filter
Token-Level Embeddings. Compared with DPR's 's compressed, one- mne-dimensional representation of queries and documents, FLMR preserves richer information by employing token-level, multi- dimensional embeddings to improve retrieval. We obtain token-level embeddings for both textual input and visual input. These are concatenated form the final embeddings of queries and documents.
To align the vision and text modalities, we train a mapping network FM that learns to project visual features from vision model Fy with hidden size dy into the latent space of the language model Fr with hidden size dz. The mapping network, 2-layer multi-layer perceptron, projects each visual representation into Nyt visual tokens, i.c. ddy + R Noide/2 + R Neids and finally reshaped into N.i X.xd
Formally, the final query embeddings Q are:
Q [FL(g), Far (lg F2, NN D) E Rlaxd,
where 10 = 4 + (Nror + 1) x Nur. ly is the length the question q. [e], uN] denotes the concatenation of N embeddings i to UN.
The documents in the knowledge base are represented by embeddings obtained from the document content d of length Ip:
D == Fz(d( e Rloxds,
Multi-Modal Late Interaction. We compute the relevance score between question-image pair q= (q, /) and a document d by a late interaction formula similar to that in ColBERT but under the multi-modal context:
r(q d) rr(((((( 1) (,
max Q.D.J
For each query token, the MAX operation selects the highest relevance score over all document tokens. In preliminary experiments, other operations (e.g. MEAN or SUM) were found be overly sensitive to the length of documents, which can be as short single sentence. We note that [PAD]

---

may dominate in the final score for short documents, whereas longer documents have an inherent advantage due to their greater number meaningful tokens.
In contrast to DPR, FLMR allows full interactions between every query emhedding vector Q; and every document embedding vector Dj. FLMR retriever also supports retrieving multi-modal documents. We leave the formulation and results to Appendix H
Training and Inference. To train the model, we treat documents d' that contain the ground-truth answer to question as gold (positive) documents. We use in-batch negative sampling for training following Karpukhin et al. [2020]. All documents in training batch other than d' are considered negative for q, denoted as N (q). We train with the contrastive loss LCL over the dataset D:
exp (r(g, ))))
LCL =
Σ
log
exp (r(q, d+) + +2 exp (r(q,2))
JED
2EN(e)
After training, all documents are indexed using PLAID (Santhanam et al. 2022b which enables fast late-interaction retrieval with a time cost similar to that of DPR.
Training the Mapping Network for Vision-Language Alignment. Directly fine-tuning the two models Fy and FL on the retrieval task leads to performance degradation the start of training, as the models are not yet aligned. Inspired by CLIP [Radford et al., (02T), where language model is trained align with vision model, we align Fv and FL the context of knowledge retrieval by pre-training the parameters of the mapping network Far with a retrieval task.
Given ground-truth image-document pairs {(pp: which can be Wikipedia images and their accompanying texts, the system is trained to retrieve the document d, associated with the input image pp- The relevance between the input image / and document d is formulated
Nut
Q = (M((((()) E R Naxdi
D = Fr(d) E Rlnxdi
r(1,d) -
Σ max
Q.DJ
(5)
where only the parameters of the mapping network Fas are trained with the contrastive loss in Eq 4
We provide details of pre-training in Appendix ( and discuss its effectiveness in Sec. 5.2
Knowledge Retrieval. We extract top-K documents from the knowledge base relevant knowledge.
The retrieval probability is defined below following the notation of Lin and Byrne [022 and Lewis
al. 2020):
exp(r(á, ds)))
po (cl (q)
ΣØ=1 exp(r(q.d)) dz))
where 9 denotes the model parameters of Fv, FL. and Fis.
3.2 Answer Generation
In principle, the knowledge retrieved by FLMR can be used by any VQA answer generator. We denote the answer generator as FA with parameters ø. Following Lin and Byree [2022], RA- RA-VQA-v2 generates answer for each retrieved document and selects the best candidate by the joint probability of retrieval and answer generation:
{dx K topK (po(dl)))
g, d= arg max da (q) arg max (slg dx) (o(del (7)
y.d.
p.d,
The training loss of the answer generator follows that of the underlying model. For example, when using BLIP 2 [Li et al. 2023), we use the cross-entropy loss of the generated sequences:
K
==
Σ
log (s)lg.
(8)
where T is the whole dataset. the set of human responses. st is the answer string that appears in the retrieved document dz or the most popular answer string if an exact match cannot be found in the document.
The most popular answer is the chosen by most annotators.
5

---

4 Experiment Setup
Datasets. We focus on the OK-VQA dataset where a large portion of questions requires external knowledge (either commonsense or domain-specific) to answer. There are no annotations of 'ground- truth' documents for OK- VQA questions. We follow the literature to use pseudo-relevance labels (a binary indicator of whether a document contains the answer string) as document annotations. We do not evaluate A-OKVQA (Schwenk et al. 2022] successor of OK-VQA. it emphasizes visually- reasoning rather than knowledge retrieval. To validate the effectiveness of our proposed approach, we test the retrieval abilities using 2 different corpora, whose statistics can be found in Appendix C
(1) Google Search Corpus for OK-VQA (Luo et al., 2021]: passage corpus collected for answering OK-VQA questions. Previous work has shown that the corpus is effective for OK- OK-VQA [Luo et al.
2021 Lin and Byrne 2022]. We use this corpus in evaluating VQA performance since it covers more
knowledse ledge for answering the OK-VQA questions.
(2) Wikipedia Corpus for OK- VQA: we collect this corpus by gathering all Wikipedia passages on common objects and concepts (c.g. umbrella, dog, hat) and those containing any of the potential answers in OK-VQA training set. This ensures the corpus covers useful knowledge for answering OK- VQA questions. We note that the collected corpus encompasses a multitude of semantically-diveer documents (>100,000) that challenge the retrieval system to identify actually useful documents. For example, all Wikipedia documents with the word 'party" are included in the corpus, ranging from descriptions of fairy tales to political parties. We also use 10% of the WIT dataset (Srinivasan] ct al. 202T], corpus based on Wikipedia with image-text pairs, to train the mapping network for multi-modal alignment.
We evaluate on two additional KB-VQA datasets demonstrate FLMR's generalizability.
(1) FVQA [Wang et al. 2017]: We follow RAVQA [Lin and Byme 2022 to preprocess the data. All knowledge triplets are serialized into text sequences form the knowledge base for retrieval. The average of 5 volss-valiotionn splits is reported.
(2) Infoseek [Chen et al.) 20236]: Infoseek is a newly proposed KB- VQA dataset that provides Wikipedia documents that can be used in answering its questions. We follow Chen et al. [2023b) in preprocessing. First, we remove questions whose answers cannot be found in the provided Wikipedia passages. Second, in additional to the documents covered in the dataset (~60,000), we include less relevant passages to form a knowledge base for retrieval (~1000,000 documents). The test set annotation has not been relcased, and so we split the official validation set again into validation and test sets (~5500 questions).
Training Setup. We use ColBERTv2 (Santhanam et al. 2022a and CLIP ViT-base [Rdforrd et al.
202T] to initialize the rext- retriever and vision encoder. For the DPR baseline, we use the official DPR checkpoints to initialize the retriever. In answer generation, we use Ta-ger [Raffel et al.) 2020 and BL----FAan--S-XL We use Nvidia 100 (80G) for all experiments. We give detailed training hyperparameters in Appendix E We use LoRA (Hu et al., 2022b to fine-tune RA-VQA-v2 (BLIP 2) on single GPU. The vision model is frozen throughout all experiments. During vision-language alignment training, only the mapping network is trainable. In training the answer generator, the re-
triever is frozen. Our implementations are released at https://githuu.ccm. Weizheeragnn//Reeiiiie Retrivvvl Augmented- Wisuull-uussionn Answering
Evaluation. We present the metrics used to assess the generated answer and the performance of our knowledge retriever below. All reported numbers are averaged from 3 runs with different seeds. We verified the significance of all mentioned improvements with scipy. stats. tttet_iendd (p < 0.05).
(1) VQA Score: To evaluate VQA performance, we use the official VQA Score [arrnn et al. 2019 which assigns score to the generated answer based its exact occurrence count in the set of human responses S:
#s(v) 3
VQAScore(y, min
1).
where #s(y) is the occurrence of y in human responses S. This score ensures that a model is partially rewarded even if it generates a less popular answer among the human responses [Luo et al. 2021).
(2) Exact Match (EM) awards point if any of the annotated answers generated exactly: EM(y, = min( #5 ssy),

---

(3) Pseudo Relevance Recall (PRRecall@K) To evaluate the retriever, we adopt pseudo relevance following Luo et al. [02T due to the absence of ground-truth knowledge documents for each query. A document is considered pseudo-relevant if it contains any human-annotated answers. PRRecall @K K measures whether the retrieved K documents contain at least one pseudo-relevant 1). where (dg. S) evaluates to 1 if the retrieved document dj contains any answer in S, and 0 otherwise. The metric is averaged across the test set.
document: PRRecall = min [Ek=
Baselines. To demonstrate the effectiveness of FLMR, we take a DPR retriever as a baseline. In later sections, FL.RR w/o Late Interaction refers to the corresponding DPR baseline. We use the same training data and hyper-parameters to build a multi-modal retriever based on DPR. For fair comparison, we keep the product No dy identical for DPR and FLMR that they have the same number of parameters in the mapping networks. Since DPR can only handle one-dimensional query and document embeddings, we sum the embeddings of the [CLS] token from Fr and the visual tokens from (Fv (.)) to reduce the dimension to 1 x dy (details in Appendix D
We also compare our VQA performance with the latest KB-VQL models. Amongst these mod- els, ConceptBERT Garderes et al., 2020]. KRISP [Marino al.) 202T), VRR (Luo et al.. 202T].
MAVEX [Wu et al. 2022]. KAT-TS [Gui et al.) [021]. TRiG-Fnsemble [Gao et al. 2022] and
RA-VQA are relatively small in model size (<1B), whereas PICa (Yagg et al.
2022]. KAT Gui
et al. 202T]. J. Prophet (Shao et al. [023), PromptCap (Hu et al. 2022a]. REVIVE Lin et al. 2022].
PALI [Chen et al. 2022b]. Flamingo [Alayrac et al. 2022]. PaLM-E Driess et al. 2023 use very
large models such as GPT-3 (175B) and PaLM-E (562B).
5
Results and Key Findings
5.1 VQA Performance
As shown in Table 1 recent models leveraged LLM or Large Multi-modal Models (LMMs) to achieve excellent performance OK-VQA. The best performance date is by PaLM-E, achieving a VQA score of 66.1 with 562 billion pre-training parameters. The original RÁ-VQA formalism achieves a lower VQA Score of 54.48 but with only 800 million parameters.
We first compare RA-VQA-v2 (with FLMR retrieval) with RA-VQA (with DPR retrieval). Our replication of RA-VQA (T5-Large) (Table 1 Row 22) achieves similar PRRecall@5 and VQA Score as the published results of RA-VQU (Table Row 8). Compared with RA-VQA (TS-large), RA-VQA v2 (T5-large) improves the PRRecall significantly from 83.08% to 89.32%, leading to a gain of 3.4 in VQA Score (51.45 to 54.85). This suggests that improvement in knowledge retrieval benefits answer generation via retrieval augmentation.
We also show the effectiveness of knowledge augmentation by comparing the underlying base models with their retreneal-augmeented version. As shown, T5-lerge and BLIP 2 (fine-tuned with OK- -K-VQA data) achieve 47.52 and 55.4 VQA Scores, respectively. Their retrieval -aggeeend-deemeeeeddde version, RA- VQA-v2 (T5-large) and RA-VQA-v2 (BLIP 2) gain 7.33 and 6.64 in VQA Score, respectively. For readers' interest, we provide more thorough analyses on the performance that the underlying answer generator model attains and the gain brought by knowledge retrieval in Appendix [
To confirm that based vision can aid LMMs such as BLIP 2 which already has its own image encoder in VQA tasks, we remove text-based vision from RA-VQA-v2 (BLIP 2) and BLIP 2 (fine- tuned). This results in a decrease in VQA performance from 62.03 to 60.37 and 55.44 to 54.10. respectively (Table 3) suggesting that text-based vision contains useful information not included in the visual features obtained by BLIP 2's own image encoders.
RA-VQA-v2 achieves comparable and even better performance when compared with systems that use very large (> 13B parameters) LLMs and LMMs With BLIP 2 (B=3), RA- -QA-Q outperforms Flamingo (80B) by 4.19 VQA Score. It also outperforms many recent systems that use GPT-3 (175B) as an answer generator or knowledge source, such as PromptCap, REVIVE, and KAT. It achieves similar performance to that of PALI (17B) (62.03 vs 64.5 VQA Score). With comparable parameter sizes, RA-VQA-V (BLIP 2, 3B) outperforms PALI (3B) by a large absolute margin (62.03 vs 52.40 VQA Score). We emphasize that RA-VQA-v2 can be used in conjunction with virtually any existing L.MS and LMMs to offer substantial improvements, as demonstrated by the Ta-le and BLIP experiments.
7

---

Table 1: Model Performance on OK- VQA. Knowledge Source abbreviations: C: ConceptNet; W: Wikipidia; GS: GoogleSearch; GI: Google Images. EM stands for Exact Match. VQA stands for VQA Score. stands for PRRecall. The best performance literature underlined.
# Model
Base Models
K
Knowl. Src.
R@5
EM
VQA
1
ConceptBERT
33.66
2
KRISP
C+ W
38.35
3
VRR
100
GS
45.08
MAVEX
W + W+C+ GI
39.40
5
KAT-TS
T3- large
40
W
44.25
6
TRiG-Ensennn Ensemble
Ta-large
100
W
54.73
50,50
7
RA-VQA (joint training)
T5-large
50
GS
82.84
59.41
54.48
8
RA-VQA
T5-large
5
GS
81.25
55.77
51.22
Systems based on large models (23B parameters)
9
PICa
GPT-3
GPT-3
48.00
10
KAT-Ensemblc
T5-large, GPT-3
40
W + GPT-3 GPT-3
54.41
11
Prophet
GPT-3
61.11
12
PromptCap
GPT-3
GPT-3
60.40
13
REVIVE
GPT-3
W + GPT-3
$8.00
14
PALI
PALI (3B)
PALI
$2.40
15
PALI
PALI (15B)
PALI
56.50
16
PALI
PALI (17B)
PALI
64.50
17
Flamingo
Flamingo (80B)
Flamingo
57.80
18
PaLM-E
PaLM-E (562B)
PaLM-E
66.10
Boselines without knowledge retrieval
19
Ta-large (fine-tuned) w/o knowledge
T5 large
51.38
47.52
20
BLIP 2 (fine-tued) w/o knowledge
BLIP 2 (T5-XL.)
59.49
55.44
Our proposed models (models w/o Late-interaction interaction use DPR instead of FLMR)
21
RA-VQA-v2 (T5-large)
T5-large
5
GS
89.32
58,85
54.85
22
w/o ROT & VE & Late-interactiin
T5-large
5
GS
83.08
55.89
51.45
23
RA-VQA-v2 (BLIP
BLIP 2 (T5-XL)
5
GS
89.32
62.01
62.08
24
w/o ROI
BLIP 2 (T5-XL)
5
GS
87.02
61.63
60.75
25
w/o ROI& & VE
BLIP 2 (T5-XL)
5
GS
85.99
59.95
60.41
26
w/o Late-interaction
BLIP 2 (T5-XL)
5
GS
82.90
59.00
58.20
27
w/o ROI & Late-interaction
BLIP 2 (TS-XL)
5
GS
83.43
60.18
$9.21
w/o ROI VE inte-interaction
BLIP 2 (T5-XL)
5
GS
83.08
59.49
58.70
5.2 Retrieval Performance
Rext-basee//eature-bassee Vision. As shown in Table previous retrievers (RA-VQA, VRR) achieve x=82.84% PRRecall @5 using only text-based vision (textual descriptions of visual scenes). We show that visual features obtained via aligned vision models (feature- vision) are equally effective as text-based vision. Relying questions only, FLMR has baseline retrieval score of 74.81 PRRecall @55. Incorporating text-based vision and frature-based vision increase PRRecall @ 85.99 and 85.08, respectively. Furthermore, feature-based vision provides information complementary to test-based vision, as demonstrated by the better PRRecall @5 at 87.02 when the two are combined. The same trend is observed for DPR-base retrieval system, though less dramatically (from 83.08 to 83.43). We note that pre-training the mapping network for vision-language alignment is crucial for good performance. Without such pre-training, performance degrades to 85.71. These results confirm that incorporating aligned vision encoders in the retrieval process compensates for the information loss in image-to-text transforms.
Effects of Late Interaction and ROIs. Late Interaction enables FLMR to capture fine-grained relevance of token-level embeddings. As shown in Table using the same query and document representations, upgrading DPR to FLMR leads to consistent improvement in retrieval performance by large margin up to ~6% (comparing Table Row 8 & 13).

---

Table 2: Retrieval performance Google Search (GS) and Wikipedia. Text-based vision refers to textual descriptions of images (such as OCR, caption, objects and attributes). Feature-base vision is obtained using neural vision model directly (e.g. ViT). R refers to PRRecall @K
#
Retriever
Text-based
Feature-based
GS
Wikipedia
Vision
Vision
R R@5
R@ 10
R@S
R@ 10
VRR
80.4
88.55
RA-VQA AA-VQA-FrDPR
81.25
88.51
3
RA-VQA
82.84
89.00
DPR
73.11
82.05
57.03
69.84
5
DPR
83.08
89.77
66.04
75.94
6
DPR
80.52
88.27
65,84
75.85
7
DPR
83.43
90.31
66.88
76.35
DPR
+9ROIs
82.90
89.95
65.86
75.90
FLMR
74.81
83.10
57.20
70.11
10
FLMR
85,99
92.79
66.50
76.80
11
FLMR
85.08
91.80
66,90
77.05
12
FLMR
87.02
92.69
67.50
77.60
13
FLMR
+9ROIs
89.32
94.02
68.10
78.01
14
w/o alignment pre-training
+9ROIs
85.71
92.41
66.40
76.10
Table 3: Removing text-based vision from
answer generation reduces the VQA perfor- mance, showing that text- based vision offers complete image understanding.
0.92
PRRecall@5
0.90
88.2,88
88.2
18.2,88.4.
88.8
89.3
88.3
0.88
Model
VQA Score
0.86
RA-VQA-v2 (BLIP w/o lext-based vision
62.03
83.8
0.84
83.0
83.2
82.6 82.8 $2.9
60.37
0.82
BLIP 2 (fine-tuned) w/o knowledge 55.44
FLMR
w/o text-based vision
$4.10
0.80
DPR
0.78
Table 4: Comparison of ROI selection meth- ods. R stands for PRRecall.
NROI
Figure PRRecall@5 versus the number of ROIs. Finer-grained ROIs cause performance degradation DPR, while FL.M captures them to improve retrieval performance.
R@5
R@ 10
4 Object-centric ROIs
88.01
93.62
Random ROIs
86.9
93.20
Evenly-split ROIs
86.96
93.16
In addition to token-level relevance, FLMR can utilize fine-grained Region-of-Interes (ROI) features with Late Interaction whereas DPR can not. This can be demonstrated by Fig. as the number of ROIs increases, DPR performance degrades. This may be because DPR's one-dinensional query and document embeddings are not expressive enough to encompass fine- -re-graided details of the ROI visual As shown in Table and Table Row 27-28. adding more ROIs effectively adds noise which adversely impacts the retrieval performance (83.43 to 82.9), and in turn worsen VQA scores (59.2 to 58.2).
Object-centric ROIs improve retrieval. We also conduct ablation studies to show that the perfor- mance improvements brought by ROIs come from the finer-graines information captured by them rather than from increases the number of features. We compare FLMR with 4 object-centric ROIs (obtained by object detection) against bascline ROI selection methods: (1) randomly crop 4 patches of size larger than 100 100 from the image as ROIs; (2) evenly split the image to obtain the top-left, top-right, bottom-left, and bottom-right of the image ROIs. As shown in Table 4 FLMR with 4 ROIs from VL object detection outperforms others, suggesting that is the object-centric, fine-grained ROIs that improve the performance.
Retrieval performance on FVQA and Infoseek. shown in Table 5 improvements with FLMR. FLMR with both text- and feature-based vision improves DPR by 2.3%
we observed similar

---

Table 5: Retrieval performance on FVQA Wang et al. 2017 and Infoseek [Chnn al. 2023b]. J
Average recall on 5 splits is reported for FVQA. FLMR outperforms DPR trained with the same data with clear margin.
FVQA Recall@5(Stte Infoseek Recall @
DPR FLMR (Visual Encoder) FLMR (Visual Encoder ROIs)
68.58(0.01)
44.88
70.88(0.01)
46.42
72.37(0.011)
47.08
Question: many does this
manny coez
bnimal
brown
RO1GROOO
RO14
ROIP
RCI
Taken pairs Document/ quory Ichert.
Document: meal-eating antiall such cal. has quite different teeth compared to grass-caling amimal, such as a horse. most cats have decidhous teeth and 30 permanent teeth.
Figure 3: Selected query tokens connected by document tokens that have the highest token-level relevance with them, computed by FLMR. For example, amongst all document tokens, '26' and '30' have the highest relevance with the query token 'how' and 'many', respectively. This shows that FLMR can capture fineggriined document relevance. Zoom in for better visualization.
and 1.54% Recall@5 FVQA and Infoseek, respectively. Incorporating ROI features further improves its performance to 72.37 on FVQA and 47.08 on Infoseek. This suggests that FLMR generalizable other KB-VQA retrieval tasks and bring steady improvements relative to baselines.
Qualitative analysis of FLMR retrieval. Figure shows FLMR retrieval in action. The orange lines connect the query token and the document token with the highest relevance score, which will be preserved after the MaxSim operation and will contribute the final retrieval score. We token-level interaction indeed captures fineggrineed relevance between the query and the document. For example, the retriever recognizes that the numbers "26" and "30" in the document relevant to "How many" in the query. We can also see that the image tokens are aligned with the text tokens: the image tokens corresponding to the cat (IMG, ROI3 and ROI4) point the words "cats" and "cat" in the document. This demonstrates the effectiveness vision-language alignment that gives rise explainable cross-modality relevance. We provide more case studies Appendix
that
highly
Conclusion
In this paper, proposed Fine-grained Late-interaction Multi-modal Retrieval (FLMR), the first of its kind leverage fine-grained token-level relevance between queries and documents for VQA tasks. FLMR incorporates feature-based vision using aligned vision model that complements text-based vision to enhance image understanding, improve retrieval performance and advance VQA performance. We achieve superior performance in OK-VQA. greatly surpassing previous systems with similar parameter size and closes gap with those systems utilizing very large (>13B) models.
10

---

7 Acknowledgement
Weizhe Lin was supported by a Research Studentship funded by Toyota Motor Europe (RG92562(24020)). We thank our colleagues. Daniel Olmeda Reino (Toyota Motor Europe) and Jonas Ambeck (Toyota Motor Europe), who provided insight and expertise in this project.
Prof. Bill Byrne holds concurrent appointments as a Professor of Information Engineering at Cambridge University and as an Amazon Scholar. This publication describes work performed at Cambridge University and is not associated with Amazon.
We would also like to thank all the reviewers for their knowledgeable reviews.
References
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised visual-retriever- reader for knowledge-based question answering. Proceedings of the 2021 Conference on Empirical Methods in Naturul Language Processing, pages 6417-6431, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https
//aclanthology-o 22g/2021....200 emnlp- main.517
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. wwasfformrrtrieee-eeeeeae Natural language-centric pusside-knowlede visual question answer- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5067-5077, 2022.
Weizhe Lin and Bill Byrne Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 2238-11254 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL httptt::/accaatttotott
org. emnlp-manie 772
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v1/2020.emml wtw.:0000..mmmmmmmim....... m50..550. URL htttps::/aclanthhll.gg//o
org. 20200 11:22..mmmlppmm emnlp-main. 550
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https::t / net//orrumPdd-iicbaall
Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE transactions on neural networks and learning systems, 29(1)):59475555 2018
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8317-8326, 2019,
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6281-6290, 2019.

---

Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned- Mealled-Mille,, and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10267-10276, 2020.
Dalu Guo, Chang Xu, and Dacheng Tao. Bilinear graph networks for visual question answering. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Hao Tan and Mohit Bansal, LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 5100-514 5111, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/w1/D19/ 1514. URL http:://ccllltttololog org/ D 1514
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adver- sarial training for vision-anddlanguuaee representation learning. Advances in Neural Information Processing Systems, 33:66 6-66288, 2020.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for rision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020a.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Sim VLM: Simple visual language model pretraining with weak supervision. International Conference on Learning Representations, 2022. httpttttt::::peeeeeeeeee
.nt./ wwwttfeuun/ Sarruuumiduuuuuuuuufe...
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 2592-2607, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.11863//1/20211.00 acl- 202. URL https: https://aclaathoooog org/ 2021 acl-long
202
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven C. H. Hoi. From images textual prompts: Zero-shot vqa with frozen large language models, 2023.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multi- modal generation on CLIP via vasion-language knowledge distillation. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 2383-2395, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1865 1.6.:////202...mnnnnnee..... URL httts://aclantthll.gg org/2022...... wtt.........nn.....ee findings----iii acl.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15617-15629, 2022. doi: 10. 10.1109/CVPR52288222222
I Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for aaiio--nd-language. arXiv preprint 2xXiv:2112.08//
2021.
12

---

Leroy Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. REVIVE: Re- gional visual representation matters in knowledge-based visual question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: htttps://ppenreviex: http://wwww...eeeonnnd//i/aa.... www.oru=wnnniindnrrro G5D
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-basee vqa. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 3081-3089 2022.
Yushi Hu, Hang Hua, Zhengyuan Yang. Weijia Shi, Noah A Smith, and Jieho Luo. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.088 1.09999, 2022a.
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing, Out of the box: Reasoning with graph convolution nets for factual visual question answering. Advances in neural information processing systems, 31, 2018.
François Garderes, Maryam Ziacefard, Baptiste Abeloos, and Freddy Lecue. Conceptbert: Concept- aware representation for visual question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: Findings, pages 489-498, 2020.
Guohao Li, Xin Wang, and Wenwu Zhu. Boosting visual question answering with context-aware knowledge aggregation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1227-1235, 2020b.
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge-basee vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2712-2722 2022.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Mareus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge- baaed vqa. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111-14121, 2021.
Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang. Jeff Z. Pan, Ningyu Zhang, and Wen Zhang. Lako: Knowledge-driven visual question answering via late knowledge-to-tee to-tttttt-text injection. In Proceedings of the / 1th International Joint Conference on Knowledge Graphs, IJCKG '22, page 20-29, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 97814708438 450399876. doi: 10.1145/3579051 35570553. URL htttt::t///ooo http://ooooorgg///// 000..1111//1111337799999
Chen Qu, Hamed Zamani, Liu Yang, W Bruce Croft, and Erik Learned-Miller. Passage retrieval for outside-knowledge knowledge visual question answering. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Informatinn Retrieval, pages 1753-1757, 2021.
Jialin Wu and Raymond Mooney. Entity-focused dense passage retrieval for outside-knowledge. visual question answering. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8061-8072, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 11116653/11222222mmlmmmm https://aclaathollog) org/2022.222222 wmn.l-mainnmmmm.
URL
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164, doi: 10.1145/3397271.
3401075. URL httph:ttt/////tt / wtt.://///.....///..... www.11111/337111.. 34010775
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon. Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715-3734, Seattle, United States, July 2022a. Association for Computational Linguisties. doi: 10. 1211166/1//22...oo www.lln.aaaaaaaaee mannnmmainn2.... URL
https: http://aaaaaathhhoo wrt.///........ooooo. nww.laaaal-aaaaaaa maalmmmin....
13

---

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-iman -mage pre-training. In International Conference on Learning Representations, 2022. URL. htttps://openreviews net/ forum??d=cpDhcsEEDD0
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. MuRAG: Multimodal retrieval- augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558-5570 Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. URL
httts://aalaanthollggg org/2022. emn www.p-mmmme..m..m..mmm halppmmmii33.
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. Plaid: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information Knowledge Management, CIKM '22, page 1747-1756, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 978 7781440332365. doi: 10.1 10.1145/351 00033511888.000 35557355. URL htt.:httts//oooo org/ 10. 1145/3511808, 3557325
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv, 2021. doi: 10.
08850/arxiv.2103.00022
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmenned genera- tion for wwwwedge-intensivg nlp tasks. Advances in Neural Information Processing Systems, 33: 1459-9474, 2020.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv preprint arXiv:2206.01718, 2022.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. arXiv preprint arXiv:2103.019133 2021.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40 (10):2413-244 2-227 2017.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming Wei Chang. Can pre-trained vision and language models answer visual information-seekee seeking questions? arXiv preprint 1171:3300..1713,, 2023b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jmlr otggpappess/v21220 074 html
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022b. URL htttpa://openrviiii wttprriwwwwweeeeeeeem net/forum?id= nZeVKeeFYfg
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3195-3204, 2019.
14

---

Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge- visual question answering. MoXXv::23030110033 Mar 2023, doi: 10. 10.48550/aXX1.23333.00000 URL [es].
arxiv. org hasg//ab//// abs/ab/s.././// 2303.01903 01903 arXiv:2303.01903
Xi Chen, Xiao Wang, Soravit Changpinyo, J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly- scaled multilingual language-image model. (arXiv:2209 067994),99.94))... Sep 2022b. doi: 10.48550/arXiv. MrXiv:2209.06799 [cs].
2209.06794. URL http::tt::/aree http:://aaxxvv.rrre
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. arXiv, 2022. doi: 00.48550/rxxv.2244..11 14198..
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint vrXiv:2303.03378 2023.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535-544) 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL Sctp://arxins arg/abs/
htte/bbb/1412..... 1412. 6980
15

---

A Limitations
We chose the Google Search corpus [Luo et al. 2021] for our question-answering system as provides good coverage of the knowledge needed and is publicly available. However, noted by the authors RA-VQA, additional knowledge bases may be required to answer some questions conrectly. Future work may address the issue by improving the quality and expanding the coverage of knowledge.
B Ethics Statement
We do not perceive any immediate ethical concerns associated with the misuse of our proposed system. There is possibility that the trained KB-V CB-VQA system might generate inappropriate or biased content as result of the training data biases during LLM and LMM pre-training and fine-tuning. Therefore, is advised to conduct an ethical review prior to deploying the system live service.
C Data Statistics
Table @ the data statistics of the OK- VQQ dataset. 7 displays the number of passages in the document collections used for evaluating the retrieval systems. Note that the WIT corpus is introduced in Appendix H which used for investigating the retrieval of multi-modal documents.
Table 7: Data statistics of document collections used retrieval.
Table 6: OK-VQA dataset statistics.
Category
Number
Corpus
passages
train questions
9,009
GS for OK-VQA [Luo et al. Wikipedia for OK-VQ- WIT for OK-VQA (Appendix H
2021
168,306
valid questions
5,046
114,637
images
14,055
87,419
D Details of DPR baselines
We build a DPR retriever as a baseline for FLMR We apply the same pre-training strategy, training data, and hyperparameters the product No. x dy and the number parameters of the vision mapping networks identical for FLMR and DPR for fair comparison. Since DPR only handle me-dimensional query and document embeddings, we sum the embeddings of the [CLS] token from F.,()) and the visual tokens from FM (-)) reduce the dimension dr, Formally, the query and document embeddings are:
construct multi molti retriever based on DPR. Particularly, keep
Qdpr Fr.cus(9) + FM((F(()))
Σ Fm((F(())) NROS
Rdi
(10)
Ddpr = FL,CLL(d) FFv E Rdc
where La is the image of the document multi-modal document collection is used and otherwise omitted. The inner product search (supported by FAISS [Johnson al. 2019 is used train and retrieve documents with DPR.
E Training and Hyperparameter Details
We use ColBERTv2 and ggenai/clip-vit-beee vision encoder. For the DPR baseline, we use secebbok/dpr--sigl--nn-ee basse to initialize the retriever. In answer generation, t5-large and Salesforce/blipt -fl---5-xtt
to initialize the text-based retriever and
16

---

With
dy 768. For FLMR, we use N,t = 32 visual tokens per image representation and d,, 128. For DPR, we use Not and dr. 768 that the number of parameters of vision mapping network is similar to that of FLMR: Not x dz, ~ 128 x 32. The mapping network consists of two fully-connected layers with tanh activation. The output of last layer is reshaped into Nur X dz visual tokens. Other model parameters are: 1 = 512, la = 512. NRRT = unless otherwise specified.
We use 1 Nvidia A100 (80G) for all experiments. The optimizer is Adam [Kingma and Ba 2013 In training the retrievers, we use learning rate 10 batch size 30, gradient accumulation steps for 10k steps (for both DPR and FLMR retrievers). When training RA-VQQA VQA-2 (T5-large), we use learning rate 6 10 batch size 2, gradient accumulation 16 for up to epochs. We use linearly-dccaying scheduler to reduce learning rate from 6 x 10-5 to 0 after 20 epochs. We use LoRA (Hu et al. 2022b) to train RA-VQA-v2 (BLIP2) with learning rate 10-1, batch size 4, gradient accumulation steps 16 for up to 6k steps. LoRA is configured to use the default huggingface-PERl PPFFT setting: r=8, lora_alpha=32, lora_dropout=0)
The vision model is frozen throughout all experiments. In pre-training the mapping network, only the mapping network is trainable. When training the answer generator, the retriever is frozen.
We report the required GPU hours on Nvidia A100 (80G): for lisoun-lagguage alignment of retrieval models, approximately 4 GPU hours needed. Training the FLMR retriever requires around 12 GPU hours (10k steps) including the time of running testing after training is complete. Training RA-VQA-v2 (BLIP 2) with LoRA requires around 12 GPU hours (6k steps) including the time of running validation per 1 steps. Training the RA-VQA-v2 (T5-large) requires around 12 GPU hours (3k steps) including the time of running validation every 500 steps.
All implementations are released at http::/gghhhbooooo////////eeeeee Visual-Question-Answeii
F Artifacts and License
We list the resources used and their License below:
(1) huggingface-transeene transformfrrs (Apache License 2.0) provides pre-trained model checkpoints for BLIP
2, DPR, T5, and their tokenizers:
(2) PLAID engine and ColBERTv2 checkpoints (MIT License): https://gthubbcoo/stttrrrr
(uturedatsCooBBERT
(3) FAISS Johnson et al. 2019) (MIT License) is used to index document embeddings for fast retrieval with DPR: https:/ http:///hhubboo////oooroesseeeee
(4) huggingface- PEFT (Apache License 2.0) for parameter-efficient LoRA fine-tuning:
htgss//gthhbbooommngiigghceceheh
(5) The official RA-VQA
implementation (GNU General Public License
v3.0):
Aug.eeete----eeeee--eeeee-eeeeee Visuual-uestion-Amee
G Computational Cost
We report the computational cost in this section.
Table Training and indexing time for FLMR and DPR. Training batch size is 30. The corpus for counting the indexing time is the Google Search Corpus for OK-VQA-
train per 1000 steps
indexing time
FLMR
1.2h
0.28h
w/o ROI
1h
0,25h
w/o ROI & VE
0.7h
0.24h
DPR
0.5h
0.2h
17

---

Though Late Interaction allows rich interactions at token level and outperforms DPR by a large margin, it also introduces additional latency retrieval. As shown by T the training time of FLMR is increased from 0.5h to 0.7h when late interaction is introduced. This latency increase comes from the more complicated token-level loss. When Vision Encoder (VE) and ROI (Region of Interest) are added, the time cost is increased to 1h and 1.2h respectively due to the additional trainable parameters of the mapping network. However, the indexing time does not increase significantly when VE and ROI are added to the FLMR retriever. We note that the FLMR spends slightly more time to build the search index when compared DPR because an extra clustering step by PLAID [Santhanam et al.
2022b is required to conduct fast retrieval.
Table 9: Training and inference time of the whole system. Please note that passages are dynamically retrieved, and thus the training and inference time already takes the retrieval latency into account. Batch size set to for both training and inference time. w/o ROI & VE means removing the vision encoder in FLMR.
Retriever
Generator
Training Speed (iterations/sec)
Inference Speed (iterations/sec)
FLMR
T5-large TS-large
1.16
1.11
DPR FLMR
1.73
1.67
BLIP 2
1,24
0.98
FLMR (w/o ROI& & VE)
BLIP 2
1.43
1.00
DPR
BLIP 2
2.14
1.30
When FLMR is integrated into the full VQA pipeline (we take the BLIP version for example), reduces the training speed from 2.14 iterations/sec to 1.24 iterations/scc (42%) since the retrieval process run on the fly. However, in retrieval, the inference speed is only reduced from 1.3 iterations/sec to 110 iterations/see, which is still affordable when considering the performance boost. The major computational cost remains that of training the answer generator with a great number of parameters.
H Retrieving Multi-modal Documents with FLMR
We additionally show that our proposed FLMR system can also be used to retrieve multi-modal documents. Since this is not the focus of our paper, we present the investigation in this appendix.
Dataset. We select a subset from WIT [Srinivasan et al., 2021], a knowledge corpus based on Wikipedia where the images associated with the documents are also present, to make an image-text corpus for retrieval. We adopt the same selection process as for the Wikipedia corpus introduced in Sec. 4. The dataset statistics is shown in Table
Multi-Modal Late Interaction. We upgrade the document embedding process to accommodate the document image. The documents in the knowledge base are represented by embeddings D which are obtained from the document content d and its associated image Ia:
D [F((d, FM (Fv e Rloxdz,
(11)
where (D La + Not. and la is the length of the document d. We compute the relevance score between question-image pair q = (g, 1) and document d (d, Ia) as follows:
1e max Q.D.J
r(q.d) r((g. r((1a)=r((g, (4,
(12)
Discussion. Both query and document embeddings are multi-modal Since the same image/text encoder is used to encode images I, Ia and texts q, d, respectively. Image-wise and text-wise relevance contribute to the final relevance score; After cross-modality alignment, the vision encoder FM (Fv should produce image embeddings close to the text embeddings produced by Fr.(.) in the latent space if the image is relevant to the question, thereby taking the relevance between J, and q, la into account during knowledge retrieval.
18

---

As shown in Table the retrieval scores see slight improvement when document images are also considered (from text-only to multi-modal). This suggests that FLMR supports retrieving mullii-mdda documents.
However, we that the gain of incorporating images is marginal. This is because WTT is strongly text-driven knowledge base as the images are already captioned by human experts. The surrounding of images are already dense and informative, which can be searched by FLMR easily, By manual inspection, we also notice that it is very rare that OK-VQA questions seek a document that can only be found by its accompanying images. This also explains the marginal gain we have observed.
In conclusion, we show that FLMR can also be applied to retrieve multi-modal documents, although more challenging questions and better datasets needed to fully exploit its potential. We leave this as future work.
Table 10: FLMR performance when retrieving documents in WIT. Models suffixed by 'uni-modal" only encode document texts, while 'multi-modie variants encode document images with vision encoders.
Model
PRRecalI@5 PRRecall @ 10
DPR-text-only
68.24
77.13
DPR-ime OPR-image-only
46.29
57.70
DPR- mpR-mmuti-modal
68.78
77.90
FLMRR-text-only
72.63
81.52
-.MM--iaag--oll
45.75
57.92
FLMR-multi-mon
73.65
81.89
I Effects of Retrieved Knowledge
Table 11: Comparing Hit Success Rate of RA- and RA-VQA.
Hit Success Rate
RA-VQA-v2 (BLIP2)
9.38
RA-VQA (BLIP2)
7.86
RA-VQA-v2 (T5-large)
17.62
RA-VQA (T5-large)
15.01
It is important understand the task performance that base model has attained and the gains from knowledge retrieval. We use the official evaluation metrics from RA-VQA: the Hit Success Ratio (HSR) which counts questions that cannot be answered by the base VQA model alone and thus require external knowledge to answer.
HSR = 1 SAÖNK $ S):
(13)
where yNk denotes the generated answer from fine-tuned base model when relevant knowledge provided. HSR reflects the net value of incorporating external documents into answer generation. We can conclude from Table that R RA-VQA-v2 steadily improves the HSR of RA-VQA by 2%, showing that the gains VQA performance come from improved knowledge retrieval. We also observe that T5-large, HSR). This is because the base language model of BLIP 2, Flan-TS-XL is significantly stronger and is able to answer more questions without the aid of external knowledge. This suggests that KB- performance can be improved by either (1) applying stronger base VQA answer generation models, and (2) collecting knowledge documents of higher quality.
earlier language model, relies more heavily retrieved knowledge (>15
We also conduct experiments while increasing K in Table aadd find that the system performance improves gradually until a saturation point. We notice that the saturation point of FLMR is at around K 10 while that of DPR is at K 20. This suggests that the useful documents are clustered around higher ranks in FLMR compared to DPR.
19

---

Table 12: Performance improvements with increasing number of retrieved documents.
K
5
10
20
50
DPR T5-large
VQA Score
51.5
51.8
52.3
52.1
Recall
83.08
89.77
94.05
97.25
FLMR T5-large
VQA Score
54.9
55.3
55.4
55.4
Recall
89.32
94.02
96.87
98.67
J
Case Study
case study is presented in Fig. case.
compares the model outputs and provides expalanations to each
20

---

RA-VOA
Closse Prediction
dhrisimas drrar?? these nouthwatering recpeis... ... the sweet and spity turkay giaza gives the bird war derful flavor It's been tradttin in my home for au long as can remember.
Christmas
turtey dosen't have rule rocet at your hollday fenet extremery french way cooking duck recommended heeneagtingg
Thankaghving
thanksghing,
to on turkey dinner also up eating turkeys during holddys??
Thankagiwing
with
RAA V4.
Chosce Production
Question: On what holiday do people traditionally
the same traditinal airkssiiing menu, with lafkey, cranberry seuce, stufting, and pumpkin pie takingg up the most real estate on our plates.
Thanksghving
Caption: large chicken walking through a field of grass.
on thankaghing, crvistmes.
to on lakey diener. the also up with the traditin eating lutkes during holddays?
Thanksgivingg
thankagiving day holiday, tirkeys began
Thankkghvvnng
dinner
Explanation: RA-VQA-12 succcecasfully captures "turkey" retrieved documents "turkey" "tradition". contrast, RA-VOA, without fine-grained information, gives high scores noisy documents.
RA-YOA
sense organs are body organs which are able hear, tasta,
Chosce
memory racegniw
reswrchers frund general undentanninng the extent which anmais thrnge mnctttiniiioonn mnmmrry ongeliion amotionn intw ligenne and altempiing answer abows queelian the swl sil out.
small
RA-YQAer2
sense animal known for?
puabaa
dog's
Caption: brown and white dog sitting front of laptop computer.
hunters
hound
powerful
hounds historically
smeel
wher 1883, the when president lincain made thankogiving national holiday, turkeys land dinner plates acrosa the counttyy.
smoll
Explanation: RA-VQA fails capture all three key concepts "dog", "animal" and "sense" in the query, which results in retrieving noisy, less relevant documents.
RA-YQ4-v2 w/e Region Interest
normally
the animal kingdom the dopartment, attracing
Chosse Prediction
once the tird had been plucked, it was abvious that half skallaoon was much bigger than the loft
bird
you' ready to give incusiiis that abuse ammels the bird. hern are some lerns to avaid and suggasaions about what to purchese instwad: boss and feather duslersen syowinnse invedliatioo largest catrich s saughte compar les the world at owed that...
catricn
RA-VQA-22
Chuen Prafiction
What animal is the woman's shirt replicating the pelt?
Question:
tastron ones which replicate the für or skin of animats like leopards, Sgere, zebras, girafles, hyenas, morkeys and mush more. after the animal intates, instance loopand print skits, zoora print top and on.
antmal mimiccing leopard, essem world behir
cheetah would
Caption: group of people sitting at table in a restaurant.
leopard
animal print ciahhing and fashion style in which the garment is made to resemble the pattem of the skin and fur on animal such as lecpard, cheetsh, snow leopard, jquar: zebra, liger, dlouded leopard. margay, ocelot, spotted hyano, strped tyena, atricnn wild dog, constrctor doot snake, grafte or monkey.
loopard
Explanation: without fine-gras fine-grained aigion-of-interest-.oo -nterret---bsedd features, FLMR fails to locate the "woman" and "shirt" in the image. When ROI features are added, FLMR successfully matches "leopard" from the image representations.
Retrieved Knowledge
Correct Final Prediction
Wrong Final Prediction
Figure 4: Case study comparing some model variants. Explanations given to each case. Please zoom for the best visualization.
21


Reviews:
[{"review_id": "OLLQwd92ME", "reviewer": "Reviewer_d3hH", "summary": "This paper focuses on retrieval augmented visual question answering and addresses two major limitations in previous RA-VQA. Specifically, they propose FLMR and overcome the limitations by obtaining image representations using a vision model. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained relevance between queries and documents. To verify the effectiveness, FLMR conducts extensive experiments on the OK-VQA dataset.  ", "questions": "1. The authors should further clarify the technical novelty of this paper. \n2. In line 163-164, please explain the improvement and details of pre-training the mapping network. ", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper solves the two limitations of previous RA-VQA. It is important for improving the effectiveness of RA-VQA.\n2. This paper is well-organized and easy to follow.\n3. The proposed model is effective with much less parameters than large models. ", "weaknesses": "Actually, the novelty of this paper is somewhat trivial. The multi-modal late interaction has been proposed by ColBERT. Besides, using multi dimensional embeddings to improve retrieval is not very novel. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness and Question 1**: \n\nPlease kindly refer to the general response where we provide a list of our technical novelties. If accepted, we will highlight these points more clearly in the camera-ready version following your suggestions.\n\n**Question 2: In line 163-164, please explain the improvement and details of pre-training the mapping network.**\n\nThank you for your question and suggestion. We will try to accommodate all details in the main content using the additional page and set pointers at Line 163-164 to direct readers to the discussion of improvements (Line 284-287) and details of hyperparameters and architecture (currently in Appendix E, Line 40-45). We apologize for not being able to fit all practical details in the main content due to limited space. \n\nHere for your convenience, we paste the information that you might be interested in:\n\n- **Details**: For FLMR, we use $N_{vt} = 32$ visual tokens per image representation and $d_L = 128$. The mapping network consists of two fully-connected layers with tanh activation. The output of last layer is reshaped into $N_{vt} × d_L$ visual tokens. We use 1 Nvidia A100 (80G) for all experiments. The optimizer is Adam. We use a learning rate of 1e−4, a batch size of 30, and 2 gradient accumulation steps for around 8k steps.\n- **Improvement**: The improvement brought by pre-training is analyzed in Line 284-287: we observe that pre-training the mapping network for vision-language alignment is crucial for good performance. Without such pre-training, performance degrades to 85.71 Recall@5 (Table 2 Row 14) from 89.32 Recall@5 (Table 2 Row 13). We will set a hyperlink in Line 164 which directs readers to this analysis in revision. Thank you for pointing this out."}, {"review_id": "BSYl46nkFY", "reviewer": "Reviewer_Bs2i", "summary": "In this paper, the authors focus on RA-VQA framework to solve the KB-VQA task. Specifically, this paper proposed Fine-grained Late-interaction Multi-modal Retrieval (FLMR) to address the limitation of inaccurate image representations and one-dimensional embeddings. FLMR leveraged late interaction and multi-dimensional representations to capture fine-grained relevance to retrieval performance. The proposed methods achieve sota performance in the OK-VQA dataset compared with systems with similar parameter sizes.", "questions": "Have the authors performed experiments on other KB-VQA datasets or is that planned? If not, do authors have any particular reason for the missing of multiple datasets?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper mainly focus on improving multimodal retrieval for the KR-VQA system, in detail, the proposed method uses an additional vision model to capture fine-grained features and align with text features instead of only using image-to-text features. The experiments show that the introduction of image representations from large vision models can lead to better retrieval and VQA performance which could benefit further research. The results on OK-VQA dataset and ablation study show the effectiveness of the proposed method.", "weaknesses": "This paper only reports results on a single dataset KR-VQA and thus didn't clearly show if the conclusion could be generalized to other related benchmarks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of the strength of our work. We hope that you enjoyed reading our paper. Below we would like to respond to your major concern and question:\n\n1. **Have the authors performed experiments on other KB-VQA datasets or is that planned? If not, do authors have any particular reason for the missing of multiple datasets?**\n    \n    We have performed experiments on additional datasets, including those that are released very recently. We provide additional results on F-VQA [1] and preliminary results on Infoseek [2] in the general response to address your concern about generalizing to other datasets.\n    \n    We chose OK-VQA for its relevance to our research focus on knowledge retrieval, as it is the largest and most challenging knowledge-based VQA dataset. The alternatives, such as F-VQA and A-OKVQA, are less suitable for evaluating our specific system due to their limitations. For instance, F-VQA's small size and biases and A-OKVQA's emphasis on visual reasoning rather than knowledge retrieval posed significant challenges. Please kindly refer to the general response for detailed rationales for our dataset choice.  We will also release the codebase which benchmarks our system on these datasets.\n    \n\nWe believe that the additional results and clarifications provided could address the concerns you raised, and we hope this strengthens the case for the acceptance of our paper.\n\n[1] FVQA: Fact-based Visual Question Answering\n\n[2] Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?"}, {"review_id": "7lrPajd7gU", "reviewer": "Reviewer_ujV3", "summary": "This paper proposed an updated version of RA-VQA by incorporating a vision model aligned with text-based retriever in a late interaction way. Extensive experiments show the effectiveness of the proposed method. Thorough ablation studies and analysis were conducted.", "questions": "See the above weakness part.", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper is well-written and easy to read.\n2.\tExtensive experiments and analysis are provided to show the effectiveness of the proposed method.\n3.\tExperimental results demonstrate the superiority of the proposed method.\n", "weaknesses": "1.\tThough the performance shows the superiority of the proposed method, the method of adding visual features is somewhat incremental compared to RA-VQA.\n2.\tWhy is the performance of adding ROI&VE in w/o late-interaction setting (row 26 in Table 1) worse than w/o them (row 27)? This result is quite strange.\n3.\tIt would be more intuitive to add the model size of each model in addition to the model names in Table 1.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your very positive review of our work. We are excited to learn that you enjoyed the presentation and the experiments.\n\nWe would like to respond to your concerns as below:\n\n**1. Though the performance shows the superiority of the proposed method, the method of adding visual features is somewhat incremental compared to RA-VQA.**\n\nThe original RA-VQA system did not consider dense image features in retrieval, and many recent KB-VQA systems (such as KAT) have similar limitations. We agree that it is an intuitive step to introduce dense image features, but how to incorporate these features in the context of multi-modal late interaction is non-trivial. We studied:\n\n- What image features are most beneficial for retrieval. We find that adding object-centric regional image representations is more effective compared to a single global representation and evenly split patches.\n- How to align the image and text modalities so that fine-grained cross-modality relevance can be assessed via late interaction. To this end, we propose a simple yet effective pre-training strategy that costs only 4 GPU hours to align the modalities under the late-interaction setting with a mapping network.\n\nPutting them together, we further show that FLMR can handle regional fine-grained image details with token-level interaction while DPR cannot, leading to superior retrieval and VQA performance. Please kindly refer to the general response for a more detailed listing of our technical contribution and novelty. \n\n**2. Why is the performance of adding ROI&VE in w/o late-interaction setting (row 26 in Table 1) worse than w/o them (row 27)? This result is quite strange.**\n\n- Thanks for your careful reading. This in fact reveals a desirable property of FLMR compared to DPR: incorporating ROI features improves the performance of FLMR, while this is not the case for DPR. We believe this is because the late-interaction mechanism allows FLMR to fully utilize the ROIs features, pin-pointing each of them to the relevant part of the document (Figure 3). In contrast, DPR’s one-dimensional embedding falls short of utilizing the rich ROI visual features.\n    \n    These findings are supported by experimental results: The retriever at Row 26, Table 1 corresponds to DPR with visual encoder and region of interest features (Row 8, Table 2), while the one at Row 27 refers to the original DPR retriever at Row 5, Table 2. If we take a closer look at Table 2, we can observe that adding feature-based vision (Visual Encoder) improves performance (comparing Row 5 and Row 7). However, the performance is reduced when ROI features are incorporated (comparing Row 7 and Row 8). We discussed this behavior in Figure 2 and Lines 292-297. We will add one more line in the ablation study in Table 1 to demonstrate the point more clearly - DPR with VE without ROI: Recall@5=83.43, EM=60.18, VQA Score=59.21. Comparing this result to Row 27, Table 1 shows that introducing ROIs to DPR significantly harms performance. We will use the additional space to explain it more clearly if accepted.\n    \n\n**3. It would be more intuitive to add the model size of each model in addition to the model names in Table 1.**\n\nThank you for your kind suggestion. We had the model sizes in the table before but we removed them to fit the table into the page. We will try to reduce the font size in the camera-ready version if accepted.\n\nFinally, thank you for your valuable support for our work."}, {"review_id": "6ZXdPP8paK", "reviewer": "Reviewer_XziC", "summary": "This paper introduces a new method, Fine-grained Late-interaction Multi-modal Retrieval, to improve knowledge retrieval in RA-VQA. The method addresses two major limitations in RA-VQA’s retriever:  incomplete image understanding and lossy compression of visual scenes and questions.", "questions": "I wrote above.", "limitations": "I wrote above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper proposes to capture fine-grained, cross-modality relevance that significantly improves retrieval performance.\n2) The paper introduces a simple yet effective alignment procedure that can complement image representations obtained via image-to-text transforms, leading to more complete image understanding, better knowledge retrieval, and higher VQA accuracy.\n3) The method improves approximately 8% in knowledge PRRecall@5 over other state-of-the-art retrievers in the OK-VQA dataset.\n", "weaknesses": "1) Method. This work used the VinVL approach, which leverages regions of interest for visual features. I could not pinpoint the main contributions of this work, which DPR does not make. For a better understanding of the differences between the two methods, perhaps the authors could present them side-by-side in a figure. \n\n2) FLMR and DPR. Why FLMR preserves richer information better than DPR representations? Because it also uses regions of the image with object class labels for predictions? I understand that FLMR allows full interaction between every query embedding vector and every document embedding vector. Is this the only difference?\n\n3) In-batch negative sampling. Has DPR also used in-batch negative sampling? If not, did you try to compare when DPR uses in-batch negative sampling? How significant is this to the proposed method?\n\n4) Results. Regarding Table 1, I am surprised to see that the comparison against BLIPv2 is not that large, while the proposed method uses the BLIPv2 caption generator. If PALI would use BLIPv2 text decoder, would it be better? I am unsure whether the main contribution is related to the BLIPv2/T5 text decoder or to better modeling.\n\n\nOverall, I appreciate the paper as it presents good results, but the contributions are largely unclear. I am open to the authors' feedback and other reviewers' opinions.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and kind suggestions. We will answer your questions point by point below:\n\n1. **This work used the VinVL approach, which leverages regions of interest for visual features. I could not pinpoint the main contributions of this work, which DPR does not make. For a better understanding of the differences between the two methods, perhaps the authors could present them side-by-side in a figure.**\n    \n    Our work uses VinVL to identify regions of interest, from which regional visual features are subsequently extracted using ViT. VinVL can be replaced by any other better object detectors in principle. Our contribution lies in the enhancement of the DPR retriever through the application of multi-dimensional embeddings and late interaction processes within the context of multi-modal retrieval, which is tangential to VinVL. \n\n    To allow a fair comparison, we add dense image features and ROI features to both FLMR and DPR in our ablation study (Table 2, Row 8 and Row 13). The DPR retriever with VE and ROI features follows the formulation in Appendix D. We find that the performance gain of FLMR is mainly derived from the multi-dimenstional late interaction mechanism, rather than the addition of global and regional visual features.\n    \n2. **FLMR and DPR. Why FLMR preserves richer information better than DPR representations? Because it also uses regions of the image with object class labels for predictions? I understand that FLMR allows full interaction between every query embedding vector and every document embedding vector. Is this the only difference?**\n    \n    Preserving token-level embeddings and allowing full interaction is indeed key to FLMR’s richer information. We rigorously investigate how FLMR compares with DPR as follows:\n    \n    First, token-level embeddings with late-interaction already improve the retrieval performance without dense visual features, as both the document and the query (which consists of not only questions, but also a series of detected objects, image captions, and OCR results) can be long in length, calling for fine-grained relevance computation. We demonstrate this by comparing FLMR and DPR in a setting where dense image features are not provided to either system (DPR: Row 5, Table 1; FLMR: Row 10, Table 1). \n    \n    Second, FLMR’s multi-dimensional representation and late interaction mechanism makes it possible to utilize region-of-interest features obtained from a vision encoder. However, we observe that DPR failed to utilize fine-grained ROI features (Line 288-297). This is because DPR’s one-dimensional query embeddings are not expressive enough to encompass fine-grained details of the ROI visual cues. As the number of ROIs increases, DPR’s performance is negatively impacted (Figure 2). In contrast, FLMR allows each ROI regional feature to interact with each of document embeddings, achieving increasingly better performance as the number of ROI increases (Figure 2). Other reviewers (xLzi) have found Figure 3 helpful in demonstrating that each ROI feature in FLMR is meaningfully connected to document token embeddings.\n    \n3. **In-batch negative sampling. Has DPR also used in-batch negative sampling? If not, did you try to compare when DPR uses in-batch negative sampling? How significant is this to the proposed method?**\n    \n    Yes. DPR used in-batch negative sampling. For a fair comparison, we strictly applied the same training configuration to FLMR and DPR (Line 155-156, Appendix D). In practice, in-batch negative sampling improves 1~2% Recall@5 for both systems.\n    \n4. **Results. Regarding Table 1, I am surprised to see that the comparison against BLIPv2 is not that large, while the proposed method uses the BLIPv2 caption generator. If PALI would use BLIPv2 text decoder, would it be better? I am unsure whether the main contribution is related to the BLIPv2/T5 text decoder or to better modeling.**\n    \n    In brief, we did not use BLIPv2 as the caption generator, although BLIPv2 and T5 are used as the answer generator. In our architecture, the answer generator takes in the query and the retrieved documents to predict answers. We do not modify the structure of the answer generator, and thus the choice of answer generator is flexible. Our main contribution lies in the improvement of the retriever part - we improved the retrieval by proposing FLMR to replace DPR, and showed that better retrieval quality enhances the ultimate VQA performance. We demonstrate by experiments in Table 1 that both large language models (such as T5) and large multi-modal models (such as BLIP2) can benefit from FLMR. In this sense, if we choose PALI as the answer generator (if one day it is publicly available), we can also augment PALI with retrieved documents from the retriever. The PALI (retrieval-augmented)’s performance will very likely be improved when we upgrade DPR to FLMR.\n    \n\nThank you very much for your support for our work. We hope that our contribution is more clear to you after the rebuttal. Please don’t hesitate to ask follow-up questions if our explanation is not clear enough."}, {"review_id": "9gYWhqAxOn", "reviewer": "Reviewer_xLzi", "summary": "This paper proposes a multi-vector (late interaction) multi-modal retrieval-based approach for Knowledge-based Visual Question Answering, inspired by the ColBERT multi-vector representations for text. The retriever improves the recall substantially over prior multi-modal methods that use single-vector encoding. The approach obtains token-level embeddings for both textual input and visual input. They also align the vision and text modalities, by training a mapping network. The paper achieves very strong SoTA scores on OK-VQA.", "questions": "n/a", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper's experiments are intuitive and extensive. They experiment test two types of visual representations (text-based and feature-based), aligning vision and text modalities, multi-modal late interaction, etc. in a series of very instructive ablations.\n\n2. The results are very strong and convincing, aligned with the strong intuitions offered in the writing.\n\n3. The paper offers nice, though minimalistic, qualitative analysis of their (nicely interpretable) retrieval.", "weaknesses": "1. What are the connections to FILIP (Yao et al., 2021), which is also a late interaction retriever for some multi-modal settings. FILIP seems to tackle zero-shot image classification and image-text retrieval, which are much simpler tasks perhaps, but the connection must be explored in the paper.\n\n2. The authors do report quite a few ablations for the retriever and the full method. That's great. But it's not necessarily great to report so many results on the test set! It might be better in the future to report such ablations on a validation set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very careful reading and the accurate summarization of our strengths. Thank you very much!\n\n1. **What are the connections to FILIP (Yao et al., 2021), which is also a late interaction retriever for some multi-modal settings. FILIP seems to tackle zero-shot image classification and image-text retrieval, which are much simpler tasks perhaps, but the connection must be explored in the paper.**\n\n    Thank you for your suggestion. We will add discussions about this work in the camera-ready version. We provide a detailed discussion for your interest.\n\n    FILIP and FLMR are similar in two aspects:\n\n    - First, both systems are dual-stream models: FILIP and FLMR encode text and image modalities with separate encoders and learn aligned, joint embedding spaces for the two modalities.\n    - Second, both systems explicitly model the interactions between token-level embeddings to capture fine-grained similarity/relevance. In a broad sense, both systems make use of late interaction in aggregating token-level interactions, keeping the computation tractable, though the mathematical formulations are not exactly the same.\n\n    The difference between FILIP and FLMR are three folds.\n\n    - First, as you mentioned, the task of FILIP is finding related texts with images. Their query is single modal (either image or text). The text being matched is usually concise and straightforward. In contrast, in our task, the query consists of not only images but also questions and text-based vision (image captioning, OCR, detected objects), leading to long and complex multi-modal queries. This is one of our motivations to employ token-level embeddings to allow full interactions between complex multi-modal queries and long knowledge documents.\n    - Second, FILIP pre-trained the whole system from scratch for text-image alignment. Conversely, our strategy is predicated upon aligning any current vision encoder (including ViT, CLIP ViT) with any late interaction text retriever (for example, ColBERT, ColBERTv2 and so on), making the retrieval performance grow with the development of general-purpose vision encoders and late-interaction text retrievers. As an additional benefit, we find that only 4 GPU hours are needed for alignment.\n    - Third, the formulations for fine-grained image features are different. We use object-centric regional image features, while FILIP uses evenly-split regional features. We demonstrate in Line 298-305 that object-centric ROIs achieved better performance than evenly-split regional features. We believe that FILIP can also benefit from our formulation.\n   \n     We will cite FILIP and discuss the connections in the Related Work section in the final version if accepted. Thank you for your knowledgeable help in improving our paper.\n\n2. **The authors do report quite a few ablations for the retriever and the full method. That's great. But it's not necessarily great to report so many results on the test set! It might be better in the future to report such ablations on a validation set.**\n\n    Thank you very much for your suggestion. In our future work, we will follow your suggestion to report ablation more rigorously.\n\nFinally, thank you for your valuable support and thorough analysis for our work! Thank you for your careful review."}, {"review_id": "v8HjjsF5H3", "reviewer": "Reviewer_shyj", "summary": "The paper proposed a fine-grained Late-interaction multimodal knowledge retriever for knowledge-based VQA. The core idea to the to both visual tokens and question tokens as token-level queries to retrieve document in late interaction fashion. The paper suggests that (1) squeezing query representation and document representation into one vectors hinder the retrieval performance. (2) fine-grained visual representation helps retrieval. The paper also show that the framework can be easily extended to retrieve multimodal documents", "questions": "Please comment on the weakness part", "limitations": "The  authors adequately addressed the limitations", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) The paper  provide a recipe to take advantage of late Interaction and multi-dimensional representations to capture fine-grained, cross-modality relevance to improve retrieval performance that further boost the (A)OK-VQA tasks\n\n(2) The paper perform very thorough analysis on the retrieval results using different corpus choices for OKVQA tasks (Google search and wikipedia) \n\n(3) The paper is well-written and easy to follow\n\n(4) The paper provide 2 baseline VQA model (T5 and BLIP2) as the answer generator and the results looks very promising\n\n(5) The paper also report the additional inference time cost in the appendix indicating that the inference time over head is not un acceptable", "weaknesses": "Overall,  I think it is a solid paper, here is some minor comments\n\n(1) While the inference time cost has been mentioned, it is also good to mention storage cost as you need to storage the features for each tokens. Hope to know if it requires very good SSD or RAM though it is cheaper than the GPUs \n\n(2) in the bottom block in table 1, I hope to understand the performance difference when the number of knowledge sentences grows. It is good to plot the performance curve with x axis denoting the number of sentences and the y shows the vqa score. This can help us understand (a) performance-knowledge saturation speed compared to DPR, showing we need to use the FLMR if we have enough bucket to hold more knowledge as inputs.\n\n\n(3) [1] also uses fine-grained visual information as multiple queries to retrieve knowledge and it is good to compare to that as well \n\n\n\n\n\n\n\n------------------------------------\npost rebuttal: I read the author's rebuttal and still willing to support the paper as my initial rating\n\n\n\nReferences:\n[1] Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your very positive review. Thank you for your careful reading as well. We respond to your constructive suggestions as below:\n\n1. **Hope to know if it requires very good SSD or RAM though it is cheaper than the GPUs.**\n    \n    Thank you for raising this point. Our cluster employs commonly-seen SSDs and RAMs. The index file which indexes the Google Search Corpus (~170k documents) is 23 GB. A more complete analysis of the tradeoff between speed and storage can be found in [2]. We will add as much detail as possible to the camera-ready version for readers’ interest.\n    \n2. **I hope to understand the performance difference when the number of knowledge sentences grows.**\n    \n    Performance-knowledge saturation is indeed interesting for understanding the behavior of FLMR and DPR. Following your suggestion, we conduct experiments increasing K in Table 1 and find that the system performance first improves gradually until a saturation point. The results are as follows:\n    \n\n    |  | K | 5 | 10 | 20 | 50 |\n    | --- | --- | --- | --- | --- | --- |\n    | DPR + T5-large | VQA Score | 51.5 | 51.8 | 52.3 | 52.1 |\n    |  | Recall | 83.08 | 89.77 | 94.05 | 97.25 |\n    | FLMR + T5-large | VQA Score | 54.9 | 55.3 | 55.4 | 55.4 |\n    |  | Recall | 89.32 | 94.02 | 96.87 | 98.67 |\n\n    We notice that the saturation point of FLMR is at around K=10 while that of DPR is at K=20. This suggests that the useful documents are clustered around higher ranks in FLMR compared to DPR. \n\n    We will provide such a figure in the main content or in the appendices if accepted. Thanks for your help in improving our paper.\n\n3. **EnFoRe [1] also uses fine-grained visual information as multiple queries to retrieve knowledge and it is good to compare to that as well**\n    \n    Thank you for your suggestions. We will compare our system to EnFoRe in the camera-ready version.\n    \n    Our work and EnFoRe share a common motivation: entities’ information can be crucial in retrieving knowledge relevant to the question. However, our approaches are quite different. The two main differences are: \n    - First, EnFoRe retrieves a list of entities from the image, the query, and the answer candidates. EnFoRe then explicitly learns scores to indicate the importance of each entity. In comparison, FLMR mainly utilizes an object detection model to extract object-centric ROIs and entity information. We employ late-interaction to learn the interaction between regional image features and document token embeddings implicitly. After training, our system also learns interpretable retrieval as depicted in Figure 3. \n    - Second, in terms of the retriever formulation, they employ one-dimensional embeddings following DPR. The scores from text embeddings and entity embeddings are added, similar to the baseline system DPR (VE+9ROIs) in Row 8, Table 2. In contrast, we employ token-level multi-dimensional embeddings for both image and text, which have been demonstrated to be more effective by comparing Row 8 and Row 13 of Table 2. \n    \n    We can also compare performance directly. Using Wikipedia passages, EnFoRe achieves ~34.88% Precision@5. In comparison, our system achieves ~46.41% Precision@5 when using Wikipedia passages. Please note that we use the scores from their paper directly instead of replicating their system.\n    \n\nFinally, thank you very much for your careful reading, and we are happy that you have read the appendices thoroughly as well. Thank you!\n\n[1] Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering\n\n[2] PLAID: An Efficient Engine for Late Interaction Retrieval"}, {"review_id": "RapsSpB4iG", "reviewer": "Reviewer_jgsy", "summary": "This paper studies two problems in knowledge-based VQA: 1) the image input is often transformed into the text; 2) the relevance score between query and document is estimated based on one-dimensional dot product.\nTo address these problems, the authors leverage several image features, including ROIs features from ViT and text descriptions.\nThe experiments are conducted on the OK-VQA benchmark dataset.\nAfter being equipped with the recent strong BLIP v2 model, the proposed method achieves very promising results.", "questions": "The proposed method achieves very promising results on the OK-VQA benchmark. \nHowever, several concerns refrain me from holding a positive view of this paper.\nI would like to see the authors' rebuttal to these concerns.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow for most parts.\n- The two studied problems are very important and practical for knowledge-based VQA models.\n- The proposed method achieves improved model performance after combining with BLIP v2.", "weaknesses": "- Is there also ground-truth knowledge for the questions in the test set? \n- With the first question, did the authors use ground-truth knowledge for testing?\n- In fact, one dataset is not fairly convincing and the A-OKVQA should also be selected as a benchmark dataset. The reason that the authors provided does not make much sense.\n- The ablation study in Table 1 is very confusing. For example, there are no detailed experiments for T5-Large. Combining line 26 and 27, it seems that ROI&VE poses a negative effect on model performance.\n- For Eqn.3 and Eqn.5, why use max rather than mean or sum?\n- The ```late interaction``` in this paper means?\n- Line 141 confuses me a lot. It seems that the authors first map each ROI image to one feature, and then transform it to $N_{vt}$ tokens. Why don't we just use the $N_{vt}$ token features from ViT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for raising these concerns.\n\n- **Is there also ground-truth knowledge for the questions in the test set? Did the authors use ground-truth knowledge for testing?**\n    - No and no. There is no ground-truth knowledge annotation of the document collection for the question in the test set. To assess retrieval performance, we adhere to established norms in the literature, using pseudo-relevance (the presence of the answer within the retrieved document) as our metric of evaluation [1]. We also assess the ultimate VQA performance, which acts as a measure of the effectiveness of the retrieved documents in aiding the QA process.\n- **In fact, one dataset is not fairly convincing and the A-OKVQA should also be selected as a benchmark dataset.**\n    - Thank you for raising this concern. Please see the general response for reports on additional datasets which we will include in the final version.\n    - As a research focusing on knowledge retrieval in VQA, we note that A-OKVQA is not suitable for our purpose as only a small number of questions (~18%) in the dataset require outside knowledge. This was reported by the dataset authors [2], and we have confirmed it in our own studies. Therefore, a knowledge retrieval system cannot be properly evaluated on this dataset.\n- **There are no detailed experiments for T5-Large.**\n    - We validate the efficiency of FLMR paired with BLIPv2, currently recognized as the SOTA, allowing for fair comparisons with the most recent literature. We report the result using T5-large since it was used in the original RA-VQA. In our experiments, we found that the trends seen with BLIPv2 also hold for T5-large. We were constrained by space and, consequently, these results were not incorporated. We share the results here:\n    \n        | Model | R@5 | EM | VQA |\n        | --- | --- | --- | --- |\n        | RA-VQA-v2 (T5-large) | 89.32 | 58.85 | 54.85 |\n        | w/o ROI | 87.02 | 57.80 | 53.75 |\n        | w/o ROI & V | 85.99 | 57.65 | 53.78 |\n        | w/o Late Interaction | 82.90 | 55.30 | 50.95 |\n        | w/o ROI & Late-interaction | 83.43 | 56.20 | 52.03 |\n        | w/o ROI & VE & Late-interaction | 83.08 | 55.89 | 51.45 |\n    \n- **it seems that ROI&VE poses a negative effect on model performance.**\n    - Thanks for your careful reading. When using the DPR approach, introducing VE alone leads to a slight improvement. However, the performance declines with the further introduction of ROIs. Row 5 and Row 7 in Table 2 show that adding VE improves retrieval performance. However, the performance is reduced when ROI features are incorporated (comparing Row 7 and 8). This regression in performance can be attributed to DPR's less effective one-dimensional representation of all the ROIs compared to FLMR's late interaction. We will add one more line in Table 1 to demonstrate the point more clearly: DPR with VE and without ROI: Recall@5=83.43, EM=60.18, VQA Score=59.21. The result, compared to Table 1 line 27, further confirms that when using DPR, adding VE is beneficial but adding ROIs adversely impacts performance. This observation in fact motivates our approach, as we believe finer-grained information should benefit retrieval. In our work, we show that FLMR’s late-interaction mechanism allows for better use of the ROI features, leading to improved retrieval and VQA performance (Table 1, Row 23&24).\n- **For Eqn.3 and Eqn.5, why use max rather than mean or sum?**\n    - The MAX operation selects the highest relevance score between a query token and all doc tokens. In our preliminary experiments, MEAN/SUM exhibits instability when dealing with a wide range of document lengths, from extremely short pieces (a sentence) to very long ones. We note that [PAD] tokens in short documents may dominate in the final score, while longer documents have an inherent advantage due to their greater number of meaningful tokens. Using SUM/MEAN in practice reduces the performance by ~3% Recall@5. Therefore, we use MAX as a straightforward and optimal solution. We will add this explanation in the final version.\n- **The `late interaction` in this paper means?**\n    - In this paper, it refers to the dual encoder architecture where the queries and documents are first encoded into token-level embeddings by Transformers. These embeddings are then aggregated to compute final relevance scores. This setup not only utilizes dense encoders but also allows full interaction. You can find a very comprehensive demonstration in Figure 2 of the ColBERT paper [3]. We will make this clear in revision.\n- **Line 141 confuses me a lot. It seems that the authors first map each ROI image to one feature, and then transform it to $N_{vt}$ tokens. Why don't we just use the $N_{vt}$ token features from ViT?**\n    - Only the first [CLS] token of ViT has been trained in its pre-training process. As a result, only ViT’s first token embeddings are meaningful. We tried in preliminary experiments to use $N_{vt}$ tokens from ViT. The model takes an extremely long time (50k steps) to converge, and the final performance is much lower (-8 Recall@5) than using the first token alone. We found that extracting regional features around objects for use with ViT’s first token embeddings resulted in readily trainable regional representations. In addition, the original ViT cannot handle object-centric ROIs since it splits an input image into evenly-split regions. We also show that object-centric ROIs are more beneficial than evenly-split regions (Line 298-305).\n    \n\nWe hope that our clarifications have addressed your concerns, and we kindly invite you to consider the value and contributions of our work in your final assessment.\n\n[1] Retrieval Augmented Visual Question Answering with Outside Knowledge.\n\n[2] A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge\n\n[3] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT."}]

Author rebuttal:
We appreciate the time and efforts of all seven reviewers in assessing our paper. The positive feedback on our work is highly encouraging, and we have taken great care to respond to each reviewer individually. Here, we hope to provide direct and concise clarifications addressing the common concerns raised by more than one reviewer. In response to reviewers’ request, we provide new experimental results showing the generalizability of our technique. We also address the concerns of technical contribution and novelty. 

1. Clarification on Technical contribution/Novelty (raised by Reviewer d3hH, Reviewer XziC) 
    
    We highlight our technical novelties and summarize the reviewers’ consensus below:
    
    - **1) Late-interaction for knowledge retrieval with multi-modal query:** To our knowledge, our work is the first to extend multi-dimensional late interaction for knowledge-based visual question answering. We tackle complex multi-modal retrieval going beyond the previous focus on exploiting fine-grained token-level textual information [5] and single modal queries [6]. As noted by Reviewer d3hH, Bs2i, ujV3, XziC, shyj, jgsy,  xLzi, extensive experiments confirm the superiority of our approach in improving knowledge retriever for KB-VQA, surpassing previous baselines on the OK-VQA dataset (Line 244-254).
    - ****2) Alignment of textual and image modalities with cost-effective mapping network pre-training:**** Reviewer ujV3, XziC, and xLzi explicitly acknowledged the novelty of this approach. Our approach captures fine-grained, cross-modality relevance via token-level interactions. We perform simple yet effective alignment pre-training which costs only 4 GPU hours (Line 160-168). In addition, we highlight that the proposed system is flexible in that it does not limit the use of vision encoders and pre-trained retrievers. This allows the retrieval performance to grow with the development of general-purpose vision encoders and text retrievers.
    - **********3) Utilizing fine-grained image information with object-centric regional representation.********** Adding ROI-based image patches for finer-grained image understanding is conceptually straightforward. However, existing DPR-based retrieval systems cannot benefit from this additional information due to the limited expressiveness of one-dimensional embedding. In contrast, our experiment shows that FLMR is able to harness the regional image representation for better multi-modal retrieval thanks to the multi-dimensional embeddings and the late interaction mechanism of FLMR (Line 298-305) as noted by Reviewer d3hH, Bs2i, ujV3, shyj. This is an intuitive but non-trivial advance from the DPR baseline. The multi-modal late-interaction retrieval process can be visually interpreted (Figure 3, as noted by Reviewer xLzi).
    - **4) Public release of implementation:** We will release the codebase upon publication, which offers a complete solution for advancing multi-modal retrieval from single-vector text-only retrieval for KB-VQA. To the best of our knowledge, this will be the first publicly available implementation for multi-modal late-interaction KB-VQA retrievers.
2. Evaluation on other KB-VQA datasets (raised by Reviewer Bs2i and Reviewer jgsy)
    
    We agree that further evaluation on other KB-VQA datasets would be beneficial and have elaborated on our consideration in the main content (Line 188-190) and Appendix A: Limitations (Line 3-9). In sum: 
    
    There are limited options in the current research community. Our primary focus was on knowledge retrieval and we evaluate the retrieval performance on OK-VQA, as it is considered the most challenging and largest KB-VQA dataset based on facts and concepts. Other KB-VQA datasets that are currently available are less suitable for the following reasons:
    
    (1) A-OKVQA is more focused on “visual reasoning” instead of “information seeking” (e.g. How many people will dine at this table? Answer: One). Only a minor portion (~18%) of its questions requires outside knowledge bases (reported by dataset authors in [2]). Therefore, it is not suitable for evaluating a knowledge retrieval system since most questions are not grounded in knowledge documents.
    
    (2) F-VQA is limited in size and quality. It only has 5,826 examples, with biases towards frequent answers (e.g. “person”) and questions, as demonstrated in [1]. Therefore, we refrain from using it for the main evaluation. 
    
    Given these constraints, we evaluated FLMR on 2 additional knowledge bases: Wikipedia (Table 2) and WIT (Appendix H, Line 94-123).  

    We further performed an analysis on F-VQA [3] on reviewers’ requests (Reviewer Bs2i and jgsy). The results are attached below and will be included in the appendices of the final version. All numbers are averaged across 5 splits and the training hyperparameters are kept the same as on OKVQA. 
    
    Furthermore, we have gained early access and obtained promising preliminary results on a newly proposed dataset, Infoseek [4], from Google after the submission of this manuscript. 

    These results further support the superiority of FLMR over DPR and establish the generalizability of FLMR.
    
    |  | FVQA [3] | Infoseek [4] |
    | --- | --- | --- |
    | Retriever | Recall@5 (Std.) | Recall@5 |
    | DPR (From RA-VQA) | 68.58 (0.01) | 45.33 |
    | FLMR (From RA-VQA-v2)  (Visual Encoder) (ours) | 70.88 (0.01) | 47.78 |
    | FLMR (From RA-VQA-v2)  (Visual Encoder + 10ROIs) (ours) | 72.37 (0.01) | 50.23 |

[1] FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering

[2] A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

[3] FVQA: Fact-based Visual Question Answering

[4] Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

[5] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

[6] FILIP: Fine-grained Interactive Language-Image Pre-Training

Meta-review: The submission focuses on the visual question answering task, where the authors proposed a retrieval-augmented visual question answering (RA-VQA) framework via Fine-grained Late-interaction Multi-modal Retrieval (FLMR). Compared to dense passage retrieval (DPR), the authors propose to capture fine-grained information with multi-dimensional embeddings, as inspired by prior work such as ColBERT. The proposed approach is evaluated on the OK-VQA benchmark and achieves competitive performance.

The reviewers found the submission to be well written, and the proposed solution to be well motivated. They found the empirical performance of the proposed method to be strong on the OK-VQA dataset, especially on knowledge retrieval performance, and that the authors provided extensive evaluations and analyses of the proposed method. However, some of the reviewers had concerns on (1) limited novelty, given the similarity of the late interaction to ColBERT, and (2) evaluations only on the OK-VQA dataset. The authors provided a rebuttal to address these concerns. Notably, they provided additional evaluations on FVQA and Infoseek benchmarks.

All reviewers responded to the authors' rebuttal, except for shyj, who is already in favor of accepting (weak accept) the paper. Among the six reviewers who acknowledged the rebuttal, five found the rebuttal to have addressed their concerns, and recommended borderline accepts (4 reviewers) and accept (1 reviewer). One reviewer decided to recommend borderline reject on the basis of the lacking of evaluation on A-OKVQA. The AC agrees with this reviewer that additional evaluations on A-OKVQA, even negative ones, would make the empirical evaluations more complete and informative, but also understands the rationales provided by the authors on why the adopted benchmarks are preferable than A-OKVQA. Overall, the AC believes the concerns on evaluation on more benchmarks to be adequately addressed.

The AC thus recommends acceptance of the submission, and encourages the authors to incorporate the additional experiments during the rebuttal, and the reviewers' feedback into the final version.
Decision: Accept (poster)