SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All implementations are released at https://github.com/LinWeizheDragon/Retrieval-Augmented-Visual-Question-Answering.
F Artifacts and License
We list the resources used and their License below:
(1) huggingface-transformers (Apache License 2.0) provides pre-trained model checkpoints for BLIP 2, DPR, T5, and their tokenizers: https://github.com/huggingface/transformers
(2) PLAID engine and ColBERTv2 checkpoints (MIT License): https://github.com/stanford-futuredata/ColBERT
(3) FAISS [Johnson et al., 2019] (MIT License) is used to index document embeddings for fast retrieval with DPR: https://github.com/facebookresearch/faiss
(4) huggingface-PEFT (Apache License 2.0) for parameter-efficient LoRA fine-tuning: https://github.com/huggingface/peft
(5) The official RA-VQA implementation (GNU General Public License v3.0): https://github.com/LinWeizheDragonRetrieval-Augmented-Visual-Question-Answering.
G Computational Cost
We report the computational cost in this section.
Table 8: Training and indexing time for FLMR and DPR. Training batch size is 30. The corpus for counting the indexing time is the Google Search Corpus for OK-VQA.
train per 1000 steps	indexing time
FLMR	1.2h	0.28h
w/o ROI	1h	0.25h
w/o ROI & VE	0.7h	0.24h
DPR	0.5h	0.2h
Though Late Interaction allows rich interactions at token level and outperforms DPR by a large margin, it also introduces additional latency in retrieval. As shown by Table 8, the training time of FLMR is increased from  $0.5\mathrm{h}$  to  $0.7\mathrm{h}$  when late interaction is introduced. This latency increase comes from the more complicated token-level loss. When Vision Encoder (VE) and ROI (Region of Interest) are added, the time cost is increased to  $1\mathrm{h}$  and  $1.2\mathrm{h}$  respectively due to the additional trainable parameters of the mapping network. However, the indexing time does not increase significantly when VE and ROI are added to the FLMR retriever. We note that the FLMR spends slightly more time to build the search index when compared to DPR because an extra clustering step by PLAID [Santhanam et al., 2022b] is required to conduct fast retrieval.
Table 9: Training and inference time of the whole system. Please note that passages are dynamically retrieved, and thus the training and inference time already takes the retrieval latency into account. Batch size is set to 1 for both training and inference time. w/o ROI & VE means removing the vision encoder in FLMR.
Retriever	Generator	Training Speed (iterations/sec)	Inference Speed (iterations/sec)
FLMR	T5-large	1.16	1.11
DPR	T5-large	1.73	1.67
FLMR	BLIP 2	1.24	0.98
FLMR (w/o ROI & VE)	BLIP 2	1.43	1.00
DPR	BLIP 2	2.14	1.30
When FLMR is integrated into the full VQA pipeline (we take the BLIP 2 version for example), it reduces the training speed from 2.14 iterations/sec to 1.24 iterations/sec  $(42\%)$  since the retrieval process is run on the fly. However, in retrieval, the inference speed is only reduced from 1.3 iterations/sec to  $\sim 1.0$  iterations/sec, which is still affordable when considering the performance boost. The major computational cost remains that of training the answer generator with a great number of parameters.
H Retrieving Multi-modal Documents with FLMR
We additionally show that our proposed FLMR system can also be used to retrieve multi-modal documents. Since this is not the focus of our paper, we present the investigation in this appendix.
Dataset. We select a subset from WIT [Srinivasan et al., 2021], a knowledge corpus based on Wikipedia where the images associated with the documents are also present, to make an image-text corpus for retrieval. We adopt the same selection process as for the Wikipedia corpus introduced in Sec. 4. The dataset statistics is shown in Table 7.
Multi-Modal Late Interaction. We upgrade the document embedding process to accommodate the document image. The documents in the knowledge base are represented by embeddings  $\mathbf{D}$  which are obtained from the document content  $d$  and its associated image  $I_{d}$ :
$$
\mathbf {D} = \left[ \mathcal {F} _ {L} (d), \mathcal {F} _ {M} \left(\mathcal {F} _ {V} \left(I _ {d}\right)\right) \right] \in \mathcal {R} ^ {l _ {D} \times d _ {L}}, \tag {11}
$$
where  $l_{D} = l_{d} + N_{vt}$ , and  $l_{d}$  is the length of the document  $d$ .
We compute the relevance score between a question-image pair  $\bar{\mathbf{q}} = (q, I)$  and a document  $\bar{\mathbf{d}} = (d, I_d)$  as follows:
$$
r \left(\bar {\mathbf {q}}, \bar {\mathbf {d}}\right) = r \left((q, I), (d, I _ {d})\right) = \sum_ {i = 1} ^ {l _ {Q}} \max  _ {j = 1} ^ {l _ {D}} \mathbf {Q} _ {i} \mathbf {D} _ {j} ^ {\top} \tag {12}
$$
Discussion. Both query and document embeddings are multi-modal. Since the same image/text encoder is used to encode images  $I, I_d$  and texts  $q, d$ , respectively. Image-wise and text-wise relevance contribute to the final relevance score; After cross-modality alignment, the vision encoder  $\mathcal{F}_M(\mathcal{F}_V(\cdot))$  should produce image embeddings close to the text embeddings produced by  $F_L(\cdot)$  in the latent space if the image is relevant to the question, thereby taking the relevance between  $I, d$  and  $q, I_d$  into account during knowledge retrieval.
As shown in Table 10, the retrieval scores see a slight improvement when document images are also considered (from text-only to multi-modal). This suggests that FLMR supports retrieving multi-modal documents.
However, we note that the gain of incorporating images is marginal. This is because WIT is a strongly text-driven knowledge base as the images are already captioned by human experts. The surrounding texts of images are already dense and informative, which can be searched by FLMR easily. By manual inspection, we also notice that it is very rare that OK-VQA questions seek a document that can only be found by its accompanying images. This also explains the marginal gain we have observed.
In conclusion, we show that FLMR can also be applied to retrieve multi-modal documents, although more challenging questions and better datasets are needed to fully exploit its potential. We leave this as future work.
Table 10: FLMR performance when retrieving documents in WIT. Models suffixed by 'uni-modal' only encode document texts, while 'multi-modal' variants encode document images with vision encoders.
Model	PRRecall@5	PRRecall@10
DPR-text-only	68.24	77.13
DPR-image-only	46.29	57.70
DPR-multi-modal	68.78	77.90
FLMR-text-only	72.63	81.52
FLMR-image-only	45.75	57.92
FLMR-multi-modal	73.65	81.89
I Effects of Retrieved Knowledge
Table 11: Comparing Hit Success Rate of RA-VQA-v2 and RA-VQA.
Hit Success Rate
RA-VQA-v2 (BLIP2)	9.38
RA-VQA (BLIP2)	7.86
RA-VQA-v2 (T5-large)	17.62
RA-VQA (T5-large)	15.01
It is important to understand the task performance that a base model has attained and the gains from knowledge retrieval. We use the official evaluation metrics from RA-VQA: the Hit Success Ratio (HSR) which counts questions that cannot be answered by the base VQA model alone and thus require external knowledge to answer.
$$
H S R = \mathbb {1} \left\{\widehat {y} \in \mathcal {S} \wedge \widehat {y} _ {N K} \notin \mathcal {S} \right\}; \tag {13}
$$
where  $y_{NK}$  denotes the generated answer from a fine-tuned base model when no relevant knowledge is provided. HSR reflects the net value of incorporating external documents into answer generation. We can conclude from Table 11 that RA-VQA-v2 steadily improves the HSR of RA-VQA by  $\sim 2\%$ , showing that the gains in VQA performance come from improved knowledge retrieval. We also observe that T5-large, as an earlier language model, relies more heavily on retrieved knowledge ( $>15$  HSR). This is because the base language model of BLIP 2, Flan-T5-XL, is significantly stronger and is able to answer more questions without the aid of external knowledge. This suggests that KB-VQA performance can be improved by either (1) applying stronger base VQA answer generation models, and (2) collecting knowledge documents of higher quality.
We also conduct experiments while increasing  $K$  in Table 1 and find that the system performance improves gradually until a saturation point. We notice that the saturation point of FLMR is at around  $K = 10$  while that of DPR is at  $K = 20$ . This suggests that the useful documents are clustered around higher ranks in FLMR compared to DPR.
Table 12: Performance improvements with increasing number of retrieved documents.
K	5	10	20	50
DPR + T5-large	VQA Score	51.5	51.8	52.3	52.1
Recall	83.08	89.77	94.05	97.25
FLMR + T5-large	VQA Score	54.9	55.3	55.4	55.4
Recall	89.32	94.02	96.87	98.67
J Case Study
A case study is presented in Fig. 4. It compares the model outputs and provides explanations to each case.
Question: On what holiday do people traditionally eat this bird?
Caption: a large chicken walking through a field of grass.
RA-VQA
christmas dinner? these mouthwatering recipes ... the sweet and spicy turkey glaze gives the bird a wonderful flavor. It's been a tradition in my home for as long as I can remember.
turkey doesn't have to rule the roost at your holiday feast; this extremely french way of cooking duck is not recommended for thanksgiving.
on thanksgiving, most of us will sit down to feast on a turkey dinner. the bird also shows up on the table at christmas. how did we end up with the tradition of eating turkeys during the holidays?
RA-VQA-v2
the same traditional thanksgiving menu, with turkey, cranberry sauce, stuffing, and pumpkin pie taking up the most real estate on our plates.
after 1863, the year when president lincoln made thanksgiving day a national holiday, turkeys began to land on dinner plates across the country.
Chosen Prediction
Explanation: RA-VQA-v2 succescessfully captures "turkey" and all retrieved documents are about "turkey" and "tradition". In contrast, RA-VQA, without fine-grained information, gives high scores to noisy documents.
Question: What sense is this animal known for?
Caption: a brown and white dog sitting in front of a laptop computer.
the sense organs are the body organs by which humans are able to see, smell, hear, taste, and touch or feel.
researchers have now discovered that humans have a much better memory to recognize and remember sequential information.
over the years researchers have found a general understanding of the extent to which animals possess things such as: • language/communication • memory • cognition • emotions • intelligence and in attempting to answer the above question the jury is, well, still out.
pixabay of the five senses, smell is a dog's predominant sense.
hound dog. Known for their powerful sense of smell and great speed, hounds were historically used by hunters to track and chase prey.
Explanation: RA-VQA fails to capture all three key concepts "dog", "animal" and "sense" in the query, which results in retrieving noisy, less relevant documents.
Question: What animal is the woman's shirt replicating the pelt? Caption: a group of people sitting at a table in a restaurant.
RA-VQA-v2 w/o Region of Interest
nature normally favours the males of the animal kingdom in the looks department, as a tool for attracting a mate.
once the bird had been plucked, it was obvious that the right half of the skeleton was much bigger than the left.
if you're ready to give industries that abuse animals the bird, here are some items to avoid and suggestions about what to purchase instead: boas and feather dustersan eyewitness investigation of the largest ostrich slaughter companies in the world showed that ...
later, animal prints mimicking the leopard, zebra, and cheetah would soon become items of fashion in the western world. printfy's t-shirt dresses are tagless, custom cut and sewn to match every style.
ones which replicate the fur or skin of animals like leopards, tigers, zebras, giraffes, hyenas, monkeys and much more. after the animal it imitates, for instance leopard print skirts, zebra print top and so on. animal print is a clothing and fashion style in which the garment is made to resemble the pattern of the skin and fur of an animal such as a leopard, cheetah, snow leopard, jaguar, zebra, tiger, clouded leopard, margay, ocelot, spotted hyena, striped hyena, african wild dog, constrictor snake, giraffe or monkey.
ostrich
Explanation: without fine-grained region-of-interest-based features, FLMR fails to locate the "woman" and "shirt" in the image. When ROI features are added, FLMR successfully matches "leopard" from the image representations.
Figure 4: Case study comparing some model variants. Explanations are given to each case. Please zoom in for the best visualization.
Retrieved Knowledge
Correct Final Prediction
Wrong Final Prediction