Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Authors: Zhuolin Yang
Wei Ping
Zihan Liu
Vijay Anand Korthikanti
Weili Nie
De-An Huang
Linxi Fan
Zhiding Yu
Shiyi Lan
Bo Li
Mohammad Shoeybi
Ming-Yu Liu
Yuke Zhu
Bryan Catanzaro
Chaowei Xiao
Anima Anandkumar
Venue: EMNLP 2023 Findings (2023)
Keywords: Retrieval
Visual Language Model
Image-to-Text

Abstract: Augmenting pretrained language models (LMs) with a vision encoder~(e.g., Flamingo) has obtained state-of-the-art results in image-to-text generation. However, these models store all the knowledge within their parameters, thus often requiring enormous model parameters to model the abundant visual concepts and very rich text descriptions. Additionally, they are inefficient in incorporating new data, requiring a computational-expensive fine-tuning process. In this work, we introduce a Retrieval-augmented Visual Language Model, Re-ViLM, built upon the Flamingo, that supports retrieving the relevant knowledge from the external database for zero and in-context few-shot image-to-text generations. By storing certain knowledge explicitly in the external database, our approach reduces the number of model parameters and can easily accommodate new data during evaluation by simply updating the database. We also construct an interleaved image and text data that facilitates in-context few-shot learning capabilities.We demonstrate that Re-ViLM significantly boosts performance for image-to-text generation tasks, especially for zero-shot and few-shot generation in out-of-domain settings with 4x less parameters compared with baseline methods.

Image-to-text generation, also known image cap- tioning (e.g., Karpathy and Fei-Fei, 2015), plays vital role in understanding visual information and enhancing human-Al interaction. This task has a wide range of practical applications such as gam- ing, virtual reality, and robotics (Luo et al., 2019; Zhao et al., 2021; Liu et al., 2021). To address this Re- V: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Zhuolin Yang*$1
Wei Ping*
Zihan Liu?
Vijay Korthikanti
Weili Nie
De-An Huang?
Linxi Fan
Zhiding Yu?
Shiyi Lan
Bo Li
Mohammad Shoeybi?
Ming-Y Mung-Yulln Liu Yuke Zhu Bryan Catanzaro
Chaowei Xiao Anima 2nandkumar
problem, numerous methods have been proposed recently and obtained great success (e.g., Alayrac et al., 2022; Wang et al., 2022c: Hu et al., 2022: Chen al., 2022b,a; Wang al., 2022b).
Abstract
Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-ane results in image- to-text generation. However, these models store all the knowledge within their parame- ters, thus often requiring enormous model pa- rameters to model the abundant visual concepts and very rich textual descriptions. Addition- ally, they inefficient in incorporating new data, requiring a computational exppensive fine- tuning process. In this work, we introduce Retrieval-augmennte Visual Language Model, Re- M., built upon the Flamingo, that sup- ports retrieving the relevant knowledge from the external database for zero and in-context few-shot image-to-ter to-text generations. By stor- ing certain knowledge explicitly in the external database, our approach reduces the number of model parameters and can casily accommodate data during evaluation by simply updating the database. We also construct an interleaved image and text data that facilitates in-context few-shot learning capabilities. We demonstrate that Re- ViLM significantly boosts performance for image-to-text generation tasks, especially for zero-shot and few-shot generation in out-of- domain settings with 4x less parameters com- pared with baseline methods.
Among them, visual language models (LMs) build on top of pretrained autoregressive LMs (e.g., GPT-3, Brown et al., 2020), and inherit its powerful text generation ability. In particular, the pretrained LM parameters are usually frozen and only some trainable layers (e.g., adaptor) are added into the large LM during multimodal pretraining (Eichen- berg et al., 2022; Mokady et al., 2021; Tsimpouklll et al., 2021; Alayrac et al., 2022). This frozen LM strategy can avoid catastrophic forgetting when the visual LM is trained on (image, text) data, where the text quality is usually lower than the text-only corpus to pretrain LM. In addition, it enables the compelling zero-shot few-shot capability of pre- trained LM (e.g., Flamingo, Alayrac et al., 2022). However, such methods have two major lim- itations: 1) They store all acquired knowledge within the model parameters, making them param- eter inefficient in modeling the abundant visual concepts (e.g., uncommon objects) and rich textual descriptions (e.g., alternative descriptions for the same scene). 2) They are inefficient incorporat- ing new data, typically requiring computationally expensive fine-tuning (Chen et al., 2022a) or per- taining on increasingly more parameters and inter- leaved (image, texi) data (Alayrac et al., 2022).
1 Introduction
In the past few years, retrieval-augmente
LMs (Guu et al., 2020; Lewis et al., 2020; Karpukhin et al., 2020; Borgeaud et al., 2022) have shown notable success in improving accu- racy while reducing model parameters by retrieving acage-scale text corpus. Despite their success, there are several issues be addressed before we apply retrieval technique to visual LMs for image-to-text generation: i) The visual LM needs to seamlessly retrieve and encode external knowledge at the be- ginning of multi-modas pretraining. Otherwise, the
"Equal contribution. @Work done during an internship at NVIDIA. 'UIUC. "NVIDIA. 'UT Austin. "University of Wisconsin-Madison
"Califoriia Institute of Tochnol- ogy. FFqua advising. Correspondence to: Zhuolin Yang <zhuolin5 @ iiliiioos..coe odu>> Wei Ping seain@@niidiaccmm

---

Query Samples
Generated Captions
Retrieved Evidence
Hamingo:
brown and blak animal in poul of wathrr.
Re-Vit Ve-VLM:
the beach.
Sea other (bnbyydng luttla) Roating water holding, Ranfiil.
A otter and sutteming together their books.
Flamingu:
picture couple difierent types
Re-VILM:
picture of two crayfish fish with difffrent colors.
Ariteally undingene obey craytich dbeyensi) near Monterex
The small spiny cnyyfish Buartacus dingodi Colls Harhour region.
Flamingo: cake sitling on of while plate.
VILM:
strawberry sitting plate.
The top view the lapered straedeerry stranbburtio top whippped topping porteen.
Home buisery cake top.
Figure 1: Examples of input images and output captions from 2.1B Flamingo (re-implemented) and 2.4B Re- ViLM. Re-ViLM can utilize the retrieved captions generate more informative and accurate captions.
powerful pretrained autoregressive LM tends to ig- nore the poorly encoded external knowledge. ii) In multi-modal datasets, there are cases where multi- ple captions describing the same image (e.g., from different annotators (Lin al., 2014)) 4)), and multiple images having the same caption (e.g., see images Figure in Appendix). Thus, simply performing standard nearest neighbor retricval tends to make the model take shortcut and copy-paste retrieval examples during training. iii) Training the model on large-scale interleaved (image, text) dataset fa- cilitates few-shot learning capability (e.g.. M3W in Alayrac et al. (2022)), but it is really expensive to collect such dataset.
at the beginning of multimodal pretraining and result in improved performance.
2. We investigate the retrieval strategy to build the multimodal retriever. At multimodal pretrain- ing, we find the best performance is obtained by retrieving k-nearest neighbor captions based on cosine similarity between image CLIP (Radford et al., 2021) embeddings, while circumventing Soopy-and-paste behavior in training by filtering out retrieved candidates with the same caption the training instance.
3. We construct both pretraining and evaluation datasets consisting of interleaved (image, text) pairs for multimodal pretraining, using exist- ing public datasets. This facilitates in-context learning where few-shot examples are given interleaved (image, tex)) pairs.
In this work, we propose -ugrenel- Visual Language Model, Re-V,LM, which en- hances the state-of-the-art visual LM, Flamingo for zero-shot and in-context few-shot image cap- tioning, by seamlessly incorporating a multimodal retriever and retrieval- LM layers that cross-attend to a encoder (see selected sam- ples in Figure/ 1, model framework in Figure 2). Specifically,, we make the following contributions:
4. We conduct extensive experiments for image- to-text generation under zero-shot, few-shot, and fine-tuning settings on various bench- marks including MSCOCO (Lin et al., 2014), Flickr30k (Plummer et al., 2015), and No- Caps (Agrawal et al., 2019). Re- ViLM consis- tently outperforms the baseline Flamingo model across all settings. The improvements are partic- ularly notable in zero-shot and few-shot settings, e.g., our Re-ViLM can outperform the Flamingo model containing even 4x more parameters zero-shot evaluation.
1. In contrast to previous work, we initialize Re-ViLM with RETRO, pretrained retrieval- augmented LM (Borgeaud et al., 2022), thus it can seamlessly integrate the retrieval capability
'Note that, no official implementation of Flamingo is avail- able. which is trained on large-scale in-house dataset (Alayrac et al., 2022). We re-implement the model on public availahle datasct.
We organize the rest of the paper as follows. In $ 2,

---

we discuss related work. We introduce Re-ViLM model in $ and multimodal dataset for pretraining and retrieval in $ 4. We present our experimental results in $ 5 and conclude the paper in
3 Re-ViLM Architecture
In this section, we begin by outlining the frame- work of Re-ViLM. After that, we delve into the details of each component in depth.
2 Related Work
3.1 Framework
We illustrate our Re- RiLM framework for image captioning in Figure 2. It consists of three essential components:
Visual Language Models Many recent work tackle the problem of generating text captions for given images (e.g., Wang al., 2022c; Alayrac
Image encoder begins with pretrained vision transformer from CLIP (Radford et al., 2021) to extract visual features from the input images. These features are subsequently fed into train- able perceiver resampler (Jaegle et al., 2021) unify the image features into the textual repre- sentation used in the retrieval-augmented LM.
al., 2022: Aghajanyan et al., 2022; Wang al.. 2022b; Hu et al., 2022; Li al., 2022; Chen et al., 2022b; Yu et al., 2022). Among these work, vi- sual language models (Tsimpoukelli et al., 2021; Alayrac et al., 2022) directly augment pretrained LMs with visual component, achieving strong re- sults in both zero-shot and few-shot generation.
augmmetel Models Retrieval has been successfully applied in various NLP tasks, in- cluding question answering (Guu et al., 2020; Karpukhin et al.. 2020), autoregressive language modeling (Borgeaud et al., 2022), and other knowledge-intensive tasks (e.g., Lewis et al., 2020). In computer vision, retrieval has also been applied for image recognition with long-tail distribution of classes (Long et al., 2022). In this work, we apply retrieval for image captioning.
Retrieval-augmennted LM is initialized with pretrained RETRO model (Borgeaud et al., 2022) to generate corresponding captions based on the image features and retrieved evidence. Among some LM layers, the module cross- attend the hidden representations from the im- age encoder or the shared text encoder. The bidirectional text encoder encodes the retrieved captions obtained from the multimodal retriever.
Multimodal retriever consists of a retrieval database storing (image, texi) pairs indexed by Faiss, a fast similarity search library (John- et al., 2019) using their CLIP embeddings. Given a query image, the retriever extracts its embedding using the CLIP-ViT module within the image encoder. then returns the top-k (image, text) pairs, measured by the cosine sim- ilarity of embeddings between the query im- age and the retrieved images. After that, the retrieved captions are encoded by the LM to generate relevant captions.
RA-CM3 (Yasunaga et al., 2022) augments the CM3 backbone (Aghajanyan et al., 2022) with re- trieval, which can perform both image-to-tert gen- eration and text-to-image synthesis. In contrast to our Re ViLM, there are the following differences: 1) We investigate the soopy-and-paster behavior of retrieval-augee -uggeeeted model during training, and pro- pose a simple filtering strategy during retrieval. In strat-
contrast, RA-CM3 proposes a query- egy that drops some tokens of the query caption used in retrieval. In our ablation study, we find our simple strategy works better than the dropout reg- ularization as shown in $ 5.5.1. 2) Our Re-ViLM uses -ugmeneel LM decoder layer with cross-attention module to attend to the retrieved similar captions. In contrast, RA-CM3 appends the retrieved captions as the prefix context on the decoder side. In our ablation study, we find ours are more effective, as shown in $ 5.5.2. 3) For image-to-text generation, RA-CM3 only provides the result of 2-shot in-context image captioning on MSCOCO (Lin et al., 2014). In contrast, we perform extensive evaluations of our Re- ViL.M on various benchmarks under zero-shot, few-shot and fine-tuning settings.
In the following subsections, we provide more de- tails about each component of our model.
3.2 Image Encoder
To fully leverage the existing pretrained model, we initialize the image encoder with CLIP-ViT (Rad- ford et al., 2021), a pretrained vision transformer that processes images by dividing them into a grid of patches and then processing each patch with transformer encoder. In our experiments, we used two different sizes of the CLIP-ViT model: ViT-B/32 and VÝT-L/114. We freeze the CLIP-VVi part during multimodal pretraining to avoid catas- trophic forgetting while making it trainable during

---

Image Encoder
Key hreeee
IIJ
CLIP VIT
capoone vecuss plane
Netrene wpp.atti telurnatte taang
Peccelver Ressmpler
database
Imsge
Multimodal Retriever
[S]]
Prompt: A pucture of
EWB
FFW
ATTN
CA
FFW
Re-ViLMi man taking phone call desktop.
XTTN-DENSE
LM Decader Layer
Retreeval-Augmented LM
The framework of R. The model first extracts CLIP embedding of the input image, and use it to retrieve similar image-text pairs from the database. Within some predetermined layers, the retrieval-augee perieval-augmented LM will cross-attend the visual representation from the image encoder, and the textual representation from text encoder, which encodes the retrieved captions.
Figure 2:
fine-tuning for better results. We then use a per- ceiver resampler to obtain fixed-length hidden rep- resentations from the CLIP-ViT ViT token embeddings. The perceiver is trainable during both the multi- modal pretraining and future fine-tuning, allowing it to adapt the visual representations for the text decoder and connects the two modalities.
in potentially degraded test performance due to the discrepancy between the training and evalua- tion setting. To avoid it, we filter out the retrieved image-text pairs if the retrieved image identical to the query image I (e.g.. in = I) during both training and inference.
Furthermore, in the image-text datasets (e.g.. Conceptual Captions (Sharma al., 2018; Chang- pinyo et al., 2021)), multiple images can have iden- tical captions. For example, one annotator may provide the same caption or alt-text for similar im- ages (see Figure 3 in Appendix for examples). If the retrieved caption from database is the same the ground-truth caption during training, it will en- courage the model take short path and simply copy-and-paste the retrieved caption the model output, hindering the training of Re-ViLM. To ad- dress this issue, we employ a filtering strategy that filters out the retrieved image-text pair its text is identical to the training image's s caption that is used as the teacher-forced input at the LM decoder layer. Thus, if the corresponding ground-truth cap- tion of I is C, the filtered retrieval results R(I) { e1 ), ... (ix, oi) Gj A C, # I,Vj [1, k}}. In 5.5.1, we show that Re- can be largely improved with this simple filtering strategy.
3.3 Multimodal Retriever
Building Database. The retrieval database is built upon a image-text paired dataset and is struc- tured as key-value map, where the keys are CLIP ViT-B/32 image embeddings, and the values descriptions. The database is indexed by Faiss library (Johnson et al.. 2019). Retrieval Given an input query image I, we perform k-nearest neighbor retrieval with cosine similarity of embeddings between query image and database images. The retrieval results are denoted R(1)- = Oj with E k] represents the retrieved image and caption, respectively.
the corresponding
(ix, cx)}. where i, and
Filtering strategy There can be multiple cap- tions for the same image from different annota- tors in some datasets (e.g., MSCOCO (Lin et al., 2014)). In this case, these retrieved captions, which are highly correlated (even near-duplicate) with the ground-truth caption, could give a false sense of retrieved evidence quality to the model, resulting
3.4 Retrieval-augmentee LM
To facilitate the model using the retrieved captions, the visual LM needs to seamlessly retrieve and en- code the external knowledge at the beginning
"We also tried to use embeddings of query image and database captions with the same CI model, and the empiri- cal results are very similar.

---

multimodal pretraining. Thus, we initialize our text encoder and LM decoder layer with pretrained RETRO (Borgeaud et al., 2022), a state-of-the- art ettrieal-auggment we use three different sizes of RETRO models: RETRObase with 148M parameters, RETRO medium with 405M, and RETRO with 1.5B parameters. To generate captions conditioned on visual input, we interleave the LM decoder layers with gated cross-attention dense layers (gated xtin-dense) as in Flamingo (Alayrac al., 2022), which take the out- put of perceiver resampler as the key and value for cross-attention. To incorporate the retrieved cap- tions as evidence, we interleave the LM decoder layers with surieval-augmentaaii take text encoder output as key and value for cross- attention. Note that, the text encoder is shared across satriivvallagmmentatiie -erieeal-augmeenee LM at multimodal pretraining, and make it trainable at fine-tuning.
4 Multimodal Data for Pretraining and Retrieval
4.1 Image-Text Pair Data
LM. In our experiments,
In this work, we pretrain our models using two multi-modal datasets: 1) CC3M + CC12M SBU, which consists of overall 15 million high- quality image-text pairs from the Conceptual Dataset (Sharma et al.. 2018; Changpinyo et al., 2021) and SBU Captions (Ordonez et al., 2011) 2) COYO-700M (Byeon et al., 2022), which con- tains 747 million image-text pairs after filtering out low-quality samples from collection of 10 billion web image-text sources. In our experiment, we find that the high-quality captions are essential for pre- training in image captioning task. Thus, we further filter out instances with irregular textual tokens or low CLIP similarity scores between image and text, and obtain 104M high-quality image-text pairs. For simplicity, we refer to the CC3M + CC12M + SBU dataset CCS and the COYO-104M dataset COYO. At retrieval, we use CCS and COYO as the main sources for our retrieval database, and utilize the Faiss library (Johnson et al., 2019) to support fast similarity- retrieval. It takes 24 2B for storing faiss index file and for each query, it takes around 50ms performing retrieval on our databasc.
layers, which
layers. We freeze the
Text encoder Given the retrieved k-nearest neigh- bor captions, use transformer-based bidirec- tional encoder model, to obtain the hidden repre- sentations. Specifically, our text encoder shares the subword embedding table with the LM de- coder. We concatenate k embeddings along with the length dimension to form the retrieval raw em- bedding tensor & C (kxm)xxd where m is se- quence length, d is hidden dimension. After ap- plying transformer layers, the encoder output is cross-attended by the LM decoder layers. We ini- tialize the text encoder with a pretrained RETRO model (Borgeaud et al., 2022) instead of training from scratch to improve model performance.
4.2 Interleaved Image- Data
In this subsection, we discuss our proposed pre- training method for enhancing the in-context few- shot ability of our model. In this scenario, the model needs to be highly conditioned on the pre- vious few-shot samples (image-text pairs) to effec- tively generate captions of test images. However, existing multimodal models generally do not use multiple image-text pairs as inputs for pretrain- ing (Tsimpoukelli et al., 2021; Yasunaga et al.. 2022). This makes the in-context few-shot learning at inference time challenging, as there is no such supervision during pretraining. While Alayrac al. (2022) built an io-house large-scale multimodal cor- pora with interlcaved images and text, collecting such a dataset is expensive.
3.5 Trainable Modules at Pretraining and Finetuning
Similar to Flamingo, our training objective to maximize the conditional likelihood of the cap- tions given the images. At pretraining, we fol- low Flamingo's strategy by freezing the pretrained components, including CLTP-VIT and retrieval- augmented LM, training only the perceiver resam- pler from scratch. At finetuning, we unfreeze all the pretrained components and increase image res- olution from 224 x 224 to 480 x 480, as suggested in (Alayrac et al., 2022). This has been shown to improve overall performance.
We construct our image-text interleaved datasets using publicly available image-text pair datasets. We make the image-text pairs in each interlcaved sample relevant, in order to explicitly teach the model how condition on previous data samples for generating the caption of the current image.
Our interleaved dataset is constructed by using CCS. For each image-text pair (query) in CCS, we

---

select four relevant data pairs from the same corpus construct each interleaved sample, which results in five data pairs for each interleaved sample. The data selection process for each query consists of two steps. Step-1: We use L2 metric to measure the distances between the CLIP embeddings of the query image and the rest of images in CCS, and se- lect data pairs where the images have normalized distance score between 0.4 and 0.6 to the query image.² Step-2: To ensure the captions an in- terleaved sample are similar, we further use CLIP embeddings to calculate the distances between the query caption and the captions from the selected data pairs. We pick the top-4 data pairs where the captions are the most similar to the query caption.
original MSCOCO, which can be used to evaluate novel object captioning performance after finetun- ing on MSCOCO. For zero-shot setting, we focus on the evaluation under MSCOCO and Flickr30k datasets. For fine-tuning setting, we evaluate on MSCOCO, Flickr30k and NoCaps datasets. We conduct our few-shot experiments on MSCOCO dataset only to assess Re-ViLM's generalization and adaptability. Throughout our experiments, we report BLEU@4, CIDEr, and SPICE scores (Lin et al., 2014) to measure the quality and relevance of the generated captions given input images. Implementation. We develop ViLL with different scales based on different size of CLIP-ViT and RETRO, named Re- (ViT-B/32, RETRO-148M), Re- Re-ViL.Mmedium (oaiiiiunmiiT--T. (ViT- L/14, RETRO-410M) and Re- Ma RETRO- Compared to Flamingo model with the same CLIP-ViT and comparable GPT-3 configu- ration (Brown et al., 2020), ViLV introduces up to 16% additional parameters while largely boost- ing the performance. We build our model with Megatron-LMM infrastructure to support large visual LM training and evaluation. We set global batch size 256 and use Adam optimizer at training. We use beam search with beam size as 3, maximum generation length as 10 for inference. In our ex- periments, we set the number of retrieved captions = 2 for Re- ViLM. We also evaluate Re- ViL. performance with larger = 5, 10, which indicates insignificant improvements. Details can be found in Appendix B.
5 Experiments
In this section, we evaluate the performance of Re- ViLM under three different settings: zero-shot, few- shot and fine-tuning, on various image captioning benchmarks. We aim to demonstrate the superiority our retrieval augmentation technique in improv- ing the quality and relevance of generated captions through retrieving relevant knowledge from exter- nal databases. We compare our results to several widely-used image captioning models such as Enc- Dec (Changpinyo al., 2021), ViM (Wang et al., 2022c), and Flamingo (Alayrac al., 2022). Through extensive evaluation of Re- ViLM, we con- clude that Re-Vil ViLM compelling under zero-shot and few-shot settings.
5.1 Experimental setup
5.2 Zero-shot Evaluation
Evaluation Dataset. We conduct our image cap- tioning evaluation on three multi-modal datasets: 1) MSCOCO (Lin et al., 2014) is a dataset for image captioning, object detection, and segmen- tation. We use the Karpathy split (Karpathy and Fei-Feii 2015), with 82k/5k/5k images for training. validation, and testing respectively. Each image is annotated with at most 5 human-generated cap- tions. 2) Flickr30k (Plummer et al., 2015) a standard benchmark for sentence-based image cap- tioning, which includes 29k/1k/lk images in its Karpathy split. 3) NoCaps contains 15k images containing nearly 400 additional novel classes to
We conduct zero-shot evaluation on MSCOCO and Flickr30k datasets. During pretraining, we include both CCS and COYO as our retrieval database and report the best number among all different set- tings. Results are shown in Table 1. We find that, Re-ViLM could achieve significant boosts (around 10.0 on CIDer score) compared to the Flamingo model, by introducing up to 16% additional param- eters. Even Re- ViLMeae outperforms the largest VLM by large margin. We leave the full results containing Re-ViLM's performance under different pretraining and retrieval database combi- nation in Appendix A.
"Note that we eliminate the most similar images since there are lots of near-duplicate images, e.g., they have very few differences in terms of resizing, cropping, color, rotation, watermark. In practice, we set thresholds [0.4, 0.6) to filter out these cases, increase the diversity of the data pairs in one interleaved sample, and still make sure the selected images are relevant.
5.3 Few-shot Evaluation
We evaluate the few-shot learning capability of Re- ViLM by pretraining it on the constructed in- terleaved CCS dataset, and evaluating it under the

---

Table Zero-shot evaluation results on MSCOCO, Flickr30k benchmarks, compared with different image captioning baselines. We report BLEU @4, CIDer, SPICE scores for different methods. Note that MSCOCO, Flickr30k were excluded from pretraining set in the following MSCOCO and Flickr30k results. We replicate Flamingo models with the same image encoder and decoder as Re- ViLM based on original paper.
Total
Trainable params.
MSCOCO karpathy test
Flickr30k karpathy test
Method
params.
BLEU@4
CIDer
CIDer
SPICE
VI.T5 (Cho al., 2021)
224M
224M
4.9
2.6
2.0
Unfied VLP (Zhou al., 2020)
122M
122M
24.9
7.2
Sim V (Wang al., 2022e)
SimVLMarg
10,5
24.9
Sim VLMagge
11.4
32.2
364M
102M
12.4
39.6
42.2
7.9
Flamingo medium (re-impl)
894M
233M
44.3
8.8
(ramingoJaggere-iinp)
2.1B
489M
16.5
49.2
46.4
9.4
Re- Mpase
420M
158M
17.0
$1.2
45.2
9.2
Re- Mie.Mm
1.0B
347M
17.9
53.6
52.0
9.8
Re- MtaVie
2.4B
806M
18.6
60.8
52.1
10.0
Table 2: Few-shot evaluation results MSCOCO benchmarks, compared with vanilla Flamingo models baseline. We report BLEU@4, CIDer scores for different methods. We pretrain our Re-ViLM on constructed CCS interleaved dataset and evaluate on constructed COCO interleaved dataset respectively. We adopt CCS as our retrieval during both pretraining and evaluate stage.
Total params.
Trainable params.
2 shots
shots
8 shots
Method
BLEU@4
CTDer
BLEUG4
CTDer
BLEU@4
CIDer
Flamingo-3B (Alayrac 2022)
3.2B
1.3B
85.0
Flamingo-9B
9.3B
166B
93.1
(ramin))sse(e-iimp)))
13.7
53.9
66.0
22.1
71.8
Re- ViLMMase
420M
158M
14.8
60.1
20.8
72.2
21.8
72.6
flaningomedium(re-immp
894M
233M
17.9
69.0
23.3
80.2
23.1
76.8
Re-ViL Vo-ViiMmeediumm
1.0B
347M
18.2
73.6
24.0
84.5
24.1
81.0
Waamingghageete-iingg
2.1B
489M
18.2
71.6
25.7
89.2
26.3
89.1
Re- Ve-ViiLaage
2.4B
806M
18.4
77.2
25.5
90.5
26.2
90.2
Table Finetuning evaluation results on MSCOCO, Flickr30k, and NoCaps benchmarks, compared with different
image captioning baselines. Note that, for NoCaps, we finetune on MSCOCO karpathy train, following prior works (Li et al., 2022), while some work mentioning this setting as zero-shot evaluation. We finetune our Re-
ViLM MSCOCO/Flickr30k karpathy train split respectively for MSCOCO and Flick30k evaluation. We report BLEU@4, CIDer, SPICE scores for different methods.
Total params.
MSCOCO karpathy test Flickr30k karpathy test
NoCaps validation CIDer
Method
BLEU@4
CIDer
BLEU@4
SPICE
SPICE
Dec-Dec (Changpinyo et al., 2021)
110.9
90.2
12.1
VinVL (Zhang et al., 2021)
38.2
129,3
13.1
VL-T5 (Cho et al., 2021)
172M
34.6
116.1
4.4
5.3
MetaLM (Hao et al., 2022)
545M
37.6
126.6
58.7
8.6
Unfied VLP (Zhou
2020)
122M
36.5
116.9
30.1
17.0
BUTD (Anderson et al., 2018)
36.2
113.5
27.3
16.0
NBT (Lu al., 2018)
34.7
107.2
27.1
15.6
SimVLM (Wang
2022c)
~1.4B
40.6
143.3
110.3
14.5
BLIP (Li al., 2022)
252M
38.6
129.7
105.1
14.4
BLIP (1.i et al., 2022)
252M
40.4
136.7
113.2
14.8
Flamingo (re-ipl))
364M
37.0
128.0
30.4
16.5
102.8
14.0
mlamingomssdium (remmpl))
17.2
FlamingOn
2.1B
38.2
129.4
31.2
17.4
109.2
14.5
Re- ViLMMasee
420M
129.1
30.6
17.3
105.2
14.2
Re- ViLMaudiim
L.0B
38.2
131.2
31.0
17.5
106.8
14.4
Re- ViLMarge
2.4B
39.4
134.2
31.6
18.0
109.5
14.7

---

interleaved MSCOCO dataset, constructed by the same process described in $ 4.2, with {2, 4, 8)- shots. Results are shown in Table 2. While the sig- nificant improvements on {2, 4}-shots setting com- pared with the comparable size Flamingo model are clearly observed, we notice that the retrieval aug- mentation benefits becomes less when the number of shots increases (i.e., 8-shot). This is not sur- prising as the few-shot in-domain examples from MSCOCO has more useful information to boost the model performance on MSCOCO, than the out-of- domain samples from our retrieval database, CCS and COYO. As the number of in-domain exam- ples increases, the benefit of retrieval from out-of- domain examples becomes marginal.
simple filtering strategy that discards retrieved samples that matches the training query image or its caption C at training (i.e. R(T) (ii...),... (f,,,))
+ C, + I,Vj E HJ}. We notice that another concurrent work RA-CM3, has also proposed the query-dropout strategy which mitigates such duplication issue by randomly dropping out retrieved caption tokens based on their similarity the query image and text. We conduct ablation study to compare our simple filtering method with the query-dropar
method. The results, as shown in Table 4, indicates that our simple filtering strategy leads to consistent im- provement in the performance of Re- ViLM, while the query- query-dropout strategy achieves competitive but slightly worse results than simple filtering strategy.
5.4 Fine-tuning Evaluation
Table 4: Comparison between VeViLM with simple fil- tering strategy, query-dropout strategy, and without any filtering method during retrieval. Models are pretrained on CCS dataset, and evaluated on MSCOCO, Flickr30k under zero-shot setting. We report @@4: BLEU 44 C: CIDer, S: SPICE scores for different methods.
We conduct fine-tuning evaluation of Re-ViLM on MSCOCO, Flickr30K and NoCaps benchmarks. For evaluation on MSCOCO and Flickr30k, we fine-tune our pretrained Re-ViLM with smaller learning rate and early-stop strategy on MSCOCO and Flickr30k dataset respectively. For NoCaps evaluation, we fine-tune our model on MSCOCO dataset, following prior works (Li et al., 2022). Results are shown in Table 3. We observe that Re- ViLM still consistently outperforms Flamingo, al- though the relative improvements becomes smaller compared to the zero-shot and few-shot settings. We leave the full results containing Re-ViLM's is per- formance under different pretraining and retrieval database combination in Appendix A.
MSCOCO
Flickr30k
Method
B@4
C
C
8
Re-ViL INo filteringl
12.3
35,5
41.4
8.1
Re. Voleees druerodrrppout]
16.5
48.6
43.4
9.1
Re Vil M
17.0
51.2
45.2
9.2
Re- ViLLMmo (No filtering]
12.3
35.5
41.4
Re- MoeMandi (Query-drpout]
52.1
50.5
9.6
Re Mmedium
5.5.2 Retrieval Augmentation as In-Context Prepending
5.5
Ablation Study
Our Re-ViLM incorporate retrieved captions through the cross attention between the bidirec- tional text encoder and LM decoder layers. A con- current work, RA-CM3, proposed an alternative retrieval augmentation method by appending the retrieved captions the prefix context on decoder side as a simpler way to utilize retrieved evidence without introducing additional parameters. We in- vestigate this prompt-like augmentation method, and replicate it by appending the top 2 retrieved evidence as prefix during pretraining and inference of Flamingo model. We compare this retrieval aug- mentation method with our retrieval- augmented LM layer approach. The results shown in in Ta- ble 5. We can observe that our retrieval-augmeented design has better zero-shot captioning performance than the retrieval augmentation method in Ya- sunaga et al. (2022). reveals the importance of our retrieval- based architecture design.
5.5.1
Filtering during Retrieval
There could exist two different types of dupli- cation scenarios in multi-modal datasets: Same image with multiple captions, which is com- monly found in MSCOCO, Flickr30k and NoCaps datasets, could lead to label leakage during training. Multiple images with identical caption, which is common in multimodal datasets such Con- ceptual Captions, as shown in Figure in Ap- pendix C. Both of these duplication can lead to a severe issue that Re- ViLM can simply copy and paste the retrieved captions to achieve 100% match to the ground-truth captions. See Appendix C for more in-depth discussion.
To mitigate these above issues, we develop
"We find that the ratio of identical captions in Conceptual Captions can be as high as 15.7%.

---

Table 5: Comparison between different retrieval aug- mentation methods: ettreval-augmmented LM layers (Re- ViLM) and in-context prepending as prompt (Flamingo prepend), along with vanilla Flamingo model. Mod- els are pretrained on CCS dataset, and evaluated on MSCOCO, Flickr30k under zero-shot setting. We re- port @ BLEU@4 C: CIDer, S: SPICE scores for different methods.
the Flamingo-like model is scaling, In this study, we haven't been able to further scale Re-ViLM 80B to address the benefits from retrieval on large scale visual language models.
References
MSCOCO
Flickr30k
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022. Cm3: A causal masked multi- modal model of the internet. arXiv preprint arXiv:2201007520.
Method
B@4
C
C
Flamingorase
12.4
39.6
42.2
Flamingohase prepend
13.4
43.4
43.5
Re-Vil Vin.M
17.0
51.2
45.2
9.2
mecding
15.6
44.3
43.2
Flamingomedium + prepend
16.4
45.6
46.6
Re- Dee-ViLMmeedine
17.9
53.6
52.0
9.8
Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. No- caps: Novel object captioning scale. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 8948-8957.
6 Conclusion
In this work, we propose Re-ViLM, a retrieval- augmented image-to-text model, with strong zero- shot and few-shot image captioning results. Re- ViLM, which is built on Flamingo, provides sub- stantial reduction in the number of parameters while obtaining compelling results across different settings, as does not need to store knowledge within the parameters. We also propose a simple yet effective filtering strategy at retrieval to circum- vent the "copy-and and-paste" behavior of retrieval- augmented model. Furthermore, we construct an in- terlcaved image-text dataset for pretraining, which is crucial for in-context few-shot learning. Ex- tensive experiments on diverse captiinat datasets shows that Re-Vil ViL. consistently outper- form the baseline Flamingo model across all set- tings. Additionally, we conduct experiments on fine-tuning settings and show promising results.
Jean-Bape nann-Baptstte Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, et al. 2022. Flamingo: A visual language model for few-shot learning. In NewrIPS.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. ceedings of the IEEE conference on computer vision and pattem recognition, pages 6077-6086.
Pro-
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprial arXiv:2106.0222
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jcan- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. 2022. Improving language models by retrieving from tril- lions of tokens. In ICMI..
7 Limitations
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. NewrIPS.
In this paper, we focus on exploring emergent zero-shot and in-context few-shot image caption- ing. To achieve this, we designed our retrieval augmented model mainly based on the Flamingo framework (Alayrac et al., 2022), and leave the application of our retrieval design to other image- to-text frameworks (Bao et al., 2021; Chen et al., 2022b; Wang et al., 2022a) as future work. Further- more, since there is no official implementation of Flamingo and its training datasets, our framework is based on our reimplemented Flamingo, trained on publicly available datasets and manually crafted interleaved image-text datasets. Also from the seal- ing perspective, comparable to GPT in the text-only domain, one of the most important advantages of
Minwoo Byeon, Beomhee Park, Haecheon Kim. Sungjun Lee, Woonhyuk Back, and Sachoon Kim. 2022. Coyo-700m: Image-text pair dataset, https: //github. www.th....o.kaaaaaaeee wtt..oo//akkkaaiinna waw.aaoobrri/ooooooooooooooooooo dataset.
Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny, 2022a. Visualgpt: Data-efficient adapta- tion pretrained language models for image caption- ing. In CVPR.

---

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. 2022b. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint arXiv:2209.06794.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 740-755. Springer.
An-An Liu, Yingehen Zhai, Ning Xu, Weizhi Nie, Wen- hui Li, and Yongdong Zhang. 2021. Region-aware image captioning via interaction learning. IEEE Transactions on Circuity and Systems for Video Tech- nology.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-len lan-lanugg tasks via text genera- tion. In International Conference Machine Learn- ing, pages 1931-1942. PMLR.
Constantin Eichenberg, Sidney Black, Samuel Wein- bach, Letitia Parcalabescu, and Anette Frank. 2022. mulmmamult augmentation of gener- ative models through adapter-based finetuning. In EMNLP.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg. Alan Blair, Chunhua Shen, and Anton van den Hengel. 2022. Retrieval augmented classification for long-tail vi- sual recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1959-69699
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Realm: Retrieval augmented language model pre-training, In ICML.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recogni- rion, pages 7219-7228.
Yaru Hao, Haoyu Song. Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. 2022. Language models are peneral-purpose interfaces. arXiv preprint urXiv:220..0633
Ren Luo, Yu- Ting Hsu, Yu- Yu-Cheng Wen, and Huan-
Jun Ye. 2019. Visual image caption generation for service robotics and industrial applications. In 2019 IEEE International Conference on Industrial Cyber Physical Systems (ICPS), pages 827-832. IEEE.
Xiaowei Hu, Zhe Gan, Jianfeng Wang. Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 17989.
Ron Mokady, Amir Hertz, and Amit Bermano. 2021, Clipcap: Clip prefix for image captioning. arXiv preprint wrxhv:2111.097
Andrew Jaegle. Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. In ICML.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 201 Im2text: Describing images using million captioned photographs. Advances neural informa- tion processing systems, 24.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEFE Transactions on Big Data, 7(3):535-547.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 31 3128- 3137.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Eduunv, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In EMNLP.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. eetrieval-augmmeneed generation for knowledge-intene intensiive NLP tasks. In NeurlPS.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In ACL.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Mul- timodal few-shot learning with frozen language mod- els. Advances Neural Information Processing Sys- lems, 34:200-212.
Junnan L., Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image prc- training for unified vasion-language understanding and generation. In ICML.

---

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022a. Git: A generative image-to-text transformer for vision and language. XrXiv preprint arXiv:2205.44100 14100.
Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed. Saksham Singhal. Subhojit Som, et al. 2022b. Image as a foreign language: Beit pretraining for all vision and vision-languar tasks. arXiv preprint arXiv:2208. 10442.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022c. SimVLM: Simple visual language model pretraining with weak supervision. ICLR.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer. and Wen-tau Yih. 2022. Retrieval- augmented multimodal language modeling. arXiv preprint 2211:2211.1250
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:2205.01917.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu. Jianwei Yang. Lei Zhang. Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual representa- tions in vision-language models, In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5579-5588.
Rui Zhao, Zhenwei Shi, and Zhengxia Zou. 2021. High- resolution remote sensing image captioning based structured attention. IEEE Transactions Geo- science and Remote Sensing. 60:1-14.
Luowei Zhou, Hamid Palangi, Lei Zhang. Houdong Hu. Jason Corso, and Jianfeng Gao. 2020. Unified vision- language pre-training for image captioning and vqa. Proceedings of the AAA/ Conference on Artificial Intelligence, volume 34, pages 13041-13049.

---

Appendix
A Re-VILM with Increasing Size of Pretraining and Retrieval Data
In this section, we present detailed results of Re-Vil Re-ViLM in both zero-shot and fine-tuning settings, as shown in Table 6, Table 7, and Table 8. The performance of Ve-ViLM continually improves with the increasing size of the pretraining set and retrieval database. As the retrieval database grows, Re-ViLM has higher chance of obtaining more relevant evidence from the retriever, resulting in better performance.
B Vi-V with Different Number of Retrieved Captions
this section, we present detailed results of Re-ViLM ViLM with different number of retrieved captions both zero-shot and fine-tuning settings on MSCOCO dataset, as shown in Table 9, We can see that, while the number of retrieved examples increases, there is additional benefit in further improving our results (sometimes even worse). We hypothesis that this is due to i) the original RETRO LM (used to initialize Re- ViLM) is pretrained with k = 2, and ii) larger may return captions that are less related to the test image.
Multiple Images with The Same Caption
In this section, we delve into the issue of data duplication, where multiple images have the identical caption. As illustrated in Figure 3, there are three distinct scenarios: #1) different views of the same event, #2) identical scenes with different poses, and #3) different scenes with the same description. In scenario 1), the provided caption describes an event with a series of pictures taking the same time, but it is not well related a specific picture, e.g., the first picture in this scenario. As result, it undesired to directly copy the retrieved caption as the model output at both training and test stage. the retrieved similar image-text pair belongs to scenario 2) and 3). the caption can be useful even with direct copy-and-paste at test time. Specifically, for scenario 2), images depict the same scene (e.g. a woman holding a cell phone in an office) and should have highly correlated or even identical caption. However, if the large part of training set consists of such examples 5, the retrieval-augmeenle -aummented models are overly encouraged to copy-and-passe the retrieved caption the output, which can undermine their generalization ability at test stage, especially for out-of-domain settings.
Table 6: Full shoo evaluation results MSCOCO dataset. CCS refers to CC3M+CC12M-SBU dataset and COYO the COYO104M. We also conduct experiments on showing "in-domain" retrieval by using MSCOCO dataset as retrieval base during inference. This brings further performance improvements. We also present the baseline results by directly evaluating with the retrieved captions from CCS+COYO database. Results indicate that the retrieved caption itself cannot be directly used for captioning generation.
Pretraining
Evaluation
Method
Training Set
Retrieval Database
Retrieval Database
BLEU@4
CIDer
Retrieved Captions
CCS+COYO
0.0
3.6
Re- ViLM
CCS
CCS
CCS
17.6
49.4
Re-ViLMMaue
CCS+COYO
CCS+COYO
CCS+COYO
17.0
51.2
Re- Re-ViLMbase
CCS+COYO
CCS+COYO
MSCOCO
17.4
55.2
Re-ViL.M Mimmium
CCS
CCS
CCS
17.3
52.8
Re- Re-ViLMmedium
CCS+COYO
CCS+COYO
CCS+COYO
17.9
53.6
Re-ViLMMaace
CCS+COYO
CCS+COYO
MSCOCO
18.2
57.4
Re- MiaeMee
CCS
CCS
CCS
19.2
59.6
Re- ViL
CCS+COYO
CCS+COYO
CCS+COYO
18.6
60.8
Re- ViLMM
CCS+COYO
CCS+COYO
MSCOCO
18.8
65.4
FFor example, we find the ratio of training examples that have the same captions others can be as high as 15.1 in Conceptual Captions (Sharma et al., 2018).

---

Table 7: Full zero-shot evaluation results Flickr30k dataset. CCS refers CC3M+CC12M++SBU dataset and
COYO the COY0104M.
Pretraining
Evaluation
Method
Training Set
Retrieval Database
Retrieval Database
CIDer
SPICE
Re-ViL.Mber bose
CCS
CCS
CCS
45.0
9.2
Re- De-ViLMasse
CCS+COYO
CCS+COYO
CCS+COYO
45.2
9.2
Re-Vil .. medium
CCS
CCS
CCS
50.8
9.5
Re- ViLMMedium
CCS+COYO
CCS+COYO
CCS+COYO
52.0
9.8
Re- ViLMargee
CCS
CCS
CCS
51.5
9.7
Re- Virgee
CCS+COYO
CCS+COYO
CCS+COYO
10.0
Table 8: Full fine-tuning evaluation results on MSCOCO dataset. CCS refers to CC3M+CCI2M+SBU dataset and
COYO the COY0104M.
Pretraining
Evaluation
Method
Training Set
Retrieval Database
Retrieval Database
BLEU@4
CIDer
Re- Re-ViLM base
CCS
CCS
CCS
37.0
127.5
Re-Vit base
CCS
CCS
CCS+COYO
37.3
128.0
Re- ViLM
CCS+COYO
CCS
CCS
37.2
128.0
Re- Vil Vil.
Mbase
CCS+COYO
CCS
CCS+COYO
37.4
128.6
Re ViLMhase
CCS+COYO
CCS+COYO
CCS+COYO
37.5
128.2
Re- Re-ViLM ViLMMase
CCS+COYO
CCS+COYO
CCS+COYO+COCO
37.8
129.1
Re- Re-ViLLMaar Mm-diim
CCS
CCS
CCS+COYO
37.5
129.0
Re- VmLMMmu
CCS+COYO
CCS
CCS+COYO
37.7
129.1
Re- Ve-ViLMmedium
CCS+COYO
CCS+COYO
CCS+COYO
38.0
129.9
Re- Re-ViLMmsdion
CCS+COYO
CCS+COYO
CCS+COYO+COCO
38.2
131.2
Re- ViL.Marge
CCS
CCS
CCS
38.4
129.8
Re- ViLMargee
CCS
CCS
CCS+COYO
38.1
128.4
Re- ViLMMage
CCS+COYO
CCS+COYO
CCS+COYO+COCO
39.4
134.2
Table Re-ViLM zero-shot evaluation and finetuning evaluation on MSCOCO dataset different number retrieved samples k. While the number of retrieved examples increases, there is additional benefit in further improving our results.
Zero-shot Evaluation
Finetuning Evaluation
Method
BLEU@-
CIDer
BLEU@4
CIDer
h: 2
17.0
51.2
37.8
129.1
Re-ViLMause
1: 5
17.0
51.4
37.5
128.4
À = 10
16.5
49.8
37.5
128.2
k 2
17.9
53.6
38.2
131.2
Re-Vil Re-ViLLmmedumm
à: = 5
17.6
52.2
37.8
128.6
k = 10
17.2
50.4
37.6
128.6
2
18.6
60.8
39.4
134.2
Re- De-ViLMaarge
1: = 5
18.5
57.2
38.4
129,2
k = 10
18.2
55.8
38.2
128.8

---

Scenario #1: Different views from same event
Caption: Hanging out near the flower market bloemenmarkt.
Scenario #2: Identical scenes with different poses
Caption: Women haiding a call phone office.
Scenario #3: Different scenes with the same description
Caption: throwing rocks in the water.
Figure 3: Set of images from conceptual captions dataset with the same captions under (Top) Scenario #1): Different views from the same event. (Middle) #2): Identical scenes different poses. (Bottom) Scenario #3): Different scenes with the same description. The example of images from Conceptual Captions (Sharma et al., 2018)


Reviews:
[{"review_id": "9HAaW7LcSP", "reviewer": "Reviewer_ag8H", "summary": "", "questions": "Why does Re-ViLM base obtain BLEU@4 of 14.8 in the 2-shot setting?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your comments and valuable suggestions. They are really helpful for improving our paper.  We will address your comments in the following.\n\n> “One important baseline is needed, which is directly evaluating with the retrieved captions (with the proposed filtering). ”\n  - Many thanks for pointing it out. We conduct the suggested evaluation on MSCOCO and show the results here.  We perform retrieval on the COYO+CCS dataset by choosing the most similar captions for each testing image under ViT-B/32 embedding space after filtering. The BLEU@4 and CIDer score is 0.0 and 3.6 respectively, which means the retrieved caption itself cannot be directly used for captioning generation. We will add this result as an important baseline into our revision.\n\n> “More examples of retrieved examples with corresponding generated results are suggested to be shown.”\n  - Many thanks for your nice suggestion. We think it’s very important. We will include more examples in the final version.\n\n> “Results with different number of retrieved images are suggested”\n  - Many thanks for this suggestion. We follow your suggestion and conduct the experiment.  We evaluated on $k={5, 10}$ settings on each size of the Re-ViLM model. While the number of retrieved examples increases, there is no additional benefit in further improving our results (sometimes even worse). We hypothesis that this is due to i) the original RETRO LM (used to initialize Re-ViLM) is pretrained with  $k=2$, and ii) larger $k$ may return captions that are less related to the test image. We will add these results into the ablation study section in our revision.\n\n|        MSCOCO Zero-shot                | BLEU@4 | CIDer |\n|------------------------|:-------------:|:------------:|\n| Re-ViLM (base, k=2)    |      17.0     |     51.2     |\n| Re-ViLM (base, k=5)    |      17.0     |     51.4     |\n| Re-ViLM (base, k=10)   |      16.5     |     49.8     |\n| Re-ViLM (medium, k=2)  |      17.9     |     53.6     |\n| Re-ViLM (medium, k=5)  |      17.6     |     52.2     |\n| Re-ViLM (medium, k=10) |      17.2     |     50.4     |\n| Re-ViLM (large, k=2)   |      18.6     |     60.8     |\n| Re-ViLM (large, k=5)   |      18.5     |     57.2     |\n| Re-ViLM (large, k=10)  |      18.2     |     55.8     |\n\n|     MSCOCO Finetuning                  | BLEU@4 | CIDer |\n|------------------------|:-------------:|:------------:|\n| Re-ViLM (base, k=2)    |      37.8     |     129.1    |\n| Re-ViLM (base, k=5)    |      37.5     |     128.4    |\n| Re-ViLM (base, k=10)   |      37.5     |     128.2    |\n| Re-ViLM (medium, k=2)  |      38.2     |     131.2    |\n| Re-ViLM (medium, k=5)  |      37.8     |     128.6    |\n| Re-ViLM (medium, k=10) |      37.6     |     128.6    |\n| Re-ViLM (large, k=2)   |      39.4     |     134.2    |\n| Re-ViLM (large, k=5)   |      38.4     |     129.2    |\n| Re-ViLM (large, k=10)  |      38.2     |     128.8    |\n\n\n> “Inference time and memory comparisons are not shown, they are suggested because extra time and memory are needed to perform the retrieval.”\n  - This is a nice suggestion. At inference, we utilize the Faiss index for fast similarity search, which takes ~50 ms per query for a database with 700M text-image pairs. The Faiss index is 241 GB. Note that the retrieval is performed on the CPU and the whole index is only loaded into CPU memory.\n\n> “Why does Re-ViLM base obtain BLEU@4 of 14.8 in the 2-shot setting?”\n  - Thanks for raising the question. Re-ViLM obtains BLEU@4 of 17.0 and CIDer of 51.2 in zero-shot setting, while worse BLEU@4 of 14.8 and better CIDer of 60.1 in 2-shot setting. We do notice this discrepancy between BLEU@4 and CIDer sometimes, especially when both of them are relatively low. This is not surprising, as BLEU@4 focuses on 4-gram, while CIDer averages {1, 2, 3, 4}-gram similarities. We will elaborate this in our revision.\n\nWe really appreciate your valuable suggestions. They are very helpful for improving our paper. We follow your suggestions and hope our response addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new method is proposed, by building the model based on pre-trained models and utilizing retrieval-augmented knowledge, the proposed method leads to better results across different settings including zero-shot, few-shot and fully fine-tuning. Some ablation studies are also conducted.", "reasons_to_reject": "One important baseline is needed, which is directly evaluating with the retrieved captions (with the proposed filtering). Meanwhile, more examples of retrieved examples with corresponding generated results are suggested to be shown.\n\n\nResults with different number of retrieved images are suggested, which can better illustrate the effectiveness of the proposed architecture and retrieval-augmentation respectively.\n\nInference time and memory comparisons are not shown, they are suggested because extra time and memory are needed to perform the retrieval.", "questions_to_authors": "Why does Re-ViLM base obtain BLEU@4 of 14.8 in the 2-shot setting?", "paper_topic_and_main_contributions": "The model proposes a new method which utilizes retrieved knowledge for image captioning task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A5alIcceo5", "reviewer": "Reviewer_uG9w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments. We will address your comments in the following.\n\n> “The novelty of this paper was limited. The Re-ViLM model basically followed the structure of the state-of-the-art visual LM and a few minor improvements were proposed.”\n\nRegarding your novelty concern, we address our opinion as follows: \n  - First, the way we introduced our retrieval modules is novel: we design our own retrieval algorithm with filtering to carry the retrieved info. into language model generation through cross-attention modules, which hasn’t been done in multimodal training before. \n  - Second, our data processing is carefully designed. In multi-modal datasets, there are cases where multiple captions describing the same image and multiple images having the same caption. Thus, simply performing standard nearest neighbor retrieval tends to make the model take a shortcut and copy-paste retrieval examples during training, hurting the model performance. \n  - Third, the interleaved dataset we constructed to empower the in-context few-shot learning capability of the model is novel and important to the community.  The original Flamingo paper crawls and constructs an in-house interleaved <image, text> dataset. However, it is not publicly available and really expensive to collect such a dataset. In this work, we propose a novel method to construct such an interleaved dataset from publicly available paired <image, text> dataset (see section 4.2). \n\n\n> “The most serious problem with this paper is that there is no section discussing limitations.”\n\nWe sincerely thank you for this reminder. We will include the limitations section in our paper as follows:\n  - In this paper, we focus on exploring emergent zero-shot and in-context few-shot image captioning. To achieve this, we designed our retrieval augmented model mainly based on the Flamingo framework (Alayrac et al., 2022), and leave the application of our retrieval design to other image-to-text frameworks (e.g., Bao et al., 2021; Chen et al., 2022c; Wang et al., 2022a) as future work. Furthermore, since there is no official implementation of Flamingo and its training datasets, our framework is based on our reimplemented Flamingo, trained on publicly available datasets and manually crafted interleaved image-text datasets. Also from the scaling perspective, comparable to GPT in the text-only domain, one of the most important advantages of the Flamingo-like model is scaling. In this study, we haven’t been able to further scale Re-ViLM to 80B to address the benefits from retrieval on large scale visual language models.\n\n[1] Alayrac, et al. \"Flamingo: a visual language model for few-shot learning.\" NeurIPS 2022.\n\n[2] Bao, et al. \"Beit: Bert pre-training of image transformers.\" arXiv:2106.08254, 2021.\n\n[3] Chen, et al. \"Pali: A jointly-scaled multilingual language-image model.\" arXiv:2209.06794, 2022c.\n\n[4] Wang, et al. \"Git: A generative image-to\u0002text transformer for vision and language.\" arXiv:2205.14100, 2022a.\n\n\nWe hope our response addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The advantages of this article are as follows:\n1. This paper uses existing public datasets to build a pre-training and evaluation dataset consisting of interleaved image-text pairs for multi-modal pre-training, facilitating contextual learning.\n2. The Re-ViLM model outperforms baselines on various downstream tasks, and results from ablation studies also support the effectiveness of the proposed improvements.", "reasons_to_reject": "The weakness of this paper is listed as follows:\n1.  The novelty of this paper was limited. The Re-ViLM model basically followed the structure of the state-of-the-art visual LM and a few minor improvements were proposed. \n2. The most serious problem with this paper is that there is no section discussing limitations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors introduce a Retrieval-augmented Visual Language Model built upon the Flamingo called \"Re-ViLM\". Re-ViLM was designed to solve enormous model parameters requirement based on Flamingo. Improvements were made on Re-ViLM:1) storing specific knowledge explicitly in the external database, 2) constructing an interleaved image and text data. Re-ViLM significantly boosts performance for image-to-text generation tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QR2VSH4Kue", "reviewer": "Reviewer_FVoX", "summary": "", "questions": "- A. The benefit of incorporating retrieval into captioning is not well demonstrated\n    - A1. For 0-shot evaluation, the previous method can achieve ~50 CIDEr on MS-COCO by only training on ~1M captions (Table 1). BLIP (ViT-B/16) can achieve 94.8 CIDEr on Flickr30K by training on 14M image-text pairs (Table 1). The authors use more data and larger models, however, no better performance is achieved. So I wonder about the benefit of such a \"retrieval augment\" technique roadmap.\n    - A2. For the finetuning case, Re-ViLM is worse than previous methods, BLIP, and SimVLM.\n    - A3. It would be nice if the authors can report the cost time of the retrieval process.  How long will it take to run a kNN algorithm with 700M entries?\n    - A4. What is the performance of the method without retrieval?\n\n- B. In Figure 2, we can see the retrieved captions have irrelevant objects with the input images, for example, \"woman\" and \"laptop\" are not in the input image. In this case, how does the method guarantee that the output caption will not be affected by these noises?\n\n- C. How can authors guarantee there are not the same samples both in the retrieval database and test set? Is the 0-shot setting really 0-shot?\n\n- D. Let's assume retrieval results are important for the final captioning performance, why only use CLIP-ViT-B/32 to extract the embedings? CLIP-ViT-L is much better than CLIP-VIT-B/32, it would be interesting to see the performance of the method with a CLIP-ViT-L augmented retrieval system.\n\n- E. Reference Lines 603-614 have two identical bib entries.\n\n[1] https://arxiv.org/pdf/2303.03032.pdf\n\n[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed review. We will address your comments in the following.\n\n> \"A1, A2. For 0-shot evaluation, the previous method can achieve ~50 CIDEr on MS-COCO by only training on ~1M captions (Table 1). BLIP (ViT-B/16) can achieve 94.8 CIDEr on Flickr30K by training on 14M image-text pairs (Table 1). The authors use more data and larger models, however, no better performance is achieved. So I wonder about the benefit of such a \"retrieval augment\" technique roadmap. For the finetuning case, Re-ViLM is worse than previous methods, BLIP, and SimVLM.\"\n\n  - Many thanks for the new reference to the DeCap paper. We didn’t notice this interesting work when we submitted our paper. We will add reference and discuss it in our final version. Regarding your performance concern on Re-ViLM, we should clarify that our paper mainly focuses on proposing a simple and novel retrieval augmentation pipeline, which can be generally applied to many other visual language models. We choose Flamingo as our base model due to its compelling in-context few-shot capability (learned from interleaved text image data) and great potential for scaling (demonstrated by a 80B model in the original paper), although it is not considered as a sample-efficient and parameter-efficient model, especially for the finetuning case. Indeed, the recent OpenFlamingo / MMC4 efforts from the open source community also underscore the importance of studying such architecture and constructing interleaved dataset. \n\n> \"A3. It would be nice if the authors can report the cost time of the retrieval process. How long will it take to run a kNN algorithm with 700M entries?\"\n- Thanks for pointing this out. With the Faiss library built on a 700M entries database, given a query image, we can retrieve its kNN neighbors in 50ms on average.\n\n> \"A4. What is the performance of the method without retrieval?\"\n- Sorry for the confusion. The performance of the method, without retrieval, can be shown as follows:\n\n|             MSCOCO Zero-shot                       | BLEU@4 | CIDer |\n|-------------------------------------|:-------------:|:------------:|\n| Re-ViLM (base, with retrieval)      |      17.0     |     51.2     |\n| Re-ViLM (base, without retrieval)   |      9.8      |     36.6     |\n| Re-ViLM (medium, with retrieval)    |      17.9     |     53.6     |\n| Re-ViLM (medium, without retrieval) |      10.6     |     39.2     |\n| Re-ViLM (large, with retrieval)     |      18.6     |     60.8     |\n| Re-ViLM (large, without retrieval)  |      10.2     |     39.6     |\n\nFrom the results we can see that, Re-ViLM without retrieval could lead to worse performance compared to Re-ViLM with retrieval during inference, even worse compared with Flamingo (re-impl). This is reasonable since we fully utilized retrieved evidence during training and once we cannot obtain any retrieved info during inference, it may cause the train/test mismatch.\n\n> \"B. In Figure 2, we can see the retrieved captions have irrelevant objects with the input images, for example, \"woman\" and \"laptop\" are not in the input image. In this case, how does the method guarantee that the output caption will not be affected by these noises?\"\n\n- Thanks for your question. Re-ViLM utilizes and fuses two sources of information to help generate relevant caption: visual features from visual encoder, and retrieved captions from retriever. While the retrieved information can be informative but noisy, Re-ViLM will unitize the visual feature through the gated cross-attention to discriminate between relevant and irrelevant key word descriptions. Thus the generation will be enriched by retrieved captions (see Figure 1), while still tends to ignore the noises within them.\n\n> \"C. How can authors guarantee there are not the same samples both in the retrieval database and test set? Is the 0-shot setting really 0-shot?\"\n  - Thanks for bringing up this question. According to [COYO-700M’s official documentation](https://github.com/kakaobrain/coyo-dataset), it states that they “removed all duplicate images based on the image pHash value from external public datasets: ImageNet-1K/21K, Flickr-30K, MS-COCO, CC-3M, CC-12M”, as shown in the Data Filtering section. We have also conducted careful verification showing that there is no leakage between the retrieval database and test set. Based on this, we believe that our 0-shot setting is really 0-shot.\n\n> \"D. Let's assume retrieval results are important for the final captioning performance, why only use CLIP-ViT-B/32 to extract the embedings? CLIP-ViT-L is much better than CLIP-VIT-B/32, it would be interesting to see the performance of the method with a CLIP-ViT-L augmented retrieval system.\"\n  - Thanks for your suggestion. We conduct additional experiments on CLIP-ViT-L retrieval system and results are shown as follows:\n\n|           MSCOCO Zero-shot                    | BLEU@4 | CIDer |\n|---------------------------------|:-------------:|:------------:|\n| Re-ViLM (base, CLIP-ViT-B/32)   |      17.0     |     51.2     |\n| Re-ViLM (base, CLIP-ViT-L/14)   |      16.8     |     50.4     |\n| Re-ViLM (medium, CLIP-ViT-B/32) |      17.9     |     53.6     |\n| Re-ViLM (medium, CLIP-ViT-L/14) |      17.9     |     54.0     |\n| Re-ViLM (large, CLIP-ViT-B/32)  |      18.6     |     60.8     |\n| Re-ViLM (large, CLIP-ViT-L/14)  |      18.4     |     59.2     |\n\n\nFrom the results, we can see that there is no significant difference between CLIP-ViT-L retrieval system and CLIP-ViT-B system. Considering additional computational and disk storage cost by introducing CLIP-ViT-L, we keep our current ViT-B embedding as retrieval measurement.\n\n> \"E. Reference Lines 603-614 have two identical bib entries.\"\n  - Thanks for pointing this out. We will remove duplicated references in our revision.\n\nWe hope our response addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It is interesting to have such a trial of captioning with retrieval.", "reasons_to_reject": "- The benefit of incorporating retrieval into captioning is not well demonstrated.", "questions_to_authors": "- A. The benefit of incorporating retrieval into captioning is not well demonstrated\n    - A1. For 0-shot evaluation, the previous method can achieve ~50 CIDEr on MS-COCO by only training on ~1M captions (Table 1). BLIP (ViT-B/16) can achieve 94.8 CIDEr on Flickr30K by training on 14M image-text pairs (Table 1). The authors use more data and larger models, however, no better performance is achieved. So I wonder about the benefit of such a \"retrieval augment\" technique roadmap.\n    - A2. For the finetuning case, Re-ViLM is worse than previous methods, BLIP, and SimVLM.\n    - A3. It would be nice if the authors can report the cost time of the retrieval process.  How long will it take to run a kNN algorithm with 700M entries?\n    - A4. What is the performance of the method without retrieval?\n\n- B. In Figure 2, we can see the retrieved captions have irrelevant objects with the input images, for example, \"woman\" and \"laptop\" are not in the input image. In this case, how does the method guarantee that the output caption will not be affected by these noises?\n\n- C. How can authors guarantee there are not the same samples both in the retrieval database and test set? Is the 0-shot setting really 0-shot?\n\n- D. Let's assume retrieval results are important for the final captioning performance, why only use CLIP-ViT-B/32 to extract the embedings? CLIP-ViT-L is much better than CLIP-VIT-B/32, it would be interesting to see the performance of the method with a CLIP-ViT-L augmented retrieval system.\n\n- E. Reference Lines 603-614 have two identical bib entries.\n\n[1] https://arxiv.org/pdf/2303.03032.pdf\n\n[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", "paper_topic_and_main_contributions": "This paper aims to enhance the 0-shot performance of an image captioning system via retrieval.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed the VL models with augmentation ability, trying to use <image, text> paired data as an additional database to augment the model's knowledge. The experiments are comprehensive with multiple experimental setups. The method is new to the reviewer while there is a discussion regarding the novelty of the method. Also, the reviewer also pointed out that the effectiveness of this method is not fully demonstrated with the empirical results.
Decision: Accept-Findings