End-to-end Knowledge Retrieval with Multi-modal Queries
Year: 2023

Abstract:
We investigate knowledge retrieval with multimodal queries, i.e. queries containing information split across image and text inputs, a challenging task that differs from previous work on cross-modal retrieval. We curate a new dataset called ReMuQ 1 for benchmarking progress on this task. ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model “ReViz” that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion without being dependent on intermediate modules such as object detectors or caption generators. We introduce a new pretraining task that is effective for learning knowledge retrieval with multimodal queries and also improves performance on downstream tasks. We demonstrate superior performance in retrieval on two datasets (ReMuQ and OK-VQA) under zeroshot settings as well as further improvements when finetuned on these datasets.

heading: 1 Introduction
text: Humans recall, retrieve, and communicate information using many indirect hints and cues. For instance, if we want to explain the concept of a “leopard” but have forgotten the name, we can relate the concept to a picture of a tiger and say “it is an animal that looks like this, but has spots instead of stripes”. Similarly, when children learn to draw a new shape like an oval, teachers often prompt them by showing a circle, but saying “make the circle stretched-out”. This method of learning new concepts from visual aids and language descriptions is a common way of reinforcing existing knowledge and allowing learners to explore and retrieve new concepts (Kinder, 1942).
We propose a task for vision-language models to retrieve knowledge with multi-modal queries,
1pronounced re–µ-queue. Data and code: https:// github.com/luomancs/ReMuQ.
i.e. queries in which hints about the information to be retrieved are split across image and text inputs. Figure 1 contains an example of this task, where the image shows the Empire State Building in New York City. If we retrieve knowledge using only the image, is it likely that the retrieved information (K1) will be related to the Empire State Building. However, K1 is insufficient to answer the question. On the other hand, if we retrieve knowledge using only the question, then the information retrieved (K2) is likely to be related to the tallest building in all cities (and not restricted to New York City). K2 by itself is also insufficient to answer the question. This example shows that the combined query containing both image and text (question) is necessary for retrieving relevant knowledge (K3).
We introduce a new benchmark and dataset called ReMuQ (Retrieval with Multimodal Queries) to train and evaluate models to retrieve the answer from a corpus given multimodal (vision + language) queries. To create multimodal queries, we start with the WebQA (Chang et al., 2022) dataset as a source – WebQA contains images annotated with questions and answers. We select questions from WebQA where the answer includes both an image and text. We then remove any image information from text and combine the image and the augmented text to form a new multimodal query. We also construct a large retrieval corpus consisting of answer options of all questions as the source of knowledge for this task.
This task requires integrating the contents from both modalities and retrieve knowledge – in this paper we denote such a system as a “VL-Retriever”. Existing VL-Retrievers (Qu et al., 2021; Luo et al., 2021; Gao et al., 2022) typically follow a two-step process to retrieve knowledge: (1) converting the image into captions or keywords, appending them to the text query, and (2) using a text-retriever system to retrieve the knowledge. However, this approach can result in a loss of important information
8573
from the image, such as context and background. Additionally, using a caption generation model trained on a particular domain does not transfer well to other domains in real-world applications.
To address these issues, we propose an end-toend VL-Retriever that has the potential to leverage the entire image, rather than just object categories, keywords, and captions. We call this model ReViz, a retriever model for “Reading and Vizualizing” the query. As part of ReViz, we use a vision transformer-based model, ViLT (Kim et al., 2021), to directly encode the image from raw pixels with context inputs, and we employ BERT (Devlin et al., 2019) as the knowledge encoder to represent the long, free-form text as a knowledge embedding. ReViz differs from previous retrieval models in two main ways. First, it does not require an extra cross-modal translator (e.g., a captioning model) or object detector to represent the images. Second, its end-to-end design allows for the flexible retraining of each submodule of the model, which can mitigate potential issues caused by domain gaps.
Unlike neural text-retrievers (Karpukhin et al., 2020; Luo et al., 2022), the query and knowledge encoders in ReViz are of different types of modality (i.e. multimodal transformer and language transformer). The different semantic spaces of the query and knowledge embeddings make alignment between them difficult. To address this, we propose a novel multimodal retrieval pretraining task. To create training data, we construct triplets of (input-image, input-text, outputknowledge) from the WiT (Srinivasan et al., 2021) dataset which contains encyclopedia-type knowledge from Wikipedia. We process the data such
that the input image and text have mutually exclusive information.
Our contributions and findings are listed below.
• We introduce a new dataset ReMuQ to facilitate research on retrieval with multimodal queries. • We propose an end-to-end VL-Retriever, ReViz, that directly acquires knowledge given multimodal query. ReViz is not dependent on any cross-modal translator, such as an image captioning model or an object detector. • We pretrain ReViz on a novel multimodal retrieval pretraining task, VL-ICT. We observe that with the proposed pretraining on the WiT dataset, our VL-Retriever is a powerful zero-shot multimodal retriever that surpasses existing singlemodal knowledge retrieval methods.

heading: 2 Related Work
text: Cross-Modal Retrieval aims to find information from a different modality than the query; for instance retrieving images from text (text-to-image), text from images (image-to-text) (Young et al., 2014; Lin et al., 2014), text-to-video and video-totext (Rohrbach et al., 2015; Xu et al., 2016; Zhou et al., 2018). In contrast, we consider retrieval of knowledge for queries comprised of both modalities (i.e. image and text) together.
Knowledge-based Question Answering. Retrievers are important for finding relevant knowledge to aid knowledge-based question-answering models for tasks such as FVQA (Wang et al., 2017) (commonsense knowledge), Text-KVQA (Singh et al., 2019) which requires knowledge of the text
in the image, and KVQA (Shah et al., 2019)(world knowledge about named entities). Both FVQA and KVQA are equipped with knowledge graph as external corpus. In OKVQA (Marino et al., 2019) and its augmented versions S3VQA (Jain et al., 2021) and A-OKVQA (Schwenk et al., 2022), models are free to use any existing knowledge bases to retrieve relevant knowledge. WebQA (Chang et al., 2022) is a multi-hop reasoning dataset that requires a system to aggregate multiple sources to answer a question, where the answers can be found either via image search or general web search. Fang et al. (2020) introduce a video question answering dataset that requires a system to answer questions using commonsense knowledge about intentions and effects of people’s actions in videos.
Knowledge-Retrieval with Multimodal Queries While there are methods for retrieving knowledge from knowledge graphs (Narasimhan et al., 2018; Li et al., 2020; Marino et al., 2021), in this work, we focus on systems that retrieve knowledge from free-form text, which is more readily available and comprehensive. Previous methods involve converting images into language representations such as captions (Qu et al., 2021; Gao et al., 2022) or object tags (Gui et al., 2022; Yang et al., 2022), and then using a text-based retriever such as BM25 (Robertson and Zaragoza, 2009) or DPR (Karpukhin et al., 2020) to find relevant knowledge. Gao et al. (2022) leverage GPT-3 (Brown et al., 2020) to generate the knowledge. Qu et al. (2021); Luo et al. (2021) use a vision and language model to obtain cross-modal representations. CLIP (Radford et al., 2021) has also been applied to retrieval tasks; however it has limitations due to its separate encoding of text and
image without a multi-modal fusion module.

heading: 3 Retrieval with Multimodal Queries
text: In this section, we define the problem statement for knowledge retrieval with multimodal queries and describe the construction of the ReMuQ dataset to assess models performing this task.

heading: 3.1 Problem Statement
text: Given a query Q = (I, T ) containing as image I and text T , we wish to learn a mapping to relevant textual knowledge K from a corpus C. Note that the two modalities I and T are such that each contains partial information about K. Both I and T are necessary for successful retrieval of K and Only using one of the two modalities is inadequate.

heading: 3.2 ReMuQ Dataset Creation
text: In ReMuQ each query has exactly one ground truth knowledge associated with it. To create such queries, we augment WebQA questions (Chang et al., 2022), and collect a large corpus to serve as the knowledge source for any retrieval systems. WebQA is a multihop and multimodalQA dataset including text questions of different types such as Yes/No, Open-ended (e.g. shape, color, etc.), and multi choice (MC) questions. The images are crawled from Wikimedia Commons, both questions and text answers are created by annotators.
To create multimodal queries, we utilize the MC questions in WebQA, which are associated with multiple choices as knowledge sources in the form of text or images. The ground truth answers of the questions include text-only, image-only, or both text and image. We adapt important steps to create
multimodal queries and explain the pipeline of the curation procedure below and in Figure 2 (more examples are given in Appendix).
(1) Question Filtering. We select multiple-choice questions which have answer choices containing both image and text.
(2) Multimodal Query Construction. The initial multimodal query is the combination of the question and the corresponding image. In order to enforce a system to integrate information from both text and images, we use tf-idf to select keywords and then remove them in the question. Our new multimodal-query is then the concatenation of the augmented question and the image, with the text-answer to be the ground-truth knowledge.
(3) Retrieval Corpus Construction. We aggregate the textual knowledge from all samples as the common knowledge corpus for multimodal retrieval, resulting in a large corpus of ∼ 199k knowledge descriptions.
(4) Dataset Train-Test Split. We divide ReMuQ into 70% for training and 30% as testing split.The new curated dataset contains 8418 training samples and 3609 testing samples, together with a knowledge corpus with 195, 837 knowledge descriptions. More statistic of ReMuQ is given in Table 1.

heading: 4 Method
text: Prior work on Vision-Language (VL)-Retrievers has focused on two-stage methods where the first stage involves feature-extraction using pretrained visual and textual encoders and the second stage learns retrieval using these features. A typical VL-
Retriever can be expressed as:
K = VL-RETRIEVER(T, F ;C), (1)
where C is the knowledge corpus, T is the text component of the query, and F denotes the extracted features of image I . This feature extraction can be done in two ways; (1) by converting the visual inputs into a human-readable textual description via an image captioning model or a series of object tags by object detector, (2) by extracting object features using an object detector.
End-to-End VL-Retriever. Instead, in this work, we are interested in building an end-to-end VLRetriever, that encodes and selects the knowledge from the corpus using a VL model:
K = VL-RETRIEVER(T, I;C). (2)
We propose ReViz, an end-to-end VL-RETRIEVER that learns to maximize the multimodal query and knowledge similarity for knowledge retrieval tasks. We introduce its architecture below.

heading: 4.1 ReViz Model Architecture
text: ReViz can read and visualize the input query, consists of two components, the multimodal query encoder and the knowledge encoder. Figure 3 illustrates the pipeline of our model.
Multimodal Query Encoder. We use ViLT (Kim et al., 2021) to jointly encode the text input T and the image I . In ViLT, an image is first partitioned into a set of a fixed size of patches – these patches are encoded as continuous visual tokens through a linear projection layer (Dosovitskiy et al., 2020). These visual tokens are concatenated with the text
tokens and summed with the position embeddings and fed into a stack of several self-attention blocks. The final multimodal representation is obtained by applying linear projection and hyperbolic tangent upon the first index token embedding.
Zq = ViLT(I, T ) (3)
Knowledge Encoder. To encode knowledge, we use a pre-trained BERT (Devlin et al., 2019) model, which produces a list of dense vectors (h1, . . . , hn) for each input token, and the final representation is the vector representation of special token [CLS].
Zk = BERT(K) (4)
After the embeddings of query and knowledge are computed by the encoders, inner-dot product of the embeddings is considered as the relevancy score.
Score(I, T,K) = Z⊤k · Zq (5)

heading: 4.2 Training
text: The training objective of ReViz draws inspiration from the instance discrimination principle based on contrastive learning. The loss function to be minimized is given below:
L = −log exp(zq · zk) exp(zq · zk) +
∑ k̂∈Bk,k̂ ̸=k exp(zq · zk̂) ,
(6) where zq denotes the query embedding, zk denotes the relevant knowledge embedding, and zk̂ is the irrelevant knowledge embedding which serves as negative instances. We use all in-batch samples (Bk) as the negative instances.
Training with Hard Negatives. Adopting random samples as negative instances may cause sub-optimal metric space. Existing work shows that mining with hard negative samples leads to discriminative representations and has been applied to a broad series of tasks like face recognition (Zhang et al., 2017), object detector (Shrivastava et al., 2016), and metric learning for retrieval tasks (Faghri et al., 2018; Harwood et al., 2017). Inspired by this, we also experiment with the hard negative technique to further boost the retrieval performance. To obtain the meaningful hard negative samples, we first train ReViz with the supervisions in eq. 6. With that, for each training question, we retrieve the top-100 knowledge instances (excluding the ground-truth) as the hard negative samples. Note that we only apply hard negative mining to fine-tuning on downstream task but not the pretraining task (introduced in the next section).

heading: 5 Pretraining Task for VL Retriever
text: Previous work (Chang et al., 2020; Lee et al., 2019; Guu et al., 2020) suggests that pretraining a retriever on unsupervised task that closely resembles retrieval can greatly improve the downstream tasks performance. We propose a pretraining task called VL-ICT, which is inspired by ICT (Lee et al., 2019) task in NLP domain.
ICT aims to train text-based information retrieval (IR) system for the open-domain question answering task. To train a model without annotated data, Lee et al. (2019) propose to construct pseudo (question, context) pairs as the training data for IR system. In particular, given a passage P , a random sentence S in the passage is selected as the pseudo question, and the remaining passage
P ′ is considered as the relevant context. Such a weakly-supervised setting enables large-scale ICT pre-training, leveraging any available knowledge base as the training corpus.
VL-ICT. We propose VL-ICT task to pre-train ReViz, which can be applied to multi-modal scenarios when both language and vision inputs exist in the query. In VL-ICT, a (I, T,K) triplet is used for training. Importantly, I and T , contain mutually exclusive information and are both necessary for knowledge retrieval. However, such condition is not naturally existing, thus, we propose an automatic procedure to construct triplet satisfying this condition in the following.
VL-ICT Training Data. Figure 4 shows a snapshot of our data construction process where we use the WiT dataset (Srinivasan et al., 2021) as the source. Each WiT entry provides a title of the page or an image caption, a passage, and an image. We use the image from this WiT entry as the image I in our VL-ICT triplet. We observe that the title or caption is usually entities, it allows us to simply use word matching to find the sentences in the page passage that include the title/caption. We take such sentences as the text (T ), then we remove this sentence from the passage and use the remaining passage as the knowledge (K). To enforce that (T ) and (I) have mutually exclusive but important information, we mask keywords in T that appear in both T as well as the caption. In our experiments, we only select the English entities in WiT and execute the above process, and this results in 3.2 million (I, T,K) training triplets.

heading: 6 Experiments and Results
text: Datasets. In addition to ReMuQ, we conduct experiments on OKVQA to obtain stronger evidence for the efficacy of our method. Here, instead of QA task, we use OKVQA as a testbed for retrieval task, i.e. to retrieve a relevant knowledge to a question such that it contains the answer span. Furthermore,
we use two corpora, a small corpus collected from Google search API introduced in Luo et al. (2021), and a large corpus which contains 21M Wikipedia knowledge used in Gao et al. (2022). The statistic of each dataset is given in Table 1.
Evaluation Metrics. Following Gao et al. (2022); Luo et al. (2021), we evaluate the performances of models by Precision@K (P@K), Recall@K (R@K), and MRR@5. We use similar metrics to evaluate the ReMuQ challenge except that P@1 is used instead of P@5 since ReMuQ has exactly one correct knowledge per query.

heading: 6.1 Zero-shot Retrieval
text: We first introduce three zero-shot baselines and then present the results.
CLIP Baseline. CLIP (Radford et al., 2021) is a vision-language model pre-trained on over 400M image-text pairs. We encode all knowledge descriptions via CLIP’s textual encoder K. Then, given an image-text pair as the query, we use the image encoder to get the visual representations (I) and use the textual encoder to get the embedding of Q. We compute the inner-dot products between all encoded visual representations (I) and K to get the top-100 knowledge for evaluation, similarly for Q. Finally we sum the scores and re-rank the top-100 knowledge. We find this performs the best than using individual modality (see Appendix).
BM25 Baseline. BM25 (Robertson and Zaragoza, 2009) is a well-known efficient retrieval algorithm for text-based retrieval task based on the sparse representation. We use the caption of the image to represent the information of the image and thus we convert the multi-modal knowledge retrieval task into a pure text-based retrieval task.
DPR Baseline. We adopt DPR (Karpukhin et al., 2020) trained on NaturalQuestions (Kwiatkowski et al., 2019) dataset as a baseline, to retrieve the knowledge given an input image-text pair. First,
we use the contextual encoder of DPR to index the corpus, then we concatenate the question and the caption of the image as a joint textual query. With that, the question encoder of the DPR extracts the dense representation of the query for later computation. Lastly, we retain the most relevant knowledge pieces by calculating the inner-dot product between the query and the knowledge embedding.
Results. Table 2 shows the performances of baselines as well as ReViz pretrained on VL-ICT task. Among the baselines, we see that DPR is the strongest baselines. Surprisingly, although CLIP has shown strong performance on many classification and cross-modality pretraining task, it does not perform well on multimodal query retrieval task, this suggests that multimodal query retrieval is a challenging task for VL model. More importantly, we observe clearly that ReViz outperforms the baselines in terms of all metrics on OKVQA task on corpus of small and large size. On the ReMuQ dataset, ReViz wins CLIP and BM25 on all metrics, and DPR on two metrics. This demonstrates
the effectiveness of our proposed pretraining task and the model design.

heading: 6.2 Fine-tuning on Downstream Tasks
text: To further demonstrate the effectiveness of VLICT pretraining task, we finetune models on downstream tasks and compare performance. We compare two versions of ReViz: (1) ReViz directly trained on the downstream task and (2) ReViz first pretrained on VL-ICT and then finetuned the downstream task. In addition, We study two senarios: in-domain, where a model is trained on the training set of X domain and evaluated on the testing set of X; out-of-domain, where a model is trained on the training set of X domain and evaluated on the testing set of Y domain.
In-Domain Results. Table 3 shows the indomain performance. On both datasets, pretrained ReViz consistently outperform vanilla ReViz, suggesting that the pretraining task equips ReViz better alignment between the multimodal queries and the relevant knowledge.
Out-of-Domain Results. We investigate if the VL-ICT pretraining task can improve the generalization of ReViz. We study the performances of ReViz under two settings: train on OKVQA (domain X) and test on ReMuQ (domain Y); and the inverse. Table 5 shows that ReViz+VL-ICT+X shows obviously better results than ReViz+X on Y, especially when X is OKVQA and Y is ReMuQ. This suggests that models pre-trained with VL-ICT tasks are more robust than models without VL-ICT. We also see that the generalization performance still has a large gap with the fine-tuning, which suggests that OKVQA and ReMuQ are quite different tasks, and ReMuQ can be a good complement to OKVQA to study multimodal query retrieval task.

heading: 6.3 Comparison with Existing Methods
text: We compare ReViz with existing retrieval methods for the OKVQA task. Note that most of the models on the leaderboard of OKVQA only report the final question answering accuracy but not the retrieval performance. In our experiments we include systems which report the retrieval performance.
Baselines. Luo et al. (2021) present two finetuned multimodal retrievers: VRR-IMG which uses LXMERT (Tan and Bansal, 2019) and VRR-CAP to convert the image into captions for knowledge re-
trieval. Both retrievers use GS-112K as the knowledge corpus. TriG (Gao et al., 2022) uses zeroshot retriever and Wikipedia 21M as the knowledge corpus. Since these systems use either fine-tuned retriever or zero-shot retrievers, for fair comparison, we compare the best fine-tuned model and zeroshot model with the corresponding corpus.
Results. In the fine-tuning scenario, in majority of the cases (only one exception, R@100), our models consistently shows better performance than previous methods overall metrics. Similarly, in the zero-shot case, our model is better than previous model on all metrics by large margins.

heading: 6.4 Effects of Mask Ratio in VL-ICT Task
text: In VL-ICT, we mask the keywords in the sentence to prevent information leakage. Despite this, we find that the certain masked sentences still somehow overlap with the retrieved knowledge. We conjecture that this overlapping makes the VL-ICL task inevitably easy, and thus impairs the effects of pre-training. To study the optimal mask ratio, we conduct experiments to randomly mask the words in the sentence by different ratios. This study is performed on a smaller corpus of 1 million VLICT training triplets and models are trained for one epoch. Figure 7 shows the results. We observe that
removing 20% of the keywords yields the best performance amongst all ratios and is also better than maintaining the sentences intact (0% masking).

heading: 6.5 Effect of Generated Captions
text: Previous systems which rely on the caption generation model are affected by the quality of generated captions. This may hamper the retrieval performance when the caption generation model is not trained on the same domain as the downstream task. In our ReMuQ dataset, the images are from Wikipedia, but the caption generator is trained on
MS-COCO (Lin et al., 2014). We compare our two baselines, BM25 and DPR, using ground-truth image captions and the generated captions. Table 6 shows that using the ground truth caption is much better than the generated caption in all cases. This suggests that the caption generator is the bottleneck of the retrieval methods to convert the image information to image captioning. This demonstrates the limitations of previous methods and justifies our exploration of end-to-end training.

heading: 7 Conclusion
text: We study knowledge retrieval with multimodal (vision and language) queries, which, compared with existing retrieval tasks, is more challenging and under-explored. In addition, multimodal-query information retrieval has numerous potential applications, not only in retrieval tasks such as image, text, and video retrieval, but also in question answering, recommendation systems, and personal assistant. The proposed dataset (ReMuQ) is ideally positioned to support the development of such functionalities. We propose an end-to-end VL-retriever model, ReViz, which does not rely on any intermediate image to text translation modules. A novel weakly-supervised task (VL-ICT) is proposed to enable large-scale pre-training. Extensive evaluations on ReMuQ and OK-VQA datasets demonstrate that ReViz exhibits strong performance amongst all retrieval models in both zero-shot and fine-tuning scenarios. Our proposed dataset and model provide a foundation for future work which could potentially lead to new findings and innovative applications in multimodal-query information retrieval.
Limitations
During the creation of the ReMuQ dataset, we simply remove the words in the question that are duplicated in the image caption – in some cases, this may result in grammatical errors in the text query. We performed the experiments for studying optimal masking ratio on a subset of the pretraining data, due to resource constraints.

heading: Acknowledgments
text: This work was supported by grants from National Science Foundation #1816039 and #2132724 and DARPA W911NF2020006. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the funding agencies and employers.

heading: A Experimental Setup
text: All ReViz models consist of a ViLT query encoder and a BERT context encoder, both with 12 transformer blocks with 12 attention heads each. For pretraining, we use Adam optimizer with 100 warm-up steps, learning rate at 1e-6, a dropout probability of 0.1, and pre-train the model in 5 epochs. For down-stream task fine-tuning, we use Adam optimizer with 10 warm-up steps in 30 epochs. Learning rate 1e-6 is applied to fine-tune a pretrained ReViz on the down-stream task, and learning rate 1e-5 is used if fine-tune a vanilla ReViz. All models use 64 batch-size in the training on a machine with eight Quadro RTX 8000 GPUs.

heading: B Effect of Hard Negative Training
text: We show the effectiveness of hard negative training in Table 6. We experiment with both OkVQA and our ReMuQ dataset and the pretrained models on VL-ICT. We see that using the hard negative examples to train the model is much better than without this training step.

heading: C Additional Visualizations
text: Examples of VL-ICT Pretraining Task. Figure 8 presents more examples of VL-ICT pretraining task. More Examples of ReMuQ Task. We present some examples of ReMuQ in Figure 9, consisting of an image, an input context and the corresponding knowledge.

heading: D Examples of Retrieval Results
text: In Table 7, we present some examples of ReViz+VL-ICT+OKVQA, the best model performing on the GS-112K corpus for OKVQA dataset. In Table 8, we present some examples of ReViz+VLICT, the best model performing on the Wiki-21M corpus for OKVQA dataset.

heading: E CLIP Performance
text: As we mention in the experiment section that CLIP is one of the baselines. We compare three methods to retrieve knowledge using CLIP. First one is
only using the image, the second one is only by question, and the last one is by both image and question. In the last method, we firstly use the image embeddings and the knowledge embeddings to obtain the top-100 relevant knowledge, then we use the question embeddings to obtain the top-100
relevant knowledge. Lastly, we obtain the final top-100 knowledge by the sum of the scores given by the image and question embeddings. Table 5 shows the performance of CLIP using three methods. Using both image and question achieves the best performance.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Left blank.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: all public datasets

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. using previously published open-source data

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our proposed dataset and model provide a foundation for future work which could potentially lead to new findings and innovative applications in multimodal-query information retrieval. Limitations
During the creation of the ReMuQ dataset, we simply remove the words in the question that are duplicated in the image caption – in some cases, this may result in grammatical errors in the text query. We performed the experiments for studying optimal masking ratio on a subset of the pretraining data, due to resource constraints