MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning (2024)

Abstract: Since commonsense information has been recorded significantly less frequently than its existence, language models pre-trained by text generation have difficulty to learn sufficient commonsense knowledge. Several studies have leveraged text retrieval to augment the models’ commonsense ability. Unlike text, images capture commonsense information inherently but little effort has been paid to effectively utilize them. In this work, we propose a novel Multi-mOdal REtrieval (MORE) augmentation framework, to leverage both text and images to enhance the commonsense ability of language models. Extensive experiments on the Common-Gen task have demonstrated the efficacy of MORE based on the pre-trained models of both single and multiple modalities.

heading: 1 Introduction
text: Language Models (LMs) have gained increasing prominence in artificial intelligence, especially Large Language Models (LLMs) such as LLaMA (Touvron et al., 2023), GPT-3.5 (OpenAI, 2022), and GPT-4 (Achiam et al., 2023) that have achieved compelling performance across various tasks. However, even LLMs still lack robust commonsense capabilities and can sometimes generate sentences that violate commonsense knowledge. Figure 1 illustrates an instance of composing a sentence given several words, where both GPT-3.5 and GPT-4 consider that music can decorate the tree, which makes nonsense.
Due to the well-recognized reporting bias (Gordon and Durme, 2013), i.e., the recording of commonsense information is significantly less than its existence in reality (Grice, 1975; Havasi et al., 2007), it is inherently difficult for LMs to learn enough commonsense knowledge from modeling text generation. To enhance their commonsense
*Corresponding Author
ability, there have been a few attempts to retrieve external commonsense text information (He et al., 2022; Li et al., 2021; Liu et al., 2022) to augment the LM generation, which have been shown to be effective on commonsense reasoning tasks.
In contrast to text, commonsense knowledge is naturally recorded in the visual data. Additionally, text is used primarily for communication and may include subjective statements while images often record the physical world more objectively. Thus, images can be supplementary to text for LMs to enhance commonsense abilities. This can also be confirmed by the fact that humans acquire knowledge from both textual and visual data (Gambrell and Bales, 1986; Bloom, 2000; Joffe et al., 2007). Aware of this, instead of retrieving text snippets to assist the models in conducting commonsense tasks (He et al., 2022; Yu et al., 2022; Li et al., 2021; Liu et al., 2022), we propose a Multi-mOdal REtrieval (MORE) augmentation framework to incorporate both text and images. For LLMs pre-trained with multi-modal data (e.g., BLIP2 (Li et al., 2023)), multi-modal retrieval augmentation can also be beneficial since it explicitly provides the text snippets and images carrying related commonsense infor-
1178
mation to the current sample. To effectively incorporate the multi-modal information into LMs, there are two major challenges:
1) How can we enable LMs to effectively extract useful knowledge from multi-modal retrieved results? This is even more challenging for text-based LMs because of the modality differences. To address this challenge, we propose a plug-andplay integrator that adopts a cross-attention mechanism to weigh each of the multi-modal results based on the query input and extract beneficial information. For text-based LMs, we employ a multi-modal encoder (e.g., the Qformer of BLIP2) to ingest results of images and text. In this case, the integrator also acts as a bridge that transforms the encoded semantic space of the retrieved results into the representation space used by the LMs.
2) Since the retrieval quality could vary considerably, how can we ensure the LMs do not ignore the retrieved results and also not trust them blindly? On the one hand, to prevent LMs from disregarding the entire retrieved results due to the noise they may contain, we introduce a training mechanism in MORE, i.e., query dropout, that masks the query input to the LMs at a certain ratio to urge the LMs to leverage the retrieved results for generation. On the other hand, to avoid too much dependence on the results that could be noisy, when queries are dropped out, we randomly replace the results with irrelevant ones and guide the LMs to output empty in such cases, so that the LMs can learn that it is not necessary to use retrieval all the time.
We evaluate our approach on a generative commonsense reasoning task, i.e., CommonGen (Lin et al., 2020). This task requires models to generate reasonable sentences using given concepts. Experimental results show that MORE can significantly boost the performance on CommonGen by incorporating multi-modal retrieved results for the LMs pre-trained with data of single or multiple modalities. MORE also significantly outperforms representative retrieval augmentation baselines and LLMs like GPT-3.5 and GPT-4, demonstrating the effectiveness of its architecture and training mechanism.
We summarize our contributions as follows: (1) We propose a novel multi-modal retrieval augmented language modeling framework for enhancing text generation of LMs. (2) Evaluations on the generative commonsense reasoning task, i.e., CommonGen, demonstrate the effectiveness of MORE
on single/multi-modal LMs. (3) We conduct comprehensive analyses to verify the effectiveness of MORE under various settings and illustrate its advantages compared to LLMs like GPT-3.5 and GPT4 through case studies.

heading: 2 Related Work

heading: 2.1 Retrieval Augmented Generation
text: The effectiveness of introducing additional contexts in the generation task has been demonstrated. Specifically, utilizing the input as a query, a retriever initially retrieves a set of documents from a corpus. Then a LM integrates these retrieved documents as supplementary information to generate a final prediction. For instance, Atlas (Izacard et al., 2022) finetunes a LM jointly with the retriever with very few training examples. RETRO (Borgeaud et al., 2022) modifies the decoder-only architecture to incorporate retrieved texts and pretrains the LM from scratch. Both methods necessitate updating model parameters through gradient descent, a process not applicable to Large Language Models (LLMs).
Given that the cost of fine-tuning LMs may not always be acceptable, recent research has explored retrieval augmentation for frozen LMs. Mallen et al. (2023); Si et al. (2023); Ram et al. (2023) demonstrate that directly prepending the documents returned by a frozen retriever to the input can improve LMs performance on open-domain QA. To support a large number of documents, FiD (Izacard and Grave, 2021) processes each input passage in parallel in the encoder. RePlug (Shi et al., 2023) further finetunes the retriever based on feedback from the frozen LM to get more helpful retrieved results. On these bases, compressing the retrieved results at the sentence level (Xu et al., 2023) or token level (Liu et al., 2023; Berchansky et al., 2023) can boost performance by filtering irrelevant information retrieved and improve computing efficiency.

heading: 2.2 Image Enhanced Text Generation
text: VisCTG (Feng et al., 2022) enhances the commonsense ability of LMs by retrieving images and using image captions as input augmentation. In addition to explicitly retrieving images, VAWI (Guo et al., 2022) leverages information from vision-language models, i.e. CLIP (Radford et al., 2021), to aid natural language understanding. I&V (Wang et al., 2022) train an imagination model to generate a scene graph given an input under the supervision
of images and then train LMs to generate sentences based on both input and scene graph. The above methods either do not directly use images as nonverbal data or require fine-tuning the whole pretrained LMs to adapt to visual input. Drawing on the importance of imagination to human writing, iNLG (Zhu et al., 2022) and LIVE (Tang et al., 2023) use images generated by an image-generative model based on text inputs as supplementary information and train the LM to generate under visual guidance. However, the generated images may not necessarily carry commonsense information, such as cartoon images.

heading: 3 Generative Commonsense Reasoning
text: We focus on the task of CommonGen (Lin et al., 2020) to investigate and enhance the common sense reasoning capabilities of LMs.

heading: 3.1 Preliminaries
text: Problem Statement: The generative commonsense reasoning task in CommonGen asks the LM to make a sentence y that contains all the concept words in the given set C = {c1, ..., cK}, where ci denotes the i-th concept and y should describe a common scenario in our daily life.
Training Objectives: It is usually modeled as a sequence generation task and is optimized by minimizing the cross-entropy loss between the predicted token distribution and the reference distribution: L = −∑|y|t=1 logP (yt|C, y<t). In this work, to ensure parameter efficiency and applicability to LLMs, we use prompt tuning (Lester et al., 2021; Liu et al., 2021) instead of fine-tuning LMs. We only tune a task prompt, which is prepended to the input word embeddings in the first layer. When retrieval augmentation is enabled, a set of retrieved items D, which is retrieved with the concepts as query words, is also used as input and the new loss function becomes:
L = − |y|∑
t=1
logP (yt|C,D, y<t). (1)

heading: 3.2 Multi-Modal Retrieval Augmention
text: As shown in Figure 2, the Multi-mOdal REtrieval (MORE) augmented framework for text generation has four core components: retrieving relevant images and texts based on the concept words (§ 3.2.1), encoding the retrieved results with an encoder (§ 3.2.2), extracting useful information to
yield a retrieval augmented prompt with an integrator (§ 3.2.3), and generating sentences based on the retrieval augmented prompt, task prompt, and concept embeddings with the frozen LM backbone (§ 3.2.4).

heading: 3.2.1 Retrieval Results for Augmentation
text: Previous work (He et al., 2022; Li et al., 2021; Liu et al., 2022) that incorporates retrieval augmentation on this task consider the image/video captions (Krishna et al., 2017; Williams et al., 2017; Wang et al., 2019; Bowman et al., 2015; Lin et al., 2014) that CommonGen is built on as the retrieval candidates, which is obviously impractical. In such a setting, we find that the captions retrieved by BM25 (Robertson et al., 2009) can achieve comparable performance with the state-of-the-art (SOTA) methods that train retrievers for augmenting LLMs (shown in Appendix A), making the investigation less meaningful.
To accommodate the task in real-world scenarios, in this paper, we retrieve the image and text results from a general Web search engine, i.e., Bing, for retrieval augmentation. We employ Bing as a reasonable off-the-shelf retriever since our focus is on how to incorporate the supporting items rather than training a powerful retriever. Formally speaking, given a concept set C, we retrieved M images and N text snippets by Bing using words in C as the query, comprising the set of items D = {dv1, ...dvM , dt1, ..., dtN}.
Specifically, after we preprocessed the retrieved results by removing duplicate images and noisy text, we collected a total of 500,100 images and 787,970 text snippets. On average, each concept set has 14 images and 23 passages, which means that M and N can be at most 14 and 23 respectively. We retain the order returned by the browser without any re-ranking. See the Appendix B for more details.

heading: 3.2.2 Multi-Modal Encoder
text: Then we use an encoder to get the initial representation erai for each retrieved image or passage di:
erai = Encode(di). (2)
This results in a sequence of representations with denc dimension. To align the encodings of text and images in the same semantic space, we adopt a multi-modal encoder - the frozen Q-Former of the pre-trained BLIP2 (Li et al., 2023). Unlike the other commonly used multi-modal encoder -
CLIP (Radford et al., 2021), that encodes the input to a single final embedding, the Q-Former of BLIP2 outputs a sequence of embeddings and thus can retain more information.

heading: 3.2.3 Retrieved Information Integrator
text: The integrator extracts useful information from the representations of multiple retrieved results and condenses it into a retrieval augmented prompt. The Integrator has a Selector module and a Former module.
Selector: This module extracts useful information from the retrieval representations based on the input concept and outputs a variable-length retrieval augmentation representation. It receives the concatenation of multiple initial representations era = [era1 ; ...; e ra M+N ] ∈ R(M+N)×denc and the embeddings of the concept words ec ∈ Rlc×denc as input, in which lc is the number of tokens in the concept set. The Selector is composed of two stacks of identical layers. Each layer consists of a self-attention network, which is used for interaction between retrieved content, a cross-attention network, which is used for interaction between retrieved content and concepts, and a fully connected feed-forward network:
hselfi = Attn(hi−1W Q i ;hi−1W K i ;hi−1W V i )
hcrossi = Attn(h self i M Q i ;E raMKi ;E raMVi )
hi = h cross i Fi.
(3) Attn(Q,K, V ) is the multi-head attention layer as in Transformer (Vaswani et al., 2017). W ∈ Rdenc×dint , M ∈ Rdenc×dint ,and F ∈ Rdint×denc
are projection matrices, in which dint is the dimension of the hidden states of Integrator, and h0 is set to ec. The output of the Selector module is a variable-length retrieval augmentation representation h2 ∈ Rlc×dint .
Former: This module converts the representation produced by the Selector into fixed-length and projects it into the input embedding space of the LM. This results in the final retrieval augmentation prompt pra. The Former comprises a cross-attention network and a fully connected feedforward network:
pra ′ = Attn(qMQ ′ ;h2M K′ ;h2M V ′)
pra = pra ′ O,
(4)
in which q ∈ Rlq×dint is a learnable embeddings with fixed-length lq. O ∈ Rdint×dlm is the projection matrix used for spatial mapping and dlm is the dimension of the input embedding of the LM.

heading: 3.2.4 Soft Prompt Based Text Generation
text: To ensure training efficiency especially based on LMs, we freeze the parameters of the LMs and adopt the Prompt-tuning (Lester et al., 2021) technique, which incorporates the fixed-length embeddings produced by the Integrator as a plug-and-play soft prompt.
During text generation, the LM receives the concatenation of the task-specific prompt ptask and the concept set C as input, and generates sentence y as output, denoted as y = LM([ptask;C]). When using retrieval augmentation, besides the task-specific prompt, we also prepend the retrieval-augmented (RA) prompt to the input. Consequently, the input to the model becomes [pra; ptask;C].

heading: 3.2.5 Training Strategy
text: Query Concept Dropout: The retrieval quality can vary considerably, so the model may simply ignore the retrieval input instead of learning to extract useful information. To enhance the utilization of retrieval augmented inputs, we propose a query dropout training strategy. Specifically, we randomly mask the query concept C input to the LMs with probability p in the initial T training steps, and let the model generate sentences only based on retrieved results. Please note that query dropout is only applied to the input of LMs, and C is always input to the Integrator to guide the model in extracting beneficial information from the retrieved results. The probability p decreases as the number of training steps increases: p = 0.5× (1− sin(π(min( tT , 1)− 0.5))).
Noisy RA Input: It is also important to ensure that the model can learn to ignore noise rather than blindly trust the retrieved results. Therefore, we artificially introduce noise during query dropout by replacing the retrieval input with irrelevant results from other samples and correspondingly changing the target output with an ‘EOS’ token with probability p̂.

heading: 4 Experiments Settings

heading: 4.1 Dataset
text: We validate our method on the CommonGen dataset (Lin et al., 2020). It is designed for generative commonsense reasoning tasks involving the composition of discrete concepts into sentences depicting everyday scenarios. The dataset comprises 32,651, 993, and 1,497 unique concept sets for training, development, and testing, respectively. Each concept set has multiple associated gold target sentences, yielding 67,389, 4,018, and 6,042 sentences for reference in total. When retrieval augmentation is enabled, we used the retrieved results from Bing as introduced in Section 3.2.1. We will release the crawled images and text to encourage future research in multi-modal retrieval augmentation under a practical setting for CommonGen.

heading: 4.2 Methods for Comparisons
text: Text-based/Multi-modal LMs: For text-based LMs, we employ T5BASE as well as T5LARGE (Raffel et al., 2019) to represent small pre-trained LMs, and OPT2.7b (Zhang et al., 2022) to represent the LLMs. We also query the close source model
https://inklab.usc.edu/CommonGen/. Under MIT license.
gpt-3.5-turbo-1106 (OpenAI, 2023a) through API with the prompt "Use the given words to make a short sentence that is consistent with commonsense. Words: {...}". For Multi-modal LMs (MLMs) we compare with BLIP2-OPT2.7b (Li et al., 2023), an open source model, and gpt-4-1106-visionpreview (OpenAI, 2023b), a close source model. Since MLMs can accept images and text as input, we directly input the retrieved items into MLMs. All the above open source models are based on huggingface and are under Apache License 2.0.
Retrieval Augmented Generation Baselines: We consider two types of textual retrieval augmented models. One is Prepend (Mallen et al., 2023; Si et al., 2023; Ram et al., 2023), which prepends the top-k text results to the concepts as input. The other one is FiD (Izacard and Grave, 2021), which concatenates each retrieved passage with the concept words separately to encode in parallel for better handling of long text. For the visual retrieval augmented model, we compare with VisCTG (Feng et al., 2022), which generates a caption for each image with an image captioning model (Luo et al., 2018) and prepends the captions to the input for augmentation. All the above models use T5BASE as the backbone and are tuned with prompt-tuning.
MORE with Various Backbones: We test MORE with various backbones to explore whether it can be effectively used in different model architectures. Specifically, T5BASE and T5LARGE represent small LMs and are encoder-decoder architecture. BLIP2-OPT2.7b represent MLMs. It should be noted that BLIP2-OPT2.7b is equivalent to OPT2.7b when not receiving image input. Therefore it can also be regarded as a variant based on OPT2.7b, which represents LLMs and is decoderonly architecture.

heading: 4.3 Evaluation Metrics
text: To assess the generation performance, we use standard metrics: BLEU (Papineni et al., 2002) quantifying the overlap between predictions and references based on n-gram precision and ROUGE (Lin, 2004) measuring the n-gram recall. METEOR (Banerjee and Lavie, 2005) is an improved version of BLEU and considers both exact word matches and semantic similarities. CIDEr (Vedantam et al., 2014) focuses on capturing sentence semantic similarity. SPICE (Anderson et al., 2016)
https://github.com/huggingface
quantifies the semantic propositional content of generations by leveraging scene graphs. Please notice that SPICE aligns closely with human evaluation and should be treated as the primary metric. We also incorporate sentence similarity metrics (Sent-Sim) with SimCSE (Gao et al., 2021) to measure semantic similarity. We use the entire test dataset to obtain the main results and randomly sample 500 data in the test set to compare with GPT-4 for the sake of a limited budget.

heading: 4.4 Implementation Details
text: The same set of hyper-parameters is used for all the models. We use the AdamW (Loshchilov and Hutter, 2017) optimizer with β1 = 0.9, β2 = 0.999 and the weight decay is 0.05. The batch size is selected from {64, 128}. Models were trained with at most 20,000 steps with a 1% warm-up period. For retrieval augmentation, we train the model with an additional T = 2000 steps with query dropout, and the noisy RA input ratio p̂ is set to 0.3. The learning rates of the task prompt and the retrieval augmented prompt are selected from {1e− 4, 5e− 4, 1e− 3} and {1e− 5, 3e− 5} respectively. During decoding, we use beam search with size 5. We train the
Code and data are publicly available at https://github. com/VickiCui/MORE
models under each setting with 3 random seeds and choose the best ones according to the performance on the development set for testing. The prompt length of task and retrieval augmentation are both set to 32.

heading: 5 Results and Analysis

heading: 5.1 Overall Results
text: As shown in Table 1 and Table 2, after incorporating text and images to LMs, our method can boost the generation performance significantly based on various backbones. Comparing images and text, we find that images are better in improving commonsense ability, and incorporating both of them yields even better performance.
As shown in Table 3, Although based on a smaller model, MORE can achieve better results than GPT-3.5 and GPT-4. This fully illustrates the effectiveness of our method. Considering that GPT4 is a multi-modal language model, we also test its performance when retrieval augmented items are provided. However, GPT-4 cannot effectively utilize the retrieved inputs, leading to deteriorated performance. Retrieval augmentation methods for the GPT-4 model are worth exploring in the future. We also tested the model with a specified length limit to avoid the tendency of LLMs to generate longer
sentences. The length of each sentence is the average length of the golden references, so the results can be regarded as an upper bound. According to the most critical SPICE metric, length constraints do not lead to better results. Further analysis revealed that length constraints result in the concept coverage decrease, indicating that LLMs face challenges in organizing concepts with simple short sentences.
In terms of incorporating retrieved results, MORE is better than Prepend and FiD, which are textual augmented models. Although previous work found that using captions can have better results, this also risks leaking the answer. The method of directly inputting retrieved results becomes invalid after the retrieval content changes. MORE is also better than VisCTG, which is a visual augmented model. As shown in Appendix C, the generated captions are not always accurate and may lack the required information, so preconverting images to captions is not a proper approach to leverage images.

heading: 5.2 Ablation Study
text: We examine the effectiveness of various components within MORE by creating several variants, selectively removing or substituting each component, with results detailed in Table 4.
First, we replace the concepts that are input into the integrator with a randomly initialized learnable token sequence (w/o concept-input). The drop in performance highlights the importance of using concept words for references when extracting beneficial information from the retrieved results. Second, we remove the query dropout strategy (w/o query-dropout). The performance drop demonstrates its importance in effectively leveraging the retrieved results. Finally, we further exclude the noisy retrieval augmented input (w/o noisy-RA). Performance degradation indicates that blindly trust in retrieved input can harm model performance. It is necessary to explicitly instruct the model to learn to ignore the irrelevant augmentation results.
Please note that when the query dropout strategy is removed, it means that noisy retrieval augmented input is also not contained. However, the results of ‘w/o query-dropout’ is better than the results of ‘w/o noisy-RA’. This emphasizes the disadvantages of blindly trusting retrieval items, and it is necessary for the model to learn to distinguish irrelevant retrieval input.

heading: 5.3 Analysis
text: Are the improvements of MORE attributed to additional parameters? Considering that our framework introduces more parameters, we investigate whether the performance improvements are attributed to these additional parameters, which arise from two aspects: 1) The retrieval augmented prompt results in an extended input length. To investigate, we adjust the task prompt length from 32 to 64, aligning with the total input length of MORE. 2) The integrator introduces more learnable parameters. To assess whether this would affect performance, we replace the retrieval inputs of each sample with irrelevant retrieved results during training, denoted as rand-RA. This maintains consistency in the learnable parameters with MORE. The experimental results are recorded in Table 5. Neither of them shows significant improvements over the backbone T5-base, showing that the benefit of MORE does not come from the extra parameters.
Will utilizing more retrieved results enhance the model performance? In retrieval-augmented methodologies, a crucial factor influencing the final results is the number of retrieval items. We inte-
grate varying numbers of retrieval content for both single-modal and multi-modal settings. The SPICE results are illustrated in Figure 3, with additional metric results available in Appendix D. No matter which modality is used, the model performance first increases and then decreases as the number of retrieval inputs increases. This is because too few retrieval inputs may lead to insufficient coverage of required information, while an excess of inputs may introduce redundancy and noise.
Is MORE robust to noise in retrieval augmentation? The retrieved results may not always be of high quality and occasionally may be even irrelevant to the query. Therefore it is also important for retrieval augmented models to be robust to the noisy retrieval outcome. We test the model’s robustness in the face of poorly retrieved results by feeding it with only irrelevant retrieval content during testing (denoted as Noisy-MORE). As illustrated in Figure 4, Noisy-MORE performs similarly to its backbone T5 when using irrelevant results for augmentation. This indicates that MORE is robust to the noise in the retrieved items by not blindly trusting the augmentation input.

heading: 5.4 Case Study
text: We conduct case studies to qualitatively analyze how MORE enhances text generation through retrieval augmentation. As shown in Figure 5, small LMs like T5LARGE make a nonsensical sentence that ‘only one blower can not drive side by side’. The retrieved images show a ’blower located on the side of the road’ scene, and the retrieved text describes ‘throw the snow to the side of the road’, thereby helping the model clarify the usage of ‘side’ and correct generation errors. More cases can be found in Appendix E.
As for LLMs, we find that they sometimes make nonsensical sentences, as shown in Figure 1. This shows that even if a huge amount of parameters
and training data are used, the LLMs are still not able to grasp commonsense knowledge completely. Therefore, it is also necessary for LLMs to use retrieval augmentation to provide reference. The other characteristic is that the sentences made by GPT-3.5 and GPT-4 are usually long. To connect the given concepts and output reasonable sentences, they may need more words or information. This also reflects the lack of commonsense knowledge that humans are well aware of.

heading: 6 Conclusion and Future Work
text: To sum up, we introduce MORE, a multi-modal retrieval augmentation framework. Our approach is capable of extracting useful information and disregarding irrelevant noise from visual and textual results of variable quality, thereby assisting language models in generating reasonable sentences. Extensive experiments on the CommonGen task demonstrated the effectiveness of our method. This novel approach may offer a new perspective for retrieval-augmented language models.
We focus on the generation task in this work and the application of multi-modal retrieval augmentation on more tasks is worth exploring in the future. Besides, the current method concentrates on ‘how to incorporate multimodal retrieved items’ and does not involve optimization of the retrieving step, which is left for future work.

heading: 7 Limitations
text: Our method uses soft-prompt, making it unsuitable for LMs accessible solely through the API as it cannot convey input in natural language form. In addition, to avoid changing the internal structure of the LMs, we adopted the p-tuning in this work. Using more advanced methods such as LoRA (Hu et al., 2021) to achieve better results can be considered in future work.
The retrieved results are from public data on the
Internet and we did not collect any privately identifiable information in our study. However, it may be inevitable to crawl some public photos and other data, which may, which may still include some personal information, such as faces. We followed Bing’s authorization requirements for the use of data and did not modify or use it commercially. We call on anyone using our framework to follow the licensing requirements and not misuse the technology.

heading: Acknowledgement
text: This work was funded by the National Natural Science Foundation of China (NSFC) under Grants No. 62302486, the Innovation Project of ICT CAS under Grants No. E361140, the CAS Special Research Assistant Funding Project, the Lenovo-CAS Joint Lab Youth Scientist Project, the project under Grants No. JCKY2022130C039, and the Strategic Priority Research Program of the CAS under Grants No. XDB0680102.

heading: A Test Result Using Captions
text: Since the Commongen dataset itself relies on caption data during the construction process, and most existing methods use the retrieved caption as a reference for generation, such as DKMR2 (He et al., 2022), KFCNet (Li et al., 2021), and KGR4 (Liu et al., 2022). We suspect and test whether the retrieved caption itself reveals the correct answer to some extent. Specifically, we use concepts as query, and then simply use BM25 as the retrieval method to retrieve captions from image captions and video captions. The retrieved caption will be directly used as the prediction result without modification and the results compared with other methods are shown in Table 6. It can be seen that even without a tunable retriever and any modification to the caption, good results can be achieved.

heading: B Retrieved Inputs Crawling and Preprocessing
text: We concatenate all concepts in a concept set to form a query. For the image, we use the template ‘a photo of {...}’ (e.g. a photo of decorate, music, background, and tree) and crawl the first 20 image results returned by the search engine. We further removed duplicate images based on the dHash algorithm. For text, we directly use the concatenation of the concept set as the query. We crawl the text results from the first two pages. Considering that the full document associated with each result may be very long, the search engine has provided a concise text summary of the webpage aligning with the search keywords, we only keep the title and description in the snapshot. We also removed the URL and non-English parts.

heading: A dog rolling on the back of the grass

heading: C Examples of Generated Captions from VisCTG
text: We use an example to illustrate why it is better to directly use the raw image than to convert the image into a caption. There are two main reasons: 1) the generated caption may be inaccurate. As shown in Figure 6, due to the error of the model and the bias in training data, when ‘dog’ appears, the caption model always generates sentences containing ‘frisbee’ or ‘ball’, even though these objects do not appear in the image. Inaccurate captions will further mislead follow-up text generation. 2) the pre-generated captions may lack the required information. In the example, the T5 model incorrectly generates ‘the back of the grass’, and the information needed is ‘dog rolls on their back‘. Although the images contain relevant information, it is not included in the captions, so that the original generation cannot be corrected.

heading: D Results of Other Metrics
text: Combining various metrics shown in Figure 7, it can be seen that using three retrieval contents is a better choice. The conclusion that using too many or too few retrieved results will not lead to optimal results has not changed.

heading: E Cases of Generation
text: We show two additional examples in Figure 8 to help intuitively understand how multi-modal retrieval augmentation helps the model generate more reasonable sentences.

heading: Future Work
text: , 2021) to achieve better results can be considered in future work. The retrieved results are from public data on the
Internet and we did not collect any privately identifiable information in our study. However, it may be inevitable to crawl some public photos and other data, which may, which may still include some personal information, such as faces. We followed Bing’s authorization requirements for the use of data and did not modify or use it commercially. We call on anyone using our framework to follow the licensing requirements and not misuse the technology