Retrieving Multimodal Information for Augmented Generation: A Survey
Authors: Ruochen Zhao
Hailin Chen
Weishi Wang
Fangkai Jiao
Do Xuan Long
Chengwei Qin
Bosheng Ding
Xiaobao Guo
Minzhi Li
Xingxuan Li
Shafiq Joty
Venue: EMNLP 2023 Findings (2023)
Keywords: Retrieval-augmented language models
Multimodality

Abstract: As Large Language Models (LLMs) become popular, there emerged an important trend of using multimodality to augment the LLMs' generation ability, which enables LLMs to better interact with the world. However, there lacks a unified perception of at which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Such methods offer a promising solution to important concerns such as factuality, reasoning, interpretability, and robustness.  By providing an in-depth review, this survey is expected to provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs.

Retrieving Multimodal Information for Augmented Generation: A Survey
Ruochen Zhaot Hailin Chen'
Weishi Wang's Fangkai Jiao!
Xuan Long Do Chengwei Qin' Bosheng Ding' Xiaobao Guo!
Minzhi Li
Xingxuan Li Shafiq Joty
Nanyang Technological University, Singapore
National University of Singapore, Singapore
*Salesforce Research
/nunchen(022, hailin001, xuanlongt00,, weishi001, fangkai002, chengwei003, bosheng00) /@e.ntu.edu.sg |xiaoba0001, xingxuan000 |@c.ntu.du.sg liminzhi@u.nus.ccie srjoty@ntu.cdu.sg
Abstract
the factuality and rationality of the generated con- tent. Recently, there have been emerging studies fo- cusing on retricval -uggeeeedd approaches (Mialon et al., 2023), which aim to provide information that is more grounded and factually dependent. Among them, most (Nakano et al., 2021; Guu et al., 2020b) retrieves textual information, which matches the data format used during pre-training and offers natural medium for interaction. However, there more world knowledge stored different struc- tures and modalities, such as images and videos. which often inaccessible, unavailable, or not de- scribable in traditional textual corpora.
As Large Language Models (LLMs) become popular, there emerged an important trend of us- ing multimodality to augment the LLMs' gen- eration ability, which enables LLMSs to better interact with the world. However, there lacks unified perception of which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment gen- erative models by retrieving multimodal knowl- edge, whose formats range from images, tables, graphs. to audio. Such methods offer promising solution to important concerns such as factuality, reasoning, interpretability, and ro- bustness. By providing an in-depth review, this survey is expected to provide scholars with deeper understanding cations and encourage them to adapt existing techniques to the fast-growing field of LLMs.
codes,
Therefore, there arises an important research in- tersection that retrieves multimodal knowledge to augment generative models. It offers promising solution to current challenges such as factuality, reasoning, interpretability. and robustness. As this field is very recent, there lacks a unified under- standing of recogmizing these methods specific group, visualizing their innate connections, con- necting their methodologies, and outlining their applications.
the methods' appli-
1 Introduction
Generative Artificial Intelligence (GAI) has demon- strated impressive performances in tasks such as text generation (Ouyang et al., 2022; Chowdhery al., 2022; Brown al., 2020) and text-to-image generation (Ramesh et al., 2021a; Poole et al., 2022). The recent advancements in Multimodal Large Language Models (MLLMs) (Driess et al., 2023; OpenAI, 2023; Huang et al., 2023b) have further improved the models' capabilities to handle multi-formal information, opening up possibilities for developing general-purpose learners.
Therefore, we survey recent advancements multimodal retrieval-augmented generation (RAG), Specifically, we discuss current research by group- ing them into different modalities, including im- age, code, structured knowledge, audio, and video. For each modality. we systematically search the ACL Anthology and Google Scholar with relevant keywords and perform manual filtering to deter- mine their relevance to the survey. As result, we collect 146 papers for detailed analysis. In Ap- pendix A.1, we include search details, statistics, and a trend analysis figure, which shows that mul- timodal RAG papers have indeed developed very fastly since the emergence of large-scale general- purpose models. Within each modality, we discuss relevant papers by grouping them under different applications. By providing an in-depth survey, we hope to help researchers recognize the importance
Nevertheless, generative models are not ex- empt from inherent limitations, including the ten- dency for generating hallucinations (Ye and Dur- rett, 2022), struggling with arithmetic tasks (Patel et al., 2021), and lacking interpretability. Conse- quently, promising solution for enhancing their capabilities lies in enabling them to interact with the external world and acquire knowledge in di- verse formats and modalities, thereby improving
NNw affiliated with NUS
Work done while the author is on leave from NTU

---

of incorporating knowledge in different formats and encourage adaptation and advancements on ex- isting techniques to the fast-growing field of LLMs.
to large amount of multimodal data and designing network that produces semantically meaningful outputs.
In summary, our contributions are as follows:
We establish retrieval augmented generation with multi-modality as an important group of methods that emerges with the recent advances in LLMs. For common modalities, we provide an in-depth review of research papers by contextualizing their innate connections and shared challenges.
2.2 Retrieval-Augmenter Generation (RAG)
RAG typically consists of two phases: retrieving contextually relevant information, and guiding the generation process using the retrieved knowledge.
Recently RAG has gained popularity in Natu- ral Language Processing (NLP) due to the rise of general-purpose Large Language Models (LLMs) (Chowdhery et al., 2022; OpenAI, 2023), which have boosted performances in wide range of NLP tasks. However, there are two primary challenges: Firstly, because generative models rely on the inner knowledge (weights), they result in a high amount of hallucinations (Zhao et al., 2023b). Secondly, due to their large parameter sizes and the high costs of updating, the traditional pretraining and finetun- ing approaches have become infeasible. As solu- tion, RAG methods (Gu et al., 2018; Weston et al., 2018; Cai al., 2019b; Lewis et al.. 2020) offer a promising solution for LLMs to effectively interact with the external world.
We provide informative analysis of future di- rections, which could contain promising solu- tions many current challenges.
2 Definitions and Background
To better understand the state and advancements that inspired multimodal retrieval augmentation, we first define and discuss the background of two key concepts: multimodal learning and retrieval- augmented generation (RAG).
2.1 Multimodal Learning
Multimodal learning refers learning unified representation of data from different modalities. aims extracting complementary information to facilitate compositional tasks (Baltrušaitis et al., 2018; Gao et al., 2020). In this survey, we include all modalities whose formats are different from natural language, including image, code, structured knowledge (e.g. tables, knowledge graphs), audio, and video.
RAG is applied to a wide range of downstream NLP tasks, including machine translation (Gu et al., 2018; Zhang et al., 2018; Xu et al., 2020; He et al.,
2021), dialogue generation (Weston et al., 2018; Wu et al., 2019; Cai et al., 2019a), abstractive summarization (Peng al., 2019), and knowledge- intensive generation (Lewis et al., 2020; Izacard and Grave, 2021). Among them, most methods fo- cus on retrieving textual information. For example, Guu et al. (2020b); Lewis et al. (2020); Borgeaud et al. (2022); Izacard et al. (2022) jointly train a retrieval system with an encoder or sequence-to- sequence LM, achieving comparable performance to larger LMs that use significantly more parame- ters. Recent research also proposes combining triever with chain-of-thought (CoT) prompting for reasoning to augment language models (He et al., 2022a; Trivedi et al., 2022; Zhao et al., 2023c).
Multimodal generative models have a wide range of applications, such as text-image generation, cre- ative writing generation, and multilingual transla- tion. For instance, the image recognition task can benefit from analyzing images and videos in con- junction with textual descriptions (Ju et al., 2022: Alayrac et al., 2022a; Jia et al., 2021; Radford al., 2021b). Conversely, incorporating visual information also aids language understanding and generation (Zhou al., 2020: Lei et al., 2021: ?). Moreover, they have the potential to significantly improve machine learning systems across various domains by enabling models to learn from and in- tegrate multiple sources of information (Tsai et al., 2019; Acosta et al., 2022; Nagrani et al., 2021 I). Additionally, there has been growing interest in de- veloping generative models that can output multiple modalities of data (Ramesh et al., 2021b; Crowson et al., 2022; Lin and Byrne, 2022a; Chen et al., 2022a). However, there remain challenges for mul- timodal generative models, such as gaining access
3 Multimodal Retrieval-Angmented Generation
For each modality, there are different retrieval and synthesis procedures, targeted tasks, and chal- lenges. Therefore, we discuss relevant methods by grouping them in terms of modality, including im- age, code, structured knowledge, audio, and video.

---

3.1 Image
reward function to train more fine-graines cap- tioning model. Beyond retrieving image elements, Sarto et al. (2022); Shi et al. (2021); Ramos et al. (2023); Yang et al. (2023b) retrieves relevant cap- tions to the inputs. Zhou et al. (2022a) tackles news image captioning by retrieving visually grounded entities in news articles.
Recent advances on pretrained models shed light on general image-text multi- modal models (Ramesh et al., 2021a; Alayrac et al., 2022b; Aghajanyan et al., 2022; Yu et al., 2022; Dou et al., 2022; Li al., 2023a). However, these models require huge computational resources for pretraining and large amounts of model parameters - as they need to memorize vast world knowledge. More critically, they cannot efficiently deal with new or out-of- domain knowledge. To this end, multiple retrieval- augmented methods have been proposed to better incorporate external knowledge from images and text documents. In general text generation tasks, image retrieval can also improve generation quality by expanding text generation contexts with more "imagination".
Visually grounded dialogue This task (Lee al.. 2021b) requires retrieving visual information produce relevant dialog responses. Fan et al. (2021) augments generative models with KNN-based In- formation Fetching (KIF) modules that retrieve im- ages and wiki knowledge. Liang et al. (2021) re- trieves a correlated image to the dialog from an im- age index to ground the response generator. Shen et al. (2021) trains word-image mapping model to retrieve response visual impressions and then uses both textual and visual information for response generation.
Visual question answering (VQA) To tackle open-domain VQA, RA-VQA (Lin and Byrne. 2022b) jointly trains the document retriever and answer generation module by approximately marginalizing predictions over retrieved docu- ments. It first uses existing tools of object de- tection, image captioning, and optical character recognition (OCR) to convert target images to tex- tual data. Then, it performs dense passage retrieval (DPR) (Karpukhin al., 2020a) fetch text docu- ments relevant to the target image in the database. Finally, each retrieved document concatenated with the initial question to generate the final predic- tion, similar to RAG (Lewis et al., 2020). Besides external documents, PICa (Yang et al., 2022b) and KAT (Gui et al., 2022) also consider LLMs as im- plicit knowledge bases and extract relevant implicit information from GPT-3. Plug-and-Play (Tiong al., 2022) retrieves relevant image patches by using GradCAM (Selvaraju et al., 2017) to localize relevant parts based on the initial question. It then performs image captioning on retrieved patches to acquire augmented context. Beyond text-only augmented context, MuRAG (Chen et al., 2022b) retrieves both text and image data and incorporates images visual tokens. RAMM (Yuan al., 2023) retrieves similar biomedical images and captions and encodes them through different networks.
Text generation For general text generation tasks, image retrieval can also help expand contexts. Yang et al. (2022a) augments text model's "imagina- tion by retrieving existing images and synthesiz- ing newly generated images. As result, fueling language models with imagination can improve per- formances in many downstream natural language tasks. Similarly, Zhu et al. (2023) compares "imag- ination" augmentation with synthetic and retrieved images and argues that machine-generann could provide better guidance due to better con- sideration of the contexts. Moreover, Fang and Feng (2022) shows that machine translation be significantly improved by retrieving visual in- formation at the phrase level, especially when the textual context is limited. Image RAG can also help low-resource tasks such as medical report genera- tion (Wu et al., 2022a) and architectural description generation (Mille et al., 2020).
images
Beyond retrieving images before generating text, Re-Imagen (Chen al., 2022c) leverages multi- modal knowledge base to retrieve image-text pairs to facilitate image generation. RA-CM3 (Yasunaga et al., 2022) can generate mixtures images and text. It shows that retrieeal-augmented image gener- ation performs much better on knowledge-inten -inteeevve generation tasks and opens up new capabilities such as multimodal in-context learning.
Image captioning To generate multi-style cap- tions, Zhou and Long (2023) uses a style-aware visual encoder retrieve image contents before generating captions. Beyond simply encoding vi- sual information, Cho et al. (2022) further uses the multimodal similarity between image-text pairs as
3.2 Code
Software developers attempt to search for rele- vant information to improve their productivity from large amounts of available resources such as expla-

---

nations for unknown terminologies, reusable code patches, and solutions to common programming bugs (Xia et al., 2017). Inspired by the progress of deep learning in NLP, a general retrieval- augmented generation paradigm has benefited a wide range of code intelligent tasks, including code completion (Lu et al., 2022b), code generation (Zhou et al., 2022b), and automatic program re- pair (APR) (Nashid et al., 2023). However, these approaches often treat programming languages and natural languages as equivalent sequences of tokens and ignore the rich semantics inherent to source code. To address these limitations, recent research work has focused on improving code generaliza- tion performance via multimodal learning, which incorporates additional modalities such as code comments, identifier tags, and abstract syntax trees (AST) into code pretrained models (Wang al.. 2021b; Guo et al., 2022; Li et al., 2022d). To this end, multimodal retrieval- -augentl- generation approach has demonstrated its feasibility in a vari- ety of code-specific tasks.
the decoding phase. This idea is further explored by Liu et al. (2021), where retrieved code-summary pairs are used to augment the original code prop- erty graph (Yamaguchi et al., 2014) of source code via local attention mechanisms. To capture the global semantics for better code structural learn- ing, global structure-aware self-attention mecha- nism (Zhu et al., 2019) is further employed.
Code Completion Recent advances in retrieval- based code completion tasks (McConnell, 2004) have gained increasing attention. No- tably, Hashimoto al. (2018) adapts the retrieve-and-edit framework to improve the model's performance in code auto-completion tasks. To address practical code completion scenar- ios, ReACC (Lu et al., 2022b) takes both lcxical and semantic information of the unfinished code snippet into account, utilizing hybrid technique to combine lexical-based sparse retriever and semantic-based dense retriever. First, the hybrid retriever searches for relevant code from the codebase based on the given incomplete code. Then, the unfinished code is concatenated with the retrieval, and an auto-regressive code completion generator will generate the completed code based on them. In order to address project relations, CoCoMIC (Ding al., 2022) decomposes a code file into four components: files, global variables, classes, and functions. context graph based on the hierarchical relations among all associated code components, forming a project-level context graph by considering both in-file and cross-file dependencies. Given an incomplete program, CoCoMIC retrieves the most relevant cross-file entities from its project-level context graph and jointly learns the incomplete program and the retrieved cross-file context for code completion.
Text-to-Code Generation Numerous research studies have investigated the utilization of relevant codes and associated documents to benefit code generation models. A prominent example is RED- CODER (Parvez al., 2021), which retrieves the top-ranked code snippets summaries from ex- isting codebase, and aggregates them with source code sequences to enhance the generation or sum- marization capabilities. As another such approach, DocPrompting (Zhou et al., 2022b) uses a set of relevant documentation as in-context prompts to generate corresponding code via retrieval. In addi- tion to these lexical modalities, Hayati et al. (2018) proposes syntax- based code generation approach reference existing subtree from the AST as tem- plates to direct code generation explicitly.
constructs an in-filc
Code-to-Tex Generation Retrieval-based code summarization methods are studied extensively. For example, RACE (Shi et al., 2022) leverages rel- evant code differences and their associated commit messages to enhance commit message generation. Besides, RACE calculates the semantic similarity between source code differences and retrieved ones to weigh the importance of different input modali- ties. Rencos (Zhang al., 2020) retrieves two sim- ilar code snippets based on the aspects of syntactic- level similarity and semantic-level similarity of a given query code. These similar contexts are then incorporated into the summarization model during
Automatic Program Repair (APR) Inspired by the nature that a remarkable portion of com- mits is comprised of existing code commits (Mar- tinez et al., 2014), APR is typically treated as a search problem by traversing the search space of repair ingredients to identify correct fix (Qi et al., 2014), based on redundancy assumption (White et al., 2019) that the target fix can often be recon- structed the search space. Recent studies have shown that mining relevant bug-fix patterns from existing search space (Jiang et al., 2018) and ex- ternal repair templates from StackOverflow (Liu and Zhong, 2018 8) can significantly benefit APR

---

models. Joshi al. (2022) intuitively ranks a col- lection of bug-fix pairs based on the similarity of error messages to develop few-shot prompts. They incorporate compiler error messages into a large programming language model Codex (Chen et al., 2021) for multilingual APR. CEDAR (Nashid et al., 2023) further extends this idea to retrieval-base based prompts design using relevant code demonstrations, comprising more modalities such as unit test, er- ror type, and error information. Additionally, Jin al. (2023) leverage a static analyzer Infer ex- tract error type, error location, and syntax hierar- chies (Clement et al., 2021) to prioritize the focal context. Then, they retrieve semantically-similar fixes from an existing bug-fix codebase and con- catenate the retrieved fixes and focal context to form the instruction prompts for program repair.
idea of transforming structured commonsense gen- cration tasks into code generation problems and employing LLMs of code as the solvers, One such work is CoCoGen (Madaan et al., 2022) which con- verts each training sample (consisting of textual input and the output structure) into a Tree class in Python. The LLMs of code then perform few-shot reasoning over the textual input to generate Python code, and the Python code is then converted back to the original structure for evaluation. Besides, the success of LLMs of code such as Codex in syn- thesizing computer code also makes them suitable for generating formal codes. Motivated by this, Wu et al. (2022b) propose to prove mathematical theorems by adopting Codex to generate formal- ized theorems from natural language mathematics for the interactive theorem prover Isabelle (Wenzel et al., 2008).
Reasoning over Codes as Intermediate Steps While large language models (LLMs) have recently demonstrated their impressive capability to per- form reasoning tasks, they are still prone to logi- cal and arithmetic errors (Gao al., 2022a; Chen et al., 2022d; Madaan et al., 2022). To mitigate this issue, emerging research papers have focused on using LLMs of code (c.g., Codex (Chen et al., 2021)) to generate the code commands for solving logical and arithmetic tasks and calling external interpreters to execute the commands to obtain the results. Notably, Gao al. (2022a) proposes to generate Python programs as intermediate reason- ing steps and offload the solution step to a Python interpreter. Additionally, Chen et al. (2022d) ex- plore generating chain-of-thought (CoTT) (Wei et al., 2022) for not only text but also programming lan- guage statements as reasoning steps to solve the problem. During the inference phase, answers obtained via an external interpreter. Similarly, Lyu et al. (2023) propose Faithful CoT that first trans- lates the natural language query to a symbolic rea- soning chain and then solves the reasoning chain by calling external executors to derive the answer. Another example is Ye et al. (2023), which uti- lizes LLMs to decompose table-based reasoning tasks into subtasks, decouples logic and numerical computations in each step through SQL queries by Codex, and calls SQL interpreters solve them (a process called mensinn-executionn -illinggg..
3.3 Structured Knowledge
An open challenge in generative models is hallu- cination, where the model is likely to output false information. Thus, A potential solution is to ground generation with retrieved structured knowledge. such as knowledge graphs, tables, and databases. Question Answering (QA) A natural setting to use knowledge is QA. To augment Knowledge Base (KB) QA by extracting the most relevant knowledge, Hu et al. (2022b) uses dense retrieval and Liu et al. (2022b) uses cross-encoder ranker. Shu et al. (2022) employs multi-grained retrieval extract relevant KB context and uses constrained decod- ing to control the output space. In table QA, Nan et al. (2022) proposes a dataset that requires re- trieving relevant tables for answer generation. Pan et al. (2021) then proposes a method that uses a transformer-based system to retrieve the most rel- evant tables and locate the correct cells. More- over, to improve Video QA, Hu et al. (2023) trieves from Knowledge Graph (KG) encodings stored in the memory. The most prominent RAG usage remains in open-domain QA, where multiple datasets are proposed (Lin et al., 2023; Ramnath et al., 2021). 1). For retrieval, Ma et al. (2022) verbal- izes the KG and then uses dense passage retrieval. Fan et al. (2019); Gupta et al. (2018) encodes KG information into dense representations. Pramanik et al. (2021); ; Jin al. (2022) builds graph embed- dings to retrieve question-relevant evidence. Xu et al. (2021); Baek et al. (2023) use semantic sim- ilarity and text- matchingg methods. Synthesis can occur at different stages. At the input stage, Xu
LLMs of code are also known as good-structured commonsense reasoners, and even better-structured reasoners than LLMs (Madaan et al., 2022). As a result, prior studies have also investigated the

---

al. (2021); Back et al. (2023) feed in the re- trieved contexts as additional inputs or prompts to the PLM. (Ma et al., 2022; Fan et al., 2019) adapt the generator to accept the context representations as inputs. At model inference stage, an interesting work is Hu et al. (2022c), which inserts an inter- action layer into PLMs to guide an external KG reasoning module.
tual premises iteratively and combines them with generation. Yang al. (2022c) proposes math reasoner that first retrieves highly-correlated alge- braic knowledge and then passes them as prompts to improve the semantic representations for the gen- cration task. With the recent advances in LLMs, He et al. (2022a); Li et al. (2023b) retrieve from KG and KB, such as Wikidata, based on reason- ing steps obtained from the chain-of-thought (CoT) prompting (Wei al., 2022).
General text generation External knowledge retrieval can improve general text generation to be more factually grounded. Liu et al. (2022a) presents memory-augmmntted approach to condi- tion an autoregressive language model on a knowl- edge graph (KG). During inference, Tan et al. (2022) selects knowledge entries through dense retrieval and then injects them into the input en- coding and output decoding stages in pretrained language models (PLMs). For domain-spp -eeciiic text generation, Frisoni et al. (2022); Yang et al. (2021); Li et al. (2019) retrieve medical report chunks or report templates to augment input prompts. Then, they use self-devised decoders or graph transform- ers to generate grounded reports. To improve in- terpretability, RAG could be used to select facts interpretable reasoning paths (Aggarwal al., 2021; Jansen and Ustalov, 2019). Moreover, RAG is especially useful for low-resource generation tasks, such as question generation (Yu and Jiang, 2021; Xin et al., 2021; Gu et al., 2019), document- to-slide (Sun et al., 2021), table-to-text (Su et al., 2021), counterargument generation (Jo et al., 2021). entity description generation (Cheng et al., 2020) and text-based games (Murugesan al., 2021).
Knowledge-grounded dialogue Dialogue gen- eration based on relevant tables and knowledge bases has been a practical research application (Wu et al., 2020b; Li et al., 2022b; Nakamura et al., 2022; Gao et al., 2022b; Lu al., 2023). To tackle the challenge, Li et al. (2022c) and Galetzka et al. (2021) retrieve relevant knowledge, process it into a dense representation and incorporate it into dia- logue generation. On top of dense representations. Gu et al. (2020) and Jung et al. (2020) leverage at- tention mechanisms to flexibly adjust which knowl- edge to depend on during generation. Some meth- ods (Zhang al., 2021; Dziri et al., 2021; Chen et al., 2020) first generate subgoals or responses and then use them to retrieve relevant knowledge. The retrieved knowledge then helps amend pre- vious responses. Besides knowledge, Cai et al. (2019b) and Wu et al. (2020a) improve dialogue re- sponse generation by retrieving templates or proto- type dialogues to augment inputs. Recently, Kang et al. (2023) retrieves relevant subgraphs from KGs, and then utilizes contrastive learning ensure that the generated texts have high similarity to the sub- graphs.
Recent research has attempted to reduce hallu- cinations in LLMs by leveraging external struc- tured knowledge. For example, during fine-tuning. LaMDA (Thoppilan et al., 2022) learns to consult external knowledge sources before responding to the user, including an information retrieval system that can retrieve knowledge triplets and web URLs. Some papers treat the generative model (often large language models) as black-box and retrieve struc- tured information without fine-tuning. For exam- ple, BINDER (Cheng et al., 2023) uses in-context learning to output designed API calls that retrieve question-relevant columns from tables.
By retrieving from relevant sources, RAG not only improves factuality but also provides the grounding contexts while generating, thus address- ing interpretability and robustness concerns. With the potential to handle more information types with recent advances LLMs (OpenAI, 2023), RAG with structured knowledge could be further en- hanced. There are still challenges to be addressed. For example, there could be new designs for bet- ter retrieval systems that could promote efficient interactions suitable for diverse knowledge bases. Synthesizing this information correctly is also an open challenge, where it is hard decide which parts need augmenting in the textual outputs.
Reasoning with knowledge By selecting knowl- edge, reasoning tasks can be solved in more grounded and interpretable way. To generate an entailment tree explanation for given hypothe- sis, Neves Ribeiro et al. (2022) retrieves from tex-
3.4 Audio
Audio RAG is useful in incorporating audio infor- mation in specific audio-land andio-language tasks, such as

---

music captioning, music and text generation, and speech recognition. Moreover, using audio RAG for audio data augmentation has also been proven useful in mitigating the lack of audio-text training data. It could be a promising future direction (Li et al., 2022a).
Music generation Royal al. (2020) uses deep neural hashing to retrieve music building blocks and then performs generation by using the current music segment to retrieve the next. In automatic speech recognition (ASR), Chan et al. (2023) uses k-nearest neighbor (KNN) approach to retrieve external knowledge related to the audio and text embeddings. The retrieved knowledge significantly reduces domain adaptation time for ASR.
Text-audio data augmentation For text-audio tasks, one of the most important challenges is the lack of training data on audio-text pairs. Therefore, retrieving audio and textual cues can alleviate the data scarcity problem and improve performance. In audio captioning, which aims translating the in- put audio into description, Koizumi et al. (2020) retrieves guidance captions similar to the input au- dio from the training set. Then, the retrieved guid- ance captions are fed into a PLM to help gener- ate new captions, which improves generation per- formance. To augment scarce speech translation (ST) data, Zhao et al. (2023a) proposes Spoken- Vocab, technique convert machine translation (MT) data to synthetic ST data. To form synthetic speech, Spoken Vocab retrieves and stitches audio snippets, corresponding to words in an MT sen- tence. Experiments show that stitched audio snip- pets can improve translation quality. Kim et al. (2023) leverages PLM to tackle the data scarcity issue. It retrieves features from the input audio, maps them to continuous vectors using mapping networks, and uses vectors as prefixes for pre- fix tuning the PLM. With the additional informa- tion from retrieved audio, it outperforms previous methods. In text-to-audio generation, Huang et al. (2023a) applies audio-text retrieval to get pseudo text prompts, which enhance audio generation in data-scarce scenarios. To augment the argumenta- tion mining (AM) task in political debates, Mestre al. (2023) integrates audio features into PLMs, which improves performance when data is scaree.
The audio modality is closely intertwined with other modalities, such as video. Therefore, recent advancements in using audio features for text-video retrieval (Falcon et al., 2022; Mithun et al., 2018) can benefit RAG tasks involving other modalities. Moreover, although audio-text retrieval has been long-standing task (Liu et al., 2015; Milde et al., 201 6a.b), exploring recently discovered techniques (Hu et al., 2022a; Lou et al., 2022; Koepke et al., 2022) could lead to further improvements.
3.5 Video
Retrieving video snippets for generation used primarily in two tasks: video-grounded dialogue and video captioning. Recently, augmenting LLMs with video retrieval also demonstrates good perfor- mances, especially in few-shot settings.
Video-grounded dialogue Given video con- texts, the model learns to engage in a relevant di- alogue. Pasunuru and Bansal (2018) introduces a video-context, many-speaker dialogue dataset, which challenges researchers to develop visually- grounded dialogue models that generate relevant responses from live videos. Similarly, Lei et al. (2020) proposes TVQA+, dataset that requires re- trieving relevant video moments answer textual questions about videos. Then, proposes a unified framework that encodes video segments into repre- sentations, uses an attention mechanism to locate relevant information, and produces textual answers. To better perform visually-grounded dialogue tasks, Le et al. (2020) retrieves visual cues from prior user queries. The cues then used contextual infor- mation to construct relevant responses. On video QA, it substantially outperforms prior approaches. Recently, Le et al. (2022) extracts visual cues from the video to augment video-grounded dialogues. The video retrieval is performed with neural mod- ule networks, which are instantiated with entities and actions in previous dialogues.
Music captioning Music captioning is the task of generating text description or lyrics given the music audio. And RAG is explored to learn better audio-lyric alignment. Manco et al. (2021) pro- poses the first music audio captioning model, Mus- Caps. Firstly, a pretrained multimodal encoder obtains audio representations that retrieve musical fcatures in the input. As the pretraining bridges the gap between the audio modality and textual under- standing, the method improves task performance. He et al. (2022h) learns an audio-lyric alignment through contrastive learning, which results in a hualet-quality generation of captions for music.
Video captioning Sharing similar motivation to RAG, Long et al. (2018) first proposes to use attention layers to automatically select the most

---

salient visual or semantic features and use them to augment caption generation. As a result, it stably outperforms previous methods. (Whitehead et al.. 2018) then develops a retrieval- based approach for video description generation. For news videos, it retrieves topically related news documents and then generates description using a knowledge-aware video description network.
into two-stage (rationale generation and answer inference) framework, surpassing GPT-3.5 by a large margin with a much smaller fine-tuned model. Similar to Zhang et al. (2023b), kosmos-1 (Huang et al., 2023b) breaks down multimodal reasoning into two steps. It first generates intermediate con- tent as the rationale based on visual information and then uses the generated rationale to induce the result. However, both methods may have difficul- ties understanding certain types of images (e.g.. maps), which could be mitigated by retrieving in- formative image-text pairs.
LLM augmentation Wang et al. (2022) attempts to augment an LLM to generalize to various video- to-text tasks from a few examples. As the LLMs cannot accept video inputs, it first translates video contents into attributes using image-language mod- els and then prompts the retrieved content to in- struct the LLM. It demonstrates good few-shot per- formances on wide range of video-language tasks. Currently, the video-text research bottleneck mainly lies in the representation gap between dif- ferent modalities. Research has been attempting to learn a better mapping between video-text via joint learning (Xu et al., 2015; Sun et al., 2019). Recent studies dense video representation learn- ing can also be useful for future video RAG. Be- sides, some papers (Yang et al., 2023a; Wang et al., 2021a) try to introduce fine-grained interaction be- tween different modalities to learn better aligned representations. Zeng et al. (2022) encourages mul- tiple pretrained models in different modalities to exchange information with each other in a zero- shot manner. Most recently, Zhang et al. (2023a) trains Video-Llama better align pretrained video and audio encoders with LLM's embedding space.
4.2 Building a Multimodal Knowledge Index
In order facilitate multimodal RAG, one of the most fundamental aspects is building a multimodal knowledge index. The goal is twofold: Firstly, dense representations should support low storage, dynamic updating of the knowledge base, and ac- curate search. Secondly, it could enable faster search speed with the help of local sensitive hash- ing (Leskovec et al., 2014), which combats scaling and robustness concerns when the knowledge base is scaled up extremely.
Currently, the dense representations for text snip- pets are widely studied for documents (Karpukhin et al., 2020b; Gao and Callan, 2021; Gao et al., 2021), entities (Sciavolino et al., 2021; Lee et al., 2021a), and images (Radford et al., 2021a). Be- sides, there are studies optimizing dense representa- tions in an end-to-end manner (Lewis et al., 2020). Nevertheless, few papers (Chen et al., 2022a) have explored building a multimodal index at the same time for downstream generation tasks. How to map multimodal knowledge index into unified space remains long-term challenge.
4 Future Directions
With the development of multi-modal LLMs,, re- trieving multimodal information augment text generation will be a promising direction to better ground textual generation in real-world contexts, contributing towards building a model that is fully aware and can better interact with the world. Specif- ically, we describe some potential directions that can be of benefit to the community.
4.3 Pretraining with Multimodal Retrieval
To better align the abilities to handle different modalities in a pre-trained model, future work could be built on employing retrieval-based ap- proaches during pre-training. Currently, some methods fine-tune the pre-trained generative model to learn to retrieve from different modalities. For example, LaMDA (Thoppilan et al., 2022) calls an external toolset for fine-tuning, including an in- formation retrieval system. Similarly, during fine- tuning, Toolformer (Schick et al., 2023) augments models with API calls to tools including a QA sys- tem and Wikipedia search engine.
4.1 Retrieval Augmented Multimodal Reasoning
One potential application of multimodal RAG is multimodal reasoning. Lu et al. (2022a) first in- troduces ScienceQA, large-scale multimodal sci- ence question dataset annotated with lectures and explanations. Then, Zhang et al. (2023b) proposes Multimodal Mlain-of-Thougee (Multimodal-Coff) which incorporates language and vision modalities
When similar retrieval abilitics are leveraged dur- ing pretraining, the generative models can interact

---

with retrieval tools much better. Then, instead of relying solely on internal weights, they could effec- tively use an external base to output more grounded information, provide relevant contexts to users, and update their information accordingly. Such pre- training techniques would also greatly improve ro- bustness for out-of-domain tasks. As an example, Guu et al. (2020a) augments pretraining with an external knowledge retriever, which outperforms previous methods.
Dinesh Garg. 2021. Explanations for Common- senseQA: New Dataset and Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3050-3065, Online. Association for Computational Linguisties.
Armen Aghajanyan. Bernie Huang. Candace Ross, Vladimir Karpukhin. Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. CM3: causal masked multimodal model of the internet. CoRR. tbs/2201.075200
To incorporate retrieval with pretraining, there remains the challenge of developing appropriate datascts labeled with retrieval API calls. To tackle this challenge, LaMDA (Thoppilan et al., 2022) uses labels developed by human annotators, which could be expensive collect. Toolformer (Schick al., 2023) uses a sampling and filtering approach for automatic labeling, which is inexpensive but could induce bias. A potential solution is to use a neuro-symbolic approach (Davoudi and Komeili, 2021), which uses prototype learning and deep- KNN to find nearest neighbors during training.
Renat Aksitov, Chung-Ching Chang, David Reitter, Sia- mak Shakeri, and Yunhsuan Sung. 2023, Character- izing attribution and fluency tradeoffs for retrieval- augmented large language models.
Jcan-Bapting Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022a. Flamingo: visual language model for few-shot learning. In Advances Neural Information Processing Systems.
Jean- Bran-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, An- drew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 20226. Flamingo: a visual language model for few-shot learning. CoRR, abs/2204. 14198.
5 Conclusions
This survey reviews research that augments gen- erative models by retrieving multi-modal informa- tion. Specifically, we categorize the current domain into enhancing with different modalities, including image, code, structured knowledge, speech, and video. With the emergence of large multi-modal models, we believe that this survey could serve as comprehensive overview of an emerging and promising field. Moreover, we hope it could en- courage future research in the domain, including multimodal reasoning, build- ing multi-modal knowledge index, and combining retrieval with pretraining.
Jinheon Back, Alham Fikri Aji, and Amir Saffari. 2023, Knowledge-augmente language model prompting for zero-shot knowledge graph question answering. arXiv preprint 2306622306.04111
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis Douis-Philippe Morency, 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-444 443.
6 Limitations
RAG also has some limitations. For example, there exists attribution-flueccy tradeoff (Aksitov et al.. 2023) where the output quality is affected due to the added constraints of the retrieved knowledge.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240, PMLR.
References
Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. 2022. Multimodal biomedical ai. Nature Medicine, 1773-1777-
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 197-1
Shourya Aggarwal, Divyanshu Mandowara, Vishwa- jeet Agrawal, Dinesh Khandelwal, Parag Singla, and

---

Deng Cai, Yan Wang. Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2019a. Skeleton- to-response: Dialogue generation guided by retrieval memory. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 1219-1228, Minneapolis, Minnesota. Association for Computational Linguistics.
Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang. Zhanming Jie, Wei Lu, and Luo Si. 2020. ENT- DESC: Entity description generation by exploring knowledge graph. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1187-1197, Online. As- sociation for Computational Linguisties.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer. Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages. ICLR.
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiao- jiang Liu, and Shuming Shi. 2019b. Retrieval- guided dialogue response generation via matching- to-generation framework. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNILP), pages 1866-1875, Hong Kong, China. Association for Computational Linguisties.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. 2022. Fine-grained image captioning with CLIP reward. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 517-527, Seattle, United States. Association for Computational Lin- guistics.
David M Chan, Shalini Ghosh, Ariya Rastrow, and Bjöm Hoffmeister. 2023. Using external off off-policy speech-to-text mappings in contextual end-to-end automated speech recognition. arXiv preprint arXiv:230.002336.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233:
Chieh- Yang Chen, Pei-Hsin Wang, Shih-Chich Chang, Da-Cheng Juan, Wei Wei, and Jia- Yu Pan. 2020. Air- Concierge:
Generating task-oriented dialogue via efficient large-scale knowledge retrieval. In Find- ings the the Association for Computational Linguistics: EMNLP 2020, pages 884 897, Online. Association for Computational I Linguistics.
Colin B Clement, Shuai Lu, Xiaoyu Liu, Michele Tu- fano, Dawn Drain, Nan Duan, Neel Sundaresan, and Alexey Svyatkovskiy. 2021. Long-range mod- eling of source code files with ewash: Extended window access by syntax hierarchy, arXiv preprint vxXiv:210900878)
Mark Chen, Jemry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022. Vggan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision-FCCC 2022: 17th European Conference, Tel Aviv, Israel, October 23- 27, 2022, Proceedings. Part XXXVII, pages 88-105. Springer.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. 2022a. MuRAG: Multimodal regrenell-uugmented generator for open question an- swering over images and text. In EMNLP, pages 5558-5570. ACL.
Seyed Omid Davoudi and Majid Komeili. 2021. To- ward faithful case-based reasoning through learning prototypes in a nearest neighbor-friemt space. International Conference on Learning Representa- tions.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William Cohen. 2022b. MuRAG: Multimodal regrieeal-augmentedd generator for open question an- swering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 5558-5570, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly mod- eling in-file and cross file context. arXiv preprint arXiv:2212.1 10007.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. 2022c. Re-imagen: Retrieval- augmented text-to-image generator. arXiv preprint arXiv:2209. 14491.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang. Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. 2022. Coarse-to-fine vision-language pre-training with fusion in the backbone. arXiv preprint 200
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022d. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks, arXiv preprint arXiv:2211.122 125888

---

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch. Aakanksha Chowdhery, Brian lchter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint wrXiv:2303.03376
Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022a. Pal: Program-aided language models. arXiv preprint arXiv:2211, 100445.5.
Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021. Neural path hunter: Re- ducing hallucination in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 2197-2214, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, and Antoine Bosselut. 2022b. Com- Fact: A benchmark for linking contextual common- sense knowledge. In Findings of the Association for Computational Linguisticy: EMNLP 2022, pages 1656-1675, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Alex Falcon, Giuseppe Serra, and Oswald Lanz. 2022. A feature-space multimodal data augmentation tech- nique for text-video retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4385-4394.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes 2019, Using local knowledge graph con- struction to scale seq2seq models to multi-document inputs. arXiv preprint arXiv: Stti::9900004333
Jia- Chen Gu, Zhenhua Ling, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2020. Filtering before iteratively response selection
referring for knowlcdge-grouned in retrieval-based chatbots. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 1412-1422, Online. Association for Computa- tional Linguisties.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2021. Augmenting transformers with KNN- based composite memory for dialog. Transactions of the Association for Computational Linguistics, 9:82- 99.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine trans- lation. In AAAI, volume 32.
Qingkai Fang and Yang Feng, 2022. Neural machine translation with phrase-level universal visual repre- sentations. Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 5687-5698, Dublin, Ireland. Association for Computational Linguisties.
Yunfan Gu, Yang Yuqiao, and Zhongyu Wei. 2019. Ex- tract, transform and filling: A pipeline model for question paraphrasing based template. In Proceed- ings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 109-114, Hong Kong. China. Association for Computational Linguistics.
Giacomo Frisoni, Miki Mizutani, Gianluca Moro, and Lorenzo Valgimigli. 2022. BioReader; a retrieval- enhanced text-to-text transformer for biomedical lit- erature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5770-5793, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Liangke Gui, Borui Wang, Qiuyuan Huang, Alexan- der Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-andllanguage. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies, pages 956-968, Seattle, United States. Association for Computational Lin- guistics.
Fabian Galetzka, Jewgeni Rose, David Schlangen, and Jens Lehmann. 2021. Space efficient context encod- ing for non-task-onn triented dialogue generation with graph attention transformer. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Pupers). pages 7028-7041, Online. As- sociation for Computational Linguistics.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross- modal pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7212-7225. Association for Computa- tional Linguistics.
Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang, 2020. A survey on deep learning for multimodal data fusion. Neural Computation, (25588829-
Luyu Gao and Jamie Callan, 2021. Condenser: pre- training architecture for dense retrieval. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981 991,
Vishal Gupta, Manoj Chinnakotla, and Manish Shri- vastava. 2018. Retrieve and re-rank: A simple and effective IR approach to simple question answer- ing over knowledge graphs. In Proceedings of the

---

First Workshop on Fact Extraction and VERification (FEVER), pages 22-27, Brussels, Belgium. Associa- tion for Computational Linguisties.
question answering. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9562-9581, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Wei Wei Chang. 2020a. Realm: Retrieval- augmented language model pre-training.
Rongjie Huang. Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xi- ang Yin, and Zhou Zhao. 2023a. Make-an-audio: Text-to-audio generation with nompt-enhanced dif- fusion models. arXiv preprint arXiv:2301. 2et.2.201...6666 12661.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020b. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
Shaohan Huang. Li Dong. Wenhui Wang, Yaru Hao. Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023b. Language is not all you need: Aligning perception with language models. arXiv preprint WxXv:2300.14445.
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. NeurIPS, 31.
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Av- varu, Pengcheng Yin, Authony Tomasic, and Graham Neubig. 2018. Retrieval-based neural code gener- ation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 925-930, Brussels, Belgium. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation Computational Linguistics: Main Volume. pages 874- 974,880, Online. Association for Computa- tional Linguistics.
Hangfeng Hc, Hongming Zhang, and Dan Roth. 2022a. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.06 .00303.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni. Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sehastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv. 2208.
Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021. Fast and accurate neural machine Proceedings
translation with translation memory. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1170-3880.
Peter Jansen and Dmitry Ustalov. 2019. TextGraphs 2019 shared task on multi-hop inference for expla- nation regeneration. In Proceedings of the Thir- teenth Workshop on Graph-Based Based Methods for Nat- ural Language Processing (TexGraphs- 13), pages 63-77, Hong Kong. Association for Computational Linguistics.
Zihao He, Weituo Hao, and Xuchen Song. 2022b. Re- cap: Retrieval augmented music captioner. arXiv preprint 22Xi::2212. 10901.
Tao Hu, Xuyu Xiang, Jiaohua Qin, and Yun Tan. 2022a. Audio-text retrieval based on contrastive learning and collaborative attention mechanism.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pages 4904-4916. PMLR.
Xixin Hu, Xuan Wu, Yiheng Shu, and Yuzhong Qu 2022b. Logical form generation via multi-task learn- ing for complex question answering over knowledge bases. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 1687- 1696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018. Shaping program repair space with existing patches and similar code. In ISSTA, pages 298-309. ACM.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023. Reveal: Retrieval- augmented visual-language pre-training with multi- source multimodal knowledge memory. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 33369-33379
Bowen Jin, Yu Zhang, Qi Zhu, and Jiawei Han. 2022. Heterformer: A transformer architecture for node representation learning on heterogeneous text-rich networks. arXiv preprint arXiv:2205.1000
Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. 2022c. Empowering language models with knowledge graph reasoning for open-domain
Matthew Jin, Syed Shahriar, Michele Tufamo, Xin
Shi, Shuai Lu, Neel Sundaresan, and Alexey Svy- atkovskiy. 2023. Inferfix: End-to-end program repair with Ims. arXiv preprint arXiv:2303.0726.

---

Yohan Jo. Haneul Yoo, Jin Yenon Bak, Alice Oh, Chris Reed, and Eduard Hovy. 2021. Knowledge-enhaneed evidence retrieval for counterargument generation. In Findings of the Association for Computational Lin- guisticx: EMNLP 2021, pages 3074-3094, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Hung Le, Nancy Chen, and Steven Hoi. 2022. Vgnmn: Video-grounde' neural module networks for video- grounded dialogue systems. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3377-3393.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2020. BiST: Bi-directional spatio-temporal rea- soning for video-grounded dialogues. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1846-1859, Online. Association for Computational Linguistics.
Harshit Joshi, José Cambronero, Sumit Gulwani, Vu Le, Ivan Radicek, and Gust Verbruggen. 2022. Repair is nearly generation: Multilingual program repair with lims. arXiv preprint arXiv:2208. 1640.
Chen Ju, Tengda Ham, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022. Prompting disual-languae mod- els for efficient video understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXV. pages 105-124, Springer.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021a. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 6634-6647, Online. Association for Computational Linguistics.
Jaehun Jung, Bokyung Son, and Sungwon Lyu. 2020. AttIO: Knowledge Graph Exploration with In-and- Out Attention Flow for Knowledge- aoowledgeGrouunded Dia- logue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3484-3497, Online. Association for Computational Linguistics.
Nyoungwoo Lee, Suwon Shin, Jaegul Choo, Ho-Jin Choi, and Sung-Hyon Myaeng. 2021b. Construct- ing multi-modal dialogue dataset by replacing text with semantically relevant images. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 897-906, Online. Association for Computational Linguisties.
Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. 2023. Knowledge graph-augmented language models for noowledge-gouundee dialogue generation.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:200..04906.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and- lan-lage learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 7331-7341.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Eduuno,, Danqi Chen, and Wen-tau Yih. 2020b. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. 2020. TVQA+: Spatio-temporal grounding for video question answering. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisticss pages 821 1-8225, Online. Association for Computational Linguistics.
Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ull- man. 2014, Mining of Massive Datasets, 2nd Ed. Cambridge University Press.
Minkyu Kim. Kim Sung-Bin, and Tae-Hyun Oh. 2023. Prefix tuning for automated audio captioning. ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. IEEE
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmentee generation for mnoweedge-ittensin nlp tasks. Advances in Neu- ral Information Processing Systems, 0049459-94722
A Sophia Koepke, Andreea-Marte Oncescu, Joao Hen- riques. Zeynep Akata, and Samuel Albanie. 2022. Audio retrieval with natural language queries: benchmark study. IEEE Transactions on Multimedia.
Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019, Knowledge- driven encode, retrieve, paraphrase for medical image report generation.
Yuma Koizumi, Yasunori Ohishi, Daisuke Nizzumi, Daiki Takeuchi, and Masahiro Yasuda. 2020. Au- dio captioning using pre-trained large-scale language model guided by audio-based similar caption re- trieval. arxiv preprint 202.v.
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022a. A survey on vetrieval-augmenter text generation. arXiv preprint arXiv:2202.01113

---

Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023a. 2: Bootstrapping tagguageimmgg pre- training with frozen image encoders and large lan- guage models. arXiv preprint (rXiv22301.15599
Qi Liu, Dani Yogatama, and Phil Blunsom. 2022a. Rela- tional memory memory-augmentee augmented language models. Trans- actions of the Association for Computational Linguis- rics, 10:555-572.
Miaoran Li, Baolin Peng, Jianfeng Gao, and Zhu Zhang. 2022b. Opera: Harmonizing task-oriented dialogs and information seeking experience. arXiv preprint arXiv:2206. 12449.
Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-auge aatreeeel-ugmeeeted generation for code summarization via hybrid GNN. In ICLR.
Liu, Kuan- Yu Chen, Berlin Chen, Hsin-Min Wang, Hsu-Chun Yen, and Wen-Lian Hsu. 2015. Combining relevance language modeling and clar- ity measure for extractive speech summarization. IEFF/ACM Transactions on Audio, Speech, and Lan- guage Processing, 236))957-96..
Xingxuan Li, Ruochen Zhao. Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Po- ria. 2023b. Chain of knowledge; A framework for grounding large language models with structured knowledge bases.
Xuliang Liu and Hao Zhong. 2018. Mining stackover- flow for program repair. In SANER, pages 118-129. IEEE Computer Society.
Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2022c. Knowledge- grounded dialogue generation with a unified knowl- edge representation. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 206-218, 218, Seattle, United States. Association for Computational Linguistics.
Ye Liu, Semih Yavuz, Rui Meng. Dragomir Radev, Caiming Xiong. and Yingbo Zhou. 2022b. Uni- parser: Unified semantic parser for question answer- ing on knowledge base and database. In Proceedings of the 2022 Conference on Empirical Methods in Nut- ural Language Processing, pages 8858-8869 8869, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder. Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundare- san. 2022d. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint Furopean Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pages 1035-1047. ACM.
Xiang Long, Chuang Gan, and Gerard De Melo. 2018. Video captioning with multi-faceted attention. Trans- actions of the Association for Computational Linguis- tics, 6:173-184.
Siyu Lou, Xuenan Xu, Mengyue Wu, and Kai Yu. 2022. Audio-text retrieval in context. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4793- 4797. IEEE
Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xi- ubo Geng. Yining Chen, Fan Liang, and Daxin Jiang. 2021. Maria: A visual experience powered conver- sational agent. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1ith International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 5596-5611, Online. Association for Computational Linguistics.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022a. Learn explain: Multi- modal reasoning via thought chains for science ques- tion answering, arXiv preprint arXiv:2209.098 03
Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung- won Hwang, and Alexey Svyatkovskiy. 2022b. ReACC: A retrieval autrivallauggme -ugmaattee code completion framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Weizhe Lin and Bill Byrne 2022a. Retrieval augmented visual question answering with outside knowledge. In EMNLP, pages 11254. Association for Computational Linguistics.
1: Long Papers), pages 6227-6240, Dublin, Ireland. Association for Computational Linguistics.
Weizhe Lin and Bill Byme. 2022b. Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1 11238-1 11254, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xing Han Lu, Siva Reddy, and Harm de Vries. 2023. The StatCan dialogue dataset: Retrieving data ta- bles through conversations with genuine intents. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 2799-2829, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Weizhe Lin, Zhilin Wang, and Bill Byme. 2023. FVQA 2.0: Introducing adversarial samples into based visual question answering. In Findings of the Asso- ciation for Computational Linguistics: EACL 2023, pages 149-157, Dubrovnik, Croatia. Association for Computational Linguistics.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Buree 2023. Faithful chain-of- thought reasoning. arXiv preprint urXiv:2301.13379

---

Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022. Open-domale question an- swering via chain of reasoning over heterogeneous knowledge. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 5360- 5374, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Stefanos Vrochidis, and Leo Wanner. 2020. A case study of NLG from multimedia data sources: Gen- erating architectural landmark descriptions. In Pro- ceedings of the 3rd International Workshop on Nat- ural Language Generation from the Semantic Web (WebNI.G+), ). pages 2-14, Dublin, Ireland (Virtual). Association for Computational Linguisties.
Aman Madaan, Shuyan Zhou, Uni Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. arXiv preprint arXiv:2210.07128.
Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K noy-Chowdhury. 2018. Learn- ing joint embedding with multimodal cues for cross- modal video-text retrieval. In Proceedings of the 2018 ACM on International Conference on Multime- dia Retrieval, pages 19 27.
Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. 2021. Muscaps: Generating cap- tions for music audio. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8.
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapani- pathi, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. 2021. Efficient text-based rein- forcement learning by jointly leveraging state and commonsense graph representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1thh International Joint Conference on Natural Language Processing ( (Volume 2: Short Pupers), pages 719-725, Online. Association for Computational Linguistics.
IBEE.
Matias Martinez, Westley Weimer, and Monperrus Mar- tin. 2014. Do the fix ingredients already exist? an empirical inquiry into the redundancy assumptions of program repair approaches. Companion Proceedings of the 36th International Conference on Software Engineering.
Steve McConnell. 2004. Code complete. Pearson Edu- cation.
Arsha Nagrani, Shan Yang, Amurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention bot- tlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34: 1200-112213
Rafael Mestre, Stuart Middleton, Matt Ryan, Masood Gheasi, Timothy Norman, and Jiatong Zhu. 2023. Augmenting pre-trained language models with audio feature embedding for argumentation mining in polit- ical debates. In Findingy of the Association for Com- putatioonll Linguistics: EACI. 2023, pages 274-288, Dubrovnik, Croatia. Association for Computational Linguistics.
Kai Nakamura, Sharon Levy, Yi-Lin Tuan, Wenhu Chen, and William Yang Wang. 2022. HybriDialogue: An information-seckinn
dialogue dataset grounded on tabular and textual data. In Findings of the Associa- tion for Computational Linguistics: ACI. 2022, pages 481-492, Dublin, Ireland. Association for Computa- tional Linguistics.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.078:4
Reicchir Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332.
Benjamin Milde, Jonas Wacker, Stefan Radomski. Max Mühlhäuser, and Chris Biemann. 201 Ambi- ent search: A document retrieval system for speech streams. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 2082-2091, Osaka, Japan. The COLING 2016 Organizing Committee.
I Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Krysciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang. Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham. Caiming Xiong. Dragomir Radev, and Dragomir Radev. 2022. FeTaQA: Free-form table question answering. Trans- actions of the Association for Computational Linguis- rics, 10:35-49.
Benjamin Milde, Jonas Wacker, Stefan Radomski, Max Mühlhäuser, and Chris Biemann. 2016b. Demon- strating ambient search: Implicit document retrieval for speech streams. In Proceedings of COLING 2016. the 26th International Conference on Computational Linguistics: System Demonstrations, pages 233-237 Osaka, Japan. The COLING 2016 Organizing Com- mittee
Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-basee prompt selection for code-related few-shot learning. In Proceedings of the 45th In- ternational Conference on Software Engineering (ICSE'23).
Simon Mille, Spyridon Symeonidis, Maria Rousi, Montserrat Marimon Felipe, Klearchos Stavrothana- sopoulos, Petros Alvanitopoulos, Roberto Car- lini Salguero, Jens Grivolla, Georgios Meditskos,
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong. Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, and Dan

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya Sutskever. 2021a. la. Lear- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Roth. 2022. Entailment tree explanations via itera- tive retrieval-gentt -eneeatilgenenttionn reasoner. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 465-475, Scattle, United States, Associ- ation for Computational Linguistics.
OpenAL. 2023. Gpt-4 technical report.
PMLR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PMLR.
Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and Peter Fox. 2021. CLTR: An end-to-end, transformer-base system for cell-level table retrieval and table question answering. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing: System Demonstrations, pages 202-209, Online. Association for Computational Linguistics.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray. Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 202 Zero-shot text-to-image gen- eration. In ICML., volume 139 of Proceedings of Ma- chine Learning Research, pages 8821-882 -831. PMLR.
Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 2719-2734, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray. Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021b. Zero-shot text-to-image gen- eration. In International Conference on Machine Learning, pages 8821 18811, PMLR.
Kiran Ramnath, Leda Sari, Mark Hosegawa-Johnson, and Chang Yoo. 2021. Worldly wise (WoW)- cross-lingual knowledge fusion for fact-based visual answering. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1908-1919, On- line. Association for Computational Linguistics.
Ramakanth Pasumuru and Mohit Bansal. 2018. Game- based video-context dialogue. arXiv preprint 180,00000
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint ar3iv:2003.07191.
Rita Ramos, Desmond Elliott, and Bruno Martins. 2023. Retrieval- -ugmentd image captioning. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics. pages 3666-366 1, Dubrovnik, Croatia. Association for Computational Linguistics.
Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-base based adaptive decoding. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2555-2565, Min- neapolis. Minnesota, Association for Computational Linguistics.
Brandon Royal. Kien Hua, and Brenton Zhang. 2020. Deep composer: Deep neural hashing and retrieval approach to automatic music generation. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2022. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988.
Sara Sarto, Marcella Comia, Lorenzo Baraldi, and Rita Cucchiara 2022. Retrieval-augmenned transformer for image captioning. In CBMI, pages 1-7. ACM.
Soumajit Pramanik, Jesujoba Alabi, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Uniqorn: unified ques- tion answering over rdf knowledge graphs and natural language text. arXiv preprint arXiv:2 1et.:220880.6144
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint 2302.0000
Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014. The strength of random search on automated program repair. In ICSE, pages 254-265. ACM.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of

---

the 2021 Conference on Empirical Methods in Natu- val Language Processing, pages 6138-66 6148, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
guage Technologies. pages 1405-1418, Online. As- sociation for Computational Linguistics.
Chao-Hong Tan, Jia-Chen Gu, Chongyang Tao, Zhen- Hua Ling, Can Xu. Huang Hu, Xiubo Geng, and Daxin Jiang. 2022. TegT Augmenting text gen- eration via task-specific and open-world knowledge. In Findings of the Association for Computational Linguisties: ACL 2022. pages 1597-1609, Dublin, Ireland, Association for Computational Linguistics.
Ramprasaath R. Selvaraju, Michael Cogswell, Ab- hishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explana- tions from deep networks via gradient-hased local- ization. In ICCV, pages 618-626, 626. IEEE Computer Society.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 2201::2200.00233
Lei Shen, Haolan Zhan, Xin Shen, Yonghao Song. and Xiaofang Zhao. 2021. Text is not enough: Integrat- ing visual impressions into open-domain dialogue generation. In Proceedings of the 29th ACM Inter- national Conference on Multimedia, MM '21, page 4287-4296, New York, NY, USA. Association for Computing Machinery.
Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Sil- vio Savarese, and Steven C.H. Hoi. 2022. Plug-and- play VQA: Zero-shot VQA by conjoining large pre- trained models with zero training. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 951 Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Ensheng Shi, Yanlin Wang. Wei Tao, Lun Du, Hongyu Zhang. Shi Han, Dongmei Zhang, and Hongbin Sun. 2022. RACE: Retrieval-augmeeteed commit message generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pages 5520-5530, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interlcav- ing retrieval with chain-of-thought reasoning for nnowledge-intensiig multi-step questions. arXiv preprint arXiv:2212. 10500
Zhan Shi, Hui Liu, Martin Renqiang Min, Christopher Malon, Li Erran Li, and Xiaodan Zhu. 2021. Re-
trieval, analogy, and composition: A framework for compositional generalization in image captioning. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1990-2000, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Pro- ceedings of the conference. Association for Computa- tional Linguistics. Meeting, volume 2019, page 6558. NIH Public Access.
Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma, Yuzhong Qu, and Chin-Yer Lin. 2022. TIARA: Multi culti-grained retrieval for robust question answering over large knowledge base. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8108-812 8121, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Xiaohan Wang, Linchao Zhu, and Yi Yang. 2021a. T2VLAD: global-local sequence alignment for text- video retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pages 5079-5088. Computer Vi- sion Foundation / IEEE.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021b. CodeT5: Identifier-awar unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pages 6696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yixuan Su, Zaiqiao Meng. Simon Baker, and Nigel Col- lier. 2021. Few-shot table-to-text generation with prototype memory. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 910-917, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Mur- phy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learn- ing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473.
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. 2022. Language models with image descriptors are strong few-shot aadoo-language learners. arXiv preprint 2rXiv:2205.1744
Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, and Nancy X. R. Wang. 2021. D2S: Document-to-slide generation via query-hased text summarization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.

---

Makarius Wenzel, Lawrence Paulson, and Tobias Nipkow, 2008. The isabelle framework. In TPHOLs, volume 5170 of LNCS, pages 33-38. Springer.
Jia Xin, Wang Hao, Yin Dawei, and Wu Yunfang, 2021. Enhancing question generation with commonsense knowledge. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 976-987, Huhhot, China. Chinese Information Processing Society of China.
Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence gener- ation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd Interna- tional Workshop on Search-Oriented Conversational AI. pages 87-92, Brussels, Belgium. Association for Computational Linguistics.
Jitao Xu, Josep-Maria Crego, and Jean Senellart. 2020. Boosting neural machine translation with similar translations. In Annual Meeting of the Association for Computational Linguistics, pages 1570-1579. As- sociation for Computational Linguistics.
Martin White, Michele Tufano, Matias Martinez, Mon- perrus Martin, and Denys Poshyvanyk. 2019. Sorting and transforming program repair ingredients via deep learning code similarities. 2019 IEEE 26th Interna- tional Conference on Software Analysis, Evolution and Reengineering (SANER), pages 490-
Ran Xu, Caiming Xiong, Wei Chen, and Jason Corso. 2015. Jointly modeling deep video and composi- tional text to bridge vision and language in a unified framework. In AAAI, volume 29.
Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu. Michael Zeng, and Xuedong Huang. 2021. Fus- ing context into knowledge graph for commonsense question answering. In Findings of the Association for Computational Linguistics: ACI-IJCNLPP 2021, pages 1201-1207, Online. Association for Computa- tional Linguistics.
Spencer Whitehead, Heng Ji, Mohit Bansal, Shih-Fu Chang, and Clare Voss. 2018. Incorporating back- ground knowledge into video description generation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3992-4001, Brussels. Belgium. Association for Com- putational Linguistics.
Fabian Yamaguchi, Nico Golde, Daniel Arp, and Kon- rad Rieck. 2014. Modeling and discovering vulner- abilities with code property graphs. In 2014 IEEF Symposium on Security and Privacy, SP 2014, Berke- ley, CA, USA, May 18-21, 2014, pages 590-604. IEEE Computer Society.
Sixing Wu, Ying Li, Dawei Zhang, and Zhonghai Wu. 2020a. Improving knowledge-aware dialogue re- sponse generation by using human-written prototype dialogues. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 1402- 1411, Online. Association for Computational Linguis- tics.
Antoine Yang, Arsha Nagrami, Paul Hongsuck Sco, An- toine Miech, Jordi Pont-Tuset Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023a. Vid2seq: Large- scale pretraining of a visual language model for dense video captioning. CoRR, abs/2302.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020b. Diverse and informative dia- logue generation with cnttext-ppecffc commonsense knowledge awareness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5811-5820, Online. Association for Computational Linguistics.
Xingyi Yang. Muchao Ye, Quanzeng You, and Feng- long Ma. 2021. Writing by memorizing: Hierar- chical retrieval- based medical report generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 5000- 5009, Online. Association for Computational Lin- guistics.
Xian Wu, Shuxin Yang, Zhaopeng Qiu, Shen Ge, Yang- tian Yan, Xingwang Wu, Yefeng Zheng, S. Kevin Zhou, and Xiao. 2022a. DeltaNet: Conditional medical report generation for COVID-19 diagno- sis. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 2952- 2961, Gyeongju. Republic of Korea. International Committee on Computational Linguistics.
Yue Yang. Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, and Jianshu Chen. 2022a. V-L Zero-shot language solver fueled by visual imagi- nation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1 1186-1203 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhou- jun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In AAAI, vol- ume 33, pages 7281-7288.
Yuhuai Wu, Albert Qiaochu Jiang. Wenda Li, Markus Norman Rabe, Charles E Staats, Mateja Jam- nik, and Christian Szegedy. 2022b. Autoformaliza- tion with large language models. In NewrIPS.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022b. An empirical study of gpt-3 for few-shot knowledge- based vqa. In AAAI, volume 36, pages 3081-3088
Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E. Hassan, and Zhenchang Xing. 2017. What do developers search for on the web? Empir Softw: Eng., 22(6): 11493335
Zhicheng Yang, Jinghui Qin, Jiaqi Chen, Liang Lin, and Xiaodan Liang. 2022c. LogicSolver: Towards interpretable math word problem solving with log- ical prompt-enhanced learning. In Findings of the

---

Association for Computational Linguistics: EMNLP 2022, pages 1-13, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
ACM/IEEE 42nd International Conference on Soft- ware Engineering. ICSE '20, page 1385-1397, New York, NY, USA. Association for Computing Machin- ery.
Zhuolin Yang, Wei Ping. Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang. Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Ming- Yu Liu, Yuke Zhu, Moham- mad Shoeybi, Bryan Catanzaro, Chaowei Xiao, and Anima Anandkumar. 2023b. Re-vilm: Retrieval- augmented visual language model for zero and few- shot image captioning. CoRR, abs/2302.04858.
Jingyi Zhang. Masao Utiyama, Eiichro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, Volume I (Long Papers), pages 1325-1335, New Orleans, Louisiana. Association for Computa- tional Linguisties.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang. Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2022. Retrieval- augmented multimodal language modeling. CoRR, abs/2211.12561.
Jun Zhang, Yan Yang, Chencai Chen, Liang He, and Zhou Yu. 2021. KERS: A knowledge-enhanced framework for recommendation dialog systems with multiple subgoals. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1092-1101, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Xi Ye and Greg Durrett. 2022. The unreliability of ex- planations in few-shot prompting for textual reason- ing. In Advances in Neural Information Processing Systems.
Yunhu Ye, Binyuan Hui. Min Yang. Binhua Li, Fei
Zhuosheng Zhang, Aston Zhang. Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023b. Multi- modal chain-of-thought reasoning in language mod- els. arXiv preprint arXiv:2302.00923.
Huang, and Yongbin Li. 2023. Large language models are versatile decomposers: Decompose evi- dence and questions for table- based reasoning. arXiv preprint arXiv:2301.
Jinming Zhao, Gholamreza Haffari, and Ehsan Shareghi. 2023a. Generating synthetic speech from Spoken Vo- cab for speech translation. In Findings of the Asso- ciation for Computational Linguistics: FACI. 2023, pages 1975-1981, Dubrovnik, Croatia. Association for Computational Linguistics.
Jiahui Yu. Yuanzhong Xu. Jing Yu Koh, Thang Lu- ong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson. Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022. Scaling autoregressive models for content-rich text- to-image generation. CoRR, abs/2206.1077 10789.
Ruochen Zhao, Xingxuan Li, Yew Ken Chia, Bosheng Ding, and Lidong Bing. 2023b. Can chatgpt-like generative models guarantee factual accuracy? on the mistakes of new generation search engines.
Xiaojing Yu and Anxiao Jiang. 2021. Expanding, re- trieving and infilling: Diversifying cross-domain question generation with flexible templates. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisticy: Main Volume, pages 3202-3212, Online. Association for Computational Linguisties.
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023c. Verify- A knowledge-enhanced chain-of-thought framework.
Luowei Zhou, Hamid Palangi. Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision- language pre-training for image captioning and vqu. In AAAI, volume 34, pages I 13041-13049.
Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, and Songfang Huang. 2023. biomedical visual ques-
RAMM: retrivvll-augmente tion answering with multi-modal pre-training. CoRR, abs/2303.0055 0003444
Mingyang Zhou, Grace Luo, Anna Rohrbach, and Zhou Yu. 2022a. Focus! relevant and sufficient context lection for news image captioning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. 2022. Socratic models: Compos- ing shot multimodal reasoning with language. arXiv preprint arXiv:2204.00:: 005988
Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022b. Docprompting: Generating code by retrieving the docs. arXiv preprini arXiv:2207.0555 009977.
Hang Zhang. Xin Li, and Lidong Bing. 2023a. Video- Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv.2306.0285..
Yucheng Zhou and Guodong Long, 2023. Style- aware contrastive learning for multi-style image captioning. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 2257-2267. Dubrovnik, Croatia. Association for Computational Linguistics.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based neural source code summarization. In Proceedings of the

---

Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text gen- cration. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLPP) pages 5459-5468, Hong Kong, China. Association for Com- putational Linguistics.
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Wang. Miguel Eckstein, and William Yang Wang. 2023. Vi- sualize before you write: Imagination-guidee ended text generation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 78-92. Dubrovnik, Croatia. Association for Compu- tational Linguistics.
open-

---

Modality
ACL
Google
Total analyzed
recently, with peaks reached around end of 2022. The observation is consistent with our hypothesis that multimodal RAG is especially important and helpful the age of large-scale models.
Image
(67) 17
6
23
Code
(177)
24
33
Structured
(108) 44
11
55
Audio
(17) 6
14
20
Video
(22)
7
14
Total (291)
62
145
Table I: Paper statistics. Number in parenthesis is the number before manual filtering. "Google" represents searching on google scholar and manually filtering. "To- tal analyzed" represents the number of total papers after manual filtering
Trend of Article Count
10
200400
201
2022
20239
She
Figure I: Paper trend analysis
A
Appendix
A.1
Search Criteria and Results
For searching the ACL anthology articles, we use a keyword search over titles and abstracts. We strictly enforce the keyword "retriev". Then, we enforce either "generat" or "ground"" to appear. For each modality, we then add mopality-specific key- words: "image" for the image modality, "code" for the code modality, any one from "structured www.lllgglahee/aabaaaektttoooooe
graph" for the structured knowledge modality, any one from "audio/speech" for the audio modality, and "vidco" for the video modality.
For searching Google Scholar, add the keyword "language models" select more NLP- related articles. We then perform manual filtering on the top 3 pages returned results.
The number of retrieved and analyzed research papers can be found in Table 1.
A trend analysis of how the number of papers change across time is shown in Figure We could observe that the domain of multimodal retrieval- augmented generation has indeed developed a lot


Reviews:
[{"review_id": "Zz37fqPort", "reviewer": "Reviewer_4Ruw", "summary": "", "questions": "See weakness above. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review. We reply to your concerns here:\n\n- As the video modality has a few RAG studies, we include some papers (e.g. L663-666 and L687-695) that share similar motivations because their tasks and methods could inspire future work in this direction. We now recognize the issue that it may confound our readers and we will adjust these works by clearly pointing out their position and including them as background methods.\n- Thank you for the note. We will put more clarifications on the scope of the survey, especially on structured data. Structured data include knowledge graphs and tables. Graphs are commonly covered in multimodal surveys, such as Guo et al, 2019. In comparison, tables are less seen in multimodal learning, but it is deemed as a heterogeneous source of information by some recent papers, such as UnifiedSKG (Xie et al, 2022) and Uniqorn (Pramanik et al, 2021). \n- Yes, the retrieval can occur at multiple stages. Some retrieve with respect to the inputs, while others retrieve with first-stage outputs. We discussed grouping the methods based on different stages, but a modality-task organization seemed clearer to the readers. We will add more stage-specific analysis for each modality. If needed, we can also add a table that clearly shows the stages in the appendix.\n- Thank you for the additional references, we aim to include all 4 of them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work aims to fill the gap by presenting a comprehensive and in-depth research on RAG with multi-modality. It provides a thorough analysis and understanding of RAG with multi-modality, which will be beneficial for the community to gain a deeper insight into its intricacies and details.\n- This work is beneficial to both the NLP and multimodal communities, as the questions and insights it presented can stimulate further development in this field.\n- This paper has a well-structured organization and good writing skills, making it easy for readers to understand.", "reasons_to_reject": "- To ensure smooth writing, the authors have included a lot of background knowledge about multimodal learning and other related fields in the paper. This helps readers better understand the topic. However, one issue is that in some places, the introduction of certain works deviates from the main focus of the discussion. For example, from line 663 to line 666, and line 687 to 695, etc. These discussions may dilute the main theme, i.e., RAG with multi-modality and cause confusion regarding some related works.\n- One issue is that the definition of multimodality may need to be clarified, as the article considers unconventional structured knowledge such as tables and structured knowledge as multimodal information used for enhancement. This may be hasty, as multimodality typically refers to heterogeneous data. There may be a discussion in the academic community regarding the broad and narrow classifications of multimodality, and this point should be explicitly stated in the paper, at least.\n- Further, the distinction of when enhanced information is retrieved needs to be carefully considered. This is because new information from retrieval can occur at various stages, such as data construction, preprocessing, feature extraction, and integration. I don't believe all of these stages can be narrowly categorized as retrieval enhancement. These statements like line 468-475, line 481-489, line 546-547, etc, can be considered to modify.", "questions_to_authors": "See weakness above. ", "paper_topic_and_main_contributions": "This paper reviews recent advances in retrieving multimodal information to augment generation, including using knowledge like image, code, structured knowledge, audio, and video, which provides a unified view for researchers to understand this field.\nThe main contributions are:\n- The authors group retrieval augmented generation with multi-modality and carry out detailed research, analyze the innate connections among these works.\n- The author has assessed and discussed the current challenges and proposed some ideas and solutions for potentially promising directions in the future.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "If the authors focus on broad discussion of *retrieval augmented*, more works in the field of AI creation[1,2], multimodal dialogue[3,4], etc, may be mentioned.\n- [1] Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences\n- [2] Multi-Modal Experience Inspired AI Creation\n- [3] Maria: A Visual Experience Powered Conversational Agent\n- [4] Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation\n- …", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written with no obvious typos and grammar mistakes. A minor style problem is that the style format of all the grouping name is not consistent. For example, line 230 “Visually grounded dialogue” and line 288 “Text-to-Code Generation”, etc. The capitalization of the first letter should be consistent. The authors are supposed to check all other places to fix this."}}, {"review_id": "vz0X75lixR", "reviewer": "Reviewer_ojn8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. \n- We will add more discussion on future directions in the later version.\n- Thank you for the suggestion. The work (Zhu, Wanrong, et al.) uses conditionally generated images to guide generation, instead of retrieval. We will consider adding it to the related work in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Retrieval-augmented generation (RAG) is an important topic.\n2. The survey broadly covers mutlimodal RAG papers.", "reasons_to_reject": "1. There is no major weakness as a survey paper, though more insights and extended future direction section would be appreciated. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper reviews methods which assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Their main contribution is a comprehensive survey.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The authors may consider adding related work in the line of text generation:\nZhu, Wanrong, et al. \"Visualize Before You Write: Imagination-Guided Open-Ended Text Generation.\" arXiv preprint arXiv:2210.03765 (2022)."}}, {"review_id": "YDJ5FlsxH7", "reviewer": "Reviewer_yUiZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review. \n1. Currently, we try to illustrate the modality-specific benefits in the beginning of each subsection, such as L183-188 and L438-L442. We will add more analysis on the specific purposes of RAG and link to the nature of each modality in a future version. \n2. Yes, the term \"multimodal\" refers to the source of retrieval. Pure text-based RAGs are currently discussed in Section 2.2 (L150-167 in particular) as background knowledge. We can add a more detailed analysis in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a timely, structured, and comprehensive overview of an important emerging research domain: retrieval-augmented generation with multi-modality.\n2. The paper analyzed over 140 recent papers in this domain and drew connections between different works and applications.\n3. It would be a valuable reference for researchers looking to adapt multimodal retrieval techniques to large language models.\n", "reasons_to_reject": "1. It could provide more analysis on the purpose and benefits of different modalities. For example, some (e.g. KGs) are useful to reduce hallucinations, while some (e.g. imagination) are useful to improve generation quality or as a hint.\n2. To my understanding, the \"multimodal\" in the title means the source of retrieval is multimodal, so it would be good to also include the literature of pure text-based RAGs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This survey paper provides a comprehensive review of research on retrieving multimodal information to augment generative language models. The key contributions are:\n\n1. The paper established retrieval-augmented generation with multi-modality as an important emerging group of methods, especially relevant with the rise of large language models.\n2. The paper analyzed over 140 recent papers in this domain and drew connections between different works and applications.\n3. The paper provided an in-depth literature review of papers using image, code, structured knowledge, audio and video modalities to improve language generation.\n4. Finally, The paper discussed future research directions like multimodal reasoning, knowledge indexing, and incorporating retrieval into pretraining.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study investigates the use of multimodality in LLM and how retrieving (inputting) multimodal information contributes to it. 

Pros:
It is a useful paper with a good description of the current situation surrounding multimodal LLM, including background knowledge.
A comprehensive study is well done. 

Cons:
As reviewer 4Ruw points out, the scope of multimodality covered by this paper needs clarification.
Decision: Accept-Findings