Section 1: References (1)Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tauYih. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
Omar Khattab, Matei Zaharia. ColBERT: Ef cient and Effective Passage Search via Contextualized Late
Interaction over BERT. SIGIR 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk.
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ICLR 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard
Grave. Unsupervised Dense Information Retrieval with Contrastive Learning. TMLR 2022.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi. When Not to Trust
Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse,
Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger,Kevin Button, Matthew Knight, Benjamin Chess, John Schulman. WebGPT: Browser-assisted questionanswering with human feedback. arXiv 2021.
fi1

Section 1: References (2)Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese,
Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese. Teaching Language Models toSupport Answers with Veri ed Quotes. arXiv 2022.
Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen. Enabling Large Language Models to Generate Text withCitations. arXiv 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, AdamRoberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel. Extracting Training Data fromLarge Language Models. UNISEX Security 2021.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George vanden Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, JacobMenick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock,Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen,
Laurent Sifre. Improving language models by retrieving from trillions of tokens. arXiv 2021.

fi2

Section 3: References (1)Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. REALM: Retrieval-AugmentedLanguage Model Pre-Training. ICML 2020.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham. InContext Retrieval-Augmented Language Models. arXiv 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau
Yih. REPLUG: Retrieval-Augmented Black-Box Language Models. arXiv 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George vanden Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, JacobMenick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock,
Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen,
Laurent Sifre. Improving language models by retrieving from trillions of tokens. arXiv 2021.

3Section 3: References (2)Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis. Nearest Neighbor MachineTranslation. ICLR 2021.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan,
Graham Neubig. Active Retrieval Augmented Generation. arXiv 2023.
Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick. Ef cient Nearest Neighbor Language Models. EMNLP
2021.
Andrew Drozdov, Shufan Wang, Razieh Rahimi, Andrew McCallum, Hamed Zamani, Mohit Iyyer. You can't pickyour neighbors, or can you? When and how to rely on retrieval in the kNN-LM. EMNLP 2022 Findings.
Uri Alon, Frank F. Xu, Junxian He, Sudipta Sengupta, Dan Roth, Graham Neubig. Neuro-Symbolic Language
Modeling with Automaton-augmented Retrieval. ICML 2022.

fi4Section 3: References (3)Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, Tom Kwiatkowski. Entities as Experts:
Sparse Memory Access with Entity Supervision. EMNLP 2020.
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, William Cohen. Mention Memory:
incorporating textual knowledge into Transformers through entity mention attention. ICLR 2022.
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy. Memorizing Transformers. ICLR 2022.
Amanda Bertsch, Uri Alon, Graham Neubig, Matthew R. Gormley. Unlimiformer: Long-Range Transformers withUnlimited Length Input. arXiv 2023.
Ohad Rubin, Jonathan Berant. Long-range Language Modeling with Self-retrieval. arXiv 2023.

5

Section 4: References (1)Juan Ramos. Using TF-IDF to Determine Word Relevance in Document Queries. The rst instructional
conference on machine learning 2003.
Robertson and Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations andTrends® in Information Retrieval 2009.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tauYih. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, EdouardGrave. Unsupervised Dense Information Retrieval with Contrastive Learning. TMLR 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau
Yih. REPLUG: Retrieval-Augmented Black-Box Language Models. arXiv 2023.

fi6

Section 4: References (2)Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham. InContext Retrieval-Augmented Language Models. arXiv 2023.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu,
Armand Joulin, Sebastian Riedel, Edouard Grave. Atlas: Few-shot Learning with Retrieval Augmented
Language Models. arXiv 2022.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu,
Armand Joulin, Sebastian Riedel, Edouard Grave. Atlas: Few-shot Learning with Retrieval Augmented
Language Models. arXiv 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George vanden Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, JacobMenick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock,Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen,
Laurent Sifre. Improving language models by retrieving from trillions of tokens. arXiv 2021.
7

Section 4: References (3)Zexuan Zhong, Tao Lei, Danqi Chen. Training Language Models with Memory Augmentation. EMNLP 2022.

Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer.
Nonparametric Masked Language Modeling. ACL 2023 Findings.
Ohad Rubin, Jonathan Berant. Long-range Language Modeling with Self-retrieval. arXiv 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. Momentum Contrast for Unsupervised Visual
Representation Learning. CVPR 2020.
Gautier Izacard, Edouard Grave. Leveraging Passage Retrieval with Generative Models for Open Domain
Question Answering. arXiv 2020.

8Section 5: References (1)Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tauYih. Dense Passage Retrieval for Open-domain Question Answering. EMNLP 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, HeinrichKüttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. Retrieval-Augmented
Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. REALM: Retrieval-Augmented
Language Model Pre-Training. ICML 2020.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu,
Armand Joulin, Sebastian Riedel, Edouard Grave. Atlas: Few-shot Learning with Retrieval Augmented
Language Models. Arxiv 2022.
Akari Asai, Matt Gardner, Hannaneh Hajishirzi. Evidentiality-guided Generation for Knowledge-Intensive NLP
Tasks. NAACL 2022.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen,Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan
Boureau, Melanie Kambadur, Jason Weston. BlenderBot 3: a deployed conversational agent that continuallylearns to responsibly engage. Arxiv 2022.
9Section 5: References (2)Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis. Nearest Neighbor MachineTranslation. ICLR 2021.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan,Graham Neubig. Active Retrieval Augmented Generation. arXiv 2023.
Zexuan Zhong, Tao Lei, Danqi Chen. Training Language Models with Memory Augmentation. EMNLP 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George vanden Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob
Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock,Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen,Laurent Sifre. . Improving language models by retrieving from trillions of tokens. Arxiv 2022.
Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang, Meng Jiang.
Retrieval Augmentation for Commonsense Reasoning: A Uni ed Approach. EMNLP 2022.
Weijia Shi, Julian Michael, Suchin Gururangan, Luke Zettlemoyer kNN-Prompt: Nearest Neighbor Zero-Shot
Inference. EMNLP 2022.
fi10Section 5: References (3)Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese,
Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese. GopherCite: Teaching language
models to support answers with veri ed quotes. Arxiv 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau
Yih. REPLUG: Retrieval-Augmented Black-Box Language Models. Arxiv 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi. When Not to TrustLanguage Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, Graham Neubig. DocPrompting:Generating Code by Retrieving the Docs. ICLR 2023.
Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, Danqi Chen. Privacy Implications of Retrieval-Based
Language Models. Arxiv 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke
Zettlemoyer, Hannaneh Hajishirzi FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long FormText Generation. Arxiv 2023.

fi11

Section 6: References (1)Akari Asai, Xinyan Yu, Jungo Kasai, Hannaneh Hajishirzi. One Question Answering Model for Many Languages
with Cross-lingual Dense Passage Retrieval. NeurIPS 2021.
Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana
Nutanong. CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question
Answering. NAACL Findings 2022.
Benjamin Muller, Luca Soldaini, Rik Koncel-Kedziorski, Eric Lind, Alessandro Moschitti. Cross-Lingual GenQA:
A Language-Agnostic Generative Question Answering Approach for Open-Domain Question Answering. AACL
2022.
Kung-Hsiang Huang, ChengXiang Zhai, Heng Ji. CONCRETE: Improving Cross-lingual Fact-checking withCross-lingual Retrieval. COLING 2022.
Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, Michael R. Lyu. RetrievalAugmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training. NAACL Findings
2022.
12

Section 6: References (2)Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross,Alireza Fathi. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal
Knowledge Memory. CVPR 2023.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William W. Cohen. MuRAG: Multimodal Retrieval-Augmented
Generator for Open Question Answering over Images and Text. EMNLP 2022.
Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, Austin Reiter. Cross-Modal Retrieval
Augmentation for Multi-Modal Classi cation. EMNLP Findings 2021.
Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, Anima Anandkumar. Retrieval-based
Controllable Molecule Generation. ICLR 2023.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, Ziwei Liu.
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. Arxiv 2023.

fi13

Section 7: References (1)Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray,
Alec Radford, Jeffrey Wu, Dario Amodei. Scaling Laws for Neural Language Models. arXiv 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diegode Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican,George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, JackW. Rae, Oriol Vinyals, Laurent Sifre. Training Compute-Optimal Large Language Models. NeurIPS 2022.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar. Accelerating LargeScale Inference with Anisotropic Vector Quantization. ICML 2020.
Felix Chern, Blake Hechtman, Andy Davis, Ruiqi Guo, David Majnemer, Sanjiv Kumar. TPU-KNN: K Nearest
Neighbor Search at Peak FLOP/s. NeurIPS 2022.
Aniket Rege, Aditya Kusupati, Sharan Ranjit S, Alan Fan, Qingqing Cao, Sham Kakade, Prateek Jain, Ali
Farhadi. AdANNS: A Framework for Adaptive Semantic Search. arXiv 2023.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan,
Graham Neubig. Active Retrieval Augmented Generation. arXiv 2023.
14Section 7: References (2)Shufan Wang, Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha, Mohit Iyyer. KNN-LM Does
Not Improve Open-ended Text Generation. arXiv 2023.
Parishad BehnamGhader, Santiago Miret, Siva Reddy. Can Retriever-Augmented Language Models Reason?
The Blame Game Between the Retriever and the Language Model. arXiv 2022.
O r Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis. Measuring and Narrowing
the Compositionality Gap in Language Models. arXiv 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. ReAct: Synergizing
Reasoning and Acting in Language Models. ICLR 2023.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia.
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv2022.

fi15