Section 7
Challenges & Opportunities

1Tutorial summary• De nition & preliminaries• Retrieval-based LMs: Architecture• What to retrieve: tokens, text blocks, entity mentions• How to use retrieval: input layer, intermediate layers, output layers• When to retrieve: once, every n tokens, every token• Retrieval-based LMs: Training• Independent training• Sequential training• Joint training with asynchronous index update• Joint training with in-batch approximationfi2

Tutorial summary (cont’d)• Applications• What tasks: knowledge-intensive NLP, code generation, classi cation etc• How to adapt: retrieval-based prompting, ne-tuning, reinforcement learning• When to use: long-tail, knowledge update, veri ability, parameter-e ciency, privacy, outof domain adaptations• Multilingual & multimodal extensions• Multilingual: retrieve and generate cross-lingually to overcome scarcity of datastore• Multimodal: multimodal extensions of retrieval-based LMs for new modality

ffi

fififi3

Challenge: Scaling retrieval-based LMs
A small LM + a large datastore ≈ a large parametric LM?

LMDatastore
vs.

LMLM

Datastore# of parameters# of tokens

kNN-LM (Khandelwal et al., 2020)

250M

≤ 3B

NPM (Min et al., 2023)

350M

1B

Atlas (Izacard et al., 2022)

11B

~30B

RETRO (Borgeaud et al., 2021)

7B

2T≤ 175B~5B

REPLUG (Shi et al., 2023)

4

Challenge: Scaling retrieval-based LMs
Scaling law?

Loss as a function of:• Training tokens• Model sizes• Training FLOPs+ Datastore sizes?
Scaling law for parametric LMs (Kalpan etal., 2020; Ho man et al., 2022)

Dependent on architecture
and training methods!

ff5Challenge: Scaling retrieval-based LMs
Measured on NVIDIA RTX 3090 GPU(Zhong et al., 2022) with a FAISS indexer(Johnson et al., 2021) with 32 CPUs
193.6k t/s

E ciency of similarity search• >12 times slower even with approximate nearest neighborsearch implemented on GPUs

18.65

• E cient similarity search on GPUs/TPUs becomes thebottleneck of scaling retrieval-based LMs

18PPL
17300 t/s
16.23

1650 t/s
15.79

15LM

LM: 250M

kNN-LM+ approx.
searchkNN-LM+ exact
searchActive research area!

Datastore: 100M

ffi

ffi

Guo et al. 2020. “Accelerating Large-Scale Inference with Anisotropic Vector Quantization”

6Challenge: Scaling retrieval-based LMs
E ciency of similarity search
E cient nearest neighbor search:• Better loss for vector quantization (Guo et al., 2020)• Optimization over di erent accelerators (Chern et al., 2022)• Adaptive representations (Rege et al., 2023)

Space e ciency?
It also requires a large disk spacefor storage if the datastore
becomes too large - storing
vectors or raw text?

Middle ground: adaptively retrieve only whennecessary (He et al., 2021; Mallen et al., 2023)

ffffi

ffi

ffi

7

Challenge: Retrieval-based LMs for applications
Open-ended text generation?
Exposure bias? Better
decoding algorithms?
16GPT-2
kNN-LM

12840

MAUVE ↑RankGen ↑ PPL (GPT-3)↓

Wang et al. 2023. “kNN-LM Does Not
Improve Open-ended Text Generation”

8Challenge: Retrieval-based LMs for applications
Complex reasoning tasks?

Retrieval-based LMs struggle with multistep entailments or logical reasoning:- Retrieving based on similarity ofqueries is insu cient 😢
- LMs also struggle with synthesizing
information from multiple statements 😢

ffi

BehnamGhader et al. 2022. “Can Retriever-Augmented Language Models Reason? The Blame
Game Between the Retriever and the Language Model.”9Challenge: Retrieval-based LMs for applications
Complex reasoning tasks?
Potential solutions?
• Iterative retrieval & query reformulation•FLARE (Jiang et al., 2023), Self-Ask (Presset al., 2022), ReAct (Yao et al., 2023)• Decompose the task into a multi-hop
program for complex reasoning•DSP: Demonstrate-Search-Predict
LM: Language Model, RM: Retrieval Model (both frozen)

Khattab et al. 2022. “Demonstrate-Search-Predict: Composing retrieval and language models forknowledge-intensive NLP”

10

Open questions• What is the best architecture & training method for
retrieval-based LMs in practice?

• We still don’t know yet how to best scale up thesemodels - Scaling law?

• We may need to explore alternative decoding or

adaptation methods in downstream tasks (e.g., openended text generation, complex reasoning)!

11

Q &AThank you for joining us today!
All the materials are athttps://acl2023-retrieval-lm.github.io/12