Retrieval-augmented LMs:
Past, Present and Future
Large Language Models: Methods and Applications
Akari Asai (akari@cs.washington.edu)

Feel free to post questions on Sli.do!
Sli.do code #2068655

How do normal parametric LLMs work?
Encapsulating everything in parameters by pre-training parameters on large-scale text corpora
Pittsburgh is a city in the county
seat of Allegheny County,
Pennsylvania, United States

Pre-trainingdata

𝑃 𝑥𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛−1 )LLM

How do normal parametric LLMs work?
Encapsulating everything in parameters by pre-training parameters on large-scale text corpora
Pittsburgh is a city in and thecounty seat of Allegheny County,
Pennsylvania, United States

Pre-training
data

AlleghenyPennsylvaniaKing

𝑃 𝑥𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛−1 )LLM
Pittsburghislocated

in

𝑥1

𝑥2

𝑥3

𝑥4

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot memorize everything in their parameters (yet), resulting in factual inaccuracy

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot memorize everything in their parameters (yet), resulting in factual inaccuracy

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot memorize everything in their parameters (yet), resulting in factual inaccuracy

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot memorize everything in their parameters (yet), resulting in factual inaccuracy

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot memorize everything in their parameters (yet), resulting in factual inaccuracy…

Limitations of parametric LLMs #1: Hallucinations
LLMs cannot encapslate everything in their parameters yet.

…

Catastrophic incidents due to LLM hallucinations
Such LLM hallucinations have been causing many critical incidents in the real world

Retrieval-augmented LMs: Definitions & Notations
A new type of LMs that can use large-scale text data (datastore) at inference-time

Input 𝒙LLM

Pre-training
data

Output 𝒚

Retrieval-augmented LMs: Definitions & Notations
A new type of LMs that can use large-scale text data (datastore) at inference-time

Input 𝒙Retriever

Datastore

LLM

Retrieval-augmented LMs: Definitions & Notations
A new type of LMs that can use large-scale text data (datastore) at inference-timeDatastore

Query 𝒒

Input 𝒙Retriever

LLM

Retrieval-augmented LMs: Definitions & Notations
A new type of LMs that can use large-scale text data (datastore) at inference-time

Query 𝒒

Input 𝒙Retriever

LLM

sim(𝑞, 𝑑)Datastore

Documents
𝒁Retrieval-augmented LMs: Definitions & Notations
A new type of LMs that can use large-scale text data (datastore) at inference-timeDatastore

Query 𝒒

Input 𝒙Retriever

LLM
Documents
𝒁

Output 𝒚Benefit of retrieval-augmented LMs #1: reduce hallucinations
Retrieval-augmented LMs can reduce hallucinations, especially in long-tail knowledge“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

Benefit of retrieval-augmented LMs #1: reduce hallucinations
Retrieval-augmented LMs can reduce hallucinations, especially in long-tail knowledge“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

Quiz:

What are the other benefits of using
retrieval-augmented LMs?

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
Parametric LMs’ knowledge gets obsolete quickly & requires continuous training“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
Parametric LMs’ knowledge gets obsolete quickly & requires continuous training“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
Parametric LMs’ knowledge gets obsolete quickly & requires continuous training

Pre-trainingdata

Rishi Sunak is a British politician andthe current Prime Minister of theUnited Kingdom (May 2024
Wikipedia)

2024 May data

𝑃 𝑥𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛−1 )LLM“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
Parametric LMs’ knowledge gets obsolete quickly & requires continuous training

Pre-trainingdata

Rishi Sunak is a British politician andthe current Prime Minister of theUnited Kingdom (May 2024
Wikipedia)

Rishi
Boris

2024 May dataLiz

𝑃 𝑥𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛−1 )LLM
The current Prime Minister UK

is“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
Parametric LMs’ knowledge gets obsolete quickly & requires continuous training
The incumbent prime minister is Keir
Starmer, who assumed the office on
5 July 2024.

2023

Pre-training
data

Rishi
Boris

2024 dataKeir

𝑃 𝑥𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛−1 )LLM
The current Prime Minister UK

is“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #2: Adaptations w/o training
We can easily swap datastores for retrieval-augmented LMs for new data distributions

2023

2024 data“RealTime QA: What's the Answer Right Now?” Kasai et al. NeurIPS (Benchmark). 2023

Benefit of retrieval-augmented LMs #3: Providing attributions
Retrieval-augmented LMs can provide a small number of documents as attributions“Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models”. Bohnet et al. ArXiv 2020.

Benefit of retrieval-augmented LMs #4: Flexible data opt-in / out
We can incorporate or remove high-risk data dynamically at inference, not training time

“SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore”. Min et al. In ICLR 2024

Benefit of retrieval-augmented LMs #5: parameter efficiency
Retrieval-augmented LMs can be much more parameter efficient and compute-optimal“Scaling Retrieval-Based Language Models with a Trillion-Token Datastore.” Shao, He, Asai et al., ArXiv 2024.

Benefit of retrieval-augmented LMs #5: parameter efficiency
Retrieval-augmented LMs can be much more parameter efficient and compute-optimal“Scaling Retrieval-Based Language Models with a Trillion-Token Datastore.” Shao, He, Asai et al., ArXiv 2024.

Retrieval-augmented LMs have been widely used!
Retrieval-augmented LMs have been widely used both in academia and industry

Retrieval-augmented LMs have been widely used!
Retrieval-augmented LMs have been widely used both in academia and industry“60% of LLM applications use some formof retrieval-augmented generation (RAG)”

Today’s outline
1. Introduction: What are retrieval-augmented LMs? Why do we want

to use them?
2. Past: Architecture and training of retrieval-augmented LMs fordownstream tasks
3. Present: Retrieval-augmented generation with LLMs
4. Future: Limitations & future directions

Feel free to post questions on Sli.do!
Sli.do code #2068655

Past: Architecture and trainingof retrieval-augmented LMs fordownstream tasks

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
Trained QA model

BM25

Retrieve and read Wikipedia articles for open-domain QA“Reading Wikipedia to Answer Open-Domain Questions.” Chen et al., ACL 2017.

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
2019: ORQA
2020: RALM, RAG

End-to-end pre-training → fine-tuning of retriever & LM

“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Lewis et al., NeurIPS 2020.

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM

New architectures for retrieval-augmented LMs“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM
2021: RETRO
New architectures for retrieval-augmented LMs“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

Brief history of retrieval-augmented LMs development
Versatile and powerful LLMs demonstrate effectiveness even without fine-tuning

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM
2020: GPT3
2021: RETRO
2022: ChatGPT
LLMs surpressed specialized QA models w/ retrieval

https://paperswithcode.com/sota/question-answering-on-triviaqa

Brief history of retrieval-augmented LMs development
Success of In-Context Retrieval-Augmented LMs (commonly referred to as RAG today)

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM
2021: RETRO

Use off-the-shelf LLMs & retrieval systems

2023: In-Context RetrievalAugmented LMs“In-Context Retrieval-Augmented Language Models.” Ram et al., TACL 2023.

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM
2021: RETRO

2023: Retrievalaugmented LLMs

Past: Developments in
Architecture and Training forSpecific Tasks

Brief history of retrieval-augmented LMs development
RAG was initially extensively studied for certain NLP tasks, namely Question Answering

2017: DrQA
2019: ORQA
2020: RALM, RAG
2020: kNN LM

Past: Architecture / training
developments for certain down or
up-stream tasks

2021: RETRO

2023: Retrievalaugmented LLMs

Current: Designing versatile andreliable LLM-based RAG systems
for diverse use cases

Diverse architectures of retrieval-augmented LMs
Classifying retrieval-augmented LMs based on “where” we incorporate retrieved context●

Input augmentaiton○ Augment the input of LMs with retrieved context○ E.g., RAG, REALM, DrQA, In-context RALM

Retriever

LLM

𝑦

𝑥

REALM: Augmenting input space of LMs
REALM is an retrieval-augmented masked LMs that predicts next tokens / spans in context“REALM: Retrieval-Augmented Language Model Pre-Training.” Guu et al., ICML 2020.

REALM: Augmenting input space of LMs
REALM finds relevant context by conducting kNN search in embedding spaces“REALM: Retrieval-Augmented Language Model Pre-Training.” Guu et al., ICML 2020.

REALM: Augmenting input space of LMs
REALM compute weighted averages of final answer distributions, using retrieval similarities“REALM: Retrieval-Augmented Language Model Pre-Training.” Guu et al., ICML 2020.

REALM: Augmenting input space of LMs
REALM compute weighted averages of final answer distributions, using retrieval similarities“REALM: Retrieval-Augmented Language Model Pre-Training.” Guu et al., ICML 2020.

RAG: Augmenting input space of LMs
RAG combines a trained retriever & autoregressive BART, starting from pre-trained weights“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Lewis et al., NeurIPS 2020.

RAG & REALM: Results
RAG and REALM show their effectiveness on open-domain QA and other tasks●

RAG outperforms REALM and other baselines on Open-domain QA such as NaturalQuestion

Open-domain QA (EM)
50

NQ

WQ

40
30
Higher is
better 20
100
T5

No retrieval

REALM

DPR

RAG

With retrieval“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Lewis et al., NeurIPS 2020.

RAG & REALM: Results
RAG and REALM show their effectiveness on open-domain QA and other tasks●●

RAG outperforms REALM and other baselines on Open-domain QA such as NaturalQuestions
RAG also show their effectiveness on generation tasks
Question generation (Tri-match)

Open-domain QA (EM)50NQMS MARCO
WQ

Jeopardy

100

40

80

30
Higher is
better 20

60
40

10

20

0

0
T5

No retrieval

REALM

DPR

RAG

With retrieval“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Lewis et al., NeurIPS 2020.

Gold

BART

No retrieval

RAG-Token

With retrieval

Recent follow-up: In-context Retrieval-augmented LMs
Similar principles as in DrQA, REALM, RAG, but completely removes retrieval●●●

Combining retrieval and off-the-shelf LMs e.g., GPT-4 at inference time without training
Often referred to as “RAG” nowadays
We’ll cover this in depth in the next section!

“In-Context Retrieval-Augmented Language Models.” Ram et al., TACL 2023.

Pros and cons of input augmentation
Input augmentation is powerful but has several limitations●

Pros○ Easy to switch to new, more powerful LMs with fine-tuning / without training○ LLMs can effectively levarage input context

Pros and cons of input augmentation
Input augmentation is powerful but has several limitations.
●

Pros○ Easy to switch to new, more powerful LMs with fine-tuning / without training○ LLMs can effectively levarage input context●

Cons○ Expensive to scale up to hundreads or thousands of documents■ LLMs also often do not fully levarage long context○ No strict attributions to specific evidences“Lost in the Middle: How Language Models Use Long Contexts.” Liu et al., TACL 2023.

Diverse architectures of retrieval-augmented LMs
Classifying retrieval-augmented LMs based on “where” we incorporate retrieved context●●

Input augmentaiton○ Augment the input of LMs with retrieved context○ E.g., RAG, REALM, DrQA, In-context RALM
Intermediate incorporation○ Incorporate retrieved context in intermediate
spaces of transformers○ E.g., RETRO, Instruct RETRO

Retriever

LLM

RETRO: Incorporating context in intermediate layers
RETRO enables more efficient incorporations of many documents“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
RETRO enables more efficient incorporations of many documents

Standard transformer block“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
RETRO enables more efficient incorporations of many documents“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
RETRO uses frozen BERT as a retriever, and retrieve nearest neighbors from 1.7T datastore“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
Given the input sequence, it first retrieves a set of relevant documents (embedding of text)“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
Use cross-attention to generate retrieved context-aware representations“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
Use cross-attention to generate retrieved context-aware representations“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
Concatnate all of the CA output (the size of input H and output CCA(H,E) remains the same

“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Incorporating context in intermediate layers
RETRO uses frozen BERT as a retriever, and retrieve nearest neighbors from 1.7T datastore“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Results
RETRO shows impressive performance improvements on upstream (language modeling) tasks●

RETRO significantly outperforms non-retrieved baselines

Lower isbetter“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Results
RETRO shows impressive performance improvements on upstream (language modeling) tasks●●

RETRO significantly outperforms non-retrieved baselines
RETRO performance continues to improve as the datastore scales from a few billion to 1.7
trillion data points

Lower isbetter“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

RETRO: Results
RETRO shows impressive performance improvements on upstream (language modeling) tasks●●●

RETRO significantly outperforms non-retrieved baselines
RETRO performance continues to improve as the datastore scales from a few billion to 1.7
trillion data points
Increasing # of docs up to 40 helps

Lower isbetter“Improving language models by retrieving from trillions of tokens.” Borgeaud et al., Arxiv 2020.

Recent follow-up: Instruct RETRO
Develop RETRO-block on top of Llama (autoregressive LMs), pre-training & multi-task training“InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.” Wang et al., ICML 2024.

Pros and cons of intermediate incorporation
Alternative way to incorporate retrieved context in a more scalable way, but requires training●

Pros○ More efficiently incorporates many passages than input augmentation○ Possibly more effective than retrieval augmentaion (i.e., Instruct RETRO results)●

Cons○ Require modification of underlying LMs○ Expensive pre-training is necessary○ Doesn’t provide strict attribution

Diverse architectures of retrieval-augmented LMs
Classifying retrieval-augmented LMs based on “where” we incorporate retrieved context●

Input augmentaiton○ Augment the input of LMs with retrieved context○ E.g., RAG, REALM, DrQA, In-context RALM●

Intermediate incorporation○ Incorporate retrieved context in intermediate
spaces of transformers○ E.g., RETRO, Instruct RETRO●

Output interpolation○ Interpolate output token probabilities with retrieved
non-parametric distributions○ E.g., kNN LM

Retriever

LLM

kNN LM: directly interpolate output token distributions
Directly interpolate output token distributions of LMs●

Given a context 𝑥, a model predicts parametric distributions for next token

Parametric distribution(LM output distribution)“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

kNN LM: directly interpolate output token distributions
Directly interpolate output token distributions of LMs●●

Given a context 𝑥, a model predicts parametric distributions for next token
kNN LM computes nonparametric distributions, by finding similar training context 𝐶𝑖

Nonparametric distribution“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

kNN LM: directly interpolate output token distributions
Directly interpolate output token distributions of LMs●●

Given a context 𝑥, a model predicts parametric distributions for next token
kNN LM computes nonparametric distributions, by finding similar training context 𝐶𝑖“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

kNN LM: directly interpolate output token distributions
Directly interpolate output token distributions of LMs●

Interpolates two token distributions, adjusting the balance using a hyperparamter 𝜆Nonparametric
distribution

Parametric
distribution“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

kNN LM: Results
kNN LM outperforms much larger parametric LMs by large margin●

kNN LM constantly outperforms parametric 100M LMs & 30x larger 3B LMs with larger datastore“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

kNN LM: Results
kNN LM outperforms much larger parametric LMs by large margin●●

kNN LM constantly outperforms parametric LMs and 30x larger 3B LMs with larger datastore
kNN LM also enables efficient & controlled domain adaptations“Generalization through Memorization: Nearest Neighbor Language Models.” Khandelwal et al., ICLR 2020.

Recent follow-up: TRIME
Training kNN LM to better learn interpolations●●

kNN LM uses pre-trained LMs without any training
TRIME introduces an efficient training method, outperforming kNN LM

Wikitext 103 (Perplexity)
20
15
10
5

0
Transformer

kNN LM

Dev Perp“Training Language Models with Memory Augmentation.” Zhong et al., EMNLP 2022.

Test Perp

TRIME

Pros and cons of output interpolation
kNN LM & variatns have unique advantages but have several empirical challenges●

Pros○ Provides token-level attributions○ Enables explicit control between parametric and non-parametric memories

Pros and cons of output interpolation
kNN LM & variatns have unique advantages but have several empirical challenges
●
●

Pros
○ Provides token-level attributions○ Enables explicit control between parametric and non-parametric memories
Cons○ Difficult to scale to large retrieval corpora (i.e., the number of embeddings equals thenumber of tokens)○ Empirically shows limited effectiveness outside of upsteam language modeling tasks

Summary
Diverse types of retrieval-augmented LMs have been studied; have pros & cons●●●

Input augmentation: widely used and effective but faces challenges when incorporating morepassagesIntermediate incorporation: can efficiently handle more passages but requires pre-training andfine-tuning
Output interpolation: provides direct control over LM output, but has limited success indownstream tasks and faces challenges of scaling the datastore
Representative
methods

Retrieval unit

Retrieval frequency

Input
augmentation

DrQA, RAG, REALM,
ICRALM

PassageOnce at the beginning

Intermediate
incorporation

RETRO,
InstructRETRO

Passage

Every k tokens

Outputinterpolation

kNNLM
TRIME

Token

Every token

Present: Retrieval-augmented
Generation with LLMs

In-context retrieval-augmented LMs
Simply augmenting input of LMs gives signficant gain across different tasks

LLM
Answer the following question, based on the reference.
Reference
Q: Who is the current PM of UK?
A:“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

In-context retrieval-augmented LMs
Simply augmenting input of LMs gives signficant gain across different tasks

LLM
Answer the following question, based on the reference.
Reference
Q: Who is the current PM of UK?
A:“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

Rishi SunakIn-context retrieval-augmented LMs
Simply augmenting input of LMs gives signficant gain across different tasks

Retriever“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

LLM

In-context retrieval-augmented LMs
Simply augmenting input of LMs gives signficant gain across different tasks

Retriever

Answer the following question, based on the reference.
Reference
The current prime minister is Keir Starmer, who succeeded
Rishi Sunak on 5 July 2024, following the 2024 generalelection
Q: Who is the current PM of UK?
A:“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

LLM

In-context retrieval-augmented LMs
Simply augmenting input of LMs gives signficant gain across different tasks

Retriever

Answer the following question, based on the reference.
Reference
The current prime minister is Keir Starmer, who succeeded
Rishi Sunak on 5 July 2024, following the 2024 generalelection
Q: Who is the current PM of UK?
A:“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

LLM

Keir Starmer

In-context Retrieval-augmented LMs: Result
Simply augmenting input-space of LMs give signficant gain across different tasks●●

In upsream language modeling task, simply adding retrieved context gives large gains, especiallysmaller models
Similar significant gains in downstream tasks such as Question Answering“In-Context Retrieval-Augmented Language Models.” Ram et al., TACL 2023.

In-context Retrieval-augmented LMs: Result
Effects of retrieval systems for downstream task performance●

On language modeling, BM 25 results in best performance“In-Context Retrieval-Augmented Language Models.” Ram et al., TACL 2023.

In-context Retrieval-augmented LMs: Result
Effects of retrieval systems for downstream task performance●●

On language modeling, BM 25 results in best performance
On downstream QA tasks, trained retrieval models eg Contriever results in best performance“When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories”. Mallen*, Asai* et al. ACL 2023

Limitations of such naïve “RAG”
Is combining off-the-shelf models sufficient?

●In-context retrieval-augmented LMs sometimes generate content that is not
fully supported by their citations“Evaluating Verifiability in Generative Search Engines”. Liu et al. Findings of EMNLP 2023.

Limitations of such naïve “RAG”
Is combining off-the-shelf models sufficient?

In-context retrieval-augmented LMs sometimes generate content that is not
fully supported by their citations● They can easily be distracted by unhelpful context●“Making Retrieval-Augmented Language Models Robust to Irrelevant Context”. ICLR 2024.

Limitations of such naïve “RAG”
Is combining off-the-shelf models sufficient?

In-context retrieval-augmented LMs generate what is not fully supported bytheir citations● They can easily get distracted by unhelpful context● Diverse tasks require diffent retrieval needs e.g., content, frequency●

Who is the current PM of UK?

Can be easily answered based on top
documents retrieved at the beginningLimitations of such naïve “RAG”
Is combining off-the-shelf models sufficient?

In-context retrieval-augmented LMs generate what is not fully supported bytheir citations● They can easily get distracted by unhelpful context● Diverse tasks require diffent retrieval needs e.g., content, frequency●

Who is the current PM of UK?
Create a table listing all previous UK Prime
Ministers, including their terms in office, politicalparty, alma mater, and notable achievements.

Can be easily answered based on top
documents retrieved at the beginningThis may require iterative retrieval, based on
the current generation

Limitations of such naïve “RAG”
Is combining off-the-shelf models sufficient?

In-context retrieval-augmented LMs generate what is not fully supported bytheir citations● They can easily be distracted by unhelpful context● Diverse tasks require diffent retrieval needs e.g., content, frequency●

Who is the current PM of UK?
Create a table listing all previous UK Prime
Ministers, including their terms in office, politicalparty, alma mater, and notable achievements.

The equation x 2 + 2x = i has two complex
solutions. Determine the product of their realparts (from MATH)

Can be easily answered based on top
documents retrieved at the beginningThis may require iterative retrieval, based on
the current generation

Questions with similar solutions may have limited
semantic similarities in embedding space

Designing and training more reliable LLM RAG
Approaches to optimize (1) LM, (2) Retrievers, or (3) prompts for LLM RAG

1. Optimizing LLMs for RAG: training / controlling LLMs with retrieved context

Retriever

LLM

SAIL: Training LMs with retrieval-augmented data
SAIL augments existing instruction-tuning data to teach the LM how to use retrieved context

SAIL sytnthetically generates
explanations by using a NLI
model“Search Augmented Instruction Learning”. Luo et al. Findings of EMNLP 2024.

Self-RAG: Teaching LLMs to learn to levarage retrieved context
Self-RAG teaches LMs to adaptively retrieve and evaluates context & own generation●

Train an arbitrary LM (e.g., Llama 3) to generate special tokens for (1) triggering retrieval only
when necessary and (2) evaluating the relevance of retrieved context and its own generations.

“Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”. Asai et al. ICLR 2024.

Advanced RAG inference algorithm
Advanced RAG inference algorithm to better incorporate retrieved context●

Levarage model-generated tokens to improve search process at inference time

“Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”. Asai et al. ICLR 2024.

Optimizing LLMs for RAG: Results
New training and advanced inference algorithm for RAG significantly boost performance●

Training with 8B and 13B models significantly boosts performance compared to off-the-shelf RAG
pipelines

Acc,
rouge

100
80
60
40
20
0
PopQA
Llama 2 13B

FActScore

ASQA

Self-RAG 13B

PubHealth

ChatGPT

SAIL“Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”. Asai et al. ICLR 2024.

Optimizing LLMs for RAG: Results
New training and advanced inference algorithm for RAG significantly boost performance●●

Training with 8B and 13B models significantly boosts performance compared to off-the-shelf RAG
pipelines
Adaptive use of retrieval also improves the efficiency of RAG systems

Acc,
rouge

100
80
60
40
20
0
PopQA
Llama 2 13B

FActScore

ASQA

Self-RAG 13B

PubHealth

ChatGPT

SAIL“Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”. Asai et al. ICLR 2024.

Designing and training more reliable LLM RAG
Approaches to optimize (1) LM, (2) Retrievers, or (3) prompts for LLM RAG

1. Optimizing LLMs for RAG: training / controlling LLMs with retrieved context
2. Optimizing Retriever for RAG: training retrievers for LLM RAG

Retriever

LLM

Optimizing retrievers for RAG
Training retrieval modules using LM feedback●●

For RAG pipelines using blackbox LLMs e.g., GPT o1, we cannot directly train the LLMs for RAG
Can we train retrievers instead?

“REPLUG: Retrieval-Augmented Black-Box Language Models”. Shi et al. NAACL 2024.

REPLUG: Training a retriever using blackbox LLM feedback
Training retrieval model using LM feedback●

Train retrievers for black-box LLMs by minimizing KL divergence between LM & retriever“REPLUG: Retrieval-Augmented Black-Box Language Models”. Shi et al. NAACL 2024.

REPLUG: Training a retriever using blackbox LLM feedback
Training retrieval model using LM feedback●

Train retrievers for black-box LLMs by minimizing KL divergence between LM & retriever“REPLUG: Retrieval-Augmented Black-Box Language Models”. Shi et al. NAACL 2024.

REPLUG: Training a retriever using blackbox LLM feedback
Training retrieval model using LM feedback●

Train retrievers for black-box LLMs by minimizing KL divergence between LM & retriever“REPLUG: Retrieval-Augmented Black-Box Language Models”. Shi et al. NAACL 2024.

RA-DIT: Combining REPLUG + retrieval-augmented LM training
Trains both retriever and LM on multiple tasks using REPLUG + retrieval-augmented training“RA-DIT: Retrieval-Augmented Dual Instruction Tuning”. Lin et al. ICLR 2024.

RA-DIT: Combining REPLUG + retrieval-augmented LM training
Trains both retriever and LM on multiple tasks using REPLUG + retrieval-augmented training“RA-DIT: Retrieval-Augmented Dual Instruction Tuning”. Lin et al. ICLR 2024.

REPLUG, RA-DIT: Results
Training retriever & LM gives large improvements across diverse tasks●

RA-DIT observes performance gain from combinations of off-the-shelf (REPLUG w/o LSR)

Acc, EM

80
75
70
65
60
55
50
MMLU
REPLUGTQA
RA-DIT (LM only)“RA-DIT: Retrieval-Augmented Dual Instruction Tuning”. Lin et al. ICLR 2024.

Avg
RA-DIT (R only)

RA-DITREPLUG, RA-DIT: Results
Training retriever & LM gives large improvements across diverse tasks●●

RA-DIT observes performance gain from combinations of off-the-shelf (REPLUG w/o LSR)
Both LM and retriever training contributes to performance gain

Acc, EM

80
75
70
65
60
55
50
MMLU
REPLUGTQA
RA-DIT (LM only)“RA-DIT: Retrieval-Augmented Dual Instruction Tuning”. Lin et al. ICLR 2024.

Avg
RA-DIT (R only)

RA-DITREPLUG, RA-DIT: Results
Training retriever & LM gives large improvements across diverse tasks●●

RA-DIT observes performance gain from combinations of off-the-shelf (REPLUG w/o LSR)
Both LM and retriever training contributes to performance gain

Acc, EM

80
75
70
65
60
55
50
MMLU
REPLUGTQA
RA-DIT (LM only)“RA-DIT: Retrieval-Augmented Dual Instruction Tuning”. Lin et al. ICLR 2024.

Avg
RA-DIT (R only)

RA-DIT

Optimizing retrievers for RAG
Alternative appraoches: introducing additional modules for reranking or filtering●●

From initial retrived docs 𝑍, select more relevant context before feeding it to LMs
Examples include: cross-encoder, context compression (Xi et al., 2024)“RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation”. Xu et al. ICLR 2024.

Designing and training more reliable LLM RAG
Approaches to optimize (1) LM, (2) Retrievers, or (3) prompts for LLM RAG

1. Optimizing LLMs for RAG: training / controlling LLMs with retrieved context
2. Optimizing Retriever for RAG: training retrievers for LLM RAG
3. Optimizing Prompts for RAG: advanced prompt techniques

Retriever

LLM

DSPy: Optimizing prompts for LLM RAG
Optimizing prompts for RAG applications●

Training-free RAG systems are brittle to prompts“DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”. Khattab et al. ICLR 2024.

DSPy: Optimizing prompts for LLM RAG
Optimizing prompts for RAG applications●

Training-free RAG systems are brittle to prompts

Scores33%
with GPT-3.5
on a multi-hop QA task

“DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”. Khattab et al. ICLR 2024.

DSPy: Optimizing prompts for LLM RAG
Optimizing prompts for RAG applications●

DSPy optimizes instructions and few-shot demonstrations to achieve the best performance“DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”. Khattab et al. ICLR 2024.

DSPy: Optimizing prompts for LLM RAG
Optimizing prompts for RAG applications“DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines”. Khattab et al. ICLR 2024.

DSPy: Optimizing prompts for LLM RAG

Scores55%
with GPT-3.5
on a multi-hop QA task

Future: Limitations & future
directions

Roadmap for more efficient & reliable retrieval-augmented LMs
Challenges of scaling up datastores & increased inference-time costs●

Evaluations●

Performance gains are achieved by scaling up the datastore
to trillions of tokens
Significantly increases inference costs, including CPU memoryand storage requirements (e.g., 24 TB for 1.7 trillion-token).

Algorithms

Infrastructure“Scaling Retrieval-Based Language Models with a Trillion-Token Datastore.” Shao, He, Asai et al., ArXiv 2024.

Roadmap for more efficient & reliable retrieval-augmented LMs
New algorithms & arthictectures to enable more efficient and effective RAG●

Evaluations●

Current “RAG” has many issues such as efficiency &redundancy
Alternative algorithms, better LM architectures, caching … etc
for improving efficiency and performance

Algorithms

Infrastructure“Generative Representational Instruction Tuning.” Muennighoff et al., ArXiv 2024.

Roadmap for more efficient & reliable retrieval-augmented LMs
New algorithms & arthictectures to enable more efficient and effective RAG●

Evaluations●

Current “RAG” has many issues such as efficiency &redundancy
Alternative algorithms, better LM architectures, caching … etc
for improving efficiency and performance

Algorithms

Infrastructure“PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design.” Jiang et al., ArXiv 2024.

Roadmap for more efficient & reliable retrieval-augmented LMs
Careful analyses on their effectiveness and limitations

Evaluations

Prior systems are often evaluated only on simple general-domain
tasks. Further exploration into their evaluation are needed●

Domains: most prior evaluations are in general-domain tasks,
where Wikipedia is a sufficient knowledge source●

Tasks: going beyond open-domain QA, multiple-choice QA●

Aspects: instead of merely evaluating final “correctness”, more
holistic evaluations of different aspects of RAG

Algorithms

InfrastructureQuestions?

Sli.do code #2068655

Acknowledgements: Some slides are adapted from our ACL 2023 tutorials https://acl2023-retrievallm.github.io/ co-taught by Akari, Sewon Min, Zexuan Zhong and Danqi Chen. We thank Omar Khattab
for sharing the DSPy slides