Advancing state of the art ofRetrieval-augmented Generation
Akari Asai
University of Washington
https://akariasai.github.io/

April 18 | CMU CS 11-711

LLMs got much more powerful as they scale

Stanford University HAI. Artificial Intelligence Index Report 2024.
2

Open AI. GPT-4 (OpenAI Blog). 2023.

Hallucinations in LM outputs (April, 2024 ver.)(*April 16, 2024)

3

Catastrophic errors as results of LM hallucinations

4

Retrieval-augmented LMs
Input
Query

LM

Retriever
Datastore
Learn more about retrieval-augmented LMs? Check out our ACL 2023 tutorial
https://acl2023-retrieval-lm.github.io/ by Akari Asai, Sewon Min, Zexuan Zhong, Danqi Chen
5

Retrieval-augmented generations (RAG)
How did US states get their names?

Answer my question using references.
References:
Question: How did US states get their names?

Of the fifty states, eleven including
New York, Georgia, Washington named
after an individual person.

LMs(e.g., GPT-4, Llama 2)

UTAH: Name taken from the Ute
people who inhabited that region

Retriever
The history of human activity in(e.g., Google,
Michigan began with settlement by
BM 25)
Paleo-Indians.

Retrieve
6

Eleven states are named after an individual person.
Some states including Utah are named after native
American tribe names.

ReadRetrieval-augmented generations (RAG)
How did US states get their names?

Answer my question using references.
References:
Question: How did US states get their names?

Of the fifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Retriever(e.g., Google
BM 25)

UTAH: Name taken from the Ute
Indians who inhabited that region
The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retrieve
7LMs(e.g., GPT-4, Llama 2)

Eleven states are named after an individual person.
Some states including Utah are named after native
American tribe names.

Read

Chen et al. Reading Wikipedia to Answer Open-domain Questions. ACL 2017.

RAG has been widely and successfully adopted
RAG has shown effective
in many benchmarks

QA

Mallen*, Asai* et al., When Not to Trust Language
Models: Investigating Effectiveness of Parametric and
Non-Parametric Memories. ACL 2023.
8

Widely applied to real-world
production systems

The shift from traditional QA to open-ended instructions

How many of US states got their names from an individual person?

Simple factoid question
Of the fifty states, eleven are named after anindividual person. Six of those are named inhonor of European monarchs: the two Carolinas,
the two Virginias, Georgia, and Louisiana.

ElevenShort answer based on single document

9

The shift from traditional QA to open-ended instructions
Make a table for me summarizing how different US states
got their names, grouping them together.

Is this fully correct?
10

The shift from traditional QA to open-ended instructions
Make a table for me summarizing how different US states
got their names, grouping them together.

Hint: Only 10 states here…?

Is this fully correct?
11

The shift from traditional QA to open-ended instructions
Make a table for me summarizing how different US states
got their names, grouping them together.

12

The shift from traditional QA to open-ended instructions
Make a table for me summarizing how different US states
got their names, grouping them together.

13

The shift from traditional QA to open-ended instructions
Make a table for me summarizing how different US states
got their names, grouping them together.
Complex instruction

Long-form answers

Requiring aggregating multiple evidence

14Challenges of the current naive RAG systems: reliability

15

Liu et al. Evaluating Verifiability in Generative
Search Engines. Findings of EMNLP 2023.

Marcus. No, RAG is probably not going to
rescue the current situation. 2024.

Challenges of the current naive RAG systems: versatility
Limited effectiveness beyond information-seeking QA-like tasks

16

GPT-2
kNN-LM

12
8
4
0

MAUVE

RankGen

PPL (GPT-3)

Wang et al. kNN-LM Does Not Improve Openended Text Generation. ACL 2023.
16

BehnamGhader et al. Can Retriever-Augmented Language Models
Reason? The Blame Game Between the Retriever and theLanguage Model. EMNLP Findings 2023.

Today’s lecture
Promises and limitations of retrieval-augmented LMs
Reliable inference: Self-reflective RAG with dynamic retrieval
Versatile Retriever: Intent-aware retrievers with LMs
Summary and future directions: RAG in the wild17

Today’s lecture
Promises and limitations of retrieval-augmented LMs
Reliable inference: Self-reflective RAG with dynamic retrieval
Versatile Retriever: Intent-aware retrievers with LMs
Summary and future directions: RAG in the wild18

Q: Why do we need RAG?

19

A: Because RAG can solve many core
limitations of parametric LMs!

20

When Not to Trust Language Models:Investigating Effectiveness of
Parametric and Non-Parametric Memories
Alex Mallen*, Akari Asai*, Victor Zhong, Rajarshi Das,
Daniel Khashabi, Hannaneh Hajishirzi* = core contributors

ACL 2023 (Oral, Best Video Award — most viewed)

Factual knowledge memorization in LLMs

If we scale up LMs, will hallucinations be eventually solved?

Petroni et al. Language Models as Knowledge Bases?.
EMNLP 2019.
22

TriviaQA (paper with code)
https://paperswithcode.com/sota/question-answering-ontriviaqa

Focus and task
Focus:
Factual knowledge

Task:
Open-domain QA(Kathy Saltzman, Occupation, Politician)Subject

RelationshipObjectSubject

What is the occupation of Kathy Saltzman?
A: Politician
Object23

RelationshipDimensions of analysis
Aspect 1:
Subject entity popularity

We created a new dataset, PopQA (17k openQA questions
with fine-grained meta data). See more details in our paper!

(Kathy Saltzman, Occupation, Politician)^(Barack Obama, Occupation, Politician)Aspect 2:
Relationship type(Barack Obama, Elementary School, St.
Francis Assisi)^(Barack Obama, Occupation, Politician)24

Does scaling solve memorization? Probably not!
On popular facts

Performance gets better as scaling.

On long-tail facts

Almost flat trends.

Scaling may not overcome
hallucination in long tail!

25

RAG can address hallucinations in such long-tail!
On popular facts

vanilla ≈ RAG or vanilla > RAG

On long-tail facts

RAG > vanilla

will discuss why inthe next section!

RAG gives large gain in
long-tail questions!

Is RAG only useful forhallucination?

26

Relative popularity

Reliable, adaptable, attributable LMs
with Retrieval
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh
Luke Zettlemoyer, Hannaneh Hajishirzi, Wen-tau Yih

Arxiv 2024
https://arxiv.org/abs/2403.03187

Core limitations of parametric LMs
Hallucinations(*February 18, 2024)

Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
28Core limitations of parametric LMs
Hallucinations

Explain Akari Asai, including the list of her famous papers.

Lack of attributions
Costs of adaptations

Language model

Copyright / privacy
Large parameter size
29

Her most famous paper is “Large-Scale Multi-Document
Summarization Dataset and Evaluation Benchmark”

Core limitations of parametric LMs
Hallucinations
Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
30

Crawled Web data in 2022

Language modelCore limitations of parametric LMs
Hallucinations
Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
31

Retrain LMs using 2024 data

Language modelCore limitations of parametric LMs
Hallucinations
Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
32

New domain corpora
Retrain LMs using new
domain data

Language modelCore limitations of parametric LMs
Hallucinations
Lack of attributions
Costs of adaptations
Copyright / privacy

Copyright-protected data?

Large parameter size
33

Dodge et al., Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. EMNLP 2021.

Core limitations of parametric LMs
Hallucinations
Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
34

New York Times lawsuits
against OpenAI

Core limitations of parametric LMs
Hallucinations

Massive scaling for better performance

Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
35

Kandpal et al., Large Language Models Struggle to Learn Long-Tail Knowledge. ICML 2023.

Q: So how can retrieval-augmented LMs
solve those challenges?

36How do retrieval-augmented LMs address them?
Hallucinations

Retrieved text can be used as attributions
Input

Lack of attributions
Costs of adaptations
Copyright / privacy

Query

LM

Index

Large parameter size
Output37

How do retrieval-augmented LMs address them?
Hallucinations

Replacing datastore (Index) for adaptations
without training
Input

Lack of attributions

Query

LM

Costs of adaptations
Copyright / privacy
Large parameter size
38

Index
(2021)
2024

OutputIzacard*, Lewis* et al., Atlas: Few-shot Learning with Retrieval Augmented Language Models. JMLR 2023.

How do retrieval-augmented LMs address them?
Hallucinations

Adding new domain corpora for domain adaptations
Input

Lack of attributions

Query

LM

Costs of adaptations
Copyright / privacy
Large parameter size
39

Index

OutputHow do retrieval-augmented LMs address them?
Hallucinations

Segregating copyright-sensitive data from pretraining data

Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
40

Min* and Gururangan* et al., SILO Language Models: Isolating Legal
Risk In a Nonparametric Datastore. ICLR 2024.

How do retrieval-augmented LMs address them?
Hallucinations

Models with much less parameters canoutperforms much larger models!

Lack of attributions
Costs of adaptations
Copyright / privacy
Large parameter size
41

Min et al., Nonparametric Masked Language Modeling. Findings of ACL 2023.

Promise and limitations of retrieval-augmented LMs
Parametric LMs have numerous challenges to build reliable systems.
Retrieval-augmented LMs such as RAG can effectively address them.

When Not to Trust Language Models:

Reliable, adaptable andInvestigating Effectiveness of Parametricattributable LMs with retrieval

and Non-Parametric Memories (ACL 2023)(Arxiv 2024)

https://arxiv.org/abs/2403.03187

https://arxiv.org/abs/2403.03187

Let’s talk about how we can improve traditional RAG!
42

Today’s lecture
Promises and Limitations of Retrieval-augmented LMs
Reliable inference: Self-reflective RAG with dynamic retrieval
Versatile Retriever: Intent-aware retrievers with LMs
Summary and Future directions: RAG in the wild
43

RAG hurts in popular knowledge memorized by vanilla LMs

44

On popular facts

vanilla ≈ RAG or vanilla > RAG

On long-tail facts

RAG > vanilla

Relative popularity

Why?

Incorrect retrieval can easily confuse LMs

Category 1:
Succeed because of RAG
45

Category 2:
Failed because of RAG

Incorrect retrieval can easily confuse LMs

Category 1:
Succeed because of RAG
46

Category 2:
Failed because of RAG

Incorrect retrieval can easily confuse LMs

In what city was Zijah
Sokolovic born?

Zijah Sokolovic wasborn in Sarajevo

Category 1:
Succeed because of RAG
47

Category 2:
Failed because of RAG

Incorrect retrieval can easily confuse LMs
Failed RAG can counterfactually make LMs answer incorrectly
In what city was Zijah
Sokolovic born?

Ali Sokol ... (born 8 May 1921 in Orahovac,
Kingdom of Serbs, Croats and Slovenes, died
23 September 1974) was a Yugoslav
pulmonologist

Zijah Sokolovic´ was bornin Orahovac, Kingdom
Category 1:
Succeed because of RAG
48

Category 2:
Failed because of RAG

A simple hack: a threshold-based adaptive retrieval
Simple adaptive RAG significantly improve RAG performance & efficiency

Popularity threshold learned on dev set

49

Self-RAG:
Learning to Retrieve, Generate andCritique through Self-Reflections
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirul Sil, Hannaneh Hajishirzi

ICLR 2024 (Oral — top 1%)
Best Paper Honorable Mention at NeurIPS Instruction workshop

Standard RAG aren’t (always) reliable
Step 1: Retrieve K documents

Step 2: Prompt LM with K docs and generate

Prompt How did US states get their names?

Prompt How did US states get their names?

Of the fifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Retriever

UTAH: Name taken from the Ute
Indians who inhabited that region
The history of human activity in
Michigan began with settlement byPaleo-Indians.

Easily distracted byunhelpful documents
51

LM

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after native
American tribe names.

Standard RAG aren’t (always) reliable
Step 1: Retrieve K documents

Step 2: Prompt LM with K docs and generate

Prompt How did US states get their names?

Prompt How did US states get their names?

Of the fifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Retriever

UTAH: Name taken from the Ute
Indians who inhabited that region
The history of human activity in
Michigan began with settlement byPaleo-Indians.

LMs can also ignore
retrieved context
52

LM

US states got their names from a variety of sources. Elevenstates such as New York and Michigan are named after anindividual
individual person.
person. Some
Some statesstates including
including Utah
Utah andand
Washington,
named
Americantribe
Washington, areare
named
after after
native native
Americantribe names.
names.

Standard RAG aren’t (always) necessary
Step 1: Retrieve K documents
Write an essay of your best summer vacation
The term summer vacation orsummer break refers to a school breakin the summer between school years.

RetrieverSummer Vacation (Korean: 여름방학)is a South Korean “home vacation”
reality show
Summer Vacations a 2023 Spanish
comedy film directed by Santiago
Segura which stars Segura, Leo Harlem,
Cristina Gallego, and Patricia Conde.

53

Always retrieving fixed number of
documents is inefficient & harmfulGoal of Self-RAG: learn to retrieve, generate and critiqueUser input

No retrieval needed

Avoid unnecessary
retrieval

Retrieval needed

User input

Identify unhelpful
evidences
54

User input

[1]

[1]

[2]

[3]

[3]

Generate output
supported by evidenceSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

Of the fifty states, eleven including
New York, Georgia, Washington named
after an individual person.
The history of human activity in
Michigan began with settlement byPaleo-Indians.

US states got their names froma variety of sources. Retrieve

Reflection tokens

55

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the fifty states, eleven states
named after an individual person.
56

Irrelevant

Many states including Michigan

are named after an individual person.

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

Relevant

Several states such as Utah andAlabama are named after native American tribes.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Supported

Of the fifty states, eleven states
named after an individual person.
57

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after native American tribes.

UTAH: Name taken from the Ute
Indians who inhabited that region

PartiallySelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

58

Supported

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after native American tribes.

PartiallySelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Generate with no retrieval

Write an essay of your best summer vacation

No Retrieval

My best summer vacation was a magical escape to the coastal town of Santorini.

The azure waters, charming white-washed building are unforgettable.
59

Util:5

Reflection tokens for retrieval and critique
US states got their names
from a variety of sources.

California
e.g.,A
Original LM vocabularies

60Reflection tokens for retrieval and critique
US states got their names
from a variety of sources.

California
e.g.,Original LM vocabularies

A

Retrieve
No Retrieval

Retrieval tokens

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withreflection tokens
61

UsefulNot useful

Critique tokens

Reflection tokens for retrieval and critique
US states got their names
from a variety of sources.

California
e.g.,A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withreflection tokens
62

UsefulNot usefulReflection tokens for retrieval and critique
US states got their names
from a variety of sources.
Of the fifty states,
eleven are named after
an individual person.

California
e.g.,A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withreflection tokens
63

UsefulNot usefulReflection tokens for retrieval and critique
US states got their names
from a variety of sources.
Of the fifty states,
eleven are named after
an individual person.

11 of 50 state names
come from persons.

California
e.g.,A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withreflection tokens
64

UsefulNot usefulReflection tokens for retrieval and critique
US states got their names
from a variety of sources.
The history of humanactivity in Michigan beganwith settlement by PaleoIndians.

California
e.g.,A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withreflection tokens
65

UsefulNot useful

So how can we train suchself-reflective LMs?

Self-RAG training
Calif
,
A

How did US states get their names?

No

Of the fifty states, eleven are named
after an individual person.

Generator LM

supporusefulRetriever66

Self-RAG training
Calif
,
A

How did US states get their names?

No

Of the fifty states, eleven are named
after an individual person.

Irrelev

Generator LM

No
Not

Critic LM teaches Generator LM
to predict reflection tokens
Training time only Critic LM
67

Self-RAG training - critic LM

Evaluate if the output y to an input x
is supported by retrieved doc d

max 𝔼((x,y,d),r)∼𝒟critic log p𝒞(r | x, y, d)𝒞

x How did US states get their names?
dyOf the fifty states, eleven are named
after an individual person.

11 of 50 state names come from persons.

Supported

Critic LM

85-90% acc. on validation set

30k fine-grained feedback, alignwith human in 90%

𝒟critic = {(x, y, d), r}
68

Self-RAG training - augmented Instruction-tuning data
150k (input, output) instances from diverse instruction-following data69

Self-RAG training - generator LM
max 𝔼(x,d,y,r)∼𝒟gen log pℳ(y, r | x, d) .
ℳ

x How did US states get their names?
Of the fifty states, eleven are namedd after
an individual person.

yr

11 of 50 state names
come from persons.

Supported

Generator LM

Train with a standard next token objective with expanded vocabulary70Self-RAG training - generator LM
max 𝔼(x,d,y,r)∼𝒟gen log pℳ(y, r | x, d) .
ℳ

x How did US states get their names?
Of the fifty states, eleven are namedd after
an individual person.

Memory-efficient & stable trainingy

11 of 50 state names
come from persons.

rGenerator LM

Easily applied to new pre-trained LM

Customize & control via reflection tokens
71

SupportedHow?

Self-reflection-guided decoding
Conduct segment-level beam search to find top k segments
Promptf(

72

Relevant

0.9

PromptSupported)f( Irrelevant )0.1

Promptf(

Relevant

0.4

Partially)Self-reflection-guided decoding
Enable simple model customization by changing weights
Promptf(

73

Relevant

0.9

PromptSupported)f( Irrelevant )0.1

Promptf(

Relevant

0.4

Partially)

Experimental details
Tasks and datasets• Closed-set tasks (classifications, multiple-choice QA)• ARC-Challenge (Clark et al., 2018)• PubHealth (Zhang et al., 2023)• Short-form generation• OpenQA - PopQA• Trivia QA (Joshi et al., 2017)• Long-form generation• ASQA-ALCE [fluency, citation accuracy, correctness] (Gao et al., 2023)• Bio generations [FactScore] (Min et al., 2023)
74

Experimental details

More details of training & test are in our paper!

Training details• Critic training data: 4k-20k instances for each type
• Generator training data: 150k instruction-following datasets• ShareGPT• OpenAssistant• Alpaca• FLANV2• Natural Questions ….
• Base LMs: Llama2-7B, 13B (Touvron et al., 2023)• Computation: 4*A100 (15 hours)
75Experimental details

More details of training & test are in our paper!

Inference-time details• Retriever Encoder: Contriever-MS MARCO (Izacard et al., 2022)• Index: HNSW Index (0.1 sec / query) and FLAT Index (5 sec / query)• Efficient LM inference: vllm (Kwon et al., 2023)• Tree decoding configuration: max 200 tokens per depth, max depth of 776

Experimental results (short-form & closed)
Self-RAG outperforms vanilla LMs incl. ChatGPT

LLama2-13B chat

ChatGPT

LLama2-13B chat RALM

ChatGPT RALM

80
60
40
20
0
77

PopQA

PubHealthOurs

Experimental results (short-form & closed)
Self-RAG outperforms standard RAG + LLMs

LLama2-13B chat

ChatGPT

LLama2-13B chat RALM

ChatGPT RALM

80
60
40
20
0
78

PopQA

PubHealthOurs

Experimental results (long-form)
Outperforms other LMs in terms of factuality & fluency correctness
LLama2-13B chat

ChatGPT

LLama2-13B chat RALM

Ours

90
67.5
45
22.5
0

79

Bio (FS)ASQA (fluency)

ASQA (rouge)

Experimental results (long-form)
Outperforms other LMs in terms of factuality & fluency correctness
LLama2-13B chat

ChatGPT

LLama2-13B chat RALM

Ours

90
67.5
45
22.5
0

80

Bio (FS)ASQA (fluency)

ASQA (rouge)

Experimental results (long-form citation precisions)
Significantly improves llama2-13B citation accuracy, matching ChatGPT

80
60
40
20
0

81

Alpaca RAG
ChatGPT RAG
ASQA (citation)

LLama2-13B chat RAG
Ours

Inference-time customization via self-reflection
Decoding-time control via reflection tokens change model behaviors

Citation accuracy

Fluency

Decreased fluency
Fully supported

x axis — Weight for Supported (larger —> more emphasis on supported)82

Impacts on academic communities and applications
New advanced RAG methods withreflections inspired by Self-RAG

Applications to expert domains(e.g., Biomedical)

CRAG(Yang et al., 2024)

EVER(Kang et al., 2024)
83

Self-BioRAG(Jeong et al., 2024)

Impacts on academic communities and applications

Self-RAG has been integrated into
LangChain, LlamaIndex … etc84Self-RAG — Self-Reflective Retrieval-Augmented Generation
An LM learns to retrieve, generate and critiqueInstruction-tuned LMs trained with fine-grained reflection tokens
Outperforms other LMs in six tasks, improving citation accuracy
https://selfrag.github.io/
https://arxiv.org/abs/2212.10511
https://huggingface.co/selfrag/selfrag_llama2_7b (13b)https://github.com/AkariAsai/self-rag (1.4k ⭐!)85

Today’s lecture
Promises and Limitations of Retrieval-augmented LMs
Reliable inference: Self-reflective RAG with dynamic retrieval
Versatile
Versatile Retriever:retriever: Intent-aware
Intent-aware retrievers
retrievers with
with LMs
LMs
Summary and future directions: RAG in the wild86

Standard RAG originally designed for a single task (e.g., QA)
How did US states get their names?

Retriever

Answer my question using references.
References:
Question: How did US states get their names?

Neural retriever

Lexical retriever(DPR)(BM 25 / TF-IDF)

Datastore(e.g., Wikipedia)

87

Capture similarities
between query and document

Modern RAG systems have been used in diverse scenarios
Implement batchnormalization in python

88

We’re planning to travel to Japan. Create afeasible 7 days travel plan to visit Tokyo,
Kyoto, Sapporo, Osaka, and Okinawa.

Modern RAG systems have been used in diverse scenarios
Implement batchnormalization in python

We’re planning to travel to Japan. Create afeasible 7 days travel plan to visit Tokyo,
Kyoto, Sapporo, Osaka, and Okinawa.

Retriever

Datastore(e.g., Github codes)

89

Datastore(e.g., Tripadvisor,
Wikipedia pages )Modern RAG systems have been used in diverse scenarios
Implement batchnormalization in python

We’re planning to travel to Japan. Create afeasible 7 days travel plan to visit Tokyo,
Kyoto, Sapporo, Osaka, and Okinawa.

Retriever Different information needs often requiredifferent information retrieval systems

Datastore(e.g., Github codes)

90

Datastore(e.g., Tripadvisor,
Wikipedia pages )Versatility of RAG systems limited by conventional retrieval
How did US states get
their names?
Implement batchnormalization in python

We’re planning to travel toJapan. Create a feasible 7days travel plan to visit
Tokyo, Kyoto, Sapporo,
Osaka, and Okinawa.

Task-specificretrievers

Versatile LM91

Trained retrieval systems typically struggle in OOD
Perform well with more task training data

Struggle in new tasks & domains
BEIR performance (BM25=100)
120
90
60
30
0
DPR

BM25

BM25+CEThakur, Nandan, et al. "BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models.” In NeurIPS (Benchmark) 2021.
92

“Similar” documents may not be always helpful
Helpful documents for some tasks aren’t necessary similar to queries

In a zoo located in a warm region, whatshould be included in the polar bear
exhibit? (Multi-hop reasoning task)
If an animal lives a certain environment
then that animal usually requires thatkind of environment.
Polar bears live in cold environments

93

BehnamGhader et al., Can Retriever-Augmented Language Models Reason? The Blame Game
Between the Retriever and the Language Model. Findings of EMNLP 2023.

Task-aware Retrieval with
Instructions
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard,
Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih

ACL Findings 2023
94

Goal of Tsk-Aware ReTriever (TART)How did US states get
their names?

Implement batchnormalization in python

Create a feasible 7 days
travel plan to visit Tokyo,
Kyoto, Sapporo, Osaka,
and Okinawa.

Versatile
Retriever

Versatile LM95Normal retrieval task
q1: How did US states
get their names?

Standard
Retriever

Datastore 𝒟
96

Topk argTopkd∈𝒟sim(q, d)New task: Retrieval with Instruction
q1: How did US states

q2 :Implement batch

get their names?

normalization in python

i1 :Wikipedia articles

i2 :Github codeabout US states names

implementing BN

q3 :Create a feasible 7days travel plan to visit
Tokyo, Kyoto, Sapporo,
Osaka, and Okinawa.

i3 :japan travel blogs

Versatile
RetrieverTopk argTopkd∈𝒟sim(q, d, i)
97New task: Retrieval with Instruction
q2 :Implement batch normalization in python

i1 : similar stack overflow
questionsi2 :Github code

implementation of BN

Topk argTopkd∈𝒟sim(q, d, i)
98

BERRI: first large-scale retrieval dataset with instructions
Curated 50 tasks with expert annotation instructions across domains

99Instruction-scheme for retrieval tasks
Propose effective instruction scheme for retrieval tasks
and annotate instructions

Intent100

DomainUnit

Instruction-aware bi-encoder retriever (TART-dual)
Inner product similarity

PositiveparagraphL(q, pEncoder

Encoder= − log

Negativeparagraphs

+

− −
−
, p1 , p2 , …, pn )+exp(sim(q, p ))n+exp(sim(q, p )) + ∑j=1 exp(sim(q, pj−))

Contrastive learning
Query
Text chunk101

Karpukhin et al.. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.

Instruction-aware bi-encoder retriever (TART-dual)
Inner product similarity

PositiveparagraphL(q, pEncoder

Encoder= − log

Negativeparagraphs

+

− −
−
, p1 , p2 , …, pn )+exp(sim(q, p ))n+exp(sim(q, p )) + ∑j=1 exp(sim(q, pj−))

Contrastive learning
Query
Instruction

102

Text chunkInstruction-aware cross-encoder retriever (TART-full)Final relevance scoref(q, p)+−L(q, P , P )MLP

FLAN-T5 Encoder

=−

∑
+
+

+

log( f(q, p )) −

p ∈P

∑−

log(1 − f(q, p ))p∈P

Cross-entropy loss
Instruction

Query
Text chunk103

retrieve candidates with an efficient BE

Nogueira and Cho. Passage Re-ranking with BERT. ArXiv 2019.

−

New negative samples: instruction un-following samples
Carefully defining negative samples is a key for successful retrievers

Follow instruction?
Relevant to the query?
104

Experimental settings

More details of training & test are in our paper!

• Our models• TART-dual (bi-encoder) - Instruction-tuning of Contriever-MS MARCO 110M
• TART-full (cross-encoder) - instruction-tuning FLAN-T5 3B Encoder (1.5B)• Evaluations• Zero-shot retrieval: generalize to new retrieval task via instruction• BEIR (Thakur et al. 2021)• LOTTE (Khattab et al., 2022)• Cross-task cross-domain retrieval: synthetically combine retrieval tasks to test

instruction following by pairing two relevant tasks (e.g., QA & question retrieval)105

Better Generalization and Instruction Following
Instruction-tuning for retrieval (3B) shows effectivenessContriever

Contriever+CE

TART Dual

TART Full

60
45
30
15
0
106

BEIR

Cross-Task

Increasing number of instruction-following retrievers
Instructor

107

ControlRetriever(Su et al., 2023)(Pang et al., 2023)

Concurrent work focusing on embeddings
trained on existing datasets

Leverage LLMs to generate training data
E5 MISTRAL Instruct (Wang et al., 2024)

Increasing number of instruction-following retrievers
Instruction following retrievers are now dominating retrieval tasks!

https://huggingface.co/spaces/mteb/leaderboard
108

Versatile retrievers improves RAG in text and multi-modal
GRIT(Muennighoff et al., 2024)

UniLR(Wei et al., 2023)

109

Are those retrievers following instructions?
Instructir (Oh et al., 2024)
Create a natural instruction-following
retrieval dataset with LLMs

110

TART (110M) outperforms other <7B
models by large margins

Are those retrievers following instructions?
Instructir (Oh et al., 2024)
Create a natural instruction-following
retrieval dataset with LLMs

111

Improvements from base retrievers
are limited

Are those retrievers following instructions?
FollowIR (Welleer et al., 2024)

All small BE models struggles tofollow instructions

112

Are those retrievers following instructions?
FollowIR (Welleer et al., 2024)

CE models or 7B BE models
have shown superiority
Introduce significant
inference latency!

113

Today’s lecture
Promises and Limitations of Retrieval-augmented LMs
Advanced Retriever: Intent-aware LM-based retrievers
Advanced RAG: Self-reflective LMs with dynamic Retrievals
Future:Summary
RAG inandthefuturewild —
directions:efficiencyRAG
andin
applications
the wild
114

Summary• Understanding retrieval-augmented LMs (Asai et al., 2024b; Mallen*, Asai et al., 2023)• Retrieval-augmented LMs can alleviate many issues in parametric LMs• More fundamental improvements for architectures or training is necessary• Advancing Retrieval-augmented LMs (Asai et al., 2024a; Asai et al., 2023)• Self-RAG to build versatile retrieval-augmented LMs addressing issues in RAG• Task-aware Retrieval with Instructions to redesign the conventional “retrieval"

115Summary• Understanding Retrieval-augmented LMs (Asai et al., 2024b; Mallen*, Asai et al.,2023)

• Retrieval-augmented LMs can alleviate many issues in parametric LMs.
• More fundamental improvements for architectures or training is necessary• Advancing RAG (Asai et al., 2024; Asai et al., 2023)• Self-RAG to build versatile retrieval-augmented LMs addressing issues in RAG• Task-aware retrievals to build versatile RAG systems

116

Reliable RAG in the wild: improving efficiency
Efficiency

Effective datastore

Vanilla

+Datastore+

Latency (sec / query)LM

New application

Retrieval-augmented

8
6
4
2
0

GPT-j 6B

GPT-neox-20B

Mallen*, Asai* et al., When Not to Trust Language Models:Investigating Effectiveness of Parametric and Non-Parametric Memories 2023.
117

Reliable RAG in the wild: efficient algorithms / arc. for RAG
Efficiency

Effective datastore

New application

GRIT

BTR

(Muennighoff et al., 2024)(Cao et al., 2024)

Caching? New architecture? Adaptivity? Quantization?

118

Reliable RAG in the wild: effective datastore (scale)
Efficiency

Effective
Scaling datastores
datastoreLM

New application

RETRO(Borgeaud et al., 2021)

Datastore
vs.

LM
119

Reliable RAG in the wild: effective datastore (scale)
Efficiency

Effective
Scaling datastores
datastore

LM

Datastore
vs.

LM

LM

Datastore# of parameters# of tokens

kNN-LM (Khandelwal et al., 2020)

250M

≤ 3B

NPM (Min et al., 2023)

350M

1B

Atlas (Izacard et al., 2022)

11B

~30B

RETRO (Borgeaud et al., 2021)

7B

2T≤ 175B~5B

How should we scale RAG?

REPLUG (Shi et al., 2023)
120

New application

Reliable RAG in the wild: effective datastore (quality)
Efficiency

Effective
Scaling datastores
datastore

New application

Data-centric approaches to build effective large-scale datastore?

121

Quality & Composition

Deduplication

Data Filtering(Longpre et al., 2023)(Lee et al., 2023)(Paster et al., 2023)

Reliable RAG in the wild: scaling datastore
Efficiency

Medical122

Scaling datastores

Science

Legal

BeyondNew application
general QA

CodeFinanceReliable RAG in the wild: scaling datastore
Efficiency

MedRAG(Xiong et al., 2024)

Scaling datastores

BeyondNew application
general QA

Automous chemistry experiments(Boiko et al., 2024)

Science
Realistic evaluation?
123

RAG pipelines optimized for expert tasks?

Thanks for listening :)• Understanding RAG (Asai et al., 2024b; Mallen*, Asai* et al., 2023)• Retrieval-augmented LMs can alleviate many issues in parametric LMs.
• More fundamental improvements for architectures or training is necessary• Advancing RAG (Asai et al., 2024a; Asai et al., 2023)• Self-RAG to build versatile retrieval-augmented LMs addressing issues in RAG• Task-aware retrievals to build versatile RAG systems
ACL 2023 tutorial: https://acl2023-retrievallm.github.io/ by Akari, Sewon, Zexuan and Danqi
RAG survey: Retrieval-augmented Generation forLarge Language Models: A Survey (Gao et al., 2024)
124

124

Contact: akari@cs.washington.edu
Website: https://akariasai.github.io/
Twitter: @AkariAsai
Public OH: Friday 6pm