Self-RAG: Learning to Retrieve, Generate andCritique through Self-Reflection
selfrag.github.io

Akari Asai

1

Zeqiu Wu
1

1

Yizhong Wang

1

2

May 7, 2024 | ICLR 2024

Avirup Sil

3

Hannaneh Hajishirzi
312Retrieval-augmented generation (RAG)

Where is ICLR 2024?

2

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

2

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

2

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

2

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

2

The Twelfth annual conference is held
Tue. May 7th through Sat the 11th, 2024 at
the Messe Wien Exhibition Congress
Center.

International conference onlearning representaiton 2024 will beheld in Vienna Austria.

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

2

The Twelfth annual conference is held
Tue. May 7th through Sat the 11th, 2024 at
the Messe Wien Exhibition Congress
Center.

International conference onlearning representaiton 2024 will beheld in Vienna Austria.

Answer my question using references.
References:
Question: Where is ICLR 2024?

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

2

The Twelfth annual conference is held
Tue. May 7th through Sat the 11th, 2024 at
the Messe Wien Exhibition Congress
Center.

International conference onlearning representaiton 2024 will beheld in Vienna Austria.

Step 2: Prompt LM with K docs and generate
Answer my question using references.
References:
Question: Where is ICLR 2024?

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

2

Step 2: Prompt LM with K docs and generate
Answer my question using references.
References:
Question: Where is ICLR 2024?

The Twelfth annual conference is held
Tue. May 7th through Sat the 11th, 2024 at
the Messe Wien Exhibition Congress
Center.

International conference onlearning representaiton 2024 will beheld in Vienna Austria.

LMs(e.g., GPT-4, Llama 3)
ICLR 2024 will be in Vienna, Austria.

factualup-to-date

attributable

Retrieval-augmented generation (RAG)
Step 1: Retrieve K documents
Where is ICLR 2024?

Retriever(e.g., Google
BM 25)

Step 2: Prompt LM with K docs and generate
Answer my question using references.
References:
Question: Where is ICLR 2024?

The Twelfth annual conference is held
Tue. May 7th through Sat the 11th, 2024 at
the Messe Wien Exhibition Congress
Center.

International conference onlearning representaiton 2024 will beheld in Vienna Austria.

LMs(e.g., GPT-4, Llama 3)
ICLR 2024 will be in Vienna, Austria.

factualup-to-date

g

g

g

g

g

g

g

2

g

Chen et al. Readin Wikipedia to Answer Open-domain Questions. ACL 2017.
Lewis et al. Retrieval-au mented Generation for Knowled e Intensive Tasks. NeurIPS 2020.
Ram et al. In-Context Retrieval-Au mented Lan ua e Models. TACL 2023
Asai et al. Reliable, Adaptable, and Attributable Lan ua e Models with Retrieval. Arxiv 2024.

attributableRAG has been widely and successfully adopted

3

RAG has been widely and successfully adopted
RAG has shown e ective
in many benchmarks

QA

Mallen*, Asai* et al., When Not to Trust Lan ua e Models: Investi atin
Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
g

g

g

g

ff

3

RAG has been widely and successfully adopted
RAG has shown e ective
in many benchmarks

Widely adopted in real-world
systems

QA

Mallen*, Asai* et al., When Not to Trust Lan ua e Models: Investi atin
Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
g

g

g

g

ff

3

RAG has been widely and successfully adopted
RAG has shown e ective
in many benchmarks

Widely adopted in real-world
systems
RAG-based search systems

QA

Mallen*, Asai* et al., When Not to Trust Lan ua e Models: Investi atin
Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
g

g

g

g

ff

3

RAG has been widely and successfully adopted
RAG has shown e ective
in many benchmarks

Widely adopted in real-world
systems
RAG-based search systems

QA

Libraries for customized RAG

Mallen*, Asai* et al., When Not to Trust Lan ua e Models: Investi atin
Effectiveness of Parametric and Non-Parametric Memories. ACL 2023.
g

g

g

g

ff

3

Standard RAG isn’t (always) reliable4Standard RAG isn’t (always) reliable

How did US states get their names?

4

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

f4

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)f

4

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)f

5

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Easily distracted by unhelpful
documentsf

5

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Easily distracted by unhelpful
documentsf

5

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Easily distracted by unhelpful
documentsf

5

US
US states
states got
got their
their names
names fromfrom aa variety
variety ofof sources.
sources.
Elevenstates
such
as
New
York
and
MichiganareEleven states such as New York and Michigan are
named
after
an
individual
person.
Somestatesnamed after an individual person. Some states
including Utah and Washington, are named after
including Utah and Washington, are named after
indigenous communities.
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)f

6

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Ignore relevant context despite
adding citationsf

6

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Ignore relevant context despite
adding citationsf

6

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
named after an individual person. Some states
including Utah and Washington, are named after
indigenous communities.

Standard RAG isn’t (always) reliable
Step 1: Retrieve K documents
How did US states get their names?
Of the ifty states, eleven including
New York, Georgia, Washington named
after an individual person.

Step 2: Prompt LM with K docs and generate
Prompt How did US states get their names?

LM

The history of human activity in
Michigan began with settlement byPaleo-Indians.

Retriever
UTAH: Name taken from the Ute(e.g., Google
Indians who inhabited that region
BM 25)

Ignore relevant context despite
adding citationsf

6

US states got their names from a variety of sources.
Eleven states such as New York and Michigan are
individual person.
person. Some
Some statesstatesnamed after an individual
including Utah
including
Utah andand Washington,
Washington, areare named
named after
indigenous communities.

Standard RAG isn’t efficient
Write an essay about your summer vacation

7

Standard RAG isn’t efficient
Write an essay about your summer vacation

The term summer vacation orsummer break refers to a school break inthe summer between school years.

RetrieverSummer Vacation (Korean: 여름방학)is a South Korean “home vacation” reality
show
Summer Vacations a 2023 Spanish
comedy ilm directed by Santiago Segura
which stars Segura, Leo Harlem, Cristina
Gallego, and Patricia Conde.

f7

Standard RAG isn’t efficient
Write an essay about your summer vacation

The term summer vacation orsummer break refers to a school break inthe summer between school years.

RetrieverSummer Vacation (Korean: 여름방학)is a South Korean “home vacation” reality
show
Summer Vacations a 2023 Spanish
comedy ilm directed by Santiago Segura
which stars Segura, Leo Harlem, Cristina
Gallego, and Patricia Conde.

fi

ffi

f

7

Always retrieving xed number of
documents is ine cient & harmfulSelf-RAG: Self-reflective Retrieval-Augmented Generation

8

Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens

fl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input

fl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval neededfl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval neededfl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

User input

fl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

User input

fl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

User input

Retrieval neededfl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

Retrieval needed

User input

Retrievefl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

Retrieval needed

User input

Retrievefl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

Retrieval needed

User input

Retrievefl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
No retrieval needed

Retrieval needed

User input

Retrievefl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Retrieve
Irrelevant

fl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Retrieve
Irrelevant

Identify unhelpful evidences

fl8

Avoid unnecessary
retrievalSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval
Relevant

Retrieve

Relevant
Irrelevant

Identify unhelpful evidencesfl8

Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval
Relevant

Retrieve

Relevant
Irrelevant

Identify unhelpful evidences

fl8

Generation 1

SupportedSelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve
Irrelevant

Identify unhelpful evidences

fl8

Relevant

Generation 1

Supported

Relevant

Generation 2

PartiallySelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve
Irrelevant

Identify unhelpful evidences

fl8

Relevant

Generation 1

Supported

Relevant

Generation 2

PartiallySelf-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve
Irrelevant

Identify unhelpful evidences

Relevant

Generation 1

Supported

Relevant

Generation 2

Partially

Generate output supported by evidence

fl8Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve

Relevant

Generation 1

Supported

Relevant

Generation 2

Partially

Irrelevant

Identify unhelpful evidences

Generate output supported by evidence

Self-RAG training: Instruction-tuning an arbitrary LM with re ection tokens

fl

fl

8

Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve

Relevant

Generation 1

Supported

Relevant

Generation 2

Partially

Irrelevant

Identify unhelpful evidences

Generate output supported by evidence

Self-RAG training: Instruction-tuning an arbitrary LM with re ection tokens

fl

fl

8

Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrieval

Retrieve

Relevant

Generation 1

Supported

Relevant

Generation 2

Partially

Irrelevant

Identify unhelpful evidences

Generate output supported by evidence

Self-RAG training: Instruction-tuning an arbitrary LM with re ection tokens

fl

fl

9

Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

10Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

10Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

US states got their names
from a variety of sources.

10Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

US states got their names
from a variety of sources. Retrieve

10Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

US states got their names
from a variety of sources. Retrieve

10Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

Of the ifty states, eleven
including New York, Georgia,
Washington named after anindividual person.
The history of human activity
in Michigan began withsettlement by Paleo-Indians.

US states got their names
from a variety of sources. Retrievef

10

UTAH: Name taken from the
Ute Indians who inhabited that
regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 1: Retrieve documents on demand
How did US states get their names?

Of the ifty states, eleven
including New York, Georgia,
Washington named after anindividual person.
The history of human activity
in Michigan began withsettlement by Paleo-Indians.

US states got their names
from a variety of sources. Retrievef

10

UTAH: Name taken from the
Ute Indians who inhabited that
regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptOf the ifty states, eleven states
named after an individual person.

f11

Prompt

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

Prompt

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

Irrelevant

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

Irrelevant

Many states including Michigan

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

Irrelevant

Many states including Michigan

are named after an individual person.

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

Irrelevant

Many states including Michigan

Relevant

are named after an individual person.

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

Irrelevant

Many states including Michigan

Relevant

Several states such as Utah andare named after an individual person.

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

11

Irrelevant

Many states including Michigan

are named after an individual person.

The history of humanactivity in Michigan began withsettlement by Paleo-Indians.

Relevant

Several states such as Utah andAlabama are named after indigenous communities.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Of the ifty states, eleven states
named after an individual person.

f

12

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after indigenous communities.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Supported

Of the ifty states, eleven states
named after an individual person.

f

12

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after indigenous communities.

UTAH: Name taken from the Ute
Indians who inhabited that regionSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 2: Generate segments in parallel
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Supported

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after indigenous communities.
Partially

Of the ifty states, eleven states
named after an individual person.

f

12

UTAH: Name taken from the Ute
Indians who inhabited that regionReflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,A
Original LM vocabularies

13

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,
A

Vocabulary expanded withre ection tokens
fl14

Original LM vocabularies

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,
A

Retrieve
No Retrieval

Vocabulary expanded withre ection tokens
fl14

Original LM vocabularies

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,

Original LM vocabularies

A

Retrieve
No Retrieval

Vocabulary expanded withre ection tokens
fl14

Retrieval tokens

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,

Original LM vocabularies

A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withre ection tokens
fl14

UsefulNot useful

Retrieval tokens

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,

Original LM vocabularies

A

Retrieve
No Retrieval

Retrieval tokens

Relevant
Irrelevant

Supported
No support

Vocabulary expanded withre ection tokens
fl14

UsefulNot useful

Critique tokens

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,
A

Retrieve
No Retrieval

Relevant
Irrelevant

Supported
No support

UsefulNot useful15

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,
A

Retrieve
No Retrieval

Relevant

Adaptive retrieval via retrieval
re ection tokens

Irrelevant

Supported
No support

UsefulNot usefulfl

15

Reflection tokens for retrieval and critique
California
US states got their names
from a variety of sources.

e.g.,
A

Retrieve
No Retrieval

Relevant

Adaptive retrieval via retrieval
re ection tokens

Irrelevant

Supported
No support

UsefulNot usefulfl15

Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 3: Evaluate and choose best k outputs using re ection tokens
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Supported

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Alabama are named after native American tribes.
Partiallyfl16

Several states such as Utah andSelf-RAG — Self-Reflective Retrieval-Augmented Generation
Step 3: Evaluate and choose best k outputs using re ection tokens
PromptPromptPromptBatch decoding

Relevant

Irrelevant

11 of 50 state names

come from persons.

Supported

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after native American tribes.
Partially

Critique token probabilities used as ne-grained feedback scores on each output
fl

fi

16

Self-RAG — Self-Reflective Retrieval-Augmented Generation
Step 3: Evaluate and choose best k outputs using re ection tokens
PromptPromptPromptBatch decoding

Relevant

11 of 50 state names

come from persons.

Supported

Irrelevant

Many states including Michigan

are named after an individual person.

Relevant

Several states such as Utah andAlabama are named after native American tribes.
Partially17

0.9

f( Irrelevant )0.1

flf( Relevant Supported )f( Relevant

Partially

0.4)

Self-RAG: Self-reflective Retrieval-Augmented Generation
Self-RAG Inference: Learn to retrieve and evaluate via special tokens
User input

User input

No retrieval needed

Retrieval needed

Avoid unnecessary
retrievalSupported

Retrieve

Partially

Irrelevant

Identify unhelpful evidences

Generate output supported by evidence

Self-RAG training: Instruction-tuning an arbitrary LM with re ection tokens

fl18

Overview of Self-RAG training
Calif
,
A

How did US states get their names?

No

Of the ifty states, eleven are
named after an individual person.

Generator LM

supporusefulRetrieverf

19Overview of Self-RAG training
Calif
,
A

How did US states get their names?

No

Of the ifty states, eleven are
named after an individual person.

Irrelev

Generator LM

No
Not

Critic LM teaches Generator LM topredict re ection tokens
Training time only Critic LM

f

fl

20

Self-RAG training - Critic LM
Evaluate if the output y to an input x is
supported by retrieved doc d:x How did US states get their names?
dOf the ifty states, eleven are named
after an individual person.

y

11 of 50 state names come from persons.

f21

Critic LM

Self-RAG training - Critic LM
Evaluate if the output y to an input x is
supported by retrieved doc d:x How did US states get their names?
dOf the ifty states, eleven are named
after an individual person.

y

11 of 50 state names come from persons.

f21

Supported

Critic LM

Self-RAG training - Critic LM
Evaluate if the output y to an input x is
supported by retrieved doc d:x How did US states get their names?
d

𝒟

y

f

21

Of the ifty states, eleven are named
after an individual person.
11 of 50 state names come from persons.

Supported

Critic LM
critic = {(x, y, d), r}

Self-RAG training - Critic LM
Evaluate if the output y to an input x is
supported by retrieved doc d:

max

|
log
p
(r
x,
y,
d)
((x,y,d),r)∼ critic

x How did US states get their names?
dSupported

Of the ifty states, eleven are named
after an individual person.

yCritic LM

𝒟

𝔼
f

21

𝒞

𝒟

𝒞

11 of 50 state names come from persons.

critic = {(x, y, d), r}

Self-RAG training - augmented instruction-tuning data
150k (input, output) instances from diverse instruction-following data

22

Self-RAG training - Generator LM

x How did US states get their names?
d

f

23

Of the ifty states, eleven are named
after an individual person.

Generator LM

Self-RAG training - Generator LM

x How did US states get their names?
d

f

23

Of the ifty states, eleven are named
after an individual person.

Generator LMy

11 of 50 state names
come from persons.

rSupportedSelf-RAG training - Generator LM
max
ℳ

|
log
p
(y,
r
x,
d)
ℳ
gen

(x,d,y,r)∼

x How did US states get their names?
Of the ifty states, eleven are named
after an individual person.

𝔼

𝒟

d

f

23

Generator LMy

11 of 50 state names
come from persons.

rSupportedSelf-RAG training - Generator LM
max
ℳ

|
log
p
(y,
r
x,
d)
ℳ
gen

(x,d,y,r)∼Train with a standard next token
objective with expanded vocabulary

x How did US states get their names?
Of the ifty states, eleven are named
after an individual person.

𝔼

𝒟

d

f

23

Generator LMy

11 of 50 state names
come from persons.

rSupportedSelf-RAG training - Generator LM
max
ℳ

|
log
p
(y,
r
x,
d)
ℳ
gen

(x,d,y,r)∼Train with a standard next token
objective with expanded vocabulary

x How did US states get their names?
Of the ifty states, eleven are named
after an individual person.

dGenerator LM

𝔼

𝒟

Easily applied to new pre-trained LM

f

23

y

11 of 50 state names
come from persons.

rSupportedExperimental details

More details of training & test are in our paper!

Tasks and datasets• Closed-set tasks (classi cations, multiple-choice QA)• ARC-Challenge (Clark et al., 2018)• PubHealth (Zhang et al., 2023)• Short-form generation• OpenQA - PopQA (Mallen* and Asai* et al., 2023)• Trivia QA (Joshi et al., 2017)• Long-form generation• ASQA-ALCE [ uency, citation accuracy, correctness] (Gao et al., 2023)• Bio generations [FactScore] (Min et al., 2023)

fi

fl

24

Experimental details

More details of training & test are in our paper!

Tasks and datasets• Closed-set tasks (classi cations, multiple-choice QA)• ARC-Challenge (Clark et al., 2018)• PubHealth (Zhang et al., 2023)• Short-form generation• OpenQA - PopQA (Mallen* and Asai* et al., 2023)• Trivia QA (Joshi et al., 2017)• Long-form generation• ASQA-ALCE [ uency, citation accuracy, correctness] (Gao et al., 2023)• Bio generations [FactScore] (Min et al., 2023)

fi

fl

24

Experimental details

More details of training & test are in our paper!

Tasks and datasets• Closed-set tasks (classi cations, multiple-choice QA)• ARC-Challenge (Clark et al., 2018)• PubHealth (Zhang et al., 2023)• Short-form generation• OpenQA - PopQA (Mallen* and Asai* et al., 2023)• Trivia QA (Joshi et al., 2017)• Long-form generation• ASQA-ALCE [ uency, citation accuracy, correctness] (Gao et al., 2023)• Bio generations [FactScore] (Min et al., 2023)

fi

fl

24

Experimental details

More details of training & test are in our paper!

Training details• Critic training data: 4k-20k instances for each type
• Generator training data: 150k instruction-following datasets• ShareGPT• OpenAssistant• Alpaca• FLANV2• Natural Questions ….
• Base LMs: Llama2-7B, 13B (Touvron et al., 2023)• Computation: 4*A100 (15 hours)
25Experimental results

Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
26

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE) ASQA (Citation Precision)
Llama2 13B
Self-RAG 13B

ChatGPT + RAG

Experimental results
Parametric LMs struggle with tasks requiring precise world knowledge
Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
26

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE) ASQA (Citation Precision)
Llama2 13B
Self-RAG 13B

ChatGPT + RAG

Experimental results
Parametric LMs struggle with tasks requiring precise world knowledge
Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
26

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE) ASQA (Citation Precision)
Llama2 13B
Self-RAG 13B

ChatGPT + RAG

Experimental results

Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
27

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results

Accuracy &precision (%)

RAG gives large gains on some tasks, but not all

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
27

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results

Accuracy &precision (%)

RAG gives large gains on some tasks, but not all

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
27

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results

Accuracy &precision (%)

RAG gives large gains on some tasks, but not all

90
67.5
45

Limited citation prec. esp. with open LMs

22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
27

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results

Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
28

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results
Self-RAG outperforms open LMs in same scale using standard RAG
Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
28

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results
Self-RAG outperforms open LMs in same scale using standard RAG
Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
28

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results
Self-RAG outperforms open LMs in same scale using standard RAG
Accuracy &precision (%)

90
67.5
45

Especially large improvements in citation precisions

22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
28

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results

Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
29

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Experimental results
Self-RAG also outperforms ChatGPT with and without RAG
Accuracy &precision (%)

90
67.5
45
22.5
0

PopQA

PubHealth

ChatGPT
LLama2 Chat 13B + RAG
29

Bio generation
LLama2 Chat 13B
Llama2 13B + RAG

ASQA (ROUGE)
Llama2 13B
Self-RAG 13B

ASQA (citation precision)
ChatGPT + RAG

Followup of Self-RAG in academia and industry
Integrated into widely-adopted RAG libraries

30

Followup of Self-RAG in academia and industry
Integrated into widely-adopted RAG libraries

Improving Self-RAG for further reliability

CRAG(Yang et al., 2024)

30

Followup of Self-RAG in academia and industry
Integrated into widely-adopted RAG libraries

Improving Self-RAG for further reliability

CRAG(Yang et al., 2024)

Applications to expert
domains

Self-BioRAG(Jeong et al., 2024)

30

Thank you so much for listening!
Self-RAG: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Retrieve

Self-re ection to improve reliability of RAG

fl

ffi

31
fl

Avoid unnecessary
retrieval for e ciency
Relevant

Generation 1

Supported

Relevant

Generation 2

Partially

Instruction-tuning to build versatile RAG

Thank you so much for listening!
Self-RAG: Learn to retrieve and re ect via special tokens
User input
User input

No retrieval needed

Retrieval needed

Retrieve

Self-re ection to improve reliability of RAG

Question?
akari@cs.washin ton.edu
akariasai. ithub.io/

Poster: Hall B #63

Generation 1

Supported

Relevant

Generation 2

Partially

Project pa e: selfra . ithub.io/
Paper (Open Review): openreview.net/forum?id=hSyW5 o0v8
HF checkpoints: hu in face.co/selfra /selfra _llama2_7b (13b)Github: ithub.com/AkariAsai/self-ra (1.4k ⭐!)
g

g

fl

g

g

g

g

g

g

g

g

ffi

g

g

g

Relevant

Instruction-tuning to build versatile RAG

31
fl

Avoid unnecessary
retrieval for e ciency