Section 5
Applications:

1

Downstream adaptation of retrieval-based LMs

The capital city of Ontario is __Datastore

LM
Toronto, which is known for …
2

Downstream adaptation of retrieval-based LMs
Open-domain QA

What are the tasks?

What is the capital of Ontario?

Datastore

LM

Toronto3

Downstream adaptation of retrieval-based LMs
What are the tasks?

Fact veri cation

Ottawa is the Ontario state capital.

Datastore

LM
False

fi

4

A range of target tasks
Question Answering

Fact veri cation

Dialogue

RETRO (Borgeaud et al., 2021)

RAG (Lewis et al, 2020)

BlenderBot3 (Shuster et al., 2022)

REALM (Gu et al, 2020)

ATLAS (Izacard et al, 2022)

ATLAS (Izacard et al, 2023)

Evi. Generator (Asai et al, 2022)

Internet-augmented generation(Komeili et a., 2022)

Retrieval-based LMs have been extensively
evaluated on knowledge-intensive tasksfi5

A range of target tasks
Question answering

Fact veri cation

Dialogue

RETRO (Borgeaud et al., 2021)

RAG (Lewis et al, 2020)

BlenderBot3 (Shuster et al., 2022)

REALM (Gu et al, 2020)

ATLAS (Izacard et al, 2022)

ATLAS (Izacard et al, 2023)

Evi. Generator (Asai et al, 2022)

Internet-augmented generation(Komeili et a., 2022)

Summarization

Machine translation

Code & proof generation

FLARE (Jiang et al, 2023)

kNN-MT (Khandelwal et al., 2020)

DocPrompting (Zhou et al., 2023)

TRIME-MT (Zhong et al., 2022)

Natural Prover(Welleck et al., 2022)

NLI

Sentiment analysis

Commonsense reasoningkNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

kNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

Raco (Yu et al, 2022)

fi

More general NLP tasks

6

A range of target tasks
Question answering

Fact veri cation

Dialogue

RETRO (Borgeaud et al., 2021)

RAG (Lewis et al, 2020)

BlenderBot3 (Shuster et al., 2022)

REALM (Gu et al, 2020)

ATLAS (Izacard et al, 2022)

ATLAS (Izacard et al, 2023)

Evi. Generator (Asai et al, 2022)

Internet-augmented generation(Komeili et a., 2022)

Summarization

Machine translation

Code & proof generation

FLARE (Jiang et al, 2023)

kNN-MT (Khandelwal et al., 2020)

DocPrompting (Zhou et al., 2023)

TRIME-MT (Zhong et al., 2022)

Natural Prover(Welleck et al., 2022)

NLI

Sentiment analysis

Commonsense reasoningkNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

kNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

Raco (Yu et al, 2022)

fi

More generations

7

A range of target tasks
Question answering

Fact veri cation

Dialogue

RETRO (Borgeaud et al., 2021)

RAG (Lewis et al, 2020)

BlenderBot3 (Shuster et al., 2022)

REALM (Gu et al, 2020)

ATLAS (Izacard et al, 2022)

ATLAS (Izacard et al, 2023)

Evi. Generator (Asai et al, 2022)

Internet-augmented generation(Komeili et a., 2022)

Summarization

Machine translation

Code & proof generation

FLARE (Jiang et al, 2023)

kNN-MT (Khandelwal et al., 2020)

DocPrompting (Zhou et al., 2023)

TRIME-MT (Zhong et al., 2022)

Natural Prover(Welleck et al., 2022)

NLI

Sentiment analysis

Commonsense reasoningkNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

kNN-Prompt (Shi et al., 2022)
NPM (Min et al., 2023)

Raco (Yu et al, 2022)

ffi

More classi ications

8

Two key questions for downstream adaptations
How can we adapt a retrieval-based LM for a task?

When should we use a retrieval-based LM?

9How to adapt a retrieval-based LM for a task
What are the tasks?

How to adapt?

- Open-domain QA- Other knowledge-- Fine-tuning- Reinforcement learning- Prompting

intensive tasks- Sentiment analysis- Code generation…

10

How to adapt a retrieval-based LM for a task
Fine-tuning (+RL)
Training LM and / or retrieveron task-data & data store

Pθ1(z | x)
RetrieverLMPθ2(y | x, z)
11How to adapt a retrieval-based LM for a task
PromptingFine-tuning (+RL)
Training LM and / or retrieveron task-data & data store

Pθ1(z | x)
Retriever

Prompt a frozen LM withretrieved knowledge

Should we update
both retriever & LM?

RetrieverWhere toincorporate?

Pθ1(z | x)LMLMPθ2(y | x, z)

How to prompt
LM?
12

How to adapt a retrieval-based LM for a task
What are the tasks?

How to adapt?

- Open-domain QA- Other knowledge-- Fine-tuning- Reinforcement learning- Prompting

intensive tasks- Sentiment analysis- Code generation…

What is data store?

Wikipedia

Training data
Codedocumentation
13

When to use a retrieval-based LM

Long-tailknowledge
updateVeri ability

Parameteref ciency

fi

fi

14

Effectiveness of retrieval-based LMs
Long-tailknowledge
update

Q: Is Toronto reallycold during winter?

Veri ability

LM

Parameteref ciency

Yes it is.

fi

fi

15

Effectiveness of retrieval-based LMs
Long-tailknowledge
update

Q: Where is Toronto
Zoo located?

Veri ability

LM

Parameteref ciency

1361A Old Finch Avenue,
in Scarborough, Ontario

fi

fi

16

Effectiveness of retrieval-based LMs
Long-tailknowledge
update

Q: Where is Toronto
Zoo located?

LM

fi

ti

RetrieverfiVeri ability

Parameteref ciency

361A Old Finch Avenue,
in Scarborough, Ontario

Toronto zoo Info
Loca on: 361A Old Finch Avenue,
Toronto, Ontario
Land Area: 287 hectares

17Effectiveness of retrieval-based LMs
Long-tail

Knowledge
update

Q: What is the populationof Toronto Metropolitan
area in 2023?

Veri ability

LM

Parameteref ciency

A: 6,255,000

Trained on the2021 corpus

fi

fi

2021

2023

18

Effectiveness of retrieval-based LMs
Long-tail

Knowledge
update

Q: What is the populationof Toronto Metropolitan
area in 2023?

fi

fi

Collected in 2023

Veri ability

Parameteref ciency

Trained on the2021 corpus

LM

Retriever

A: 6,372,000

2021

2023

19

Effectiveness of retrieval-based LMs
Long-tailknowledge
update

Q: Where is Toronto
Zoo located?

LM

fi

ti

RetrieverfiVeri ability

Parameteref ciency

361A Old Finch Avenue,
in Scarborough, Ontario

Toronto zoo Info
Loca on: 361A Old Finch Avenue,
Toronto, Ontario
Land Area: 287 hectares

20Effectiveness of retrieval-based LMs
Long-tailknowledge
updateLM

Veri ability>>>>

Parameteref ciency

LM

Retriever

fi

fi

21

Two key questions for downstream adaptations
How can we adapt a retrieval-based LM for a task?

When should we use a retrieval-based LM?

22

Downstream adaptation of retrieval-based LMs
What are the tasks?
- Open-domain QA- Other knowledgeintensive tasks- General NLU- Language Modeling &other generation tasks

How to adapt?

- Fine-tuning- Reinforcement
learning- Prompting

What is data store?

- Unlabeled

Wikipedia / CC
- Web (Google / Bing
Search Results)- Training data

23

Adapting retrieval-based LMs for tasks
Fine-tuning
Training LM and / or retrieveron task-data & data store

RetrieverPθ1(z | x)LMPθ2(y | x, z)
24Adapting retrieval-based LMs for tasks
Fine-tuning
Training LM and / or retrieveron task-data & data store

RetrieverPθ1(z | x)LM

Costs of retrieval-based LM
training (Section 4)
Independent training (DPR)
Asynchronous updates (REALM)…

Pθ2(y | x, z)
25

ATLAS (Izacard et al., 2022; Section 4)
Input x

Few-shot task data
(e.g., k=64)

Query qLM

Queryencoder
IndexDoc
encoderDocs zOutput y

QA

QA QA(xi , yi )Factveri cation

FV FV(xi , yi )Dialoguedial dial(xi , yi )fiIzacard et al. 2022. “Few-shot learning with retrieval augmented language models”

26

ATLAS (Izacard et al., 2022; Section 4)
Input x

Few-shot task data
(e.g., k=64)

Query qLM

QueryencoderDocs zIndex
Doc
encoder

QA

QA QA(xi , yi )Factveri cation

FV FV(xi , yi )Dialoguedial dial(xi , yi )Backpropagate

Output yfi

27

ATLAS (Izacard et al., 2022; Section 4)
Fully updating index
during training is
expensive!

Input x

Few-shot task data
(e.g., k=64)

Query qLM

QueryencoderDocs zIndex
Doc
encoder

QA

QA QA(xi , yi )Factveri cation

FV FV(xi , yi )Dialoguedial dial(xi , yi )Backpropagate

Output yfi

28

ATLAS: Fixed retrieval with ne-tuned LM
Input x

Few-shot task data
(e.g., k=64)

Query qLM
Index is xed

Query
encoderDocs z

Index

QA

QA QA
(xi , yi )Factveri cation

FV FV(xi , yi )Dialoguedial dial(xi , yi )No back-propDoc
encoder

Output y

fi

fi

fi

29

ATLAS: Query-side ne-tuning
Input x

Few-shot task data
(e.g., k=64)

Query qLM

QueryencoderDocs z

Index

QA

QA QA
(xi , yi )Factveri cation

FV FV(xi , yi )Dialoguedial dial(xi , yi )No back-propDoc
encoder

Output y

fi

fi

30

Ablations of ef cient retrieval training
80

60

EM (NQ)
F1 (F1)
Accuracy (FEVER)

40

20

0

Standard FTFixed FT
Query-side FTNQWoW (Dialogue)

FEVER (Fact veri cation)

Fixed FT shows large performance drop on QA.

fi

fi

31

Ablations of ef cient retrieval training
80

60

EM (NQ)
F1 (F1)
Accuracy (FEVER)

40

20

0

Standard FTFixed FT
Query-side FTNQWoW (Dialogue)

FEVER (Fact veri cation)

Query-side ne-tuning matches or outperforms full ne-tuning

fi

fi

fi

fi

32

ATLAS: Few-shot v.s. full v.s. transfer setups
Few-shot task data
(e.g., k = 64)

NQ

QA QA
(x1 , y1 )QA QA
… (xk , yk )MMLU
MMNU MMNU
(x1
, y1
)

…
MMNU MMNU
(xk
, yk
)

Full-shot task data
(e.g., N = 70,000)

Transfer task data
(e.g., MCTest —> MMLU)NQ

MCTest

QA QA(x1 , y1 )TF TF
(x1 , y1 )…
QA QA
(xN , yN )

…

TF TF
(xN , yN )Kwiatkowski et al. 2019. “Natural Questions: A Benchmark for Question Answering Research”
Hendrycks et al. 2021. “Measuring Massive Multitask Language Understanding”

33

Task results
80

On QA, ATLAS largely outperforms
other LLMs in few-shot

60

EM

40

20

Chinchilla (70B)
ATLAS (Few; 11B)
ATLAS (Full; 11B)

0

NQ

TriviaQA
34

Task results80

Full-shot ne-tuning further improves
performance

60

EM

40

20

Chinchilla (70B)
ATLAS (Few; 11B)
ATLAS (Full; 11B)

0

NQ

TriviaQA

fi

35

Task results
MMLU(Multiple-choice NLU benchmark)80

On MMLU, ATLAS few-shot largely
underperforms Chinchilla / GPT-4.

72.5

Accuracy 65

Room for improvements for diverse
task adaptations!

57.5

50

GPT-4 (2023)
ATLAS (Few; 11B)

Chinchilla (70B)
ATLAS (Transfer; 11B)
36

Task results
MMLU(Multiple-choice NLU benchmark)80

Transferring from relevant tasks givelarge improvements, matching Chinchilla

72.5

Accuracy 65
57.5

50

GPT-4 (2023)
ATLAS (Few; 11B)

Chinchilla (70B)
ATLAS (Transfer; 11B)

37

Summary of downstream adaptations

ATLAS (Izacard et al., 2022)

Target task

Adaptation method

Datastore

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

Fine-tuning for QA & knowledge-intensive tasks often
gives strong performance (even in few-shot)38

Summary of downstream adaptations

ATLAS (Izacard et al., 2022)

Target task

Adaptation method

Datastore

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

Fine-tuning a retriever for a task matters!

39

Downstream adaptation of retrieval-based LMs
What are the tasks?
- Open-domain QA- Other knowledgeintensive tasks- General NLU- Language Modeling &other generation tasks

How to adapt?

- Fine-tuning- Reinforcement
learning- Prompting

What is data store?

- Unlabeled

Wikipedia / CC
- Web (Google / Bing
Search Results)- Training data40

GopherCite (Menick et al., 2022)

fi

Menick et al. 2022. “GopherCite: Teaching language models to support answers with veri ed quotes”

41GopherCite (Menick et al., 2022)

Extract and generate aquote to support an answer

fi

Menick et al. 2022. “GopherCite: Teaching language models to support answers with veri ed quotes”

42GopherCite: RLHF for answering with veri ed quotes

q
Index

x
z

LM

Supervised ne-tuning (SFT)y1

O -the-shelf
Google Search

We cannot update(x, y)

fi

fi

ff

43

q
Index

x
z

Supervised ne-tuning (SFT)y1

LM

O -the-shelf
Google Search

Model generated training dataltered by human(x, y)

y=

fi

fi

44
ff

fi

GopherCite: RLHF for answering with veri ed quotes

GopherCite: RLHF for answering with veri ed quotes

q
Index

x
z

LM

Reinforcement Learning with human feedback(e.g., Instruct GPT)y1

Reward
Model

y2

y2

Human
preferencedata

O -the-shelf
Google Search(x, y)

1

2(x, y , y , r)

fi

ff

45

GopherCite: RLHF for answering with veri ed quotes
Human rater
1

2(x, y , y )Plausibler ∈ y1, y2

as an answer to the input

Supported by accompanying evidence

2(x, y , y , r)
46fi

33k Human preference data

1Effects of RL
S&P

Natural Questions

80

RL w/ human feedback improves thequality of top 1 generations

60

40

20

0

FT (best 1)
FT (best 64)
NQ

RL (best 1)
RL (best 64)

ELI5(S&P = Supported & Plausible)

47Effects of RL
S&P

Natural Questions

80

Sampling & reranking many generations
using a reward model gives gains fromTop 1

60

40

20

0

FT (best 1)
FT (best 64)
NQ

RL (best 1)
RL (best 64)

ELI5
48

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results(Menick et al., 2022), also
WebGPT (Nakano et al., 2021)

Customizable

Bene it of ine-tuning

Competitive w/ more dataffRequiring training

49

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results(Menick et al., 2022), also
WebGPT (Nakano et al., 2021)

Bene it of RL

Better alignment with user preferences
Requiring additional data collection (preference)f50

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results(Menick et al., 2022), also
WebGPT (Nakano et al., 2021)

What if we cannot train LMs for downstream tasks?
(e.g., lack of computational resources / proprietary LM … etc)
51

Downstream adaptation of retrieval-based LMs
What are the tasks?

How to adapt?

What is data store?

- Open-domain QA- Other knowledgeintensive tasks- General NLU- Language Modeling &other generation tasks- Fine-tuning- Reinforcement
learning- Prompting- Wikipedia- Web (Google / Bing
Search Results)- Training data52

Prompting
k-shot instances (k=0, 32 … etc)
Doesn’t require LM training on tasks!

Q: who Is the US president
A: Joe Biden##
Q: What is the capital of US?
A: Washington DC.
##
Q: what is the Ontario capital?
A:

Training instances (demonstrations)

LM

Toronto

Test instances53

Retrieval-based prompting
k-shot instances (k=0, 32 … etc)
Retriever

Q: who Is the US president
A: Joe Biden##
Q: What is the capital of US?
A: Washington DC.
##
Q: what is the Ontario capital?
A:

LM

Toronto54

Design choice of retrieval-based Prompting

Input

LM

Input space:
Incorporate retrieved context in input space
Intermediate layers:
N/A

OutputOutput space:Interpolate token probability
distributions in output space55

Design choice of retrieval-based Prompting

Input

LM

Input space:
Incorporate retrieved context in input space
Intermediate layers:
N/A

OutputExtending kNN-LM fordownstream tasks

Output space:Interpolate token probability
distributions in output space56

kNN-Prompt (Shi et al., 2022)
kNN LM with fuzzy verbalizers
for zero-/few-shot classi cationfi

Shi et al. 2022. “Nearest Neighbor Zero-shot Inference”

57

kNN-Prompt (Shi et al., 2022)

LM predicts next token

58kNN-Prompt (Shi et al., 2022)

kNN predicts next tokens

59kNN-Prompt (Shi et al., 2022)
kNN-LM(Khandelwal et al., 2020)

Interpolate output token
probability
60

kNN-Prompt (Shi et al., 2022)
The kNN token
distributions arequite sparse!

61kNN-Prompt (Shi et al., 2022)
The kNN token
distributions arequite sparse!

Positive
Negative62kNN-Prompt (Shi et al., 2022)
Fuzzy verbalizer maps
token probability to target
class labels

Find similar tokens using GloVe& ConceptNet

63

Results on diverse classi cation tasks
AccuracyLM

kNN-LM

kNN-Prompt90

Signi cant gains fromkNN-LM

67.5
45
22.5
0

RTE

SST-2

AGNews

NLI/ entailment

Sentiment
analysis

Topicclassi cation

fi

fi

fi

64

Results on diverse classi cation tasks
AccuracyLM

kNN-LM

kNN-Prompt

90
67.5

kNN-prompt largely
outperforms vanilla LM inzero-shot classi cation

45
22.5
0

RTE

SST-2

AGNews

NLI/ entailment

Sentiment
analysis

Topicclassi cation

fi

fi

fi

65

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results

Classi ication

Prompting (output)

Wikipedia | Pile

(Menick et al., 2022)

kNN-prompt (Shi et al.,2022)

Retrieval-based LMs are e ective in general NLU tasks!
ff

f

66

Retrieval-based Prompting

Input

LM

Input space:
Incorporate retrieved context in input space
Intermediate layers:
N/A

OutputOutput space:Interpolate token probability
distributions in output space67

Retrieval-in-contextqRetrieverInput context

x

What is the capital of
Ontario?

z

Toronto is in Canada. It isthe capital city of theprovince of Ontario.

LM

Toronto(Shi et al., 2023; Ram et al., 2022; Mallen et al., 2022; Yu et al., 2022; Press et al., 2022; inter alia)
68

REPLUG (Shi et al., 2023; Section 3&4)

x What is the capital of Ontario?
RetrieverOttawa Toronto Ontario

Toronto is in Canada. It
is the capital city of theprovince of Ontario.

LM69

REPLUG (Shi et al., 2023; Section 3&4)

x What is the capital of Ontario?
RetrieverOttawa Toronto Ontario

Toronto is in Canada. It
is the capital city of theprovince of Ontario.
Ontario is home to thenation’s capital city,
Ottawa, and the mostpopulous city Toronto.
…
Top 10 documents

LM

70

REPLUG (Shi et al., 2023; Section 3&4)

x What is the capital of Ontario?
RetrieverOttawa Toronto Ontario

Toronto is in Canada. It
is the capital city of theprovince of Ontario.
Ontario is home to thenation’s capital city,
Ottawa, and the mostpopulous city Toronto.
…
Top 10 documents

LM

∑
i

p(Toronto | x, pi) >∑ip(Ottawa | x, pi)
71

REPLUG: Results on QA & MMLU
80

+3.5

Large performance gainfrom base LM

68

55

+4.6

Acc.
43

Base LM (CodeX)+ REPLUG LSR
30

MMLU

NQ

72

REPLUG: Comparison with ATLAS
80

Outperforms ATLAS in fewshot, especially in MMLU

60

Acc.

40

20

0

ATLAS (Few-shot)
REPLUGATLAS (Full / Transfer)
MMLU

NQ

73

REPLUG: Comparison with ATLAS
80

ATLAS (Full / Transfer)outperforms REPLUG

60

Acc.

40

20

0

ATLAS (Few-shot)
REPLUGATLAS (Full / Transfer)
MMLU

NQ

74

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results

Classi ication

Prompting (output)

Wikipedia | CC

Knowledge-intensive

Prompting (input)

Wikipedia | CC

(Menick et al., 2022)

kNN-prompt (Shi et al.,2022)

REPLUG (Shi et al., 2023)

f

f

Bene it of retrievalbased prompting

No training & strong performance
Hard to control, underperforming full FT model 75

Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results

Classi ication

Prompting (output)

Wikipedia | CC

Knowledge-intensive

Prompting (input)

Wikipedia | CC

(Menick et al., 2022)

kNN-prompt (Shi et al.,2022)

REPLUG (Shi et al., 2023)

What can be other types of datastores?
f76

DocPrompting (Zhou et al., 2023)

Retrieve code documentations about related functions

Zhou et al. 2023. “DocPrompting: Generating the Codes by Retrieving the Docs”

77DocPrompting (Zhou et al., 2023)

Datastore
RetrieverLMPθ1(z | x)(x, z*)Pθ2(y | x, z)
78

DocPrompting (Zhou et al., 2023)

RetrieverLMPθ1(z | x)

Fine-tuning (e.g., CodeT5)

Pθ2(y | x, z)

Prompting (CodeX)(x, y)

79

DocPrompting (Zhou et al., 2023)
TLDR (NL —> bash)
32

Large gain given by DocPrompting
for both CodeT5 & CodeX

24

BLEU

16

8

0

CodeT5+ DocPrompting
CodeX+ DocPrompting
80

DocPrompting (Zhou et al., 2023)
CoNaLA (NL —> Python)
50

Room for improvement in the
retrieval component

37.5

BLEU

25

12.5

Active research in OOD / Zero-shot retrieval!
(BEIR; Thakur et al., 2021)+ DocPrompting+ DocPrompting (Oracle)

081Summary of downstream adaptations
Target task

Adaptation method

Datastore

ATLAS (Izacard et al., 2022)

Knowledge-intensiveFine-tuning(Retriever & LM)

Wikipedia | CC

GopherCite

Open-domain QA,
Long-form QA

Fine-tuning + RL (LM)

Google Search
Results

Classi ication

Prompting (output)

Wikipedia | CC

Knowledge-intensive

Prompting (input)

Wikipedia | CC

Code Generation

Fine-tuning (DS & LM),
Prompting (Input)

Code
documentations(Menick et al., 2022)

kNN-prompt (Shi et al.,2022)

REPLUG (Shi et al., 2023)
DocPrompting(Zhou et al., 2023)

f

82

How to adapt a retrieval-based LM for a task

Datastore

LM

Retrieval-based prompting is easy and simple; noneed to train but has higher variance

Fine-tuning (+ RL) requires training but lessvariance & is completive with more data83How to adapt a retrieval-based LM for a task

RetrieverTraining a retriever on downstream tasks helps

LM

Datastore can be diverse (also in Section 6) whilechallenges remain in OOD retrieval

84

Two key questions for downstream adaptations
How can we adapt a retrieval-based LM for a task?

When should we use a retrieval-based LM?

85

When to use a retrieval-based LM

Long-tailknowledge
updateVeri ability

fi

fi

86

Parameteref ciency

Key effectiveness in downstream tasks
Long-tail

LLMs often struggle in long-tail/less frequent entities<— less popularMore popular —>Kandpal et al. 2023. “Large language models struggle to learn long-tail knowledge”

87Key effectiveness in downstream tasks
Long-tailScaling LLMs only helps for popular knowledge; forlong tail, scaling gives marginal performance improvements
PopQA (Open-domain QA)

Performance on less popular questions(blue) doesn’t improve over scale

Mallen* and Asai* et al. 2023. “When Not to Trust Language Models: Investigating
Effectiveness of Parametric and Non-Parametric Memories”

88Key effectiveness in downstream tasks
Long-tail

Retrieval gives large performance gain in such long-tailRetrieval-in-context<— less popularMore popular —>Mallen* and Asai* et al. 2023. “When Not to Trust Language Models: Investigating
Effectiveness of Parametric and Non-Parametric Memories”

89Key effectiveness in downstream tasks
Long-tail

Retrieval gives large performance gain in such long-tail
NPM

Google RE<— more popular

Less popular —>Min et al. 2023. “Nonparametric Masked Language Modeling” ; Petroni et al. 2019. “Language Models as Knowledge Bases?”90

Key effectiveness in downstream tasks
Long-tailLargely reduce hallucinations in long-form generations
FactScore
80

60

InstructGPT
ChatGPT
Perplexity

40

20

0

FactScore

Min et al. 2023. “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation”

91Key effectiveness in downstream tasks
UpdateStandard LLMs need to be trained again to adapt to
evolving world knowledge
Temp LAMA

Test- 2017

Test - 2020

13
9.75

Huge performance drop when test
knowledge needs to be updated

6.5
3.25

Izacard et al. 2022. “Few-shot learning with retrieval augmented language models”
Dhingra et al. 2022. “Time-Aware language models as temporal knowledge bases”

0

T5-Fine-tuned on 2017 data

92

Key effectiveness in downstream tasks
UpdateSwapping the knowledge corpus to accommodate
temporal changes without additional training.
Train 2020, DS 2020
Train 2017, DS 2020

Temp LAMA61
45.75

Swapping test datastore only retains
strong performance

30.5
15.25

Izacard et al. 2022. “Few-shot learning with retrieval augmented language models”
Dhingra et al. 2022. “Time-Aware language models as temporal knowledge bases”

093Key effectiveness in downstream tasks
Parameteref ciency

Much smaller LMs with retrieval can outperform
much larger LMs in fact completions.
50

Retrieval + GPT-Neo 1.3B outperforms
vanilla GPT3 on PopQA

37.5

25

12.5

w/o retrieval
w/ Contriever0

GPT-Neo (1.3B)

GPT-Neo (1.3B)

fiMallen* and Asai* et al. 2023. “When Not to Trust Language Models: Investigating
Effectiveness of Parametric and Non-Parametric Memories”

94Key effectiveness in downstream tasks
Parameteref ciency

Much smaller LMs with retrieval can outperform
much larger LMs in fact completions.

NPM (354 M) outperforms
GPT-3 on T-Rex.

 fi

Min et al. 2023. “Nonparametric Masked Language Modeling”
Petroni et al. 2019. “Language Models as Knowledge Bases?”95

Key effectiveness in downstream tasks
Parameteref ciency

Much smaller LMs with retrieval can outperform
much larger LMs in fact completions.

Room for improvements fordiverse task adaptations!

80
72.5
65
57.5
50

fi

Izacard et al. 2022. “Few-shot learning withretrieval augmented language models”

MMLU (Multiple choice

GPT-4 (2023)
ATLAS (Few; 11B)

Chinchilla (70B)
ATLAS (Full; 11B)
96

Key effectiveness in downstream tasks
Veri ability

Human and model can reliably assess the factuality of thegenerations using the retrieved evidence.

fiNakano et al. 2021. “WebGPT:
Browser-assisted question-answeringwith human feedback”

97Attributions: AttributedQA (Bohnet et al., 2022)
Expected Model Output

g(x) = (a, c)
Q: The population ofToronto is _LM

6,255,000

Datastore

Attribution: The
current metro area
population of Toronto
in 2021 is 6,255,000.

Answer:Bohnet et al. 2022. “Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models”

98Attributions: AttributedQA (Bohnet et al., 2022)
Human Evaluation(AIS)

Automatic Evaluation(AutoAIS)NLI model
1. Are all (a,c) interpretable?
2. Is any information in a
supported by c?
99

AttributedQA (Bohnet et al., 2022)
Retrieval-based LM

Datastore

Q: The population ofToronto in 2023 is _Evidence: The current
metro area populationof Toronto in 2023 is
6,372,000.
A:

Post-hoc retrieval

LM

6,372,000

DatastoreLM6,372,000

Attribution: The
current metro area
population of Toronto
in 2021 is 6,255,000.

100

AttributedQA (Bohnet et al., 2022)
70

Retrieval in context yields higher
AIS than post-hoc retrieval

52.5

35

17.5

Retrieval-based LM

0EM

AutoAIS

Post-hoc retrieval

AIS

Attributions

101

When to use a retrieval-based LM

Long-tailknowledge
updateVeri ability

fi

fi

102

Parameteref ciency

When to use a retrieval-based LM

Long-tailknowledge
updateVeri ability

Parameteref ciency

Out of domain adaptations(Shi et al., 2022; Zheng et al., 2021)
Khandelwal, et al.2020. “Nearest Neighbor Zero-shot Inference”
Shi et al. 2022. “Nearest Neighbor Zero-shot Inference”

 

fi

fi

103

When to use a retrieval-based LM

Long-tailknowledge
updateVeri ability

Parameteref ciency

Out of domain adaptations

Privac(Shi et al., 2022; Zheng et al., 2021)(Huang et al., 2023)

Khandelwal, et al.2020. “Nearest Neighbor Zero-shot Inference”
Shi et al. 2022. “Nearest Neighbor Zero-shot Inference”
Huang et al. 2023. “Privacy Implications of Retrieval-Based Language Models”
 

y

fi

fi

104