Section 3:
Retrieval-based LM: Architecture1Categorization of retrieval-based LMs

2

Categorization of retrieval-based LMs
What to retrieve?
Query

2

Categorization of retrieval-based LMs
What to retrieve?
Query

Text chunks (passages)?

2Categorization of retrieval-based LMs
What to retrieve?
Query

Text chunks (passages)?
Tokens?

2Categorization of retrieval-based LMs
What to retrieve?
Query

Text chunks (passages)?
Tokens?
Something else?

2Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?

When to retrieve?

Input

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

When to retrieve?
w/ retrieval
The capital city of Ontario is Toronto.

LM
Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

When to retrieve?
w/ retrieval
The capital city of Ontario is Toronto.

LM

w/ retrieval w/ r w/r w/r w/ r w/r w/rThe capital city of Ontario is Toronto.

Text chunks (passages)?
Tokens?
Something else?

Output2

Categorization of retrieval-based LMs
What to retrieve?
Query

How to use retrieval?
Input

When to retrieve?
w/ retrieval
The capital city of Ontario is Toronto.

LM

w/ retrieval w/ r w/r w/r w/ r w/r w/rThe capital city of Ontario is Toronto.

Text chunks (passages)?
Tokens?
Something else?

Outputw/ retrieval

w/r

w/r

The capital city of Ontario is Toronto.

2Roadmap

3

Roadmap

What toretrieve?

Text chunks

3Roadmap
How to use
retrieval?
What toretrieve?

Text chunks

Input layer(concatenation)

3

Roadmap
When toretrieve?

How to use
retrieval?
What toretrieve?

Text chunks

Input layer(concatenation)

Once

3

Roadmap
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once

3

Roadmap
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens
Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

3

Roadmap
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

3

Roadmap
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

3

Roadmap
Févry et al. 2020,de Jong et al. 2021

When toretrieve?

How to use
retrieval?

Entities orentity mentions

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)
Adaptively
He et al. 2021, Drozdov et al. 2022, Alon et al. 2022

Intermediate layers(soft incorporation)
RETRO (Borgeaud et al. 2022)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)
Adaptively

Retrieve its own input

Jiang et al. 2023

Wu et al. 2022, Bertsch et al. 2023, Rubin & Brent, 2023

3

Roadmap
Févry et al. 2020,de Jong et al. 2021

When toretrieve?

How to use
retrieval?

Entities orentity mentions

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)
Adaptively

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

He et al. 2021, Drozdov et al. 2022, Alon et al. 2022

Adaptively

Retrieve its own input

Jiang et al. 2023

Wu et al. 2022, Bertsch et al. 2023, Rubin & Brent, 2023

This is only about “architecture”
Section 4 will categorize & discuss “training"

3

REALM (Guu et al 2020)

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

4REALM (Guu et al 2020)
x = World Cup 2022 was the last with 32 teams before the increase to [MASK] in 2026.

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

4REALM (Guu et al 2020)
x = World Cup 2022 was the last with 32 teams before the increase to [MASK] in 2026.

World Cup 2022 was … the increase to [MASK] in 2026.

LM

48

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

4REALM (Guu et al 2020)
x = World Cup 2022 was the last with 32 teams before the increase to [MASK] in 2026.

q (=x)
World Cup 2022 was … the increase to [MASK] in 2026.

RetrievalLM

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

5

REALM (Guu et al 2020)
x = World Cup 2022 was the last with 32 teams before the increase to [MASK] in 2026.

q (=x)
World Cup 2022 was … the increase to [MASK] in 2026.

RetrievalLMk chunks of text
(passages)

FIFA World Cup 2026
will expand to 48 teams.

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

5

REALM (Guu et al 2020)
x = World Cup 2022 was the last before the increase to [MASK] in the 2026 tournament.

xFIFA World Cup 2026 will expand to 48 teams.
…
World Cup 2022 was … the increase to [MASK] in 2026.

RetrievalLMk chunks of text
(passages)

FIFA World Cup 2026
will expand to 48 teams.

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

6REALM (Guu et al 2020)
x = World Cup 2022 was the last before the increase to [MASK] in the 2026 tournament.

xFIFA World Cup 2026 will expand to 48 teams.
…
World Cup 2022 was … the increase to [MASK] in 2026.

RetrievalLMk chunks of text
(passages)

FIFA World Cup 2026
will expand to 48 teams.

48

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

6REALM (Guu et al 2020)
x = World Cup 2022 was the last before the increase to [MASK] in the 2026 tournament.

xFIFA World Cup 2026 will expand to 48 teams.
…
World Cup 2022 was … the increase to [MASK] in 2026.

RetrievalLMk chunks of text
(passages)

FIFA World Cup 2026
will expand to 48 teams.

Retrieve stage

48

Read stage

Guu et al. 2020. “REALM: Retrieval-Augmented Language Model Pre-Training”

6REALM: (1) Retrieve stage

FIFA World Cup 2026
will expand to 48 teams.
In 2022, the 32 national
teams involved in thetournament.
Team USA celebrated
after winning its matchagainst Iran …
Wikipedia
13M chunks (passages)(called documents in the paper)

7

REALM: (1) Retrieve stage

FIFA World Cup 2026
will expand to 48 teams.

Encoder

In 2022, the 32 national
teams involved in thetournament.

Encoder

Team USA celebrated
after winning its matchagainst Iran …

z = Encoder(z)Encoder

Wikipedia
13M chunks (passages)(called documents in the paper)

7

REALM: (1) Retrieve stage
x = World Cup 2022 was … the increase to [MASK] in 2026.

FIFA World Cup 2026
will expand to 48 teams.

Encoder

In 2022, the 32 national
teams involved in thetournament.

Encoder

Team USA celebrated
after winning its matchagainst Iran …

z = Encoder(z)Encoder

Wikipedia
13M chunks (passages)(called documents in the paper)

7

REALM: (1) Retrieve stage
x = World Cup 2022 was … the increase to [MASK] in 2026.
EncoderFIFA World Cup 2026
will expand to 48 teams.

Encoder

In 2022, the 32 national
teams involved in thetournament.

Encoder

Team USA celebrated
after winning its matchagainst Iran …

Encoderz = Encoder(z)x = Encoder(x)Wikipedia
13M chunks (passages)(called documents in the paper)

7

REALM: (1) Retrieve stage
x = World Cup 2022 was … the increase to [MASK] in 2026.
EncoderFIFA World Cup 2026
will expand to 48 teams.

Encoder

In 2022, the 32 national
teams involved in thetournament.

Encoder

Team USA celebrated
after winning its matchagainst Iran …

Encoder

Fast nearest neighbor searchz = Encoder(z)x = Encoder(x)Wikipedia
13M chunks (passages)(called documents in the paper)

7

REALM: (1) Retrieve stage
x = World Cup 2022 was … the increase to [MASK] in 2026.
EncoderFIFA World Cup 2026
will expand to 48 teams.

Encoder

In 2022, the 32 national
teams involved in thetournament.

Encoder

Team USA celebrated
after winning its matchagainst Iran …
Wikipedia
13M chunks (passages)(called documents in the paper)

Encoder

Fast nearest neighbor searchz = Encoder(z)x = Encoder(x)z1, . . . , zk = argTop-k (x ⋅ z)

k retrieved chunks

7

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)…

[MASK] z1 [SEP] x[MASK] zk [SEP] x

LM

P(y | x, zk)

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] x

LM

P(y | x, zk)

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] x∑

LM

P(y | x, zk)P(z | x)P(y | x, z)

𝒟

z∈

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] x∑

P(y | x, zk)P(z | x)P(y | x, z)
from theretrieve stage

𝒟

z∈

LM

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] x∑

P(z | x)P(y | x, z)
from theretrieve stage

from theread stage𝒟z∈

P(y | x, zk)

LM

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] xz∈P(z | x)P(y | x, z)
from theretrieve stage

from theread stage

𝒟

Need to approximate→ Consider top k chunks only∑

P(y | x, zk)

LM

8

REALM: (2) Read stageLM

P(y | x, z1)[MASK] z2 [SEP] x

LM

P(y | x, z2)

Weighted average…

[MASK] z1 [SEP] x[MASK] zk [SEP] x

P(y | x, zk)LM

0 if not one of top kz∈from theretrieve stage

from theread stage

𝒟

Need to approximate→ Consider top k chunks only∑

P(z | x)P(y | x, z)

8REALM (Guu et al 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

9REALM (Guu et al 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

10REALM (Guu et al 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

11

REALM (Guu et al 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

12

REALM and subsequent work

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning on open-domain QA

fi

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning on open-domain QA* DPR (Karpukhin et al 2020): Pipeline training instead of joint training, ne-tuned on open-domain QA(no explicit language modeling)

fi

fi

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning on open-domain QA* DPR (Karpukhin et al 2020): Pipeline training instead of joint training, ne-tuned on open-domain QA(no explicit language modeling)* RAG (Lewis et al 2020): “Generative” instead of “masked language modeling”, ne-tuned on opendomain QA & knowledge intensive tasks (no explicit language modeling)

fi

fi

fi

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning on open-domain QA* DPR (Karpukhin et al 2020): Pipeline training instead of joint training, ne-tuned on open-domain QA(no explicit language modeling)* RAG (Lewis et al 2020): “Generative” instead of “masked language modeling”, ne-tuned on opendomain QA & knowledge intensive tasks (no explicit language modeling)* Atlas (Izcard et al 2022): Combine RAG with retrieval-based language model pre-training based on
the encoder-decoder architecture (more to come in Section 4), ne-tuned on open-domain QA &other QA tasks* Papers that follow this approach focusing on LM perplexity have come out quite recently — Shi etal. 2023, Ram et al. 2023

fi

fi

fi

fi

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning on open-domain QA* DPR (Karpukhin et al 2020): Pipeline training instead of joint training, ne-tuned on open-domain QA(no explicit language modeling)* RAG (Lewis et al 2020): “Generative” instead of “masked language modeling”, ne-tuned on opendomain QA & knowledge intensive tasks (no explicit language modeling)* Atlas (Izcard et al 2022): Combine RAG with retrieval-based language model pre-training based on
the encoder-decoder architecture (more to come in Section 4), ne-tuned on open-domain QA &other QA tasks* Papers that follow this approach focusing on LM perplexity have come out quite recently — Shi etal. 2023, Ram et al. 2023

For a while, mainly evaluated on
knowledge-intensive tasks (e.g. open-domain QA) with ine-tuning(more context in Section 5)

fi

f

fi

fi

fi

13

REALM and subsequent work
* REALM (Guu et al 2020): MLM followed by ne-tuning, focusing on open-domain QA* DPR (Karpukhin et al 2020): Pipeline training instead of joint training, focusing on open-domain QA(no explicit language modeling)* RAG (Lewis et al 2020): “Generative” instead of “masked language modeling”, focusing on opendomain QA & knowledge intensive tasks (no explicit language modeling)* Atlas (Izcard et al 2022): Combine RAG with retrieval-based language model pre-training based on
the encoder-decoder architecture (more to come in Section 4), focusing on open-domain QA &knowledge intensive tasks* Papers that follow this approach focusing on LM perplexity have come out quite recently (Shi et al.
2023, Ram et al. 2023)

fi

Ram et al. 2023. “In-Context Retrieval-Augmented Language Models”
Shi et al. 2023. “REPLUG: Retrieval-Augmented Black-Box Language Models”

14Retrieval-in-context LM
x = World Cup 2022 was the last with 32 teams, before the increase to

Ram et al. 2023. “In-Context Retrieval-Augmented Language Models”
Shi et al. 2023. “REPLUG: Retrieval-Augmented Black-Box Language Models”

15

Retrieval-in-context LM
x = World Cup 2022 was the last with 32 teams, before the increase to
World Cup 2022 was the last with 32 teams, before the increase to

Retrieval* Can use multiple text blocks too (see the papers!)

FIFA World Cup 2026 will expand to 48 teams. World Cup 2022 was the last with 32 teams, before the increase to

Ram et al. 2023. “In-Context Retrieval-Augmented Language Models”
Shi et al. 2023. “REPLUG: Retrieval-Augmented Black-Box Language Models”

15

Retrieval-in-context LM
x = World Cup 2022 was the last with 32 teams, before the increase to
World Cup 2022 was the last with 32 teams, before the increase to

Retrieval* Can use multiple text blocks too (see the papers!)

FIFA World Cup 2026 will expand to 48 teams. World Cup 2022 was the last with 32 teams, before the increase toLM
48 in the 2026 tournament.

Ram et al. 2023. “In-Context Retrieval-Augmented Language Models”
Shi et al. 2023. “REPLUG: Retrieval-Augmented Black-Box Language Models”

16Retrieval-in-context LM
Perplexity: The lower the better

Varying sizes of LMs

Retrieval helps over all sizes of LMs
Graphs from Ram et al. 2023
17

Retrieval-in-context LM
Is q=x necessary?

18Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase to18

Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

Retrieval

18Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

Retrieval
The U.S. national team defeated Iran 1-0.
18

Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

Retrieval
The U.S. national team defeated Iran 1-0.
Does not cover “tokens that will come next”

18Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

World Cup 2022 was the last with 32 teams,
before the increase to

Retrieval

Retrieval

The U.S. national team defeated Iran 1-0.
Does not cover “tokens that will come next”

19Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

World Cup 2022 was the last with 32 teams,
before the increase to

Retrieval

Retrieval

The U.S. national team defeated Iran 1-0.

FIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
before the increase toDoes not cover “tokens that will come next”

19Retrieval-in-context LM
Is q=x necessary?

x = Team USA celebrates after winning its match against Iran at Al Thumama Stadium in Group B play of theFIFA World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022 was the last with 32 teams, before the increase toTeam USA celebrates after winning its match against
Iran at Al Thumama Stadium in Group B play of the FIFA
World Cup 2022 on Nov. 29, 2022. (..) World Cup 2022
was the last with 32 teams, before the increase to

World Cup 2022 was the last with 32 teams,
before the increase to

Retrieval

Retrieval

The U.S. national team defeated Iran 1-0.

FIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
more relevant to what will come next
before the increase toDoes not cover “tokens that will come next”

19Retrieval-in-context LM

Shorter pre x (more recent tokens) as a query helps
Graphs from Ram et al. 2023
fi

20

Retrieval-in-context LM

but not too short

Shorter pre x (more recent tokens) as a query helps
Graphs from Ram et al. 2023
fi

20

Retrieval-in-context LM
How frequent should retrieval be?

21Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last with21

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last with22

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withLM
32 teams before the increase to 48 in the 2026 tournament.
22

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withLM
32 teams before the increase to 48 in the 2026 tournament.
explained by retrieval

22

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withLM
32 teams before the increase to 48 in the 2026 tournament.
explained by retrieval
not really covered22Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last with

Retrieval
The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withLM
32 teams before the increase23

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last withWorld Cup 2022 was the last with 32 teams
before the increase

Retrieval

Retrieval

The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withFIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
before the increaseLM
32 teams before the increase24Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last withWorld Cup 2022 was the last with 32 teams
before the increase

Retrieval

Retrieval

The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withFIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
before the increaseLM
32 teams before the increase25Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last withWorld Cup 2022 was the last with 32 teams
before the increase

Retrieval

Retrieval

The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withFIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
before the increaseLMLM

32 teams before the increase

to 48 in the 2026 tournament.
25

Retrieval-in-context LM
How frequent should retrieval be?
World Cup 2022 was the last withWorld Cup 2022 was the last with 32 teams
before the increase

Retrieval

Retrieval

The 2022 FIFA World Cup (...) 32 national teams
involved in the tournament. World Cup 2022
was the last withFIFA World Cup 2026 will expand to 48 teams.
World Cup 2022 was the last with 32 teams,
before the increaseLMLM

32 teams before the increase

to 48 in the 2026 tournament.
Retrieval results from a new query explain them!

25Retrieval-in-context LM

Retrieving more frequently helps
Graphs from Ram et al. 2023
26

Retrieval-in-context LM

with cost in inference time

Retrieving more frequently helps
Graphs from Ram et al. 2023
26

Retrieve-in-context LM (Shi et al 2023, Ram et al 2023)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

27Retrieve-in-context LM (Shi et al 2023, Ram et al 2023)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

28Retrieve-in-context LM (Shi et al 2023, Ram et al 2023)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

29Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

Applying the same approach to LM raised new questions
which mattered less in prior work (e.g. REALM) with short inputs & short outputs

30Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

can be very inef cient to retrieve many text chunks, frequentlyfi

31

RETRO (Borgeaud et al. 2021)

Borgeaud et al. 2021. “Improving language models by retrieving from trillions of tokens”

32RETRO (Borgeaud et al. 2021)
Incorporation in the “intermediate layer” instead of the “input” layer→ designed for many chunks, frequently, more ef cientlyfiBorgeaud et al. 2021. “Improving language models by retrieving from trillions of tokens”

32RETRO (Borgeaud et al. 2021)
Incorporation in the “intermediate layer” instead of the “input” layer→ designed for many chunks, frequently, more ef ciently
Scale the datastore (1.8T tokens)fi

Borgeaud et al. 2021. “Improving language models by retrieving from trillions of tokens”

32RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to33

RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to

x1

x2

x3

33

RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to

x1

x2

x3

(k chunks of text per split)1k
p1 . . . p1

x1
x2
x3

Retrieval
Encoder1kIndex p2 . . . p2
1kp3 . . . p3

33

RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to

x1

x3

x2

(k chunks of text per split)1k
p1 . . . p1

x1
x2
x3

Retrieval
Encoder1kIndex p2 . . . p2

LM
Encoder1kp3 . . . p3

33

RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to

x1

x3

x2

(k chunks of text per split)1k
p1 . . . p1

x1
x2
x3

Retrieval
Encoder1kIndex p2 . . . p2
1kp3 . . . p3

E1
LM
Encoder

E2
E3

33

RETRO (Borgeaud et al. 2021)
x = World Cup 2022 was the last with 32 teams, before the increase to

x1

x3

x2

(k chunks of text per split)1k
p1 . . . p1

x1
x2
x3

Retrieval
Encoder1kIndex p2 . . . p2
1kp3 . . . p3

E1
LM
Encoder

E2
E3(A r × k × d matrix)(r = # tokens per text chunk)(d = hidden dimension)(k = # retrieved chunks per split)
33

Regular decoder

x1
x2
x3

EMB

ATTN

FFN

HEAD

Transformers blocks (xL)34

Decoder in RETRO
E1 E2 E3
x1
x2
x3

EMB

ATTN

CCA
EMB

FFN

HEAD

RETRO blocks (xL)

Chunked Cross Attention (CCA)35

Chunked Cross Attention

Outputs from the previous layer

36

Chunked Cross Attention

Outputs from the previous layer

37

Chunked Cross Attention

Outputs from the previous layer

38

Chunked Cross Attention

Outputs from the previous layer

Inputs to the next layer

39

Chunked Cross Attention

Outputs from the previous layer

Inputs to the next layer

Cross-attention can be computed in parallel40

Chunked Cross Attention

If you generated until here

Outputs from the previous layer

Inputs to the next layer

Cross-attention can be computed in parallel40

Chunked Cross Attention
You get thisIf you generated until here

Outputs from the previous layer

Inputs to the next layer

Cross-attention can be computed in parallel40

Chunked Cross Attention
You get thisIf you generated until here

Outputs from the previous layer

Inputs
to
the
next
layer
and go through thisCross-attention can be computed in parallel40

Chunked Cross Attention

This part can be re-used

If you generated until here

Outputs from the previous layer

Inputs to the next layer

Cross-attention can be computed in parallel
41

Results
Perplexity: The lower the better

Signi cant improvements by retrieving from 1.8 trillion tokens

fi42Results
Perplexity: The lower the better

Signi cant improvements by retrieving from 1.8 trillion tokens

fi42Results

Gains are constant with model scale

The larger datastore is, the better
43

RETRO (Borgeaud et al. 2021)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

44

RETRO (Borgeaud et al. 2021)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

45

RETRO (Borgeaud et al. 2021)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

46

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

47

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

47

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Can use many blocks, more frequently, more e iciently

ff

47

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Can use many blocks, more frequently, more e iciently
Additional complexity; Can’t be used without training (more in section 4)

ff

47

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

What else?

48kNN-LM (Khandelwal et al. 2020)

Khandelwal et al. 2020. “Generalization through Memorization: Nearest Neighbor Language Models”

49

kNN-LM (Khandelwal et al. 2020)
A different way of using retrieval, where the LM outputs a
nonparametric distribution over every token in the data.

Khandelwal et al. 2020. “Generalization through Memorization: Nearest Neighbor Language Models”

49

kNN-LM (Khandelwal et al. 2020)
A different way of using retrieval, where the LM outputs a
nonparametric distribution over every token in the data.
Can be seen as an incorporation in the “output” layer

Khandelwal et al. 2020. “Generalization through Memorization: Nearest Neighbor Language Models”

49

kNN-LM (Khandelwal et al. 2020)50

kNN-LM (Khandelwal et al. 2020)

51

f

kNN-LM (Khandelwal et al. 2020)… Obama was senator for Illinois from 1997 to
2005, …. Barack is Married to Michelle and theirirst daughter, … Obama was born in Hawaii, andgraduated from Columbia University. … Obama is anative of Hawaii, ….

52

fkNN-LM (Khandelwal et al. 2020)… Obama was senator for Illinois from 1997 to
2005, …. Barack is Married to Michelle and theirirst daughter, … Obama was born in Hawaii, andgraduated from Columbia University. … Obama is anative of Hawaii, ….

52

fkNN-LM (Khandelwal et al. 2020)
The size of the datastore = # of tokens in the corpus (>1B)… Obama was senator for Illinois from 1997 to
2005, …. Barack is Married to Michelle and theirirst daughter, … Obama was born in Hawaii, andgraduated from Columbia University. … Obama is anative of Hawaii, ….

52

kNN-LM (Khandelwal et al. 2020)53

kNN-LM (Khandelwal et al. 2020)

Which tokens in a datastore are close to the next token?

53kNN-LM (Khandelwal et al. 2020)

Which tokens in a datastore are close to the next token?

=
Which pre xes in a datastore are close to the pre x we have?

fi

fi

54

kNN-LM (Khandelwal et al. 2020)

Which tokens in a datastore are close to the next token?

=
Which pre xes in a datastore are close to the pre x we have?

=
Which vectors in a datastore are close to the vector we have?

fi

fi

55

kNN-LM (Khandelwal et al. 2020)

Which vectors in a datastore are close to the vector we have?

56kNN-LM (Khandelwal et al. 2020)

Which vectors in a datastore are close to the vector we have?

57

kNN-LM (Khandelwal et al. 2020)58

kNN-LM (Khandelwal et al. 2020)

𝕀

𝒟

PkNN(y | x) ∝∑[v = y]sim(k, x)(k,v)∈

58kNN-LM (Khandelwal et al. 2020)

𝕀

𝒟

PkNN(y | x) ∝∑(k,v)∈[v = y]sim(k, x)sim(k, x) = exp (−d(Enc(k), Enc(x)))
58kNN-LM (Khandelwal et al. 2020)

𝕀

𝒟

PkNN(y | x) ∝∑(k,v)∈[v = y]sim(k, x)sim(k, x) = exp (−d(Enc(k), Enc(x)))
58kNN-LM (Khandelwal et al. 2020)

𝕀

𝒟

PkNN(y | x) ∝∑(k,v)∈[v = y]sim(k, x)sim(k, x) = exp (−d(Enc(k), Enc(x)))
58kNN-LM (Khandelwal et al. 2020)

distance function𝕀

𝒟

PkNN(y | x) ∝∑(k,v)∈[v = y]sim(k, x)sim(k, x) = exp (−d(Enc(k), Enc(x)))
58kNN-LM (Khandelwal et al. 2020)

PkNN−LM(y | x) = (1 − λ)PLM(y | x) + λPkNN(y | x)
Later work, e.g., NPM (Min et al. 2023) removed interpolation (more in Section 4)59

kNN-LM (Khandelwal et al. 2020)

PkNN−LM(y | x) = (1 − λ)PLM(y | x) + λPkNN(y | x)
Later work, e.g., NPM (Min et al. 2023) removed interpolation (more in Section 4)59

kNN-LM (Khandelwal et al. 2020)

PkNN−LM(y | x) = (1 − λ)PLM(y | x) + λPkNN(y | x)
Later work, e.g., NPM (Min et al. 2023) removed interpolation (more in Section 4)59

kNN-LM (Khandelwal et al. 2020)

λ: hyperparameter

PkNN−LM(y | x) = (1 − λ)PLM(y | x) + λPkNN(y | x)
Later work, e.g., NPM (Min et al. 2023) removed interpolation (more in Section 4)59

kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

60

kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

61

kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

61

kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

61

kNN-LM - why?
Dense vector space

Training contexts… a ordable

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… nice… good… bad… poor… terribleff62

kNN-LM - why?
Dense vector space
10/10, would buy this cheap

Training contexts… a ordable

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… nice… good… bad… poor… terribleff62

kNN-LM - why?
Dense vector space
10/10, would buy this cheap

Training contexts… a ordable

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… nice… good… bad… poor… terrible

Item delivered broken. Very cheapff

62

kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

63kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

63kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

63kNN-LM - why?

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch

63kNN-LM - why?
Dense vector space

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… computer… tool… ire… infection… machine… pregnancyf64kNN-LM - why?
Dense vector space
PyTorch, you can use torch

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… computer… tool… ire… infection… machine… pregnancyf64kNN-LM - why?
Dense vector space
PyTorch, you can use torch… permitted to bring a torch

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… computer… tool… ire… infection… machine… pregnancyf64kNN-LM - why?
Dense vector space
PyTorch, you can use torch… permitted to bring a torch

Training contexts

Targets

10/10, would buy this cheap
Item delivered broken. Very cheap
To check the version of PyTorch, you can use torch
You are permitted to bring a torch
A group of infections … one of the torch… computer… tool… ire… infection… machine… pregnancy… a group of infections … torchf64kNN-LM - results

The lower the better65kNN-LM - results
No-retrieval LM

The lower the better65kNN-LM - results
No-retrieval LM

The lower the better

30x larger No-retrieval LM

65

kNN-LM - results
No-retrieval LM

The lower the better

30x larger No-retrieval LM
kNN-LM65

kNN-LM - results
No-retrieval LM

The lower the better

30x larger No-retrieval LM
kNN-LM

Outperforms no-retrieval LM
65

kNN-LM - results
No-retrieval LM

The lower the better

30x larger No-retrieval LM
kNN-LM

Outperforms no-retrieval LM

Better with bigger datastore
65kNN-LM - results

Better withbigger k
66

kNN-LM - results

Better withbigger k

Helps more
out-of-domain
66Can use in-domain datastore
kNN-LM
resultseven if parameters were not trained in-domain

Better withbigger k

Helps more
out-of-domain
66kNN-LM (Khandelwal et al. 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

67kNN-LM (Khandelwal et al. 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

68kNN-LM (Khandelwal et al. 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

69kNN-LM (Khandelwal et al. 2020)
What to retrieve?

How to use retrieval?

When to retrieve?

- Chunks- Tokens- Others- Input layer- Intermediate layers- Output layer- Once- Every n tokens (n>1)- Every token

70Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)71

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)71

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

More ine-grained; Can be better at rare patterns & out-of-domainf71Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

More ine-grained; Can be better at rare patterns & out-of-domain
Can be very e icient (as long as kNN search is fast)

ff

f

71

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

More ine-grained; Can be better at rare patterns & out-of-domain
Can be very e icient (as long as kNN search is fast)
Datastore is expensive in space: given the same data, # text chunks vs. # tokens

ff

f

71

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

More ine-grained; Can be better at rare patterns & out-of-domain
Can be very e icient (as long as kNN search is fast) (Wikipedia) 13M vs. 4B
Datastore is expensive in space: given the same data, # text chunks vs. # tokens

ff

f

71

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

More ine-grained; Can be better at rare patterns & out-of-domain
Can be very e icient (as long as kNN search is fast)
Datastore is expensive in space: given the same data, # text chunks vs. # tokens

ff

f

No cross attention between input and retrieval results

71Extensions
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)72

Extensions
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

kNN-LM (Khandelwal et al. 2020)

It’s xed! Can we do adaptively?

fi72Adaptive retrieval for ef ciency

Adaptive retrieval oftext chunks

Adaptive retrieval of
tokens(following retrieve-in-context)(following kNN-LM)fi73

Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)
Language Model

Jiang et al. “Active Retrieval Augmented Generation”

74Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States.

Language Model

Jiang et al. “Active Retrieval Augmented Generation”

75Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States.

I am con dent!

Language Modelfi

Jiang et al. “Active Retrieval Augmented Generation”

75Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States.

I am con dent!

Language Modelfi

Jiang et al. “Active Retrieval Augmented Generation”

76Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended the University of Pennsylvania,
where he earned a law degree.

Language Model

Jiang et al. “Active Retrieval Augmented Generation”

77Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended the University of Pennsylvania,
where he earned a law degree.

Unsure…

Language Model

Jiang et al. “Active Retrieval Augmented Generation”

77Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended [mask], where he earned [mask].
Unsure…

Language Model

Jiang et al. “Active Retrieval Augmented Generation”

78Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended [mask], where he earned [mask].
Unsure…

Language Model

Jiang et al. “Active Retrieval Augmented Generation”

79Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended [mask], where he earned [mask].
Unsure…

Language Model

Joe Biden

At the University of Delaware in Newark, Biden …
earned a Bachelor of Arts degree in 1965 with adouble major in history and political science.

Jiang et al. “Active Retrieval Augmented Generation”

79Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended [mask], where he earned [mask].
Unsure…

Language Model

Joe Biden

At the University of Delaware in Newark, Biden …
earned a Bachelor of Arts degree in 1965 with adouble major in history and political science.

Jiang et al. “Active Retrieval Augmented Generation”

80Adaptive retrieval of chunks- Judge necessity

Input: Generate a summary about Joe Biden.

FLARE (Jiang et al. 2023)
Retrieval (Datastore + Index)
Language Model

Joe Biden (born November 20, 1942) is the 46th president of theUnited States. Joe Biden attended [mask], where he earned [mask].
He graduated from the University of Delaware in 1965 with a Bachelorof Arts in history and political science.

Joe Biden

At the University of Delaware in Newark, Biden …
earned a Bachelor of Arts degree in 1965 with adouble major in history and political science.

Jiang et al. “Active Retrieval Augmented Generation”

81Adaptive retrieval of tokens- Judge necessity82Adaptive retrieval of tokens- Judge necessity

Joe Biden graduated from the University of

Delaware .

82

Adaptive retrieval of tokens- Judge necessity
retrieve retrieve

retrieve

retrieve

retrieve retrieve

Joe Biden graduated from the University of

retrieve

Delaware .

82

Adaptive retrieval of tokens- Judge necessity
retrieve retrieve

retrieve

retrieve

retrieve retrieve

Joe Biden graduated from the University of
retrieveLMLM

retrieve

Delaware .

retrieve retrieve

Joe Biden graduated from the University of

retrieve

LM

Delaware .

He et al. 2021. “Efficient Nearest Neighbor Language Models”

82Adaptive retrieval of tokens- Judge necessity
retrieve retrieve

retrieve

retrieve

retrieve retrieve

Joe Biden graduated from the University of
retrieveLMLM

retrieve

Delaware .

retrieve retrieve

Joe Biden graduated from the University of

retrieve

LM

Delaware .

PkNN−LM(y | x) = (1 − λ(x))PLM(y | x) + λ(x)PkNN(y | x)

He et al. 2021. “Efficient Nearest Neighbor Language Models”

82Adaptive retrieval of tokens- Judge necessity
retrieve retrieve

retrieve

retrieve

retrieve retrieve

Joe Biden graduated from the University of
retrieveLMLM

retrieve

Delaware .

retrieve retrieve

Joe Biden graduated from the University of

retrieve

LM

Delaware .

PkNN−LM(y | x) = (1 − λ(x))PLM(y | x) + λ(x)PkNN(y | x)
A function of the input x→ λ = 0 if λ < γ
He et al. 2021. “Efficient Nearest Neighbor Language Models”

82Adaptive retrieval of tokens- Use local info

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

83Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

83Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrieve

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

84Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrieve retrieve

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

85Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrieve retrieve

retrieve

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

86Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrieve retrieve

retrieve retrieve

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

87Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrieve

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

88Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrievepointerpointer

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

89Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrievepointerpointerpointer

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

90Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrievepointerpointerpointerpointer

Joe Biden graduated from the University of Delaware.

Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

91Adaptive retrieval of tokens- Use local info
Training contexts

Targets
At theAt the University
At the Universty ofAt the University of Delaware
At the University of Delaware in
At the University of Delaware in Newark

retrievepointerpointerpointerpointer

Joe Biden graduated from the University of Delaware.
Retrieve once, and save other searches!
Alon et al. 2022. “Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval”

91Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

Output layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)
Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Text chunks
Tokens92

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

Output layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)
Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Text chunks
TokensMore e icientff

92Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

Output layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)
Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

More e icient

Text chunks
Tokens

Decision may not always be optimal

ff

92Summary
REALM (Guu et al 2020)
Retrieve-in-context LM (Shi et al2023, Ram et al 2023)
RETRO (Borgeaud et al. 2021)
kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)
Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

What do retrieve?

How to use retrieval?

When to retrieve?

Text chunks
Text chunks
Text chunks

Input layer

Once

Input layer

Every n tokens

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

Output layer

Every n tokens(adaptive)

Text chunks
TokensWhat else beyond text chunks and tokens?
93

Entities as Experts (Fevry et al. 2020)

94

Entities as Experts (Fevry et al. 2020)

94

Entities as Experts (Fevry et al. 2020)

94

Entities as Experts (Fevry et al. 2020)

94

Entities as Experts (Fevry et al. 2020)

94

Entities as Experts (Fevry et al. 2020)

Dense vector space
94

Entities as Experts (Fevry et al. 2020)

Fevry et al. 2020. “Entities as Experts: Sparse Memory Access with Entity Supervision”

95Entities as Experts (Fevry et al. 2020)

Fevry et al. 2020. “Entities as Experts: Sparse Memory Access with Entity Supervision”

96Entities as Experts (Fevry et al. 2020)(Wikipedia)
chunks: 13 millions
tokens: 4 billions
entities: 6 millions

Fevry et al. 2020. “Entities as Experts: Sparse Memory Access with Entity Supervision”

96Entities as Experts (Fevry et al. 2020)(Wikipedia)
chunks: 13 millions
tokens: 4 billions
entities: 6 millions

Need text withentity detected

Fevry et al. 2020. “Entities as Experts: Sparse Memory Access with Entity Supervision”

96Entities as Experts (Fevry et al. 2020)(Wikipedia)
chunks: 13 millions
tokens: 4 billions
entities: 6 millions

Need text withentity detected

Need entity linker
Fevry et al. 2020. “Entities as Experts: Sparse Memory Access with Entity Supervision”

96Mention Memory (de Jong et al. 2022)
One vector per entity → One vector per entity mention

de Jong et al. 2022. “Mention Memory:
incorporating textual knowledge into Transformers through entity mention attention”

97Mention Memory (de Jong et al. 2022)
One vector per entity → One vector per entity mention

de Jong et al. 2022. “Mention Memory:
incorporating textual knowledge into Transformers through entity mention attention”

98Mention Memory (de Jong et al. 2022)
One vector per entity → One vector per entity mention

de Jong et al. 2022. “Mention Memory:
incorporating textual knowledge into Transformers through entity mention attention”

98Mention Memory (de Jong et al. 2022)(Wikipedia)
One
vector per entity → One vector per entity mention
chunks:13M
tokens: 4B
entities: 6Mentity mentions: 150M

de Jong et al. 2022. “Mention Memory:
incorporating textual knowledge into Transformers through entity mention attention”

98Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

99

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

Most effective for entity-centric tasks & space-ef icientf99

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

Most effective for entity-centric tasks & space-ef icient

Additional entity detection requiredf99Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

All models retrieve from the external text
100

Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

All models retrieve from the external text
What else can we do with these models?

100Retrieval for long-range LM

Wu et al. 2022. Memorizing Transformers (Figure source)
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval

101

Retrieval for long-range LM

Wu et al. 2022. Memorizing Transformers (Figure source)
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval

101

Retrieval for long-range LM

Wu et al. 2022. Memorizing Transformers (Figure source)
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval

102

Retrieval for long-range LM
Datastore is based on “input”(instead of external text corpus)

Wu et al. 2022. Memorizing Transformers (Figure source)
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval

102

Retrieval for long-range LM
Datastore is based on “input”(instead of external text corpus)

kNN search incorporated in
the attention layer

Wu et al. 2022. Memorizing Transformers (Figure source)
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval

102

Retrieval for long-range LM

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)

103Retrieval for long-range LM

Chunked Cross Attention

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)

104Retrieval for long-range LM

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)

105Retrieval for long-range LM

Every chunk is assigned
a similarity score

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)

105Retrieval for long-range LM
Top-k nearest neighbor search

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)

106Retrieval for long-range LM
Fed into CCA

Wu et al. 2022. Memorizing Transformers
Bertsch et al. 2023. Unlimiformer: Long-Range Transformers with Unlimited Length Input
Rubin & Berant. 2023. Long-range Language Modeling with Self-retrieval (Figure source)107Summary
What do retrieve?

How to use retrieval?

When to retrieve?

REALM (Guu et al 2020)

Text chunks

Input layer

Once

Retrieve-in-context LM (Shi et al2023, Ram et al 2023)

Text chunks

Input layer

Every n tokens

RETRO (Borgeaud et al. 2021)

Text chunks

Intermediate layers

Every n tokens

Tokens

Output layer

Every token

Input layer

Every n tokens(adaptive)

kNN-LM (Khandelwal et al. 2020)
FLARE (Jiang et al. 2023)

Text chunks

Adaptive kNN-LM (He et al 2021,
Alon et al 2022, etc)

Tokens

Output layer

Every n tokens(adaptive)

Entities as Experts (Fevry et al.
2020), Mention Memory (de Jonget al. 2022)

Entities or entity
mentions

Intermediate layers

Every entity mentions

Wu et al. 2022, Bertsch et al. 2023,
Rubin & Berant. 2023

Text chunks fromthe input

Intermediate layers

Once or every ntokens108

Wrapping up

109

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens
Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

109

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens
Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

More frequent retrieval = better in performance, but slower

109

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

110

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)• Input layer: Simple but can be slower• Intermediate layers: More complex (need training) but can be designed to be more e icientff

110

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

111

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)• Text blocks: Datastore can be space-e icient, more computation• Tokens: More ine-grained, compute-e icient, but datastore can be space-expensive
ff

ff

f

111

Wrapping up
When toretrieve?

How to use
retrieval?

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)
Adaptively

Intermediate layers(soft incorporation)
RETRO (Borgeaud et al. 2022)

He et al. 2021, Alon et al. 2022

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)
Adaptively
Jiang et al. 2023

Adaptive retrieval can improve e iciency

ff

112

Wrapping up
Févry et al. 2020,de Jong et al. 2021

When toretrieve?

How to use
retrieval?

Entities orentity mentions

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)
Adaptively

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

He et al. 2021, Alon et al. 2022

Adaptively
Jiang et al. 2023

Entities or entity mentions instead of every token or chunk113

Wrapping up
Févry et al. 2020,de Jong et al. 2021

When toretrieve?

How to use
retrieval?

Entities orentity mentions

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokenskNN-LM (Khandelwal et al. 2020)
Adaptively

Intermediate layers(soft incorporation)

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)

RETRO (Borgeaud et al. 2022)

He et al. 2021, Alon et al. 2022

Adaptively

Retrieve its own input

Jiang et al. 2023

Wu et al. 2022, Bertsch et al. 2023, Rubin & Brent, 2023

We can use a similar approach for long-sequence modeling114

Verga et al. 2020

Extend to use
fact triples

Wrapping up

Izcard et al. 2022

(not covered in this section)

Févry et al. 2020,de Jong et al. 2021

When toretrieve?

How to use
retrieval?

Entities orentity mentions

Use encoder-decoder to scale# of chunks to process

REALM (Guu et al. 2020)

What toretrieve?

Text chunks

Input layer(concatenation)

Once
Every n tokens

TokensIntermediate layers(soft incorporation)

kNN-LM (Khandelwal et al. 2020)
Adaptively
He et al. 2021, Alon et al. 2022
Drozdov et al. 2022
Soft adaptation for
better expressivity

RETRO (Borgeaud et al. 2022)
Min et al. 2023
Removing interpolation

Retrieve-in-context(Ram et al. 2023, Shi et al. 2023)
Adaptively

Retrieve its own input

Jiang et al. 2023

Wu et al. 2022, Bertsch et al. 2023, Rubin & Brent, 2023

115

Wrapping up

116

Wrapping up

116

Wrapping up

Still largely under-explored!
116

Wrapping up

We didn’t cover anything about training →Section 4!
We brie y saw some results but not extensively
on downstream tasks → Section 5!
fl117