https://tinyurl.com/
retrieval-lm-tutorial
Join sli.do for Q & A!

ACL 2023 Tutorial:

Retrieval-based Language Models
and Applications
Akari Asai, Sewon Min, Zexuan Zhong, Danqi Chen
https://acl2023-retrieval-lm.github.io/

July 9, 2023

About presenters

Akari Asai

Sewon Min

Zexuan Zhong

Danqi Chen

PhD student@UW

PhD student@UW

PhD student@Princeton

Faculty@Princeton

2Participation and Q & A• All tutorial slides and reading list are available at:https://acl2023-retrieval-lm.github.io/• Throughout the tutorial, we will provide Q & A on sli.do (also on the website):https://tinyurl.com/retrieval-lm-tutorial

Ask & upvote questions anytime!
(+10 minutes Q & A in 1st and 2nd half)3Retrieval-based language models (LMs)
Retrieval-based LMs = Retrieval + LMs• It is a language model P(xn | x1, x2, ⋯, xn−1)
The capital city of Ontario is ___(can be broadly extended to masked language
models or encoder-decoder models)• It retrieves from an external datastore (at least during inference time)(Also referred to semiparametricand non-parametric models)4

The age of large language models (LLMs)• Transformers-based, fullyparametric• Trained on next-token
prediction tasks (+ RLHF; notthe focus today)• Model size ↑, data size↑
Image: https://lifearchitect.ai/models/5

Retrieval for knowledge-intensive NLP tasks
Representative tasks: open-domain QA, fact checking, entity linking, ..

Image: http://ai.stanford.edu/blog/retrieval-based-NLP/

Drives a lot of research on better algorithms for dense
retrieval, e.g., DPR (Karpukhin et al., 2020), ColBERT(Khattab and Zaharia, 2020), ANCE (Xiong et al., 2021),
Contriever (Izacard et al., 2022), …

Why retrieval → LMs?

6

Why retrieval-based LMs?
LLMs can’t memorize all (long-tail) knowledge in their parameters
List 5 important papers authored by Geo rey Hinton
Geo rey Hinton is a renowned computer scientist …
Here are ve important papers authored by him:

What is Kathy Saltzman’s occupation?

1. "Learning Internal Representations by Error
Propagation" (with D. E. Rumelhart and R. J.
Williams) - This paper, published in 1986, ..
2. "Deep Boltzmann Machines" (with R.
Salakhutdinov) - Published in 2009, ..
…
4. "Deep Learning" (with Y. Bengio and A. Courville) Published as a book in 2016,…
5. "Attention Is All You Need" (with V. Vaswani, N.
Shazeer, et al.) - Published in 2017, this paper
introduced the Transformer model,…

(Mallen et al., 2023)

GPT-3 davinci-003: 20%-30% accuracy

fffiff7

Why retrieval-based LMs?
LLMs’ knowledge is easily outdated and hard to update
Who is the CEO of Twitter?

As of my knowledge cuto in
September 2021, the CEO of
Twitter is Jack Dorsey….

• Existing knowledge editing methods are still
NOT scalable (active research!)• The datastore can be easily updated andexpanded - even without retraining!

New!

ff

8

Why retrieval-based LMs?
LLMs’ output is challenging to interpret and verify

Generating text with citations

Can trace knowledge sourcefrom retrieval results - betterinterpretability & control(Nakano et al. 2021; Menick et al.,2022; Gao et al., 2023)
9

Why retrieval-based LMs?
LLMs’ output is challenging to interpret and verify

10Why retrieval-based LMs?
LLMs are shown to easily leak private training data

Individualization on private data by storing it in the datastore
Carlini et al. 2021. “Extracting Training Data from Large Language Models”

11Why retrieval-based LMs?
LLMs are *large* and expensive to train and run

LMvs.

LMDatastore
Long-term goal: can we possibly reduce the training andinference costs, and scale down the size of LLMs?
e.g., RETRO (Borgeaud et al., 2021): “obtains
comparable performance to GPT-3 on the Pile,despite using 25x fewer parameters”
12Goals of the tutorial• We will cover a number of key developments on retrieval-based LMs (mostly 2020–2023)• Problem de nition• Architecture design• Training methods• Applications and extensions• This tutorial is cutting-edge, and we are still far fromunderstanding how to best develop retrieval-based LMs
compared to parametric LLMs:• Taxonomies of existing research and key insights• Our perspectives on the current challenges & open problemsfi13

ScheduleQ&A

30min co ee breakff14