Section 6
Multilingual & Multimodal:

1Retrieval-based LM for diverse knowledge sources
Raw text corpus

QueryLM

Retriever

Datastore2

Retrieval-based LM for diverse knowledge sources
Raw text corpus
Toronto (pronunciación en inglés: /tʲəˈɹɑntʲoʊ/ ( escuchar), localmente /tʲəˈɹɑnoʊ/, /ˈtʲɹɑnoʊ/) es la capital de la provincia
de Ontario3টরউ র আেমিরকা মহােদেশররা কানাডার দ ণ-পূ অংেশঅব ত অ িরও েদেশর রাজধানীশহর। এ কানাডার হ ম মহানগর
এলাকা ( ের ল ২ হ ম)

QueryLM

Retriever

Datastoreত্তত্তর্ববৃবৃয়প্রক্ষিয়ান্টাত্তটিমোঁস্থিন্টোষ্ট্র3Retrieval-based LM for diverse knowledge sources
Raw text corpus

KB

Toronto (pronunciación en inglés: /tʲəˈɹɑntʲoʊ/ ( escuchar), localmente /tʲəˈɹɑnoʊ/, /ˈtʲɹɑnoʊ/) es la capital de la provincia
de Ontario3টরউ র আেমিরকা মহােদেশররা কানাডার দ ণ-পূ অংেশঅব ত অ িরও েদেশর রাজধানীশহর। এ কানাডার হ ম মহানগর
এলাকা ( ের ল ২ হ ম)

QueryLMVideo

RetrieverImages
AudioDatastoreত্ত

ত্ত

র্ব

বৃ

বৃ
য়

প্র

ক্ষি

য়া

ন্টাত্ত

টি

মোঁ

স্থি

ন্টো

ষ্ট্র

4

Multilingual Retrieval-based LMInput x
Multilingual
RetrieverDatastore(Japanese)Index

トロント名物のプーティンはどこ発祥？
(Where did Toronto's famous poutine come from?
MultilingualLM

この料理はケベック州で1950年に生まれた．

(Limited amount of data is available in Japanese datastore…)
Output y

プーティンはケベック州で1950年代に生まれた．
5

Multilingual Retrieval-based LM
Input x

トロント名物のプーティンはどこ発祥？
(Where did Toronto's famous poutine come from?

Multilingual
RetrieverDatastore(English)

Index

Datastore(Japanese)

Index

MultilingualLM

Several restaurants in the area claim to be the originators of the dish
- Le Lutin qui rit, Warwick – Restaurateur Fernand Lachance of Le Café Idea
- Le Roy Jucep, Drummondville – This drive-in restaurant served french fries with grav
La Petite Vache, Princeville – Customers would mix cheese curds with their fries

この料理はケベック州で1950年に生まれた．

プーティンはケベック州で1950年代に生まれた．発祥については諸説あるが，Output y

Le Lutin qui rit,

Le Roy Jucep, La Petite Vacheの三つのレストランが一番最初にプーティンを提供したレストランとして有力である．

yl:

6

Language biases in representation spaces
この料理はケベック州で1950年に生まれた．

Several restaurants in the area claim to be
the originators of the dish
- Le Lutin qui rit, Warwick – Restaurateur
Fernand Lachance of Le Café Idea
- Le Roy Jucep, Drummondville – This drivein restaurant served french fries with grav
La Petite Vache, Princeville – Customers
would mix cheese curds with their fries

Datastore(Japanese)

Datastore(English)yl:7

CORA (Asai et al., 2021)

Asai et al. 2021. “One question answering model for many languages with cross-lingual dense passage retrieval” 8CORA: Iteratively training multilingual LM & retriever
Initial ne-tuning of retriever and LMusing task datafi9

CORA: Iteratively training multilingual LM & retriever
Retrieve positive paragraphs in otherlanguages using language links

Links10CORA: Iteratively training multilingual LM & retriever
Add new positive / negative paragraphs
based on whether LMs can answer correctly

Links
11

CORA: Iteratively training multilingual LM & retriever
Repeat DA+training

12Resultsbase LM

Prior SOTA

CORA

50

Large gains from ne-tuned
LM without retrieval

37.5

25

12.5

0Bn

Ja

TeAvg.
13fi

F1

Resultsbase LM

Prior SOTA

CORA50Signi cantly outperforms
prior SOTA

37.5

F1

25

12.5

0Bn

JaTeAvg.

fi

14

Ablations: Effects of iterative retrieval
CORAw/o iterativeIterative training of retriever and LMgives large performance improvements

4433

F122110

XOR-TyDi

MKQA

15

Multilingual retrieval-based LMs for diverse tasks
Question Answering* CL-ReLKT (Limkonchotiwat et al., 2022): knowledge transfer for better cross-lingual retrieval training* Gen-TyDi QA (Muller et al., 2023): generate full sentence answers for cross-lingual QA.
* AfriQA (Ogundepo et al., 2023):: Cross-lingual Open-Retrieval Question Answering for African
Languages: the rst open-domain QA datasets for African languages
Fact veri cation* CONCRETE (Hung et al., 2022): Improving cross-lingual fact-checking with cross-lingual retrieval
Dialogue* Cross-lingual Knowledge-grounded Dialogue (Kim et al 2021): a Korean knowledge-grounded dialoguesystem that learns to generate Korean response given English & Korean knowledge
Event extraction* R-GQA (Du and Ji, 2022): retrieve similar QA pairs for event argument extraction.
Key-phrase generations* Retrieval-augmented Multilingual Key phrase Generation (Gao et al 2022): Using iterative training toimprove retrieval & LM for key phrase generations
fifi16

Multi-modal retrieval-based LMs
Query qMultimodal
RetrieverIndex(Text)

Input xMultimodalLM

Index(Image)

Output y

Retrieved docs z17

RA-CM3 (Yasunaga et al., 2023)

A realistic photo of statue of liberty standing
next to Washington Monument

Index(Text)
Index(Image)Yasunaga et al. 2023. “Retrieval-Augmented Multimodal Language Modeling”

18

RA-CM3 (Yasunaga et al., 2023)
Input xA realistic photo of statue of liberty standing
next to Washington Monument

Multimodal
Retriever
Output yIndex(Text)
Index(Image)

Yasunaga et al. 2023. “Retrieval-Augmented Multimodal Language Modeling”

19Results
Caption to image (MS COCO)
30

Signi cantly outperforms baseLM (CM3)

22.5

FiD

15

Lower better

DALL-E
Vanilla CM3RA-CM3Parti

7.5

005

1015

20

Parameter counts
fiYasunaga et al. 2023. “Retrieval-Augmented Multimodal Language Modeling”

20Results
Caption to image (MS COCO)
30

Outperforms DALL-E (12B) usingmuch less parameters (2.7 B)

22.5

FiD

15

Lower better

DALL-E
Vanilla CM3RA-CM3Parti

7.5

005

1015

20

Parameter countsYasunaga et al. 2023. “Retrieval-Augmented Multimodal Language Modeling”

21Results
Caption to image (MS COCO)

Achieves signi cantly better
training ef ciency

fifi

Yasunaga et al. 2023. “Retrieval-Augmented Multimodal Language Modeling”

22More applications beyond text
Multi-modal Retrieval-augmented Pre-training* RAVEAL (Hu et al 2023): Pre-training visual-language model using knowledge memory
Multi-modal Question Answering* MuRAG (Chen et al., 2022)
Multi-modal Classi cation* ALIGN (Gur et al., 2021)
Retrieval-augmented training
for molecules
Multimodalusing image and text have been actively studied* Retrieval-based Molecule Generation (Wang et al., 2023)
Retrieval-augmented 3D motion generations* ReMoDiffus (Zhang et al., 2023)

fi23

More applications beyond text
Multi-modal Retrieval-augmented Pre-training* RAVEAL (Hu et al 2023): Pretraining visual-language model using knowledge memory
Multi-modal Question Answering* MuRAG (Chen et al., 2022)
Multi-modal Classi cation* ALIGN (Gur et al., 2021)
Retrieval-augmented training for molecules* Retrieval-based Molecule Generation (Wang et al., 2023)
Retrieval-augmented 3D motion generations* ReMoDiffus (Zhang et al., 2023)

New extensions for new input / output modality!

fi24

Wrapping up
Raw text corpus

KB

Extension to multilingual
Cross-lingual retrieval and generation to overcome
datastore scarcity in many world languages

Toronto (pronunciación en inglés: /tʲəˈɹɑntʲoʊ/ ( escuchar), localmente /tʲəˈɹɑnoʊ/, /ˈtʲɹɑnoʊ/) es la capitalটরউ রআেমিরকা মহােদেশররা কানাডার দ ণ-পূঅংেশ

VideoImages

Extension to multimodal
Key effectivenesses (Section 5; long-tail, ef ciency)
apply to diverse modality

Audiofiর্বক্ষিত্তন্টোষ্ট্র

25