Few-shot Learning with Multilingual
Generative Language Models
Meta AI

Presented by Abdullah Al Ishtiaq

Overview●Motivation●

Model architecture and training●Dataset●Tasks●Results●Strengths●

Weaknesses

Motivation●

Natural Language Research are, in many cases, dominated byhigh-resource languages like English●

Training data of Large Language Models (LLM), e.g., GPT-3, mostly
include English (~93%)●

Performance on mid- and low-resource languages is not adequate●

Other multilingual models, e.g., mBERT, XLM-R, mT5, mBART, etc.,
require ﬁnetuning for downstream applications●

Multilingual few-shot learning capabilities of LLMs are not well studied

XGLM
●

XGLM presents four multilingual generative models of different sizes●

Training corpus includes 30 diverse languages with 500B tokens●Achieves state-of-the-art on diverse multilingual NLP tasks○ Commonsense reasoning○ Anaphora resolution○ Natural language inference○ Paraphrasing○ Machine translation●

Comprehensively studies zero-shot and few-shot applications andprompt generation techniquesXGLM Models●Decoder-only CausalLanguage Model (CLM)●Transformer architecture
similar to GPT-3●Four models with 564M, 1.7B,
2.9B and 7.5B parameters●256 A100 GPUs for about 3weeks

Fig: Model details. l: layers, h: hidden dim

DataTasks: Commonsense reasoning (XCOPA) [1]Tasks: Anaphora resolution (XWinograd) [2]Tasks: Natural language inference (XNLI) [3]Tasks: Paraphrasing (PAWS-X) [4]Prompts●

Three approaches for obtaining prompts for non-English tasks○ Handcrafted prompts○ Translating from English prompts○ Cross-lingual prompts●Also evaluates Cross lingual demonstrations●This enables cheap transfer from high-resource to low-resourcelanguage

Prompts

Results: Prompt strategy

Prompt Language

Results: Prompt Language

Results: Comparison

Results: Comparison

Results: Machine translation

Results: English Tasks

Results: XGLM scale

Results: XGLM scale

Strengths●Provides a balanced dataset for multilingual NLP research●

Demonstrates different prompting strategies for multilingual tasks●Evaluates zero and few shot performance of XGLM comprehensivelyon diverse languages

Weaknesses●Does not provide any new model or does not discuss how the priormodels are appropriate for multilingual generation tasks●Uses only 30 languages, whereas, despite uneven ratio, GPT-3 istrained on 118 languages●No comparison or reference to the zero or few-shot performance ofother multilingual models like mBERT, XLM-R, mT5, mBART●Degrades performance on English tasks

Thanks
Questions?

References
[1]
[2]

[3]
[4]

Ponti, Edoardo Maria, et al. "XCOPA: A multilingual dataset for causal commonsense reasoning." arXiv
preprint arXiv:2005.00333 (2020).
Emelin, Denis, and Rico Sennrich. "Wino-X: Multilingual Winograd schemas for commonsense reasoning
and coreference resolution." Proceedings of the 2021 Conference on Empirical Methods in NaturalLanguage Processing. 2021.
Conneau, Alexis, et al. "XNLI: Evaluating cross-lingual sentence representations." arXiv preprint
arXiv:1809.05053 (2018).
Yang, Yinfei, et al. "PAWS-X: A cross-lingual adversarial dataset for paraphrase identiﬁcation." arXiv
preprint arXiv:1908.11828 (2019).