SlowGuess/NAACL_2021_OCR huggingface dataset:
ISO Code	Language	Tokens (B)	Pages (M)	mT5 (%)	ISO Code	Language	Tokens (B)	Pages (M)	mT5 (%)
en	English	2,733	3,067	5.67	mk	Macedonian	1.8	2.1	0.62
ru	Russian	713	756	3.71	ml	Malayalam	1.8	2.1	0.62
es	Spanish	433	416	3.09	mn	Mongolian	2.7	2.1	0.62
de	German	347	397	3.05	ur	Urdu	2.4	1.9	0.61
fr	French	318	333	2.89	be	Belarusian	2.0	1.7	0.59
it	Italian	162	186	2.43	la	Latin	1.3	1.7	0.58
pt	Portuguese	146	169	2.36	eu	Basque	1.4	1.6	0.57
pl	Polish	130	126	2.15	tg	Tajik	1.4	1.3	0.54
nl	Dutch	73	96	1.98	te	Telugu	1.3	1.2	0.52
tr	Turkish	71	88	1.93	fy	West Frisian	0.4	1.1	0.51
ja	Japanese	164	87	1.92	kn	Kannada	1.1	1.1	0.51
vi	Vietnamese	116	79	1.87	ky	Kyrgyz	1.0	1.0	0.50
id	Indonesian	69	70	1.80	sw	Swahili	1.0	1.0	0.50
cs	Czech	63	60	1.72	so	Somali	1.4	0.9	0.48
zh	Chinese	39	55	1.67	my	Burmese	0.9	0.8	0.47
fa	Persian	52	54	1.67	uz	Uzbek	0.9	0.8	0.46
ar	Arabic	57	53	1.66	km	Khmer	0.6	0.8	0.46
sv	Swedish	45	49	1.61	-	Russian (Latin)	0.9	0.7	0.46
ro	Romanian	52	46	1.58	sd	Sindhi	1.6	0.7	0.45
el	Greek	43	42	1.54	gu	Gujarati	0.8	0.6	0.43
uk	Ukrainian	41	39	1.51	-	Hindi (Latin)	0.6	0.6	0.43
hu	Hungarian	39	37	1.48	jv	Javanese	0.3	0.6	0.42
da	Danish	29	29	1.38	zu	Zulu	0.2	0.6	0.42
fi	Finnish	25	27	1.35	si	Sinhala	0.8	0.5	0.41
no	Norwegian	27	25	1.33	-	Japanese (Latin)	0.3	0.5	0.41
bg	Bulgarian	22	23	1.29	eo	Esperanto	0.7	0.5	0.40
hi	Hindi	24	19	1.21	co	Corsican	0.2	0.5	0.40
sk	Slovak	18	18	1.19	ga	Irish	0.5	0.5	0.40
ko	Korean	26	16	1.14	-	Greek (Latin)	0.4	0.4	0.39
th	Thai	11	15	1.14	-	Chinese (Latin)	0.2	0.4	0.37
ca	Catalan	13	14	1.12	pa	Punjabi	0.6	0.4	0.37
ms	Malay	13	13	1.09	ceb	Cebuano	0.2	0.4	0.36
iw	Hebrew	17	12	1.06	mg	Malagasy	0.2	0.3	0.36
lt	Lithuanian	11	11	1.04	ps	Pashto	0.4	0.3	0.36
sl	Slovenian	8.8	8.5	0.95	sn	Shona	0.2	0.3	0.35
mr	Marathi	14	7.8	0.93	gd	Scottish Gaelic	0.4	0.3	0.35
bn	Bengali	7.3	7.4	0.91	ku	Kurdish	0.4	0.3	0.34
et	Estonian	6.9	6.9	0.89	hmn	Hmong	0.2	0.3	0.34
lv	Latvian	7.0	6.4	0.87	su	Sundanese	0.1	0.3	0.34
az	Azerbaijani	4.4	5.3	0.82	ht	Haitian Creole	0.2	0.3	0.33
gl	Galician	2.4	4.6	0.79	ha	Hausa	0.2	0.2	0.33
cy	Welsh	4.9	4.1	0.76	ny	Chichewa	0.1	0.2	0.29
sq	Albanian	4.0	4.1	0.76	am	Amharic	0.3	0.2	0.29
ta	Tamil	3.4	3.5	0.73	-	Bulgarian (Latin)	0.09	0.2	0.29
sr	Serbian	4.3	3.4	0.72	yi	Yiddish	0.3	0.1	0.28
ne	Nepali	3.2	2.9	0.69	lo	Lao	0.1	0.1	0.28
lb	Luxembourgish	1.0	2.7	0.68	mi	Maori	0.1	0.1	0.25
hy	Armenian	2.4	2.4	0.65	sm	Samoan	0.09	0.1	0.25
kk	Kazakh	3.1	2.4	0.65	ig	Igbo	0.09	0.09	0.24
ka	Georgian	2.5	2.3	0.64	haw	Hawaiian	0.09	0.08	0.24
mt	Maltese	5.2	2.3	0.64	xh	Xhosa	0.06	0.07	0.22
af	Afrikaans	1.7	2.2	0.63	st	Sotho	0.08	0.07	0.22
fil	Filipino	2.1	2.1	0.62	yo	Yoruba	0.05	0.05	0.20
is	Icelandic	2.6	2.1	0.62
Table 6: Statistics of the mC4 corpus, totaling 6.6B pages and 6.3T tokens. The "mT5" column indicates the percentage of mT5 training data coming from a given language, using the default exponential smoothing value of  $\alpha = 0.3$ . We list 107 "languages" as detected by cld3, but note six of these (marked "Latin") are just Romanized variants of existing languages.
A mC4 Corpus Language Distribution
B Per-Language Results on All Tasks
Model	en	ar	bg	de	el	es	fr	hi	ru	sw	th	tr	ur	vi	zh	avg
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	80.8	64.3	68.0	70.0	65.3	73.5	73.4	58.9	67.8	49.7	54.1	60.9	57.2	69.3	67.8	65.4
XLM	82.8	66.0	71.9	72.7	70.4	75.5	74.3	62.5	69.9	58.1	65.5	66.4	59.8	70.7	70.2	69.1
XLM-R	88.7	77.2	83.0	82.5	80.8	83.7	82.2	75.6	79.1	71.2	77.4	78.0	71.7	79.3	78.2	79.2
mT5-Small	79.6	65.2	71.3	69.2	68.6	72.7	70.7	62.5	70.1	59.7	66.3	64.4	59.9	66.3	65.8	67.5
mT5-Base	84.7	73.3	78.6	77.4	77.1	80.3	79.1	70.8	77.1	69.4	73.2	72.8	68.3	74.2	74.1	75.4
mT5-Large	89.4	79.8	84.1	83.4	83.2	84.2	84.1	77.6	81.5	75.4	79.4	80.1	73.5	81.0	80.3	81.1
mT5-XL	90.6	82.2	85.4	85.8	85.4	81.3	85.3	80.4	83.7	78.6	80.9	82.0	77.0	81.8	82.7	82.9
mT5-XXL	91.6	84.5	87.7	87.3	87.3	87.8	86.9	83.2	85.1	80.3	81.7	83.8	79.8	84.6	83.6	85.0
Translate-train (models fine-tune on English training data plus translations in all target languages)
mT5-Small	78.3	70.3	74.8	73.6	73.6	74.9	74.1	68.3	73.6	67.6	72.0	70.8	65.1	70.2	73.2	72.0
mT5-Base	85.8	78.8	82.2	81.6	81.4	83.0	82.1	77.0	81.1	74.8	78.6	78.4	73.3	78.9	80.2	79.8
mT5-Large	90.1	83.3	86.8	85.9	85.8	87.2	86.1	82.6	84.7	79.7	82.9	83.8	78.8	84.0	84.4	84.4
mT5-XL	91.0	84.0	87.5	87.2	86.7	88.5	87.4	83.1	85.3	80.9	83.2	84.7	80.3	84.8	85.0	85.3
mT5-XXL	92.4	87.1	88.7	89.2	88.7	89.4	88.7	85.3	86.4	83.4	84.5	86.4	82.9	86.6	86.2	87.1
Table 7: XNLI accuracy scores for each language.
Model	en	de	es	fr	ja	ko	zh	avg
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	94.0	85.7	87.4	87.0	73.0	69.6	77.0	81.9
XLM	94.0	85.9	88.3	87.4	69.3	64.8	76.5	80.9
XLM-R	94.7	89.7	90.1	90.4	78.7	79.0	82.3	86.4
mT5-Small	92.2	86.2	86.1	86.6	74.7	73.5	77.9	82.4
mT5-Base	95.4	89.4	89.6	91.2	79.8	78.5	81.1	86.4
mT5-Large	96.1	91.3	92.0	92.7	82.5	82.7	84.7	88.9
mT5-XL	96.0	92.8	92.7	92.4	83.6	83.1	86.5	89.6
mT5-XXL	96.3	92.9	92.6	92.7	84.5	83.9	87.2	90.0
Translate-train (models fine-tune on English training data plus translations in all target languages)
mT5-Small	87.9	81.4	83.1	84.1	74.2	71.7	76.7	79.9
mT5-Base	95.5	90.9	91.4	92.5	83.6	84.8	86.4	89.3
mT5-Large	96.4	92.7	93.3	93.6	86.5	87.4	88.4	91.2
mT5-XL	96.4	92.5	93.1	93.6	85.5	86.9	89.0	91.0
mT5-XXL	96.1	92.9	93.6	94.2	87.0	87.9	89.0	91.5
Table 8: PAWS-X accuracy scores for each language.
Model	en	af	ar	bg	bn	de	el	es	et	eu	fa	fi	fr	he	hi	hu	id	it	ja	jv
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	85.2	77.4	41.1	77.0	70.0	78.0	72.5	77.4	75.4	66.3	46.2	77.2	79.6	56.6	65.0	76.4	53.5	81.5	29.0	66.4
mT5-Small	80.6	67.0	36.2	59.8	60.0	66.1	54.0	63.6	58.4	42.3	25.3	64.5	74.6	39.6	57.4	61.5	46.6	73.2	28.8	49.6
mT5-Base	83.2	73.8	45.4	62.1	67.1	72.5	57.0	70.3	67.3	49.2	30.4	68.6	78.6	46.1	67.6	64.7	49.7	78.9	35.0	56.9
mT5-Large	84.2	74.7	55.0	60.6	64.5	75.2	68.2	74.2	66.4	48.4	51.4	65.8	82.4	55.8	69.0	67.3	51.1	80.6	43.0	57.1
mT5-XL	86.3	79.3	60.2	80.3	78.1	80.4	78.3	74.5	71.8	52.2	61.5	70.1	85.9	65.3	76.3	71.9	56.8	83.2	47.7	63.2
mT5-XXL	86.6	81.1	66.5	85.1	78.8	82.0	79.1	85.8	74.1	55.1	59.6	70.5	86.8	66.1	78.4	74.2	75.0	86.3	51.0	69.1
ka	kk	ko	ml	mr	ms	my	nl	pt	ru	sw	ta	te	th	tl	tr	ur	vi	yo	zh	avg
mBERT	64.6	45.8	59.6	52.3	58.2	72.7	45.2	81.8	80.8	64.0	67.5	50.7	48.5	3.6	71.7	71.8	36.9	71.8	44.9	42.7	62.2
mT5-Small	53.2	22.6	26.6	38.2	38.1	68.8	28.9	75.0	70.5	46.5	54.8	37.5	32.5	7.0	68.7	56.0	24.8	63.8	58.8	37.7	50.5
mT5-Base	49.9	22.1	33.9	45.5	43.8	68.9	36.4	80.1	76.0	53.2	62.4	40.8	41.8	8.5	74.1	58.4	38.4	72.1	56.5	41.0	55.7
mT5-Large	58.2	23.3	36.2	46.3	46.5	65.8	32.2	82.7	79.6	50.2	72.3	46.4	44.5	9.1	79.0	65.1	44.2	77.1	47.2	44.0	58.5
mT5-XL	66.0	31.6	38.1	54.1	57.6	74.5	42.6	85.5	85.2	66.9	72.8	49.0	54.7	9.6	84.1	67.4	64.7	79.6	59.0	53.9	65.5
mT5-XXL	66.1	39.2	43.2	54.1	62.8	77.4	44.1	87.6	86.8	71.4	73.1	56.5	59.4	10.2	85.1	71.6	81.2	84.6	66.4	56.9	69.2
Model	en	af	ar	bg	bn	de	el	es	et	eu	fa	fi	fr	he	hi	hu	id	it	ja	jv
In-language multitask (models fine-tuned on gold data in all target languages)
mBERT	85.4	92.0	89.6	93.5	95.3	90.1	91.1	93.3	92.4	97.6	92.4	91.6	86.5	88.9	98.9	93.5	93.8	92.6	74.6	91.5
mT5-Small	80.8	92.1	87.8	91.9	92.8	87.2	85.5	91.6	91.4	90.2	73.7	89.2	88.8	83.5	87.9	90.9	93.1	90.1	73.0	89.4
mT5-Base	84.2	92.1	89.6	93.4	94.2	89.4	87.1	93.1	92.9	92.3	74.8	91.5	91.2	86.2	90.6	92.7	93.8	92.2	73.5	89.4
mT5-Large	86.0	93.6	91.3	94.4	94.0	91.1	88.6	93.9	94.3	94.1	76.1	93.1	92.4	88.9	92.3	94.4	95.0	93.6	75.2	92.0
mT5-XL	87.7	94.4	93.0	95.2	94.4	92.6	89.7	94.7	95.4	95.1	77.0	94.4	93.4	91.2	93.2	95.2	95.5	94.7	78.6	94.9
mT5-XXL	88.5	95.2	94.1	96.0	95.4	93.3	90.5	95.4	96.0	95.8	77.5	95.2	94.0	92.8	94.3	96.0	96.1	95.6	80.6	92.8
ka	kk	ko	ml	mr	ms	my	nl	pt	ru	sw	ta	te	th	tl	tr	ur	vi	yo	zh	avg
mBERT	88.0	88.2	89.0	84.3	88.5	94.8	78.1	93.0	93.5	89.6	91.8	86.0	82.3	75.3	94.9	93.1	94.4	92.9	84.8	82.5	89.2
mT5-Small	87.1	85.8	84.2	79.9	85.0	93.8	64.8	90.4	90.9	86.2	77.9	87.1	75.0	76.4	95.4	91.9	94.8	91.3	86.4	78.7	86.4
mT5-Base	88.8	88.7	86.1	81.8	87.2	94.7	72.2	92.3	92.4	88.2	79.4	88.4	78.1	73.9	96.4	93.1	96.0	92.3	91.9	80.6	88.2
mT5-Large	91.1	89.8	89.2	84.1	89.3	96.0	74.4	93.9	93.8	90.4	80.7	91.1	81.4	73.9	97.3	94.8	96.1	93.8	91.5	82.4	89.7
mT5-XL	92.6	91.7	91.1	85.3	91.2	95.9	84.7	94.8	94.4	91.6	80.9	92.6	84.3	78.5	98.0	95.6	97.4	94.9	93.7	85.0	91.3
mT5-XXL	93.8	94.3	92.7	86.6	93.1	97.3	83.3	95.5	95.4	92.7	83.0	93.2	86.1	79.7	98.0	96.2	97.4	95.5	93.3	86.2	92.2
Table 9: WikiAnn NER F1 scores for each language.
Model	en	ar	de	el	es	hi	ru	th	tr	vi	zh	avg
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	83.5 / 72.2	61.5 / 45.1	70.6 / 54.0	62.6 / 44.9	75.5 / 56.9	59.2 / 46.0	71.3 / 53.3	42.7 / 33.5	55.4 / 40.1	69.5 / 49.6	58.0 / 48.3	64.5 / 49.4
XLM	74.2 / 62.1	61.4 / 44.7	66.0 / 49.7	57.5 / 39.1	68.2 / 49.8	56.6 / 40.3	65.3 / 48.2	35.4 / 24.5	57.9 / 41.2	65.8 / 47.6	49.7 / 39.7	59.8 / 44.3
XLM-R	86.5 / 75.7	68.6 / 49.0	80.4 / 63.4	79.8 / 61.7	82.0 / 63.9	76.7 / 59.7	80.1 / 64.3	74.2 / 62.8	75.9 / 59.3	79.1 / 59.0	59.3 / 50.0	76.6 / 60.8
mT5-Small	78.5 / 66.1	51.4 / 34.0	63.8 / 45.9	53.8 / 33.4	67.0 / 50.3	47.8 / 34.5	50.5 / 30.1	54.0 / 44.5	55.7 / 38.9	58.1 / 41.3	58.9 / 48.7	58.1 / 42.5
mT5-Base	84.6 / 71.7	63.8 / 44.3	73.8 / 54.5	59.6 / 35.6	74.8 / 56.1	60.3 / 43.4	57.8 / 34.7	57.6 / 45.7	67.9 / 48.2	70.7 / 50.3	66.1 / 54.1	67.0 / 49.0
mT5-Large	88.4 / 77.3	75.2 / 56.7	80.0 / 62.9	77.5 / 57.6	81.8 / 64.2	73.4 / 56.6	74.7 / 56.9	73.4 / 62.0	76.5 / 56.3	79.4 / 60.3	75.9 / 65.5	77.8 / 61.5
mT5-XL	88.8 / 78.1	77.4 / 60.8	80.4 / 63.5	80.4 / 61.2	82.7 / 64.5	76.1 / 60.3	76.2 / 58.8	74.2 / 62.5	77.7 / 58.4	80.5 / 60.8	80.5 / 71.0	79.5 / 63.6
mT5-XXL	90.9 / 80.1	80.3 / 62.6	83.1 / 65.5	83.3 / 65.5	85.1 / 68.1	81.7 / 63.6	79.3 / 66.1	77.8 / 66.1	80.2 / 60.9	83.1 / 63.6	83.1 / 73.4	82.5 / 66.8
Translate-train (models fine-tune on English training data plus translations in all target languages)
mT5-Small	74.0 / 61.2	61.0 / 45.0	66.0 / 50.2	64.1 / 47.2	67.5 / 50.8	60.2 / 43.7	64.4 / 46.7	58.9 / 52.9	59.0 / 39.4	63.5 / 46.0	68.2 / 61.2	64.3 / 49.5
mT5-Base	83.1 / 70.3	72.4 / 55.2	76.9 / 59.7	76.8 / 58.8	79.0 / 61.2	71.4 / 53.4	76.1 / 58.5	67.9 / 62.0	72.5 / 51.4	75.9 / 56.3	76.9 / 69.7	75.3 / 59.7
mT5-Large	87.3 / 75.5	79.4 / 62.7	82.7 / 66.0	81.8 / 63.5	83.8 / 66.1	78.0 / 59.8	81.9 / 66.3	74.7 / 68.2	80.2 / 59.2	80.4 / 60.8	83.2 / 76.9	81.2 / 65.9
mT5-XL	88.5 / 77.1	80.9 / 65.4	83.4 / 66.7	83.6 / 64.9	84.9 / 68.2	79.6 / 63.1	82.7 / 67.1	78.5 / 72.9	82.4 / 63.8	82.4 / 64.1	83.2 / 75.9	82.7 / 68.1
mT5-XXL	91.3 / 80.3	83.4 / 68.2	85.0 / 68.2	85.9 / 68.9	87.4 / 70.8	83.7 / 68.2	85.2 / 70.4	80.2 / 74.5	84.4 / 67.7	85.3 / 67.1	85.7 / 80.0	85.2 / 71.3
Table 10: XQuAD results (F1/EM) for each language.
Model	en	ar	de	es	hi	vi	zh	avg
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	80.2 / 67.0	52.3 / 34.6	59.0 / 43.8	67.4 / 49.2	50.2 / 35.3	61.2 / 40.7	59.6 / 38.6	61.4 / 44.2
XLM	68.6 / 55.2	42.5 / 25.2	50.8 / 37.2	54.7 / 37.9	34.4 / 21.1	48.3 / 30.2	40.5 / 21.9	48.5 / 32.6
XLM-R	83.5 / 70.6	66.6 / 47.1	70.1 / 54.9	74.1 / 56.6	70.6 / 53.1	74.0 / 52.9	62.1 / 37.0	71.6 / 53.2
mT5-Small	77.2 / 63.0	44.7 / 27.3	53.3 / 35.7	60.1 / 41.5	43.0 / 29.2	52.9 / 33.2	51.3 / 29.7	54.6 / 37.1
mT5-Base	81.7 / 66.9	57.1 / 36.9	62.1 / 43.2	67.1 / 47.2	55.4 / 37.9	65.9 / 44.1	61.6 / 38.6	64.4 / 45.0
mT5-Large	84.9 / 70.7	65.3 / 44.6	68.9 / 51.8	73.5 / 54.1	66.9 / 47.7	72.5 / 50.7	66.2 / 42.0	71.2 / 51.7
mT5-XL	85.5 / 71.9	68.0 / 47.4	70.5 / 54.4	75.2 / 56.3	70.5 / 51.0	74.2 / 52.8	70.5 / 47.2	73.5 / 54.4
mT5-XXL	86.7 / 73.5	70.7 / 50.4	74.0 / 57.8	76.8 / 58.4	75.6 / 57.3	76.4 / 56.0	71.8 / 48.8	76.0 / 57.4
Translate-train (models fine-tune on English training data plus translations in all target languages)
mT5-Small	70.5 / 56.2	49.3 / 31.0	55.6 / 40.6	60.5 / 43.0	50.4 / 32.9	55.2 / 36.3	54.4 / 31.6	56.6 / 38.8
mT5-Base	80.7 / 66.3	61.1 / 40.7	65.5 / 49.2	70.7 / 52.1	63.6 / 44.3	68.0 / 47.6	63.5 / 39.4	67.6 / 48.5
mT5-Large	85.3 / 72.0	68.5 / 47.7	71.6 / 55.8	75.7 / 57.1	71.8 / 52.6	74.3 / 54.0	70.1 / 47.1	73.9 / 55.2
mT5-XL	86.0 / 73.0	70.0 / 49.8	72.7 / 56.8	76.9 / 58.3	73.4 / 55.0	75.4 / 55.0	71.4 / 48.4	75.1 / 56.6
mT5-XXL	86.5 / 73.5	71.7 / 51.4	74.9 / 58.7	78.8 / 60.3	76.6 / 58.5	77.1 / 56.3	72.5 / 49.8	76.9 / 58.3
Table 11: MLQA results (F1/EM) for each language.
Model	en	ar	bn	fi	id	ko	ru	sw	te	avg
Cross-lingual zero-shot transfer (models fine-tune on English data only)
mBERT	75.3 / 63.6	62.2 / 42.8	49.3 / 32.7	59.7 / 45.3	64.8 / 45.8	58.8 / 50.0	60.0 / 38.8	57.5 / 37.9	49.6 / 38.4	59.7 / 43.9
XLM	66.9 / 53.9	59.4 / 41.2	27.2 / 15.0	58.2 / 41.4	62.5 / 45.8	14.2 / 5.1	49.2 / 30.7	39.4 / 21.6	15.5 / 6.9	43.6 / 29.1
XLM-R	71.5 / 56.8	67.6 / 40.4	64.0 / 47.8	70.5 / 53.2	77.4 / 61.9	31.9 / 10.9	67.0 / 42.1	66.1 / 48.1	70.1 / 43.6	65.1 / 45.0
mT5-small	58.9 / 48.2	44.3 / 28.2	18.2 / 9.7	42.0 / 25.8	46.4 / 32.0	27.5 / 18.5	43.7 / 27.5	35.4 / 22.8	16.1 / 10.5	36.9 / 24.8
mT5-Base	72.8 / 60.2	68.9 / 50.3	44.9 / 28.3	67.9 / 53.1	73.3 / 55.2	48.6 / 34.4	58.0 / 35.7	59.9 / 42.5	46.2 / 34.1	60.0 / 43.8
mT5-Large	75.1 / 63.0	67.2 / 45.9	51.9 / 31.9	69.6 / 53.1	72.5 / 55.9	57.4 / 44.2	62.8 / 37.9	71.2 / 51.7	65.0 / 46.3	65.8 / 47.8
mT5-XL	79.6 / 68.9	82.4 / 65.9	72.8 / 54.9	79.9 / 65.7	82.6 / 68.5	68.3 / 57.6	73.9 / 49.5	77.3 / 59.7	79.1 / 60.2	77.3 / 61.2
mT5-XXL	84.5 / 73.2	84.6 / 68.8	82.5 / 70.8	82.8 / 70.1	85.8 / 73.3	77.2 / 66.3	77.4 / 57.0	83.8 / 69.7	79.6 / 59.8	82.0 / 67.7
Translate-train (models fine-tune on English training data plus translations in all target languages)
mT5-Small	58.2 / 47.3	55.9 / 39.3	40.3 / 23.0	51.7 / 37.9	62.2 / 46.0	41.5 / 30.8	51.6 / 35.0	51.8 / 37.1	34.8 / 24.2	49.8 / 35.6
mT5-Base	71.0 / 59.1	71.8 / 55.5	56.8 / 36.3	71.5 / 58.8	76.8 / 60.5	61.5 / 49.3	66.1 / 47.5	67.0 / 50.7	55.3 / 41.7	66.4 / 51.0
mT5-Large	77.2 / 65.7	80.3 / 64.1	71.8 / 54.9	75.9 / 61.3	81.7 / 68.0	69.7 / 56.9	75.0 / 56.8	76.9 / 60.3	73.1 / 53.1	75.7 / 60.1
mT5-XL	81.7 / 68.9	82.1 / 66.0	79.0 / 64.6	79.5 / 65.3	84.9 / 71.2	71.8 / 57.6	78.7 / 60.6	82.4 / 67.3	80.8 / 63.8	80.1 / 65.0
mT5-XXL	83.3 / 72.3	83.9 / 66.6	83.3 / 71.7	83.0 / 69.1	85.9 / 71.5	77.6 / 63.4	81.1 / 64.4	86.0 / 75.4	85.2 / 70.7	83.3 / 69.4
In-language multitask (models fine-tuned on gold data in all target languages)
mT5-Small	67.8 / 57.0	79.5 / 67.2	73.1 / 59.3	72.3 / 59.5	78.7 / 67.8	59.1 / 51.1	71.2 / 58.0	79.1 / 70.9	84.1 / 72.5	74.0 / 62.7
mT5-Base	74.6 / 63.2	82.8 / 69.7	79.7 / 67.3	78.5 / 66.4	84.9 / 73.5	70.7 / 62.7	76.1 / 62.3	81.7 / 72.5	87.2 / 77.4	79.7 / 68.4
mT5-Large	81.9 / 71.1	87.3 / 75.6	86.7 / 79.6	85.1 / 73.5	87.3 / 77.5	79.2 / 70.3	83.5 / 70.2	85.8 / 78.0	90.6 / 81.9	85.3 / 75.3
mT5-XL	83.8 / 74.3	88.4 / 76.7	88.7 / 83.2	86.7 / 75.6	90.1 / 81.4	82.9 / 74.6	85.3 / 73.2	90.1 / 82.8	92.4 / 84.0	87.6 / 78.4
mT5-XXL	85.4 / 75.2	89.4 / 77.6	90.3 / 85.0	87.7 / 77.1	90.7 / 82.8	84.2 / 75.0	86.9 / 75.5	90.8 / 83.6	92.4 / 83.7	88.7 / 79.5
(Human)	84.2 / -	85.8 / -	94.8 / -	87.0 / -	92.0 / -	82.0 / -	96.3 / -	92.0 / -	97.1 / -	90.1 / -
Table 12: TyDi QA GoldP results (F1/EM) for each language.
C Per-Language Results of Ablation Models
Model	ar	bg	de	el	en	es	fr	hi	ru	sw	th	tr	ur	vi	zh	avg
Baseline (mT5-Large)	79.8	84.1	83.4	83.2	89.4	84.2	84.1	77.6	81.5	75.4	79.4	80.1	73.5	81.0	80.3	81.1
Dropout 0.1	76.4	82.1	81.7	81.0	88.0	70.8	80.3	74.4	79.0	72.3	75.8	75.9	70.6	78.6	76.5	77.6
Sequence length 512	78.1	83.4	83.1	82.1	88.8	84.5	82.8	77.3	81.2	75.4	78.2	79.6	73.8	80.0	78.9	80.5
Span length 10	77.6	81.5	80.5	81.2	87.2	83.0	81.2	74.7	79.8	73.6	76.7	75.9	71.3	78.6	76.5	78.6
α = 0.7	79.3	84.1	84.5	83.1	89.4	85.3	84.4	76.4	82.8	70.6	78.7	79.8	71.7	80.3	79.9	80.7
α = 0.2	78.7	83.8	83.3	82.5	89.3	83.4	83.6	77.3	81.2	75.4	78.6	79.4	73.9	79.9	79.7	80.7
No line length filter	78.4	83.3	81.5	81.4	88.9	83.8	82.5	74.4	80.5	69.4	77.6	76.9	71.3	78.8	78.3	79.1
Add Wikipedia data	79.3	83.1	83.1	82.7	88.6	80.1	83.2	77.3	81.4	75.0	78.9	79.3	73.5	80.2	79.2	80.3
Table 13: XNLI zero-shot accuracy of various ablations on our mT5-Large model.
Model	en	ar	de	el	es	hi	ru	th	tr	vi	zh	avg
Baseline (mT5-Large)	88.4 / 77.3	75.2 / 56.7	80.0 / 62.9	77.5 / 57.6	81.8 / 64.2	73.4 / 56.6	74.7 / 56.9	73.4 / 62.0	76.5 / 56.3	79.4 / 60.3	75.9 / 65.5	77.8 / 61.5
Span length 10	88.1 / 76.3	70.0 / 50.6	78.1 / 60.2	68.8 / 44.0	79.0 / 60.8	67.3 / 48.4	65.4 / 43.3	68.1 / 57.2	74.4 / 53.6	77.9 / 57.7	76.6 / 66.4	74.0 / 56.2
Dropout 0.1	87.3 / 76.0	54.9 / 33.9	77.6 / 60.2	64.4 / 40.1	79.2 / 60.6	59.1 / 40.4	59.5 / 38.4	65.7 / 51.0	73.6 / 52.8	75.8 / 55.8	77.0 / 64.5	70.4 / 52.1
Sequence length 512	88.0 / 76.9	77.0 / 59.6	80.2 / 62.4	79.8 / 60.0	81.7 / 64.4	75.1 / 57.5	77.4 / 58.5	72.7 / 59.8	75.3 / 53.9	79.4 / 58.9	78.5 / 67.2	78.6 / 61.7
α = 0.7	88.4 / 77.1	76.5 / 58.8	78.5 / 59.8	77.2 / 55.5	78.7 / 59.5	74.6 / 56.8	73.1 / 54.5	72.5 / 60.2	75.7 / 55.0	79.2 / 58.3	78.6 / 66.2	77.5 / 60.2
α = 0.2	87.9 / 76.8	75.5 / 57.3	80.2 / 62.4	76.2 / 54.0	81.6 / 63.7	73.7 / 57.0	70.7 / 50.8	72.2 / 60.4	75.5 / 55.7	79.7 / 59.7	78.3 / 67.5	77.4 / 60.5
No line length filter	88.9 / 77.4	73.8 / 54.0	80.8 / 62.7	74.2 / 51.8	80.9 / 62.8	74.1 / 56.6	75.0 / 56.4	71.7 / 60.3	76.7 / 56.0	78.8 / 58.6	78.5 / 67.1	77.6 / 60.3
Add Wikipedia data	89.3 / 78.4	69.6 / 48.9	79.6 / 61.1	59.5 / 36.0	80.6 / 61.0	73.6 / 55.0	68.7 / 47.0	70.5 / 58.1	76.7 / 56.9	78.6 / 56.4	77.5 / 66.3	74.9 / 56.8
Table 14: XQuAD zero-shot F1/EM of various ablations on our mT5-Large model.