SlowGuess/NAACL_2022_OCR huggingface dataset:
Haoran Xu and Philipp Koehn. 2021a. Cross-lingual bert contextual embedding space mapping with isotropic and isometric conditions. arXiv preprint arXiv:2107.09186.
Haoran Xu and Philipp Koehn. 2021b. Zero-shot crosslingual dependency parsing through contextual embedding transformation. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 204-213, Kyiv, Ukraine. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. 2021. Improving multilingual translation by representation and gradient regularization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7266-7279, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Hao ran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, et al. 2021. Everything is all it takes: A multipronged strategy for zero-shot cross-lingual information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1950-1967.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33.
Yu Zhang and Qiang Yang. 2018. An overview of multitask learning. National Science Review, 5:30-43.
Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulic, Roi Reichart, Anna Korhonen, and Hinrich Schütze. 2021. A closer look at few-shot crosslingual transfer: The choice of shots matters. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5751-5767, Online. Association for Computational Linguistics.
A Size of Dev Sets
In Figure 4, we show the size of dev sets that we used in our experiments, which are also the dev sets used by Zhao et al. (2021). Data of all tasks are downloaded from the XTREME-R benchmark (Hu et al., 2020; Ruder et al., 2021), where train/dev/test sets are already split. We can notice that the dev size of all languages in all tasks are tremendously higher than the largest number (10) of shots we pick in few-shot cross-lingual learning. However, in reality, if we only have access to a few training instance, we usually do not have a such large dev set. For tasks such as NER, POS and XNLI, we sample shots from the target training sets and directly use their supported dev sets. For TyDiQA which only supports train and dev sets in XTREME-R, we sample shots from the target training sets but use the remaining training data as dev sets, and we use dev sets for test.
B Language Statistics
In this paper, we cover a total of 49 languages in our whole experiments, including NER, POS, TyDiQA, and XNLI tasks. The list of full names of languages is shown in Table 3, with their ISO 639-1 code, script, and language families. We checkmark under the column of the task in the Table if the language is involved in the task.
C Full Results
The full results of NER, POS, TyDiQA and XNLI are shown in Table 4, Table 5, Table 6 and Table 7, respectively. In each task, we report F1 scores (or accuracy) of all covered languages in 1-,5-, or 10 shot learning by using all introduced methods. Best score among methods in each language is bold.
D Languages Benefits Least
In Table 8, we show the list of top-5 language which benefits least by using gradient-mix-train in 5-shot learning. In NER and XNLI, we can notice a reverse phenomenon in the top-5 languages which benefit most — most of the languages are linguistically closer to English, at least using the same (Latin) script. In NER and TyDiQA tasks, although the gap left by gradient-mix-train is much smaller than top-5 languages which benefits most, the improvements are still significant.
E Mixed Training with Subsets of Languages
Here, we show the numeric results of Figure 2a and Figure 2b in Table 9 and Table 10, respectively.
F Our Methods with Dev Sets
We take  $ar$  in the NER task as an example to show that gradient-mix-train can be further improved by utilizing large dev sets (around 10K). Figure 5 shows F1 scores of gradient-mix-train and ord-FS both with and without dev sets with increasing epoch numbers. Methods with the help of the dev set start showing its effectiveness in model selection when it comes to large enough epoch numbers. Importantly, the gap led by the dev set in gradient-mix-train is smaller than the one in ord-FS, which shows that gradient-mix-train is able to select approximately optimal model even without target dev sets by using the source dev set. It is also worth mentioning that gradient-mix-train even significantly outperforms the best performance of ord-FS with only 2 epoch of source (and target) data training. Still, ord-FS starts training based on 10-epoch source-trained model.
Language	ISO 639-1 code	Script	Language Family	NER	POS	TyDiQA	XNLI
Afrikaans	af	Latin	IE:Germanic	✓	✓	✓
Arabic	ar	Arabic	Afro-Asiatic	✓	✓	✓
Azerbaijani	az	Latin	Turkic	✓
Bulgarian	bg	Cyrillic	IE:Slavic	✓	✓	✓
Bengali	bn	Brahmic	IE:Germanic	✓	✓	✓
Greek	el	Greek	IE:Greek	✓	✓	✓
English	en	Latin	IE:Germanic	✓	✓	✓	✓
Spanish	es	Latin	IE:Romance	✓	✓	✓
Estonian	et	Latin	Uralic	✓	✓
Basque	eu	Latin	Basque	✓	✓
Persian	fa	Perso-Arabic	IE:Iranian	✓	✓
Finnish	fi	Latin	Uralic	✓	✓	✓
French	fr	Latin	IE:Romance	✓	✓	✓
Gujarati	gu	Brahmic	IE:Indo-Aryan	✓
Hebrew	he	Jewish	Afro-Asiatic	✓	✓
Hindi	hi	Devanagari	IE:Indo-Aryan	✓	✓	✓
Hungarian	hu	Latin	Uralic	✓	✓
Indonesian	id	Latin	Austronesian	✓	✓	✓
Italian	it	Latin	IE:Romance	✓	✓
Japanese	ja	Ideograms	Japonic	✓	✓
Javanese	jv	Brahmic	Austronesian	✓
Georgian	ka	Georgian	Kartvelian	✓
Kazakh	kk	Arabic	Turkic	✓	✓
Korean	ko	Hangul	Koreanic	✓	✓	✓
Lithuanian	lt	Latin	IE:Baltic	✓	✓
Malayalam	ml	Brahmic	Dravidian	✓
Marathi	mr	Devanagari	IE:Indo-Aryan	✓	✓
Malay	ms	Latin	Austronesian	✓
Burmese	my	Brahmic	Sino-Tibetan	✓
Dutch	nl	Latin	IE:Germanic	✓	✓
Punjabi	pa	Brahmic	IE:Indo-Aryan	✓
Polish	pl	Latin	IE:Slavic	✓	✓
Portuguese	pt	Latin	IE:Romance	✓	✓
CuscoQuechua	qu	Latin	Quechuan	✓
Romanian	ro	Latin	IE:Romance	✓	✓
Russian	ru	Cyrillic	IE:Slavic	✓	✓	✓	✓
Swahili	sw	Latin	Niger-Congo	✓	✓	✓
Tamil	ta	Brahmic	Dravidian	✓	✓
Telugu	te	Brahmic	Dravidian	✓	✓	✓
Thai	th	Brahmic	Kra-Dai	✓	✓
Tagalog	tl	Brahmic	Austronesian	✓
Turkish	tr	Latin	Turkic	✓	✓	✓
Ukrainian	uk	Cyrillic	IE:Slavic	✓	✓
Urdu	ur	Perso-Arabic	IE:Indo-Aryan	✓	✓	✓
Vietnamese	vi	Latin	Austro-Asiatic	✓	✓	✓
Wolof	wo	Latin	Niger-Congo	✓
Yoruba	yo	Arabic	Niger-Congo	✓
Mandarin	zh	Chinese	ideograms	✓	✓	✓
Table 3: Statistics about languages considered in this paper, including the scripts and language family of every language. A language used in a task is checkmarked under the column of the task.
Table 5: Full results (F1) of the POS task.
K	Methods	ar	bn	fi	id	ko	ru	sw	te	en	Avg.
K=0	Zero-Shot	62.53	42.24	61.82	70.62	42.99	57.75	56.40	43.23	65.51	55.80
K=1	ord-FS+dev	62.20	45.92	59.33	71.15	38.70	58.70	53.63	47.12	65.51	55.81
ord-FS	48.53	33.91	54.56	63.36	40.43	49.59	47.56	23.49	65.51	47.44
mix-FT	50.93	36.83	54.04	61.72	38.46	51.35	46.26	33.48	65.51	48.73
naive-mix-train	62.46	42.52	62.32	72.32	43.44	58.28	54.87	49.86	67.17	57.03
gradient-mix-train	62.60	45.07	62.88	72.43	46.05	58.82	55.47	47.64	67.81	57.64
K=5	ord-FS+dev	58.59	46.68	59.23	69.87	41.19	59.33	54.47	45.50	65.51	55.60
ord-FS	54.07	36.85	54.82	65.39	40.91	53.46	47.61	42.04	65.51	51.19
mix-FT	58.67	43.59	57.46	67.09	44.04	54.65	56.17	43.23	65.51	54.49
naive-mix-train	62.42	47.51	61.64	72.39	46.06	59.16	57.62	53.96	66.58	58.59
gradient-mix-train	63.52	49.11	63.87	73.29	46.17	59.09	58.20	53.19	67.58	59.34
K=10	ord-FS+dev	61.78	44.67	59.32	69.96	41.29	59.23	52.73	45.79	65.51	55.59
ord-FS	59.46	43.21	56.21	65.88	40.67	52.64	53.45	41.61	65.51	53.17
mix-FT	60.51	44.64	58.42	67.23	44.99	56.39	58.12	44.09	65.51	55.54
naive-mix-train	64.87	48.02	62.12	72.63	47.91	60.43	60.44	46.18	67.32	58.88
gradient-mix-train	64.17	47.46	63.37	72.77	47.26	60.48	60.13	52.73	66.85	59.47
Table 6: Full results (F1) of the TyDiQA task.
K	Methods	ar	bg	de	el	es	fr	hi	ru	sw	th	tr	ur	vi	zh	en	Avg.
K=0	Zero-Shot	72.28	77.15	75.97	74.71	78.56	77.19	69.10	73.95	62.08	71.52	72.32	65.39	74.15	73.67	85.19	73.55
K=1	ord-FS+dev	72.08	77.49	76.19	75.47	79.21	77.99	69.16	74.69	62.29	72.31	72.46	66.00	74.60	74.20	85.19	73.95
ord-FS	71.60	77.43	76.09	75.29	78.95	77.64	69.42	74.47	61.46	72.27	72.10	65.55	74.72	74.20	85.19	73.70
mix-FT	71.56	77.28	75.88	74.81	78.42	77.33	69.34	74.39	61.58	71.64	72.08	65.40	74.33	73.84	85.19	73.54
naive-mix-train	71.52	76.83	75.89	74.74	77.88	77.25	68.99	74.67	62.80	71.07	71.77	65.17	73.74	72.23	83.84	73.29
gradient-mix-train	71.97	77.76	76.12	75.27	78.47	77.74	70.06	75.47	64.08	72.49	72.10	66.25	74.90	74.48	84.20	74.09
K=5	ord-FS+dev	71.98	77.72	76.55	75.48	78.69	77.48	70.16	74.76	62.29	72.68	72.30	65.82	75.31	74.77	85.19	74.08
ord-FS	71.57	77.25	76.18	75.39	78.64	77.03	69.94	74.44	61.59	72.33	71.92	65.21	74.93	74.37	85.19	73.73
mix-FT	70.85	76.37	75.23	74.20	77.41	76.79	69.09	74.19	61.89	71.27	71.13	65.28	74.00	74.05	85.19	73.13
naive-mix-train	72.02	77.43	76.12	74.74	78.19	77.41	69.63	74.67	62.95	72.02	72.22	65.86	74.33	73.86	83.90	73.69
gradient-mix-train	72.05	77.89	76.54	75.48	78.83	77.94	70.64	76.12	63.94	72.83	72.40	67.15	75.31	75.15	83.90	74.41
K=10	ord-FS+dev	71.74	77.51	76.73	75.33	79.03	77.69	70.11	75.09	62.46	72.92	72.76	66.00	75.28	75.03	85.19	74.19
ord-FS	71.31	77.65	76.38	74.83	79.20	77.43	70.12	75.20	62.43	72.77	72.72	65.58	75.14	74.84	85.19	74.03
mix-FT	71.32	76.77	75.80	74.58	77.77	77.11	69.72	74.73	62.15	72.36	71.78	65.96	74.44	74.67	85.19	73.62
naive-mix-train	72.22	77.67	76.47	75.39	78.29	77.52	70.53	75.57	63.05	72.51	72.35	66.76	74.69	74.65	84.23	74.23
gradient-mix-train	71.74	78.04	76.61	75.29	78.89	77.79	70.95	75.90	63.74	73.15	72.41	67.07	75.48	75.43	84.10	74.44
Table 7: Full results (accuracy) of the XNLI task.
NER	POS	TyDiQA	XNLI
lang.	Δ F1 (%)	lang.	Δ F1 (%)	lang.	Δ F1 (%)	lang.	Δ Acc. (%)
pl	0.90	vi	-4.35	ko	5.25	es	-0.32
ms	1.51	ja	-1.85	ru	5.62	tr	-0.31
nl	2.66	he	-1.36	id	7.90	de	0.23
af	2.73	hu	-1.02	fi	9.05	vi	0.34
sw	2.76	zh	-0.71	ar	9.45	fr	0.36
Table 8: Top-5 languages that achieve the least improvement by using gradient-mix-train compared to ord-FS on all tasks in 5-shot learning.
ar	hi	my	pa	hu	nl	fr	tr
ord-FS	57.69	64.08	52.44	52.28	79.91	82.11	76.99	79.97
gradient-single	71.98	72.82	57.67	63.60	80.28	83.89	81.51	80.10
gradient-subset	70.57	71.79	61.18	62.99	81.65	83.96	81.90	82.37
gradient-all	71.83	76.76	63.78	69.88	83.06	84.77	83.64	85.21
Table 9: Numeric results of Figure 2a.
Figure 4: The size of dev sets that we use in the experiments for each language in each task.
ar	fi	id	sw	te
ord-FS	54.07	54.82	65.39	47.61	42.04
gradient-single	64.43	62.52	72.78	56.91	54.28
gradient-subset	64.04	63.17	72.44	57.48	54.49
gradient-all	63.52	63.87	73.29	58.20	53.19
Table 10: Numeric results of Figure 2b.
Figure 5: F1 scores of gradient-mix-train(+dev) and ord-FS(+dev) with increasing number of epochs. The large dev set helps model selection after certain epochs. Gradient-mix-train shows less gap led by the dev set than ord-FS and can select approximately optimal model by only using the source dev set.