SlowGuess/NAACL_2022_OCR huggingface dataset:
the average (bottom, excluding the English performance). This is here the case even when further fine-tuning a model on English samples results in a decreased English performance as seen for the 2nd model initialization. The least negative sets of samples still correspond to the best performing shots. The results generalize for varying sizes of few-shot sets e.g. 1000 samples in Appendix Figure 20.
5.2.2 Correlation between manual and machine translation performance
Another possible indicator of the best performing set of samples could be the performance of the samples in the set when they are machine translated to the target language. Artetxe et al. (2020a) has shown that subtle patterns in the (machine or manual) translated samples can have a notable impact on the model performance, so it is important to empirically study the relation between both. Similar to the above, we calculate the correlation between the performance for both manual and machine translation of the same set of samples for each target language. As seen for XNLI in Table 3, there is an even higher correlation than with the English performance. A somewhat lower correlation is seen for PAWS-X in Appendix Table 10. Lower correlation might be a result of lower-quality MT or a result of the different patterns introduced by MT as mentioned before.
5.2.3 Gain from choosing shots
We show in Table 4 the few-shot performance gain resulting from choosing the shots with the highest English performance and the highest MT
performance. Random samples are used for few-shot cross-lingual transfer in related work, so we compare to the average few-shot gain across the different shots in no choosing (avg), and also to the minimum in no choosing (min), because an important aspect of choosing shots is avoiding the worst-performing ones (Comparing to the average hides the fact that we might accidentally use a very bad set of shots). We can see a clear gain in most cases across all models when using en performance or mt performance. When there is no gain compared to no choosing (avg), the performance is still comparable and the benefit of not choosing the worst performing shots is still there as compared to no choosing (min). The few-shot gain with chosen-shots is most significant at smaller number of samples where the gain is almost double that from no choosing (avg).
Combining both En and MT performance when choosing the shots is expected to result in more gain, so we investigate feeding the performance values as features to a linear model that takes as input the performance of a set of samples and predicts the performance gain when this set is manually translated and used for few-shot. Predicting the performance gain is also helpful to avoid translating any set of samples if all are expected to result in a negative or low gain. We use the performance metrics as a dataset: collecting the performance of En/MT of random sets of samples along with the performance of the actual manual translation. This is done using 5 different random sets of samples for 5 different XLM- $\mathbf{R}_B$  initialization with varying
10	50	100	500(*)	1000
mBERT
XNLI	no choosing (avg)	0.22	0.51	0.92	1.35	2.21
no choosing (min)	-0.18	-0.00	0.17	0.11	0.99
en performance	0.59	0.84	1.06	1.37	2.43
mt performance	0.76	0.87	1.23	1.87	2.52
PAWS-X	no choosing (avg)	0.22	-0.02	-0.02	0.89	1.76
no choosing (min)	0.04	-0.24	-0.28	0.24	0.67
en performance	0.03	0.37	-0.02	1.21	1.89
mt performance	0.46	0.54	0.27	0.72	2.00
XQuAD	no choosing (avg)	0.64	1.68	2.28	3.55
no choosing (min)	-0.15	-0.35	-0.59	-0.59
en performance	0.80	1.58	2.34	3.52
XLM-RB
XNLI	no choosing (avg)	0.36	0.64	1.13	1.38	1.71
no choosing (min)	0.04	-0.15	0.10	0.36	0.65
en performance	0.71	1.15	1.32	1.82	1.90
mt performance	0.88	1.08	1.36	1.81	2.01
en + mt model	0.85	1.11	1.42	1.85	2.01
+ lang features	0.83	1.13	1.44	1.85	2.03
PAWS-X	no choosing (avg)	0.19	0.02	-0.20	0.44	0.76
no choosing (min)	-0.34	-0.43	-1.05	-0.23	0.10
en performance	0.17	0.10	0.23	0.53	0.71
mt performance	0.38	0.19	0.09	0.42	0.73
en + mt model	0.32	0.09	0.13	0.44	0.76
+ lang features	0.26	0.04	0.00	0.52	0.84
XQuAD	no choosing (avg)	0.04	0.06	-0.02	0.17
no choosing (min)	-0.36	-0.71	-0.76	-1.31
en performance	0.04	0.08	-0.15	0.17
XLM-RL
XNLI	no choosing (avg)	0.53	0.76	1.35	1.84	2.10
no choosing (min)	0.28	0.32	0.70	0.97	1.29
en performance	0.71	1.09	1.51	1.78	2.11
PAWS-X	no choosing (avg)	0.06	-0.10	-0.52	0.46	0.84
no choosing (min)	-0.56	-0.75	-0.78	0.02	0.04
en performance	-0.14	0.27	0.18	0.55	1.04
XQuAD	no choosing (avg)	0.05	0.06	-0.08	0.08
no choosing (min)	-0.20	-0.42	-0.97	-1.17
en performance	0.09	0.10	-0.01	0.06
(*) 250 for XQuAD
Table 4: Chosen-shots performance gain. Gain over zero-shot performance when choosing the best set of shots using a heuristic (en or mt performance) or a linear model that predicts the performance.
sample sizes across all languages (excluding English) resulting in 1750, 750, 1100 data points for XNLI, PAWS-X and XQuAD. For each language, we train the model using the data from all other languages and evaluate on the selected language. Cross-validation is done on the data after excluding the selected language to choose the best hyperparameters. The following features are considered as input: En and/or MT performance gain for the set of samples corresponding to each data point. In all cases, we consider: the zero-shot performance (since the gain is usually larger when the zero-shot performance is lower), and the number of samples used for that data point. We also investigate
whether adding language features<sup>4</sup> can improve the prediction in a way similar to the analysis by Lauscher et al. (2020). lang2vec<sup>5</sup> from Littell et al. (2017) is used to obtain the feature vectors for each language. The cosine similarity between the English vectors and the vectors for each language are added as 5 new scalar features (values are in Appendix Table 6). Those features can help the model better use the English performance depending on the similarity between the language and English. The prediction error of the linear models is reported in Appendix Table 13. We can see in Table 4 that using the models improves the chosen-shots performance gain for XNLI with the best result, as before, using a combination of all features. This is not the case for PAWS-X and could be partially due to having a smaller performance data and fewer languages to train on (7 as compared to 15 for XNLI).
The detailed results for the different languages are in the Appendix Figure 21. Choosing the shots improves the few-shot performance on XNLI for all languages across almost all sample sizes. For PAWS-X, there is mixed gain/loss but the improvement when using English performance at maximum size is concentrated in the European languages.
This work conducted a systematic comparison between translate-train and few-shot cross-lingual transfer. It quantified the performance gain for each and showed that starting from 1k samples, MT data could be used to improve over zero-shot performance, and that at  $10\mathrm{k} - 100\mathrm{k}$ , there is an advantage for translate-train over few-shot.
For the tasks that benefit from few-shot, we show that there are random sets of samples that perform better across languages and that the English performance of the samples in those sets can help us identify them. The performance of the MT of the samples can also be used as another indicator. When not incurring gain, both help at least avoid the worst performing samples.
Further analysis in the future could help identify why some datasets do not benefit from few-shot transfer with certain models, and analysing the samples might lead to uncovering interesting properties in the best/worst performing sets of samples.
4Using syntax, phonology, inventory, family and geographical location as features
$^{5}$ https://github.com/antonisa/lg2vec
We acknowledge funding by the Bundesministerium fur Bildung und Forschung (BMBF) through the project E-DELIB.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1994. Active learning with statistical models. In NIPS.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020. Don't use English dev: On the zero-shot cross-lingual evaluation of contextual embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 549-554, Online. Association for Computational Linguistics.
Anders Krogh and Jesper Vedelsby. 1994. Neural network ensembles, cross validation, and active learning. In NIPS.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In NeurIPS.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Peter Rousseeuw. 1984. Least median of squares regression. Journal of the American Statistical Association, 79:871-880.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215-10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Computational Linguistics.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. 2019. Selfie: Refurbishing unclean samples for robust deep learning. In ICML.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulic, Roi Reichart, Anna Korhonen, and Hinrich Schütze. 2021. A closer look at few-shot crosslingual transfer: The choice of shots matters. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5751-5767, Online. Association for Computational Linguistics.
Software: We use the Huggingface Transformers  $^{6}$  for fine-tuning the pretrained language models. We use scikit-learn  $^{7}$  to train the performance prediction models. Our code is made publicly available  $^{8}$ .
Hardware: NVIDIA GeForce GTX 1080 Ti with 11G memory is used for most experiments. The linear model is trained locally on a CPU.
Model: mBERT (base, cased) has 12-layer, 768-hidden, 12-heads, 110M parameters. Trained on cased text in the top 104 languages with the largest Wikipedias. XLM- $\mathbf{R}_B$  (base) has  $\sim 270\mathrm{M}$  parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 8-heads, and trained on on 2.5 TB of newly created clean CommonCrawl data in 100 languages. XLM- $\mathbf{R}_L$  (large)  $\sim 355\mathrm{M}$  parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads, $^9$ .
Hyperparameters: For the two classification tasks, we use a maximum sequence length of 128. We limit hyperparameter tuning to a search for the learning rate in  $\{7e - 6, 1e - 5, 3e - 5\}$  and use a batch size of 32. For Question Answering, we use a maximum sequence length of 384 with a paragraph slide of 128. We train using a learning rate of  $3e - 5$  and a batch size of 12 for 2 epochs. The used learning rate for XLM- $\mathbf{R}_B$  along with the dev performance for a model with seed=42 is reported in Table 5. We use four other models fine-tuned on the English train split with seed  $\in \{2, 4, 8, 16\}$
XNLI	PAWS-X	XQuAD
1e-5	7e-6	3e-5
84.82	92.45	89.10
Accuracy	Accuracy	F1
Training & Evaluation Runs: Starting from each of the 5 source fine-tuned models, we fine-tune on the target language for 5 different sets of samples. This is repeated for each size resulting in 25 runs per size. The runtime for the target language fine-tuning varies based on the number of samples used and the number of languages in each dataset. For smaller sample sizes, most runtime is spent for the evaluation on the large test set.
Table 5: learning rate and English dev performance
code	language properties(1)	cosine similarity to English(2)	XNLI	PAWS-X	XQuAD
name	size(3)	script	language family	syntax	phonology	inventory	family	geo
ar	Arabic	1.02	Arabic	Afro-Asiatic	0.65	0.70	0.71	0.00	0.97	x	x
vi	Vietnamese	1.24	Latin	Austro-Asiatic	0.66	0.78	0.75	0.00	0.85	x	x
de	German	2.37	Latin	IE: Germanic	0.90	0.81	0.76	0.54	1.00	x	x	x
en	English	5.98	Latin	IE: Germanic	1.00	1.00	1.00	1.00	1.00	x	x	x
el	Greek	0.17	Greek	IE: Greek	0.78	0.95	0.65	0.15	0.99	x	x
hi	Hindi	0.13	Devanagari	IE: Indo-Aryan	0.62	0.78	0.71	0.13	0.91	x	x
ur	Urdu	0.15	Perso-Arabic	IE: Indo-Aryan	0.62	0.86	0.72	0.13	0.93	x
es	Spanish	1.56	Latin	IE: Romance	0.82	0.86	0.64	0.10	1.00	x	x	x
fr	French	2.16	Latin	IE: Romance	0.81	0.75	0.74	0.10	1.00	x	x
ru	Russian	1.58	Cyrillic	IE: Slavic	0.81	0.86	0.65	0.17	0.96	x	x
bg	Bulgarian	0.26	Cyrillic	IE: Slavic	0.86	0.86	0.68	0.14	0.99	x
ja	Japanese	1.18	Ideograms	Japonic	0.50	0.67	0.65	0.00	0.86	x
ko	Korean	0.47	Hangul	Koreanic	0.55	0.75	0.71	0.00	0.87	x
th	Thai	0.13	Brahmic	Kra-Dai	0.64	0.78	0.75	0.00	0.85	x	x
sw	Swahili	0.05	Latin	Niger-Congo	0.46	0.91	0.76	0.00	0.92	x
zh	Mandarin	1.09	Chinese ideograms	Sino-Tibetan	0.71	0.73	0.70	0.00	0.88	x	x	x
tr	Turkish	0.34	Latin	Turkic	0.51	0.82	0.67	0.00	0.98	x	x
(1) properties taken from XTREME
(2) similarity calculated using lang2vec
(3) size is the #wikipedia articles in millions
Table 6: Languages in the Datasets
<sup>6</sup>https://github.com/huggingface/transformers
<sup>7</sup>https://github.com/scikit-learn/scikit-learn
<sup>8</sup>https://github.com/imanjundi/cross-lingual-transfer
from https://huggingface.co/transformers/pretrained_models.html
(a) Average Accuracy on XNLI across languages
(b) Average Accuracy on PAWS-X across languages
(c) Average F1 on XQuAD across languages
(c) Average F1 on XQuAD across languages
Figure 7: Average performance across languages for translate-train vs. few-shot using mBERT. The biggest performance boost comes from using translate-train
Figure 8: Average performance across languages for translate-train vs. few-shot using  $\mathbf{XLM - R}_L$ . The biggest performance boost comes from using translate-train
Figure 9: Detailed Results on XNLI using mBERT. Gains in performance over zero-shot for few-shot and translate-train. Low-resource languages like Swahili have the most gains in both cases
Figure 10: Detailed Results on XNLI using  $\mathbf{XLM - R}_L$ . Gains in performance over zero-shot for few-shot and translate-train. Low-resource languages like Swahili have the most gains in both cases
de-es-fr-ja-ko-zh-avg-	84.86±1.27	0.23±1.17	0.12±1.54	1.97±0.98	0.20±1.07	0.36±0.85	0.47±0.88	1.82±0.98	2.70±0.66
87.87±1.31	0.04±1.06	-0.04±1.18	0.71±0.96	-0.06±1.27	-0.28±1.22	0.26±0.86	1.53±0.64	2.62±0.52
87.03±0.64	0.07±0.63	-0.12±0.90	0.67±0.54	-0.03±0.57	-0.04±0.60	0.39±0.35	1.40±0.38	2.78±0.56
74.23±2.36	0.14±1.54	-0.28±2.15	2.76±1.28	-0.14±1.58	0.20±1.62	1.91±0.90	5.02±0.73	7.42±1.07
74.51±2.31	0.75±1.43	0.15±2.28	2.28±1.43	0.55±1.80	0.20±1.56	1.04±1.40	3.73±1.29	6.30±1.26
78.96±2.22	0.08±1.37	0.05±1.71	2.18±1.25	0.25±1.25	0.34±1.11	1.75±0.94	3.53±0.59	5.18±0.48
81.24	0.22	-0.02	1.76	0.13	0.13	0.97	2.84	4.50
0	10	100	1k	10	100	1k	10k	50k
Figure 11: Detailed Results on PAWS-X using mBERT. Gains in performance over zero-shot for few-shot and translate-train. Non-European language show the most gain especially Japanese and Korean.
de -es -fr -ja -ko -zh -avg	86.75±0.95	-0.34±0.97	-0.42±1.18	0.17±0.74	-0.29±1.00	-0.33±1.02	-0.52±1.12	0.46±1.00	1.29±0.82
87.94±0.65	0.02±0.53	-0.52±1.18	0.24±0.55	-0.82±0.97	-0.91±1.17	-0.01±0.64	0.77±0.70	1.77±0.38
88.74±0.85	-0.16±0.73	-0.18±0.86	0.10±0.65	-0.07±0.58	-0.59±0.93	0.11±0.57	0.68±0.56	1.58±0.57
75.91±0.59	0.07±0.56	-1.05±1.60	0.63±0.87	0.10±0.51	0.10±0.85	0.34±0.85	1.96±0.68	3.31±0.80
73.95±1.32	1.02±0.93	0.85±0.77	1.92±0.88	-0.18±1.75	0.81±0.94	0.96±1.03	4.05±0.78	6.43±1.07
79.16±1.43	0.52±0.66	0.11±1.13	1.49±0.63	0.20±1.43	0.07±1.08	1.14±1.00	2.65±0.57	3.54±0.59
82.07	0.19	-0.20	0.76	-0.18	-0.14	0.34	1.76	2.99
0	10	100	1k	10	100	1k	10k	50k
Figure 12: Detailed Results on PAWS-X using XLM-  $\mathbf{R}_B$ . Gains in performance over zero-shot for few-shot and translate-train. Non-European language show the most gain especially Korean.
de -es -fr -ja -ko -zh -avg -	90.26±0.25	-0.56±0.86	-0.78±0.87	0.37±0.54	-0.21±0.59	-0.48±0.60	-0.27±0.45	0.46±0.35	1.25±0.17
90.47±0.47	-0.05±0.88	-0.75±1.32	0.46±0.48	-0.56±1.15	-0.33±1.30	0.17±0.57	1.06±0.34	1.69±0.47
90.76±0.53	-0.31±0.99	-0.73±0.91	0.04±0.51	-0.23±0.76	-0.06±0.62	0.07±0.45	0.93±0.42	1.65±0.39
80.36±0.76	0.41±0.42	-0.72±1.56	1.02±0.58	0.23±1.00	-0.03±1.29	0.55±0.53	2.33±0.57	4.00±0.29
80.31±1.55	0.70±1.57	0.34±1.77	2.11±0.64	0.34±1.65	0.79±1.11	1.50±0.77	3.36±0.36	4.94±0.24
83.21±0.82	0.16±1.01	-0.46±1.67	1.03±0.88	0.06±0.98	0.03±0.85	0.79±0.64	2.06±0.38	3.46±0.35
85.89	0.06	-0.52	0.84	-0.06	-0.01	0.47	1.70	2.83
0	10	100	1k	10	100	1k	10k	50k
Figure 13: Detailed Results on PAWS-X using XLM-  $\mathbf{R}_L$ . Gains in performance over zero-shot for few-shot and translate-train. Non-European language show the most gain especially Japanese and Korean.
Figure 14: Detailed Results on XQuAD using mBERT. Gains in performance over zero-shot for few-shot and translate-train. Non-European languages show the most gain especially Chinese. Thai shows a significant degrade when using the full machine-translated dataset. This might be due to lower-quality machine translation for Thai.
Figure 15: Detailed Results on XQuAD using XLM-  $\mathbf{R}_B$ . Gains in performance over zero-shot for few-shot and translate-train. Non-European languages show the most gain especially Chinese. Thai shows a significant degrade when using the full machine-translated dataset. This might be due to lower-quality machine translation for Thai.
Figure 16: Detailed Results on XQuAD using XLM-  $\mathbf{R}_L$ . Gains in performance over zero-shot for few-shot and translate-train. Non-European languages show the most gain especially Chinese. Thai shows a significant degrade when using the full machine-translated dataset. This might be due to lower-quality machine translation for Thai.
Figure 17: Detailed Results on XNLI using a part of the available data as dev. The few-shot performance only changes slightly with minor increases and decreases for across the languages. The highest increase on average is at 10 samples with an increase of  $0.05\%$ . Translate-train performance decreases for almost all languages and on average.
Figure 18: Detailed Results on PAWS-X using a part of the available data as dev. The few-shot performance shows mixed gains decreasing by  $\sim 0.60\%$  for 10 samples, increasing by  $\sim 0.40\%$  at 100 then decreasing again by  $\sim 0.10\%$ . Translate-train performance decreases until the full dataset is used where it increases by  $\sim 1\%$ .
Figure 19: Performance variance on different shots
(a) PAWS-X Performance variance on different shots. (b) XQuAD Performance variance on different shots.
Variance increases with an increased data size
lang	ar	bg	de	el	es	fr	hi	ru	sw	th	tr	ur	vi	zh	avg
10	0.64	0.42	0.42	0.51	0.63	0.75	0.49	-0.25	0.63	0.48	0.70	0.19	0.48	0.42	0.50
50	0.82	0.59	0.59	0.76	0.74	0.87	0.66	0.11	0.70	0.81	0.84	0.54	0.73	0.56	0.69
100	0.76	0.53	0.47	0.46	0.64	0.77	0.69	-0.50	0.58	0.58	0.64	0.06	0.73	0.33	0.52
500	0.84	0.64	0.77	0.79	0.73	0.81	0.77	0.18	0.67	0.71	0.88	0.54	0.77	0.73	0.72
1000	0.72	0.63	0.74	0.69	0.72	0.84	0.60	0.10	0.06	0.51	0.80	0.03	0.51	0.75	0.58
all	0.77	0.59	0.62	0.69	0.73	0.79	0.66	0.15	0.62	0.57	0.79	0.38	0.65	0.55	0.64