SlowGuess/NAACL_2022_OCR huggingface dataset:
Iman Jundi and Gabriella Lapesa  
Institute for Natural Language Processing  
University of Stuttgart  
first.last@ims.uni-stuttgart.de
Translate-train or few-shot cross-lingual transfer can be used to improve the zero-shot performance of multilingual pretrained language models. Few-shot utilizes high-quality low-quality samples (often manually translated from the English corpus). Translate-train employs a machine translation of the English corpus, resulting in samples with lower quality that could be scaled to high quantity. Given the lower cost and higher availability of machine translation compared to manual professional translation, it is important to systematically compare few-shot and translate-train, understand when each has an advantage, and investigate how to choose the shots to translate in order to increase the few-shot gain. This work aims to fill this gap: we compare and quantify the performance gain of few-shot vs. translate-train using three different base models and a varying number of samples for three tasks/datasets (XNLI, PAWS-X, XQuAD) spanning 17 languages. We show that scaling up the training data using machine translation gives a larger gain compared to using the small-scale (higher-quality) few-shot data. When few-shot is beneficial, we show that there are random sets of samples that perform better across languages and that the performance on English and on the machine-translation of the samples can both be used to choose the shots to manually translate for an increased few-shot gain. $^{1}$
With the emergence of large-scale multilingual Pre-trained Language Models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), a significant amount of research went into exploring the cross-lingual transfer capabilities of these models, allowing for an easier adaptation to a task in many various languages. This is achieved through a number of approaches.
Zero-shot cross-lingual transfer has become a research focus, e.g. XTREM / XTREM-R benchmark (Hu et al., 2020; Ruder et al., 2021). In this approach, transfer to new languages is done by fine-tuning a multilingual PLM on the task at issue, using only an English corpus (source language) and reporting the performance on multiple target languages.
Few-shot cross-lingual transfer was recently shown to give an advantage over zero-shot cross-lingual transfer (Lauscher et al., 2020). In this approach, it is shown that fine-tuning the model using a small amount of target-language task data (few-shot) improves the performance, especially for low-resource languages.
Translate-train is another common approach to improve the performance. Here the full training dataset is machine translated to the target language and used for fine-tuning. There exists relatively good Machine Translation (MT) systems for the languages that are usually studied in the few-shot approach² that could be used in translate-train.
In the following, we use few-shot to refer to fine-tuning using fewer samples of high-quality professional manual translation. Translate-train is used to refer to fine-tuning using lower-quality machine translation that has the potential to be scaled to a larger number of samples. Although some research has dealt with few-shot cross-lingual transfer and analyzing it (Lauscher et al., 2020; Zhao et al., 2021), no systematic study was done to compare it to translate-train. Given that both zero-shot and few-shot cross-lingual transfer assume the availability of a large-scale English corpus of the task for source training, we hypothesize that the translate-train approach might have an advantage over few-shot given the scale of data that would be available even if not at the best quality.
2All target languages in the studied datasets are supported by e.g. Google Translate: https://cloud.google.com/translate/docs/languages
$^{1}$ Code available under: https://github.com/imanjundi/cross-lingual-transfer
Findings of the Association for Computational Linguistics: NAACL 2022, pages 129 - 150 July 10-15, 2022 ©2022 Association for Computational Linguistics
On the other hand, when there is a need for few-shot cross-lingual transfer for some task and therefore a need for professional translation of some training samples, this entails significantly more effort and cost compared to MT. It is then important to find out which samples to manually translate given the high variance in performance depending on the choice of samples as shown in (Zhao et al., 2021).
We investigate both those research directions using 3 base models (mBERT<sub>base</sub>, XLM-R<sub>base</sub>, XLM-R<sub>large</sub>) on 3 high-level semantic tasks and datasets: XNLI (Natural Language Inference), X-PAWS (Paraphrase Detection) and XQUAD (Question Answering), spanning 17 diverse languages. We investigate the following research questions:
Q1. How does the performance of few-shot crosslingual transfer compare to that of translate-train? We show that there is a performance advantage for few-shot transfer over translate-train given the same number of samples, but that with the increase of samples used for translate-train, this gap shrinks, and using the full large-scale corpus in translate-train results in a clear advantage over few-shot. We show that at a scale of  $10\mathrm{x} - 100\mathrm{x}$  of machine-translation to manual-translation, quantity trumps quality and it is recommended in this case to use translate-train if MT is available for the language. Few-shot transfer still has an advantage when less source data is available and it is therefore not possible to benefit from the scale gain of using MT.
Q2. Are there sets of samples that have better few-shot performance if translated and how can those sets be identified?
We show that when few-shot transfer is beneficial for the task, there are random sets of samples that perform better across most target languages and across different model initializations. We investigate using the performance on the English version of the samples and the machine-translated version to choose the best candidates to manually translate and use for few-shot transfer. We show that there is a correlation between the performance of the same set of shots across languages and that the few-shot samples that perform better on the source language, English, perform also better across languages. A similar observation is made also using MT of the samples. We further show empirically that choosing the sets of samples for few-shot transfer using those heuristics or a model, using such features of the samples, results in more bang for your shots.
Cross-lingual transfer: The cross-lingual transfer capabilities of multilingual pretrained language models have led to major recent advances and a growing number of such models have been introduced, e.g., mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), mT5 (Xue et al., 2021) etc. The cross-lingual transfer is usually exploited in a zero-shot setup, and benchmarks are built based on this assumption e.g. XTREME/XTREME-R (Hu et al., 2020; Ruder et al., 2021).
Few-shot: There has been recently some focus on few-shot cross-lingual transfer and its analysis. Lauscher et al. (2020) shows the effectiveness of few-shot compared to zero-shot cross-lingual transfer especially in lower-resource and distant languages, where zero-shot is least effective and few-shot gives a large gain. Zhao et al. (2021) analyzes few-shot cross-lingual transfer emphasizing that the choice of shots has a significant effect on the performance. The experiments are conducted at a small scale of around 10 samples. Compared to this, we conduct larger-scale few-shot experiments with a size up to hundreds of samples and focus on choosing the best-performing samples.
Translate-train: is commonly used to boost the performance for a target language using a machine translation of the source corpus (Conneau et al., 2018; Lample and Conneau, 2019; Conneau et al., 2020; Hu et al., 2020). Xue et al. (2021) shows that, similar to zero-shot, translate-train performance increases with the scale of the model. No systematic study tested the effect of the scale of the translated data in comparison with few-shot to understand the interplay of data quality vs. quantity in this context.
Choosing samples: Two related areas are sample selection (Rousseeuw, 1984) which is used for robust training on noisy data (Song et al., 2019) and active learning (Cohn et al., 1994; Krogh and Vedelsby, 1994) used to choose the best potential samples to annotate (Siddhant and Lipton, 2018). Both assume access to the actual sample input (with or without label). On the other hand, this work investigates choosing samples while only having access to the source-language sample input/output.
We focus on high-level tasks and conduct our experiments on 2 classification tasks and a question answering task (Table 1) from the XTREME benchmark. The details and properties of the languages
Dataset	|Train|	|mDev|	|mTest|	|Langs|	metric
XNLI	392,702	2,490	5,010	15	Acc
PAWS-X	49,401	2,000	2,000	7	Acc
XQuAD	87,599	261	1,190-261=930	11	F1
Table 1: Datasets statistics. Train is the English training dataset. |mDev| and |mTest| are used to indicate the size of the multilingual split of the dataset.
can be found in Appendix Table 6. When attempting to choose the shots, we rely on measuring the performance of the same set of samples across different languages, so we are limited to datasets with parallel corpus, i.e. the target language corpus is created by translating the English corpus as opposed to collecting and annotating the target language corpus from scratch: XNLI (Conneau et al., 2018) is a professional translation of the dev and test set of the MultiNLI dataset (Williams et al., 2018) into 14 languages. The dataset consists of pairs of sentences, a premise and a hypothesis, where the task is to predict whether the premise entails, contradicts, or is neutral to the hypothesis. PAWS-X (Yang et al., 2019) dataset is a professional translation of the dev and test set of the PAWS dataset (Zhang et al., 2019) into 6 languages. The dataset consists of pairs of sentences and the task is to predict whether those two sentences are paraphrases of each other. XQuAD (Artetxe et al., 2020b) is a professional translation of the dev set from SQuAD v1.1 (Rajpurkar et al., 2016) into 10 other languages. The dataset consists of a paragraph and a set of questions. The task is to select the span of the paragraph that answers the question. 10 paragraphs from the multilingual split are reserved, similar to Lauscher et al. (2020), as dev (total of 261 question/answer samples) and the rest as test set.
Three main models are used: mBERT (base), XLM- $\mathbf{R}_B$  (base) and XLM- $\mathbf{R}_L$  (large). We report results on XLM- $\mathbf{R}_B$  if not specified otherwise, because it strikes a balance between good performance and efficient training. For each task, we fine-tune the model on the source language (English) corpus for 5 epochs with early stopping using the loss on the English dev set. We then continue fine-tuning the model on the target language either in a few-shot or translate-train setup as explained in the following sections. Training details are in Appendix A.
4.1 Few-shot experiments
We use samples from the multilingual dev set as training samples. Few-shot fine-tuning is done as follows: for each language, we separately continue fine-tuning the source model for one epoch on  $n \in \{10, 50, 100, 500, 1k\}$  samples from the target language corpus for the two classification tasks and for  $n \in \{10, 50, 100, 250\}$  for the Question Answering task, given the smaller amount of data available for training in this case. We report the results on the test set for each target language. For each  $n$  number of samples, the performance is averaged across 5 different sets of random samples using 5 different fine-tuned models with different random initializations, 25 runs in total. This is to ensure more robust results when measuring the gain over zero-shot given the high variance across different sets of samples (Zhao et al., 2021) as well as the variance in zero-shot performance across all random initializations (Keung et al., 2020). For comparing the performance across shots, we make sure to use the same set of parallel samples across languages, using the sample ids, to compare how a set of samples performs when translated to different languages. This is possible due to our selection of tasks and datasets that have a parallel corpus for the various target languages.
4.2 Translate-train experiments
We train using MT of the source train set to each target language $^3$  and adapt a similar setup as few-shot: for each language, continue fine-tuning separately on  $n \in \{10, 50, 100, 500, 1k, 10k, |dataset|\}$  samples from the machine-translated train set and report the results on the test set of the target language.
5.1 How to translate your samples? Few-shot vs. translate-train
To demonstrate the full potential for each approach, Table 2 shows a performance summary for zero-shot, few-shot and translate-train when the maximum possible number of samples is used. The gap to English performance is the average of the gap between the target language performance and the performance on the English test set. Both few-shot and translate-train help bridge the gap, but using translate-train on a large scale has an advantage
<sup>3</sup>We use the Machine Translation provided by the XTREME Benchmark:
https://console.cloud.google.com/storage/browser/xtreme_translations
XNLI	PAWS-X	XQuAD
langa	engap	gain	langa	engap	gain	langa	engap	gain
English performance
mBERT	81.85±0.99	-	-	92.81±0.33	-	-	83.74±0.36	-	-
XLM-RB	84.04±0.65	-	-	93.99±0.35	-	-	83.10±0.29	-	-
XLM-RL	88.98±0.29	-	-	95.13±0.35	-	-	87.07±0.59	-	-
zero-shot (only en training)
mBERT	65.75±0.31	-16.10	-	81.24±1.58	-11.57	-	61.51±0.24	-22.22	-
XLM-RB	73.79±0.34	-10.26	-	82.08±0.92	-11.92	-	70.51±0.32	-12.60	-
XLM-RL	79.61±0.61	-9.37	-	85.89±0.59	-9.24	-	76.98±0.25	-10.10	-
few-shot (max 1k high-quality translated samples)
mBERT	67.96±0.29	-13.89	2.21	83.01±0.91	-9.80	1.76	65.06±0.27	-18.67	3.55
XLM-RB	75.50±0.30	-8.54	1.71	82.83±0.53	-11.16	0.76	70.68±0.28	-12.42	0.17
XLM-RL	81.70±0.14	-7.27	2.10	86.73±0.17	-8.40	0.84	77.06±0.16	-10.01	0.08
translate-train (full machine-translated training set)
mBERT	72.81±0.24	-9.04	7.06	85.74±0.67	-7.07	4.50	69.84±0.34	-13.89	8.33
XLM-RB	76.95±0.32	-7.09	3.16	85.06±0.66	-8.93	2.99	72.16±0.16	-10.95	1.65
XLM-RL	82.46±0.19	-6.51	2.86	88.73±0.20	-6.40	2.83	77.26±0.35	-9.81	0.29
Table 2: Performance summary. The average performance on all languages along with the gap to the English performance and the gain over zero-shot performance. Using translate-train on a large scale has a clear advantage.
in further narrowing the gap as compared to the small scale of few-shot transfer. This results in translate-train having the best performance for all models across all datasets. The highest gain is seen for the model with the highest en gap (mBERT) for both few-shot and translate-train. For XLM-R on XQuAD, the gain is low and negligible. Given that there is a significant gain for mBERT and the same experimental setup is used for all models, the lack of gain is probably not dataset-specific and possible happens with some models.
To see the effect of the available dataset size in each scenario, Figure 1 shows the average performance across languages for few-shot vs. translate-train across varying number of samples. We can see an advantage of having manual over machine translation resulting in a clear performance gap between both on XNLI for the same number of samples. This gap increases with the increase of the number of samples as seen at  $1\mathrm{k}$ . The availability of manual translation for few-shot is limited though and starting from  $10\mathrm{k} - 100\mathrm{k}$ , the scale of translate-train has an advantage for all tasks (similar results for the other models are in Appendix Figure 7, 8). The performance on PAWS-X and XQuAD does not improve much with few-shot as shown in Figure 1b and 1c, and the clear boost comes from using the large scale machine-translated dataset. We discuss the observed large variance on XQuAD across languages near the end of the following section.
Detailed results & language analysis: Figure 2 shows the translate-train performance gain (over zero-shot) across a varying sample size for each language family (More details about the languages
(a) Average Accuracy on XNLI
(b) Average Accuracy on PAWS-X
(c) Average F1 on XQuAD
Figure 1: Avg performance across langs for translate-train vs. few-shot using XLM-R $_B$ . The biggest performance boost comes from using translate-train
(a) XNLI / mBERT
(b) PAWS-X / mBERT
(c) XQuAD / mBERT
(d) XNLI / XLM-  $R_B$
(e) PAWS-X / XLM-  $R_B$
(f) XQuAD / XLM-  $\mathbb{R}_B$
(g) XNLI / XLM-  $\mathbb{R}_L$
(h) PAWS-X / XLM-  $\mathbb{R}_L$
(i) XQuAD / XLM-  $\mathbf{R}_L$
Figure 2: Translate-train gain on language families. European languages especially Germanic have the least gain, whereas various other families like Koreanic and Niger-Kongo have the most gain (detailed results in Appendix C)
in Appendix Table 6). We can see, across all tasks and models, that European languages have a small gain compared to non-European languages which show the largest gain e.g. Swahili (Niger-kongo) in XNLI, Korean and Japanese in PAWS-X, and Turkish and Chinese for XQuAD. Those languages also tend to have a larger zero-shot performance gap to English and are more distant to it (the source language). Those results are comparable to the few-shot results of Lauscher et al. (2020). We can see that the languages with the most gain differ be
tween mBERT and XLM-R mainly because XLM-R extends the pre-training corpus using CommonCrawl to have more data that less-spoken languages benefit especially from e.g. Turkish zero-shot performance on XQuAD is low with mBERT as compared to XLM-R models which result in more gain for Turkish with mBERT on XQuAD (detailed results on XQuAD in Appendix Figure 15, 14, 16).
Appendix C contains the detailed performance gains for few-shot and translate-train over zero-shot for each language across varying sizes of sam
Figure 3: Detailed Results on XNLI using  $\mathbf{XLM - R}_B$ . Gains in performance over zero-shot for few-shot and translate-train. Low-resource languages like Swahili have the most gains in both cases
plies. Figure 3 shows the detailed results for XNLI as an example, where we see that once the full machine-translated training set is used, a clear advantage for translate-train is seen across almost all languages and in all tasks. We can see that the gain for Urdu (ur) is the highest on XNLI up until  $100\mathrm{k}$  when it starts decreasing. We think this might be due to a lower-quality MT. The same effect is seen for Thai (th) on XQuAD with a significant performance degrade when the full training dataset is used (details in the Appendix in Figure 15). This is also the reason for the degrade and high variance in performance seen at this point in Figure 19b.
We investigate whether longer training would have changed the results and would have been beneficial, especially for few-shot where longer training on the high-quality manual translation might be beneficial. We split the available set of samples into train/dev and train for 10 epochs with early stopping on dev. Although some languages benefit from this setup, it still yields comparable results and translate-train still has a clear advantage. (results in Appendix Figure 17 and 18).
5.2 How to choose your shots? Which samples to translate for few-shot?
Few-shot can still have an advantage over translate-train when the English dataset is not large enough to benefit from the scale effect of translate-train. It can also be necessary when adapting to a target language that does not have an existing machine translation system or does not have a good one. Creating few-shot samples, in this case, can be
done by collecting and labeling new samples or by translating samples from the available English dataset. The latter is a common method and 4 out of the 7 non-retrieval datasets in XTREME use manual professional translation to create samples in the target languages (all of which high-level semantic tasks). It is beneficial then to support in selecting the samples with higher performance-potential to translate and do few-shot training on.
To emphasize the significance of choosing the samples, we plot in Figure 4 the XNLI performance variance on different shots (using the same model initialization) across 20 sets of random few-shot samples varying in size from 10 to 1k samples. The performance varies, sometimes significantly, depending on the set of samples used. Zhao et al. (2021) shows similar variance obser
Figure 4: XNLI accuracy variance on different shots. High variance decreases with an increased data size
Figure 5: XNLI Pearson correlation between the performance on English and the performance on other languages.
ar	bg	de	el	es	fr	hi	ru	sw	th	tr	ur	vi	zh
0.65	0.86	0.89	0.75	0.88	0.88	0.80	0.75	0.72	0.85	0.86	0.61	0.89	0.89
Table 3: XNLI Pearson correlation between the performance of machine translation and manual translation
vations on a smaller number of samples (around 10). We consider a larger size range that is more representative of the data size if a manual translation is conducted. The performance variance across shots decreases with the increased number of shots. This means that choosing the shots to translate is more important when a smaller size of samples is used. (similar results on PAWS-X and XQuAD are in Appendix Figure 19 although for XQuAD the variance increases with the size). In the following, we focus mainly on XNLI as the task that had the most few-shot gain. We investigate whether there are sets of samples that have a potential for better performance across languages and what could be an indication of that. For a set of shots, we consider two indicators: the performance of this set in another language, and the performance on the MT of the samples in the set.
5.2.1 Correlation between performance across languages
If the performance of a set of samples for one language can be an indication of its performance on another language, a high correlation between the performance for both languages is expected. To estimate this, we calculate the performance using the manual translations across languages of the same set of training samples. We then calculate the Pear
son correlation of the performance across 5 random sets of samples (with varying sample-set sizes) using 5 models with different random initialization. As seen in Figure 5, there is a high positive correlation between the performance on XNLI for the various languages (using  $\mathrm{XLM - R}_B$ ). This is also the case, but to a lesser degree for PAWS-X as seen in the Appendix Table 8. XQuAD, on the other hand, has low and sometimes even negative correlation (Appendix Table 11), which might be due to the QA task being harder and requiring more data and the fact that we have less data in this case for both training and test. It is also worth noting that the correlation is lower for both tasks, PAWS-X and XQuAD, which had low few-shot gain.
A breakdown of the English correlation based on data size is shown in the Appendix Table 7 and 9. As an example of this, Figure 6 shows XNLI few-shot gain over zero-shot performance for 5 random sets of samples  $\{A,B,C,D,E\}$  each containing 10 samples. The performance is shown for 3 different XLM-  $\mathbf{R}_B$  initializations. The sets  $\{A,C,E\}$  perform better than  $\{B,D\}$  across target languages and on average as well as across different initializations. The performance on English can be used as an indicator of the best shots to choose as seen when comparing the English performance (top) to
Figure 6: XNLI few-shot gain over zero-shot across 5 sets of samples (size=10) for 3 different model initializations. Sets C, A, and E yield better performance for the 3 different initializations. The English performance can be used as an indicator when choosing samples to translate