Revisiting Machine Translation for Cross-lingual Classification
Authors: Mikel Artetxe
Vedanuj Goswami
Shruti Bhosale
Angela Fan
Luke Zettlemoyer
Venue: EMNLP 2023 Main (2023)
Keywords: multilinguality
cross-lingual classification
machine translation

Abstract: Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate-test), or translating the training set into the target languages and finetuning a multilingual model (translate-train). However, most research in the area focuses on the multilingual models rather than the MT component. We show that, by using a stronger MT system and mitigating the mismatch between training on original text and running inference on machine translated text, translate-test can do substantially better than previously assumed. The optimal approach, however, is highly task dependent, as we identify various sources of cross-lingual transfer gap that affect different tasks and approaches differently. Our work calls into question the dominance of multilingual models for cross-lingual classification, and prompts to pay more attention to MT-based baselines.

Revisiting Machine Translation for Cross-lingual Classification
Mikel Artetxe 1.2*
Vedanuj Goswami Shruti Bhosale
Angela Fan Luke Zettlemoyer
Reea AI
HiTZ Center, University of the Basque Country (UPV/EHU) mikelêreka.al ai
Meta AI
[vedanuj, angelafan,,, @mm ta. com
Abstract
Conneau et (2023)
Ours
86
Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate- test), or translating the training into the target languages and finetuning a multilingual model (franslate-train). However, most re- search in the focuses the multilingual models rather than the MT component. We show that, by using stronger MT system and mitigating the mismatch between train- ing on original text and running inference on machine translated text, translate-test can substantially better than previously assumed. The optimal approach, however, is highly task identify various sources of cross-lingual transfer gap that affect different tasks and approaches differently. Our work calls into question the dominance of multi- lingual models for cross-lingual classification, and prompts to pay more attention MT- based baselines.
84
XNLL aoLuaa
80
78
z0rc-shot (XLM-R)
yanslass-trai train (XLM-R)
transbitakisi (RoBERTa)
Figure XNLI accuracy. We show that translate-test can do substantially better than previously reported.
The conventional wisdom is that translate-train tends bring modest improvements zero-shot, while translate-test has been largely overlooked and is often not even considered et al., 2020; Ruder et al., 2021). In this context, most recent research in multilingual NLP has fo- cused on pretraining stronger multilingual mod- els (Conneau et al., 2020; Xue et al., 2021: Chi et al., 2022), and/or designing better fine-tuning techniques elicit tross-lingual transfer (Liu al., 2021; Yu and Joty, 2021: Zheng et al., 2021; Fang et al., 2021). However, little attention has been paid the MT component despite its central role in both translate-train and translate-test. Most authors use the official translations that some mul- tilingual benchmarks come with, and it is unclear the extent to which better results could be obtained by using stronger MT systems developing better integration techniques.
dependent,
baseline (Hu
1 Introduction
Recent work in cross-lingual learning has pivoted around multilingual models, which are typically pretrained on unlabeled сorpora in multiple lan- guages using some form of language modeling objective (Doddapaneni al., 2021). When fine- tumed on downstream data in a single language- typically English-tte these models are able to gen- cralize to the rest the languages thanks to the aligned representations learned at pretraining time. an approach known as zero-shot transfer. The called translate-train is an extension of this method that augments the downstream training data by translating it all target languages through MT. A third approach, translate-test, uses MT trans- late the test data into English, and runs inference using an English-only model.
In this work, we revisit the use of MT for cross- lingual learning through extensive experiments on 6 classification benchmarks. We find evidence that translate-test is more sensitive than franslate-train the quality of the MT engine, and show that better results can be obtained by mitigating the mis- match between training on original downstream data and running inference on machine translated
Work done at Meta Al

---

data, As exemplified by Figure we demonstrate that, with enough care. translate-test can work sub- stantially better than previously assumed (e.g., out- performing both zero-shot and translate-train on XNLJ for the first time), calling into question the dominance of multilingual pretraining in the area. However, these trends are not universal, as we find that the optimal approach is highly task depen- dent. We introduce new methodology to quantify the underlying sources of cross- lingull transfer gap that cause this discrepancy across tasks. We find that translate-test excels on complex tasks requir- ing commonsense or real world knowledge, as it benefits from the use of a stronger English-only model. In contrast, translate-train performs best shallower tasks like sentiment analysis, for which the noise introduced by MT outweighs the benefit of using a better pretrained model.
Evaluation. We use the following 6 datasets for evaluation: XNLI (Conneau al., 2018). a Natural Language Inference (NLI) dataset covering 15 lan- guages; PAWS-X (Yang et al., 2019), an adversar- ial paraphrase identification dataset covering 7 lan- guages; MARC (Keung et al., 2020), a sentiment analysis dataset in 6 languages where one needs to predict the star rating of an Amazon review; XCOPA (Ponti et al., 2020), causal common- sense reasoning dataset covering 12 languages; XS- toryCloze (Lin et al., 2021), commonsense rea- soning dataset in 11 languages where one needs to predict the correct ending to a four-sentence story; and EXAMS (Hardalov et al., 2020), a dataset of high school multiple choice exam questions in 16 languages. For fair comparison between different approaches, we exclude Quechua and Haitian Cre- ole from XCOPA, as the former not supported by NLLB and the latter is not supported by XLM-R. In all cases, we do 5 finetuning runs with different ran- dom seeds, and report accuracy numbers in the test set averaged across all languages and runs. Follow- ing common practice, we use MultiNLI (Williams et al., 2018) as our training data for XNLI, and PAWS (Zhang al., 2019) as our training data for PAWS-X. For MARC, use the English portion of the training set. XCOPA, XStoryCloze and EX- AMS do not come with a sizable training set in English, so we train on the combination of the fol- lowing multiple choice datasets: Social IQa (Sap et al., 2019), SWAG (Zellers et al., 2018), COPA (Roemmele et al., 201 1), OpenBookQA (Mihaylov et al., 2018), ARC (Clark et al., 2018) and PIQA (Bisk et 2020).
2 Experimental setup
Pre-trained models. We use RoBERTa large (Liu et al., 2019) and XLM-R large (Conneau et al., 2020) as our primary English-only and multilingual models, respectively. XLM-R can be considered the multilingual version of RoBERTa, as they are both encoder-only masked language models trained with the same codebase. So to understand the benefits from using a stronger English-only model, we also experiment with DeBERTaV3 large (He et al., 2021). All the models have 304M back- bone parameters, although they differ in the size of their vocabulary.
Machine translation. We use the 3.3B NLLB model (NLLB Team et al., 2022) as our MT en- gine. We explore two decoding strategies in- dicated in each experiment: beam search with beam size of 4, and nucleus sampling (Holtzman et al., 2020) with top-p- = 0.8. Unless otherwise indicated, we translate at the sentence level using the w__sent_ud_sii _m model in spaCy for sentence segmentation.? 2 For variants translating at the doc- ument level, we concatenate all fields (e.g., the premise and the hypothesis in XNLI) using spe- cial word <sep> separate them. For experiments involving MT fine-tuning, we use a learning rate of 5e-05 and a batch size of 32k tokens with dropout disabled, and use the final checkpoint after 25k steps.
Fine-tuning, We use HuggingFace's Transform- ers library (Wolf et al., 2019) for all of our ex- periments. So as to handle variable number of candidates in multiple choice tasks (XCOPA, XStoryCloze and EXAMS), we feed each input- candidate pair independently into the model, take its final representation from the first token, down- project into a scalar score through a linear projec- tion, and apply the softmax function over the scores of all candidates. For the remaining tasks, we sim- ply learn a regular classification head. In all cases, use batch size of 64 and truncate examples longer than 256 tokens. We train with a learning rate of 66-6 with linear decay and 50 warmup steps, and use the final checkpoint without any model selection. When using the original English training set, we finetune for two epochs. For experiments
'In fact, XLM-R stands for XLM-RRBERTa. Pnctps://sanc so/mdeess/xxtte sont__d_eeet

---

involving some form of data augmentation, where each training example (back-) )translated into mul- tiple instances, we finetune for a single epoch.
MT engine
xnli
pwsx
mare
Xeop
Xsto
exm
Official NLLB
76.8
75.9
79.9
87.3
57.6
72.9
89.3
36.3
+ dom udapt
80.8
86.9
58.2
72.9
88.4
36.5
3
Main results
doc level
83.8
87.8
58.3
76.3
91.3
36.3
We next present our main results revisiting MT for cross-lingual classification, both for translate-test ($3.1) and translate-train ($3.2). Finally, we put everything together and reconsider the state-of-the- art in the light of our findings ($3.3).
Table 1: Translate-test results with different MT en- gines. All variants use RoBERTa model finetuned on the original English data.
the document level test time, whereas the rest of the systems translate at the sentence level. 3
3.1 Revisiting translate-test
We find that document-level fine-tuned NLLB obtains the best results across the board, obtain- ing consistent improvements over vanilla NLLB in all benchmarks except EXAMS. We remark that the former does not have any unfair advantage in terms of the data it sees, as it leverages the ex- act same downstream data that RoBERTa is fine- tuned on. Sentence-leve fine-tuned NLLB is only able to outperform vanilla NLLB on XNLI, MARC and EXAMS, and does worsc on PAWS-X and XStoryCloze. This shows that a large part of the improvement from fine-tuning NLLB can be at- tributed to learning to jointly translate all sentences and fields from each example.
While translate-test might be regarded as a simple baseline, we argue that there are two critical as- pects of it that have been overlooked in prior work. First, little attention has been paid to the MT engine itself: most existing works use the official transla- tions from each dataset without any consideration about their quality, and the potential for improve- ment from using stronger MT systems is largely unknown. Second, translate-tent uses original (hu- man generated) English data to fine-tune the model, but the actual input that fed into the model at test time is produced by MT. Prior work has shown that original and machine translated data have different properties, and this mismatch is detrimental for per- formance (Artetxe et al., 2020), but there is barely any work addressing this issue. In what follows, we report results strong MT system, and explore two different ap- proaches to mitigate the train/test mismatch issue: adapting MT to produce translations that are more similar to the original training data 153.1.1), and adapting the training data to make it more similar to the translations from MT 1.3.1.2).
Finally, also find high variance in the qual- ity of the official translations in the two datasets that include them. More concretely, the official translations are 3.0 points better than vanilla NLLB on XCOPA, and 3.1 points worse on XNLI. De- spite overlooked to date, we argue that this fac- tor has likely played an important role in some of the results from prior work. For instance, Ponti et al. (2020) obtain their best results on XCOPA using translate-lest with RoBERTa, whereas Con- neau et al. (2020) find that this approach obtains the worst results on XNLII. These seemingly con- tradictory findings can partly be explained by the higher quality of the official XCOPA translations, and would have been less divergent if the same MT engine was used for both datasets.
6 evaluation tasks using
3.1.1 MT adaptation
Table reports translate-test results with RoBERTa using translations from 4 different MT systems: the official ones from each dataset (if any), the vanilla NLLB 3.3B model, and two fine-tuned variants of it. In the first variant (+dom adapt), we segment the downstream training data into sentences, back- translate them into all target languages using beam search, and fine-tune NI.LB on it as described in $2. The second variant (+doc level) is similar, except that we concatenate the back-translated sentences first, and all the input fields (e.g. the premise and the hypothesis) afterwards, separated by <sep>, and fine-tune NLLB on the resulting synthetic par- allel data. For this last variant we also translate at
3.1.2 Training data adaptation
We experiment with form of data augmentation where we translate each training example into an- other language and then back into English. The
We also tried translating the document level with the rest the systems, but this worked poorly in our preliminary experiments, as the model would often only translate the first sentence and ignore the rest.
'In the forward (out-of- lout-of-English) direction, we use beam search for multiplechoice tasks (XCOPA, XStoryCloze, EX-

---

Train data
xnli
pwSx
mare
xeep
xsto
MT engine
xnli
pwsx
mare
xcop
xsto
exm
Original
79.9
87.3
57.6
72.9
89.3
36.3
None (zero-shot)
80.1
87.1
60.6
69.1
84.6
36.0
Roundtrip MT
85.2
89.9
58.8
74.3
91.2
36.2
Official
83.3
90.7
+ MT adapt (doc)
85.9
89.3
59.1
75.7
91.2
36.4
NLLB (beam)
83.5
90.4
60.5
68.5
86.4
36.0
NLLB (samp)
84.3
90.7
60.8
67.6
86.7
35.2
Table 2:
Translate- test results different training data. All methods vanilla NLLB for translation.
Table 3: Translate-train results using XLM-R
resulting data is aimed to be more similar to the in- put that the model will be exposed to at test time, as they are both produced by the same MT model. For each task, we use all target languages it covers as the pivot, and further combine the back-translace examples with the original data English. We compare this approach to the baseline (training on the original English data), and report our results in Table 2. So as to understand how complementary training data adaptation and MT adaptation are, we also include system combining the two, where we use the document-level fine-tuned model from $3.1.1 translate the test examples.
and nucleus sampling. In all cases, translate the downstream training data into all target languages covered by each task, and fine-tune XLM-R on this combined data (including a copy of the origi- nal data English). We also include a zero-shot system as a baseline, which fine-tunes the same XLM-R model on the original English data only. results.
Table 3 reports
We find that translate-train obtains substantial improvements over zero-shot half of the tasks (XNLI, PAWS-X X and XStoryCloze), but performs at par or even worse in the other half (MARC, XCOPA and EXAMS). This suggests that translate- Irain might not be as generally helpful found in prior work (Hu et al., 2020). In those tasks where MT does help, nucleus sampling outperforms beam search. This is in line with prior work in MT find- ing that sampling is superior beam search for back-translation (Edunov et al., 2018) but, to the best of our knowledge, we are first to show that this also holds for cross-lingual classification.
We find that roundtrip MT outperforms the base- line by a considerable margin in all tasks but EX- AMS. While prior work has shown that multilin- gual benchmarks created through (professional) translation have artifacts that can be exploited through similar techniques (Artetxe et al., 2020), it is worth noting that also get improvements on MARC, which was not generated through trans- lation. Finally, we observe that combining this approach with MT adaptation is beneficial in most cases, but the improvements are small and incon- sistent across tasks. This suggests that the two techniques are little complementary, which is not surprising as they both try to mitigate the same underlying issue.
Finally, find that translation quality had considerably larger impact for translate-test than does for translate-train. More concretely, the XNLI accuracy gap between the official transla- tions and vanilla NLLB was 3.1 for translate-traine and is only 0.2 for translate-test when using beam search, or 1.0 when using nucleus sampling. This suggests that the use of relatively weak MT engines in prior work might have resulted in underestimat- ing the potential of translate-test relative to other approaches.
3.2 Revisiting translate-train
As seen in $3.1, the MT engine can have a big impact in translate-test. To get the full picture, we next explore using the same MT engines for translate-train including the official translations (when available) as well as NLLB with beam search
3.3 Reconsidering the state-of-the-art
We have so far analyzed translate-test and translate- individually. We next put all the pieces together, and compare different pretrained models (XLM-R, RoBERTa and DeBERTa) using zero-shot. translate-train with nucleus sampling. and two variants of translate-test: the naive ap- proach using vanilla NI.LB, and our improved ap- proach combining document-level MT adaptation and training data adaptation. We report our results
AMS), and nucleus sampling for the rest of the tasks. In the backward direction, we beam search for all tasks. We made this decision based on preliminary experiments on the development set. Sampling the forward direction produced more diverse translations, tasks, where some options are very short. The use beam search when translating into English is consistent with the decoding method used at test time.
was noisy multiplechoice
As such, if the original dataset has examples and we have n target languages, the resulting data will consist of & * (1 + 1) examples.

---

xnli
pwsx
mare
xeоp
Xsto
exm
avg
zero-shot
80.1
87.1
60.6
69.1
84.6
36.0
69.6
translate-train
84.3
90.7
60.8
67.6
86.7
35.2
70.9
XLM-R
vanilla
79.3
86.9
58.0
68.6
84.8
34.9
68.8
translate-test
ours
84.6
89.3
58.8
69.0
87.9
35.0
70.8
vanilla ours
79.9
87.3
57.6
89.3
36.3
70.6
RoBERTa
translate-test
85.9
89.3
59.1
75.7
91.2
36.4
72.9
vanilla
81.0
87.1
58.2
77.7
92.1
46.1
73.7
DeBERTa translate-test
86.7
90.3
59.2
81.3
93.8
46.0
76.2
Table 4: Main results. All systems use NLLB for MT. Best model results underlined, best overall results in bold.
in Table 4.
is slightly better in the other 2. For instance, De- BERTa translate-test is 13.7 points better than XLM-R translate-train on XCOPA, but 1.6 points worse MARC. This suggests that different tasks have different properties, which make different ap- proaches more or less suitable.
We observe that our improved variant of translate-test consistently outperforms the vanilla approach. Interestingly, the improvements are gen- erally bigger for stronger pretrained models: an av- erage of 2.0 points for XLM-R, 2.3 for RoBERTa, and 2.5 for DeBERTa.
Analyzing the variance across tasks
When comparing different approaches, we find that zero-shot obtains the worst results in average, while translate-tred train is only 0.1 points better than translate-test with XLM-R. However, this compar- ison unfair to translate-test, there is no point in using a multilingual pretrained model when one is translating everything into English at test time. When using RoBERTa (a comparable English-only model), translate-test outperforms the best XLM- R results by 2.0 points. Using the stronger De- BERTa model further pushes the improvements to 5.3 points.
As have just seen, the optimal cross-lingual learning approach is highly task dependent, In this section, we try to characterize the specific factors that explain this different behavior. To that end, we build on the concept of cross-lingual transfer gap, which is defined as the difference in performance between the source language that we have training data in (typically English) and the target language that
evaluating (Hu et al., 2020). While prior work has used this as an absolute metric compare the cross- less-lingual generalization capabili- ties of different multilingual models, we argue that such a transfer gap can be attributed to different sources depending on the approach used, which we try to quantify empirically.
These results evidence that translate-tess can be considerably more competitive than suggested by prior work. For instance, the seminal work from Conneau al. (2020) reported that RoBERTa translate-test lagged behind XLM-R zero-shot and translate-train on XNLI. As illustrated in Figure 1, we show that, with enough care, can actually obtain the best results of all, outperforming the original numbers from Conneau et al. (2020) by 8. points. Our approach is also 3.9 points better than Ponti et al. (2021), the previous state-of-the-art for translate-test in this task. While most work in cross-lingual classification focuses on multilingual models, this shows that competitive or even supe- rior results can also be obtained with English-only models.
In what follows, we identify the specific sources of transfer gap that each approach sensitive ($4.1), 1), propose methodology to estimate their impact using monolingual dataset ($4.2), and present the estimates that we obtain for various tasks and target languages ($4.3).
4.1 Sources of cross-lingual transfer gap
We next dissect the specific sources of cross-lingual transfer gap that each approach is sensitive to:
Translate-test.
All the degradation comes from MT, no multilingual model is used. We distin- guish between the information lost in the trans- lation process (either caused by translation errors or superficial patterns removed by MT), and (ii) the
Nevertheless, we also find a considerable vari- ance across tasks. The best results are obtained by translate-test in 4 out of 6 benchmarks in most cases by a large margin- but translate-train

---

distribution shift between the original data seen during training and the machine translated data seen during evaluation (e.g., stylistic differences like translationese that existing models might strug- gle generalizing to, even no information is lost).
created through translation (Artetxe al., 2020), but can also be result of the task having nat- urally different properties in different languages (e.g., question answering datasets in different lan- guages might cover different topics that are rele- vant to their corresponding speaker communities). Quantifying this factor would require comparing translated and original annotations for each task, which we do not consider in analysis given the lack of such data.
Zero-shot. All the degradation comes from the multilingual model, as no MT is used. We dis- tinguish between (i) source-language represen- tation quality" relative to a monolingual model (English-only models being typically stronger than their multilingual counterparts, but only us- able with translate-test), (ii) target-language rep- resentation quality relative to the source lan- guage (the representations of the target language- typically less-resourced-beeing worse than those of the source language), and (iii) representation misalignment between the source and the target language (even when a model has a certain capabil- ity in both languages, there can be a performance gap when generalizing from the source to the target language if the languages are not well-aligned).
4,2 Methodology
Let Tyre be some training data in the source language, Tigs
(Tare) its MT- generated translation into the target language, and The = MTTggt-rre (Teg:) its MT-generated transla- tion back into the source language. Similarly, let Eare. Etgt and Eot be analogously defined evalu- ation sets. We define acc(T, E) as the accuracy of our multilingual model when training T and evaluating on E, and accmono (T, E) the accu- racy of its equivalent monolingual model when training on T and evaluating on E. Given this, we estimate the cross-lingual gap from each of the sources discussed in $4.1 as follows:
Translate-train. The degradation comes from both MT and the multilingual model. However, while both source and target language represen- tation quality have an impact, ? this approach is not sensitive to representation misalignment, as the model is trained and tested in the same language. Regarding MT, there is no translation and therefore no information lost at test time, potential translation errors training time to be further inducing a distribution shift.
MT information lost. We take the difference be- tween training and testing on original data, and training and testing on back-translated data. Given that there is not distribution shift between train and test induced by MT, the difference in perfor- mance can be solely attributed to the information lost. MT is used twice when translating into the target language and then back into the source lan- guage, so we make the assumption that each of them introduces a similar error and divide the total gap by two to estimate the impact of single step:
consider
Finally, there can also be an inherent distri- bution mismatch across languages the bench- mark itself (e.g., the source language training data and the target language evaluation data having dif- ferent properties). This can be due to annotation artifacts in multilingual datasets, in particular those
MT acc (Trr: Esrc) acc(Tor, Fbt) info
2
"We consider that pretrained model A has learned higher quality representations than model B if fine-tuning in our target task results in better downstream performance. In this case, English-only models like RoBERTa generally su- perior to their multilingual counterparts like XLM-R when evaluated downstream English tasks, we say that their source-language representation quality is higher.
MT distribution shift. We take the difference between training and testing on back-translater data, and training on original data and evaluating on back-translated data. Similar to info we divide this difference by two, assuming that each MT step introduces the same error:
"Even model is trained and tested on the target lan- guage, we consider that there is still a degradation from the source-language representation quality under our framework. This is because we are measuring the degradation from us- ing a multilingual model in the target language as opposed to monolingual model in the source language, which is de- composed into the difference between the monolingual and multilingual model in the source language. plus the difference hetween the source and the target language in the multilingual model.
A MT acc(Tir, EMd) acc(Tsre, Ebt) dist
Source representation quality. We take the dif- ference between the monolingual and the multilin- gual model, training and testing on original data:
rер are
-CC (T we:: ESTc) - acc Exre)

---

10
onta
MT lasz sheto
Rep quailty
quality sec)
quallty
quality Pgp
Hop
mitmmtch
Xni
Xnli
snterycloow
(a) All languages
(b) High-resource related languages
MT Info lut
20
HT (ints
MT 1012 shino
(cien
Rep quality (sn))
Rep quallty
Kop quality fag
15
Rep quality
Rop mismotch
Rep mismatch
10
(c) High-resouree unrelated languages
(d) Low-resource unrelated languages
Figure 2: Estimation the impact of different sources cross-lingual transfer gap.
Target representation quality. We take the dif- ference between training and testing on original data, and training and testing translated data, and further subtract A MT to account for the error from MT:
English (German, Dutch, French, Spanish, Italian), 5 high-resource unrelated languages (Turkish, Viet- namese, Japanese, Finnish, Arabic), and 5 low- resouree unrelated languages (Malagasy, Oromo, Javanese, Uyghur, Odia). Figure 2 reports the esti- mates that we obtain, and we next summarize our main findings:
fo
A rep = acce Esre) acc(Tigt, Fngt) A MT info
Representation misalignment. We take the dif- ference between training and testing on translated data, and training original data and testing on translated data, and further subtract A MT' dist to ac- count for the error from the distribution mismatch in the later:
Degradation from MT. We find that the impact of MT greatly varies across tasks. After manual inspection, believe that the most relevant fac- tors causing these differences are the format, lin- guistic complexity and domain of each task. For instance, degradation from MT is most prominent in OpenBookQA, a dataset of multiple choice sci- ence questions. We find two reasons for this: (i) the questions and answers often contain domain- specific terminology that is hard to translate, and (ii) the answer candidates are often unnatural lack the necessary context be translated in iso- lation. In contrast, each example in XStoryCloze consists of short and simple sentences that form story, and we find the impact of MT to be small in this dataset. As expected, MT has a larger abso-
rcp
aee(Trge, -aco) Eege) A MT
align
dist
4.3 Results and discussion
We experiment on our usual set of tasks, us- ing the exact same training data as in $3 as Tare. and the English portion of the respective test sets E.rc. EXAMs does not have English test set, we the analogous OpenBookQA (Mi- haylov al., 2018) instead. We use XLM-R as our main model and RoBERTa as our monolin- gual model. We explore a diverse set of target languages: 5 highures languages related to
DDifferent from our previous experiments, we do not per- form multiple finetuning runs with different random seeds due to the high computational cost. However, our results are averaged across languages each of them requiring separate finetuning runs.

---

lute impact in distant and low-resource languages. While degradation from MT comes predominantly from information lost during the translation pro- cess, a non-negligible part can be attributed to the distribution shift, which we addressed $3.1.
5 Related work
While using MT for cross-lingual classification is long standing idea (Fortuna and Shawe- Shawe-Taylor, 2005; Banea et al., 2008; Shi et al., 2010; Duh et al., 2011), 1),, there is relatively little work focus- ing it in the era of language model pretraining. Ponti et al. (2021) improve translate-test by treat- ing the translations as a latent variable, which al- lows them to finetune the MT model for the end task through minimum risk training and combine multiple translations at inference. Our approach is simpler, but obtains substantially better results on XNLI and PAWS-X. X. Artetxe et al. (2020) explore simpler variant of our training data adaptation ($3. (,2,, but their focus is translation artifacts and our numbers are considerably stronger. Oh al. (2022) show that translate- trainn and translate-test are complementary and better results can be ob- tained by combining them. Isbister et al. (2021) report that translate-test outperforms monolingual models fine-tuned on the target language, but their work is limited to sentiment analysis in Scandina- vian languages. While we focus classification tasks, there is also considerable body of work exploring MT for cross-lingual sequence labeling, which has the additional challenge of projecting the labels (Jain et al., 2019; Fei et al., 2020; García- Ferrero et al., b022a,b)
Degradation from the multilingual model. The impact of both the source and target represen- tation quality greatly varies across tasks, and the two are highly correlated. This suggests that there are certain tasks for which representation quality (both source and target) is generally important. By definition, the degradation from the source repre- sentation quality is invariant across languages, but the degrudation from the target representation qual- ity becomes dramatically higher for unrelated and, more importantly, low-resource languages. In gen- eral, we find that tasks requiring commonsense or factual knowledge, like XStoryCloze and Open- BookQA, heavily sensitive to source and target representation quality, whereas shallower tasks like sentiment analysis (MARC) are barely affected. Fi- nally, we find that, in most tasks, representation misalignment has a relatively small impact com- pared to reprexentation quality. This suggests that multilingual models learn well-aligned representa- tions that allow cross-lingual transfer at fine-tuning time, but puts into question the extent to which cross-lingual transfer happening pre-training time, as the target language representations can be considerably worse than the source language representations.
Conclusions
Contrary to the conventional wisdom in the area, we have shown that translate-test can outperform both zero-shot and translate-train in most clas- sification tasks. While most research in cross- lingual learning pivots around multilingual mod- els, these results evidence that using an English- only model through MT is a strong-and often auperior-altermative that should not be overlooked. However, there no one method that is optimal across the board, as not all tasks are equally sensi- tive to the different sources of lingull trans- fer gap. Using a new approach to quantify such sources of transfer gap, we find evidence that com- plex tasks like commonsense reasoning are more sensitive to representation quality, making them more suitable for translate-test, whereas shallower tasks like sentiment analysis work better with mul- tilingual models. In the future, we would like extend our study to other types of NLP problems like generation and sequence labelling, and study how the different approaches work at scale.
Explaining the variance across tasks. Our re- sults can explain why the optimal cross- tross-lingual learning approach is highly task dependent as ob- served in $3.3. For instance, we find that source representation quality is the most important factor in XStoryCloze, but does not have any impact in only
MARC. This explains why translate-test- approach that not sensitive this source of trans- fer obtinss the best results on XStoryCloze, but lags behind other approaches on MARC. Simi- larly, we find that the impact of the MT distribution shift is highest on XNLI, which is also the tasks at which our improved translate-test approach miti- gating this issuc brings the largest improvements. We remark that our analysis is only using the En- glish portion of the benchmarks. This shows that is feasible to characterize the cross-lingual learning behavior of downstream tasks even if no multilin- gual data is available.

---

Limitations
Kian-Ling Mao, Heyan Huang, and Furu Wei. 2022. XLM-E: Cross- lingusllnalll language model pre- pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 61 6170-6182, Dublin, Ireland. Association for Computational Lin- guistics.
Our study limited to classification tasks, an im- portant yet incomplete subset of NLP problems. Translate-train and translate-test can also be ap- plied to other types of problems like generation or sequence labeling, but require additional steps (e.g. projecting labels in the case of sequence labeling). We believe that it would be interesting conduct similar studies on these other types of problems. At the same time, most multilingual benchmarks- inclluding some used in our study-have been created through translation, which prior work has shown to suffer from annotation artifacts (Artetxe et al., 2020). It should be noted that the artifacts characterized by Artetxe et al. (2020) favor translate-trai and harm translate-test, so we believe that our strong results with the latter are not a result of exploiting such artifacts. However, other types of more subtle interactions might be possible (c.g. translating a text that is itself a translation might be easier than translating an original text). As such, we encourage future research to revisit the topic as better multilingual benchmarks become available. Finally, our experiments are limited to the tra- ditional pretrain-finetune paradigm with encoder- only models. There is some early evidence that translate-ted test also outperforms current multilingual autoregressive language models (Lin et al., 2021; Shi et al., 2022), but further research is necessary to draw more definitive conclusions.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zeule- moyer, and Veselin Stoyanow. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2483 Brussels, Belgium. Association for Computational Linguistics.
Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2021. A primer on pretrained multilingual language models.
Kevin Duh, Akinori Fujino, and Masaaki Nagata. 201 Is machine translation ripe for cross-lings tross-lingual senti- ment classification? In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 429 433, Portland, Oregon, USA. Association for Computational Linguistics.
References
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross- linsull transfer learn- ing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 7674-7684, Online. Associa- tion for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 Understanding back-translation at scale. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Carmen Banea, Rada Mihalcea Janyce Wiebe, and Samer Hassan. 2008. Multilingual subjectivity anal- ysis using machine translation. Proceedings of the 2008 Conference on Empirical Methods in Natu- ral Language Processing, pages 127-135, Honolulu, Hawaii. Association for Computational Linguistics.
Yuwei Fang, Shuohang Wang. Zhe Gan, Siqi Sun, and Jingjing Liu. 2021. Filter: An enhanced fu- sion method for cross coss-lingual language understand- ing. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 35(14):12776-127) 12784
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. PIQA: Reasoning about physi- cal commonsense in natural language. In Proceed- ings of the AAAI conference on artificial intelligence, pages 7432-7439.
Hao Fei, Mcishan Zhang. and Donghong Ji. 2020. Cross-lingus semantic role labeling with high- quality translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7014-7026, On- line. Association for Computational Linguistics.
Zewen Chi, Shaohan Huang. Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song,

---

Blaz Fortuma and John Shawe-Taylor. 2005. The use of machine translation tools for cross- lingual text mining. In Proceedings of the ICML Workshop on Learning with Multiple Views.
reviews corpus. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568, Online. As- sociation for Computational Linguisties.
Iker García-Ferrero, Rodrigo Agerii, and German Rigau. 2022a. Model and data transfer for cross- lingual sequence labelling in zero-resource settings. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 6403-64) 6416, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig. Myle Oit, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo. Jeff Wang. Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2021. Few-shot learning with multilingual language models.
Iker Garcia-Ferrero, Rodrigo Ageri,, and German Rigau. 2022b. T-projection: High quality annotation projection for sequence labeling tasks.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427-5444, Online. Association for Computational Linguistics.
Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2021. Preserving cross-linguality of pre-trained models via continual learning. In Pro- ceedings of the 6th Workshop on Representation Learning for NIP (RepLANT.P-202)) pages 64-71, Online. Association for Computational Linguistics.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentanlled embedding sharing.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learn- ing Representations.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Blahe Kalbassi, Janice Lam, Daniel Licht, Jean Mailtard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jar- rett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat. and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411 4411-4421. PMLR.
Tim Isbister, Fredrik Carlsson, and Magnus Sahlgren. 2021. Should we stop training more monolingual models, and simply use machine translation instead? In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDal.iDa), pages 385- 390, Reykjavik, Iceland (Online). Linköping Univer- sity Electronic Press, Sweden.
Jachoon Oh, Jongwoo Ko, and Se- Young Yun. 2022. Synergy with translation artifacts for training and inference in multilingual tasks. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6747-6754, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Alankar Jain, Bhargavi Paranjape, and Zachary C. Lip- ton. 2019. Entity projection via machine transla- tion for cross-lingual NER. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 1083-1092, Hong Kong, China. Association for Computational Linguistics.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online, As- sociation for Computational Linguistics.
Phillip Keung, Yichao Lu, György Szarvas, and
Noah A. Smith. 2020. The multilingual Amazon

---

Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulié, and Siva Reddy, 2021. Modelling latent translations for cross-lingual transfer.
Barua, and Colin Raffel. 2021. mT5: A massively multilingual tre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 483-498, Online. Association for Computa- tional Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible altemna- tives: An evaluation of commonsense causal reason- ing. In AAAI Spring Symposium: Logical Formal- izations of Commonsense Reasoning, pages 90-95,
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual ad- versarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-JCCNLP), pages 3687- 3692. Hong Kong, China. Association for Computa- tional Linguisties.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Jun- jie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more chal- lenging and nuanced multilingual evaluation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10215- 10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tao Yu and Shafiq Joty. 2021. Effective fine-tuning methods for cross-lingual adaptation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8492-8501, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Maarten Sap. Hamnah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCNIP)), pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversar- ial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93- 104, Brussels, Belgium. Association for Computa- tional Linguistics.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, Di- panjan Das, and Jason Wei. 2022. Language models are multilingual thain-of-thought reasoners.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 1298-1308, Minncapolis, Minnesota. Association for Computational Linguistics.
Lei Shi. Rada Mihalcea, and Mingjun Tian. 2010. Cross language text classification by model trans- lation and sumi-super learning. Proceed- ings of the 2010 Conference on Empirical Meth- ods in Natural Language Processing, pages 1057- 1067, Cambridge, MA. Association for Computa- tional Linguistics.
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang. Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Consistency regularization for cross-lingue fine-tuning. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volume Long Papers), pages 3403- 3417, Online. Association for Computational Lin- guistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya


Reviews:
[{"review_id": "olbpnAWjaq", "reviewer": "Reviewer_p9fP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review. We next address the various points that were raised:\n\n> As far as I can tell, the authors only report experiments across 1 seed.\n\nThis is not accurate, we actually report results across 5 seeds as described in Section 2 (L133-135: *we do 5 finetuning runs with different random seeds, and report accuracy numbers in the test set averaged across all languages and runs*).\n\n> Some of the margins of improvements in quantitative experiments are slim\n\nWe agree that the differences in some datasets are small, but we do not think this contradicts our claims, which we believe are well supported. In fact, one of our main conclusions is that *there is a considerable variance across tasks* (L360-361), and *the optimal cross-lingual learning approach is highly task dependent* (L371-372), which also includes cases where two approaches perform similarly.\n\n> one specific experiment (§3.1.2) corresponds to a much larger training set than other baselines, making it more difficult to compare results across experiments\n\nWhile it is true that this variant uses a larger training set, it is still comparable in terms of the human supervision it uses. More concretely, this variant uses the exact same set of original examples as the others, and all the additional examples are automatically derived from them using MT-based paraphrasing. As such, it does not require any additional (human-labelled) data. This is similar to how translate-train works, which generates a larger training set by translating the original English examples into all other languages and combining them all. This other approach is widely used in the literature, and also covered in our experiments.\n\n> the authors assume that test and training data will have comparable distributions\n\nWhen defining the MT information lost, the paper says that the training and test data come from the same distribution, but what we actually meant is that there is no distribution shift between the training and test set induced by translation, which is true by design. It is not necessary to assume that the training and test data come from the exact same distribution, so we realize that our statement is misleading. Thanks for pointing it out, we will fix it in the camera ready version!\n\n> I'm having some difficulties in conceptualizing what is the relevance of a translate-test approach\n\nWe think that research on translate-test can be relevant from multiple perspectives.\n\nFirst of all, translate-test has an inherent practical interest as an alternative to multilingual pretrained models. While we naturally focus on academic benchmarks, similar multilingual classification problems are very common in industry. Our improvements to translate-test, and the finding that this approach can be competitive and even superior to others in most cases, are thus directly relevant to solve such practical multilingual classification tasks in the real world.\n\nBesides that, while it is true that our work does not propose a new method to learn better multilingual representations, we believe that having strong baselines is important to advance the field. In addition, our analysis in Section 4 shows that different tasks have different properties that explain why existing multilingual models do better or worse in them, and understanding this can be useful to develop better multilingual models in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very cogent and well written,. The results they provide clearly exhibit that a more careful formulation of MT-based approaches to cross-lingual classification can bring about significant improvements over more naive approaches\n- The methodology introduced in §4.2 to evaluate which factors contribute to cross-lingual transfer gaps can be of interest to the community in general", "reasons_to_reject": "**A/** There are some methodological flaws in the current state of the paper, in particular:\n- ~As far as I can tell, the authors only report experiments across 1 seed~ I was mistaken and appreciate the authors' pointing this out.\n- Some of the margins of improvements in quantitative experiments (tables 1, 2, 3 and 4) are slim enough that they call for significance testing\n\nOn a related though less important note:\n- Footnote 5 suggest that one specific experiment (§3.1.2) corresponds to a much larger training set than other baselines, making it more difficult to compare results across experiments\n- In the definition of $\\Delta^{MT}_{info}$  the authors assume that test and training data will have comparable distributions. \n  + This overlooks potential train test mismatches or biases that could in principle be mitigated or obfuscated by the backtranslation procedure\n  + This is problematic, as the same value is then used to calculate $\\Delta^{rep}_{tgt}$.\n\n**B/** On a more theoretical level, I'm having some difficulties in conceptualizing what is the relevance of a translate-test approach. \n- If one considers cross-lingual datasets like XNLI as tools for benchmarking the cross-lingual capabilities of a given model, then a translate-test approach, by converting the task at hand to a monolingual one, ends up being at most marginally relevant to cross-lingual studies. It may constitute a supplementary baseline but I fail to see how it could advance the field's understanding of how to develop efficient multilingual representations.\n- Perhaps a more fruitful line of thought would be to consider cross-lingual datasets like XNLI as engineering problem &mdash; some toy example of a real-world situation where large quantities of data are available in one language (English), but not in the lower-resource target language of interest. Under such a view, I assume that translate test might have an interest, but the paper in its current format does not present relevant point of comparisons: either the study should focus on low resource languages where no pretrained model is available, or the study should compare a translate-test baseline to target-language monolingual approaches, rather than the purely English-centric point of view.\n\nLong story short: I am uncertain what point the authors are trying to make. It seems the paper would benefit from a more explicit stance as to what is the intended reach of the proposed approach, and how relevant it is for the intended readership.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on cross-lingual classification, and demonstrates how machine translation approaches can prove very effective. In particular, the authors argue that translating the test set into English and using English monolingual PLMs can constitute a much stronger baseline than previously thought. Lastly, the authors also introduce a methodology for studying cross-lingual transfer gaps.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yCdd5V8x4U", "reviewer": "Reviewer_S3YL", "summary": "", "questions": "- For the MT engines, do you train a single adapted MT per dataset (e.g. xnli) or together for all six datasets", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful and encouraging review. We address the main points raised next:\n\n> For the MT engines, do you train a single adapted MT per dataset (e.g. xnli) or together for all six datasets\n\nWe train a separate adapted MT per dataset.\n\n> Missing references\n\nThanks for the suggested references. Our current version of the related work section focuses on recent work that is more directly relevant to our work, but we will extend it to give a broader perspective, covering the papers suggested by the reviewer.\n\n> I think a figure on the different ways you generated data for MT would be helpful\n\nThanks for the suggestion, which we will consider for the camera ready version.\n\n> I'm not sure if \"doc-level\" is a good term because in MT it refers to a real document involving many sentences. How about pair-level or something that emphasizes the NLI aspect?\n\nWe agree that, in the MT literature, “doc-level” usually refers to a real document involving many sentences. In our case, we have various fields in some of the tasks (e.g., the premise and hypothesis in NLI), but in some instances each of these fields can contain more than one sentence too. For instance, MARC consists of Amazon reviews, which typically contain many sentences. For the baseline system, we run sentence segmentation and translate each of these sentences separately, whereas for the doc-level variant we translate the entire input as a whole, which can involve both more than one field and more than one sentence. For that reason, we believe that “pair-level” would not be an appropriate term. Having said that, we understand that this is not entirely clear in the current version of the paper, and we will clarify it for the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thought-provoking claims on the relative strengths of train-test and importance of multilingual pretraining. \n- Nice small contribution to better MT for XNLI like tasks\n- Well-designed methodology and clear writing that makes this \"revisitation\" paper an interesting read\n- Exemplar analysis framework ", "reasons_to_reject": "None", "questions_to_authors": "- For the MT engines, do you train a single adapted MT per dataset (e.g. xnli) or together for all six datasets", "paper_topic_and_main_contributions": "This paper re-examines cross-lingual classification for the XNLI task by using stronger MT systems. In particular, it revisits the discussion about the relative merits of the translate-test, translate-train, and zero-shot approaches. One important conclusion, in my opinion, is that the translate-test approach is more competitive than previously thought; while I'm not sure if this was the common sentiment in the field, in any case it suggests that stronger MT could be helpful and the multilingual pretraining of XNLI models may be less important. This suggestion by itself is a thought-provoking idea and I think it benefits the field to have \"revisiting\" papers like this. \n\nGenerally, I think the paper is written very clearly. The approach is very methodological, and each experimental design decision is justified. The main contributions are (1) the stronger MT engines, including the doc-level and roundtrip adaptation, (2) the main results in Table 4, and (3) the analysis in Section 4. \n\nI did not get so much out of the results of Section 4 (i.e. Fig 2 is challenging to interpret and some might draw different conclusions from yours), though I do very much like the way the authors approach different kinds of degradation (Sec 4.2). I would give this to new PhD students as a good reference for how to think about error analysis. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- \"Is Machine Translation Ripe for Sentiment Analysis\", ACL 2011, http://www.aclweb.org/anthology/P11-2075 - this is an older paper that focuses on cross-lingual sentiment classification, but it also investigates both MT and domain issues like you did in this paper. It may be interesting to refer to. \n- There have been much discussions on \"translate-test\" vs \"translate-train\" in the context of Cross-Language Information Retrieval. It is a very different problem than NLI or classification, but just for your reference. See for example the book by Nie: https://link.springer.com/book/10.1007/978-3-031-02138-1 \n\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- I think a figure on the different ways you generated data for MT would be helpful. It is relatively clear from the text (Sec 3.1.1 and 3.1.2), but it took me a few readings. Specifically, a figure indicating examples of how sentences are backtranslated, then forward-translated, and which parts are used for which adaptation system would be helpful.  \n- I'm not sure if \"doc-level\" is a good term because in MT it refers to a real document involving many sentences. How about pair-level or something that emphasizes the NLI aspect? "}}, {"review_id": "mBCeLYVLPE", "reviewer": "Reviewer_chth", "summary": "", "questions": "Question A: Could you explain the meaning of the negative values in Figure 2 for MT distribution shift and Representation misalignment?\nQuestion B: Could you explain the motivation behind the formulas depicted for MT distribution shift, Target representation quality and Representation misalignment?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review, we next address the questions that were asked:\n\n> The analysis of the variance across tasks (section 4) lacks theoretical motivation [...] and performs an oversimplification by assuming that the impact of the different sources of degradation can be directly summed up\n\nWe agree that there are simplifying assumptions in our analysis. In fact, we intentionally use the term “estimate” throughout the entire section, as we are approximating the impact of each factor rather than computing a ground truth value. Having said that, we believe that such simplifying assumptions are unavoidable in this type of analysis. We think that the methodology we propose is sensible and can provide interesting insights despite the (unavoidable) simplification. To give an example, we do not claim that the degradation from MT can be exactly computed using the formulas we propose, but our analysis does provide strong evidence that some tasks like OpenBookQA are much more sensitive to this type of degradation than others like XStoryCloze. We will clarify this point in the final version of the paper.\n\n> Could you explain the meaning of the negative values in Figure 2 for MT distribution shift and Representation misalignment?\n\nThe values we report should be taken as an estimate or approximation rather than an exact ground truth value for various reasons. First of all, our framework makes some simplifying assumptions as rightfully pointed out by the reviewer and discussed in the previous point. In addition, we use a finite sample to estimate these values, so there is some variance. While it is true that we obtain a few negative values, they are all small, and we think that they can be attributed to such noise.\n\n> Could you explain the motivation behind the formulas depicted for MT distribution shift, Target representation quality and Representation misalignment?\n\nThe MT distribution shift accounts for the mismatch between the train and test distribution induced by MT. For that reason, we take the difference in the back-translated evaluation accuracy between training on original data (so there is a distribution shift between train and test induced by MT) and training on back-translated data (so there is no distribution shift between train and test induced by MT). MT is used twice in back-translation, so we divide the resulting value by two to estimate the impact of a single step.\n\nFor the target representation quality, we take the difference between training and testing in the source language and training and testing in the target language. Say we obtain an accuracy of 80% in English and 70% in Swahili. The resulting difference of 10% can be attributed to the Swahili representations being worse than the English ones, which is what we refer to as the target representation quality. However, given that the target language data is generated through MT, which can introduce some errors due to the information lost in the translation process, we further subtract this value as estimated separately.\n\nRegarding the representation misalignment, we take the difference in the target language accuracy between training on the target language itself and training on the source language. Say we obtain an accuracy of 70% in Swahili when the training data is also in Swahili, and an accuracy of 65% when the training data is in English. The resulting difference of 5% can be attributed to the misalignment between English and Swahili in the multilingual model or, in other words, the degradation from generalizing from English to Swahili. However, given that the target language data is generated through MT, but the source language training data is not, we further need to subtract the value from the MT distribution shift as estimated separately.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contains very useful advice for building text classification systems for low-resource languages.", "reasons_to_reject": "The analysis of the variance across tasks (section 4) lacks theoretical motivation. The analysis leverages the concept of cross-lingual transfer gap, i.e., comparing the classification performance between English and the target language, and performs an oversimplification by assuming that the impact of the different sources of degradation can be directly summed up. Unless it is empirically proved in some way, it is safe to think that interactions between the different factors that degrade quality during cross-lingual transfer is not so simple. Some of the sources of degradation have a negative value as depicted in Figure 2: this fact is not explained by the authors and casts doubt on whether the way of calculating the impact of each factor is appropriate.", "questions_to_authors": "Question A: Could you explain the meaning of the negative values in Figure 2 for MT distribution shift and Representation misalignment?\nQuestion B: Could you explain the motivation behind the formulas depicted for MT distribution shift, Target representation quality and Representation misalignment?", "paper_topic_and_main_contributions": "The paper assesses the role of MT in cross-lingual classification by comparing the two most prominent ways of using it: translating the training data into the target language and finetuning the classification model on machine-translated target language data (“translate-train”), or translating the test set into English so as to use a classification model finetuned on English data (“translate-test”). \n\nBy means of systematic experiments, the paper shows, unlike previous works in the literature, that translate-test can outperform translate-train in many tasks provided that the output of the MT system is adapted to be similar to the English data used for finetuning, or the data used for finetuning resembles the output of the MT system.\n\nThe paper also tries to estimate where the loss of performance in classification comes from when moving to a language different from English, defining 5 categories and computing their influence by simple subtraction operations between model accuracies, obtaining different relative impacts of the categories for different tasks.\n\nThe main contributions are the following:\n- Re-assessment of the different strategies for using MT for cross-lingual classification\n- Definition of simple but effective methods for adapting MT for cross-lingual classification\n- Outlining the type of classification tasks that more suitable for each method (translate-train and translated-test)", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper finds that "translate-test" could be a stronger method than what people previously expected. The reviewers appreciate the comprehensive analysis on various classification tasks and the paper is well written. The paper could be further improved by provide more theoretical justification of the results. Moreover, it would be useful to clarify the languages included in the task in the paper. It might be interesting to understand whether this conclusion still holds for very under-represented languages other than those studied in the paper, or if it still holds for tasks other than classification.
Decision: Accept-Main