Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution
Year: 2023

Abstract:
Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language. Although the zero-shot cross-lingual transfer approach has achieved success in various classification tasks (Wu and Dredze, 2019), its performance on natural language generation tasks falls short in quality (Rönnqvist et al., 2019; Vu et al., 2022) and sometimes outputs an incorrect language (Xue et al., 2021). In our study, we show that the fine-tuning process learns language invariant representations, which is beneficial for classification tasks but harmful for generation tasks. Motivated by this, we propose a simple method to regularize the model from learning language invariant representations and a method to select model checkpoints without a development set in the target language, both resulting in better generation quality. Experiments on three semantically diverse generation tasks show that our method reduces the accidental translation problem by 68% and improves the ROUGE-L score (Lin, 2004) by 1.5 on average.

heading: 1 Introduction
text: Language Models (LMs) pre-trained on multilingual corpora (Devlin et al., 2019a; Conneau et al., 2020a; Liu et al., 2020; Xue et al., 2021) exhibit zero-shot cross-lingual transfer ability (Wu and Dredze, 2019). Given only annotated data in one language for a task, multilingual LMs are able to perform this task in languages seen only during the pre-training stage. The cross-lingual transferability of multilingual LMs reduces the need for annotated data in low-resource languages, which is valuable for building practical multilingual NLP systems.
Existing studies on cross-lingual transfer select tasks such as word alignment (Artetxe et al., 2020b), POS tagging (Pires et al., 2019), dependency parsing and sentence classification (Wu and Dredze, 2019) to investigate cross-lingual transferability of multilingual LMs (Hu et al., 2020),
and few works focus on cross-lingual transfer in generation tasks (Maurya et al., 2021; Maurya and Desarkar, 2022). Cross-lingual transfer approach in generation tasks are known to produce incoherent text (Rönnqvist et al., 2021), generate in a wrong language (Xue et al., 2021), and suffer from catastrophic forgetting (Vu et al., 2022). Table 1 illustrates a common problem where the multilingual LM generates text in an incorrect language. Moreover, such a problem becomes more severe when under a true zero-shot setting (Zhao et al., 2021; Schmidt et al., 2022), where we do not have annotated data in the target language to guide model selection.
We show that the reason why zero-shot crosslingual transfer in text generation fails is because the fine-tuning process learns language invariant representations, which is beneficial for classification tasks, but detrimental to generation tasks. In our paper, we use the cosine similarity between parallel sentence representations in different languages to measure the Cross-Lingual Representation Similarity (XLRS). We use a range of tasks from classification to extractive question answering, then to abstractive generation to show that in the best performing model, the XLRS after finetuning decreases as we move from classification to generation.
The fact that language invariant representations causes the degradation in generation tasks challenges the common belief that invariant representations generally enhance cross-lingual transfer on all downstream tasks (Cao et al., 2020; Conneau et al., 2020b; Yang et al., 2022; Xian et al., 2022). To the best of our knowledge, our work is the first to provide an analysis of how XLRS affects cross-lingual transfer in language generation tasks.
Motivated by our findings, we propose to use an auxiliary source language that implicitly regularizes the XLRS being too large and results in better generation performance over three complex natural
12461
language generation tasks (Summarization, Story Completion, and Title Generation). Under a true zero-shot setting, choosing the model checkpoint with the lowest XLRS results in an average of 4.1 point increase in ROUGE-L over using a source development set in two generation datasets.
To sum up, our contributions are threefold:
• We show that fine-tuning on a single source language increases the cosine similarity between sentence representations of different languages (XLRS).
• We show that the increase in XLRS causes degradation of cross-lingual transfer in generation tasks, and argue that the prevalent understanding of the benefit of similar representations does not apply to generation tasks.
• We empirically show that using two goldannotated source languages instead of one regularizes the XLRS, resulting in an average increase of 1.5 in ROUGE-L.

heading: 2 Related Works
text: Multilingual Language Models. One line of work is to train multilingual versions of modern Language Models. mBERT (Devlin et al., 2019b) is the multilingual version of BERT (Devlin et al., 2019a), which uses the same encoder-only model architecture but is only trained on multilingual corpora. XLM-R (Conneau et al., 2020a) is the multilingual version of RoBERTa (Liu et al., 2019), which implements multiple optimization tricks and is larger in scale, resulting in better performance than BERT. mBART (Liu et al., 2020) is the multilingual version of BART (Lewis et al., 2020), an encoder-decoder model trained to reconstruct the original text through various types of artificially introduced noises. mT5 (Xue et al., 2021) is the
multilingual version of T5 (Raffel et al., 2020), an encoder-decoder model trained on a span denoising objective.
Cross-lingual Transfer. Multilingual models are able to be fine-tuned on annotated data of a task in only one source language and transfer the knowledge to other target languages to perform the same task without any supervision. While Pires et al. (2019) states that sub-word overlap between source and target facilitates cross-lingual transfer, K et al. (2020) shows that cross-lingual transfer manifests in pairs of source and target with zero sub-word overlap and word order is instead the most crucial ingredient. The performance of cross-lingual transfer between languages with a different order severely drops. Although the importance of word order is echoed by later studies (Artetxe et al., 2020b; Dufter and Schütze, 2020), recent studies have also debated in favor of the importance of matching script also contributing to cross-lingual transfer (Lauscher et al., 2020; Fujinuma et al., 2022). Wu et al. (2022) points out that the optimal set of parameters that generalizes well to all languages is a subset of parameters that achieves good performance on the source language. Therefore it is hard to find the optimal zero-shot cross-lingual transfer parameters by only optimizing source language performance. Chen and Ritter (2021) train a scoring model with the input features being the model’s hidden representations and the output score being how well it generalizes to a given target language. However, previous studies focus on lower-level NLP tasks, which include text classification, dependency parsing, and extractive question answering (Hu et al., 2020) and rarely touch on language generation.
Another line of work focuses on applying crosslingual transfer to a wide range of multilingual
NLP applications, which include sequence tagging (Yang et al., 2016), Named Entity Recognition (Xie et al., 2018), dependency parsing (Ahmad et al., 2019), sentence classification (Conneau et al., 2018; Yang et al., 2019), and information retrieval (Izacard et al., 2022). Empirical studies also train ranking models (Lin et al., 2019), use meta-learning (Nooralahzadeh et al., 2020), or use Shapley Value (Parvez and Chang, 2021) to predict which sources perform the best for a given target language.
Natural Language Generation. Multilingual LMs are prone to produce text that is repetitive (Xu et al., 2022), contains hallucinations (Raunak et al., 2021), or is in the wrong language (Zhang et al., 2020; Xue et al., 2021; Vu et al., 2022). Vu et al., 2022 proposed to use parameter efficient finetuning methods (Lester et al., 2021; Qin and Eisner, 2021; Li and Liang, 2021) to regularize the model to generate in a desired language. Other ways to improve generation quality include using back translation (Gu et al., 2019; Zhang et al., 2020), and transliteration (Sun et al., 2022) as data augmentation techniques, mixing in the pretrain objective during fine-tuning (Xue et al., 2021) and using an auxiliary source language in machine translation (Xu et al., 2021). Two concurrent efforts are close to our work: Xu and Murray (2022) and Schmidt et al. (2022) both empirically show that using multiple languages during fine-tuning in few-shot crosslingual transfer improves performance in text classification. Our work differs in that we evaluated text generation under a true zero-shot setting, where we have access to neither a few examples to train on nor an annotated development set to guide model checkpoint selection.

heading: 3 Setup
text: The consensus of the literature (Cao et al., 2020; Conneau et al., 2020b; Tiyajamorn et al., 2021; Yang et al., 2022; Xian et al., 2022) is that if a model can produce similar representations for parallel sentences, the model would be able to achieve good cross-lingual transfer performance. Intuitively, if a model maps parallel sentences in English and French into nearly identical representations, and is able to predict the sentiment of the English sentence, it will also be able to predict the sentiment of the French sentence.
We hypothesize that the fine-tuning process in-
creases the similarity between sentence representations of different languages. We use the following setups and tasks to verify our hypothesis.

heading: 3.1 Models and Datasets
text: Models We select the state-of-the-art multilingual language model: mT5-base (Xue et al., 2021). We use the Huggingface (Wolf et al., 2020) implementation. We use a uniform learning rate of 7e-5, a batch size of 32 for 10 epochs for all tasks described below.
Datasets Table 2 describes the tasks we used in the following section to show the transition from classification to generation1. We use the UDPOS (Nivre et al., 2018) dataset containing sentences and the part-of-speech tag of each word. For sentence-level classification, we use the PAWS-X (Yang et al., 2019) dataset containing pairs of sentences and a binary tag on whether the second sentence entails the first sentence. For extractive generation, we use the TyDiQA-GoldP (Clark et al., 2020) dataset which contains paragraphs and questions whose answers are spans extracted from the paragraphs. For abstractive generation, we use the WikiLingua (Ladhak et al., 2020) dataset, which contains WikiHow instructions and their summaries in 17 languages.
We use the story completion (SG) and title generation (TG) task in the MTG benchmark (Chen et al., 2022), a recently introduced benchmark to evaluate multilingual text generation. We follow Vu et al., 2022, which uses the WikiLingua dataset to construct WikiLingua-0, where the model is only finetuned on English and evaluated on other languages. We extend WikiLingua-0 and use languages besides English as the source and evaluate the zero-shot directions.
In all of our experiments, we report the results averaged across three runs with different random seeds. For each source language, we only use the
1We follow (Vu et al., 2022) and report the SP-ROUGE score.
top 10k training examples to train our model to ablate the effect of training data size on cross-lingual transfer. Unless specified otherwise, we evaluate under a true zero-shot setting, where we select the model checkpoint based on its performance on a dev set of the source language.

heading: 3.2 Sequence to Sequence Learning
text: We cast sequence labeling and sentence classification tasks into a text-to-text format using templates described in Table 10 in the Appendix. We follow Raman et al. (2022) and cast sequence labeling tasks into the sentinel + tag format. We follow Schick and Schütze (2021) and cast the sentence entailment task into a cloze question, supervising the model to predict the word "yes" for entailment and the word "no" for non-entailment.

heading: 4 Learning Dynamics of Cross-lingual Transfer
text: We plot the average cosine similarity between representations of parallel sentences (XLRS)2 for each training iteration in two classification tasks: POS tagging and paraphrase identification (PAWS-X) at Figure 1 and Figure 2, respectively.
In both tasks, the plot displays an increasing trend of XLRS between parallel sentences between English and all the other languages. Notably, languages that have the same script have a higher similarity. Our findings show that the fine-tuning process on classification tasks does make the sentence representations of different languages more similar.
We then plotted the XLRS between representations of parallel sentences of a model when finetuned on WikiLingua: a summarization dataset in Figure 3. The average similarity gradually increases as we progress further into the training iterations, confirming our hypothesis that fine-tuning on a single source language increases the XLRS between the source and other languages.
Based on our findings, we conjecture that the model jointly minimizes two metrics, resulting in cross-lingual transfer:
• The Cross-Entropy loss between the predicted labels and the ground-truth labels, given an input in the source language (the standard training objective).
2We use the mean-pooled token encoder hidden states as the sentence representation. We randomly sample 512 sentences from the test set of the MTG story completion task.
• The distance between parallel sentences of the source and target languages (increase in XLRS).
And as a result, the cross-entropy loss between the predicted and ground-truth labels, given a context in the target language, is minimized, enabling cross-lingual transfer.

heading: 5 Unified View of Tasks
text: With the intuition of how the model does crosslingual transfer in classification tasks, we note that language generation is a classification on a large vocabulary set, rather than a small label set. Thus, we point out that the reason why good performance
with cross-lingual transfer on generation tasks is harder to achieve is actually caused by increasing XLRS.
Figure 4 illustrates our intuition. In classification tasks, the model needs to map parallel sentences to the same label. Ideally, the model produces identical representations for parallel sentences, resulting in the highest possible XLRS of 1. This is why model cross-lingual transfers better with a high XLRS. However, in generation tasks, if we view it as classifying over the entire vocabulary set, we are mapping parallel sentences to different labels. In an extreme case when XLRS is 1, the model fails to identify the source language, resulting in the common problem of the model producing an incorrect language (Xue et al., 2021). We introduce the notion of label overlap, d, to measure the percentage of examples in a dataset where the model needs to map parallel sentences into the same label.
We use C to denote the set of all discrete contexts and Cs to denote the set of all discrete contexts in language s. In classification tasks, the model learns to predict the ground-truth label ŷ = l(c) over a set of candidate labels Y for context c ∈ C. Similarly, in a simplified view of generation, the model learns to predict the next word v given context c. Therefore, we can essentially view language generation as classification where the label set is the entire vocabulary. In both cases, given context c, the model learns a probability distribution p·|c. The only difference is between their classification label set cardinality.
We define cross-lingual label overlap as an indicator of difficulty to cross-lingual transfer for
a given task at §5.1. We then use a range of tasks: word-level classification (POS tagging §5.2) - Sentence level classification (Entailment classification §5.3) - Span Extraction (Extractive Generation §5.4) - Summarization (Abstractive generation §5.5) to show an increasing level of difficulty to perform cross-lingual transfer.

heading: 5.1 Cross-lingual Label Overlap
text: We denote a task’s difficulty in transferring knowledge from one language to another by the percentage of overlap of their label set for parallel sentences. Given n parallel sentences {c1s, c2s, ..., cns } and {c1t , c2t , ..., cnt } in source language s and target language t, the cross-lingual overlap d for task α is defined as:
dα(s, t) =
∑n i=1 1(lα(c i s) = lα(c i t))
n
In our analysis, we use English as the source language and evaluate the difficulty of performing cross-lingual transfer on other languages. The total label overlap for each task is the average label overlap for each target language.
dα =
m∑
j=1
dα(english, tj)
Where tj is the jth target language. A higher dα indicates an easier task to transfer knowledge from one language to another, whereas a lower dα indicates a more difficult task to crosslingual transfer.

heading: 5.2 POS tagging
text: The word-level label overlap for part-of-speech tagging should be close to 100%. With such a high percentage of label overlap, the model benefits from producing identical representations for parallel sentences to predict the same labels for different languages without supervision of the target language.
For example, if the model maps the English sentence the dog ran and the french sentence le chien courir into nearly identical representations and simultaneously learns a function to map the English words to their respective POS tags "DET NOUN VERB", the model would also be able to predict the correct label for french even without supervision. We denote the amount of "label overlap" as a metric defining the difficulty for a model to perform cross-lingual transfer on it.

heading: 5.3 Sentence Classification
text: The classification task discussed in this section (PAWS-X) includes sentiment classification of a single sentence and entailment classification between two sentences. For semantically equivalent parallel sentences, their sentiment or entailment labels are always the same. Therefore, d = 100%.
Ideally, in sentence classification tasks, parallel sentences in different languages should map to the same probability distribution. For example, if the English sentence I am happy and the french sentence Je suis content maps to nearly identical representations and the model learns to predict the sentiment in English, the model would be able to cross-lingual transfer the ability to predict sentiment in English to French without any supervision.

heading: 5.4 Span Extraction
text: Span extraction requires a model to select a correct answer span from a passage given a question. Even though the data in TyDiQA is in different languages, and not parallel, 16% of the answer spans are pure numbers, and 50.6% of answer spans are mainly composed of numbers (Time and Dates).
This indicates that span extraction is a harder task than sentence classification, but with such a high amount of label overlap, the task is solvable through cross-lingual transfer.

heading: 5.5 Generation
text: The amount of label overlap in abstractive generation tasks (e.g. summarization, story completion,
title generation) is close to zero as the model needs to predict words in completely different languages. The amount of label overlap for a subset of five languages (En, De, Fr, Es, Zh) of the WikiLingua (Ladhak et al., 2020) dataset is d = 0.13%3.
In a generation task, if the model maps the source and the target into identical representations, the model predicts the next word to be the same. Even if this is correct in semantics and possibly results in the code-switched results as shown in Figure 1, the model fails to generate in the correct language.

heading: 5.6 Analysis
text: We plotted the XLRS between English and four different languages in the best-performing English supervised models for the four tasks and the pretrained model at Figure 5.
The plot confirms our belief that for tasks (POS tagging, PAWS-X, TyDiQA) with large label overlap, the model cross-lingual transfers from increasing XLRS, whereas in generation tasks with label overlap close to zero (title generation), the bestperforming model has a lower XLRS.
Following Yang et al. (2022), we calculate the Spearman’s rank correlation score between (a) XLRS between English and 4 target languages (German, French, Chinese, Spanish), and (b) The averaged zero-shot cross-lingual transfer performance in each task. The results are reported at Table 4. In both classification tasks, XLRS positively correlates with cross-lingual performance. In contrast, in our three generation tasks, XLRS negatively correlates with cross-lingual performance4,
3This small percentage represents mainly numbers and named entities (i.e. cities) that are the same across languages.
4We observed a stronger negative correlation between cosine similarity and ROUGE-2 in generation tasks but opted to report the ROUGE-L results to be consistent with our main results.
indicating that XLRS is strongly correlated to transfer performance in classification tasks but is detrimental to cross-lingual transfer in generation tasks.

heading: 6 Text Generation Experiments
text: Now that we know XLRS is negatively correlated with cross-lingual transfer in text generation, since calculating XLRS during every iteration is computationally expensive, we wonder if we can regularize XLRS implicitly. Motivated by using auxiliary source languages improves machine translation (Xu et al., 2021) and few-shot cross-lingual transfer (Xu and Murray, 2022; Schmidt et al., 2022),
we propose to use an additional source language to regularize XLRS.
To verify our hypothesis, we plot the XLRS between English and French during training on two different sources (En, De)in the story completion task, compared to training only on one source (En) in Figure 6. We observe that when the model is only given one language as the source, the XLRS increases, whereas using two source languages allows the model to learn to control the XLRS from being too high, resulting in fewer accidental translations and better quality.
To show that regularizing XLRS does result in better generation quality, we experiment with three semantically diverse generation tasks: Summarization (WikiLingua), Title Generation (TG), and Story Completion (SG).

heading: 6.1 Results
text: Table 3 shows the results of fine-tuning with multiple languages. We observe that adding Chinese to English data improves the performance in 13 out of 15 zero-shot directions5 compared to only using English. We point out that our improvement is not due to an increase in the amount of training data since we used the same amount of training data for all experiments. We further observe that adding Chinese as an additional language to German also improves the performance in all 14 zero-shot directions, which often results in the best zero-shot performance.
Table 5 and 6 show the ROUGE-L results in the title generation (TG) and story completion (SG) in the MTG (Chen et al., 2022) benchmark, respectively. Again, we are able to observe that using two source languages almost always improves the ROUGE-L score. Notably, using two related
5The results when the target language are Chinese (ZH) or English (EN) is not zero-shot.
languages often results in degraded performance than using two unrelated languages with different scripts. We hypothesize that a language with a different script and order provides a more substantial regularization effect, preventing the cosine similarity between the source and target sentence representations from being too high.
To verify that our method helps against the accidental translation problem, we follow previous work (Vu et al., 2022) and calculate the language id confidence score on the source language and target language on the title generation task. The results are shown at Table 9 in Appendix A. Fine-tuning with multiple source languages helps the model learn which language it should produce.

heading: 6.2 Model Selection Using Parallel Sentences
text: Since XLRS negatively correlates with the performance of cross-lingual generation, we use it as a criterion for model selection in the absence of an annotated dev set. We report the performance on the WikiLingua dataset and the story completion task in MTG benchmark at Figure 7 and 8, when selecting the model using English dev set performance (en-dev), selecting the model with the lowest XLRS between English and the target lan-
guage (cos-sim), and selecting the model using an annotated dev set on each target language (tgt-dev), which serves as an upper bound for true zero-shot cross-lingual transfer.
In both tasks, selecting the model checkpoint with the lowest XLRS results in better performance than using an English development set on all target languages. The performance is on average less than one ROUGE-L point less on Spanish and German in both datasets, compared to using an annotated dev set. Our method results in an average increase of 5 ROUGE-L points in a distant language (Chinese).

heading: 7 Conclusion
text: We show that multilingual LMs transfer supervision from one language to another by increasing Cross-Lingual Representation Similarity (XLRS). Such a learning process results in decent zero-shot cross-lingual transfer performance in classification tasks but is harmful to text generation performance . We demonstrate that regularizing XLRS improves text generation quality and use parallel sentences to guide model selection without annotated data in the target languages. We believe that this is valuable under a practical setting (Artetxe et al., 2020c) where we have access to parallel data between the source and target languages, but not task-specific data in the target language.
Limitations
Our work sheds light on understanding the training dynamics of cross-lingual transfer learning of multilingual LMs. In our work, we selected to use English as the source of cross-lingual transfer following previous work (Vu et al., 2022). We acknowledge that using other languages as the source language can provide benefits depending on the task (Lin et al., 2019; Turc et al., 2021). Our work does not focus on choosing source language to maximize downstream performance but instead focuses on the difference between classification tasks and generation tasks in cross-lingual transfer.
Secondly, we acknowledge that some of the datasets (Yang et al., 2019; Chen et al., 2022) used in our work are created by machine translation and human annotation. Previous studies have pointed out that translationese in datasets affects cross-lingual transfer performance (Artetxe et al., 2020a; Artetxe et al., 2020c). We believe that translationese in datasets also have impact on XLRS. We leave the study of how dataset features (size, quality, translationese) affect cross-lingual transfer for future work.

heading: Acknowledgements
text: We sincerely thank Haoran Xu, Kelly Marchisio and Daniel Khashabi for their helpful suggestions.

heading: A Language Identification Scores
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, in the section "Limitations"
7 A2. Did you discuss any potential risks of your work? We believe that our study is an Engineering study and is not applicable to this question.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes. Under the "Abstract" and "Introduction"
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.1 Models and Datasets

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.1 Models and Datasets
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets we used are very common open-sourced datasets. This information can be found in the citations we provided.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.1
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we used are very common open-sourced datasets.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No. Since the datasets we used are very popular open-sourced datasets, we believe that
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.1
C 3 Did you run computational experiments? Yes. In Section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We did report the number of parameters. But we did not discuss the computational budget because the experiment we do are fine-tuning, which should be fairly lightweight.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We stated that we report the mean result averaged across three runs. But we did not provide error bars.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the study of how dataset features (size, quality, translationese) affect cross-lingual transfer for future work