prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 1: Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3   Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4   Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios 
However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child)  On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs  However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En)  Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9  
In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor  Furthermore, the amount of indirect in-domain parallel corpora, i e , Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small  As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT  To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9  
We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows:
To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it 
Our Japanese–Russian Setting
In this paper, we deal with Ja INLINEFORM0 Ru news translation  This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax  Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers  Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences  To establish and evaluate existing methods, we also involved English as the third language  As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10  
There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation  Therefore, we manually compiled development and test sets using News Commentary data as a source  Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data  Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line  Note that we have never merged two or more lines  As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8   Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets 
Our manually aligned development and test sets are publicly available 
Related Work
koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT  Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available  This restricts one of the source or the target languages to be English which, in our case, is not possible  Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11   However, like most other, this work focuses on translation from and into English 
Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10   The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings  Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings  Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons  Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT  However, this approach requires base MT systems that can generate somewhat accurate translations  It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation  MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages  This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains  Section SECREF4 empirically confirms the limit of these existing approaches 
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9   In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods  By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality 
Limit of Using only In-domain Data
This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data  We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT  Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora 
Data
To train MT systems among the three languages, i e , Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS  Table TABREF9 summarizes the size of train/development/test splits used in our experiments  The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k  Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations  This puts a limit on the positive impact that the helping corpora can have on the translation quality 
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language  Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4   Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain  Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus 
https://dumps wikimedia org/backup-index html (20180501) http://www statmt org/wmt18/translation-task html https://www yomiuri co jp/database/glossary/ http://www cs jhu edu/~kevinduh/a/multitarget-tedtalks/ http://opus nlpl eu/Tatoeba-v2 php
We tokenized English and Russian sentences using tokenizer perl of Moses BIBREF3   To tokenize Japanese sentences, we used MeCab with the IPA dictionary  After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages 
MT Methods Examined
We began with evaluating standard MT paradigms, i e , PBSMT BIBREF3 and NMT BIBREF1   As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14  
As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18   In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8  
After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17  
First, we built a PBSMT system for each of the six translation directions  We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction  Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model  We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13  
Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja 
2-step decoding using the source-to-English and English-to-target systems 
Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system 
Compile a new phrase table combining those for the source-to-English and English-to-target systems 
Among these three, triangulation is the most computationally expensive method  Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs  To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07  For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7  
We also investigated the utility of recent advances in unsupervised MT  Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons  We therefore followed another method described in marie:usmt-unmt  Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt  To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words  For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap  For each of the retained phrase pair, INLINEFORM5 was computed analogously  We also computed lexical translation probabilities relying on those learned from the given small parallel corpus 
Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses  Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations  Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3   In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4  
We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor  A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus  Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8  
Table TABREF22 contains some specific hyper-parameters for our baseline NMT models  The hyper-parameters not mentioned in this table used the default values in tensor2tensor  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 
Context 2: 10pt
1 10pt
[ Characterizing Political Fake News in Twitter by its Meta-DataJulio Amador Díaz LópezAxel Oehmichen Miguel Molina-Solana( j amador, axelfrancois oehmichen11, mmolinas@imperial ac uk ) Imperial College London This article presents a preliminary approach towards characterizing political fake news on Twitter through the analysis of their meta-data  In particular, we focus on more than 1 5M tweets collected on the day of the election of Donald Trump as 45th president of the United States of America  We use the meta-data embedded within those tweets in order to look for differences between tweets containing fake news and tweets not containing them  Specifically, we perform our analysis only on tweets that went viral, by studying proxies for users' exposure to the tweets, by characterizing accounts spreading fake news, and by looking at their polarization  We found significant differences on the distribution of followers, the number of URLs on tweets, and the verification of the users 
]
Introduction
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e g  it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0   Since then, governments and corporations alike (e g  Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3   Yet, the ability to define, identify and stop fake news from spreading is limited 
Since the Obama campaign in 2008, social media has been pervasive in the political arena in the United States  Studies report that up to 62% of American adults receive their news from social media BIBREF4   The wide use of platforms such as Twitter and Facebook has facilitated the diffusion of fake news by simplifying the process of receiving content with no significant third party filtering, fact-checking or editorial judgement  Such characteristics make these platforms suitable means for sharing news that, disguised as legit ones, try to confuse readers 
Such use and their prominent rise has been confirmed by Craig Silverman, a Canadian journalist who is a prominent figure on fake news BIBREF5 : “In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlet” 
Our current research hence departs from the assumption that social media is a conduit for fake news and asks the question of whether fake news (as spam was some years ago) can be identified, modelled and eventually blocked  In order to do so, we use a sample of more that 1 5M tweets collected on November 8th 2016 —election day in the United States— with the goal of identifying features that tweets containing fake news are likely to have  As such, our paper aims to provide a preliminary characterization of fake news in Twitter by looking into meta-data embedded in tweets  Considering meta-data as a relevant factor of analysis is in line with findings reported by Morris et al  BIBREF6   We argue that understanding differences between tweets containing fake news and regular tweets will allow researchers to design mechanisms to block fake news in Twitter 
Specifically, our goals are: 1) compare the characteristics of tweets labelled as containing fake news to tweets labelled as not containing them, 2) characterize, through their meta-data, viral tweets containing fake news and the accounts from which they originated, and 3) determine the extent to which tweets containing fake news expressed polarized political views 
For our study, we used the number of retweets to single-out those that went viral within our sample  Tweets within that subset (viral tweets hereafter) are varied and relate to different topics  We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al  BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious  The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties 
From our results, the following main observations can be made:
Our findings resonate with similar work done on fake news such as the one from Allcot and Gentzkow BIBREF9   Therefore, even if our study is a preliminary attempt at characterizing fake news on Twitter using only their meta-data, our results provide external validity to previous research  Moreover, our work not only stresses the importance of using meta-data, but also underscores which parameters may be useful to identify fake news on Twitter 
The rest of the paper is organized as follows  The next section briefly discusses where this work is located within the literature on fake news and contextualizes the type of fake news we are studying  Then, we present our hypotheses, the data, and the methodology we follow  Finally, we present our findings, conclusions of this study, and future lines of work 
Defining Fake news
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news  In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news  On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news  On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news 
With respect to social sciences, efforts from psychology, political science and sociology, have been dedicated to understand why people consume and/or believe misinformation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14   Most of these studies consistently reported that psychological biases such as priming effects and confirmation bias play an important role in people ability to discern misinformation 
In relation to the production and distribution of fake news, a recent paper in the field of Economics BIBREF9 found that most fake news sites use names that resemble those of legitimate organizations, and that sites supplying fake news tend to be short-lived  These authors also noticed that fake news items are more likely shared than legitimate articles coming from trusted sources, and they tend to exhibit a larger level of polarization 
The conceptual issue of how to define fake news is a serious and unresolved issue  As the focus of our work is not attempting to offer light on this, we will rely on work by other authors to describe what we consider as fake news  In particular, we use the categorization provided by Rubin et al  BIBREF7   The five categories they described, together with illustrative examples from our dataset, are as follows:
Research Hypotheses
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news  Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Taking those three dimensions into account, we propose the following hypotheses about the features that we believe can help to identify tweets containing fake news from those not containing them  They will be later tested over our collected dataset 
Exposure 
Characterization 
Polarization 
Data and Methodology
For this study, we collected publicly available tweets using Twitter's public API  Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data  Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets  Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016  Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton  The data collection ran for just one day (Nov 8th 2016) 
One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers  Among the reasons for retweeting, Body et al  BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others  As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news)  For our study, we consider that a tweet went viral if it was retweeted more than 1000 times 
Once we have the dataset of viral tweets, we eliminated duplicates (some of the tweets were collected several times because they had several handles) and an expert manually inspected the text field within the tweets to label them as containing fake news, or not containing them (according to the characterization presented before)  This annotated dataset BIBREF8 is publicly available and can be freely reused 
Finally, we use the following fields within tweets (from the ones returned by Twitter's API) to compare their distributions and look for differences between viral tweets containing fake news and viral tweets not containing fake news:
In the following section, we provide graphical descriptions of the distribution of each of the identified attributes for the two sets of tweets (those labelled as containing fake news and those labelled as not containing them)  Where appropriate, we normalized and/or took logarithms of the data for better representation  To gain a better understanding of the significance of those differences, we use the Kolmogorov-Smirnov test with the null hypothesis that both distributions are equal 
Results
The sample collected consisted on 1 785 855 tweets published by 848 196 different users  Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users  Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window 
The 1327 `viral' tweets were manually annotated as containing fake news or not  The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset  Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'  Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization  Because of this, we do not claim that this dataset can be considered a ground truth 
The following results detail characteristics of these tweets along the previously mentioned dimensions  Table TABREF23 reports the actual differences (together with their associated p-values) of the distributions of viral tweets containing fake news and viral tweets not containing them for every variable considered 
Exposure
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently  As such, Twitter users were exposed to fake news related to the election for a shorter period of time 
However, in terms of retweets, Figure FIGREF25 shows no apparent difference between containing fake news or not containing them  That is confirmed by the Kolmogorov-Smirnoff test, which does not discard the hypothesis that the associated distributions are equal 
In relation to the number of favourites, users that generated at least a viral tweet containing fake news appear to have, on average, less favourites than users that do not generate them  Figure FIGREF26 shows the distribution of favourites  Despite the apparent visual differences, the difference are not statistically significant 
Finally, the number of hashtags used in viral fake news appears to be larger than those in other viral tweets  Figure FIGREF27 shows the density distribution of the number of hashtags used  However, once again, we were not able to find any statistical difference between the average number of hashtags in a viral tweet and the average number of hashtags in viral fake news 
Characterization
We found that 82 users within our sample were spreading fake news (i e  they produced at least one tweet which was labelled as fake news)  Out of those, 34 had verified accounts, and the rest were unverified  From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted  Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs  not containing fake news)  From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts 
Turning to friends, accounts distributing fake news appear to have, on average, the same number of friends than those distributing tweets with no fake news  However, the density distribution of friends from the accounts (Figure FIGREF29 ) shows that there is indeed a statistically significant difference in their distributions 
If we take into consideration the number of followers, accounts generating viral tweets with fake news do have a very different distribution on this dimension, compared to those accounts generating viral tweets with no fake news (see Figure FIGREF30 )  In fact, such differences are statistically significant 
A useful representation for friends and followers is the ratio between friends/followers  Figures FIGREF31 and FIGREF32 show this representation  Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers  The distribution of those accounts not generating fake news is more evenly distributed 
With respect to the number of mentions, Figure FIGREF33 shows that viral tweets labelled as containing fake news appear to use mentions to other users less frequently than viral tweets not containing fake news  In other words, tweets containing fake news mostly contain 1 mention, whereas other tweets tend to have two)  Such differences are statistically significant 
The analysis (Figure FIGREF34 ) of the presence of media in the tweets in our dataset shows that tweets labelled as not containing fake news appear to present more media elements than those labelled as fake news  However, the difference is not statistically significant 
On the other hand, Figure FIGREF35 shows that viral tweets containing fake news appear to include more URLs to other sites than viral tweets that do not contain fake news  In fact, the difference between the two distributions is statistically significant (assuming INLINEFORM0 ) 
Polarization
Finally, manual inspection of the text field of those viral tweets labelled as containing fake news shows that 117 of such tweets expressed support for Donald Trump, while only 8 supported Hillary Clinton  The remaining tweets contained fake news related to other topics, not expressing support for any of the candidates 
Discussion
As a summary, and constrained by our existing dataset, we made the following observations regarding differences between viral tweets labelled as containing fake news and viral tweets labelled as not containing them:
These findings (related to our initial hypothesis in Table TABREF44 ) clearly suggest that there are specific pieces of meta-data about tweets that may allow the identification of fake news  One such parameter is the time of exposure  Viral tweets containing fake news are shorter-lived than those containing other type of content  This notion seems to resonate with our findings showing that a number of accounts spreading fake news have already been deleted or suspended by Twitter by the time of writing  If one considers that researchers using different data have found similar results BIBREF9 , it appears that the lifetime of accounts, together with the age of the questioned viral content could be useful to identify fake news  In the light of this finding, accounts newly created should probably put under higher scrutiny than older ones  This in fact, would be a nice a-priori bias for a Bayesian classifier 
Accounts spreading fake news appear to have a larger proportion of friends/followers (i e  they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only  Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter  Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising) 
Finally, it is interesting to notice that the content of viral fake news was highly polarized  This finding is also in line with those of Alcott et al  BIBREF9   This feature suggests that textual sentiment analysis of the content of tweets (as most researchers do), together with the above mentioned parameters from meta-data, may prove useful for identifying fake news 
Conclusions
With the election of Donald Trump as President of the United States, the concept of fake news has become a broadly-known phenomenon that is getting tremendous attention from governments and media companies  We have presented a preliminary study on the meta-data of a publicly available dataset of tweets that became viral during the day of the 2016 US presidential election  Our aim is to advance the understanding of which features might be characteristic of viral tweets containing fake news in comparison with viral tweets without fake news 
We believe that the only way to automatically identify those deceitful tweets (i e  containing fake news) is by actually understanding and modelling them  Only then, the automation of the processes of tagging and blocking these tweets can be successfully performed  In the same way that spam was fought, we anticipate fake news will suffer a similar evolution, with social platforms implementing tools to deal with them  With most works so far focusing on the actual content of the tweets, ours is a novel attempt from a different, but also complementary, angle 
Within the used dataset, we found there are differences around exposure, characteristics of accounts spreading fake news and the tone of the content  Those findings suggest that it is indeed possible to model and automatically detect fake news  We plan to replicate and validate our experiments in an extended sample of tweets (until 4 months after the US election), and tests the predictive power of the features we found relevant within our sample 
Author Disclosure Statement
No competing financial interest exist 
Context 3: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1

ground_truth: pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17

M2M Transformer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is their definition of tweets going viral?
Context 3: 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them

ground_truth: Viral tweets are the ones that are retweeted more than 1000 times

those that contain a high number of retweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 3:  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish