prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What language model architectures are used?
Context 1: Introduction
Pre-training of language models has been shown to provide large improvements for a range of language understanding tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   The key idea is to train a large generative model on vast corpora and use the resulting representations on tasks for which only limited amounts of labeled data is available  Pre-training of sequence to sequence models has been previously investigated for text classification BIBREF4 but not for text generation  In neural machine translation, there has been work on transferring representations from high-resource language pairs to low-resource settings BIBREF5  
In this paper, we apply pre-trained representations from language models to language generation tasks that can be modeled by sequence to sequence architectures  Previous work on integrating language models with sequence to sequence models focused on the decoder network and added language model representations right before the output of the decoder BIBREF6   We extend their study by investigating several other strategies such as inputting ELMo-style representations BIBREF0 or fine-tuning the language model (§ SECREF2 ) 
Our experiments rely on strong transformer-based language models trained on up to six billion tokens (§ SECREF3 )  We present a detailed study of various strategies in different simulated labeled training data scenarios and observe the largest improvements in low-resource settings but gains of over 1 BLEU are still possible when five million sentence-pairs are available  The most successful strategy to integrate pre-trained representations is as input to the encoder network (§ SECREF4 ) 
Strategies to add representations
We consider augmenting a standard sequence to sequence model with pre-trained representations following an ELMo-style regime (§ SECREF2 ) as well as by fine-tuning the language model (§ SECREF3 ) 
ELMo augmentation
The ELMo approach of BIBREF0 forms contextualized word embeddings based on language model representations without adjusting the actual language model parameters  Specifically, the ELMo module contains a set of parameters INLINEFORM0 to form a linear combination of the INLINEFORM1 layers of the language model: ELMo = INLINEFORM2 where INLINEFORM3 is a learned scalar, INLINEFORM4 is a constant to normalize the INLINEFORM5 to sum to one and INLINEFORM6 is the output of the INLINEFORM7 -th language model layer; the module also considers the input word embeddings of the language model  We also apply layer normalization BIBREF7 to each INLINEFORM8 before computing ELMo vectors 
We experiment with an ELMo module to input contextualized embeddings either to the encoder () or the decoder ()  This provides word representations specific to the current input sentence and these representations have been trained on much more data than is available for the text generation task 
Fine-tuning approach
Fine-tuning the pre-trained representations adjusts the language model parameters by the learning signal of the end-task BIBREF1 , BIBREF3   We replace learned input word embeddings in the encoder network with the output of the language model ()  Specifically, we use the language model representation of the layer before the softmax and feed it to the encoder  We also add dropout to the language model output  Tuning separate learning rates for the language model and the sequence to sequence model may lead to better performance but we leave this to future work  However, we do tune the number of encoder blocks INLINEFORM0 as we found this important to obtain good accuracy for this setting  We apply the same strategy to the decoder: we input language model representations to the decoder network and fine-tune the language model when training the sequence to sequence model () 
Datasets
We train language models on two languages: One model is estimated on the German newscrawl distributed by WMT'18 comprising 260M sentences or 6B tokens  Another model is trained on the English newscrawl data comprising 193M sentences or 5B tokens  We learn a joint Byte-Pair-Encoding (BPE; Sennrich et al , 2016) vocabulary of 37K types on the German and English newscrawl and train the language models with this vocabulary 
We consider two benchmarks: Most experiments are run on the WMT'18 English-German (en-de) news translation task and we validate our findings on the WMT'18 English-Turkish (en-tr) news task  For WMT'18 English-German, the training corpus consists of all available bitext excluding the ParaCrawl corpus and we remove sentences longer than 250 tokens as well as sentence-pairs with a source/target length ratio exceeding 1 5  This results in 5 18M sentence pairs  We tokenize all data with the Moses tokenizer BIBREF8 and apply the BPE vocabulary learned on the monolingual corpora 
For WMT'18 English-Turkish, we use all of the available bitext comprising 208K sentence-pairs without any filtering  We develop on newstest2017 and test on newstest2018  For en-tr we only experiment with adding representations to the encoder and therefore apply the language model vocabulary to the source side  For the target vocabulary we learn a BPE code with 32K merge operations on the Turkish side of the bitext  Both datasets are evaluated in terms of case-sensitive de-tokenized BLEU BIBREF9 , BIBREF10  
We consider the abstractive document summarization task comprising over 280K news articles paired with multi-sentence summaries  is a widely used dataset for abstractive text summarization  Following BIBREF11 , we report results on the non-anonymized version of rather than the entity-anonymized version BIBREF12 , BIBREF13 because the language model was trained on full text  Articles are truncated to 400 tokens BIBREF11 and we use a BPE vocabulary of 32K types BIBREF14   We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L BIBREF15  
Language model pre-training
We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder  Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right  The bi-directional model solves a cloze-style token prediction task at training time BIBREF17   The model consists of two towers, the forward tower operates left-to-right and the tower operating right-to-left as backward tower; each tower contains INLINEFORM1 transformer blocks  The forward and backward representations are combined via a self-attention module and the output of this module is used to predict the token at position INLINEFORM2   The model has access to the entire input surrounding the current target token  Models use the standard settings for the Big Transformer BIBREF16   The bi-directional model contains 353M parameters and the uni-directional model 190M parameters  Both models were trained for 1M steps using Nesterov's accelerated gradient BIBREF18 with momentum INLINEFORM3 following BIBREF19   The learning rate is linearly warmed up from INLINEFORM4 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0 0001 BIBREF20   We train on 32 Nvidia V100 SXM2 GPUs and use the NCCL2 library as well as the torch distributed package for inter-GPU communication  Training relies on 16-bit floating point operations BIBREF21 and it took six days for the bi-directional model and four days for the uni-directional model 
Sequence to sequence model
We use the transformer implementation of the fairseq toolkit BIBREF22   The WMT en-de and en-tr experiments are based on the Big Transformer sequence to sequence architecture with 6 blocks in the encoder and decoder  For abstractive summarization we use a base transformer model BIBREF16   We tune dropout values of between 0 1 and 0 4 on the validation set  Models are optimized with Adam BIBREF23 using INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 and we use the same learning rate schedule as BIBREF16 ; we perform 10K-200K depending on bitext size  All models use label smoothing with a uniform prior distribution over the vocabulary INLINEFORM3 BIBREF24 , BIBREF25   We run experiments on 8 GPUs and generate translations with a beam of size 5 
Machine translation
We first present a comparison of the various strategies in different simulated parallel corpus size settings  For each experiment, we tune the dropout applied to the language model representations, and we reduce the number of optimizer steps for smaller bitext setups as models converge faster; all other hyper-parameters are equal between setups  Our baseline is a Big Transformer model and we also consider a variant where we share token embeddings between the encoder and decoder (; Inan et al , 2016; Press & Wolf, 2016) 
Figure FIGREF10 shows results averaged over six test sets relative to the baseline which does not share source and target embeddings (Appendix SECREF6 shows a detailed breakdown)  performs very well with little labeled data but the gains erode to practically zero in large bitext settings  Pre-trained language model representations are most effective in low bitext setups  The best performing strategy is ELMo embeddings input to the encoder ()  This improves the baseline by 3 8 BLEU in the 160K bitext setting and it still improves the 5 2M setting by over 1 BLEU 
We further improve by sharing learned word representations in the decoder by tying input and output embeddings ()  This configuration performs even better than with a gain of 5 3 BLEU in the 160K setup  Sharing decoder embeddings is equally applicable to   Language model representations are much less effective in the decoder: improves the 160K bitext setup but yields no improvements thereafter and performs even worse  We conjecture that pre-trained representations give much easier wins in the encoder  Table TABREF14 shows additional results on newstest2018 
Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix SECREF7 ): slows throughput during training by about 5 3x and is even slower because of the need to backpropagate through the LM for fine-tuning (9 2x)  However, inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (, )  This is because the LM computation can be paralelized for all input tokens  Inference is much slower when adding representations to the decoder because the LM needs to be invoked repeatedly  Our current implementation does not cache LM operations for the previous state and can be made much faster 
The baseline uses a BPE vocabulary estimated on the language model corpora (§ SECREF3 )  Appendix SECREF6 shows that this vocabulary actually leads to sligtly better performance than a joint BPE code learned on the bitext as is usual 
Next, we validate our findings on the WMT'18 English-Turkish task for which the bitext is truly limited (208K sentence-pairs)  We use the language model vocab for the the English side of the bitext and a BPE vocabulary learned on the Turkish side  Table TABREF15 shows that ELMo embeddings for the encoder improve English-Turkish translation 
Abstractive summarization
Following BIBREF11 , we experiment on the non-anonymized version of   When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14   For this task we train language model representations on the combination of newscrawl and the training data  Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer  We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline  Pre-trained representations are complementary to their method 
Conclusion
We presented an analysis of different strategies to add pre-trained language model representations to sequence to sequence models for neural machine translation and abstractive document summarization  Adding pre-trained representations is very effective for the encoder network and while returns diminish when more labeled data becomes available, we still observe improvements when millions of examples are available  In future research we will investigate ways to improve the decoder with pre-trained representations 
Context 2: Introduction
Data annotation is a major bottleneck for the application of supervised learning approaches to many problems  As a result, unsupervised methods that learn directly from unlabeled data are increasingly important  For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4   While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6   In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure 
Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories  However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand  For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9   However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4  
In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies  To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e g  a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function  The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion 
By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model  In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss 
Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12   BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings  BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization  However, they do not represent the embeddings as latent variables 
In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV  We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags  Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available 
Model
As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations  Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector  Lastly, we extend our approach to more general syntactic structures 
Example: Gaussian HMM
We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions  Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4   The joint distribution of data and latent variables factors as:
DISPLAYFORM0
where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability 
While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap  Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables 
Markov Structure with Neural Projector
To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space  We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3   In the case of sequential Markov structure, our new model corresponds to the following generative process:
For each time step INLINEFORM0 ,
[noitemsep, leftmargin=*]
Draw the latent state INLINEFORM0
Draw the latent embedding INLINEFORM0
Deterministically produce embedding
INLINEFORM0
The graphical model is depicted in Figure FIGREF6   The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0   The joint distribution of our model is: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0
General Structure with Neural Projector
Our approach can be applied to a broad family of structured syntax models  We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4   The joint probability of our model factors as:
DISPLAYFORM0
where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model  As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper  The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision 
The marginal data likelihood of our model is: DISPLAYFORM0
While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq  ( EQREF17 ), which means inference and learning may be difficult  In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation 
Learning & Inference
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge  Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists  Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data  Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 )  Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function  Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation 
Learning with Invertibility
For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality  As shown in Eq  ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0   If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm  We can apply Eq  ( EQREF14 ) and obtain : INLINEFORM3
By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant  This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists 
Eq  ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0   Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0
where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2   Eq  ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term  From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term  Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model  The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection  Maximizing it can be thought of as preventing information loss  In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection  Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable 
More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0
If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0   Therefore, our joint model inherits the tractability of the underlying syntax model 
Invertible Volume-Preserving Neural Net
For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice  However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive  To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19   Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”  This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property) 
From Eq  ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1   Thus, we directly define the architecture of INLINEFORM2   As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer  The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively  A single coupling layer is defined as: DISPLAYFORM0
where INLINEFORM0 is the coupling function and can be any nonlinear form  This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal  Thus the Jacobian determinant is always equal to one (i e  volume-preserving) and the invertibility condition is naturally satisfied 
To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16   Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24   For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same  Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties  Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20  
Experiments
In this section, we first describe our datasets and experimental setup  We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively  Lastly, we analyze the learned latent embeddings 
Data
For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank  To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9   Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information  The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus 
General Experimental Setup
For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16   We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units  The number of coupling layers are varied as 4, 8, 16 for both tasks
Context 3: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case

ground_truth: uni-directional model to augment the decoder

bi-directional language model to augment the sequence to sequence encoder ,  uni-directional model to augment the decoder

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What datasets do they evaluate on?
Context 2:  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 

ground_truth:  Wall Street Journal (WSJ) portion of the Penn Treebank

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 3:  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish