Introduction
Transfer learning has been shown to work well in Computer Vision where pre-trained components from a model trained on ImageNet BIBREF0 are used to initialize models for other tasks BIBREF1 . In most cases, the other tasks are related to and share architectural components with the ImageNet task, enabling the use of such pre-trained models for feature extraction. With this transfer capability, improvements have been obtained on other image classification datasets, and on other tasks such as object detection, action recognition, image segmentation, etc BIBREF2 . Analogously, we propose a method to transfer a pre-trained component - the multilingual encoder from an NMT system - to other NLP tasks.
In NLP, initializing word embeddings with pre-trained word representations obtained from Word2Vec BIBREF3 or GloVe BIBREF4 has become a common way of transferring information from large unlabeled data to downstream tasks. Recent work has further shown that we can improve over this approach significantly by considering representations in context, i.e. modeled depending on the sentences that contain them, either by taking the outputs of an encoder in MT BIBREF5 or by obtaining representations from the internal states of a bi-directional Language Model (LM) BIBREF6 . There has also been successful recent work in transferring sentence representations from resource-rich tasks to improve resource-poor tasks BIBREF7 , however, most of the above transfer learning examples have focused on transferring knowledge across tasks for a single language, in English.
Cross-lingual or multilingual NLP, the task of transferring knowledge from one language to another, serves as a good test bed for evaluating various transfer learning approaches. For cross-lingual NLP, the most widely studied approach is to use multilingual embeddings as features in neural network models. However, research has shown that representations learned in context are more effective BIBREF5 , BIBREF6 ; therefore, we aim at doing better than just using multilingual embeddings in the cross-lingual tasks. Recent progress in multilingual NMT provides a compelling opportunity for obtaining contextualized multilingual representations, as multilingual NMT systems are capable of generalizing to an unseen language direction, i.e. zero-shot translation. There is also evidence that the encoder of a multilingual NMT system learns language agnostic, universal interlingua representations, which can be further exploited BIBREF8 .
In this paper, we focus on using the representations obtained from a multilingual NMT system to enable cross-lingual transfer learning on downstream NLP tasks. Our contributions are three-fold:
Proposed Method
We propose an Encoder-Classifier model, where the Encoder, leveraging the representations learned by a multilingual NMT model, converts an input sequence ${\mathbf {x}}$ into a set of vectors C, and the Classifier predicts a class label $y$ given the encoding of the input sequence, C.
Multilingual Representations Using NMT
Although there has been a large body of work in building multilingual NMT models which can translate between multiple languages at the same time BIBREF29 , BIBREF30 , BIBREF31 , BIBREF8 , zero-shot capabilities of such multilingual representations have only been tested for MT BIBREF8 . We propose a simple yet effective solution - reuse the encoder of a multilingual NMT model to initialize the encoder for other NLP tasks. To be able to achieve promising zero-shot classification performance, we consider two factors: (1) The ability to encode multiple source languages with the same encoder and (2) The ability to learn language agnostic representations of the source sequence. Based on the literature, both requirements can be satisfied by training a multilingual NMT model having a shared encoder BIBREF32 , BIBREF8 , and a separate decoder and attention mechanism for each target language BIBREF30 . After training such a multilingual NMT model, the decoder and the corresponding attention mechanisms (which are target-language specific) are discarded, while the multilingual encoder is used to initialize the encoder of our proposed Encoder-Classifier model.
Multilingual Encoder-Classifier
In order to leverage pre-trained multilingual representations introduced in Section "Analyses" , our encoder strictly follows the structure of a regular Recurrent Neural Network (RNN) based NMT encoder BIBREF33 with a stacked layout BIBREF34 . Given an input sequence ${\mathbf {x}} = (x_{1}, x_{2}, \ldots , x_{T_x})$ of length $T_x$ , our encoder contextualizes or encodes the input sequence into a set of vectors C, by first applying a bi-directional RNN BIBREF35 , followed by a stack of uni-directional RNNs. The hidden states of the final layer RNN, $h_i^l$ , form the set C $~=\lbrace  h_i^l \rbrace _{i=1}^{T_x}$ of context vectors which will be used by the classifier, where $l$ denotes the number of RNN layers in the stacked encoder.
The task of the classifier is to predict a class label $y$ given the context set C. To ease this classification task given a variable length input set C, a common approach in the literature is to extract a single sentence vector $\mathbf {q}$ by making use of pooling over time BIBREF36 . Further, to increase the modeling capacity, the pooling operation can be parameterized using pre- and post-pooling networks. Formally, given the context set C, we extract a sentence vector $\mathbf {q}$ in three steps, using three networks, (1) pre-pooling feed-forward network $f_{pre}$ , (2) pooling network $f_{pool}$ and (3) post-pooling feed-forward network $f_{post}$ , $ \mathbf {q} = f_{post}( f_{pool} ( f_{pre} (\textbf {C}) ) ). $
Finally, given the sentence vector $\mathbf {q}$ , a class label $y$ is predicted by employing a softmax function.
Corpora
We evaluate the proposed method on three common NLP tasks: Amazon Reviews, SST and SNLI. We utilize parallel data to train our multilingual NMT system, as detailed below.
For the MT task, we use the WMT 2014 En $\leftrightarrow $ Fr parallel corpus. The dataset contains 36 million En $\rightarrow $ Fr sentence pairs. We swapped the source and target sentences to obtain parallel data for the Fr $\rightarrow $ En translation task. We use these two datasets (72 million sentence pairs) to train a single multilingual NMT model to learn both these translation directions simultaneously. We generated a shared sub-word vocabulary BIBREF37 , BIBREF38 of 32K units from all source and target training data. We use this sub-word vocabulary for all of our experiments below.
The Amazon reviews dataset BIBREF39 is a multilingual sentiment classification dataset, providing data for four languages - English (En), French (Fr), German (De), and Japanese. We use the English and French datasets in our experiments. The dataset contains 6,000 documents in the train and test portions for each language. Each review consists of a category label, a title, a review, and a star rating (5-point scale). We only use the review text in our experiments. Following BIBREF39 , we mapped the reviews with lower scores (1 and 2) to negative examples and the reviews with higher scores (4 and 5) to positive examples, thereby turning it into a binary classification problem. Reviews with score 3 are dropped. We split the training dataset into 10% for development and the rest for training, and we truncate each example and keep the first 200 words in the review. Note that, since the data for each language was obtained by crawling different product pages, the data is not aligned across languages.
The sentiment classification task proposed in BIBREF9 is also a binary classification problem where each sentence and phrase is associated with either a positive or a negative sentiment. We ignore phrase-level annotations and sentence-level neutral examples in our experiments. The dataset contains 6920, 872, and 1821 examples for training, development and testing, respectively. Since SST does not provide a multilingual test set, we used the public translation engine Google Translate to translate the SST test set to French. Previous work by BIBREF40 has shown that replacing the human translated test set with a synthetic set (obtained by using Google Translate) produces only a small difference of around 1% absolute accuracy on their human-translated French SNLI test set. Therefore, the performance measured on our `pseudo' French SST test set is expected to be a good indicator of zero-shot performance.
Natural language inference is a task that aims to determine whether a natural language hypothesis $\mathbf {h}$ can justifiably be inferred from a natural language premise $\mathbf {p}$ . SNLI BIBREF10 is one of the largest datasets for a natural language inference task in English and contains multiple sentence pairs with a sentence-level entailment label. Each pair of sentences can have one of three labels - entailment, contradiction, and neutral, which are annotated by multiple humans. The dataset contains 550K training, 10K validation, and 10K testing examples. To enable research on multilingual SNLI, BIBREF40 chose a subset of the SNLI test set (1332 sentences) and professionally translated it into four major languages - Arabic, French, Russian, and Spanish. We use the French test set for evaluation in Section "Zero-Shot Classification Results" and "Analyses" .
Model and Training Details
Here, we first describe the model and training details of the base multilingual NMT model whose encoder is reused in all other tasks. Then we provide details about the task-specific classifiers. For each task, we provide the specifics of $f_{pre}$ , $f_{pool}$ and $f_{post}$ nets that build the task-specific classifier.
All the models in our experiments are trained using Adam optimizer BIBREF42 with label smoothing BIBREF43 and unless otherwise stated below, layer normalization BIBREF44 is applied to all LSTM gates and feed-forward layer inputs. We apply L2 regularization to the model weights and dropout to layer activations and sub-word embeddings. Hyper-parameters, such as mixing ratio $\lambda $ of L2 regularization, dropout rates, label smoothing uncertainty, batch sizes, learning rate of optimizers and initialization ranges of weights are tuned on the development sets provided for each task separately.
Our multilingual NMT model consists of a shared multilingual encoder and two decoders, one for English and the other for French. The multilingual encoder uses one bi-directional LSTM, followed by three stacked layers of uni-directional LSTMs in the encoder. Each decoder consists of four stacked LSTM layers, with the first LSTM layers intertwined with additive attention networks BIBREF33 to learn a source-target alignment function. All the uni-directional LSTMs are equipped with residual connections BIBREF45 to ease the optimization, both in the encoder and the decoders. LSTM hidden units and the shared source-target embedding dimensions are set to 512.
Similar to BIBREF30 , multilingual NMT model is trained in a multi-task learning setup, where each decoder is augmented with a task-specific loss, minimizing the negative conditional log-likelihood of the target sequence given the source sequence. During training, mini-batches of En $\rightarrow $ Fr and Fr $\rightarrow $ En examples are interleaved. We picked the best model based on the best average development set BLEU score on both of the language pairs.
The Encoder-Classifier model here uses the encoder defined previously. With regards to the classifier, the pre- and post-pooling networks ( $f_{pre}$ , $f_{post}$ ) are both one-layer feed forward networks to cast the dimension size from 512 to 128 and from 128 to 32, respectively. We used max-pooling operator for the $f_{pool}$ network to pool the activation over time.
We extended the proposed Encoder-Classifier model to a multi-source model BIBREF46 since SNLI is an inference task of relations between two input sentences, “premise" and “hypothesis". For the two sources, we use two separate encoders, which are initialized with the same pre-trained multilingual NMT encoder, to obtain their representations. Following our notation, the encoder outputs are processed using $f_{pre}$ , $f_{pool}$ and $f_{post}$ nets, again with two separate network blocks. Specifically, $f_{pre}$ consists of a co-attention layer BIBREF47 followed by a two-layer feed-forward neural network with residual connections. We use max pooling over time for $f_{pool}$ and again a two-layer feed-forward neural network with residual connections as $f_{post}$ . After processing two sentence encodings using two network blocks, we obtain two vectors representing premise $\mathbf {h}_{premise}$ and hypothesis $\mathbf {h}_{hypothesis}$ . Following BIBREF48 , we compute two types of relational vectors with $\mathbf {h}_{-} = |\mathbf {h}_{premise} - \mathbf {h}_{hypothesis}|,$ and $\mathbf {h}_{\times } = \mathbf {h}_{premise} \odot \mathbf {h}_{hypothesis}$ , where $f_{pool}$0 denotes the element-wise multiplication between two vectors. The final relation vector is obtained by concatenating $f_{pool}$1 and $f_{pool}$2 . For both “premise" and “hypothesis" feed-forward networks we used 512 hidden dimensions.
For Amazon Reviews, SST and SNLI tasks, we picked the best model based on the highest development set accuracy.
Transfer Learning Results
In this section, we report our results for the three tasks - Amazon Reviews (English and French), SST, and SNLI. For each task, we first build a baseline system using the proposed Encoder-Classifier architecture described in Section "Proposed Method" where the encoder is initialized randomly. Next, we experiment with using the pre-trained multilingual NMT encoder to initialize the system as described in Section "Analyses" . Finally, we perform an experiment where we freeze the encoder after initialization and only update the classifier component of the system.
Table 1 summarizes the accuracy of our proposed system for these three different approaches and the state-of-the-art results on all the tasks. The first row in the table shows the baseline accuracy of our system for all four datasets. The second row shows the result from initializing with a pre-trained multilingual NMT encoder. It can be seen that this provides a significant improvement in accuracy, an average of 4.63%, across all the tasks. This illustrates that the multilingual NMT encoder has successfully learned transferable contextualized representations that are leveraged by the classifier component of our proposed system. These results are in line with the results in BIBREF5 where the authors used the representations from the top NMT encoder layer as an additional input to the task-specific system. However, in our setup we reused all of the layers of the encoder as a single pre-trained component in the task-specific system. The third row shows the results from freezing the pre-trained encoder after initialization and only training the classifier component. For the Amazon English and French tasks, freezing the encoder after initialization significantly improves the performance further. We hypothesize that since the Amazon dataset is a document level classification task, the long input sequences are very different from the short sequences consumed by the NMT system and hence freezing the encoder seems to have a positive effect. This hypothesis is also supported by the SNLI and SST results, which contain sentence-level input sequences, where we did not find any significant difference between freezing and not freezing the encoder.
Zero-Shot Classification Results
In this section, we explore the zero-shot classification task in French for our systems. We assume that we do not have any French training data for all the three tasks and test how well our proposed method can generalize to the unseen French language without any further training. Specifically, we reuse the three proposed systems from Table 1 after being trained only on the English classification task and test the systems on data from an unseen language (e.g. French). A reasonable upper bound to which zero-shot performance should be compared to is bridging - translating a French test text to English and then applying the English classifier on the translated text. If we assume the translation to be perfect, we should expect this approach to perform as well as the English classifier.
The Amazon Reviews and SNLI tasks have a French test set available, and we evaluate the performance of the bridged and zero-shot systems on each French set. However, the SST dataset does not have a French test set, hence the `pseudo French' test set described in Section UID14 is used to evaluate the zero-shot performance. We use the English accuracy scores from the SST column in Table 1 as a high-quality proxy for the SST bridged system. We do this since translating the `pseudo French' back to English will result in two distinct translation steps and hence more errors.
Table 2 summarizes all of our zero-shot results for French classification on the three tasks. It can be seen that just by using the pre-trained NMT encoder, the zero-shot performance increases drastically from almost random to within 10% of the bridged system. Freezing the encoder further pushes this performance closer to the bridged system. On the Amazon Review task, our zero-shot system is within 2% of the best bridged system. On the SST task, our zero-shot system obtains an accuracy of 83.14% which is within 1.5% of the bridged equivalent (in this case the English system).
Finally, on SNLI, we compare our best zero-shot system with bilingual and multilingual embedding based methods evaluated on the same French test set in BIBREF40 . As illustrated in Table 3 , our best zero-shot system obtains the highest accuracy of 73.88%. INVERT BIBREF23 uses inverted indexing over a parallel corpus to obtain crosslingual word representations. BiCVM BIBREF25 learns bilingual compositional representations from sentence-aligned parallel corpora. In RANDOM BIBREF24 , bilingual embeddings are trained on top of parallel sentences with randomly shuffled tokens using skip-gram with negative sampling, and RATIO is similar to RANDOM with the one difference being that the tokens in the parallel sentences are not randomly shuffled. Our system significantly outperforms all methods listed in the second column by 10.66% to 15.24% and demonstrates the effectiveness of our proposed approach.
Analyses
In this section, we try to analyze why our simple Encoder-Classifier system is effective at zero-shot classification. We perform a series of experiments to better understand this phenomenon. In particular, we study (1) the effect of shared sub-word vocabulary, (2) the amount of multilingual training data to measure the influence of multilinguality, (3) encoder/classifier capacity to measure the influence of representation power, and (4) model behavior on different training phases to assess the relation between generalization performance on English and zero-shot performance on French.
Conclusion
In this paper, we have demonstrated a simple yet effective approach to perform cross-lingual transfer learning using representations from a multilingual NMT model. Our proposed approach of reusing the encoder from a multilingual NMT system as a pre-trained component provides significant improvements on three downstream tasks. Further, our approach enables us to perform surprisingly competitive zero-shot classification on an unseen language and outperforms cross-lingual embedding base methods. Finally, we end with a series of analyses which shed light on the factors that contribute to the zero-shot phenomenon. We hope that these results showcase the efficacy of multilingual NMT to learn transferable contextualized representations for many downstream tasks.

Question: Do the other multilingual baselines make use of the same amount of training data?
Answer: Unanswerable

Question: How big is the impact of training data size on the performance of the multilingual encoder?
Answer: Unanswerable

Question: What data were they used to train the multilingual encoder?
Answer: WMT 2014 En-Fr parallel corpus