Title: LEARNING CROSS-LINGUAL SENTENCE REPRESEN-TATIONS VIA A MULTI-TASK DUAL-ENCODER MODEL

Abstract: A significant roadblock in multilingual neural language modeling is the lack of labeled non-English data. One potential method for overcoming this issue is learning cross-lingual text representations that can be used to transfer the performance from training on English tasks to non-English tasks, despite little to no task-specific non-English data. In this paper, we explore a natural setup for learning crosslingual sentence representations: the dual-encoder. We provide a comprehensive evaluation of our cross-lingual representations on a number of monolingual, crosslingual, and zero-shot/few-shot learning tasks, and also give an analysis of different learned cross-lingual embedding spaces.

1. INTRODUCTION:

There has been a significant amount of recent work on developing models that can produce sentence representations that are useful for a number of language processing tasks (Kiros et al., 2015; Conneau et al., 2017; Subramanian et al., 2018; Logeswaran & Lee, 2018; Cer et al., 2018) . However, these models are trained on largely monolingual data, and can thus only be used for tasks in a single language. A promising direction for extending the previous models to multiple languages is learning cross-lingual embedding spaces (Schwenk et al., 2017; Eriguchi et al., 2018; Singla et al., 2018) , which could be used to transfer performance in one language to others.

We develop a novel approach for cross-lingual representation learning by combining the dualencoder architectures used for learning sentence representations (Logeswaran & Lee, 2018; Cer et al., 2018) and for bi-text retrieval (Guo et al., 2018) . By doing so, we learn representations that maintain state-of-the-art performance in tasks for a source language while simultaneously obtaining state-of-the-art performance in zero-shot learning tasks for a target language. For a given sourcetarget language pair, we construct a multi-task training scheme using native source language tasks, native target language tasks, and a bridging source-target translation retrieval task to learn sentence representations that are aligned between the source and target languages. We then evaluate the learned representations on several monolingual and cross-lingual tasks, and also provide a graphbased analysis of the learned representations.

We find that multi-task training using additional monolingual tasks improves performance over models that only make use of parallel data on both cross-lingual semantic textual similarity (STS) (Cer et al., 2017) and Søgaard et al. (2018) 's cross-lingual eigen-similarity metric. The results show that the addition of monolingual data improves the embedding alignment of sentences and their translations. Furthermore, we find that cross-lingual training with additional monolingual data leads to far better transfer learning performance, and we show that our cross-lingual representations outperform state-of-the-art zero-shot learning models in sentiment classification and natural language inference.

2. MULTI-TASK DUAL-ENCODER MODEL:

The core of our approach is the idea of modeling various tasks as ranking input-response pairs by encoding them via two encoders, with the crucial task for learning cross-lingual representations being translation ranking. For translation ranking, as well as for our other tasks, we take an input sentence s I i and an associated response sentence s R i , and we seek to rank s R i over all other possible response sentences s R j ∈ S R . To do so, we model the conditional probability P (s R i | s I i ) as: 

Encoder s I i s R i g(s I i ) g(s R i ) g R (s R i ) (g(s I i )) T g R (s R i )

s R i g(s R i ) g R (s R i ) Encoder Fully-connected layers (g I (s I i )) T g R (s R i )

Encoder

s i t i g(s i ) g(t i ) (g(s i )) T g(t i ) Translation Encoder Encoder s I i s i R g(s I i ) = u 1 (u 1 , u 2 , |u 1 -u 2 |, u 1 *u 2 ) NLI Encoder Fully-connected layers Softmax g(s R i ) = u 2

Native Tasks Bridging Task

Figure 1 : Multi-task dual-encoder model. It consists of a group of native tasks in each language and a bridging task using translation pair data. The encoders in the gray box all share their parameters, and thus constitute g.

EQUATION

Where g I and g R are the input and response sentence encoding functions that compose the dualencoder. Since the normalization term in equation 1 is computationally intractable, we follow the approaches of Henderson et al. (2017) and instead choose to model an approximate conditional probability P (s R i | s I i ):

EQUATION

Where K denotes the size of a single batch of training examples, and the s R j correspond to the response sentences associated with the other input sentences in the same batch as s I i . We parametrize g I and g R as deep neural networks that are trained to minimize the negative log-likelihood of P (s R i | s I i ) for each task. In order to produce a single sentence encoding function g that can be evaluated on downstream tasks, we share several layers between the input and response encoders and treat the final output of these shared layers as g. Additionally, these layers are modeled after the Universal Sentence Encoder (USE) model of Cer et al. (2018) , since it is the state-of-the-art model that is most amenable to our setup. To learn cross-lingual representations, we train g on several tasks from mirrored corpora across languages1 for the source-target language pairs English-French (en-fr), English-Spanish (enes), and English-German (en-de). The resulting model structure is illustrated in Figure 1 .

2.1. ENCODER ARCHITECTURE:

Word and Character Embeddings. As part of the training process for learning the cross-lingual sentence encoding function g, we learn embeddings for the words and characters present in the training data for a given source-target language pair. Word embeddings are learned end-to-end, as we noticed that pre-trained embeddings did not make a difference for final performance. Character embeddings are learned in a similar manner, but with the added stipulation that we consider character n-gram embeddings instead of single character embeddings by using a single feedforward layer with tanh activation on top of character n-grams. Each word in an input sentence then obtains a character representation by having its character n-gram representations summed together. To have the sentence encoder g leverage the word and character representations together without drastically increasing its number of parameters, we sum the word and character embeddings before using them as input to g.

Transformer Encoder. The actual architecture of the shared encoder g consists of three2 layers of transformer stacks, which contain the feed-forward and multi-head attention sub-layers described in Vaswani et al. (2017) . The transformer encoder output is a variable-length sequence at each stack. We average encodings of all sequence positions in the final layer as the final sentence encoding. This embedding is then fed into different sets of feedforward layers that are used for each task. For our transformer layers, we use 8 attentions heads, a hidden size of 512, and a filter size of 2048.

2.2. MULTI-TASK TRAINING SETUP:

To learn a function g that is capable of strong cross-lingual matching and transfer learning performance for a source-target language pair while also maintaining monolingual downstream task performance, we employ four unique task types for each language pair. Specifically, we employ a conversation response prediction task, a quick thought task, a natural language inference task, and a bridging task -translation ranking. Six total tasks are used in training, as the first two tasks are mirrored across languages.

Conversation Response Prediction. We model the conversation response prediction task in the same manner as Yang et al. (2018) . We minimize the negative log-likelihood of P (s R i | s I i ), where s I i is a single comment and s R i is its associated response comment. For the response side, we model g R (s R i ) as two fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g(s R i ). For the input side, however, we simply let g I (s I i ) = g(s I i ), as we noticed in early experiments that letting the optimization of the conversational response task more directly influence the parameters of the underlying sentence encoder g led to better downstream task performance.

Quick Thought. We use a modified version of the Quick Thought task detailed by Logeswaran & Lee (2018) . We minimize the sum of the negative log-likelihoods of P (s R i | s I i ) and P (s P i | s I i ), where s I i is a sentence taken from an article and s P i and s R i are its predecessor and successor sentences respectively. For this task, we model all three of g P (s P i ), g I (s I i ), and g R (s R i ) using separate, fully-connected feedforward layers of size 320 and 512 with tanh activation on top of g, as we did for g R (s R i ) in our conversational modeling task. Natural Language Inference (NLI). We also include an English-only natural language inference task based on Bowman et al. (2015) . For this task, we first encode an input sentence s I i and its corresponding response hypothesis s R i into vectors u 1 and u 2 using g. The vectors u 1 , u 2 are then used to construct a feature vector (u 1 , u 2 , |u 1 -u 2 |, u 1 * u 2 ), where (•) represents concatenation and * represents element-wise multiplication. The form of this feature vector is derived from the original experiments of Bowman et al. (2015) . This feature vector is then fed into a single feedforward layer of size 512 that is used to perform the 3-way NLI classification.

Translation Ranking. Our translation task setup is identical to the one used by Guo et al. (2018) for bi-text retrieval. We minimize the negative log-likelihood of P (s i | t i ), where (s i , t i ) is a sourcetarget translation pair. Since the translation task is intended to align the sentence representations for the source and target languages, we do not use any kind of task-specific feedforward layers and instead use g as both g I and g R . Following Guo et al. (2018) , we append 5 translations that are similar to the correct translation to each training example as "hard-negatives". Similarity is determined via a version of our model trained only on the translation ranking task. We did not see additional gains from using more than 5 hard-negatives.

3. EXPERIMENTS:



3.1. CORPORA:

We draw upon multiple, openly available data sources and training corpora for the training of the tasks mentioned above. For each of our datasets, we use 90% of the data for training, and the remaining 10% for development/validation. Our data preprocessing procedures are described in detail in the supplementary material.

Reddit. We preprocess the Reddit data extracted by Al-Rfou et al. (2016) into 600 million inputresponse comment pairs for training our conversation response prediction task. We also translate this data using the Google neural machine translation (NMT) system of Wu et al. (2016) .

Wikipedia. To get native, non-English data, we extract triplets of contiguous sentences from English, French, Spanish, and German articles take from Wikipedia. Our final extracted corpus of Wikipedia sentence triplets consists of 127.9, 49.5, 29.8, and 49.3 million triplets for English, French, Spanish, and German respectively, which we use to train our Quick Thought task.

Stanford Natural Language Inference (SNLI). The NLI data we use is taken from the Stanford Natural Language Inference (SNLI) dataset of Bowman et al. (2015) , which consists of 570K sentence pairs associated with one of three labels: entailment, contradiction, or neutral. The corpus is split into training (550K), validation (10K), and testing sets (10K).

Translation. The data for training the translation task is constructed using a system similar to the approach described by Guo et al. (2018) . The final constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs.

3.2. MODEL CONFIGURATION:

In all of our experiments, multi-task training is done by cycling through the different tasks (translation pairs, Reddit, Wikipedia, NLI) and performing an optimization step for a single task at a time. We train all of our models with a batch size of 100 using stochastic gradient descent with a learning rate of 0.008. All of our models are trained for 30 million steps. All input text is tokenized prior to being used for training. We build a vocab containing 200 thousand unigram tokens with 10 thousand hash buckets for out-of-vocabulary tokens. The character n-gram vocab contains 200 thousand hash buckets used for 3 and 4 grams. Both the word and character n-gram embedding sizes are 320. All hyperparameters are tuned based on performance on the development portion (random 10% slice) of our datasets. Finally, as an additional training heuristic, we multiply the gradients to the word and character embeddings by a factor of 1003 . We found that using this embedding gradient multiplier alleviated vanishing gradient issues and greatly improved training.

We compare the proposed cross-lingual multi-task (referred to simply as multi-task) models with baseline models that are trained using only the translation ranking task, which we dub as the "translation-ranking" models.

3.3. MODEL PERFORMANCE ON ENGLISH DOWNSTREAM TASKS:

We first evaluated all of our cross-lingual models on several downstream English tasks taken from SentEval (Conneau & Kiela, 2018) to verify the impact of cross-lingual training. Each task is described in the supplementary material. Results on the tasks are summarized in Table 1 . We note that cross-lingual training does not hinder the effectiveness of our encoder on English tasks, as the multi-task models are close to state-of-the-art in each of the downstream tasks. For the Text REtrieval Conference (TREC) eval, we actually find that our multi-task models outperform the previous state-of-the-art models by a sizable amount.

3.4. CROSS-LINGUAL RETRIEVAL:

We also evaluate both the multi-task and translation-ranking models' efficacy in performing crosslingual retrieval by using held-out translation pair data. Following Henderson et al. (2017) , we use precision at N (P@N) as the evaluation metric by checking if a source sentence's target translation ranks (where ranking is done using dot product) in the top N scored candidates when considering K other randomly selected target sentences. Unlike Henderson et al. (2017) , we set K to be 999 instead of 99 because using K = 99 results in all metrics quickly shooting up to 99%.

The translation-ranking model remains as a strong baseline for finding the true translation, with 95.4%, 87.5%, 97.5% P@1 for en-fr, en-es and en-de retrieval tasks respectively. The multi-task model performs almost identical with 95.1%, 88.8% and 97.8%, which provides empirical justifica- tion that it is possible to maintain embedding space alignment despite optimizing for native tasks in each individual language. We also experimented with P@3 and P@10, the results are identical.

3.5. MULTILINGUAL STS:

We further test whether our learned cross-lingual representations can also perform well in their associated non-English language tasks by evaluating semantic textual similarity (STS) performance on French, Spanish, and German.

To evaluate Spanish-Spanish (es-es) STS, we use SemEval-2017 task 1 (STS17) track 3 of Cer et al. (2017) , which contains 250 Spanish sentence pairs with human labeled similarity scores. We also evaluate es-en STS by using the track 4(a) task4 , which contains 250 en-es sentence pairs.

Beyond English and Spanish, however, there are no standard STS datasets available for other languages. As such, we evaluate on a translated version of the STS Benchmark dataset from Cer et al. (2017) for French, Spanish, and German. We use Google's translation system to translate the STS Benchmark sentences to French, Spanish and German. We believe that the results on our pseudomultilingual STS Benchmark dataset are expected to still be a reasonable indicator of multilingual semantic similarly performance, since the NMT encoder-decoder architecture for translation differs significantly from our dual-encoder approach.

Following Cer et al. (2018) , we first compute the sentence encodings u, v of an STS sentence pair, and then score the sentence pair similarity based on the angular distance between the two vectors, -arccos uv ||u|| ||v|| . Table 2 shows the Pearson's correlation coefficient of the STS tasks for all models. The first column shows the trained model performance on original English STS Benchmark data. Columns 2 to 4 shows the the performance on the other languages. All multi-task models remain strong on the translated STS tasks, with around 0.77 for dev and 0.74 for test in all languages. Lastly, columns 5 and 6 shows the results of en-es models on STS17 tasks. The un-tuned multi-task models achieve 0.827 for the es-es task and 0.769 for the es-en task. As a point of reference, we also list the two best performing STS systems, Tian et al. (2017) (ECNU) and Wu et al. (2017) (BIT), reported from Cer et al. (2017) . Our results are very close to these state-of-the-art feature engineered and mixed systems, which we describe in greater detail in the supplementary material.

4. ZERO-SHOT CLASSIFICATION:

To evaluate the transfer learning capabilities of our models, we examine how well the multi-task and translation-ranking encoders perform on zero-shot and few-shot classification tasks. 

4.1. MULTILINGUAL NLI:

We evaluate the zero-shot classification performance of our multi-task models on two multilingual natural language inference (NLI) tasks. However, prior to doing so, we first train a modified version5 of our multi-task models that also includes training on the English Multi-genre NLI (MultiNLI) dataset of Williams et al. (2018) in addition to SNLI. We train with MultiNLI to be consistent with the baselines we compare to, which also use MultiNLI.

First, we make use of the professionally translated French and Spanish subsets of SNLI created by Agić & Schluter (2017) for an initial cross-lingual zero-shot evaluation of French and Spanish. We refer to these translated subsets as SNLI-X. There are 1000 examples in the translated subsets for each language. To evaluate, we simply feed the French and Spanish examples into the pre-trained English NLI sub-network of our multi-task models.

We also use the more recent dataset (XNLI) of Conneau et al. (2018) , which provides a means for multilingual NLI evaluation in Spanish, French, German, Chinese and more. Since XNLI provides non-European-language evaluations, we also train an English-Chinese (en-zh) version of our multitask model. There are 5000 examples in each XNLI test set, and zero-shot evaluation is once again done by feeding non-English examples into the pre-trained English NLI sub-network.

Table 3 lists the accuracy on the English SNLI test set as well as on SNLI-X and XNLI for all of our multi-task models. The original English SNLI accuracies are around 84% for all of our multitask models, indicating that English SNLI performance remains stable in the multi-task training setting. The zero-shot accuracy on SNLI-X is around 74% for both the en-fr and en-es models.

The zero-shot accuracy on XNLI is around 65% for en-es, en-fr, and en-de, and around 63% for en-zh, thereby significantly outperforming the pretrained sentence encoding baselines (X-CBOW) described in Conneau et al. (2018) . The X-CBOW baselines use fixed sentence encoders that are the result of averaging tuned multilingual word embeddings.

Row 4 of Table 3 shows the zero-shot French NLI performance of Eriguchi et al. (2018) , which is a state-of-the-art zero-shot NLI classifiers based on multilingual NMT embeddings. Our multi-task model shows comparable performance to the NMT-based model in both English and French.

4.2. AMAZON REVIEW:

Zero-shot Learning. We also conduct a zero-shot evaluation based on the Amazon review data extracted by Prettenhofer & Stein (2010) . We preprocess the Amazon reviews and convert the data into a sentiment classification task by considering reviews with strictly more than three stars as positive and strictly less than three stars as negative, in the same manner as Prettenhofer & Stein (2010) . Each review contains a summary field and a text field, which we concatenate to produce a single input. As the multi-task models are trained with sentence lengths clipped to 64, we only take the first 64 tokens from the the concatenated text as the input. There are 6000 training reviews in English, which we split into 90% for training and 10% for development. compute pairwise distance matrices within each sampled set of encodings, and use these distance matrices to construct graph Laplacians7 . Finally, we obtain the similarity Ψ(S, T ) between each model's source and target language embedding subsets by comparing the eigenvalues of the source language graph Laplacians to the eigenvalues of the target language graph Laplacians as follows:

Ψ(S, T ) = 1 N N i=1 K j=1 (λ j (L (s) i ) -λ j (L (t) i )) 2 (3) Where L (s) i and L (t)

i refer to the graph Laplacians of the source language and target language sentences obtained from the i th sample of source-target translation pairs. A smaller value of Ψ(S, T ) indicates higher eigen-similarity of the source language and target language embedding subsets. Following Søgaard et al. (2018) we use a sample size of K = 10 translation pairs, but we choose to use N = 1000 samples instead of N = 10 (as was done in their work) since we found Ψ(S, T ) to have very high variance at N = 10. The computed values of Ψ(S, T ) for our multi-task and translation-ranking models are summarized in Table 6 .

Table 6 : Average eigen-similarity values of source and target embedding subsets.

Model:

en-fr en-es en-de multi-task 0.592 0.526 0.761 translation-ranking 1.036 0.572 2.187

We find that the source and target embedding subsets constructed from the multi-task models exhibit greater average eigen-similarity than those resulting from the translation-ranking models for all source-target language pairs. This result is not necessarily intuitive, since one might expect the translation-ranking model to optimize more for alignment. Given that eigen-similarity correlates with the better performance of the multi-task models in almost all tasks, a potential direction for future work could be to introduce regularization penalties based on graph similarity in multitask training. Interestingly, we also observe that the eigen-similarity gaps between the multi-task and translation-ranking models are not uniform across language pairs (although it may be that translation-ranking requires even more training). Thus, another direction could be to further study differences in the difficulty of aligning different source-target language embeddings.

6. CONCLUSION:

In this work, we explored a straightforward framework for training cross-lingual, multi-task dualencoder models. We showed that by training English-French, English-Spanish, and English-German multi-task models using our setup, we can achieve near-state-of-the-art or state-of-the-art performance in a variety of English tasks while also being able to produce similar caliber results in zeroshot transfer learning tasks for other languages. Finally, we note that the fact that multi-task training can actually improve performance on some downstream English tasks (TREC) is particularly interesting, and believe that there are many possibilities for future explorations of cross-lingual model training.

A SUPPLEMENTARY MATERIAL:

A.1 DATA PREPROCESSING In order to effectively use Reddit, Wikipedia, and translation data, we do a significant amount of preprocessing on the raw data. We describe our preprocessing procedures for each dataset in the following paragraphs.

Reddit. The raw Reddit corpus extracted by Al-Rfou et al. (2016) consists of 2.4 billion posts and comments from Reddit between 2007 and 2016, making it our largest data source by an order of magnitude. To preprocess this data for training our conversation response prediction task, we follow the same procedure as Yang et al. (2018) . Essentially, we consider Reddit comments and their children (response comments) as input-response pairs. We filter out comments that have ≥ 350 characters, due to the limitations on the number of input tokens our models can accept. We also remove comments that start with "https", "@", or "/r/", and also remove comments whose authors have "bot" in their usernames. Lastly, we find that Reddit comments also contain a small mix of non-English text, and we filter out comments where the percentage of alphabetic characters is ≤ 70%. As mentioned in the paper, the final, processed Reddit data consists of 600 million input-response pairs. Since Reddit comments are predominantly English, we found the original data to be unsuitable for training encoders for non-English languages. To create conversational corpora for other languages, we translate the entire Reddit dataset using Google's neural machine translation (NMT) system of Wu et al. (2016) .

Wikipedia. One concern with using the translated Reddit corpus as a monolingual task for non-English languages is that the translated data will propagate any errors made by the NMT system used for translation. To ameliorate this problem, we crawl Wikipedia and extract triplets of contiguous sentences from Wikipedia articles. We use Wikipedia due to it having well-formed articles with a broad coverage of several languages. We extract all Wikipedia articles for English, French, Spanish and German from the Wikipedia dump of May 5, 2018, and use the sentence splitter model of Gillick (2009) to split the articles into sentences. All article titles and article section names are treated as sentences inline. As mentioned in the paper, our final extracted Wikipedia corpus consists of 127.9 million English sentence triplets, 49.5 million French triplets, 29.8 million Spanish triplets, and 49.3 million German triplets for French, Spanish and German respectively.

Stanford Natural Language Inference (SNLI). We use the English Stanford Natural Language Inference (SNLI) dataset consisting of 570K sentence pairs of Bowman et al. (2015) as is, without any further preprocessing. We also use the provided splits for training (550K), validation (10K), and testing (10K).

Translation. The data for training the translation task is crawled from the public web using a system similar to the approach described by Uszkoreit et al. (2010) . The extracted data is further cleaned by a pre-trained translation pair scoring system. We then generate "hard-negatives" following Guo et al. (2018) by using a pre-trained coarse translation-ranking model to determine translations that are close to a correct translation. As mentioned in the paper, the final constructed corpus contains around 600M en-fr pairs, 470M en-es pairs and 500M en-de pairs. A.2 DOWNSTREAM TASK DESCRIPTIONS Each of the English downstream tasks shown in Table 1 in the main body of the paper are described briefly below:

• Movie Reviews (MR): Multi-class prediction (scale of one to five stars) on movie review snippet data from Pang & Lee (2005) .

• Customer Reviews (CR): Binary sentiment classification of sentences from customer reviews mined by Hu & Liu (2004) .

• Subjectivity (SUBJ): Binary classification of sentences from movie reviews (Pang & Lee, 2004) as either objective or subjective. (Cer et al., 2017) .

A.3 STATE-OF-THE-ART STS SYSTEM DESCRIPTIONS:

For the STS evaluations provided in the body of our paper, we compared to the state-of-the-art systems of Tian et al. (2017) and Wu et al. (2017) . We refer the reader to Cer et al. (2017) for a detailed summary of the characteristics of these systems, but we provide a brief characterization below for convenience:

• ECNU: The ECNU model is an ensemble of random forest, gradient boosted decision trees, and deep learning models that leverages several manually constructed features such as ngram overlap, edit distance, longest common prefix, common substrings, word alignments, and more. • BIT: The BIT model relies mostly on a feature called information content (IC), which is based on the likelihood of occurrence of a concept (in this case, a word). Information content is then computed and combined for sentences through a hierarchical approach, which Wu et al. ( 2017) also augment with word embedding alignment to produce a final ensemble model.

A.4 ABLATION TESTS AND FURTHER EXPERIMENTS:

No NLI. Given that the multi-task models detailed in the body of the paper are trained with an English NLI task but no non-English NLI tasks, we evaluate how much training without this English NLI task affects model performance on downstream English tasks. The performance of these no-NLI models is summarized under the "Cross-lingual Multitask Transformer No SNLI" section of Table 7 . We find that training without SNLI leads to comparable or better performance on all English downstream tasks except for STS, where we find training with SNLI provides a significant bump in performance.

No Wikipedia. We similarly investigate whether there is a significant advantage gained from using the only native target language data source, Wikipedia. To do so, we train en-fr, en-es, and en-de multi-task models without the Wikipedia-based Quick Thought task. The performance of these nowiki models is shown under the "Cross-lingual Multitask Transformer No Wiki" section of Table 7 . We find that training without the Wikipedia Quick Thought task does not have a strong impact on model performance, with some tasks being marginally better and others being marginally worse.

No target language native tasks. Additionally, the cross-lingual, multi-task models discussed in the body of our paper use a largely parallel task setup, where for each source-target language pair the monolingual target tasks are mirrored from the monolingual source tasks (the only exception being NLI, which is English-only). To test how useful this mirroring of monolingual tasks is, we train several models without target language monolingual tasks. We label these models as the "no non-English native task" models, and also summarize their downstream English task performance in Table 7 . We note that removing the non-English native tasks actually leads to significant decreases in performance on TREC, SST, and STS. We find this quite interesting, as it provides some more empirical justification for the notion that cross-lingual training for a source-target language pair can actually improve monolingual source task performance.

Finally, we also evaluate the zero-shot performance of the no non-English native task models on Amazon review sentiment classification. As can be seen in Table 8 , training without the non-English native tasks leads to lower performance on English and roughly the same zero-shot classification performance as training with only translation pair data. 

:

We explore some training setups without mirroring in the supplementary material.

We tried up to six layers, but did not notice a significant difference beyond three.

We tried different orders of magnitude for the multiplier and found 100 to work the best.

The es-en task is split into track 4(a) and track 4(b), we only use track 4(a) here. The track 4(b) task contains sentence pairs from WMT with only one annotator for each pair. The previous reported numbers are also particularly low for this task. We suspect it is not a very good evaluation set for current systems.

Training with additional MultiNLI data did not significantly impact SNLI or downstream task performance.

Eriguchi et al. (2018) also train a shallow classifier, but use only review text and truncate their inputs to 200 tokens. Our setup is slightly different, as our models can take a maximum of only 64 tokens.

See Zhang (2011) for an overview of graph Laplacians.

Figures and Tables:

Performance on classification transfer tasks.

Table content: <table><tr><td>Model</td><td>MR</td><td>CR</td><td colspan="4">SUBJ MPQA TREC SST</td><td>STS Bench (dev / test)</td></tr><tr><td/><td/><td colspan="4">Cross-lingual Multi-task Models</td><td/></tr><tr><td>en-fr</td><td colspan="2">77.9 82.9</td><td>95.5</td><td>89.3</td><td>95.3</td><td colspan="2">84.0 0.803 / 0.763</td></tr><tr><td>en-es</td><td colspan="2">80.1 85.9</td><td>94.6</td><td>86.5</td><td>96.2</td><td colspan="2">85.2 0.809 / 0.770</td></tr><tr><td>en-de</td><td colspan="2">78.8 84.0</td><td>95.9</td><td>87.6</td><td>96.1</td><td colspan="2">85.0 0.802 / 0.764</td></tr><tr><td/><td/><td colspan="3">Translation-ranking Models</td><td/><td/></tr><tr><td>en-fr</td><td colspan="2">68.7 79.3</td><td>87.0</td><td>81.8</td><td>89.4</td><td colspan="2">74.2 0.668 / 0.558</td></tr><tr><td>en-es</td><td colspan="2">67.7 75.7</td><td>83.5</td><td>86.0</td><td>94.4</td><td colspan="2">72.6 0.669 / 0.631</td></tr><tr><td>en-de</td><td colspan="2">67.8 75.2</td><td>84.4</td><td>83.6</td><td>86.8</td><td colspan="2">74.6 0.673 / 0.632</td></tr><tr><td/><td/><td colspan="3">State-of-the-art Models</td><td/><td/></tr><tr><td>InferSent</td><td colspan="2">81.1 86.3</td><td>92.4</td><td>90.2</td><td>88.2</td><td colspan="2">84.6 0.801 / 0.758</td></tr><tr><td colspan="3">Skip-Thought LN 79.4 83.1</td><td>93.7</td><td>89.3</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Quick-Thought</td><td colspan="2">82.4 86.0</td><td>94.8</td><td>90.2</td><td>92.4</td><td>87.6</td><td>-</td></tr><tr><td colspan="3">USE Transformer 81.4 87.4</td><td>93.9</td><td>87.0</td><td>92.5</td><td colspan="2">85.4 0.814 / 0.782</td></tr></table>

Pearson's correlation coefficients on translated STS Benchmark and STS17 tasks. The first column shows the results on the original STS Benchmark data in English.

Table content: <table><tr><td>Model</td><td>en-en</td><td colspan="4">Translated STS Benchmark (dev / test) fr-fr es-es</td><td>de-de</td><td>STS17 es-es es-en</td></tr><tr><td>Multi-task en-fr</td><td colspan="3">0.803 / 0.763 0.777 / 0.738</td><td>-</td><td/><td>-</td><td>-</td><td>-</td></tr><tr><td>Trans.-ranking en-fr</td><td colspan="3">0.668 / 0.558 0.641 / 0.579</td><td>-</td><td/><td>-</td><td>-</td><td>-</td></tr><tr><td>Multi-task en-es</td><td colspan="2">0.809 / 0.770</td><td>-</td><td>0.779 / 0.744</td><td/><td>-</td><td>0.827 0.769</td></tr><tr><td colspan="3">Trans.-ranking en-es 0.669 / 0.631</td><td>-</td><td>0.622 / 0.611</td><td/><td>-</td><td>0.642 0.587</td></tr><tr><td>Multi-task en-de</td><td colspan="2">0.802 / 0.764</td><td>-</td><td>-</td><td colspan="2">0.768 / 0.722</td><td>-</td><td>-</td></tr><tr><td colspan="3">Trans.-ranking en-de 0.673 / 0.632</td><td>-</td><td>-</td><td colspan="2">0.630 / 0.526</td><td>-</td><td>-</td></tr><tr><td>ECNU</td><td>-</td><td/><td>-</td><td>-</td><td/><td>-</td><td>0.856 0.813</td></tr><tr><td>BIT</td><td>-</td><td/><td>-</td><td>-</td><td/><td>-</td><td>0.846 0.749</td></tr></table>

Sentiment classification accuracy(%) on target language Amazon review test data after training on English Amazon review data and a portion of French of German data. The second row shows the percent of French (fr) or German (de) data is used for training in each model.

Table content: <table><tr><td>Model</td><td>0%</td><td colspan="2">fr 10% 20% 40% 80% 100%</td><td>0%</td><td colspan="2">de 10% 20% 40% 80% 100%</td></tr><tr><td colspan="3">Few-shot 82.3 84.4 84.4 84.8 85.2</td><td>85.8</td><td colspan="2">81.0 81.6 83.3 84.0 84.7</td><td>84.5</td></tr><tr><td>Baseline</td><td>-</td><td>79.2 80.0 82.7 84.3</td><td>84.9</td><td>-</td><td>75.5 77.7 81.6 83.5</td><td>84.4</td></tr></table>

Multi-Perspective Question Answering (MPQA): Multi-class opinion polarity prediction on news data from Wiebe et al. (2005). • Text REtrieval Conference (TREC): Multi-class classification of questions obtained from TREC by Li & Roth (2002). • Stanford Sentiment Treebank (SST): Phrase-level binary sentiment classification of text from Socher et al. (2013). • Semantic Text Similarity (STS): Semantic textual similarity of sentence pairs in the form of Pearson correlation with human judgments

Table content: <table/>

English task performance of different model configurations. Notably, removing non-English tasks in encoder training actually hurts performance on downstream English tasks.

Table content: <table><tr><td colspan="2">Model MR</td><td>CR</td><td colspan="4">SUBJ MPQA TREC SST</td><td>STS Benchmark (dev / test)</td></tr><tr><td/><td/><td colspan="5">Cross-lingual Multitask Transformer</td></tr><tr><td/><td/><td/><td colspan="4">SNLI + Native Tasks (report above)</td></tr><tr><td>en-fr</td><td colspan="2">77.9 82.9</td><td>95.5</td><td>89.3</td><td>95.3</td><td>84.0</td><td>0.803 / 0.763</td></tr><tr><td>en-es</td><td colspan="2">80.1 85.9</td><td>94.6</td><td>86.5</td><td>96.2</td><td>85.2</td><td>0.809 / 0.770</td></tr><tr><td>en-de</td><td colspan="2">78.8 84.0</td><td>95.9</td><td>87.6</td><td>96.1</td><td>85.0</td><td>0.802 / 0.764</td></tr><tr><td/><td/><td colspan="5">Cross-lingual Multitask Transformer</td></tr><tr><td/><td/><td/><td/><td>No SNLI</td><td/><td/></tr><tr><td>en-fr</td><td colspan="2">80.5 86.5</td><td>94.0</td><td>89.1</td><td>96.6</td><td>85.0</td><td>0.747 / 0.722</td></tr><tr><td>en-es</td><td colspan="2">80.0 89.4</td><td>95.6</td><td>90.5</td><td>94.5</td><td>85.7</td><td>0.754 / 0.730</td></tr><tr><td>en-de</td><td colspan="2">80.5 85.5</td><td>94.0</td><td>90.0</td><td>92.8</td><td>82.6</td><td>0.737 / 0.723</td></tr><tr><td/><td/><td colspan="5">Cross-lingual Multitask Transformer</td></tr><tr><td/><td/><td/><td/><td>No Wiki</td><td/><td/></tr><tr><td>en-fr</td><td colspan="2">78.9 85.8</td><td>92.1</td><td>90.0</td><td>94.7</td><td>82.7</td><td>0.804 / 0.774</td></tr><tr><td>en-es</td><td colspan="2">79.1 87.4</td><td>94.5</td><td>89.6</td><td>94.3</td><td>83.0</td><td>0.809 / 0.769</td></tr><tr><td>en-de</td><td colspan="2">79.6 85.9</td><td>94.1</td><td>89.7</td><td>92.2</td><td>82.5</td><td>0.801 / 0.741</td></tr><tr><td/><td/><td colspan="3">Cross-lingual Multitask</td><td/><td/></tr><tr><td/><td/><td/><td colspan="3">No non-English native tasks</td><td/></tr><tr><td>en-fr</td><td colspan="2">79.4 84.0</td><td>93.5</td><td>89.7</td><td>93.8</td><td>83.4</td><td>0.797 / 0.758</td></tr><tr><td>en-es</td><td colspan="2">77.4 81.7</td><td>94.9</td><td>89.4</td><td>94.0</td><td>82.3</td><td>0.796 / 0.761</td></tr><tr><td>en-de</td><td colspan="2">79.3 83.7</td><td>94.0</td><td>88.1</td><td>91.2</td><td>82.5</td><td>0.760 / 0.732</td></tr><tr><td/><td/><td/><td colspan="3">Translation-ranking Models</td><td/></tr><tr><td>en-fr</td><td colspan="2">68.7 79.3</td><td>87.0</td><td>81.8</td><td>89.4</td><td>74.2</td><td>0.668 / 0.558</td></tr><tr><td>en-es</td><td colspan="2">67.7 75.7</td><td>83.5</td><td>86.0</td><td>94.4</td><td>72.6</td><td>0.669 / 0.631</td></tr><tr><td>en-de</td><td colspan="2">67.8 75.2</td><td>84.4</td><td>83.6</td><td>86.8</td><td>74.6</td><td>0.673 / 0.632</td></tr></table>

Zero-shot sentiment classification accuracy(%) on target language Amazon review test data after training on only English Amazon review data.

Table content: <table><tr><td>Model</td><td>en</td><td>fr</td><td>de</td></tr><tr><td>Multi-task en-fr</td><td colspan="2">87.4 82.3</td><td>-</td></tr><tr><td colspan="3">Multi-task No non-English native tasks en-fr 85.6 65.4</td><td>-</td></tr><tr><td>Translation-ranking en-fr</td><td colspan="2">74.4 66.3</td><td>-</td></tr><tr><td>Multi-task en-de</td><td>87.1</td><td>-</td><td>81.0</td></tr><tr><td colspan="2">Multi-task No non-English native tasks en-fr 85.1</td><td>-</td><td>67.2</td></tr><tr><td>Translation-ranking en-de</td><td>73.8</td><td>-</td><td>67.0</td></tr></table>

Peer review (iclr2019):
 This paper proposes a novel cross-lingual multi-tasking framework based on a dual-encoder model that can learn cross-lingual sentence representations which are useful in monolingual tasks and cross-lingual tasks for both languages involved in the training, as observed on the experiments for three language pairs. The main idea of the approach is to model all tasks as input-response ranking tasks and introduce cross-lingual representation tying through the translation ranking task, introduced by Guo et al. (2018). All components of the framework are quite standard and deja-vu, but I like the paper in general, and the results seem quite encouraging. I have several comments on how to further strengthen the paper and improve the presentation of the main findings. The proposed framework does not offer any substantial modeling contribution (i.e., all major components are based on SOTA models), but the framework is still quite interesting as a mixture of these SOTA components. I believe that some additional experiments would make the main contributions clearer and would also provide additional insights into the main properties of the proposed framework: 1) cross-linguality and 2) multi-tasking. *Most of all, I am surprised not to see any ablation studies. For instance, what happens if we remove one of the two monolingual tasks in each language? How does that reduced model compare to the full model? Which monolingual task is more beneficial for the final performance in downstream tasks? Can we think of adding another monolingual task to boost performance further? I think that this sort of experiment would be more beneficial for the paper than a pretty long analysis from Section 5 (this analysis is still valid, but should be shortened substantially). Evaluating only multi-tasking without any cross-lingual training would also be very beneficial to recognise the extent of improvement achieved by adding cross-linguality to the model. *How much does the proposed architecture depend on the choice of the encoding model for the function g? Have the authors experimented with other (recent and (near-)SOTA) encoding models? I would like to see a comparative analysis of this *hyper-parameter*. *I would like to see more experiments on more distant language pairs. This would make the paper even more interesting imho. I am also curious whether there would be a drop in performance reported conditioned on the distance/proximity between two languages in a language pair. *I would like to see a more detailed description of the two best performing STS systems (ECNU and BIT). In what respect are these systems state-of-the-art feature engineered and mixed? I am not sure what this means without providing any additional context to the claim and description. *How does the monolingual English STS model trained with the cross-lingual multi-task framework compare to the work of Conneau et al. (EMNLP 2017) which also used SNLI as the task on which to learn universal sentence representations. This would be a good experiment imho as it would show how much we gain from cross-lingual training and multi-tasking. Minor: *Page 3: Could you add a short footnote discussing how hard-negatives for the translation ranking task are selected? How do you compute similarity here? *Do you expect performance to improve further by training MultiNLI instead of SNLI (or combining the two datasets)? *"All hyperparameters are tuned based on preliminary experiments on a development set." -> What is used as the development set? More details needed. *"Finally, as an additional training heuristic, we multiply the gradients to the word and character embeddings by a factor of 100." -> How is the value for the embedding gradient multiplier determined? Is there an automatic procedure to fine-tune this hyper-parameter or has this been done in a completely empirical way? *Table 1: please define the task abbreviations before showing them in the table. It is not clear what each task is by relying only on the abbreviation. *This dataset was not available at the time of the submission, but for the revision it would make sense to also evaluate on the new XNLI dataset of Conneau et al. (EMNLP 2018) for multilingual NLI experiments. (After the first revision) I have raised the score after the very detailed author response (thanks for that!), but this is also conditioned on the authors making the actual revisions promised in their response. I am still quite interested to check how well the method works in a setup with distant language pairs.