Multi-task training setup	Multi-task training with different data sources for each task stills poses open questions. For example: When does one switch to training on a different task? Should the switching be periodic? Do we weight each task equally? If not, what training ratios do we use?   use periodic task alternations with equal training ratios for every task. In contrast,  alter the training ratios for each task based on the size of their respective training sets. Specifically, the training ratio for a particular task, INLINEFORM0 , is the fraction of the number of training examples in that task to the total number of training samples across all tasks. The authors then perform INLINEFORM1 parameter updates on task INLINEFORM2 before selecting a new task at random proportional to the training ratios, where N is a predetermined constant. We take a simpler approach and pick a new sequence-to-sequence task to train on after every parameter update sampled uniformly. An NLI minibatch is interspersed after every ten parameter updates on sequence-to-sequence tasks (this was chosen so as to complete roughly 6 epochs of the dataset after 7 days of training). Our approach is described formally in the Algorithm below.

Semantic Textual Similarity	In this evaluation, we measure the relatedness of two sentences using only the cosine similarity between their representations. We use the similarity textual similarity (STS) benchmark tasks from 2012-2016 (STS12, STS13, STS14, STS15, STS16, STSB). The STS dataset contains sentences from a diverse set of data sources. The evaluation criteria is Pearson correlation.

Quora Duplicate Question Classification	In addition to the above tasks which were considered by  , we also evaluate on the recently published Quora duplicate question dataset since it is an order of magnitude larger than the others (approximately 400,000 question pairs). The task is to correctly identify question pairs that are duplicates of one another, which we formulate as a binary classification problem. We use the same data splits as in  . Given the size of this data, we consider a more expressive classifier on top of the representations of both questions. Specifically, we train a 4 layer MLP with 1024 hidden units, with a dropout rate of 0.5 after every hidden layer. The evaluation criterion is classification accuracy. We also artificially create a low-resource setting by reducing the number of training examples between 1,000 and 25,000 using the same splits as  .

Models were trained for 7 days on an Nvidia Tesla P100-SXM2-16GB GPU. While  report close to a month of training, we only train for 7 days, made possible by advancements in GPU hardware and software (cuDNN RNNs). We did not tune any of the architectural details and hyperparameters owing to the fact that we were unable to identify any clear criterion on which to tune them. Gains in performance on a specific task do not often translate to better transfer performance.

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning	A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

Related Work	The problem of learning distributed representations of phrases and sentences dates back over a decade. For example,  present an additive and multiplicative linear composition function of the distributed representations of individual words.  combine symbolic and distributed representations of words using tensor products. Advances in learning better distributed representations of words  ,  combined with deep learning have made it possible to learn complex non-linear composition functions of an arbitrary number of word embeddings using convolutional or recurrent neural networks (RNNs). A network's representation of the last element in a sequence, which is a non-linear composition of all inputs, is typically assumed to contain a squashed “summary” of the sentence. Most work in supervised learning for NLP builds task-specific representations of sentences rather than general-purpose ones. Notably, skip-thought vectors  , an extension of the skip-gram model for word embeddings  to sentences, learn re-usable sentence representations from weakly labeled data. Unfortunately, these models take weeks or often months to train.  address this by considering faster alternatives such as sequential denoising autoencoders and shallow log-linear models.  , however, demonstrate that simple word embedding averages are comparable to more complicated models like skip-thoughts.

In this work, as in  ,  , we do not employ an attention mechanism. This enables us to obtain a single, fixed-length, distributed sentence representation. To diminish the effects of vanishing gradient, we condition every decoding step on the encoder hidden representation INLINEFORM0 . We use a GRU for the encoder and decoder in the interest of computational speed. The encoder is a bidirectional GRU while the decoder is a unidirectional conditional GRU whose parameterization is as follows: DISPLAYFORM0  The encoder representation INLINEFORM0 is provided as conditioning information to the reset gate, update gate and hidden state computation in the GRU via the parameters INLINEFORM1 , INLINEFORM2 and INLINEFORM3 to avoid attenuation of information from the encoder.

Image-caption retrieval	Image-caption retrieval is typically formulated as a ranking task wherein images are retrieved and ranked based on a textual description and vice-versa. We use 113k training images from MSCOCO with 5k images for validation and 5k for testing. Image features are extracted using a pre-trained 110 layer ResNet. The evaluation criterion is Recall@K and the median K across 5 different splits of the data.

Model details can be found in section SECREF7 in the Appendix. 0ptcenterline   A set of INLINEFORM0 tasks with a common source language, a shared encoder INLINEFORM1 across all tasks and a set of INLINEFORM2 task specific decoders INLINEFORM3 . Let INLINEFORM4 denote each model's parameters, INLINEFORM5 a probability vector ( INLINEFORM6 ) denoting the probability of sampling a task such that INLINEFORM7 , datasets for each task INLINEFORM8 and a loss function INLINEFORM9 .   INLINEFORM0 has not converged [1] Sample task INLINEFORM1 . Sample input, output pairs INLINEFORM2 . Input representation INLINEFORM3 . Prediction INLINEFORM4 INLINEFORM5 Adam INLINEFORM6 .

Entailment and Semantic Relatedness	To test if similar sentences share similar representations, we evaluate on the SICK relatedness (SICK-R) task where a linear model is trained to output a score from 1 to 5 indicating the relatedness of two sentences. We also evaluate using the entailment labels in the same dataset (SICK-E) which is a binary classification problem. The evaluation metric for SICK-R is Pearson correlation and classification accuracy for SICK-E.

In Table TABREF20 , we probe our sentence representations to determine if certain sentence characteristics and syntactic properties can be inferred following work by  and  . We observe that syntactic properties are better encoded with the addition of multi-lingual NMT and parsing. Representations learned solely from NLI do appear to encode syntax but incorporation into our multi-task framework does not amplify this signal. Similarly, we observe that sentence characteristics such as length and word order are better encoded with the addition of parsing. In Appendix Table TABREF30 , we note that our sentence representations outperform skip-thoughts and are on par with Infersent for image-caption retrieval. We also observe that comparing sentences using cosine similarities correlates reasonably well with their relatedness on semantic textual similarity benchmarks (Appendix Table TABREF31 ). We also present qualitative analysis of our learned representations by visualizations using dimensionality reduction techniques (Figure FIGREF11 ) and nearest neighbor exploration (Appendix Table TABREF32 ). Figure FIGREF11 shows t-sne plots of our sentence representations on three different datasets - SUBJ, TREC and DBpedia. DBpedia is a large corpus of sentences from Wikipedia labeled by category and used by  .

Multi-task Sequence-to-sequence Learning	  present a simple one-to-many multi-task sequence-to-sequence learning model for NMT that uses a shared encoder for English and task-specific decoders for multiple target languages.  extend this by also considering many-to-one (many encoders, one decoder) and many-to-many architectures. In this work, we consider a one-to-many model since it lends itself naturally to the idea of combining inductive biases from different training objectives. The same bidirectional GRU encodes the input sentences from different tasks into a compressed summary INLINEFORM0 which is then used to condition a task-specific GRU to produce the output sentence.

First, like  , their use of an attention mechanism prevents learning a fixed-length vector representation for a sentence. Second, their work aims for improvements on the same tasks on which the model is trained, as opposed to learning re-usable sentence representations that transfer elsewhere. We further present a fine-grained analysis of how different tasks contribute to the encoding of different information signals in our representations following work by  and  .   similarly present a multi-task framework for textual entailment with task supervision at different levels of learning. “Universal" multi-task models have also been successfully explored in the context of computer vision problems  ,  .

+STN +Fr +De +NLI +L +STP : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without STP. +STN +Fr +De +NLI +2L +STP : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a 2-layer bidirectional GRU with 2048-dimensional hidden vectors and a 1-layer bidirectional GRU with 2048-dimensional hidden vectors trained without STP. +STN +Fr +De +NLI +L +STP +Par : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without Par. In tables TABREF18 and TABREF20 we do not concatenate the representations of multiple models.

Evaluation Strategy	We follow a similar evaluation protocol to those presented in  ,  ,  which is to use our learned representations as features for a low complexity classifier (typically linear) on a novel supervised task/domain unseen during training without updating the parameters of our sentence representation model. We also consider such a transfer learning evaluation in an artificially constructed low-resource setting. In addition, we also evaluate the quality of our learned individual word representations using standard benchmarks  ,  . The choice of transfer tasks and evaluation framework are borrowed largely from  . We provide a condensed summary of the tasks in section SECREF10 in the Appendix but refer readers to their paper for a more detailed description. https://github.com/kudkudak/word-embeddings-benchmarks/wiki

The addition of constituency parsing improves performance on sentence relatedness (SICK-R) and entailment (SICK-E) consistent with observations made by  . In Table TABREF19 , we show that simply training an MLP on top of our fixed sentence representations outperforms several strong & complex supervised approaches that use attention mechanisms, even on this fairly large dataset. For example, we observe a 0.2-0.5% improvement over the decomposable attention model  . When using only a small fraction of the training data, indicated by the columns 1k-25k, we are able to outperform the Siamese and Multi-Perspective CNN using roughly 6% of the available training set. We also outperform the Deconv LVM model proposed by  in this low-resource setting. Unlike  , who use pretrained GloVe word embeddings, we learn our word embeddings from scratch. Somewhat surprisingly, in Table TABREF18 we observe that the learned word embeddings are competitive with popular methods such as GloVe, word2vec, and fasttext  on the benchmarks presented by  and  .

We merely hope to capture some of its characteristics that could be of use in a variety of tasks. The primary contribution of our work is to combine the benefits of diverse sentence-representation learning objectives into a single multi-task framework. To the best of our knowledge, this is the first large-scale reusable sentence representation model obtained by combining a set of training objectives with the level of diversity explored here, i.e. multi-lingual NMT, natural language inference, constituency parsing and skip-thought vectors. We demonstrate through extensive experimentation that representations learned in this way lead to improved performance across a diverse set of novel tasks not used in the learning of our representations. Such representations facilitate low-resource learning as exhibited by significant improvements to model performance for new tasks in the low labelled data regime - achieving comparable performance to a few models trained from scratch using only 6% of the available training set on the Quora duplicate question dataset.

Vocabulary Expansion & Representation Pooling	In addition to performing 10-fold cross-validation to determine the L2 regularization penalty on the logistic regression models, we also tune the way in which our sentence representations are generated from the hidden states corresponding to words in a sentence. For example,  use the last hidden state while  perform max-pooling across all of the hidden states. We consider both of these approaches and pick the one with better performance on the validation set. We note that max-pooling works best on sentiment tasks such as MR, CR, SUBJ and MPQA, while the last hidden state works better on all other tasks. We also employ vocabulary expansion on all tasks as in  by training a linear regression to map from the space of pre-trained word embeddings (GloVe) to our model's word embeddings.

also present evidence that sequence-to-sequence parsers  more strongly encode source language syntax. Similarly,  probe representations extracted by sequence autoencoders, word embedding averages, and skip-thought vectors with a multi-layer perceptron (MLP) classifier to study whether sentence characteristics such as length, word content and word order are encoded. To generalize across a diverse set of tasks, it is important to build representations that encode several aspects of a sentence. Neural approaches to tasks such as skip-thoughts, machine translation, natural language inference, and constituency parsing likely have different inductive biases. Our work exploits this in the context of a simple one-to-many multi-task learning (MTL) framework, wherein a single recurrent sentence encoder is shared across multiple tasks. We hypothesize that sentence representations learned by training on a reasonably large number of weakly related tasks will generalize better to novel tasks unseen during training, since this process encodes the inductive biases of multiple models. This hypothesis is based on the theoretical work of  . While our work aims at learning fixed-length distributed sentence representations, it is not always practical to assume that the entire “meaning” of a sentence can be encoded into a fixed-length vector.

More recently,  show that a completely supervised approach to learning sentence representations from natural language inference data outperforms all previous approaches on transfer learning benchmarks. Here we use the terms “transfer learning performance" on “transfer tasks” to mean the performance of sentence representations evaluated on tasks unseen during training.  demonstrated that representations learned by state-of-the-art large-scale NMT systems also generalize well to other tasks. However, their use of an attention mechanism prevents the learning of a single fixed-length vector representation of a sentence. As a result, they present a bi-attentive classification network that composes information present in all of the model's hidden states to achieve improvements over a corresponding model trained from scratch.  and  demonstrate that discourse-based objectives can also be leveraged to learn good sentence representations. Our work is most similar to that of  , who train a many-to-many sequence-to-sequence model on a diverse set of weakly related tasks that includes machine translation, constituency parsing, image captioning, sequence autoencoding, and intra-sentence skip-thoughts. There are two key differences between that work and our own.

Sequence-to-Sequence Learning	Five out of the six tasks that we consider for multi-task learning are formulated as sequence-to-sequence problems  ,  . Briefly, sequence-to-sequence models are a specific case of encoder-decoder models where the inputs and outputs are sequential. They directly model the conditional distribution of outputs given inputs INLINEFORM0 . The input INLINEFORM1 and output INLINEFORM2 are sequences INLINEFORM3 and INLINEFORM4 . The encoder produces a fixed length vector representation INLINEFORM5 of the input, which the decoder then conditions on to generate an output. The decoder is auto-regressive and breaks down the joint probability of outputs into a product of conditional probabilities via the chain rule: INLINEFORM6    and  use encoders and decoders parameterized as RNN variants such as Long Short-term Memory (LSTMs)  or Gated Recurrent Units (GRUs)  . The hidden representation INLINEFORM0 is typically the last hidden state of the encoder RNN.   alleviate the gradient bottleneck between the encoder and the decoder by introducing an attention mechanism that allows the decoder to condition on every hidden state of the encoder RNN instead of only the last one.

Model Training	We present some architectural specifics and training details of our multi-task framework. Our shared encoder uses a common word embedding lookup table and GRU. We experiment with unidirectional, bidirectional and 2 layer bidirectional GRUs (details in Appendix section SECREF9 ). For each task, every decoder has its separate word embedding lookups, conditional GRUs and fully connected layers that project the GRU hidden states to the target vocabularies. The last hidden state of the encoder is used as the initial hidden state of the decoder and is also presented as input to all the gates of the GRU at every time step. For natural language inference, the same encoder is used to encode both the premise and hypothesis and a concatenation of their representations along with the absolute difference and hadamard product (as described in  ) are given to a single layer MLP with a dropout  rate of 0.3. All models use word embeddings of 512 dimensions and GRUs with either 1500 or 2048 hidden units. We used minibatches of 48 examples and the Adam  optimizer with a learning rate of 0.002.

Acknowledgements	The authors would like to thank Chinnadhurai Sankar, Sebastian Ruder, Eric Yuan, Tong Wang, Alessandro Sordoni, Guillaume Lample and Varsha Embar for useful discussions. We are also grateful to the PyTorch development team  . We thank NVIDIA for donating a DGX-1 computer used in this work and Fonds de recherche du Québec - Nature et technologies for funding.

Description of evaluation tasks	  and  provide a detailed description of tasks that are typically used to evaluate sentence representations. We provide a condensed summary and refer readers to their work for a more thorough description.

Sentence Characteristics & Syntax	In an attempt to understand what information is encoded in by sentence representations, we consider six different classification tasks where the objective is to predict sentence characteristics such as length, word content and word order  or syntactic properties such as active/passive, tense and the top syntactic sequence (TSS) from the parse tree of a sentence  . The sentence characteristic tasks are setup in the same way as described in  . The length task is an 8-way classification problem where sentence lengths are binned into 8 ranges. The content task is formulated as a binary classification problem that takes a concatenation of a sentence representation INLINEFORM0 and a word representation INLINEFORM1 to determine if the word is contained in the sentence. The order task is an extension of the content task where a concatenation of the sentence representation and word representations of two words in sentence is used to determine if the first word occurs before or after the second. We use a random subset of the 1-billion-word dataset for these experiments that were not used to train our multi-task representations. The syntactic properties tasks are setup in the same way as described in  .The passive and tense tasks are characterized as binary classification problems given a sentence's representation.

The former's objective is to determine if a sentence is written in active/passive voice while the latter's objective is to determine if the sentence is in the past tense or not. The top syntactic sequence (TSS) is a 20-way classification problem with 19 most frequent top syntactic sequences and 1 miscellaneous class. We use the same dataset as the authors but different training, validation and test splits.

Text Classification	We evaluate on text classification benchmarks - sentiment classification on movie reviews (MR), product reviews (CR) and Stanford sentiment (SST), question type classification (TREC), subjectivity/objectivity classification (SUBJ) and opinion polarity (MPQA). Representations are used to train a logistic regression classifier with 10-fold cross validation to tune the L2 weight penalty. The evaluation metric for all these tasks is classification accuracy.

Introduction	Transfer learning has driven a number of recent successes in computer vision and NLP. Computer vision tasks like image captioning  and visual question answering typically use CNNs pretrained on ImageNet  ,  to extract representations of the image, while several natural language tasks such as reading comprehension and sequence labeling  have benefited from pretrained word embeddings  ,  that are either fine-tuned for a specific task or held fixed. Many neural NLP systems are initialized with pretrained word embeddings but learn their representations of words in context from scratch, in a task-specific manner from supervised learning signals. However, learning these representations reliably from scratch is not always feasible, especially in low-resource settings, where we believe that using general purpose sentence representations will be beneficial. Some recent work has addressed this by learning general-purpose sentence representations  ,  ,  ,  ,  ,  ,  ,  . However, there exists no clear consensus yet on what training objective or methodology is best suited to this goal. Understanding the inductive biases of distinct neural models is important for guiding progress in representation learning.  and  demonstrate that neural machine translation (NMT) systems appear to capture morphology and some syntactic properties.

Experimental Results & Discussion	Table 2 presents the results of training logistic regression on 10 different supervised transfer tasks using different fixed-length sentence representation. Supervised approaches trained from scratch on some of these tasks are also presented for comparison. We present performance ablations when adding more tasks and increasing the number of hidden units in our GRU (+L). Ablation specifics are presented in section SECREF9 of the Appendix. It is evident from Table 2 that adding more tasks improves the transfer performance of our model. Increasing the capacity our sentence encoder with more hidden units (+L) as well as an additional layer (+2L) also lead to improved transfer performance. We observe gains of 1.1-2.0% on the sentiment classification tasks (MR, CR, SUBJ & MPQA) over Infersent. We demonstrate substantial gains on TREC (6% over Infersent and roughly 2% over the CNN-LSTM), outperforming even a competitive supervised baseline. We see similar gains (2.3%) on paraphrase identification (MPRC), closing the gap on supervised approaches trained from scratch.

Conclusion & Future Work	We present a multi-task framework for learning general-purpose fixed-length sentence representations. Our primary motivation is to encapsulate the inductive biases of several diverse training signals used to learn sentence representations into a single model. Our multi-task framework includes a combination of sequence-to-sequence tasks such as multi-lingual NMT, constituency parsing and skip-thought vectors as well as a classification task - natural language inference. We demonstrate that the learned representations yield competitive or superior results to previous general-purpose sentence representation methods. We also observe that this approach produces good word embeddings. In future work, we would like understand and interpret the inductive biases that our model learns and observe how it changes with the addition of different tasks beyond just our simple analysis of sentence characteristics and syntax. Having a rich, continuous sentence representation space could allow the application of state-of-the-art generative models of images such as that of  to language. One could also consider controllable text generation by directly manipulating the sentence representations and realizing it by decoding with a conditional language model.

Evaluation Strategies, Experimental Results & Discussion	In this section, we describe our approach to evaluate the quality of our learned representations, present the results of our evaluation and discuss our findings.

Sentences appear to cluster reasonably well according to their labels. The clustering also appears better than that demonstrated in Figure 2 of  on TREC and SUBJ. Appendix Table TABREF32 contains sentences from the BookCorpus and their nearest neighbors. Sentences with some lexical overlap and similar discourse structure appear to be clustered together.

Multi-task model details	This section describes the specifics our multi-task ablations in the experiments section. These definitions hold for all tables except for TABREF18 and TABREF20 . We refer to skip-thought next as STN, French and German NMT as Fr and De, natural language inference as NLI, skip-thought previous as STP and parsing as Par. +STN +Fr +De : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a forward GRU with 1500-dimensional hidden vectors and a bidirectional GRU, also with 1500-dimensional hidden vectors. +STN +Fr +De +NLI : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a bidirectional GRU with 1500-dimensional hidden vectors and another bidirectional GRU with 1500-dimensional hidden vectors trained without NLI. +STN +Fr +De +NLI +L : The sentence representation INLINEFORM0 is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without NLI.

Paraphrase Identification	We also evaluate on pairwise text classification tasks such as paraphrase identification on the Microsoft Research Paraphrase Corpus (MRPC) corpus. This is a binary classification problem to identify if two sentences are paraphrases of each other. The evaluation metric is classification accuracy and F1.

Training Objectives & Evaluation	Our motivation for multi-task training stems from theoretical insights presented in  . We refer readers to that work for a detailed discussion of results, but the conclusions most relevant to this discussion are (i) that learning multiple related tasks jointly results in good generalization as measured by the number of training examples required per task; and (ii) that inductive biases learned on sufficiently many training tasks are likely to be good for learning novel tasks drawn from the same environment. We select the following training objectives to learn general-purpose sentence embeddings. Our desiderata for the task collection were: sufficient diversity, existence of fairly large datasets for training, and success as standalone training objectives for sentence representations.

Question: Which data sources do they use?
Answer: - En-Fr (WMT14)
- En-De (WMT15)
- Skipthought (BookCorpus)
- AllNLI (SNLI + MultiNLI)
- Parsing (PTB + 1-billion word)