Title: LEARNING GENERAL PURPOSE DISTRIBUTED SEN-TENCE REPRESENTATIONS VIA LARGE SCALE MULTI-TASK LEARNING

Abstract: A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

1. INTRODUCTION:

Transfer learning has driven a number of recent successes in computer vision and NLP. Computer vision tasks like image captioning (Xu et al., 2015) and visual question answering typically use CNNs pretrained on ImageNet (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) to extract representations of the image, while several natural language tasks such as reading comprehension and sequence labeling (Lample et al., 2016) have benefited from pretrained word embeddings (Mikolov et al., 2013; Pennington et al., 2014) that are either fine-tuned for a specific task or held fixed.

Many neural NLP systems are initialized with pretrained word embeddings but learn their representations of words in context from scratch, in a task-specific manner from supervised learning signals. However, learning these representations reliably from scratch is not always feasible, especially in low-resource settings, where we believe that using general purpose sentence representations will be beneficial.

Some recent work has addressed this by learning general-purpose sentence representations (Kiros et al., 2015; Wieting et al., 2015; Hill et al., 2016; Conneau et al., 2017; McCann et al., 2017; Jernite et al., 2017; Nie et al., 2017; Pagliardini et al., 2017) . However, there exists no clear consensus yet on what training objective or methodology is best suited to this goal.

Understanding the inductive biases of distinct neural models is important for guiding progress in representation learning. Shi et al. (2016) and Belinkov et al. (2017) demonstrate that neural machine translation (NMT) systems appear to capture morphology and some syntactic properties. Shi et al. (2016) also present evidence that sequence-to-sequence parsers (Vinyals et al., 2015) more strongly encode source language syntax. Similarly, Adi et al. (2016) probe representations extracted by sequence autoencoders, word embedding averages, and skip-thought vectors with a multi-layer perceptron (MLP) classifier to study whether sentence characteristics such as length, word content and word order are encoded.

To generalize across a diverse set of tasks, it is important to build representations that encode several aspects of a sentence. Neural approaches to tasks such as skip-thoughts, machine translation, natural language inference, and constituency parsing likely have different inductive biases. Our work exploits this in the context of a simple one-to-many multi-task learning (MTL) framework, wherein a single recurrent sentence encoder is shared across multiple tasks. We hypothesize that sentence representations learned by training on a reasonably large number of weakly related tasks will generalize better to novel tasks unseen during training, since this process encodes the inductive biases of multiple models. This hypothesis is based on the theoretical work of Baxter et al. (2000) . While our work aims at learning fixed-length distributed sentence representations, it is not always practical to assume that the entire "meaning" of a sentence can be encoded into a fixed-length vector. We merely hope to capture some of its characteristics that could be of use in a variety of tasks.

The primary contribution of our work is to combine the benefits of diverse sentence-representation learning objectives into a single multi-task framework. To the best of our knowledge, this is the first large-scale reusable sentence representation model obtained by combining a set of training objectives with the level of diversity explored here, i.e. multi-lingual NMT, natural language inference, constituency parsing and skip-thought vectors. We demonstrate through extensive experimentation that representations learned in this way lead to improved performance across a diverse set of novel tasks not used in the learning of our representations. Such representations facilitate low-resource learning as exhibited by significant improvements to model performance for new tasks in the low labelled data regime -achieving comparable performance to a few models trained from scratch using only 6% of the available training set on the Quora duplicate question dataset.

2. RELATED WORK:

The problem of learning distributed representations of phrases and sentences dates back over a decade. For example, Mitchell & Lapata (2008) present an additive and multiplicative linear composition function of the distributed representations of individual words. Clark & Pulman (2007) combine symbolic and distributed representations of words using tensor products. Advances in learning better distributed representations of words (Mikolov et al., 2013; Pennington et al., 2014) combined with deep learning have made it possible to learn complex non-linear composition functions of an arbitrary number of word embeddings using convolutional or recurrent neural networks (RNNs). A network's representation of the last element in a sequence, which is a non-linear composition of all inputs, is typically assumed to contain a squashed "summary" of the sentence. Most work in supervised learning for NLP builds task-specific representations of sentences rather than general-purpose ones.

Notably, skip-thought vectors (Kiros et al., 2015) , an extension of the skip-gram model for word embeddings (Mikolov et al., 2013) to sentences, learn re-usable sentence representations from weakly labeled data. Unfortunately, these models take weeks or often months to train. Hill et al. (2016) address this by considering faster alternatives such as sequential denoising autoencoders and shallow log-linear models. Arora et al. (2016) , however, demonstrate that simple word embedding averages are comparable to more complicated models like skip-thoughts. More recently, Conneau et al. (2017) show that a completely supervised approach to learning sentence representations from natural language inference data outperforms all previous approaches on transfer learning benchmarks. Here we use the terms "transfer learning performance" on "transfer tasks" to mean the performance of sentence representations evaluated on tasks unseen during training. McCann et al. (2017) demonstrated that representations learned by state-of-the-art large-scale NMT systems also generalize well to other tasks. However, their use of an attention mechanism prevents the learning of a single fixedlength vector representation of a sentence. As a result, they present a bi-attentive classification network that composes information present in all of the model's hidden states to achieve improvements over a corresponding model trained from scratch. Jernite et al. (2017) and Nie et al. (2017) demonstrate that discourse-based objectives can also be leveraged to learn good sentence representations.

Our work is most similar to that of Luong et al. (2015) , who train a many-to-many sequenceto-sequence model on a diverse set of weakly related tasks that includes machine translation, constituency parsing, image captioning, sequence autoencoding, and intra-sentence skip-thoughts. There are two key differences between that work and our own. First, like McCann et al. (2017) , their use of an attention mechanism prevents learning a fixed-length vector representation for a sentence. Second, their work aims for improvements on the same tasks on which the model is trained, as opposed to learning re-usable sentence representations that transfer elsewhere.

We further present a fine-grained analysis of how different tasks contribute to the encoding of different information signals in our representations following work by Shi et al. (2016) and Adi et al. (2016) . Hashimoto et al. (2016) similarly present a multi-task framework for textual entailment with task supervision at different levels of learning. "Universal" multi-task models have also been successfully explored in the context of computer vision problems (Kokkinos, 2016; Eigen & Fergus, 2015) .

3. SEQUENCE-TO-SEQUENCE LEARNING:

Five out of the six tasks that we consider for multi-task learning are formulated as sequence-tosequence problems (Cho et al., 2014; Sutskever et al., 2014) . Briefly, sequence-to-sequence models are a specific case of encoder-decoder models where the inputs and outputs are sequential. They directly model the conditional distribution of outputs given inputs P (y|x). The input x and output y are sequences x 1 , x 2 , . . . , x m and y 1 , y 2 . . . , y n . The encoder produces a fixed length vector representation h x of the input, which the decoder then conditions on to generate an output. The decoder is auto-regressive and breaks down the joint probability of outputs into a product of conditional probabilities via the chain rule: Cho et al. (2014) and Sutskever et al. (2014) use encoders and decoders parameterized as RNN variants such as Long Short-term Memory (LSTMs) (Hochreiter & Schmidhuber, 1997) or Gated Recurrent Units (GRUs) (Chung et al., 2014) . The hidden representation h x is typically the last hidden state of the encoder RNN. Bahdanau et al. (2014) alleviate the gradient bottleneck between the encoder and the decoder by introducing an attention mechanism that allows the decoder to condition on every hidden state of the encoder RNN instead of only the last one. In this work, as in Kiros et al. (2015) ; Hill et al. (2016) , we do not employ an attention mechanism. This enables us to obtain a single, fixed-length, distributed sentence representation. To diminish the effects of vanishing gradient, we condition every decoding step on the encoder hidden representation h x . We use a GRU for the encoder and decoder in the interest of computational speed. The encoder is a bidirectional GRU while the decoder is a unidirectional conditional GRU whose parameterization is as follows:

P (y|x) = n i=1 P (y i |y <i , h x )

r t = σ(W r x t + U r h t-1 + C r h x ) (1) z t = σ(W z x t + U z h t-1 + C z h x ) (2) ht = tanh(W d x t + U d (r t h t-1 ) + C d h x )

(3)

EQUATION

The encoder representation h x is provided as conditioning information to the reset gate, update gate and hidden state computation in the GRU via the parameters C r , C z and C d to avoid attenuation of information from the encoder.

3.1 MULTI-TASK SEQUENCE-TO-SEQUENCE LEARNING Dong et al. (2015) present a simple one-to-many multi-task sequence-to-sequence learning model for NMT that uses a shared encoder for English and task-specific decoders for multiple target languages. Luong et al. (2015) extend this by also considering many-to-one (many encoders, one decoder) and many-to-many architectures. In this work, we consider a one-to-many model since it lends itself naturally to the idea of combining inductive biases from different training objectives. The same bidirectional GRU encodes the input sentences from different tasks into a compressed summary h x which is then used to condition a task-specific GRU to produce the output sentence.

4. TRAINING OBJECTIVES & EVALUATION:

Our motivation for multi-task training stems from theoretical insights presented in Baxter et al. (2000) . We refer readers to that work for a detailed discussion of results, but the conclusions most relevant to this discussion are (i) that learning multiple related tasks jointly results in good generalization as measured by the number of training examples required per task; and (ii) that inductive biases learned on sufficiently many training tasks are likely to be good for learning novel tasks drawn from the same environment.

We select the following training objectives to learn general-purpose sentence embeddings. Our desiderata for the task collection were: sufficient diversity, existence of fairly large datasets for training, and success as standalone training objectives for sentence representations.

Skip-thought vectors Skip-thought vectors (Kiros et al., 2015) are an extension of skip-gram word embedding models to sentences. The task typically requires a corpus of contiguous sentences, for which we use the BookCorpus (Zhu et al., 2015) . The learning objective is to simultaneously predict the next and previous sentences from the current sentence. The encoder for the current sentence and the decoders for the previous (STP) and next sentence (STN) are typically parameterized as separate RNNs. We also consider training skip-thoughts by predicting only the next sentence given the current, motivated by results in Tang et al. (2017) where it is demonstrated that predicting the next sentence alone leads to comparable performance.

Neural Machine Translation Sutskever et al. (2014) and Cho et al. (2014) demonstrated that NMT can be formulated as a sequence-to-sequence learning problem where the input is a sentence in the source language and the output is its corresponding translation in the target language. We use a parallel corpus of around 4.5 million English-German (De) sentence pairs from WMT15 and 40 million English-French (Fr) sentence pairs from WMT14. We train our representations using multiple target languages motivated by improvements demonstrated by Dong et al. (2015) .

Constituency Parsing (linearized parse tree construction) Vinyals et al. (2015) demonstrated that a sequence-to-sequence approach to constituency parsing is viable. The input to the encoder is the sentence itself and the decoder produces its linearized parse tree. We train on 3 million weakly labeled parses obtained by parsing a random subset of the 1-billion word corpus with the Puck GPU parser1 along with gold parses from sections 0-21 of the WSJ section of Penn Treebank. Gold parses are duplicated 5 times and shuffled in with the weakly labeled parses to have a roughly 1:5 ratio of gold to noisy parses.

Natural Language Inference Natural language inference (NLI) is a 3-way classification problem. Given a premise and a hypothesis sentence, the objective is to classify their relationship as either entailment, contradiction, or neutral. In contrast to the previous tasks, we do not formulate this as sequence-to-sequence learning. We use the shared recurrent sentence encoder to encode both the premise and hypothesis into fixed length vectors u and v, respectively. We then feed the vector

[u; v; |u -v|; u * v],

which is a concatenation of the premise and hypothesis vectors and their respective absolute difference and hadamard product, to an MLP that performs the 3-way classification. This is the same classification strategy adopted by Conneau et al. (2017) . We train on a collection of about 1 million sentence pairs from the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) corpora. We take a simpler approach and pick a new sequence-to-sequence task to train on after every parameter update sampled uniformly. An NLI minibatch is interspersed after every ten parameter updates on sequence-to-sequence tasks (this was chosen so as to complete roughly 6 epochs of the dataset after 7 days of training). Our approach is described formally in the Algorithm below.

Model details can be found in section 7 in the Appendix.

Require: A set of k tasks with a common source language, a shared encoder E across all tasks and a set of k task specific decoders D1 . . . D k . Let θ denote each model's parameters, α a probability vector (p1 . . . p k ) denoting the probability of sampling a task such that Σ k i pi = 1, datasets for each task IP1 . . . IP k and a loss function L. 

5. EVALUATION STRATEGIES, EXPERIMENTAL RESULTS & DISCUSSION:

In this section, we describe our approach to evaluate the quality of our learned representations, present the results of our evaluation and discuss our findings.

5.1. EVALUATION STRATEGY:

We follow a similar evaluation protocol to those presented in Kiros et al. (2015) ; Hill et al. (2016) ; Conneau et al. (2017) which is to use our learned representations as features for a low complexity classifier (typically linear) on a novel supervised task/domain unseen during training without updating the parameters of our sentence representation model. We also consider such a transfer learning evaluation in an artificially constructed low-resource setting. In addition, we also evaluate the quality of our learned individual word representations using standard benchmarks (Faruqui & Dyer, 2014; Tsvetkov et al., 2015) .

The choice of transfer tasks and evaluation framework2 are borrowed largely from Conneau et al. (2017) . We provide a condensed summary of the tasks in section 10 in the Appendix but refer readers to their paper for a more detailed description.

5.2. EXPERIMENTAL RESULTS & DISCUSSION:

Table 2 presents the results of training logistic regression on 10 different supervised transfer tasks using different fixed-length sentence representation. Supervised approaches trained from scratch on some of these tasks are also presented for comparison. We present performance ablations when adding more tasks and increasing the number of hidden units in our GRU (+L). Ablation specifics are presented in section 9 of the Appendix.

It is evident from Table 2 that adding more tasks improves the transfer performance of our model. Increasing the capacity our sentence encoder with more hidden units (+L) as well as an additional layer (+2L) also lead to improved transfer performance. We observe gains of 1.1-2.3% on the sentiment classification tasks (MR, CR, SUBJ & MPQA) over Infersent and smaller improvement of 0.5% on SST. We demonstrate substantial gains on TREC (6% over Infersent and roughly 2% over the CNN-LSTM), outperforming even a competitive supervised baseline. We see similar gains (2.3%) on paraphrase identification (MPRC), closing the gap on supervised approaches trained from scratch. The addition of constituency parsing improves performance on sentence relatedness (SICK-R) and entailment (SICK-E) consistent with observations made by Bowman et al. (2016) .

In Table 4 , we show that simply training an MLP on top of our fixed sentence representations outperforms several strong & complex supervised approaches that use attention mechanisms, even on this fairly large dataset. For example, we observe a 0.2-0.5% improvement over the decomposable attention model (Parikh et al., 2016) . When using only a small fraction of the training data, indicated by the columns 1k-25k, we are able to outperform the Siamese and Multi-Perspective CNN using roughly 6% of the available training set. We also outperform the Deconv LVM model proposed by Shen et al. (2017) in this low-resource setting.

Unlike Conneau et al. (2017) , who use pretrained GloVe word embeddings, we learn our word embeddings from scratch. Somewhat surprisingly, in Table 3 we observe that the learned word embeddings are competitive with popular methods such as GloVe, word2vec, and fasttext (Bojanowski et al., 2016) on the benchmarks presented by Faruqui & Dyer (2014) and Tsvetkov et al. (2015) .

In Table 5 , we probe our sentence representations to determine if certain sentence characteristics and syntactic properties can be inferred following work by Adi et al. (2016) and Shi et al. (2016) . We observe that syntactic properties are better encoded with the addition of multi-lingual NMT and parsing. Representations learned solely from NLI do appear to encode syntax but incorporation into our multi-task framework does not amplify this signal. Similarly, we observe that sentence characteristics such as length and word order are better encoded with the addition of parsing.

In Appendix Table 6 , we note that our sentence representations outperform skip-thoughts and are on par with Infersent for image-caption retrieval. We also observe that comparing sentences using cosine similarities correlates reasonably well with their relatedness on semantic textual similarity benchmarks (Appendix Table 7 ).

We also present qualitative analysis of our learned representations by visualizations using dimensionality reduction techniques (Figure 1 ) and nearest neighbor exploration (Appendix Table 8 ). Figure 1 shows t-sne plots of our sentence representations on three different datasets -SUBJ, TREC and DBpedia. DBpedia is a large corpus of sentences from Wikipedia labeled by category and used by Zhang et al. (2015) . Sentences appear to cluster reasonably well according to their labels. The clustering also appears better than that demonstrated in Figure 2 of Kiros et al. (2015) on TREC and SUBJ. Appendix Table 8 contains sentences from the BookCorpus and their nearest neighbors. Sentences with some lexical overlap and similar discourse structure appear to be clustered together. 

6. CONCLUSION & FUTURE WORK:

We present a multi-task framework for learning general-purpose fixed-length sentence representations. Our primary motivation is to encapsulate the inductive biases of several diverse training signals used to learn sentence representations into a single model. Our multi-task framework includes a combination of sequence-to-sequence tasks such as multi-lingual NMT, constituency parsing and skip-thought vectors as well as a classification task -natural language inference. We demonstrate that the learned representations yield competitive or superior results to previous general-purpose sentence representation methods. We also observe that this approach produces good word embeddings. Shi et al. (2016) . Numbers reported are the accuracy with which the models were able to predict certain characteristics.

In future work, we would like understand and interpret the inductive biases that our model learns and observe how it changes with the addition of different tasks beyond just our simple analysis of sentence characteristics and syntax. Having a rich, continuous sentence representation space could allow the application of state-of-the-art generative models of images such as that of Nguyen et al. (2016) to language. One could also consider controllable text generation by directly manipulating the sentence representations and realizing it by decoding with a conditional language model. from the hidden states corresponding to words in a sentence. For example, Kiros et al. (2015) use the last hidden state while Conneau et al. (2017) perform max-pooling across all of the hidden states.

We consider both of these approaches and pick the one with better performance on the validation set. We note that max-pooling works best on sentiment tasks such as MR, CR, SUBJ and MPQA, while the last hidden state works better on all other tasks.

We also employ vocabulary expansion on all tasks as in Kiros et al. (2015) by training a linear regression to map from the space of pre-trained word embeddings (GloVe) to our model's word embeddings.

9. MULTI-TASK MODEL DETAILS:

This section describes the specifics our multi-task ablations in the experiments section. These definitions hold for all tables except for 3 and 5. We refer to skip-thought next as STN, French and German NMT as Fr and De, natural language inference as NLI, skip-thought previous as STP and parsing as Par.

+STN +Fr +De : The sentence representation h x is the concatenation of the final hidden vectors from a forward GRU with 1500-dimensional hidden vectors and a bidirectional GRU, also with 1500-dimensional hidden vectors.

+STN +Fr +De +NLI : The sentence representation h x is the concatenation of the final hidden vectors from a bidirectional GRU with 1500-dimensional hidden vectors and another bidirectional GRU with 1500-dimensional hidden vectors trained without NLI.

+STN +Fr +De +NLI +L : The sentence representation h x is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without NLI.

+STN +Fr +De +NLI +L +STP : The sentence representation h x is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without STP.

+STN +Fr +De +NLI +2L +STP : The sentence representation h x is the concatenation of the final hidden vectors from a 2-layer bidirectional GRU with 2048-dimensional hidden vectors and a 1-layer bidirectional GRU with 2048-dimensional hidden vectors trained without STP.

+STN +Fr +De +NLI +L +STP +Par : The sentence representation h x is the concatenation of the final hidden vectors from a bidirectional GRU with 2048-dimensional hidden vectors and another bidirectional GRU with 2048-dimensional hidden vectors trained without Par.

In tables 3 and 5 we do not concatenate the representations of multiple models.

10 DESCRIPTION OF EVALUATION TASKS Kiros et al. (2015) and Conneau et al. (2017) provide a detailed description of tasks that are typically used to evaluate sentence representations. We provide a condensed summary and refer readers to their work for a more thorough description.

10.1. TEXT CLASSIFICATION:

We evaluate on text classification benchmarks -sentiment classification on movie reviews (MR), product reviews (CR) and Stanford sentiment (SST), question type classification (TREC), subjectivity/objectivity classification (SUBJ) and opinion polarity (MPQA). Representations are used to train a logistic regression classifier with 10-fold cross validation to tune the L2 weight penalty. The evaluation metric for all these tasks is classification accuracy.

10.2. PARAPHRASE IDENTIFICATION:

We also evaluate on pairwise text classification tasks such as paraphrase identification on the Microsoft Research Paraphrase Corpus (MRPC) corpus. This is a binary classification problem to identify if two sentences are paraphrases of each other. The evaluation metric is classification accuracy and F1.

10.3. ENTAILMENT AND SEMANTIC RELATEDNESS:

To test if similar sentences share similar representations, we evaluate on the SICK relatedness (SICK-R) task where a linear model is trained to output a score from 1 to 5 indicating the relatedness of two sentences. We also evaluate using the entailment labels in the same dataset (SICK-E) which is a binary classification problem. The evaluation metric for SICK-R is Pearson correlation and classification accuracy for SICK-E.

10.4. SEMANTIC TEXTUAL SIMILARITY:

In this evaluation, we measure the relatedness of two sentences using only the cosine similarity between their representations. We use the similarity textual similarity (STS) benchmark tasks from 2012-2016 (STS12, STS13, STS14, STS15, STS16, STSB). The STS dataset contains sentences from a diverse set of data sources. The evaluation criteria is Pearson correlation.

10.5. IMAGE-CAPTION RETRIEVAL:

Image-caption retrieval is typically formulated as a ranking task wherein images are retrieved and ranked based on a textual description and vice-versa. We use 113k training images from MSCOCO with 5k images for validation and 5k for testing. Image features are extracted using a pre-trained 110 layer ResNet. The evaluation criterion is Recall@K and the median K across 5 different splits of the data.

10.6 QUORA DUPLICATE QUESTION CLASSIFICATION In addition to the above tasks which were considered by Conneau et al. (2017) , we also evaluate on the recently published Quora duplicate question dataset5 since it is an order of magnitude larger than the others (approximately 400,000 question pairs). The task is to correctly identify question pairs that are duplicates of one another, which we formulate as a binary classification problem. We use the same data splits as in Wang et al. (2017) . Given the size of this data, we consider a more expressive classifier on top of the representations of both questions. Specifically, we train a 4 layer MLP with 1024 hidden units, with a dropout rate of 0.5 after every hidden layer. The evaluation criterion is classification accuracy. We also artificially create a low-resource setting by reducing the number of training examples between 1,000 and 25,000 using the same splits as Shen et al. (2017) .

10.7. SENTENCE CHARACTERISTICS & SYNTAX:

In an attempt to understand what information is encoded in by sentence representations, we consider six different classification tasks where the objective is to predict sentence characteristics such as length, word content and word order (Adi et al., 2016) or syntactic properties such as active/passive, tense and the top syntactic sequence (TSS) from the parse tree of a sentence (Shi et al., 2016) .

The sentence characteristic tasks are setup in the same way as described in Adi et al. (2016) . The length task is an 8-way classification problem where sentence lengths are binned into 8 ranges. The content task is formulated as a binary classification problem that takes a concatenation of a sentence representation u ∈ R k and a word representation w ∈ R d to determine if the word is contained in the sentence. The order task is an extension of the content task where a concatenation of the sentence representation and word representations of two words in sentence is used to determine if the first word occurs before or after the second. We use a random subset of the 1-billion-word dataset for these experiments that were not used to train our multi-task representations.

The syntactic properties tasks are setup in the same way as described in Shi et al. (2016) .The passive and tense tasks are characterized as binary classification problems given a sentence's representation. The former's objective is to determine if a sentence is written in active/passive voice while the latter's objective is to determine if the sentence is in the past tense or not. The top syntactic sequence

:

https://github.com/dlwh/puck

https://www.github.com/facebookresearch/SentEval 4 https://github.com/kudkudak/word-embeddings-benchmarks/wiki

https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs

Figures and Tables:

Figure 1: T-SNE visualizations of our sentence representations on 3 different datasets. SUBJ (left), TREC (middle), DBpedia (right). Dataset details are presented in the Appendix.



Table content: <table><tr><td>Task</td><td>Sentence Pairs</td></tr><tr><td>En-Fr (WMT14)</td><td>40M</td></tr><tr><td>En-De (WMT15)</td><td>5M</td></tr><tr><td>Skipthought (BookCorpus)</td><td>74M</td></tr><tr><td>AllNLI (SNLI + MultiNLI)</td><td>1M</td></tr><tr><td>Parsing (PTB + 1-billion word)</td><td>4M</td></tr><tr><td>Total</td><td>124M</td></tr></table>

An approximate number of sentence pairs for each task. 4.1 MULTI-TASK TRAINING SETUP Multi-task training with different data sources for each task stills poses open questions. For example: When does one switch to training on a different task? Should the switching be periodic? Do we weight each task equally? If not, what training ratios do we use? Dong et al. (2015) use periodic task alternations with equal training ratios for every task. In contrast, Luong et al. (2015) alter the training ratios for each task based on the size of their respective training sets. Specifically, the training ratio for a particular task, α i , is the fraction of the number of training examples in that task to the total number of training samples across all tasks. The authors then perform α

Table content: <table/>

For MRPC and STSB we consider only the F1 score and Spearman correlations respectively and we also multiply the SICK-R scores by 100 to have all differences in the same scale. Bold numbers indicate the best performing transfer model on a given task. Underlines are used for each task to indicate both our best performing model as well as the best performing transfer model that isn't ours.

Table content: <table/>

Evaluation of word embeddings. All results were computed using Faruqui & Dyer (2014) with the exception of the Skipgram, NMT, Charagram and Attract-Repel embeddings. Skipgram and NMT results were obtained from Jastrzebski et al. (2017) 4 . Charagram and Attract-Repel results were taken from Wieting et al.(2016)  andMrkšić et al. (2017) respectively. We also report QVEC benchmarks(Tsvetkov et al., 2015)

Table content: <table><tr><td>Model</td><td/><td colspan="2">Accuracy</td><td/><td/></tr><tr><td/><td>1k</td><td>5k</td><td>10k</td><td colspan="2">25k All (400k)</td></tr><tr><td>Siamese-CNN</td><td>-</td><td>-</td><td>-</td><td>-</td><td>79.60</td></tr><tr><td>Multi-Perspective-CNN</td><td>-</td><td>-</td><td>-</td><td>-</td><td>81.38</td></tr><tr><td>Siamese-LSTM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>82.58</td></tr><tr><td>Multi-Perspective-LSTM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>83.21</td></tr><tr><td>L.D.C</td><td>-</td><td>-</td><td>-</td><td>-</td><td>85.55</td></tr><tr><td>BiMPM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>88.17</td></tr><tr><td>DecAtt (GloVe)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>86.52</td></tr><tr><td>DecAtt (Char)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>86.84</td></tr><tr><td>pt-DecAtt (Word)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>87.54</td></tr><tr><td>pt-DecAtt (Char)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>88.40</td></tr><tr><td>CNN (Random)</td><td colspan="4">56.3 59.2 63.8 68.9</td><td>-</td></tr><tr><td>CNN (GloVe)</td><td colspan="4">58.5 62.4 66.1 70.2</td><td>-</td></tr><tr><td>LSTM-AE</td><td colspan="4">59.3 63.8 67.2 70.9</td><td>-</td></tr><tr><td>Deconv-AE</td><td colspan="4">60.2 65.1 67.7 71.6</td><td>-</td></tr><tr><td>Deconv LVM</td><td colspan="4">62.9 67.6 69.0 72.4</td><td>-</td></tr><tr><td>Infersent (LogReg)</td><td colspan="4">68.8 73.8 75.7 76.4</td><td>-</td></tr><tr><td>Infersent (MLP)</td><td colspan="4">71.8 75.7 77.2 78.9</td><td>84.79</td></tr><tr><td>Our Models</td><td/><td/><td/><td/><td/></tr><tr><td colspan="5">+STN +Fr +De +NLI +L +STP (LogReg) 70.4 74.8 75.9 77.2</td><td>-</td></tr><tr><td>+STN +Fr +De +NLI +L +STP (MLP)</td><td colspan="4">74.7 77.7 78.3 81.4</td><td>87.01</td></tr></table>

Supervised & low-resource classification accuracies on the Quora duplicate question dataset. Accuracies are reported corresponding to the number of training examples used. The first 6 rows are taken from Wang et al. (2017), the next 4 are from Tomar et al. (2017), the next 5 from Shen et al. (2017) and The last 4 rows are our experiments using Infersent(Conneau et al., 2017) and our models.

Table content: <table/>

Evaluation of sentence representations by probing for certain sentence characteristics and syntactic properties. Sentence length, word content & word order from Adi et al. (2016) and sentence active/passive, tense and top level syntactic sequence (TSS) from

Table content: <table><tr><td>Model</td><td colspan="6">Length Content Order Passive Tense TSS</td></tr><tr><td/><td colspan="3">Sentence characteristics</td><td colspan="3">Syntatic properties</td></tr><tr><td>Majority Baseline</td><td>22.0</td><td>50.0</td><td>50.0</td><td>81.3</td><td>77.1</td><td>56.0</td></tr><tr><td>Infersent (AllNLI)</td><td>75.8</td><td>75.8</td><td>75.1</td><td>92.1</td><td>93.3</td><td>70.4</td></tr><tr><td>Skipthought</td><td>-</td><td>-</td><td>-</td><td>94.0</td><td>96.5</td><td>74.7</td></tr><tr><td>Our Models</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Skipthought +Fr +De</td><td>86.8</td><td>75.7</td><td>81.1</td><td>97.0</td><td>97.0</td><td>77.1</td></tr><tr><td>Skipthought +Fr +De +NLI</td><td>88.1</td><td>75.7</td><td>81.5</td><td>96.7</td><td>96.8</td><td>77.1</td></tr><tr><td>Skipthought +Fr +De +NLI +Par</td><td>93.7</td><td>75.5</td><td>83.1</td><td>98.0</td><td>97.6</td><td>80.7</td></tr></table>

Peer review (iclr2018):
 ---- updates: ---- I had a ton of comments and concerns, and I think the authors did an admirable job in addressing them. I think the paper represents a solid empirical contribution to this area and is worth publishing in ICLR. ---- original review follows: ---- This paper is about learning sentence embeddings by combining a bunch of training signals: predicting the next & previous sentences (skip-thought), predicting the sentence*s translation, classifying entailment relationships between two sentences, and predicting the constituent parse of a sentence. This is a simple idea that combines a bunch of things from prior work into one framework and yields strong results, outperforming most prior work on most tasks. I think this paper is impressive in how it scales up training to use so many tasks and such large training sets for each task. That and its strong experimental results make it worthy of publication. It*s not very surprising that adding more tasks and data improves performance on average across downstream tasks, but it is nice to see the experimental results in detail. While many people would think of this idea, few would have the resources and expertise necessary to do it justice. I also like how the authors move beyond the standard sentence tasks to evaluate also on the Quora question duplicate task with different amounts of training data and also consider the sentence characteristic / syntactic property tasks. It would be great if the authors could release their pretrained sentence representation model so that other researchers could use it. I do have some nitpicks here and there with the presentation and exposition, and I am concerned that at times the paper appears to be minimizing its weaknesses, but I think these are things that can be addressed in the next revision. I understand that sometimes it*s tempting to minimize one*s weaknesses in order to get a paper accepted because the reviewers may not understand the area very well and may get hung up on the wrong things. I understand the area well and so all the feedback I offer below comes from a place of desiring this paper*s publication while also desiring it to be as accurate and helpful for the community as possible. Below I*ll discuss my concerns with the experiments and description of the results. Regarding the results in Table 2: The results in Table 2 seem a little bit unstable, as it is unclear which setting to use for the classification tasks; maybe it depends on the kind of classification being performed. One model seems best for the sentiment tasks ("+2L +STP") while other models seem best for SUBJ and MPQA. Adding parsing as a training task hurts performance on the sentence classification tasks while helping performance on the semantic tasks, as the authors note. It is unclear which is the best general model. In particular, when others write papers comparing to the results in this paper, which setting should they compare to? It would be nice if the authors could discuss this. The results reported for the CNN-LSTM of Gan et al. do not exactly match those of any single row from Gan et al, either v1 or v2 on arxiv or the published EMNLP version. How were those specific numbers selected? The caption of Table 2 states "All results except ours are taken from Conneau et al. (2017)." However, Conneau et al (neither the latest arxiv version nor the published EMNLP version) does not include many of the results in the table, such as CNN-LSTM and DiscSent mentioned in the following sentence in the caption. Did the authors replicate the results of those methods themselves, or report them from other papers? What does bold and underlining indicate in Table 2? I couldn*t find this explained anywhere. At the bottom of Table 2, in the section with approaches trained from scratch on these tasks, I*d suggest including the 89.7 SST result of Munkhdalai and Yu (2017) and the 96.1 TREC result of Zhou et al. (2016) (as well as potentially other results from Zhou et al, since they report results on others of these datasets). The reason this is important is because readers may observe that the paper*s new method achieves higher accuracies on SST and TREC than all other reported results and mistakenly think that the new method is SOTA on those tasks. I*d also suggest adding the results from Radford et al. (2017) who report 86.9 on MR and 91.4 on CR. For other results on these datasets, including stronger results in non-fixed-dimensional-sentence-embedding transfer settings, see results and references in McCann et al. (2017). While the methods presented in this paper are better than prior work in learning general purpose, fixed-dimensional sentence embeddings, they still do not produce state-of-the-art results on that many of these tasks, if any. I think this is important to note. For all tasks for which there is additional training, there*s a confound due to the dimensionality of the sentence embeddings across papers. Using higher-dimensional sentence embeddings leads to more parameters in the linear model being trained on the task data. So it is unclear if the increase in hidden units in rows with "+L" is improving the results because of providing more weights for the linear model or whether it is learning a better sentence representation. The main sentence embedding results are in Table 2, and use the SentEval framework. However, not all tasks are included. The STS Benchmark results are included, which use an additional layer trained on the STS Benchmark training data just like the SICK tasks. But the other STS results, which use cosine similarity on the embedding space directly without any retraining, are only included in the appendix (in Table 7). The new approach does not do very well on those unsupervised tasks. On two years of data it is better than InferSent and on two years it is worse. Both are always worse than the charagram-phrase results of Wieting et al (2016a), which has 66.1 on 2012, 57.2 on 2013, 74.7 on 2014, and 76.1 on 2015. Charagram-phrase trains on automatically-generated paraphrase phrase pairs, but these are generated automatically from parallel text, the same type of resource used in the "+Fr" and "+De" models proposed in this submission, so I think it should be considered as a comparable model. The results in the bottom section of Table 7, reported from Arora et al (2016), were in turn copied from Wieting et al (2016b), so I think it would make sense to also cite Wieting et al (2016b) if those results are to be included. Also, it doesn*t seem appropriate to designate those as "Supervised Approaches" as they only require parallel text, which is a subset of the resources required by the new model. There are some other details in the appendix that I find concerning: Section 8 describes how there is some task-specific tuning of which function to compute on the encoder to produce the sentence representation for the task. This means that part of the improvement over prior work (especially skip-thought and InferSent) is likely due to this additional tuning. So I suppose to use these sentence representations in other tasks, this same kind of tuning would have to be done on a validation set for each task? Doesn*t that slightly weaken the point about having "general purpose" sentence representations? Section 9 provides details about how the representations are created for different training settings. I am confused by the language here. For example, the first setting ("+STN +Fr +De") is described as "A concatenation of the representations trained on these tasks with a unidirectional and bidirectional GRU with 1500 hidden units each." I*m not able to parse this. I think the authors mean "The sentence representation h_x is the concatenation of the final hidden vectors from a forward GRU (with 1500-dimensional hidden vectors) and a bidirectional GRU (also with 1500-dimensional hidden vectors)". Is this correct? Also in Sec 9: I found it surprising how each setting that adds a training task uses the concatenation of a representation with that task and one without that task. What is the motivation for doing this? This seems to me to be an important point that should be discussed in Section 3 or 4. And when doing this, are the concatenated representations always trained jointly from scratch with the special task only updating a subset of the parameters, or do you use the fixed pretrained sentence representation from the previous row and just concatenate it with the new one? To be more concrete, if I want to get the encoder for the second setting ("+STN +Fr +De +NLI"), do I have to train two times or can I just train once? That is, the train-once setting would correspond to only updating the NLI-specific representation parameters when training on NLI data; on other data, all parameters would be updated. The train-twice setting would first train a representation on "+STN +Fr +De", then set it aside, then train a separate representation on "+STN +Fr +De +NLI", then finally concatenate the two representations as my sentence representation. Do you use train-once or train-twice? Regarding the results in Table 3: What do bold and underline indicate? What are the embeddings corresponding to the row labeled "Multilingual"? In the caption, I can*t find footnote 4. The caption includes the sentence "our embeddings have 1040 pairs out of 2034 for which atleast one of the words is OOV, so a comparison with other embeddings isn*t fair on RW." How were those pairs handled? If they were excluded, then I think the authors should not report results on RW. I suspect that most of the embeddings included in the table also have many OOVs in the RW dataset but still compute results on it using either an unknown word embedding or some baseline similarity of zero for pairs with an OOV. I think the authors should find some way (like one of those mentioned, or some other way) of computing similarity of those pairs with OOVs. It doesn*t make much sense to me to omit pairs with OOVs. There are much better embeddings on SimLex than the embeddings whose results are reported in the table. Wieting et al. (2016a) report SimLex correlation of 0.706 and Mrkšić et al. (2017) report 0.751. I*d suggest adding the results of some stronger embeddings to better contextualize the embeddings obtained by the new method. Some readers may mistakenly think that the embeddings are SOTA on SimLex since no stronger results are provided in the table. The points below are more minor/specific: Sec. 2: In Sec. 2, the paper discusses its focus on fixed-length sentence representations to distinguish itself from other work that produces sentence representations that are not fixed-length. I feel the motivation for this is lacking. Why should we prefer a fixed-length representation of a sentence? For certain downstream applications, it might actually be easier for practitioners to use a representation that provides a representation for each position in a sentence (Melamud et al., 2016; Peters et al., 2017; McCann et al., 2017) rather than an opaque sentence representation. Some might argue that since sentences have different lengths, it would be appropriate for a sentence representation to have a length proportional to the length of the sentence. I would suggest adding some motivation for the focus on fixed-length representations. Sec. 4.1: "We take a simpler approach and pick a new task to train on after every parameter update sampled uniformly. An NLI minibatch is interspersed after every ten parameter updates on sequence-to-sequence tasks" These two sentences seem contradictory. Maybe in the first sentence "pick a new task" should be changed to "pick a new sequence-to-sequence task"? Sec. 5.1: typo: "updating the parameters our sentence" --> "updating the parameters of our sentence" Sec. 5.2: typo in Table 4 caption: "and The" --> ". The" typo: "parsing improvements performance" --> "parsing improves performance" In general, there are many missing citations for the tasks, datasets, and prior work on them. I understand that the authors are pasting in numbers from many places and just providing pointers to papers that provide more citation info, but I think this can lead to mis-attribution of methods. I would suggest including citations for all datasets/tasks and methods whose results are being reported. References: McCann, Bryan, James Bradbury, Caiming Xiong, and Richard Socher. "Learned in translation: Contextualized word vectors." CoRR 2017. Melamud, Oren, Jacob Goldberger, and Ido Dagan. "context2vec: Learning Generic Context Embedding with Bidirectional LSTM." CoNLL 2016. Mrkšić, Nikola, Ivan Vulić, Diarmuid Ó. Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. "Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints." TACL 2017. Munkhdalai, Tsendsuren, and Hong Yu. "Neural semantic encoders." EACL 2017. Pagliardini, Matteo, Prakhar Gupta, and Martin Jaggi. "Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features." arXiv preprint arXiv:1703.02507 (2017). Peters, Matthew E., Waleed Ammar, Chandra Bhagavatula, and Russell Power. "Semi-supervised sequence tagging with bidirectional language models." ACL 2017. Radford, Alec, Rafal Jozefowicz, and Ilya Sutskever. "Learning to generate reviews and discovering sentiment." arXiv preprint arXiv:1704.01444 2017. Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. "Charagram: Embedding words and sentences via character n-grams." EMNLP 2016a. Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. "Towards universal paraphrastic sentence embeddings." ICLR 2016b. Zhou, Peng, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. "Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling." COLING 2016.