Title: LEARNING ROBUST, TRANSFERABLE SENTENCE REP-RESENTATIONS FOR TEXT CLASSIFICATION

Abstract: Despite deep recurrent neural networks (RNNs) demonstrate strong performance in text classification, training RNN models are often expensive and requires an extensive collection of annotated data which may not be available. To overcome the data limitation issue, existing approaches leverage either pre-trained word embedding or sentence representation to lift the burden of training RNNs from scratch. In this paper, we show that jointly learning sentence representations from multiple text classification tasks and combining them with pre-trained word-level and sentence level encoders result in robust sentence representations that are useful for transfer learning. Extensive experiments and analyses using a wide range of transfer and linguistic tasks endorse the effectiveness of our approach.

1. INTRODUCTION:

Recent advances in deep neural networks have demonstrated the capability to build highly accurate models by training on vast amounts of data. The efficiency of these techniques comes from their ability to learn an encoder to convert raw inputs into useful continuous feature representations effectively. These successes primarily credit to the availability of ample resources, such as an extensive collection of training data. However, collecting a sufficient amount of manually annotated data is not always feasible, especially for domains requiring expert annotators.

While human annotated data is limited, there are abundant resources that can be used to lift the burden of learning representations from scratch and thus subsidize the requirement of having a large amount of training data. In the context of modeling natural languages, many successful stories showed that learned representations in both word and sentence levels are transferable to other tasks. These pre-trained representations enable us to model many natural language processing (NLP) tasks such as text classification (Bailey & Chopra, 2018) and named entity recognition (Cherry & Guo, 2015) with only a few thousands of examples.

At the word level, pre-trained word embeddings (Mikolov et al., 2013; Pennington et al., 2014) encode each word into a continuous vector representation, have been widely used in many applications (Seo et al., 2017; Lee et al., 2017; Venugopalan et al., 2017; Teney et al., 2017) . A few recent methods propose to construct contextualized word vectors to address the issue that the meaning of a word should be context-dependent. For example, Peters et al. (2018) leveraged a large unannotated corpus to train such contextualized word vectors by feeding word sequences into a deep recurrent neural network (RNNs) and generating representations based on the hidden states of the RNNs correspond to the respective words. This results in impressive performances in many NLP applications (Lee et al., 2017; Peters et al., 2017; He et al., 2017) .

At the sentence level, Conneau et al. (2017) showed that an LSTM-based sentence encoder (Hochreiter & Schmidhuber, 1997) trained on an annotated corpus for natural language inference (NLI) (Bowman et al., 2015) can capture useful features that are transferable to a wide range of text classification tasks. A few follow-up studies (Subramanian et al., 2018; Logeswaran & Lee, 2018; Cer et al., 2018) extended the approach by leveraging large-scale data and studied how to learn better transferable sentence representations.

However, all the existing approaches considered training word or sentence level representations from scratch. In contrast, we argue that by leveraging pre-trained embeddings/encoders and employing multiple large-scale supervised text classification datasets, we can learn more robust and transferable sentence representations. The primary research question that we address is how to build robust and transferable representations for sentence classification. The key challenges are two folds: 1) how to transfer only salient features by distinguishing generic information from task-specific information when learning an encoder and 2) how to combine representations at word and sentence levels to build strong transferable representations for sentence classification.

To address the first challenge, we propose to leverage multi-task learning (MTL) to jointly train sentence encoders on three large-scale text classification corpora, which cover a variety of domains and two language classification tasks -textual entailment and question paraphrasing. We exploit an MTL architecture that learns to separate generic representations from task-specific representations using adversarial training. While generic representations capture language-specific information, i.e., language structure, syntax, and semantics that are useful uniformly across a variety of language tasks, the task-specific representations encode domain knowledge that is helpful if the source and transfer tasks are homogeneous. Our experimental results show that when the shared and task-specific encoders are combined, they become more effective and applicable to a wide range of transfer tasks.

Besides, we combine our MTL-based sentence encoders with another existing sentence encoder (Subramanian et al., 2018) trained with different learning signals, and contextualized word vectors (Peters et al., 2018) , to build a more robust and transferable sentence encoder. We evaluate our encoder on 15 transfer (Conneau & Kiela, 2018) and 10 linguistic probing (Conneau et al., 2018) tasks. Experimental results demonstrate that our proposed sentence encoder better captures linguistic information and provides a significant improvement over existing transfer learning approaches.

2. RELATED WORK:

Our work is closely related to sentence representation learning, multi-task learning, and transfer learning and we briefly review each of these areas in this section.

• Sentence Representations Learning. Training neural networks to form useful sentence representations has become a core component in many machine learning models. Learning distributional sentence representations such that they capture the syntactic and semantic regularities has been proposed. These approaches range from models that compose of word embeddings (Le & Mikolov, 2014; Arora et al., 2017; Wieting et al., 2016) to models with complex network architectures (Zhao et al., 2015; Wang & Jiang, 2016; Liu et al., 2016c; Lin et al., 2017) . Unsupervised approaches are also proposed in literature by utilizing a large collection of unlabeled text corpora to learn distributional sentence representations. For example, Kiros et al. (2015) revised the skip-gram model (Mikolov et al., 2013) to learn a generic sentence encoder, called SkipThought that is further improved by using layer normalization (Ba et al., 2016) . Among other closely related works, the technique proposed by (Hill et al., 2016) fell short to SkipThought while (Logeswaran & Lee, 2018) showed improvement over skip-thought vectors.

Unlike word embeddings, learning sentence representations in an unsupervised fashion lack the reasoning about semantic relationships between sentences. To this end, Conneau et al. (2017) proposed to train a universal sentence encoder in the form of a bidirectional LSTM using the supervised natural language inference data, outperforming unsupervised approaches like SkipThought. Subramanian et al. (2018) propose to build general purpose sentence encoder by learning from a joint objective of classification, machine translation, parse tree generation and unsupervised skip-thought tasks. Compared to their approach, we propose to utilize multiple text classification datasets by leveraging a multi-task learning approach and combine them with existing contextualized word vectors (McCann et al., 2017; Peters et al., 2018) to learn robust and transferable sentence representations. Recent works (Cer et al., 2018; Perone et al., 2018) explored RNN free sentence encoders and evaluated sentence representations learning methods by using a variety of downstream and linguistic tasks.

• Multi-task Learning (MTL). Multi-task learning has been successfully used in a wide-range of natural language processing applications, including text classification (Liu et al., 2017) , machine translation (Luong et al., 2016) , sequence labeling (Rei, 2017) , sequence tagging (Peng & Dredze, 2017) , dependency parsing (Peng et al., 2017) etc. Recent works (Liu et al., 2016b; Zhang et al., 2017b; Liu et al., 2016a) proposed multi-task learning architectures with different methods of sharing information across the participant tasks. To facilitate scaling and transferring when a large number of tasks are involved, Zhang et al. (2017a) proposed to embed labels by considering semantic correlations among tasks. To investigate how much transferable an end-to-end neural network architectures are for NLP applications, Mou et al. (2016) propose to use multi-task learning on sentence classification tasks. In contrast to these prior work, we aim to learn a universal sentence encoders via multi-task learning that are transferable to a wide range of heterogeneous tasks.

• Transfer Learning. Transfer learning stores the knowledge gained from solving source tasks (usually with abundant annotated data), and apply it to other tasks (usually suffer from insufficient annotated data to train complex models) to combat the inadequate supervision problem. It has become prevalent in many computer vision applications (Sharif Razavian et al., 2014; Antol et al., 2015) where image features were trained on ImageNet (Deng et al., 2009) , and applications where word vectors (Pennington et al., 2014; Mikolov et al., 2013) were trained on large unlabeled corpora. Despite the benefits of using pre-trained word embeddings, many NLP applications still suffer from lacking high quality generic sentence representations that can help unseen tasks. In this work, we combine sentence representations learned using MTL and contextualized word vectors to obtain more robust sentence representations that transfer better.

3. SENTENCE REPRESENTATIONS LEARNING:

Our goal is to leverage available text copora and existing sentence and word encoders to build a universal sentence encoder. In the following, we first define the sentence encoder and then describe a multi-task learning approach that learns sentence representations jointly on multiple text classification tasks. Then we discuss how to combine the learned sentence representations with the existing sentence and contextualized word vectors.

3.1. SENTENCE ENCODER:

A typical text classification model consists of two parts: a representation learning component, also known as encoders that convert input text sequences into fixed-size vectors, and a classifier component that takes the vector representations and predicts the final class labels. The encoder is usually realized by a high complexity neural network architecture and requires a large amount of data to train, as opposed to the classifier which is generally simple (e.g., a linear model). When enough training examples are provided, the encoder and the classifier can be trained jointly from scratch in an end-to-end fashion.1 However, when data is insufficient, this approach is unfeasible. Instead, we can pre-train the encoder on other tasks (a.k.a source tasks) and transfer the learned encoder to the target task. In this case, we only require a few labeled examples to train the low-complexity classifier on top of the pre-trained encoder. We discuss how to build the pre-train encoder in below.

We follow (Conneau et al., 2017) to build a transferable encoder based on an one layer bidirectional LSTM with max pooling (BiLSTM-max). Formally, given a sentence with T words, [w 1 , w 2 , ..., w T ], the encoder first runs two LSTM models on input text from both directions.

-

EQUATION

and

h t = [ - → h t , ← - h t ] ∈ R 2d

is the t-th hidden vectors in BiLSTM, d is the dimensionality of the LSTM hidden units. To form a fixed-size vector representation of variable length sentences, the maximum value is selected over each dimension of the hidden units:

EQUATION

where s j is the j-th element of the sentence embedding s.

For some transfer tasks (e.g., textural entailment and similarity measuring), the goal is to predict the relationship between two sentences. Therefore, the input involves two sentences (s 1 , s 2 ). We generate the representation of input instances by [s 1 , s 2 , s 1 -s 2 , s 1 s 2 ] where denotes the element-wise multiplication, and [•, •] denotes vector concatenation.

Multi-task learning. Multi-task learning was shown efficient in many text classification tasks. However, its effectiveness in learning transferable sentence representations is comparably less studied. In this paper, we investigate the utility offered by several large-scale text classification tasks. We show that learning signals from various text classification tasks results in robust and transferable sentence representations. Inspired by (Liu et al., 2017) , we study two variants of shared-private (SP) multi-task learning (MTL) frameworks.The shared-private MTL framework maintains private and shared encoders with their own task-specific layers to encourage task-specific and generic features being learned by the private and shared encoders, respectively. In this study, we design one private BiLSTM-max sentence encoder for each task, and one shared BiLSTM-max encoder for all the tasks to capture task-dependent and generic features, respectively. Sentence embeddings produced by private and shared encoders are concatenated to form the final sentence representations. In this way, the shared encoder provides task-independent information, while the private encoders are helpful when the target task is proximity to some source tasks. Formally, for any sentence in a given task k, its shared representation s k s and private representation s k p can be computed using Eq. ( 1) -( 2), and they are concatenated to construct the sentence embedding:

s k = [s k s , s k p ].

Adversarial Training. Ideally, we want the private encoders to learn only task-specific features, and the shared encoder to learn generic features. To achieve this goal, we adopt the adversarial training strategy proposed by Liu et al. (2017) to introduce a discriminator on top of the shared BiLSTM-max sentence encoder. The goal of the discriminator, D is to identify which task an encoded sentence s k comes from, and the adversarial training requires the shared sentence encoder to generate representations that can "fool" the discriminator. In this way, the shared encoder is forced not to carry task-related information. The discriminator is defined as,

D(s k ) = softmax(W s k + b),

where W ∈ R d×d and b ∈ R d are model parameters. Optimizing the adversarial loss,

L adv = min θ E max θ D K k=1 N k i=1 d k i log[D(E(s))]

has two competing goals: the discriminator tries to maximize the classification accuracy (inside the parentheses), and the sentence encoder tries to confuse it (and thus minimize the classification accuracy). E and D represents the shared sentence encoder and the discriminator respectively and θ E and θ D are the model parameters of E and D. d k i denotes the ground-truth label indicating the type of the current task. To encourage the shared and private encoders to capture different aspects of the sentences, the following term is added.

L dif f = K k=1 H k s H k p 2 F

where • 2 F is the squared Frobenius norm. Here, H k s and H k p are matrices where rows are the hidden vectors (see Eq. ( 1)) generated by the shared and private encoders given an input sentence of task k. The final loss function is a weighted combination of three parts:

L = L multi-task + βL adv + γL dif f

where β and γ are hyper-parameters, L multi-task refers to a simple summation over the cross entropy loss for each task. We tune β and γ in the range [0.001, 0.005, 0.01, 0.05, 0.1, 0.5] and present the best β and γ values in table 4 (provided in the appendix) for different multi-task learning settings.

3.2. UNIFYING SENTENCE EMBEDDINGS AND CONTEXTUALIZED VECTORS:

Existing studies (Subramanian et al., 2018; Peters et al., 2018) leverage large amount of data to train sentence or word representations. However, sometimes it is impractical to assume there is an access to these large-scale data or computation resources. In these circumstances, we can combine existing encoders in a post-processing step to leverage the humongous data sources. In this work, we show that by combining our MTL based sentence encoder with an existing sentence encoder (Subramanian et al., 2018) and a contextualized word representation (Peters et al., 2018 ) encoder, we achieve state-of-the-art transfer performance on a wide variety of text classification tasks.

The contextualized word vectors refer to the hidden states generated by a BiLSTM (as in Eq. ( 1)) given a sequence of words (sentences) as inputs. To form a fixed size sentence representation from the contextual word vectors, we apply average pooling2 . Although Peters et al. (2018) suggested learning the weights of the contextual word vectors, we do not learn any additional weights because we consider scenarios when there is no training example available to learn such weights. To get a universal sentence representation, we concatenate the sentence embeddings provided by our MTL based encoders, an existing sentence encoder (Subramanian et al., 2018) trained with another set of tasks, and the fixed size vector constructed from contextual word vectors (Peters et al., 2018) . We will investigate more effective ways to combine multiple representations in our future work.

4. EXPERIMENTS:

In this section, we first show that our proposed sentence encoder can achieve state-of-the-art transfer performances. Then we demonstrate that combining the multi-task trained sentence representations with other sentence and word vectors yield better universal sentence representations. To better understand our results, we provide a thorough ablation study and confirm our combined encoder can learn robust and transferable sentence representations.

4.1. EXPERIMENTAL SETUP:

We use three large-scale textual entailment and paraphrasing tasks to train sentence encoders with multi-task learning, and combine these with an existing sentence encoder and contextualized word embeddings to compose the final sentence representations. We test the generalizability of the sentence embeddings on fifteen transfer tasks. A detailed description of the source and transfer tasks are presented in table 5 in the appendix. In addition, we perform a quantitative analysis using ten probing tasks to show what linguistic information is captured by our proposed sentence encoders.

Source tasks. The first two source tasks are natural language inference (NLI) which determines whether a natural language hypothesis can be inferred from a natural language premise. We consider the SNLI (Bowman et al., 2015) and the Multi-Genre NLI (MNLI) (Williams et al., 2017) which consist of sentence pairs, manually labeled with one of the three categories: entailment, contradiction and neutral. Following Conneau et al. (2017) , we also conduct experiments that combine SNLI and MNLI datasets, which is denoted as AllNLI. The second task is the Quora question paraphrase (QQP)3 detection based on a dataset of 404k question pairs. We use the Quora dataset split as that in Wang et al. (2017) . We present and discuss the source task performances in appendix A.

Transfer and probing tasks. We evaluate the sentence encoders on fifteen transfer and ten probing tasks using the SentEval toolkit (Conneau & Kiela, 2018) . Among the transfer tasks, six are text classification tasks for sentiment analysis (MR, SST), question-type (TREC), product reviews (CR), subjectivity/objectivity (SUBJ) and opinion polarity (MPQA). Rest of the transfer tasks (SICK-E, SICK-R, MRPC, STSB, and STS12-16) are semantic relatedness and textual similary tasks. We test our sentence encoder on capturing linguistic (surface, syntactic, and semantic) information using the ten probing tasks suggested in (Conneau et al., 2018) .

Hyper-parameter tuning. We carefully tune the parameters and report the testing performance with best parameters. We use SGD with an initial learning rate of 0.1 and a weight decay of 0.99. At each epoch, we divide the learning rate by 5 if the development accuracy decreases. We use minibatches of size 128 and training is stopped when the learning rate goes below the threshold of 10 -5 . For the task-specific classifier, we use a multi-layer perceptron with 1 hidden-layer of 512 hidden units. We consider the range [256, 512, 1024, 2048] for the number of hidden units in BiLSTM and found 2048 results in best performance. We use 300 dimensional GloVe word vectors (Pennington et al., 2014) trained on 840 billions of tokens as fixed word embeddings.

4.2. EVALUATION ON TRANSFER TASKS:

We benchmark the performances of our proposed encoders on fifteen transfer tasks in comparison with several baselines including unsupervised models, sentence encoders, contextualized word encoders and supervised models. Table 1 and 2 summarize the results. From the block 4 of table 1, we see our combined encoders achieve best performance on 5/8 transfer tasks, which demonstrates the efficacy of our proposed unified sentence encoder.

We further analyze the performance of our proposed combined encoders from two aspects: the improvement achieved by multi-task learned sentence encoders and the efficiency of the combination. From block 2 of table 1, we see the MTL based sentence encoders, Sent2vec outperforms the single task based sentence encoder InferSent on 7 out of 8 transfer tasks (comparing row 2.1 with 2.3-2.4). Sent2Vec also provides competitive performance on the other 7 tasks as shown in table 2. The

Model Type:

MR CR SUBJ MPQA SST TREC SICK-E MRPC 1.Unsupervised sentence representations learning 1.1. FastSent (Hill et al., 2016) 70.8 78.4 88.7 80.6 -76.8 -72.2/80.3 1.2. SkipThought (Kiros et al., 2015) 76.5 80.1 93.6 87.1 82.0 92.2 82.3 73.0/82.0 1.3. USE (Transformer) (Cer et al., 2018) 81.4 87.4 93.9 87.0 85.4 92.5 --1.4. Byte mLSTM (Radford et al., 2017) 86.9 91.4 94.6 88.5 ---75.0/82.8 2. Supervised sentence representations learning 2.1. InferSent (Conneau et al., 2017) 81.6 85.9 92.4 90.4 85.3 87.0 85.6 75.5/82.2 2.2. GenSen (Subramanian et al., 2018) 82 results demonstrate that learning from multiple tasks helps to capture more generalizable features that are suitable for transfer learning. In addition, when using adversarial training, we observe improvements in 9 out of 15 transfer tasks comparing to the non-adversarial setting (see table 1 and 2 ).

To investigate the advantages of adversarial training, we provide a detailed comparison between the shared and private encoders with and without adversarial training in appendix C.

Combining sentence encoders and contextualized word vectors improves the transfer learning performance significantly (comparing row 4.3 to 2.2, 2.4, and 3.2 in table 1 ). First, we see from table 2, when there are no training examples available for tasks (STS12-16 and STSB tasks), sentence embeddings (blocks 1 and 3) perform better than contextualized vectors (block 2). The result demonstrates the necessity of learning generic sentence representations so that they can be directly used (without training) in transfer tasks. Although Sent2vec outperforms ELMo on 12/15 tasks (comparing row 2.4 to 3.2 in table 1 and row ELMo to Sent2vec (ASP) in table 2), it fell short to GenSen on most of the transfer tasks since GenSen is trained using 124M sentence pairs while Sent2vec is trained on 1.4M pairs. Because training an encoder on massive datasets require large computational resources, the direct utilization of GenSen through a post-processing step can bring benefits from large resources in a computational efficient way. Second, contextualized vectors (ELMo) perform better in specific tasks like SST. Also contextualized vectors are shown capable to capture specific linguistic properties (more details in the analysis part). These indicate that utilizing contextualized vectors may help learn better sentence representations. As a result, in the block 4 of table 1, when Sent2vec, GenSen, and ELMo are combined, we observe a significant improvement on 4 out of 8 tasks (MR, CR, SST, and SICK-E) over its individual components and competitive performance on the other tasks, which confirms the efficiency of the combination.

4.3. ANALYSIS:

Impact of sentence embeddings and contextualized word vectors. To analyze the contributions of our proposed sentence encoder and the sentence representations learned from contextualized word vectors (CoVe, ELMo) in a combined encoder during transfer learning, we design a classifier with a different network architecture. The classifier first generates predicted class probabilities based on a softmax layer using each sentence representations as input. Then the predictions are combined by a pooling layer with a weight parameter for each encoder. By investigating the learned weights in the pooling layer, we can understand which encoder contributes the most. The learned weights are shown in Figure 1 (a). Although contextualized vectors have higher weights in 7/10 tasks, sentence vectors have > 20% contributions for each task and play dominant roles in tasks like SST and SICK-R. As we have shown in table 1, the combined encoder performs better than individual encoders (comparing row 4.1 to 2.4, 3.1, 3.2), indicating the contribution of the sentence and contextual word encoders are quite complementary.

Impact of source tasks on transfer tasks. To understand the influences of the source tasks on the transfer tasks, We conduct a similar analysis as in Figure 1 (a) and show the learned weights assigned for the private (task-specific) and shared (generic) encoders in the ASP model in Figure 1 (b). In general, for target tasks that are similar to the source tasks, the private encoders get higher weights, otherwise, the shared encoder is better. The combination of the shared and private encoders enables the transfer task to choose the best combination, thus achieved the best results. Most of the transfer tasks assign large weights on the SNLI task-specific and low weights on the QQP taskspecific encoder, which explains why representations learned on SNLI are especially efficient for transfer tasks as noted in (Conneau et al., 2017) . Besides, with adversarial training enforced, the shared encoder gets a lower weight from most of the transfer tasks than non-adversarial training (see figure 3 in the appendix) demonstrating the efficiency of adversarial training to separate generic and task-specific representations.

Probing for linguistic properties. Table 3 : Probing task accuracies with MLP as the classifier. For ELMo, the same bag-of-words averaging technique is employed as used for the downstream transfer tasks. When ELMo is combined with Sent2Vec and GenSen, features only from the top layer are used to fit in single GPU (Titan X).

Bold-faced values denote the best results across the board. 

5. CONCLUSION:

In this paper, we propose to leverage available large-scale text classification datasets and existing word and sentence encoding models to learn a universal sentence encoder. We utilize multi-task learning (MTL) to train sentence encoders that learn both generic and task-specific sentence representations from three heterogeneous text classification corpora. Experiments show that the MTL trained representations outperforms sentence encoders trained on singe task on a variety of transfer sentence classification tasks. We then further combine these sentence encoders with an existing multi-task pre-trained sentence encoder (with a different set of tasks) and a contextualized word representation learner. Our proposed unified sentence encoder yields significant improvements over the state-of-the-art sentence representations on transfer learning tasks. Extensive comparisons and thorough analysis using 15 transfer datasets and 10 linguistic probing tasks endorse the robustness of our proposed universal sentence encoder. Table 5 : Statistics of the datasets for multi-task learning and the transfer tasks. N is the number of samples, V is the vocabulary size, and C is the number of classes or score range. † denotes the datasets that are used in multi-task learning.

A EVALUATION ON SOURCE TASKS:

In this section, we discuss the performance of the shared-private multi-task learning (MTL) frameworks on different combinations of QQP, SNLI and MNLI datasets as the source tasks. We concatenate the representations generated by the shared and private encoders to form sentence embeddings.

The results are presented in table 6. We compare the performance of MTL with the models trained training data is the same. To verify the hypothesis that the improvements in transfer learning do not solely come from having more annotated data, we design an experiment that samples equal amount of data (225k training examples) from SNLI and QQP to match the size of full SNLI dataset. We found 0.26% average improvement in transfer tasks compared to single task learning (STL) on the SNLI dataset. With full SNLI and QQP dataset, we observe a larger (0.69% on average) improvement in transfer tasks compared to STL on SNLI dataset. The first row of table 8 shows that MTL is beneficial in this setting and the second row demonstrates that with additional data, MTL achieves larger improvements.

C PRIVATE ENCODERS VS. SHARED ENCODER:

To verify our hypothesis that shared encoder learns generic features that are more suitable for transfer learning and with adversarial training enforced, shared encoder becomes more effective; we provide a detailed comparison of the private and shared encoders in table 9. However, by concatenating the shared and task-specific representations, we can achieve better transfer performance which indicates that transfer tasks also get benefited from task-specific features, specially when the source and transfer tasks are homogenous (more details are provided in the ablation analysis). 

:

In this case, the classifier is the last layer in the network architecture.

We tried max pooling, but it consistently performed more inferior compared to average pooling.

https://www.kaggle.com/quora/question-pairs-dataset

Figures and Tables:

Figure 1: (a) Weights learned by transfer tasks for contextualized vectors (ELMo, CoVe), and sentence vectors which refer to a concatenation of all the private and shared encoders of the adversarial shared-private multi-task model. (b) Weights learned by the transfer tasks for private (task-specific) and shared encoders of adversarial shared private multi-task model.

Figure 2: Comparing test performances of supervised learning (using BCN), word (ELMo) and sentence level representations (Sent2vec, GenSen) and their combination (ComboRep refers to Sent2vec + GenSen + ELMo) on SST and SICK-E tasks as the training dataset size is varied.

Figure 3: Weights learned by the transfer tasks for private (task-specific) and shared encoders of shared private multi-task model (a) with and (b) without adversarial training.

Evaluation of sentence representations on a set of 8 tasks using a logistic regression classifier. "SP" and "ASP" in row 2.3 and 2.4 refers to the shared-private and adversarial shared private multi-task learning models. Values indicate the accuracy (accuracy/F1 for MRPC) for the test sets and bold-faced values denote the best transfer performances. We employ an averaging bag-of-words technique to form sentence embeddings, using features from all three layers of ELMo.

Table content: <table><tr><td>.7 87.6 94.1 91.1 83.5 93.0</td><td>87.8 78.0/84.2</td></tr></table>

Transfer evaluation of the semantic relatedness and textual similarity tasks. "SP" and "ASP" in block 3 refers to the shared-private and adversarial shared private multi-task learning models. In block 4, we use ASP setting for Sent2vec. We use features from the top layer of the ELMo to produce sentence embeddings. Values indicate the Pearson correlation coefficient for the test sets and bold-faced values denote the best performance across all the models.

Table content: <table/>

To understand what linguistic properties are captured by our proposed sentence encoders and the unified encoders, we conduct experiments on probing tasks proposed inConneau et al. (2018). Results are provided in table 3. Sentence representations show superiority in hard semantic tasks like SOMO and CoordInv, while contextualized word vectors perform better on embedding surface and syntactic properties.Moreover, MTL encoders (Sent2Vec)

Table content: <table><tr><td>Model Type</td><td colspan="8">SentLen WC TreeDepth TopConst BShift Tense SubjNum ObjNum SOMO CoordInv</td></tr><tr><td>InferSent</td><td>84.0 90.5</td><td>38.6</td><td>47.3</td><td>62.3 87.1</td><td>85.9</td><td>81.5</td><td>59.8</td><td>68.5</td></tr><tr><td>GenSen</td><td>93.9 96.7</td><td>44.0</td><td>64.1</td><td>75.5 90.0</td><td>90.1</td><td>90.3</td><td>53.6</td><td>69.0</td></tr><tr><td>USE (Transformer)</td><td>79.8 54.2</td><td>30.5</td><td>68.7</td><td>60.5 86.2</td><td>77.8</td><td>74.6</td><td>58.5</td><td>58.2</td></tr><tr><td>CoVe</td><td>88.4 19.0</td><td>41.2</td><td>39.1</td><td>68.8 86.2</td><td>84.6</td><td>85.4</td><td>50.8</td><td>61.3</td></tr><tr><td>ELMo</td><td>95.3 76.0</td><td>42.6</td><td>50.4</td><td>85.1 89.6</td><td>91.4</td><td>89.1</td><td>59.6</td><td>67.8</td></tr><tr><td>Sent2Vec (SP)</td><td>87.3 88.1</td><td>41.9</td><td>51.9</td><td>62.8 88.1</td><td>87.7</td><td>83.9</td><td>60.6</td><td>71.1</td></tr><tr><td>Sent2Vec (ASP)</td><td>88.0 85.5</td><td>41.0</td><td>53.2</td><td>53.1 88.3</td><td>87.3</td><td>83.3</td><td>49.9</td><td>70.8</td></tr><tr><td>Sent2Vec (ASP) + ELMo + GenSen</td><td>91.0 86.2</td><td>45.2</td><td>59.2</td><td>85.4 91.0</td><td>92.4</td><td>91.7</td><td>49.9</td><td>73.3</td></tr></table>

Best β and γ values for adversarial shared private model on different set of tasks.

Table content: <table><tr><td>Name</td><td>N</td><td>V</td><td>Task</td><td>C</td></tr><tr><td colspan="4">Binary and multi-class classification tasks</td><td/></tr><tr><td>MR</td><td>11k</td><td colspan="2">20.3k sentiment</td><td>2</td></tr><tr><td>CR</td><td>4k</td><td colspan="2">5.7k product review</td><td>2</td></tr><tr><td>SUBJ</td><td>10k</td><td colspan="2">22.6k subj/obj</td><td>2</td></tr><tr><td>MPQA</td><td>11k</td><td colspan="2">6.2k opinion</td><td>2</td></tr><tr><td>SST</td><td>70k</td><td colspan="2">17.5k sentiment</td><td>2</td></tr><tr><td>TREC</td><td>6k</td><td colspan="2">9.7k question-type</td><td>6</td></tr><tr><td colspan="4">Recognizing textual entailment tasks</td><td/></tr><tr><td>SNLI  †</td><td>560k</td><td colspan="2">42.7k entailment</td><td>3</td></tr><tr><td colspan="4">Multi-NLI  † 433k 102.7k entailment</td><td>3</td></tr><tr><td>SICK-E</td><td>10k</td><td colspan="2">2.4k entailment</td><td>3</td></tr><tr><td colspan="3">Paraphrase identification tasks</td><td/><td/></tr><tr><td>QQP  †</td><td colspan="3">404k 127.5k paraphrasing</td><td>2</td></tr><tr><td>MRPC</td><td>5.8k</td><td colspan="2">19.5k paraphrasing</td><td>2</td></tr><tr><td colspan="3">Semantic textual similarity tasks</td><td/><td/></tr><tr><td>SICK-R</td><td>10k</td><td colspan="2">2.4k similarity</td><td>0 -5</td></tr><tr><td>STSB</td><td>8.6k</td><td colspan="2">15.9k similarity</td><td>5</td></tr><tr><td>STS-12</td><td>399</td><td colspan="2">735 similarity</td><td>0 -5</td></tr><tr><td>STS-13</td><td>561</td><td colspan="2">1.6k similarity</td><td>0 -5</td></tr><tr><td>STS-14</td><td>750</td><td colspan="2">3.8k similarity</td><td>0 -5</td></tr><tr><td>STS-15</td><td>750</td><td colspan="2">1.3k similarity</td><td>0 -5</td></tr><tr><td>STS-16</td><td>209</td><td colspan="2">868 similarity</td><td>0 -5</td></tr></table>

Detailed analysis of the transfer test results for shared-private models trained on different combinations of QQP, SNLI and MNLI datasets. Combined encoder refers to the concatenation of shared encoder and all private encoders. Underlined values indicate the best performance among different encoders of the shared-private models trained on the same set of tasks. Bold-faced values indicate the best performance among all models in this table.

Table content: <table/>

Peer review (iclr2019):
 The paper introduces an approach to train sentence representations from multiple text classification tasks. Two variants of shared private multi-task learning are employed to train BiLSTM based sentence encoders. Experiments were conducted with care and nicely compared to existing results from the literature. However, these show only incremental improvements over recent baselines. The idea of using shared and private encoders in the multitask setup is interesting, however, results do not indicate much improvements over simple MTL settings like the one from Conneau et al. Overall, the paper seems to be a straight-forward combination of 2 existing ideas: the model and datasets (except for QQP) from Conneau et al. and the SP setup from Liu et al. The additional QQP dataset actually doesn*t seem to offer all that much looking at the appendix. Furthermore, the authors simply concatenate all embeddings that exist to build their universal representations which is fairly obvious that this helps. They also concatenate all private embeddings with the shared one, which is not all that fair in comparison to other results, because the larger dimensionality typically helps in these tasks, even with random projections [1]. Nevertheless there is no real gain over previous approaches. So overall I believe that the incremental contributions in this paper as well as the lack of favorable results that would support the additions made over Conneau et al. make this paper rather an interesting workshop addition than a conference full-paper. Comment: On a sidenote, as a line of research I would also like to raise awareness on how little trained sentence representations actually offer over simple BoW representations. See for instance the following parallel submission [1] that uses BoW representations with random projections. This paper also points out a problem in InferSent that prevents proper evaluation when using max-pooling. In case the InferSent library was used to obtain results for this paper, this would make it hard to trust the results. [1] https://openreview.net/forum?id=BkgPajAcY7