Title: IMPROVING SENTENCE REPRESENTATIONS WITH MULTI-VIEW FRAMEWORKS

Abstract: Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

1. INTRODUCTION:

Multi-view learning methods provide the ability to extract information from different views of the data and enable self-supervised learning of useful features for future prediction when annotated data is not available (de Sa, 1993) . Minimising the disagreement among multiple views helps the model to learn rich feature representations of the data and, also after learning, the ensemble of the feature vectors from multiple views can provide an even stronger generalisation ability.

Distributional hypothesis (Harris, 1954) noted that words that occur in similar contexts tend to have similar meaning (Turney & Pantel, 2010) , and distributional similarity (Firth, 1957) consolidated this idea by stating that the meaning of a word can be determined by the company it has. The hypothesis has been widely used in machine learning community to learn vector representations of human languages. Models built upon distributional similarity don't explicitly require humanannotated training data; the supervision comes from the semantic continuity of the language data. Large quantities of annotated data are usually hard and costly to obtain, thus it is important to study unsupervised and self-supervised learning. Our goal is to propose learning algorithms built upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelled data. We draw inspiration from the lateralisation and asymmetry in information processing of the two hemispheres of the human brain where, for most adults, sequential processing dominates the left hemisphere, and the right hemisphere has a focus on parallel processing (Bryden, 2012) , but both hemispheres have been shown to have roles in literal and non-literal language comprehension (Coulson et al., 2005; Coulson & van Petten, 2007) .

Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models, which have been widely applied in sentiment analysis tasks (Yang et al., 2016) , and the linear/loglinear models, which have excelled at capturing attributional similarities of words and sentences (Arora et al., 2016; 2017; Hill et al., 2016; Turney & Pantel, 2010) for learning sentence representations. Previous work on unsupervised sentence representation learning based on distributional hypothesis can be roughly categorised into two types:

Generative objective: These models generally follow the encoder-decoder structure. The encoder learns to produce a vector representation for the current input, and the decoder learns to generate sentences in the adjacent context given the produced vector (Kiros et al., 2015; Hill et al., 2016; Gan et al., 2017; Tang et al., 2018) . The idea is straightforward, yet its scalability for very large corpora is hindered by the slow decoding process that dominates training time, and also the decoder in each model is discarded after learning as the quality of generated sequences is not the main concern, which is a waste of parameters and learning effort.

Our first multi-view framework has a generative objective and uses an RNN as the encoder and an invertible linear projection as the decoder. The training time is drastically reduced as the decoder is simple, and the decoder is also utilised after learning. A regularisation is applied on the linear decoder to enforce invertibility, so that after learning, the inverse of the decoder can be applied as a linear encoder in addition to the RNN encoder.

Discriminative Objective: In these models, a classifier is learnt on top of the encoders to distinguish adjacent sentences from those that are not (Li & Hovy, 2014; Jernite et al., 2017; Nie et al., 2017; Logeswaran & Lee, 2018) ; these models make a prediction using a predefined differentiable similarity function on the representations of the input sentence pairs or triplets.

Our second multi-view framework has a discriminative objective and uses an RNN encoder and a linear encoder; it learns to maximise agreement among adjacent sentences. Compared to earlier work on multi-view learning (de Sa, 1993; Dhillon et al., 2011; Wang et al., 2015) that takes data from various sources or splits data into disjoint populations, our framework processes the exact same data in two distinctive ways. The two distinctive information processing views tend to encode different aspects of an input sentence; forcing agreement/alignment between these views encourages each view to be a better representation, and is beneficial to the future use of the learnt representations.

Our contribution is threefold:

• Two multi-view frameworks for learning sentence representations are proposed, in which one framework uses a generative objective and the other one adopts a discriminative objective. Two encoding functions, an RNN and a linear model, are learnt in both frameworks.

• The results show that in both frameworks, aligning representations from two views gives improved performance of each individual view on all evaluation tasks compared to their single-view trained counterparts, and furthermore ensures that the ensemble of two views provides even better results than each improved view alone.

• Models trained under our proposed frameworks achieve good performance on the unsupervised tasks, and overall outperform existing unsupervised learning models, and armed with various pooling functions, they also show solid results on supervised tasks, which are either comparable to or better than those of the best unsupervised transfer model.

It is shown (Hill et al., 2016) that the consistency between supervised and unsupervised evaluation tasks is much lower than that within either supervised or unsupervised evaluation tasks alone and that a model that performs well on supervised evaluation tasks may fail on unsupervised tasks. It is subsequently showed (Conneau et al., 2017; Subramanian et al., 2018) that, with large-scale labelled training corpora, the resulting representations of the sentences from the trained model excel in both supervised and unsupervised tasks, while the labelling process is costly. Our model is able to achieve good results on both groups of tasks without labelled information.

2. MODEL ARCHITECTURE:

Our goal is to marry RNN-based sentence encoder and the avg-on-word-vectors sentence encoder into multi-view frameworks with simple objectives. The motivation for the idea is that, RNN-based encoders process the sentences sequentially, and are able to capture complex syntactic interactions, while the avg-on-word-vectors encoder has been shown to be good at capturing the coarse meaning of a sentence which could be useful for finding paradigmatic parallels (Turney & Pantel, 2010) .

We present two multi-view frameworks, each of which learns two different sentence encoders; after learning, the vectors produced from two encoders of the same input sentence are used to compose the sentence representation. The details of our learning frameworks are described as follows:

2.1 ENCODERS In our multi-view frameworks, we first introduce two encoders that, after learning, can be used to build sentence representations. One encoder is a bi-directional Gated Recurrent Unit (Chung et al., 2014) f (s; φ), where s is the input sentence and φ is the parameter vector in the GRU. During learning, only hidden state at the last time step is sent to the next stage in learning. The other encoder is a linear avg-on-word-vectors model g(s; W ), which basically transforms word vectors in a sentence by a learnable weight matrix W and outputs an averaged vector.

2.2. GENERATIVE OBJECTIVE:

Given the finding (Tang et al., 2018) that neither an autoregressive nor an RNN decoder is necessary for learning sentence representations that excel on downstream tasks, our learning framework only learns to predict words in the next sentence. The framework has an RNN encoder f , and a linear decoder h. Given an input sentence s i , the encoder produces a vector z f i = f (s i ; φ), and the decoder h projects the vector to x i = h(z f i ; U ) = U z f i , which has the same dimension as the word vectors v w . Negative sampling is applied to calculate the likelihood of generating the j-th word in the (i + 1)-th sentence, shown in Eq. 1.

EQUATION

where v w k are pretrained word vectors for w k , the empirical distribution P e (w) is the unigram distribution raised to power 0.75 (Mikolov et al., 2013) , and K is the number of negative samples. The learning objective is to maximise the likelihood for words in all sentences in the training corpus.

Ideally, the inverse of h should be easy to compute so that during testing we can set g = h -1 . As h is a linear projection, the simplest situation is when U is an orthogonal matrix and its inverse is equal to its transpose. Often, as the dimensionality of vector z f i doesn't necessarily need to match that of word vectors v w , U is not a square matrix 1 . To enforce invertibility on U , a row-wise orthonormal regularisation on U is applied during training, which leads to U U = I, where I is the identity matrix, thus the inverse function is simply h -1 (x) = U x, which is easily computed. The regularisation formula is ||U U -I|| F , where || • || F is the Frobenius norm. Specifically, the update rule (Cissé et al., 2017) for the regularisation is:

U := (1 + β)U -β(U U )U (2)

where β is set to 0.01. After learning, we set W = U , then the inverse of the decoder h becomes the encoder g. Compared to prior work with generative objective, our framework reuses the decoding function rather than ignoring it for building sentence representations after learning, thus information encoded in the decoder is also utilised.

2.3. DISCRIMINATIVE OBJECTIVE:

Our multi-view framework with discriminative objective learns to maximise the agreement between the representations of a sentence pair across two views if one sentence in the pair is in the neighbourhood of the other one. An RNN encoder f (s; φ) and a linear avg-on-word-vectors g(s; W ) produce a vector representation z f i and z g i for i-th sentence respectively. The agreement between two views of a sentence pair (s i , s j ) is defined as

a ij = a ji = cos(z f i , z g j ) + cos(z g i , z f j ).

The training objective is to minimise the loss function:

L(φ, W ) = - |i-j|≤c log p ij ,

where

p ij = e aij /τ i+N/2-1 n=i-N/2 e ain/τ

(3)

where τ is the trainable temperature term, which is essential for exaggerating the difference between adjacent sentences and those that are not. The neighbourhood/context window c, and the batch size N are hyperparameters.

The choice of cosine similarity based loss is based on the observations (Turney & Pantel, 2010) that, of word vectors derived from distributional similarity, vector length tends to correlate with frequency of words, thus angular distance captures more important meaning-related information. Also, since our model is unsupervised/self-supervised, whatever similarity there is between neighbouring sentences is what is learnt as important for meaning.

2.4. POSTPROCESSING:

The postprocessing step (Arora et al., 2017) , which removes the top principal component of a batch of representations, is applied on produced representations from f and g respectively after learning with a final l 2 normalisation.

In addition, in our multi-view framework with discriminative objective, in order to reduce the discrepancy between training and testing, the top principal component is estimated by the power iteration method (Mises & Pollaczek-Geiringer, 1929) and removed during learning.

3. EXPERIMENTAL DESIGN:

Three unlabelled corpora from different genres are used in our experiments, including BookCorpus (Zhu et al., 2015) , UMBC News (Han et al., 2013) and Amazon Book Review2 (McAuley et al., 2015) ; six models are trained separately on each of three corpora with each of two objectives. The summary statistics of the three corpora can be found in Table 1 . Adam optimiser (Kingma & Ba, 2014) and gradient clipping (Pascanu et al., 2013) are applied for stable training. Pretrained word vectors, fastText (Bojanowski et al., 2017) , are used in our frameworks and fixed during learning.

Table 1 : Summary statistics of the three corpora used in our experiments. For simplicity, the three corpora will be referred to as 1, 2 and 3 in the following tables respectively. Bi-GRU f :

z f i [max(H i ); mean(H i ); min(H i ); h M i i ] mean(H i ) Linear g: z g i [max(W X i ); mean(W X i ); min(W X i )] mean(W X i ) Ensemble Concatenation Averaging

All of our experiments including training and testing are done in PyTorch (Paszke et al., 2017) . The modified SentEval (Conneau & Kiela, 2018) package with the step that removes the first principal component is used to evaluate our models on the downstream tasks. Hyperparameters, including negative samples K in the framework with generative objective, context window c in the one with discriminative objective, are tuned only on the averaged performance on STS14 of the model trained on the BookCorpus; STS14/G1 and STS14/D1 results are thus marked with a in Table 3 

3.1. UNSUPERVISED EVALUATION -TEXTUAL SIMILARITY TASKS:

Representation: For a given sentence input s with M words, suggested by (Pennington et al., 2014; Levy et al., 2015) , the representation is calculated as z = ẑf + ẑg /2, where ẑ refers to the post-processed and normalised vector, and is mentioned in Table 2 .

Tasks: The unsupervised tasks include five tasks from SemEval Semantic Textual Similarity (STS) in 2012-2016 (Agirre et al., 2012; 2013; 2014; 2015; 2016) and the SemEval2014 Semantic Relatedness task (SICK-R) (Marelli et al., 2014) .

Comparison: We compare our models with: • Unsupervised learning: We selected models with strong results from related work, including fastText, fastText+WR. • Semi-supervised learning: The word vectors are pretrained on each task (Wieting et al., 2015) without label information, and word vectors are averaged to serve as the vector representation for a given sentence (Arora et al., 2017) .

• Supervised learning: ParaNMT (Wieting & Gimpel, 2018) is included as a supervised learning method as the data collection requires a neural machine translation system trained in supervised fashion. The InferSent3 (Conneau et al., 2017) trained on SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) is included as well.

The results are presented in Table 3 . Since the performance of FastSent (Hill et al., 2016) and QT (Logeswaran & Lee, 2018) were only evaluated on STS14, we compare to their results in Table 4 .

All six models trained with our learning frameworks outperform other unsupervised and semisupervised learning methods, and the model trained on the UMBC News Corpus with discriminative objective gives the best performance likely because the STS tasks contain multiple news-and headlines-related datasets which is well matched by the domain of the UMBC News Corpus.

3.2. SUPERVISED EVALUATION:

The evaluation on these tasks involves learning a linear model on top of the learnt sentence representations produced by the model. Since a linear model is capable of selecting the most relevant dimensions in the feature vectors to make predictions, it is preferred to concatenate various types of representations to form a richer, and possibly more redundant feature vector, which allows the linear model to explore the combination of different aspects of encoding functions to provide better results.

Representation: Inspired by prior work (McCann et al., 2017; Shen et al., 2018) , the representation z f is calculated by concatenating the outputs from the global mean-, max-and min-pooling on top of the hidden states H, and the last hidden state, and z g is calculated with three pooling functions as well. The post-processing and the normalisation step is applied individually. These two representations are concatenated to form a final sentence representation. Table 2 presents the details.

Tasks: Semantic relatedness (SICK) (Marelli et al., 2014) , paraphrase detection (MRPC) (Dolan et al., 2004) , question-type classification (TREC) (Li & Roth, 2002) , movie review sentiment (MR) (Pang & Lee, 2005) , Stanford Sentiment Treebank (SST) (Socher et al., 2013) , customer product reviews (CR) (Hu & Liu, 2004) , subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004) , opinion polarity (MPQA) (Wiebe et al., 2005) . The results are presented in Table 5 .

Comparison: Our results as well as related results of supervised task-dependent training models, supervised learning models, and unsupervised learning models are presented in Table 5 . Note that, for fair comparison, we collect the results of the best single model of MC-QT (Logeswaran & Lee, 2018) trained on BookCorpus.

Six models trained with our learning frameworks either outperform other existing methods, or achieve similar results on some tasks. The model trained on the Amazon Book Review gives the best performance on sentiment analysis tasks, since the corpus conveys strong sentiment information. 

a ij = cos(z f 1 i , z f 2 j ) + cos(z f 2 i , z f 1 j ) Multi-view with g 1 and g 2 : a ij = cos(z g 1 i , z g 2 j ) + cos(z g 2 i , z g 1 j ) z f

4. DISCUSSION:

In both frameworks, RNN encoder and linear encoder perform well on all tasks, and generative objective and discriminative objective give similar performance.

4.1. GENERATIVE OBJECTIVE: REGULARISATION ON INVERTIBILITY:

The orthonormal regularisation applied on the linear decoder to enforce invertibility in our multiview framework encourages the vector representations produced by f and those by h -1 , which is g in testing, to agree/align with each other. A direct comparison is to train our multi-view framework without the invertible constraint, and still directly use U as an additional encoder in testing. The results of our framework with and without the invertible constraint are presented in Table 6 .

The ensemble method of two views, f and g, on unsupervised evaluation tasks (STS12-16 and SICK14) is averaging, which benefits from aligning representations from f and g by applying invertible constraint, and the RNN encoder f gets improved on unsupervised tasks by learning to align with g. On supervised evaluation tasks, as the ensemble method is concatenation and a linear model is applied on top of the concatenated representations, as long as the encoders in two views process sentences distinctively, the linear classifier is capable of picking relevant feature dimensions from both views to make good predictions, thus there is no significant difference between our multi-view framework with and without invertible constraint.

4.2. DISCRIMINATIVE OBJECTIVE: MULTI-VIEW VS. SINGLE-VIEW:

In order to determine if the multi-view framework with two different views/encoding functions is helping the learning, we compare our framework with discriminative objective to other reasonable variants, including the multi-view model with two functions of the same type but parametrised independently, either two f -s or two g-s, and the single-view model with only one f or g. Table 6 presents the results of the models trained on UMBC News Corpus.

As specifically emphasised in previous work (Hill et al., 2016) , linear/log-linear models, which include g in our model, produce better representations for unsupervised evaluation tasks than RNNbased models do. This can also be observed in Table 6 as well, where g consistently provides better results on unsupervised tasks than f . In addition, as expected, multi-view learning with f and g, improves the resulting performance of f on unsupervised tasks, also improves the resulting g on supervised evaluation tasks.

Provided the results of models with generative and discriminative objective in Table 6 , we confidently show that, in our multi-view frameworks with f and g, the two encoding functions improve each other's view.

4.3. ENSEMBLE IN MULTI-VIEW FRAMEWORKS:

In general, aligning the representations generated from two distinct encoding functions ensures that the ensemble of them performs better. The two encoding functions f and g encode the input sentence with emphasis on different aspects, and the trained linear model for each of the supervised downstream tasks benefits from this diversity leading to better predictions. However, on unsupervised evaluation tasks, simply averaging representations from two views without aligning them during learning leads to poor performance and it is worse than g (linear) encoding function solely. Our multi-view frameworks ensure that the ensemble of two views provides better performance on both supervised and unsupervised evaluation tasks.

Compared with the ensemble of two multi-view models, each with two encoding functions of the same type, our multi-view framework with f and g provides slightly better results on unsupervised tasks, and similar results on supervised evaluation tasks, while our model has much higher training efficiency. Compared with the ensemble of two single-view models, each with only one encoding function, the matching between f and g in our multi-view model produces better results.

5. CONCLUSION:

We proposed multi-view sentence representation learning frameworks with generative and discriminative objectives; each framework combines an RNN-based encoder and an average-on-wordvectors linear encoder and can be efficiently trained within a few hours on a large unlabelled corpus.

The experiments were conducted on three large unlabelled corpora, and meaningful comparisons were made to demonstrate the generalisation ability and transferability of our learning frameworks and consolidate our claim. The produced sentence representations outperform existing unsupervised transfer methods on unsupervised evaluation tasks, and match the performance of the best unsupervised model on supervised evaluation tasks.

Our experimental results support the finding (Hill et al., 2016) that linear/log-linear models (g in our frameworks) tend to work better on the unsupervised tasks, while RNN-based models (f in our frameworks) generally perform better on the supervised tasks. As presented in our experiments, multi-view learning helps align f and g to produce better individual representations than when they are learned separately. In addition, the ensemble of both views leveraged the advantages of both, and provides rich semantic information of the input sentence. Future work should explore the impact of having various encoding architectures and learning under the multi-view framework.

Our multi-view learning frameworks were inspired by the asymmetric information processing in the two hemispheres of the human brain, in which the left hemisphere is thought to emphasise sequential processing and the right one more parallel processing (Bryden, 2012). Our experimental results raise an intriguing hypothesis about how these two types of information processing may complementarily help learning.

APPENDIX 1 EVALUATION TASKS:

The details, including size of each dataset and number of classes, about the evaluation tasks are presented below1 . 

2. POWER ITERATION:

The Power Iteration was proposed in Mises & Pollaczek-Geiringer (1929) , and it is an efficient algorithm for estimating the top eigenvector of a given covariance matrix. Here, it is used to estimate the top principal component from the representations produced from f and g separately. We omit the superscription here, since the same step is applied to both f and g.

Suppose there is a batch of representations Z = [z 1 , z 2 ..., z N ] ∈ R 2d×N from either f or g, the Power Iteration method is applied here to estimate the top eigenvector of the covariance matrix2 : C = ZZ , and it is described in Algorithm 1:

Algorithm 1 Estimating the First Principal Component (Mises & Pollaczek-Geiringer, 1929) Input: Covariance matrix C ∈ R 2d×2d , number of iterations T Output: First principal component u ∈ R 2d 1: Initialise a unit length vector u ∈ R 2d 2: for t ← 1, T do In our experiments, T is set to be 5.

3. TRAINING & MODEL DETAILS:

The hyperparameters we need to tune include the batch size N , the dimension of the GRU encoder d, and the context window c, and the number of negative samples K. The results we presented in this paper is based on the model trained with N = 512, d = 1024. Specifically, in discriminative objective, the context window is set c = 3, and in generative objective, the number of negative samples is set K = 5. It takes up to 8GB on a GTX 1080Ti GPU.

The initial learning rate is 5 × 10 -4 , and we didn't anneal the learning rate through the training. All weights in the model are initialised using the method proposed in He et al. (2015) , and all gates in the bi-GRU are initialised to 1, and all biases in the single-layer neural network are zeroed before training. The word vectors are fixed to be those in the FastText (Bojanowski et al., 2017) , and we don't finetune them. Words that are not in the FastText's vocabulary are fixed to 0 vectors through training. The temperature term is initialised as 1, and is tuned by the gradient descent during training.

The temperature term is used to convert the agreement a ij to a probability distribution p ij in Eq. 1 in the main paper. In our experiments, τ is a trainable parameter initialised to 1 that decreased consistently through training. Another model trained with fixed τ set to the final value performed similarly.

4. EFFECT OF POST-PROCESSING STEP:

Table 2 : Effect of the Post-processing Step. 'WR' refers to the post-processing step (Arora et al., 2017) which removes the principal component of a set of learnt vectors. The postprocessing step overall improves the performance of our models on unsupervised evaluation tasks, and also improves the models with generative objective on supervised sentence similarity tasks. However, it doesn't have a significant impact on single sentence classification tasks. 

:

As often the dimension of sentence vectors are equal to or large than that of word vectors, U has more columns than rows. If it is not the case, then regulariser becomes ||U U -I||F .

Largest subset of Amazon Review.

The released InferSent(Conneau et al., 2017) model is evaluated with the postprocessing step.

Provided by https://github.com/facebookresearch/SentEval

In practice, often N is less than 2d, thus we estimate the top eigenvector of Z Z ∈ R N ×N .

Figures and Tables:

Representation

Table content: <table><tr><td>Name</td><td colspan="2"># of sentences mean # of words per sentence</td></tr><tr><td>BookCorpus (1)</td><td>74M</td><td>13</td></tr><tr><td>UMBC News (2)</td><td>134.5M</td><td>25</td></tr><tr><td>Amazon Book Review (3)</td><td>150.8M</td><td>19</td></tr><tr><td>Phase</td><td>Testing Supervised</td><td>Unsupervised</td></tr></table>

Results on unsupervised evaluation tasks (Pearson's r × 100) . Bold numbers are the best results among unsupervised transfer models, and underlined numbers are the best ones among all models. 'G' and 'D' refer to generative and discriminative objective respectively. 'WR' refers to the post-processing step that removes the top principal component. Arora et al. (2017); 2 Wieting et al. (2015); 3 Wieting & Gimpel (2018); 4 Conneau et al. (2017); 5 Wieting & Gimpel (2018); 6-10 Agirre et al. (2012; 2013; 2014; 2015; 2016); 11 Marelli et al. (2014); 12 Mikolov et al. (2017)

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">Un. Training</td><td/><td/><td/><td colspan="2">Semi.</td><td/><td>Su.</td></tr><tr><td>Task</td><td/><td/><td colspan="2">Multi-view</td><td/><td/><td colspan="2">fastText</td><td colspan="2">2 PSL</td><td>4 Infer</td><td>3 ParaNMT</td></tr><tr><td/><td>G1</td><td>G2</td><td>G3</td><td>D1</td><td>D2</td><td>D3</td><td colspan="2">12 avg 1 WR</td><td>1 avg</td><td>1 WR</td><td>Sent</td><td>(concat.)</td></tr><tr><td>6 STS12</td><td>60.0</td><td>61.3</td><td>60.1</td><td>60.9</td><td>64.0</td><td>60.7</td><td>58.3</td><td>58.8</td><td>52.8</td><td>59.5</td><td>58.2</td><td>67.7</td></tr><tr><td>7 STS13</td><td>60.5</td><td>61.8</td><td>60.2</td><td>60.1</td><td>61.7</td><td>59.9</td><td>51.0</td><td>59.9</td><td>46.4</td><td>61.8</td><td>48.5</td><td>62.8</td></tr><tr><td>8 STS14</td><td>71.1</td><td>72.1</td><td>71.5</td><td>71.5</td><td>73.7</td><td>70.7</td><td>65.2</td><td>69.4</td><td>59.5</td><td>73.5</td><td>67.1</td><td>76.9</td></tr><tr><td>9 STS15</td><td>75.7</td><td>76.9</td><td>75.5</td><td>76.4</td><td>77.2</td><td>76.5</td><td>67.7</td><td>74.2</td><td>60.0</td><td>76.3</td><td>71.1</td><td>79.8</td></tr><tr><td>10 STS16</td><td>75.4</td><td>76.1</td><td>75.1</td><td>75.8</td><td>76.7</td><td>74.8</td><td>64.3</td><td>72.4</td><td>-</td><td>-</td><td>71.2</td><td>76.8</td></tr><tr><td>11 SICK14</td><td>73.8</td><td>73.6</td><td>72.7</td><td>74.7</td><td>74.9</td><td>72.8</td><td>69.8</td><td>72.3</td><td>66.4</td><td>72.9</td><td>73.4</td><td>-</td></tr><tr><td>Average</td><td>69.4</td><td>70.3</td><td>69.2</td><td>69.9</td><td>71.4</td><td>69.2</td><td>62.7</td><td>67.8</td><td>-</td><td>-</td><td>64.9</td><td>-</td></tr></table>

Comparison with FastSent and QT on STS14 (Pearson's r × 100).

Table content: <table><tr><td colspan="2">FastSent (Hill et al., 2016)</td><td colspan="2">QT (Logeswaran &amp; Lee, 2018)</td><td/><td/><td colspan="2">Multi-view</td><td/><td/></tr><tr><td/><td>+AE</td><td>RNN</td><td>BOW</td><td>G1</td><td>G2</td><td>G3</td><td>D1</td><td>D2</td><td>D3</td></tr><tr><td>61.2</td><td>59.5</td><td>49.0</td><td>65.0</td><td>71.1</td><td>72.1</td><td>71.5</td><td>71.5</td><td>73.7</td><td>70.7</td></tr></table>

Supervised evaluation tasks. Bold numbers are the best results among unsupervised transfer models, and underlined numbers are the best ones among all models. " †" refers to an ensemble of two models. " ‡" indicates that additional labelled discourse information is required. Our models perform similarly or better than existing methods, but with higher training efficiency. Conneau et al. (2017); 2 Arora et al. (2017); 3 Hill et al. (2016); 4 Kiros et al. (2015); 5 Gan et al. (2017); 6 Jernite et al. (2017); 7 Nie et al. (2017); 8 Tai et al. (2015); 9 Zhao et al. (2015) 10 Le & Mikolov (2014); 11 Logeswaran & Lee (2018); 12 Shen et al. (2018);13 Mikolov et al. (2017)

Table content: <table><tr><td>Model</td><td>Hrs</td><td>SICK-R</td><td>SICK-E</td><td>MRPC</td><td colspan="2">TREC MR</td><td>CR</td><td colspan="3">SUBJ MPQA SST</td></tr><tr><td/><td/><td colspan="6">Supervised task-dependent training -No transfer learning</td><td/><td/><td/></tr><tr><td>9 AdaSent</td><td>-</td><td>-</td><td>-</td><td>-</td><td>92.4</td><td>83.1</td><td>86.3</td><td>95.5</td><td>93.3</td><td>-</td></tr><tr><td>8 TF-KLD</td><td>-</td><td>-</td><td>-</td><td>80.4/85.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>12 SWEM-concat</td><td>-</td><td>-</td><td>-</td><td>71.5/81.3</td><td>91.8</td><td>78.2</td><td>-</td><td>93.0</td><td>-</td><td>84.3</td></tr><tr><td/><td/><td/><td colspan="4">Supervised training -Transfer learning</td><td/><td/><td/><td/></tr><tr><td>1 InferSent</td><td>&lt;24</td><td>88.4</td><td>86.3</td><td>76.2/83.1</td><td>88.2</td><td>81.1</td><td>86.3</td><td>92.4</td><td>90.2</td><td>84.6</td></tr><tr><td/><td/><td colspan="5">Unsupervised training with unordered sentences</td><td/><td/><td/><td/></tr><tr><td>10 ParagraphVec</td><td>4</td><td>-</td><td>-</td><td>72.9/81.1</td><td>59.4</td><td>60.2</td><td>66.9</td><td>76.3</td><td>70.7</td><td>-</td></tr><tr><td>2 GloVe+WR</td><td>-</td><td>86.0</td><td>84.6</td><td>-/ -</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>82.2</td></tr><tr><td>13 fastText+bow</td><td>-</td><td>-</td><td>-</td><td>73.4/81.6</td><td>84.0</td><td>78.2</td><td>81.1</td><td>92.5</td><td>87.8</td><td>82.0</td></tr><tr><td>3 SDAE</td><td>72</td><td>-</td><td>-</td><td>73.7/80.7</td><td>78.4</td><td>74.6</td><td>78.0</td><td>90.8</td><td>86.9</td><td>-</td></tr><tr><td/><td/><td colspan="5">Unsupervised training with ordered sentences</td><td/><td/><td/><td/></tr><tr><td>3 FastSent</td><td>2</td><td>-</td><td>-</td><td>72.2/80.3</td><td>76.8</td><td>70.8</td><td>78.4</td><td>88.7</td><td>80.6</td><td>-</td></tr><tr><td>4 Skip-thought</td><td>336</td><td>85.8</td><td>82.3</td><td>73.0/82.0</td><td>92.2</td><td>76.5</td><td>80.1</td><td>93.6</td><td>87.1</td><td>82.0</td></tr><tr><td>5 CNN-LSTM  †</td><td>-</td><td>86.2</td><td>-</td><td>76.5/83.8</td><td>92.6</td><td>77.8</td><td>82.1</td><td>93.6</td><td>89.4</td><td>-</td></tr><tr><td>6 DiscSent  ‡</td><td>8</td><td>-</td><td>-</td><td>75.0/ -</td><td>87.2</td><td>-</td><td>-</td><td>93.0</td><td>-</td><td>-</td></tr><tr><td>7 DisSent  ‡</td><td>-</td><td>79.1</td><td>80.3</td><td>-/ -</td><td>84.6</td><td>82.5</td><td>80.2</td><td>92.4</td><td>89.6</td><td>82.9</td></tr><tr><td>11 MC-QT</td><td>11</td><td>86.8</td><td>-</td><td>76.9/84.0</td><td>92.8</td><td>80.4</td><td>85.2</td><td>93.9</td><td>89.4</td><td>-</td></tr><tr><td>Multi-view G1</td><td>3.5</td><td>88.1</td><td>85.2</td><td>76.5/83.7</td><td>90.0</td><td>81.3</td><td>83.5</td><td>94.6</td><td>89.5</td><td>85.9</td></tr><tr><td>Multi-view G2</td><td>9</td><td>87.8</td><td>85.9</td><td>77.5/83.8</td><td>92.2</td><td>81.3</td><td>83.4</td><td>94.7</td><td>89.5</td><td>85.9</td></tr><tr><td>Multi-view G3</td><td>9</td><td>87.7</td><td>84.4</td><td>76.0/83.7</td><td>90.6</td><td>84.0</td><td>85.6</td><td>95.3</td><td>89.7</td><td>88.7</td></tr><tr><td>Multi-view D1</td><td>3</td><td>87.9</td><td>84.8</td><td>77.1/83.4</td><td>91.8</td><td>81.6</td><td>83.9</td><td>94.5</td><td>89.1</td><td>85.8</td></tr><tr><td>Multi-view D2</td><td>8.5</td><td>87.8</td><td>85.2</td><td>76.8/83.9</td><td>91.6</td><td>81.5</td><td>82.9</td><td>94.7</td><td>89.3</td><td>84.9</td></tr><tr><td>Multi-view D3</td><td>8</td><td>87.7</td><td>85.2</td><td>75.7/82.5</td><td>89.8</td><td>85.0</td><td>85.7</td><td>95.7</td><td>90.0</td><td>89.6</td></tr></table>

Ablation study on our multi-view frameworks. Variants of our frameworks are tested to illustrate the advantage of our multi-view learning frameworks. In general, under the proposed frameworks, learning to align representations from both views helps each view to perform better and an ensemble of both views provides stronger results than each of them. The arrow and value pair indicate how a result differs from our multi-view learning framework. Better view in colour. Multi-view with f 1 and f 2 :

Table content: <table><tr><td/><td/><td>Unsupervised tasks</td><td/><td>Supervised tasks</td><td/></tr><tr><td>UMBC News</td><td>Hrs</td><td>Avg of STS tasks (STS12-16, SICK14)</td><td>Avg of SICK-R, STS-B</td><td>Avg of Binary-CLS tasks (MR, CR, SUBJ, MPQA, SST)</td><td>MRPC</td></tr><tr><td/><td/><td colspan="3">Our Multi-view with Generative Objective + Invertible Constraint</td><td/></tr><tr><td>z f</td><td/><td>66.6</td><td>82.0</td><td>86.1</td><td>74.7/83.1</td></tr><tr><td>z g</td><td>9</td><td>67.8</td><td>82.3</td><td>85.3</td><td>74.8/82.2</td></tr><tr><td>en(z f , z g )</td><td/><td>70.3</td><td>82.7</td><td>87.0</td><td>77.5/83.8</td></tr><tr><td/><td/><td colspan="3">Generative Objective without Invertible Constraint</td><td/></tr><tr><td>z f</td><td/><td>55.7 (↓10.9)</td><td>79.9 (↓2.1)</td><td>86.0 (↓0.1)</td><td>73.2/81.7</td></tr><tr><td>z g</td><td>9</td><td>70.1 (↑2.3)</td><td>82.8 (↑0.5)</td><td>85.0 (↓0.3)</td><td>74.3/82.0</td></tr><tr><td>en(z f , z g )</td><td/><td>67.8 (↓2.5)</td><td>82.9 (↑0.2)</td><td>86.4 (↓0.7)</td><td>74.8/83.2</td></tr><tr><td/><td colspan="4">Our Multi-view Discriminative Objective: a ij = cos(z f i , z g j ) + cos(z g i , z f j )</td><td/></tr><tr><td>z f</td><td/><td>67.4</td><td>83.0</td><td>86.6</td><td>75.5/82.7</td></tr><tr><td>z g</td><td>8</td><td>69.2</td><td>82.6</td><td>85.2</td><td>74.3/82.7</td></tr><tr><td>en(z f , z g )</td><td/><td>71.4</td><td>83.0</td><td>86.6</td><td>76.8/83.9</td></tr></table>

Details about the evaluation tasks used in our experiments.

Table content: <table><tr><td colspan="3">Task name |Train| |Test|</td><td>Task</td><td>Classes</td></tr><tr><td/><td/><td colspan="2">Relatively Small-scale</td><td/></tr><tr><td>MR</td><td>11k</td><td>11k</td><td>sentiment (movies)</td><td>2</td></tr><tr><td>CR</td><td>4k</td><td>4k</td><td>product reviews</td><td>2</td></tr><tr><td>SUBJ</td><td>10k</td><td>10k</td><td>subjectivity/objectivity</td><td>2</td></tr><tr><td>MPQA</td><td>11k</td><td>11k</td><td>opinion polarity</td><td>2</td></tr><tr><td>TREC</td><td>6k</td><td>0.5k</td><td>question-type</td><td>6</td></tr><tr><td>SICK-R</td><td>4.5k</td><td colspan="2">4.9k semantic textual similarity</td><td>6</td></tr><tr><td>STS-B</td><td>5.7k</td><td colspan="2">1.4k semantic textual similarity</td><td>6</td></tr><tr><td>MRPC</td><td>4k</td><td>1.7k</td><td>paraphrase</td><td>2</td></tr><tr><td>SICK-E</td><td>4.5k</td><td>4.9k</td><td>NLI</td><td>3</td></tr><tr><td/><td/><td colspan="2">Relatively Large-scale</td><td/></tr><tr><td>SST-2</td><td>67k</td><td>1.8k</td><td>sentiment (movies)</td><td>2</td></tr></table>

Peer review (iclr2019):
 After reading through the authors* comments and rereading parts of the submission, I have become a little more positive about this paper. I am still unsure about the contributions to the ICLR community. The authors merely state "We believe our contributions to ICLR community are clear and valuable" without backing up this claim with an argument. But in the rest of the authors* comment, they make some good points. I think those points should be made more prominently in the paper itself. I would suggest that the authors describe their approach as using different, complementary encoders of the input sentence and consensus maximization. If they wish to describe this as multi-view learning, that*s fine, but I think using the term "consensus maximization" (or something more descriptive like that) in prominent places would be helpful. If the approach is applicable beyond sentence embedding learning, then it would behoove the authors to describe the approach in a general way so that readers will see how they can apply it to their own tasks. As currently written, the paper is very much focused on sentence embedding learning, which causes me to think that the paper is more appropriate for an NLP venue. But it is true that ICLR publishes papers that are application-specific, so I can*t consider this to be a deal-breaker for the paper. I raised my score to a 6. --------------------------------- original review follows: ---------------------------------- This paper describes experiments in learning sentence embeddings from unlabeled text. The paper compares a few different compositional architectures and training objectives. The story of the paper focuses on the training of multiple architectures jointly for a single sentence, then ensembling those architectures at test time to represent sentences. One architecture is an RNN and the other is a word averaging model, and the idea is that these two architectures capture different "views" of the sentence. Pros: As a general-purpose method to get sentence embeddings without using any resources other than unannotated text documents, this approach has strong results, including solid performance on the SentEval tasks and relatively-low training times. It was also nice to see how the results depend on the domain of the training data. Review data definitely helps on the several sentiment-related tasks, which provides further evidence of a worrisome aspect of SentEval. Cons: Overall, the paper feels incremental and is likely a better fit for an NLP conference. What are the generalizable contributions to the ICLR community? Given the known differences between RNNs and word averaging models for sentences (especially on the SentEval tasks, which, as the authors note, was discussed by Hill et al.), it*s entirely unsurprising that combining the two would be a good idea. But even if this were not the case, the ubiquity of ensembles outperforming single models in deep learning also makes it unsurprising that combining these two kinds of model architectures would be beneficial. So I*m just not sure if there is a significant contribution beyond the NLP results. These results seem solid (though a bit incremental), but if the primary contribution is empirical, then the paper would be a better fit for an NLP venue. In addition, I*m not sure if "multi-view" is an appropriate description of the approach. In Sec. 1, we find the sentence "Compared to earlier work on multi-view learning (de Sa, 1993; Dhillon et al., 2011) that takes data from various sources or splits data into disjoint populations, our framework processes the exact same input data in two distinctive ways." Therefore, maybe it*s not quite accurate to describe this approach using the term "multi-view learning"? I think it would make more sense to use a different term rather than stretch the definition for a well-known one. I kept expecting the paper to present results when combining the generative and discriminative objectives, but as far as I can tell, this was never done. What would happen if one were to use multitask learning and just optimize the sum of the two losses? I*d suggest citing and comparing to the results from "Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning" by Subramanian et al. (ICLR 2018) and the Byte mLSTM from "Learning to generate reviews and discovering sentiment" by Radford et al. I*m not sure how excited we should get about not using any annotations or structured resources for learning sentence embeddings. The authors do not motivate this goal. Below are more specific comments/questions: Sec. 2.2 contains the sentence "Ideally, the inverse of h should be easy to compute so that during testing we can set g = h^-1." At this point in the paper, it is not clear what g is going to be applied to at test time, since presumably the following sentence is not going to be available at test time, right? I think it would be good to discuss how the model is going to be used at test time before discussing the inverse of h. Sec. 2.3: In Eq. (3), why does the denominator sum always start at 1 no matter what i and j are? That is, why would the denominator always sum over the first N sentences in the dataset? I think the pooling methods in Table 2 should be described in Section 2. In Table 2, it is not clear what h_i^{M_i} is. If M_i is the number of words in sentence i, that should be mentioned somewhere. Sec. 3.1: "For a given sentence input s with M words, suggested by (Pennington et al., 2014; Levy et al., 2015), the representation is calculated as z = (.hat{z}_f + .hat{z}_g) / 2, where .hat{z} refers to the post-processed and normalised vector, and is mentioned in Table 2." I don*t understand. Where is this mentioned in Table 2? Minor issues follow: Sec. 1: "Distributional hypothesis" --> "the distributional hypothesis" "in machine learning community" --> "in the machine learning community" "and distributional hypothesis" --> "and the distributional hypothesis" "the linear/log-linear models" --> "linear/log-linear models" "based on distributional hypothesis" --> "based on the distributional hypothesis" "contraint" --> "constraint" Sec. 2: "marry RNN-based sentence encoder" --> "marry RNN-based sentence encoders" or "marry the RNN-based sentence encoder" Sec. 2.1: "only hidden state" --> "only the hidden state" Sec. 2.2: "prior work with generative objective" --> "prior work with generative objectives" Sec. 2.3: "with discriminative objective learns" --> "with the discriminative objective learns" Sec. 3.1: In Table 3, I don*t see where superscript 5 is shown in the table. Sec. 3.2: In Table 5, the numbered superscripts at the top of the table do not show up next to the methods in the actual table rows.