Improving Sentence Representations with Multi-view Frameworks

Multi-view learning can provide self-supervision when different views are available of the same data . Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora . Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning , we present two multi-view frameworks for learning sentence representations in an unsupervised fashion . One framework uses a generative objective and the other a discriminative one . In both frameworks , the final representation is an ensemble of two views , in which , one view encodes the input sentence with a Recurrent Neural Network ( RNN ) , and the other view encodes it with a simple linear model . We show that , after learning , the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts , and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks . 1 INTRODUCTION . Multi-view learning methods provide the ability to extract information from different views of the data and enable self-supervised learning of useful features for future prediction when annotated data is not available ( de Sa , 1993 ) . Minimising the disagreement among multiple views helps the model to learn rich feature representations of the data and , also after learning , the ensemble of the feature vectors from multiple views can provide an even stronger generalisation ability . Distributional hypothesis ( Harris , 1954 ) noted that words that occur in similar contexts tend to have similar meaning ( Turney & Pantel , 2010 ) , and distributional similarity ( Firth , 1957 ) consolidated this idea by stating that the meaning of a word can be determined by the company it has . The hypothesis has been widely used in machine learning community to learn vector representations of human languages . Models built upon distributional similarity don ’ t explicitly require humanannotated training data ; the supervision comes from the semantic continuity of the language data . Large quantities of annotated data are usually hard and costly to obtain , thus it is important to study unsupervised and self-supervised learning . Our goal is to propose learning algorithms built upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelled data . We draw inspiration from the lateralisation and asymmetry in information processing of the two hemispheres of the human brain where , for most adults , sequential processing dominates the left hemisphere , and the right hemisphere has a focus on parallel processing ( Bryden , 2012 ) , but both hemispheres have been shown to have roles in literal and non-literal language comprehension ( Coulson et al. , 2005 ; Coulson & van Petten , 2007 ) . Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models , which have been widely applied in sentiment analysis tasks ( Yang et al. , 2016 ) , and the linear/loglinear models , which have excelled at capturing attributional similarities of words and sentences ( Arora et al. , 2016 ; 2017 ; Hill et al. , 2016 ; Turney & Pantel , 2010 ) for learning sentence representations . Previous work on unsupervised sentence representation learning based on distributional hypothesis can be roughly categorised into two types : Generative objective : These models generally follow the encoder-decoder structure . The encoder learns to produce a vector representation for the current input , and the decoder learns to generate sentences in the adjacent context given the produced vector ( Kiros et al. , 2015 ; Hill et al. , 2016 ; Gan et al. , 2017 ; Tang et al. , 2018 ) . The idea is straightforward , yet its scalability for very large corpora is hindered by the slow decoding process that dominates training time , and also the decoder in each model is discarded after learning as the quality of generated sequences is not the main concern , which is a waste of parameters and learning effort . Our first multi-view framework has a generative objective and uses an RNN as the encoder and an invertible linear projection as the decoder . The training time is drastically reduced as the decoder is simple , and the decoder is also utilised after learning . A regularisation is applied on the linear decoder to enforce invertibility , so that after learning , the inverse of the decoder can be applied as a linear encoder in addition to the RNN encoder . Discriminative Objective : In these models , a classifier is learnt on top of the encoders to distinguish adjacent sentences from those that are not ( Li & Hovy , 2014 ; Jernite et al. , 2017 ; Nie et al. , 2017 ; Logeswaran & Lee , 2018 ) ; these models make a prediction using a predefined differentiable similarity function on the representations of the input sentence pairs or triplets . Our second multi-view framework has a discriminative objective and uses an RNN encoder and a linear encoder ; it learns to maximise agreement among adjacent sentences . Compared to earlier work on multi-view learning ( de Sa , 1993 ; Dhillon et al. , 2011 ; Wang et al. , 2015 ) that takes data from various sources or splits data into disjoint populations , our framework processes the exact same data in two distinctive ways . The two distinctive information processing views tend to encode different aspects of an input sentence ; forcing agreement/alignment between these views encourages each view to be a better representation , and is beneficial to the future use of the learnt representations . Our contribution is threefold : • Two multi-view frameworks for learning sentence representations are proposed , in which one framework uses a generative objective and the other one adopts a discriminative objective . Two encoding functions , an RNN and a linear model , are learnt in both frameworks . • The results show that in both frameworks , aligning representations from two views gives improved performance of each individual view on all evaluation tasks compared to their single-view trained counterparts , and furthermore ensures that the ensemble of two views provides even better results than each improved view alone . • Models trained under our proposed frameworks achieve good performance on the unsupervised tasks , and overall outperform existing unsupervised learning models , and armed with various pooling functions , they also show solid results on supervised tasks , which are either comparable to or better than those of the best unsupervised transfer model . It is shown ( Hill et al. , 2016 ) that the consistency between supervised and unsupervised evaluation tasks is much lower than that within either supervised or unsupervised evaluation tasks alone and that a model that performs well on supervised evaluation tasks may fail on unsupervised tasks . It is subsequently showed ( Conneau et al. , 2017 ; Subramanian et al. , 2018 ) that , with large-scale labelled training corpora , the resulting representations of the sentences from the trained model excel in both supervised and unsupervised tasks , while the labelling process is costly . Our model is able to achieve good results on both groups of tasks without labelled information . 2 MODEL ARCHITECTURE . Our goal is to marry RNN-based sentence encoder and the avg-on-word-vectors sentence encoder into multi-view frameworks with simple objectives . The motivation for the idea is that , RNN-based encoders process the sentences sequentially , and are able to capture complex syntactic interactions , while the avg-on-word-vectors encoder has been shown to be good at capturing the coarse meaning of a sentence which could be useful for finding paradigmatic parallels ( Turney & Pantel , 2010 ) . We present two multi-view frameworks , each of which learns two different sentence encoders ; after learning , the vectors produced from two encoders of the same input sentence are used to compose the sentence representation . The details of our learning frameworks are described as follows : 2.1 ENCODERS . In our multi-view frameworks , we first introduce two encoders that , after learning , can be used to build sentence representations . One encoder is a bi-directional Gated Recurrent Unit ( Chung et al. , 2014 ) f ( s ; φ ) , where s is the input sentence and φ is the parameter vector in the GRU . During learning , only hidden state at the last time step is sent to the next stage in learning . The other encoder is a linear avg-on-word-vectors model g ( s ; W ) , which basically transforms word vectors in a sentence by a learnable weight matrixW and outputs an averaged vector . 2.2 GENERATIVE OBJECTIVE . Given the finding ( Tang et al. , 2018 ) that neither an autoregressive nor an RNN decoder is necessary for learning sentence representations that excel on downstream tasks , our learning framework only learns to predict words in the next sentence . The framework has an RNN encoder f , and a linear decoder h. Given an input sentence si , the encoder produces a vector z f i = f ( si ; φ ) , and the decoder h projects the vector to xi = h ( z f i ; U ) = Uz f i , which has the same dimension as the word vectors vw . Negative sampling is applied to calculate the likelihood of generating the j-th word in the ( i+ 1 ) -th sentence , shown in Eq . 1. logP ( wj |si ) = log σ ( x > i vwj ) + ∑K k=1 Ewk∼Pe ( w ) log σ ( −x > i vwk ) ( 1 ) where vwk are pretrained word vectors for wk , the empirical distribution Pe ( w ) is the unigram distribution raised to power 0.75 ( Mikolov et al. , 2013 ) , and K is the number of negative samples . The learning objective is to maximise the likelihood for words in all sentences in the training corpus . Ideally , the inverse of h should be easy to compute so that during testing we can set g = h−1 . As h is a linear projection , the simplest situation is when U is an orthogonal matrix and its inverse is equal to its transpose . Often , as the dimensionality of vector zfi doesn ’ t necessarily need to match that of word vectors vw , U is not a square matrix1 . To enforce invertibility on U , a row-wise orthonormal regularisation on U is applied during training , which leads to UU > = I , where I is the identity matrix , thus the inverse function is simply h−1 ( x ) = U > x , which is easily computed . The regularisation formula is ||UU > − I||F , where || · ||F is the Frobenius norm . Specifically , the update rule ( Cissé et al. , 2017 ) for the regularisation is : U : = ( 1 + β ) U − β ( UU > ) U ( 2 ) where β is set to 0.01 . After learning , we set W = U > , then the inverse of the decoder h becomes the encoder g. Compared to prior work with generative objective , our framework reuses the decoding function rather than ignoring it for building sentence representations after learning , thus information encoded in the decoder is also utilised . 2.3 DISCRIMINATIVE OBJECTIVE . Our multi-view framework with discriminative objective learns to maximise the agreement between the representations of a sentence pair across two views if one sentence in the pair is in the neighbourhood of the other one . An RNN encoder f ( s ; φ ) and a linear avg-on-word-vectors g ( s ; W ) produce a vector representation zfi and z g i for i-th sentence respectively . The agreement between two views of a sentence pair ( si , sj ) is defined as aij = aji = cos ( z f i , z g j ) + cos ( z g i , z f j ) . The training objective is to minimise the loss function : L ( φ , W ) = − ∑ |i−j|≤c log pij , where pij = eaij/τ∑i+N/2−1 n=i−N/2 e ain/τ ( 3 ) where τ is the trainable temperature term , which is essential for exaggerating the difference between adjacent sentences and those that are not . The neighbourhood/context window c , and the batch size N are hyperparameters . 1As often the dimension of sentence vectors are equal to or large than that of word vectors , U has more columns than rows . If it is not the case , then regulariser becomes ||U > U − I||F . The choice of cosine similarity based loss is based on the observations ( Turney & Pantel , 2010 ) that , of word vectors derived from distributional similarity , vector length tends to correlate with frequency of words , thus angular distance captures more important meaning-related information . Also , since our model is unsupervised/self-supervised , whatever similarity there is between neighbouring sentences is what is learnt as important for meaning .

Summary: This paper is about a multi-view framework for learning sentence representations. Two objective functions (a generative one and a discriminative one) are proposed that make use of two encoders, one of them is based on an RNN and the other on a linear projection of averaged word embeddings. Each of these objective functions has a multi-view framework where their respective objective functions are in part based on making sure their is some relationship between the two different encoders. This multi-view framework is shown to be helpful over having independent encoders in their ablation study.