Unsupervised Learning  of Sentence Representations Using Sequence Consistency

1 INTRODUCTION . In natural language processing , the use of distributed representations has become standard through the effective use of word embeddings . In a wide range of NLP tasks , it is beneficial to initialize the word embeddings with ones learned from large text corpora like word2vec ( Mikolov et al . ( 2013 ) ) or GLoVe ( Pennington et al . ( 2014 ) ) and tune them as a part of a target task e.g . text classification . It is therefore a natural question to ask whether such standardized representations of whole sentences that can be widely used in downstream tasks is possible . There are several of approaches to this problem . Taking cue from word2vec , an unsupervised learning approach is taken by SkipThought ( Kiros et al . ( 2015 ) ) , FastSent ( Hill et al . ( 2016 ) ) and QuickThoughts ( Logeswaran & Lee ( 2018 ) ) , exploiting the closeness of adjacent sentences in a text corpus . Models trained using a language modeling objective have also been shown to produce excellent sentence representations ( Ruder & Howard ( 2018 ) ; Radford ( 2018 ) ) . More recently , the work of Conneau et al . ( 2017 ) takes a supervised learning approach . They train a sentence encoder on large scale natural language inference datasets ( Bowman et al . ( 2015 ) ; Williams et al . ( 2018 ) ) and show that the learned encoding perform well on a set of transfer tasks . This is reminiscent of the approach taken by ImageNet ( Deng et al . ( 2009 ) ) in the computer vision community . In Subramanian et al . ( 2018 ) , the authors train a sentence encoder on multiple tasks to get improved performance . In this paper , we take a slightly different unsupervised approach to learning sentence representations . We define a sequence of tokens to be consistent if they form a valid sentence . While defining consistency precisely is difficult , we generate approximately inconsistent sequences of tokens by randomly perturbing consistent ones . We extend the notion of consistency to pairs of sequences – two sequences are defined to be consistent if they can me merged to form a consistent sequence . We then train a sentence encoder to discriminate between consistent and inconsistent sequences or pairs of sequences . Note that external supervised labels are not required for training as we generate our own labels using the notion of consistency . We call our proposed method ConsSent . We generate inconsistent sequences in four different ways – by deleting , permuting , inserting and replacing random tokens in a consistent sequence . For pairs of sequences , we generate inconsistent examples by randomly partitioning sentences into two parts and pairing parts from two different sentences , where the method of partitioning can be contiguous or non-contiguous . We vary the complexity of the training tasks by choosing a different number of tokens and pairs to generate inconsistent examples and train several sentence encoders on a set of unordered sentences from the Billionword corpus ( Chelba et al . ( 2014 ) ) . We also combine the training tasks and encoders in two different ways – one by training an encoder in a multitask setting and the other by creating an ensemble of models trained on the individual tasks . We evaluate the trained sentence encoders on a set of transfer learning and linguistic probing tasks encapsulated in the SentEval benchmark ( Conneau & Kiela ( 2018 ) ) . Our models achieve strong results in both settings , improving upon existing unsupervised and supervised methods of learning sentence representations in several cases . 2 RELATED WORK . Learning general sentence encoders is a fundamental problem in NLP and there is a long list of works that addresses this problem . To begin with , an untrained BiLSTM model with max pooling of the intermediate states performs fairly well on several transfer and linguistic probing tasks ( Conneau et al . ( 2018 ) ) . The next set of simple models consists of a bag-of-words approach using learned word embeddings like GloVe ( Pennington et al . ( 2014 ) ) or FastText ( Joulin et al . ( 2017 ) ) , where a simple average of the word embeddings in a sentence define the sentence embedding . Although very fast , these approaches are limited by the dimensions of the word embeddings and do not take into account the order of the words . Due to the availability of practically unlimited textual data , learning sentence encoders using unsupervised learning is an attractive proposition . The SkipThought model of Kiros et al . ( 2015 ) learns sentence encoders by using an encoder-decoder architecture . Exploiting the relatedness inherent in adjacent sentences , the model is trained by using the encoder to encode a particular sentence and then using the decoder to decode words in adjacent sentences . This approach is directly inspired by a similar objective for learning word embeddings like word2vec ( Mikolov et al . ( 2013 ) ) . In more recent work , Logeswaran & Lee ( 2018 ) propose QuickThoughts , which uses a form of discriminative training on encodings of sentences , by biasing the encodings of adjacent sentences to be closer to each other than non-adjacent sentences , where closeness is defined by the dot product . The Bag-of-Words approach is developed further in the FastSent model ( Hill et al . ( 2016 ) ) which uses a bag-of-words representation of a sentence to predict words in adjacent sentences . In work by Arora et al . ( 2017 ) it is shown that a simple post processing of the average word-embeddings can perform comparably or better than skip-thought like objectives . The work of Pagliardini et al . ( 2018 ) , where the authors use n-gram features for unsupervised learning is also relevant . The other notable unsupervised approach is that of using a denoising autoencoder ( Hill et al . ( 2016 ) ) . Language modeling is a very natural task for natural language for which models can be trained in an unsupervised manner . The context encoder trained on a large corpus with a language modeling objective can be used to generate sentence representations by encoding the sentence as a context . Work by Ruder & Howard ( 2018 ) and Radford ( 2018 ) has shown that training language models on large text corpora produce excellent sentence encoders that perform very well on downstream tasks . The recent work of Devlin et al . ( 2018 ) pushes this approach even further by training large transformers ( Vaswani et al . ( 2017 ) ) on very large text corpora using a slightly modified masked language modeling objective . The language modeling based approaches typically require corpora with ordered sentences or document level corpora , which is not a requirement in our case . Supervised learning has been used to learn better sentence encoders . Conneau et al . ( 2017 ) use the SNLI ( Bowman et al . ( 2015 ) ) and MultiNLI ( Williams et al . ( 2018 ) ) corpus to train a sentence encoder on the natural language inference task that performs well on several transfer tasks . Another domain where large datasets for supervised training is available is machine translation and the work of McCann et al . ( 2017 ) exploits this in learning sentence encoders by training it on the machine translation task . Subramanian et al . ( 2018 ) combine several supervised and unsupervised objectives in a multi-task framework to obtain some of the best results on learning general sentence representations . Our approach is based on automatically generating ( possibly noisy ) training data by perturbing sentences . Such an approach was used by Wagner et al . ( 2009 ) to train a classifier to judge the grammaticality of sentences . The ungrammatical sentences were generated by , among other things , dropping and inserting words . Recent work by Warstadt et al . ( 2018 ) extend this approach by using neural network classifiers . Finally , in parallel to our work , a recent report Ranjan et al . ( 2018 ) uses word dropping and word permutation to generate fake sentences and learn sentence encoders . Our work is substantially more general and exhaustive . 3 CONSISTENT SEQUENCES AND DISCRIMINATIVE TRAINING . Let S = { w1 , w2 , · · · , wn } be an ordered sequence of n tokens . We define this sequence to be consistent if the tokens form a valid sentence . Let E be an encoder that encodes any sequence of tokens into a fixed length distributed representation . Our goal is to train E to discriminate between consistent sequences and inconsistent ones . We argue that such an encoder will have to encapsulate many structural properties of sentences , thereby resulting in a good sentence representation . Whether a sequence of tokens S is consistent is a notoriously hard problem to solve . We take a slightly different approach . We start from a consistent sequence ( e.g . from some text corpus ) and introduce random perturbations to generate sequences S′ that are not consistent . We take inspiration from the standard operations in string edit distance and consider the following variations for generating S′ . • ConsSent-D ( k ) : We pick k random tokens in S and delete them . • ConsSent-P ( k ) : We pick k random tokens in S and permute them randomly ( avoiding the identity permutation ) . • ConsSent-I ( k ) : We pick k random tokens not from S and insert them at random positions in S. • ConsSent-R ( k ) : We pick k random tokens in S and replace them with other random tokens not in S. It is important to note that it is possible that in some cases S′ will itself form a valid sentence and hence violate the definition of inconsistency . We do not address this issue and assume that such cases will be a relatively few and will not influence the encoder in a substantial manner . Also , with larger values of k , the chance of this happening goes down . We train E to distinguish between S and S′ by using a binary classifier on top of the representations generated by E. We extend the definition of consistency to pairs of sequences . Given two sequences , we define them to be consistent if they can be merged into a consistent sequence without changing the order of the tokens . Similar to the above definitions , we generate consistent and inconsistent pairs by starting from a consistent sequence and splitting them into two parts . We consider the following variations . • ConsSent-N ( k ) : If n is the number of tokens in a sequence S1 , let S11 be a random subsequence of S1 , and let S21 = S1\S11 be the complementary subsequence . For a consistent sequence S1 , S11 and S 2 1 form a consistent pairs of sequences . Let S 1 2 and S 2 2 be a partition for a different consistent sequence S2 , such that S21 6= S22 . Then S11 and S22 form an inconsistent pair of sequences , by virtue of the fact they belong to two different consistent sequences . We can vary the complexity of the encoder E by training it to discriminate between a consistent pair ( S11 , S 2 1 ) and k−1 other inconsistent pairs ( S11 , S22 ) , ( S11 , S23 ) , · · · , ( S11 , S2k ) for different values of k. It is possible to pose the task of discriminating the consistent pair ( S11 , S 2 1 ) from the k − 1 inconsistent pairs as a classification problem with a classification layer applied to encodings of the pairs . But this introduces additional parameters which is avoidable for sentence pairs . Instead , we train E by enforcing the constraint that E ( S11 ) ·E ( S21 ) ≥ E ( S11 ) ·E ( S2j ) ∀j ∈ { 2 , k } In other words , we train the encoder to place the representations of consistent pairs of sequences closer in terms of dot product than inconsistent pairs . A similar procedure was also used in training sentence representations in ( Logeswaran & Lee ( 2018 ) ) , but with whole sentences appearing adjacent to each other in a larger body of text . Our method does not require the individual sentences to be ordered for training . • ConsSent-C ( k ) : We generate S11 and S21 from S1 by partitioning it at a random point . Thus both S11 and S 2 1 are contiguous subsequences of S1 . If S1 is consistent , then the two partitioned sequences form a consistent pair . We generate inconsistent pairs by pairing S11 with k − 1 other S2j originating from the partition of different consistent sequences Sj . In addition to training an encoder on each individual task , we also consider training encoders in a multitask setting . We split the tasks into two groups , the first containing ConsSent-D ( k ) , ConsSentP ( k ) , ConsSent-I ( k ) and ConsSent-R ( k ) and the other containing ConsSent-N ( k ) and ConsSentC ( k ) . We then train an encoder E1 for the first group and another encoder E2 for the second group , cycling through the tasks in a round robin manner within each group . Note that for the first group , we use different classification layers for the four different tasks . After training , the sentence representations from E1 and E2 is concatenated to generate the final representation for any sentence . We denote this model with multitask training by ConsSent-MT ( k ) . In Table1 , we show toy positive and negative training examples for each of these training tasks . The choice of the encoder E is important to generate good sentence representations . Following Conneau et al . ( 2017 ) , we use a bidirectional LSTM to process a sequence of tokens and take a max-pool of the intermediate hidden states to compute a distributed representation .

Summary: This submission presents a model for self-supervised learning of sentence representations. The core idea is to train a sentence encoder to predict sequence consistency. Sentences from a text corpus are considered consistent (positive examples), while simple editions of these make the negative samples. Six different ways to edit the sequence are proposed. The network is trained to solve this binary classification task, separately for all six possible editions.