Sentence embedding with contrastive multi-views learning

1 INTRODUCTION . We propose to learn sentence embeddings by contrasting multiple linguistic representations . The motivation is to benefit from linguistic structures diversity to discard noises inherent to each representation . We aim at encoding high-level representations by aligning the underlying shared information from multiple views . As illustrated in Figure 1 , we train our model with a contrastive framework which aims at mapping close input sentences to close representations while separating unrelated sentences . In Natural Language Processing ( NLP ) , this framework has been widely used to learn word representations ( Mikolov et al. , 2013a ; b ) for example . This model relies on the distributional hypothesis which conjectures that words within similar context share similar meaning . Such framework has also been extended to sentences with the similar hypothesis that the meaning can be inferred from the context sentences ( Logeswaran & Lee , 2018 ; Kiros et al. , 2015 ) . We propose to extend this framework by assuming that different views of the same sentence should lead to close representation . We considered the dependency trees , a linguistic framework that describes the compositional structure of a sentence . As illustrated in Figure 1 , in this framework , the sentence is mathematically described as an oriented acyclic graph where the nodes are words and edges describe the relations between words . Such structure has benefited from an important attention in the NLP community and efficient parser tools for various languages are available , which makes it possible to obtain such information almost freely in the sense it does not require additional hand annotated data . Tree representations are then mapped in a shared embedding space using appropriate Tree LSTM networks introduced in Tai et al . ( 2015 ) . Model parameters are learned using a discriminating objective as proposed in Logeswaran & Lee ( 2018 ) . 2 RELATED WORK . 2.1 REPRESENTATION LEARNING . Representation learning has gained significant attention from the NLP community ( Bengio et al. , 2013 ) . As illustrated in Figure 3 it is structured in two-steps : ( A ) a representation is learned using a proxy objective on a usually very large corpora ( B ) the representation is then used to solve a variety of downstream tasks . Literature comes with a variety of proxy tasks to learn representations . Proxy objectives usually fall in two categories : ( i ) predicting a contextual information or reconstructing an altered input or ( ii ) discriminating multiple representations of the same data among negative examples . Proxy ( i ) has been declined into multiple variations in the literature . Words embeddings methods Mikolov et al . ( 2013b ) proposes that words with close distributions should lead to close representations . For images , Noroozi & Favaro ( 2016 ) proposes to learn representations by solving jigsaw puzzles . With a close taste to NLP tasks , Pathak et al . ( 2016 ) aims at learning representations by trying to fill the missing part of an image . van den Oord et al . ( 2018 ) ; Walker et al . ( 2016 ) methods aim at predicting the future to represent video or audio . 2.2 CONTRASTIVE LEARNING . The second proxy objective ( ii ) is motivated by the observation that end-to-end frameworks tend to learn semantic similarities among classes . For example Wu et al . ( 2018 ) reports that the hidden layers of a supervised neural networks assigns close representations to leopard and jaguar images . However an image from lifeboat and leopard will be apart from each other . This motivates for contrastive learning methods where the network aims at distinguishing individual samples to insure a global consistency in the dataset . Many contrastive learnings have been proved effective in a variety of domains : image processing ( Tian et al. , 2019 ; Wu et al. , 2018 ) , audio input ( van den Oord et al. , 2018 ) , sentence representation ( Logeswaran & Lee , 2018 ) or word embedding ( Mikolov et al. , 2013a ; b ; Mnih & Kavukcuoglu , 2013 ) . Besides the nature of data , formalisms might differ on two main aspects : ( i ) How to measure the proximity in both the original and the representation space and ( ii ) What training objective should be used to achieve well distinction between samples . One possible answer on how to select samples that should lead to close representation is to consider different views of the same sample as proposed in Tian et al . ( 2019 ) ; Li et al . ( 2018 ) who propose to combine multi-views of images with contrastive learning . In NLP , Logeswaran & Lee ( 2018 ) ; van den Oord et al . ( 2018 ) propose contrastive frameworks for sentences but without multi-views setting . Multiple metrics to measure the proximity in the representation space are proposed in the literature : Tschannen et al . ( 2019 ) enumerates multiple so called critic functions . The most straight forward also used in Logeswaran & Lee ( 2018 ) beeing the scalar product : hθ ( u , v ) = uT v. But other critics are considered such as the separable critic hθ ( u , v ) = φ1 ( u ) Tφ2 ( v ) . The objective function for contrastive learning has also been declined in multiple work : Many relate to Mutual Information ( MI ) ( Hjelm et al. , 2019 ; Tschannen et al. , 2019 ; Bachman et al. , 2019 ) or methods based on triplet loss or max-margin ( Chopra et al. , 2005 ; Weinberger et al. , 2005 ) . 3 METHOD . 3.1 CONTRASTIVE LEARNING . Contrastive learning is a self-supervised learning method which aims at learning a semantic mapping . Data is embedded with compact representations such that close samples are mapped to nearby points while unrelated ones are affected to apart points . In practice different methods exist to learn such mapping . A straight forward method is to treat each data sample as a distinct class and train a classifier to distinguish between individual instance classes . Such approach is used for word embedding where the goal is to predict a word given his context . However this is computationally difficult and the outputs are not a finite set in our case since an infinite number of correct sentences might be expressed . A method to approximate a full instance discrimination is to use Negative sampling and Noise Contrastive estimation methods . For every data point x from a dataset D we have access to one paired sample x+ which should be close . We then draw K negative samples x−1 , x − 2 , · · · , x − K which should be apart . We learn a scoring function hθ which assigns large scores to positive paired samples ( x , x+ ) while low values for negative pairs ( x , x−k ) . In practice the function is trained using a classifier which aims at identifying the correct pair hθ ( x + , x− ) among negatives pairs { hθ ( x+ , x−k ) } ( k=1 , K ) . In our setup we use a softmax classifier as follow : p ( ·|x , { x−1 , x − 2 , · · · , x − K } ) = exp ( hθ ( x , · ) ) hθ ( x , · ) + ∑K k=1 hθ ( x − k , · ) ( 1 ) The classifier is trained using the negative log likelihood loss : L = ∑ x∈D logp ( x+|x , { x−1 , x − 2 , · · · ∈ , x − K } ) ( 2 ) As in Logeswaran & Lee ( 2018 ) the used scoring function is the inner product hθ ( u , v ) = uT v in order to avoid the situation where the model learns poor representations but compensates with an excellent classifier on the proxy task . However some authors report excellent results with other critic functions : for example Tai et al . ( 2015 ) uses a combination of absolute distance and angle measures . Given the scoring vectors for each pair representation we use a simple softmax classifier with a log negative likelihood loss . 3.2 MULTI-VIEWS LEARNING . In our setup , the positive pair ( x , x+ ) is built from different views of the input data . From the original sentences , we construct multiple representations from different views of the object . For every sentence we consider two encoders . xs which is the representation obtained with a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) who assumes the underlying structure of the sentence to be a sequence , while allowing for long term dependencies . The final representation of the sentence is the concatenation of both direction last state . xd is obtained with the dependency tree representation combined with a ChildSum Tree LSTM ( Tai et al. , 2015 ) . The final representation is the state from the root node of the sentence graph . Negative examples are obtained using the dependency Tree LSTM . The dataset D is therefore augmented with the positive examples and multi-views examples leading to D = { xs , xd , x−d1 , x − d2 , · · · , x − dK } . Similarly to Logeswaran & Lee ( 2018 ) , the model is trained to identify the sentence appearing in the context from the target sentence . The key difference and our contribution is to build different views of the data . The target sequence is encoded using the sequential Tree LSTM , while the positive and negative samples are encoded using the ChildSum Tree LSTM . As described in the previous section , the objective is to maximize the log probability of the correct sentence to appear in the context . 4 EXPERIMENTAL SETUP . 4.1 DATA . Models are trained on the BookCorpus dataset Zhu et al . ( 2015 ) . Since the dataset is no longer distributed , a similar dataset is generated using smashword open book data 1 . The obtained dataset contains 78M sentences from 17.000 books . We define a 20k words vocabulary , containing the corpus most frequent tokens . Sentences are tokenized using Microsoft BlingFire tool 2 . Models are trained on a single epoch on the entire corpus without any train-test split . The use of Tree LSTM models supposes to parse sentences in dependency which is operated using the publicly available Spacy parser . The contrastive learning method supposes to draw negative examples for each sentence pair . We follow the training procedure proposed in Logeswaran & Lee ( 2018 ) and build mini-batches respecting books sentences order . Therefore samples from the mini-batch are used as negative examples . 4.2 TRAINING PARAMETERS . Models are trained using the Adam optimizer Kingma & Ba ( 2015 ) with a 5e-3 learning rate . Gradient is clipped with a value of 5.0 . The hidden layer for both Tree LSTM is fixed to 1000 and the embedding dimension to 300 . The batch size is 100 and the model is trained on a 1080Ti Nvidia GPU . All model weights are initialized with a xavier distribution Glorot & Bengio ( 2010 ) and biases to 0 . We observe biases initialization to have a significant impact on the training . Indeed leaf nodes are initialized with zero vectors for state and hidden and initial biases with too significant values tend to slow the training convergence . The Tree LSTM network was implemented using a batch procedure described in Section A . However the training was significantly slower than vanilla LSTM . The training phase was stopped after 33 1Corpus dataset was generated using https : //github.com/soskek/bookcorpus 2https : //github.com/Microsoft/BlingFire hours of training . The training phase was completed on only 4.6M sentences among the 78M available . We monitored the performance on downstream tasks during training as illustrated in Figure 2 .

Summary: This paper describes a self-supervised sentence embedding approach that incorporates a different view from plain text where some extent of linguistic knowledge is incorporated through the application of tree LSTM. The training procedure is standard contrastive framework where the model is encouraged to distinguish between context sentence (sentences appearing close to the target sentence) and negative samples. Evaluations are conducted on 1) downstream tasks, but with a simple logistic regression model on top of sentence embeddings; 2) probing tasks that more focus on surface information prediction, syntactic and semantic tasks; and 3) qualitative analysis with nearest 5 sentences.