Sentence embedding with contrastive multi-views learning

1 INTRODUCTION . We propose to learn sentence embeddings by contrasting multiple linguistic representations . The motivation is to benefit from linguistic structures diversity to discard noises inherent to each representation . We aim at encoding high-level representations by aligning the underlying shared information from multiple views . As illustrated in Figure 1 , we train our model with a contrastive framework which aims at mapping close input sentences to close representations while separating unrelated sentences . In Natural Language Processing ( NLP ) , this framework has been widely used to learn word representations ( Mikolov et al. , 2013a ; b ) for example . This model relies on the distributional hypothesis which conjectures that words within similar context share similar meaning . Such framework has also been extended to sentences with the similar hypothesis that the meaning can be inferred from the context sentences ( Logeswaran & Lee , 2018 ; Kiros et al. , 2015 ) . We propose to extend this framework by assuming that different views of the same sentence should lead to close representation . We considered the dependency trees , a linguistic framework that describes the compositional structure of a sentence . As illustrated in Figure 1 , in this framework , the sentence is mathematically described as an oriented acyclic graph where the nodes are words and edges describe the relations between words . Such structure has benefited from an important attention in the NLP community and efficient parser tools for various languages are available , which makes it possible to obtain such information almost freely in the sense it does not require additional hand annotated data . Tree representations are then mapped in a shared embedding space using appropriate Tree LSTM networks introduced in Tai et al . ( 2015 ) . Model parameters are learned using a discriminating objective as proposed in Logeswaran & Lee ( 2018 ) . 2 RELATED WORK . 2.1 REPRESENTATION LEARNING . Representation learning has gained significant attention from the NLP community ( Bengio et al. , 2013 ) . As illustrated in Figure 3 it is structured in two-steps : ( A ) a representation is learned using a proxy objective on a usually very large corpora ( B ) the representation is then used to solve a variety of downstream tasks . Literature comes with a variety of proxy tasks to learn representations . Proxy objectives usually fall in two categories : ( i ) predicting a contextual information or reconstructing an altered input or ( ii ) discriminating multiple representations of the same data among negative examples . Proxy ( i ) has been declined into multiple variations in the literature . Words embeddings methods Mikolov et al . ( 2013b ) proposes that words with close distributions should lead to close representations . For images , Noroozi & Favaro ( 2016 ) proposes to learn representations by solving jigsaw puzzles . With a close taste to NLP tasks , Pathak et al . ( 2016 ) aims at learning representations by trying to fill the missing part of an image . van den Oord et al . ( 2018 ) ; Walker et al . ( 2016 ) methods aim at predicting the future to represent video or audio . 2.2 CONTRASTIVE LEARNING . The second proxy objective ( ii ) is motivated by the observation that end-to-end frameworks tend to learn semantic similarities among classes . For example Wu et al . ( 2018 ) reports that the hidden layers of a supervised neural networks assigns close representations to leopard and jaguar images . However an image from lifeboat and leopard will be apart from each other . This motivates for contrastive learning methods where the network aims at distinguishing individual samples to insure a global consistency in the dataset . Many contrastive learnings have been proved effective in a variety of domains : image processing ( Tian et al. , 2019 ; Wu et al. , 2018 ) , audio input ( van den Oord et al. , 2018 ) , sentence representation ( Logeswaran & Lee , 2018 ) or word embedding ( Mikolov et al. , 2013a ; b ; Mnih & Kavukcuoglu , 2013 ) . Besides the nature of data , formalisms might differ on two main aspects : ( i ) How to measure the proximity in both the original and the representation space and ( ii ) What training objective should be used to achieve well distinction between samples . One possible answer on how to select samples that should lead to close representation is to consider different views of the same sample as proposed in Tian et al . ( 2019 ) ; Li et al . ( 2018 ) who propose to combine multi-views of images with contrastive learning . In NLP , Logeswaran & Lee ( 2018 ) ; van den Oord et al . ( 2018 ) propose contrastive frameworks for sentences but without multi-views setting . Multiple metrics to measure the proximity in the representation space are proposed in the literature : Tschannen et al . ( 2019 ) enumerates multiple so called critic functions . The most straight forward also used in Logeswaran & Lee ( 2018 ) beeing the scalar product : hθ ( u , v ) = uT v. But other critics are considered such as the separable critic hθ ( u , v ) = φ1 ( u ) Tφ2 ( v ) . The objective function for contrastive learning has also been declined in multiple work : Many relate to Mutual Information ( MI ) ( Hjelm et al. , 2019 ; Tschannen et al. , 2019 ; Bachman et al. , 2019 ) or methods based on triplet loss or max-margin ( Chopra et al. , 2005 ; Weinberger et al. , 2005 ) . 3 METHOD . 3.1 CONTRASTIVE LEARNING . Contrastive learning is a self-supervised learning method which aims at learning a semantic mapping . Data is embedded with compact representations such that close samples are mapped to nearby points while unrelated ones are affected to apart points . In practice different methods exist to learn such mapping . A straight forward method is to treat each data sample as a distinct class and train a classifier to distinguish between individual instance classes . Such approach is used for word embedding where the goal is to predict a word given his context . However this is computationally difficult and the outputs are not a finite set in our case since an infinite number of correct sentences might be expressed . A method to approximate a full instance discrimination is to use Negative sampling and Noise Contrastive estimation methods . For every data point x from a dataset D we have access to one paired sample x+ which should be close . We then draw K negative samples x−1 , x − 2 , · · · , x − K which should be apart . We learn a scoring function hθ which assigns large scores to positive paired samples ( x , x+ ) while low values for negative pairs ( x , x−k ) . In practice the function is trained using a classifier which aims at identifying the correct pair hθ ( x + , x− ) among negatives pairs { hθ ( x+ , x−k ) } ( k=1 , K ) . In our setup we use a softmax classifier as follow : p ( ·|x , { x−1 , x − 2 , · · · , x − K } ) = exp ( hθ ( x , · ) ) hθ ( x , · ) + ∑K k=1 hθ ( x − k , · ) ( 1 ) The classifier is trained using the negative log likelihood loss : L = ∑ x∈D logp ( x+|x , { x−1 , x − 2 , · · · ∈ , x − K } ) ( 2 ) As in Logeswaran & Lee ( 2018 ) the used scoring function is the inner product hθ ( u , v ) = uT v in order to avoid the situation where the model learns poor representations but compensates with an excellent classifier on the proxy task . However some authors report excellent results with other critic functions : for example Tai et al . ( 2015 ) uses a combination of absolute distance and angle measures . Given the scoring vectors for each pair representation we use a simple softmax classifier with a log negative likelihood loss . 3.2 MULTI-VIEWS LEARNING . In our setup , the positive pair ( x , x+ ) is built from different views of the input data . From the original sentences , we construct multiple representations from different views of the object . For every sentence we consider two encoders . xs which is the representation obtained with a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) who assumes the underlying structure of the sentence to be a sequence , while allowing for long term dependencies . The final representation of the sentence is the concatenation of both direction last state . xd is obtained with the dependency tree representation combined with a ChildSum Tree LSTM ( Tai et al. , 2015 ) . The final representation is the state from the root node of the sentence graph . Negative examples are obtained using the dependency Tree LSTM . The dataset D is therefore augmented with the positive examples and multi-views examples leading to D = { xs , xd , x−d1 , x − d2 , · · · , x − dK } . Similarly to Logeswaran & Lee ( 2018 ) , the model is trained to identify the sentence appearing in the context from the target sentence . The key difference and our contribution is to build different views of the data . The target sequence is encoded using the sequential Tree LSTM , while the positive and negative samples are encoded using the ChildSum Tree LSTM . As described in the previous section , the objective is to maximize the log probability of the correct sentence to appear in the context . 4 EXPERIMENTAL SETUP . 4.1 DATA . Models are trained on the BookCorpus dataset Zhu et al . ( 2015 ) . Since the dataset is no longer distributed , a similar dataset is generated using smashword open book data 1 . The obtained dataset contains 78M sentences from 17.000 books . We define a 20k words vocabulary , containing the corpus most frequent tokens . Sentences are tokenized using Microsoft BlingFire tool 2 . Models are trained on a single epoch on the entire corpus without any train-test split . The use of Tree LSTM models supposes to parse sentences in dependency which is operated using the publicly available Spacy parser . The contrastive learning method supposes to draw negative examples for each sentence pair . We follow the training procedure proposed in Logeswaran & Lee ( 2018 ) and build mini-batches respecting books sentences order . Therefore samples from the mini-batch are used as negative examples . 4.2 TRAINING PARAMETERS . Models are trained using the Adam optimizer Kingma & Ba ( 2015 ) with a 5e-3 learning rate . Gradient is clipped with a value of 5.0 . The hidden layer for both Tree LSTM is fixed to 1000 and the embedding dimension to 300 . The batch size is 100 and the model is trained on a 1080Ti Nvidia GPU . All model weights are initialized with a xavier distribution Glorot & Bengio ( 2010 ) and biases to 0 . We observe biases initialization to have a significant impact on the training . Indeed leaf nodes are initialized with zero vectors for state and hidden and initial biases with too significant values tend to slow the training convergence . The Tree LSTM network was implemented using a batch procedure described in Section A . However the training was significantly slower than vanilla LSTM . The training phase was stopped after 33 1Corpus dataset was generated using https : //github.com/soskek/bookcorpus 2https : //github.com/Microsoft/BlingFire hours of training . The training phase was completed on only 4.6M sentences among the 78M available . We monitored the performance on downstream tasks during training as illustrated in Figure 2 .

Summary: The paper proposes a new sentence embedding method. The novelty is to use dependency trees as examples in the self-supervised method based on contrastive learning. The idea to use linguistic knowledge in the design of sentence embeddings is attractive. The sentence representation is computed by a bi-LSTM and dependency tree representations are computed by Tree LSTM. The softmax classifier is trained using the negative log-likelihood loss.