Contrasting distinct structured views to learn sentence embeddings

1 INTRODUCTION . We propose a self-supervised method that builds sentence embeddings from the combination of diverse explicit syntactic structures . Such a method aims at improving the ability of models to perform compositional knowledge . In particular , we evaluate the embedding potential to solve downstream tasks . Building generic sentence embeddings remains an open question . Many training methods have been explored : generating past and previous sentences ( Kiros et al. , 2015 ; Hill et al. , 2016 ) , discriminating context sentences ( Logeswaran & Lee , 2018 ) , predicting specific relations between pairs of sentences ( Conneau et al. , 2017 ; Nie et al. , 2019 ) . While all these methods propose efficient training objectives , they all rely on a similar RNN as encoder architecture . Nonetheless , model architectures have been subject to extensive work as well ( Tai et al. , 2015 ; Zhao et al. , 2015 ; Arora et al. , 2017 ; Lin et al. , 2017 ) , and in supervised frameworks , many encoder structures outperform standard RNN networks . We hypothesize structure is a crucial element to perform compositional knowledge . In particular , the heterogeneity of performances given models and tasks makes us assume that some structures may be better adapted for a given example or task . Therefore , combining diverse structures should be more robust for tasks requiring complex word composition to derive their meaning . Hence , we aim here to evaluate the potential benefit from interactions between pairs of encoders . In particular , we propose a training method for which distinct encoders are learned jointly . We conjecture this association might improve our embeddings ’ power of generalization and propose an experimental setup to corroborate our hypothesis . We take inspiration from multi-view learning , which is successfully applied in a variety of domains . In such a framework , the model learns representations by aligning separate observations of the same object . Traditionally , views are issued from a complementary natural perception of the data . For example , a picture and a sound recording of a dog . However , it can be extended to any pair of samples that share similar semantic content , such as the translation of the same sentence in two different languages . The definition can be extended to synthetic views , which are derived from the same unimodal data . In our case , we derived multiple views from a single sentence by pairing it with a distinct syntactic framework . We illustrated in Figure 2 , two views derived from the same input sentence by applying respectively a constituent or dependency parser . As proposed in image processing ( Tian et al. , 2019 ; Bachman et al. , 2019 ) , we propose to align the different views using a contrastive learning framework . Indeed , contrastive learning is broadly used in NLP Mikolov et al . ( 2013b ; a ) ; Logeswaran & Lee ( 2018 ) . We proposed to enhance the sentence embedding framework proposed in Logeswaran & Lee ( 2018 ) with a multi-view paradigm . As detailed in Section 2 , composing multiple views has demonstrated its effectiveness in many NLP applications . However , as far as we are aware , combining distinct structured models to build standalone embeddings has not yet be explored . Nevertheless , this paradigm benefits from several structural advantages : as already mentioned , it pairs nicely with contrastive learning . It might thus be trained in a self-supervised manner that does not require data annotation . Moreover , contrary to models presented in Section 2 , our method is not specific to a certain kind of encoder architecture . It does not require , for example , the use of attention layers or tree-structured models . More generally , it could be extended to any notion of view , even in other domains than language processing . Our setup could therefore be extended with any encoding function . Finally , our training method induces an interaction between models during inference and , paramountly , during the training phase . Our paper is organized as follows : we detail our contrastive multi-view framework in Section 3 . In Section 4 , we propose an evaluation of our framework on standard evaluation benchmarks and propose qualitative analysis from our embeddings . 2 RELATED WORK . Multi-view is effectively used in a broad variety of domains . For image processing , some methods aim to learn representations by filling the missing part of an image or solving jigsaw puzzles . For video , Tian et al . ( 2019 ) propose to build image tuples using video frames and flow . For audio , van den Oord et al . ( 2018 ) maximize the mutual information between the embedding of the signal at different time steps . Regarding NLP , combining different structural views has already been proven to be successful . Kong & Zhou ( 2011 ) provide a heuristic to combine dependency and constituency analysis for coreference resolution . Zhou et al . ( 2016 ) ; Ahmed et al . ( 2019 ) combine Tree LSTM and standard sequential LSTM with a cross-attention method and observe improvement on a semantic textual similarity task . Chen et al . ( 2017a ) combine CNN and Tree LSTM using attention methods on a sentiment classification task , and CNN outperforms both Tree-LSTM and CNN separately . Finally , Chen et al . ( 2017b ) combine sequential LSTM and Tree LSTM for natural language inference tasks . However , to our knowledge , combining distinct structured models in a contrastive learning setup was not attempted to build sentence embeddings . 3 METHOD . Given a sentence s , the model aims at discriminating the sentences s+ in the neighborhood of s from sentences s− outside of this neighborhood . This is contrastive learning ( Section 3.1 ) . The representation of each sentence is acquired by using multiple views ( Section 3.2 ) . 3.1 CONTRASTIVE LEARNING . Contrastive learning is successfully applied in a variety of domains including audio van den Oord et al . ( 2018 ) , image ( Wu et al. , 2018 ; Tian et al. , 2019 ) , video or natural language processing for word embedding ( Mikolov et al. , 2013b ) or sentence embedding ( Logeswaran & Lee , 2018 ) . Some mathematical foundations are detailed in Saunshi et al . ( 2019 ) . The idea is to build a dataset such that each sample x is combined with another sample x+ , which is somehow close . For word or sentence embeddings , the close samples are the words or the sentences appearing in the given textual context . For image processing , close samples might be two different parts of the same image . Systems are trained to bring close sentences together while dispersing negative examples . In particular , a sentence embedding framework is proposed by Logeswaran & Lee ( 2018 ) . The method takes inspiration from the distributional hypothesis successfully applied for word , but this time , to identify context sentences . The network is trained using a contrastive method . Given a sentence s , a corresponding context sentence s+ and a set of K negative samples s−1 · · · s − K , the training objective is to maximize the probability of discriminate the correct sentence among negative samples : p ( s+|s , s−1 · · · s − K ) . The algorithm architecture used to estimate p is close to word2vec ( Mikolov et al. , 2013b ; a ) . Two sentences encoders f and g are defined and the conditional probability is estimated as follow : p ( s+|s , s−1 · · · s − K ) = ef ( s ) T g ( s+ ) ef ( s ) T g ( s+ ) + ∑N i=1 e f ( s ) T g ( s−i ) At inference time , the sentence representation is obtained as the concatenation of the two encoders f and g such as s → [ f ( s ) ; g ( s ) ] . In Logeswaran & Lee ( 2018 ) , f and g are chosen identical and consist in two RNN . However , the authors observe that the encoders might learn redundant features . To limit this effect , they propose to use a distinct set of embeddings for each encoder . We propose addressing this aspect by enhancing the method with a multi-view framework and using a distinct structured model for the encodes f and g. We hypothesize that some structures may be better adapted for a given example or task . For example , Figure 2 illustrates that dependency parsing uses the verb ” filled ” as the root node . Whereas in constituency parsing , subject and verb are respectively , the right and left child from the root node . Therefore , the combination of different structures should be more robust for tasks requiring complex word composition and be less sensitive to lexical variations . Consequently , we propose a training procedure that allows the model to benefit from the interaction of various syntactic structures . The choice for the encoder architecture is detailed in the following section . 3.2 LANGUAGE VIEWS . Multi-view aims as learning representations from data represented by multiple independent sets of features . The method is not specific for any particular nature of data and can be applied to a broad scale of domains , making it an efficient framework in self-supervised representation learning . As depicted in Section 1 , we generalize the notion of view for a sentence as the application of a specific syntactic framework . For each view , we use an ad-hoc algorithm that maps the structured representation of the sentence into an embedding space . For tree-based views , we consider both phrase structure trees and dependency trees . The phrase structure of a sentence is represented as nested multi-word constituents . The dependency tree represents the relationship between individual words . Although equivalences might be derived between the two representations schemes , we hypothesize that , in our context , the corresponding sequence of operations might allow capturing rather distinct linguistic properties . The various models may , therefore , be complementary and their combination allows for more fine-grained analysis . Together with the trees , we consider the following views of a sentence . Bag Of Word ( BOW ) This setup does not assume any underlying structure . The sentence is modeled as an unordered set of words . The associated encoding method is a simple commutative sum operation of the word embeddings . In our case , vectors are initialized using GloVe vectors ( Pennington et al. , 2014 ) publicly available1 . We used 300-dimensional word vectors trained on the common crawl dataset ( 840B tokens ) with a vocabulary of 2.2M case sensitive words . Vanilla LSTM ( SEQ ) assumes a sequential structure where each word depends on the previous words in the sentence . The framework is a bidirectional sequential LSTM ( Hochreiter & Schmidhuber , 1997 ) . The concatenation of the forward and backward last hidden state of the model is used as sequence embedding . Dependency tree ( DEP ) In the dependency tree model , words are connected through dependency edges . A word might have an arbitrary number of dependants . As illustrated in Figure 2 , the sentence can be represented as a tree where nodes correspond to words and edges indicate whether or not the words are connected in the dependency tree . In our case , the dependency tree is obtained using the deep biaffine parser from Dozat & Manning ( 2017 ) 2 For this view , we compute sentence embeddings with the Child-Sum Tree LSTM model described in Tai et al . ( 2015 ) : Each node is assigned an embedding given its dependent with a recursive function . The recursive node function is derived from standard LSTM formulations but adapted for tree inputs . In particular , the hidden state is computed as the sum of all children hidden states : h̃j = ∑ k∈C ( j ) hk ( 1 ) with C ( j ) , the set of children of node j . All equations are detailed in Tai et al . ( 2015 ) . However , we slightly modify the computation of h̃j using Equation 2 . As in Zhou et al . ( 2016 ) , we propose to 1https : //nlp.stanford.edu/projects/glove/ 2We use an open-source implementation of the parser and replace the pos-tags features with features obtained with BERT . Therefore we do not need pos-tags annotations to parse our corpus : https : //github . com/yzhangcs/biaffine-parser compute h̃j as the weighted sum of children vectors in order to allow the model to filter semantically less relevant children . h̃j = ∑ k∈C ( j ) αkjhk ( 2 ) The parameters αkj are attention weights computed using a soft attention layer . Given a node j , we consider h1 , h2 , . . . , hn the corresponding children hidden states . the soft attention layer produces a weight αk for each child ’ s hidden state . We did not use any external query to compute the attention but instead use a projection from the current node embedding . The attention mechanism is detailed in equations below : qj =W ( q ) xj + b ( q ) ( 3 ) pk =W ( p ) hk + b ( p ) ( 4 ) akj = qj · pᵀk ‖qj‖2 · ‖pk‖2 ( 5 ) αkj = softmaxk ( a1j · · · anj ) ( 6 ) The embedding at the root of the tree is used as the sentence embedding as the Tree LSTM model computes representations bottom up . Constituency tree ( CONST ) Constituent analysis describes the sentence as a nested multi-word structure . In this framework , words are grouped recursively in constituents . In the resulting tree , only leaf nodes correspond to words , while internal nodes encode recursively word sequences . The structure is obtained using the constituency neural parser from Kitaev & Klein ( 2018 ) . The framework is associated with the N-Ary Tree LSTM , which is defined in Tai et al . ( 2015 ) . Similarly to the original article , we binarize the trees to ensure that every node has exactly two dependents . The binarization is performed using a left markovization and unary productions are collapsed in a single node . Again the representation is computed bottom-up and the embedding of the tree root node is used as sentence embedding . The equations detailed in Tai et al . ( 2015 ) make the distinction between right and left nodes . Therefore we do not propose to enhance the original architecture with a weighted sum as on the DEP view .

Summary: The authors introduce a pretrianing paradigm based on contrastive learning between multiple syntactic views of the same sentence. The method maximizes representations between different setence encoders when given the same sentence, and minimize the similarity to all other sentence repre sentations. The results on the infersent benchmark show competitive performance of the approach when compared to non-syntactic pretraining methods.