Sentence Encoding with Tree-Constrained Relation Networks

1 INTRODUCTION . Sentence representations play a crucial role in many NLP tasks , including sentiment analysis , paraphrase detection , entailment recognition , summarisation , and machine translation . Popular approaches for encoding sentences compose words using recurrent units ( Cho et al. , 2014 ; Sutskever et al. , 2014 ) or convolutions ( Kalchbrenner et al. , 2014 ; Kim , 2014 ) . A series of papers has also explored improving sentence representations by incorporating structural information into the encoders , including both supervised syntactic information ( Tai et al. , 2015 ; Bowman et al. , 2016 ; Ma et al. , 2015 ; Socher et al. , 2013 ) , and unsupervised syntax where the structure is learned to maximise task performance ( Yogatama et al. , 2017 ; Choi et al. , 2017 ) . We approach the problem of sentence representation starting from the insight that the meaning of a sentence is an aggregation of the set of relations that hold between words . For example , in the simple sentence “ John saw Mary ” , most of the semantics is captured by knowing that “ John is doing seeing ” and that “ Mary is what is being seen ” . Treating sentences as a conjunction of simple relational predicates has been influential in both formal semantics ( Copestake et al. , 2005 ) and roll-filler computational models of semantics ( Smolensky , 1990 ) . While the prior sentence representation architectures discussed above can implicitly discover and represent such relational facts , the recently proposed relation networks model ( Santoro et al. , 2017 , RNs ) lets us work with them more directly . RNs represent a set of objects by aggregating representations of pairwise relations of all pairs of objects . For sentence representation , the “ set ” we seek to represent is a sentence , and the objects are words . There are two problems with this setup , and we develop extensions of RNs to address these shortcomings ( §3 ) . First , RNs assume equal contributions from all pairs of objects . In unstructured sets , this uniformity is justified , but sentences are structured sequences , not unstructured sets , and intuitively the relationship between some word pairs is more meaningful than the relationship between others . To address this structural deficiency , we extend RNs to be sensitive to pairwise attachment information based on a dependency tree structure . Our formulation lets us either use dependency trees obtained from external parsers , or learn trees as latent variables so as to optimise task performance . Tree learning is tractable in this model because we exploit the fact that RNs operate on representations of pairs of words , and we can efficiently obtain marginal probabilities of attachment decisions ( i.e. , word pairs ) in pair-factored parsing models using Kirchoff ’ s matrix tree theorem ( Tutte , 1984 ; Koo et al. , 2007 ; Liu & Lapata , 2017 ) . A second objection to RNs is that they aggregate representations of pairs , but the meaning of a sentence may require understanding higher-order relational information ( i.e. , trees can be deeply nested ) . To enable representations that can directly capture the salient aspects of higher-order relations , we use recurrent RNs ( Palm et al. , 2017 ) , but again augmented with supervised or latent dependency trees . We carefully evaluate and analyse different versions of our RN-based models on machine translation and three text classification tasks , namely textual entailment , question type classification , and duplicate question detection ( §4 ) . Our experimental results demonstrate that in general sentence encoders based on recurrent RNs outperform those based on vanilla RNs , and models with latent tree constraints mostly achieve better performance than their supervised counter-parts . Overall , our models outperform sentence encoders based on birectional LSTMs and those incorporating an intra-sentence attention mechanism . 2 BACKGROUND . Relations networks ( Santoro et al. , 2017 ) are designed to model relational reasoning . Neural networks augmented with RNs have achieved success across various tasks such as visual question answering and text-based question answering on the bAbI suite of tasks , on which even models with sophisticated architecture could not obtain satisfactory results due to the requirement for complex relational inference . An RN is formalised as RN ( O ) = fφ ∑ i , j gθ ( oi , oj ) , ( 1 ) where the input is a set of “ objects ” O = { o1 , o2 , . . . , on } , oi ∈ Rd , and f and g are functions parameterised by φ and θ using feed-forward neural networks ( MLPs ) . The function gθ computes the relatedness of two objects oi and oj . RNs aggregate the relations between all the pairs of objects via the summation operation in Eq . 1 . The embeddings of objects oi and oj differ depending on the task and the nature of the objects to be compared . Although RNs perform well on problems that require basic relational reasoning , they are sub-optimal for tasks that demand multiple steps of reasoning as only pair-wise relational operations are involved . To address this limitation , Palm et al . ( 2017 ) proposed a recurrent variant for RNs , denoted as recurrent RNs , and demonstrated the model ’ s ability for multi-step reasoning , which is essential for the tasks they studied , namely Sudoku and question answering . Recurrent RNs are formulated as follows : mti = ∑ j gθ ( h t−1 j , h t−1 i ) , h t i = fφ ( ht−1i , oi , m t i ) . ( 2 ) At timestep t , hti is the hidden state vector corresponding to the i-th object . In addition to the previous hidden state ht−1i and the input oi , it also takes into account the message m t i , which summarises relations between the i-th object and all its neighbours ( e.g . all objects in the simplest form ) in the previous timestep t − 1 . The number of recurrent steps is a hyperparameter , where higher values allow the modelling of longer chains of relations . In the standard setting , the vector oi encodes timestep invariant information , such as position and ordering , and gθ and fφ are parameterised as MLPs and LSTMs ( Hochreiter & Schmidhuber , 1997 ) , respectively . In this architecture , the recurrent dependencies accumulate information of relations enabling multi-step reasoning . 3 MODELS . A straightforward application of RNs for sentence representation is to treat words as objects and consider all relations between pairs of words in a sentence following Eq . 1 . Intuitively , this approach is unappealing as not all direct links between pairs of words are equally important . Thus , we propose to use a dependency grammar as a constraint in RNs and argue that the relations considered in the model should comply with a valid dependency tree structure . In the vanilla RNs , this means that the aggregations will operate on the subset of relations that form a valid tree ( §3.1 ) and in the recurrent RNs , we want the messages ( Eq . 2 ) only coming from the syntactical neighbours of the node corresponding to a consistent tree structure across different timesteps ( §3.2 ) . In a supervised setting we use an off the shelf parser to generate trees for all sentences . In an unsupervised setting , we model them as latent variables . 3.1 RNS WITH TREE CONSTRAINTS . Let x = [ x1 , x2 , . . . , xn ] denote a sentence , where xi represents the i-th word in the sentence . Let y∗ denote a directed dependency tree , which is represented by a set of head-modifier index tuples ( h , m ) .1 To incorporate syntactic information , we modify Eq . 1 by restricting the summation function to span over ( h , m ) ∈ y∗ : s = fφ ∑ ( h , m ) ∈y∗ gθ ( oh , om ) , ( 3 ) where s denotes the vector representation of the sentence x , and oh and om are embeddings corresponding to the head word xh and the modifier word xm , respectively . To enable task-optimal trees to be learned we introduce a distribution over trees given the sentence , pψ ( y | x ) , and rewrite Eq . 3 in terms of expectations under this distribution : s = fφ ∑ y∈Y ( x ) pψ ( y | x ) ∑ ( h , m ) ∈y gθ ( oh , om ) , where Y ( x ) denotes the set of dependency trees for a sentence x . Because the inner loop depends only on ( h , m ) pairs and the linearity of expectation , we can rewrite this in terms of the marginal edge probabilities pψ ( h → m | x ) = ∑ y∈Y ( x ) pψ ( y | x ) [ ( h , m ) ∈ y ] , where [ q ] = 1 if q is true and 0 otherwise . This becomes s = fφ ∑ ( h , m ) ∈D ( x ) pψ ( h→ m | x ) gθ ( oh , om ) , where D ( x ) denote all possible dependencies arcs for a sentence x : D ( x ) = { ( h , m ) | h ∈ [ 0 . . . n ] , m ∈ [ 1 . . . n ] } . Fortunately , the required attachment marginals can be calculated efficiently in O ( n3 ) using Kirchhoff ’ s matrix-tree theorem ( Tutte , 1984 ) , provided the conditional tree probability is a conditional random field with potentials that only look at a single pair of nodes at a time ( Koo et al. , 2007 ; Smith & Smith , 2007 ; Liu & Lapata , 2017 ) . Although this restriction on the potential functions may seem limiting ( the model can not look at higher order dependency structures when assigning probability to the tree ) , representations derived from neural networks employing such models can produce state-of-the-art supervised dependency parsers ( Dozat & Manning , 2017 ) , suggesting this is an acceptable restriction . For our model , we define the potential associated with an edge using a bilinear form ( Ma & Hovy , 2017 ) , Ψ ( h , m ) = eo T hWom+U T oh+V T om+b , ( 4 ) where oi denotes the BiLSTM output corresponding to the input word xi ; W , U , V are weight matrices , and b is the bias term . Structured intra-sentence attention For many applications , it is useful to represent each word in the sentence in terms of its context in the surrounding sentence . While simple mechanisms like bidirectional RNNs can produce satisfactory results ( Bahdanau et al. , 2015 ) , attention-based word-in-context representations have received increasing interest ( Vaswani et al. , 2017 ) , including 1For the experiments below that use syntax trees , we obtain dependency parses from SyntexNet ( Weiss et al. , 2015 ; Kong et al. , 2017 ) , an open-source parser trained on the Universal Dependencies English dataset ( Nivre et al. , 2016 ) . those that impose structural constraints on attention ( Liu & Lapata , 2017 ; Kim et al. , 2017 ) . Our tree-constrained RN can likewise be used to construct word-in-context representations . Intuitively , we create a contextualised representation for each word by aggregating information from its possible parents and children according to their attachment marginal probability . These context vectors and word vectors can either be operated on by models that require them or pooled over to generate a sentence representation . For RNs with supervised tree constraints , we define the context vector for the word xi in terms of the representation of the relation between xi and its parent xh , i.e. , ri = fφ ( gθ ( oh , oi ) ) . For latent tree RNs , the context vector for each word xi contains two components : a weighted sum of relations between xi and its parents , and a weighted sum of relations between xi and its children : pi = fφ ( n∑ h=1 pψ ( h→ i|x ) gθ ( oh , oi ) ) , ci = fφ ( n∑ m=1 pψ ( i→ m|x ) gθ ( oi , om ) ) , ri = [ pi ; ci ] . Finally , the ri ’ s are concatenated with the original object representations and transformed into the final contextual word representations : si = tanh ( WTr [ ri ; oi ] ) . For applications where a single vector is required , a subsequent pooling operation is applied to the sequence of vectors [ s1 , s2 , . . . , sn ] to generate an alternative representation of a sentence s. We discuss the pooling operations used below ( §4 and §A.1 ) . In our model , the context vectors are computed from the representations of word pairs ( i.e. , components of RNs ) , whereas in the related prior work of Liu & Lapata ( 2017 ) and Kim et al . ( 2017 ) , they are the expected value of LSTM hidden state vectors under structured attention distributions .

Summary: The paper presents an extension of relation networks (RNs) for natural language processing. RNs are designed to represent a set as a function of the representations of their elements. This paper treats a sentence as a set of words. Whereas regular RNs assume that the representation of a set is a uniform aggregation of the representation of the pairs of elements in the set, this paper proposes to weight the relevance of the pairs according to their tree-structured dependency relations between the words. The authors evaluate on a suite of NLP tasks, including SNLI, Quora duplicate question ID, and machine translation. They show marginal improvements over naive baselines, and no improvement over SOTA.