Graph Neural Networks with Generated Parameters for Relation Extraction

1 INTRODUCTION . Recent years , graph neural networks ( GNNs ) have been applied to various fields of machine learning , including node classification ( Kipf & Welling , 2016 ) , relation classification ( Schlichtkrull et al. , 2017 ) , molecular property prediction ( Gilmer et al. , 2017 ) , few-shot learning ( Garcia & Bruna , 2018 ) , and achieve promising results on these tasks . These works have demonstrated GNNs ’ strong power to process relational reasoning on graphs . Relational reasoning aims to abstractly reason about entities/objects and their relations , which is an important part of human intelligence . Besides graphs , relational reasoning is also of great importance in many natural language processing tasks such as question answering , relation extraction , summarization , etc . Consider the example shown in Fig . 1 , existing relation extraction models could easily extract the facts that Luc Besson directed a film Léon : The Professional and that the film is in English , but fail to infer the relationship between Luc Besson and English without multi-hop relational reasoning . By considering the reasoning patterns , one can discover that Luc Besson could speak English following a reasoning logic that Luc Besson directed Léon : The Professional and this film is in English indicates Luc Besson could speak English . However , most existing GNNs can only process multi-hop relational reasoning on pre-defined graphs and can not be directly applied in natural language relational reasoning . Enabling multi-hop relational reasoning in natural languages remains an open problem . To address this issue , in this paper , we propose graph neural networks with generated parameters ( GP-GNNs ) , to adapt graph neural networks to solve the natural language relational reasoning task . GP-GNNs first constructs a fully-connected graph with the entities in the sequence of text . After that , it employs three modules to process relational reasoning : ( 1 ) an encoding module which enables edges to encode rich information from natural languages , ( 2 ) a propagation module which propagates relational information among various nodes , and ( 3 ) a classification module which makes predictions with node representations . As compared to traditional GNNs , GP-GNNs could learn edges ’ parameters from natural languages , extending it from performing inferring on only non-relational graphs or graphs with a limited number of edge types to unstructured inputs such as texts . In the experiments , we apply GP-GNNs to a classic natural language relational reasoning task : relation extraction from text . We carry out experiments on Wikipedia corpus aligned with Wikidata knowledge base ( Vrandečić & Krötzsch , 2014 ) and build a human annotated test set as well as two distantly labeled test sets with different levels of denseness.Experiment results show that our model outperforms other state-of-the-art models on relation extraction task by considering multihop relational reasoning . We also perform a qualitative analysis which shows that our model could discover more relations by reasoning more robustly as compared to baseline models . Our main contributions are in two-fold : ( 1 ) We extend a novel graph neural network model with generated parameters , to enable relational message-passing with rich text information , which could be applied to process relational reasoning on unstructured inputs such as natural languages . ( 2 ) We verify our GP-GNNs in the task of relation extraction from text , which demonstrates its ability on multi-hop relational reasoning as compared to those models which extract relationships separately . Moreover , we also present three datasets , which could help future researchers compare their models in different settings . 2 RELATED WORK . 2.1 GRAPH NEURAL NETWORKS ( GNNS ) . GNNs were first proposed in ( Scarselli et al. , 2009 ) and are trained via the Almeida-Pineda algorithm ( Almeida , 1987 ) . Later the authors in Li et al . ( 2016 ) replace the Almeida-Pineda algorithm with the more generic backpropagation and demonstrate its effectiveness empirically . Gilmer et al . ( 2017 ) propose to apply GNNs to molecular property prediction tasks . Garcia & Bruna ( 2018 ) shows how to use GNNs to learn classifiers on image datasets in a few-shot manner . Gilmer et al . ( 2017 ) study the effectiveness of message-passing in quantum chemistry . Dhingra et al . ( 2017 ) apply message-passing on a graph constructed by coreference links to answer relational questions . There are relatively fewer papers discussing how to adapt GNNs to natural language tasks . For example , Marcheggiani & Titov ( 2017 ) propose to apply GNNs to semantic role labeling and Schlichtkrull et al . ( 2017 ) apply GNNs to knowledge base completion tasks . Zhang et al . ( 2018 ) apply GNNs to relation extraction by encoding dependency trees , and De Cao et al . ( 2018 ) apply GNNs to multi-hop question answering by encoding co-occurence and co-reference relationships . Although they also consider applying GNNs to natural language processing tasks , they still perform message-passing on predefined graphs . Johnson ( 2017 ) introduces a novel neural architecture to generate a graph based on the textual input and dynamically update the relationship during the learning process . In sharp contrast , this paper focuses on extracting relations from real-world relation datasets . 2.2 RELATIONAL REASONING . Relational reasoning has been explored in various fields . For example , Santoro et al . ( 2017 ) propose a simple neural network to reason the relationship of objects in a picture , Xu et al . ( 2017 ) build up a scene graph according to an image , and Kipf et al . ( 2018 ) model the interaction of physical objects . In this paper , we focus on the relational reasoning in natural language domain . Existing works ( Zeng et al. , 2014 ; 2015 ; Lin et al. , 2016 ) have demonstrated that neural networks are capable of capturing the pair-wise relationship between entities in certain situations . For example , ( Zeng et al. , 2014 ) is one of the earliest works that applies a simple CNN to this task , and ( Zeng et al. , 2015 ) further extends it with piece-wise max-pooling . Nguyen & Grishman ( 2015 ) propose a multi-window version of CNN for relation extraction . Lin et al . ( 2016 ) study an attention mechanism for relation extraction tasks . Peng et al . ( 2017 ) predict n-ary relations of entities in different sentences with Graph LSTMs . Le & Titov ( 2018 ) treat relations as latent variables which are capable of inducing the relations without any supervision signals . Zeng et al . ( 2017 ) show that the relation path has an important role in relation extraction . Miwa & Bansal ( 2016 ) show the effectiveness of LSTMs ( Hochreiter & Schmidhuber , 1997 ) in relation extraction . Christopoulou et al . ( 2018 ) proposed a walk-based model to do relation extraction . The most related work is ( Sorokin & Gurevych , 2017 ) , where the proposed model incorporates contextual relations with attention mechanism when predicting the relation of a target entity pair . The drawback of existing approaches is that they could not make full use of the multi-hop inference patterns among multiple entity pairs and their relations within the sentence . 3 GRAPH NEURAL NETWORK WITH GENERATED PARAMETERS ( GP-GNNS ) . We first define the task of natural language relational reasoning . Given a sequence of text with m entities , it aims to reason on both the text and entities and make a prediction of the labels of the entities or entity pairs . In this section , we will introduce the general framework of GP-GNNs . GP-GNNs first build a fullyconnected graph G = ( V , E ) , where V is the set of entities , and each edge ( vi , vj ) ∈ E , vi , vj ∈ V corresponds to a sequence s = xi , j0 , x i , j 1 , . . . , x i , j l−1 extracted from the text . After that , GP-GNNs employ three modules including ( 1 ) encoding module , ( 2 ) propagation module and ( 3 ) classification module to proceed relational reasoning , as shown in Fig . 2 . 3.1 ENCODING MODULE . The encoding module converts sequences into transition matrices corresponding to edges , i.e . the parameters of the propagation module , by A ( n ) i , j = f ( E ( xi , j0 ) , E ( xi , j1 ) , · · · , E ( xi , jl−1 ) ; θne ) , ( 1 ) where f ( · ) could be any model that could encode sequential data , such as LSTMs , GRUs , CNNs , E ( · ) indicates an embedding function , and θne denotes the parameters of the encoding module of n-th layer . 3.2 PROPAGATION MODULE . The propagation module learns representations for nodes layer by layer . The initial embeddings of nodes , i.e . the representations of layer 0 , are task-related , which could be embeddings that encode features of nodes or just one-hot embeddings . Given representations of layer n , the representations of layer n+ 1 are calculated by h ( n+1 ) i = ∑ vj∈N ( vi ) σ ( A ( n ) i , j h ( n ) j ) , ( 2 ) where N ( vi ) denotes the neighbours of node vi in graph G and σ ( · ) denotes non-linear activation function . 3.3 CLASSIFICATION MODULE . Generally , the classification module takes node representations as inputs and outputs predictions . Therefore , the loss of GP-GNNs could be calculated as L = g ( h00 : |V|−1 , h10 : |V|−1 , . . . , hK0 : |V|−1 , Y ; θc ) , ( 3 ) where θc denotes the parameters of the classification module , K is the number of layers in propagation module and Y denotes the ground truth label . The parameters in GP-GNNs are trained by gradient descent methods . 4 RELATION EXTRACTION WITH GP-GNNS . Relation extraction from text is a classic natural language relational reasoning task . Given a sentence s = ( x0 , x1 , . . . , xl−1 ) , a set of relations R and a set of entities in this sentence Vs = { v1 , v2 , . . . , v|Vs| } , where each vi consists of one or a sequence of tokens , relation extraction from text is to identify the pairwise relationship rvi , vj ∈ R between each entity pair ( vi , vj ) . In this section , we will introduce how to apply GP-GNNs to relation extraction . 4.1 ENCODING MODULE . To encode the context of entity pairs ( or edges in the graph ) , we first concatenate the position embeddings with word embeddings in the sentence : E ( xi , jt ) = [ xt ; p i , j t ] , ( 4 ) where xt denotes the word embedding of word xt and p i , j t denotes the position embedding of word position t relative to the entity pair ’ s position i , j ( Details of these two embeddings are introduced in the next two paragraphs . ) After that , we feed the representations of entity pairs into encoder f ( · ) which contains a bi-directional LSTM and a multi-layer perceptron : A ( n ) i , j = [ MLPn ( BiLSTMn ( ( E ( xi , j0 ) , E ( xi , j1 ) , · · · , E ( xi , jl−1 ) ) ] , ( 5 ) where n denotes the index of layer 1 , [ · ] means reshaping a vector as a matrix , BiLSTM encodes a sequence by concatenating tail hidden states of the forward LSTM and head hidden states of the backward LSTM together and MLP denotes a multi-layer perceptron with non-linear activation σ . Word Representations We first map each token xt of sentence { x0 , x1 , . . . , xl−1 } to a kdimensional embedding vector xt using a word embedding matrix We ∈ R|V |×dw , where |V | is the size of the vocabulary . Throughout this paper , we stick to 50-dimensional GloVe embeddings pre-trained on a 6 billion corpus ( Pennington et al. , 2014 ) . Position Embedding In this work , we consider a simple entity marking scheme2 : we mark each token in the sentence as either belonging to the first entity vi , the second entity vj or to neither of those . Each position marker is also mapped to a dp-dimensional vector by a position embedding matrix P ∈ R3×dp . We use notation pi , jt to represent the position embedding for xt corresponding to entity pair ( vi , vj ) . 1Adding index to neural models means their parameters are different among layers . 2As pointed out by Sorokin & Gurevych ( 2017 ) , other position markers lead to no improvement in perfor- mance .

Summary: The paper describes a Graph NN method for information extraction from sentences. The objective is to label couples of entities (token or multiple tokens)  according to a given set of relations. The GNN processes token representations through one or more layers and a final classification layer scores the relations between entity couples in the sentence. Parameters of GNN – transition matrices between layers operating on node representations – are learned from the data.  Experiments are performed on different variants of an extraction corpus, for the task consisting in extracting all the relations between identified token couples in a sentence.