Iterated graph neural network system

1 INTRODUCTION . GNN ( Scarselli et al. , 2009 ) has been proved to be effective in processing graph structured data , and has been widely used in natural language processing , computer vision , data mining , social network and biochemistry . In recent years , GNN has developed a variety of architectures , such as GCN ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) , DGI ( Veličković et al. , 2019 ) , GIN ( Xu et al. , 2019 ) , GCNII ( Ming Chen et al. , 2020 ) and GEN ( Li et al. , 2020 ) . These architectures have a common feature , that is , the representation of each node is updated using messages from its neighbors but without distinguishing the direction ( or angle ) of message passing between two nodes . Recent studies have shown that considering directed message passing between nodes can improve the performance of GNN and achieve success in related fields . For example , DimeNet ( Klicpera et al. , 2020 ) considers the spatial direction from one atom to another and can learn both molecular properties and atomic forces . R-GCN ( Schlichtkrull et al. , 2018 ) and Bi-GCN ( Marcheggiani & Titov , 2017 ; Fu et al. , 2019 ) are models for directed graph , applied in the field of natural language processing . We note that the above direction based model does not consider the bidirectional mixed passing of messages . But in real life , message passing is interactive in different directions . For example , node A obtains a message from node B . After processing the message , node A not only passes it to the next node C , but also feeds back to node B . Suppose there are only two directions for message passing , forward and backward , represented by 0 or 1 , respectively . The symbol space of the first generation message passing path is { 0 , 1 } = { 0 , 1 } 1 , and that of the second generation message passing path is { 00 , 01 , 10 , 11 } = { 0 , 1 } 2 . Generally , the symbol space of the n-th generation message passing path is { 0 , 1 } n and the size of the symbol space is 2n . This means that the scope of message passing spreads with exponent 2 . However , in Bi-GCN ( similar to Bi-LSTM ) and R-GCN architectures , the symbol space is { { 0 } n , { 1 } n } , and its size is 2 , which indicates that a lot of information will be lost in the process of message passing ( see Appendix A ) . How to characterize the above message passing patterns ? We use two mappings to represent message passing process in two directions . Then the interactive passing of messages in different directions is equivalent to the composite operation of corresponding mappings . In addition , the direction of message passing is often random , so we endow the two mappings with an adjoint probability vector to reflect the randomness . Because the symbol space of the iterative path of the Iterated Function System ( IFS ) with two mappings is also { 0 , 1 } n and the mapping is selected with a certain probability , the iterative process of IFS is similar to the message passing process . In other words , the above message passing pattern can be described perfectly by IFS with probabilities . We naturally present the Iterative Graph Neural Network System ( IGNNS ) , whose core layer is constructed by IFS . Figure 1 describes the differences in message passing patterns among GCN , Bi-GCN and IGNNS . At the same time , we regard undirected graph as a directed graph with equal probability of bidirectional message passing ( see Figure 1 ( a ) ) , so the IGNNS architecture can handle directed graph and undirected graph in a unified way . 2 PRELIMINARIES . A graph G = ( V , E ) is defined by its note set V = { v1 , v2 , ... , vN } and edge set E = { ( vi , vj ) |vi , vj ∈ V } . Let A ∈ RN denote the adjacency matrix of G , providing with relational information between nodes . A [ i , j ] denote i , jth element of A , A [ i , : ] means the ith row , and A [ : , j ] means the jth column . In this paper , we assume that all nodes of G are self adjacent , that is A [ i , i ] = 1 , i = 1 , 2 , ... , N . let D = diag ( d1 , d2 , ... , dN ) be the degree matrix of A , where di = ∑N j=1 A [ i , j ] . Neighborhood Normalization . There are two ways to normalize A . One approach is the following mean-pooling employed by Hamilton et al . ( 2017 ) and Veličković et al . ( 2019 ) for inductive learning : Amp = D −1A . Another approach is the following symmetric normalization employed by Kipf & Welling ( 2017 ) : Asym = D − 12AD− 1 2 . Iterated Function System . A mapping f : RN → RN is said to be a contractive mapping on RN if there exists a constant 0 < c < 1 such that ‖f ( x1 ) − f ( x2 ) ‖2 < c‖x1 − x2‖2 for all x1 , x2 ∈ RN . An iterated function system ( Hutchinson , 1981 ) is defined by IFS = { RN ; f1 , f2 , ... , fm ; p } , where each fi : RN → RN is a contractive mapping and p = ( p1 , p2 , ... , pm ) is an adjoint probability vector meaning that fi is selected by probability pi for each iteration . Hutchinson ( 1981 ) showed that there exists a unique nonempty compact set F such that F = m⋃ i=1 fi ( F ) . A = 1 1 0 01 1 0 10 1 1 1 0 1 1 1 , A0 = 1 2 1 2 0 0 0 12 0 1 2 0 0 12 1 2 0 0 0 1 and A1 = 1 0 0 0 1 2 1 2 0 0 0 12 1 2 0 0 13 1 3 1 3 . We call F the fractal set or invariant set of IFS . More conclusions on IFS can be found in the Appendix D. It is well known that there exists a unique probability measure µ with support F satisfying the equation µ = m∑ i=1 piµ ◦ f−1i . ( 1 ) The probability measure µ in ( 1 ) is called the self-similar measure of IFS with probability vector p . 3 IGNNS ARCHITECTURE . In this section , we will introduce the architecture of the IGNNS according to the input layer , IFS layer , representation layer and output layer , which is described in Figure 2 . 3.1 INPUT LAYER . Given a graph structure data X ∈ RN×F of G = ( V , E ) , called as the feature matrix of node set V . A row of X represents the F -dimensional feature vector of a node in V . Let W int ∈ RF×H be a learnable parameter matrix , where H is the dimension of the latent space . Then XW int ∈ RN×H . The output of input layer is defined by X int = σ ( XW int ) ∈ RN×H , ( 2 ) where σ ( · ) is the activation function . Generally , ReLU ( x ) = max ( 0 , x ) is used as the nonlinear activation function . Here , each column of X int is regarded as a point in RN , so X int is the set of H points in RN and arranged in a certain order . The vector composed of the ith component of these points ( the ith row of X int ) is a feature representation of the ith node of graph G . 3.2 IFS LAYER . Let A be the adjacency matrix of G. Let triu ( A ) denote the upper triangular matrix of A and tril ( A ) denote the lower triangular matrix of A . The symmetric normalization of triu ( A ) and tril ( A ) are A0 = D − 12 0 triu ( A ) D − 12 0 and A1 = D − 12 1 tril ( A ) D − 12 1 , where D0 and D1 are degree matrices of triu ( A ) and tril ( A ) respectively . Sometimes , we use the mean-pooling of triu ( A ) and tril ( A ) , i.e . A0 = D−10 triu ( A ) , A1 = D −1 1 tril ( A ) . Let f0 , f1 be the two affine transformations on RN , induced by A0 , A1 respectively , defined as follows : f0 : x→ A0x+ b0 , x ∈ RN , b0 ∈ R , f1 : x→ A1x+ b1 , x ∈ RN , b1 ∈ R , where b0 and b1 are learnable biases , namely add constants b0 and b1 to each component of A0x and A1x respectively . Constructing iterated function system IFS = { RN ; f0 , f1 ; p } , where p = ( p0 , p1 ) is a learnable adjoint probability vector , satisfying p0 > 0 , p1 > 0 and p0+p1 = 1 . Using the symbol space Ωm = { 0 , 1 } m , then for each i = ( i1 , i2 , ... , im ) ∈ Ωm the length of i is m , denoted as |i| = m , and defining pi = pi1pi2 · · · pim and fi = fi1 ◦ fi2 ◦ · · · ◦ fim . Let n be the number of iterations of IFS . For IGNNS , n is a preset parameter . The iterative process of IFS is described as follows : The first iteration ( |i| = 1 ) . The result of the first iteration is denoted by H ( 1 ) = { f0 ( X int ) , f1 ( X int ) } = { Hi } |i|=1 , where Hi = fi ( X int ) , ∀i ∈ Ω1 . Since IFS selects the iteration branch fi with probability pi , the mathematical expectation of H ( 1 ) is computed by E1 = p0f0 ( X int ) + p1f1 ( X int ) = p0H0 + p1H1 = ∑ |i|=1 piHi . If choose to use bias in iterations , then H0 = A0X int + b0 , H1 = A1X int + b1 , where b0 and b1 are learnable H-dimensional vectors . The second iteration ( |i| = 2 ) . Using the results of the first iteration as the input of the second iteration , then the result of the second iteration is denoted by H ( 2 ) = { f0 ( f0 ( X int ) ) , f0 ( f1 ( X int ) ) , f1 ( f0 ( X int ) ) , f1 ( f1 ( X int ) ) } = { f00 ( X int ) , f01 ( X int ) , f10 ( X int ) , f11 ( X int ) } = { Hi } |i|=2 , where Hi = fi ( X int ) , ∀i ∈ Ω2 . Note that IFS selects the iteration path fi with probability pi , then the mathematical expectation of H ( 2 ) is computed by E2 = ∑ |i|=2 pifi ( X int ) = ∑ |i|=2 piHi . We expand the expression of E2 and perceive its powerful feature representation ability . First , H00 = f0 ( H0 ) = A0 ( A0X int + b0 ) + b0 , H01 = f0 ( H1 ) = A0 ( A1X int + b1 ) + b0 , H10 = f1 ( H0 ) = A1 ( A0X int + b0 ) + b1 , H11 = f1 ( H1 ) = A1 ( A1X int + b1 ) + b1 . Then E2 = p00H00 + p01H01 + p10H10 + p11H11 = ( p00A00 + p01A01 + p10A10 + p11A11 ) X int + ( p00A0b0 + p01A0b1 + p10A1b0 + p11A1b1 ) + ( p00b0 + p01b0 + p10b1 + p11b1 ) , where Ai = Ai1Ai2 , ∀i = ( i1 , i2 ) ∈ Ω2 . The n-th iteration ( |i| = n ) . Inductively , we have H ( n ) = { Hi } |i|=n , Hi = fi ( X int ) , En = ∑ |i|=n piHi . ( 3 ) Note that Hi ∈ RN×H and each column of Hi is a point in RN , so we regard it as a subset of RN withH elements . Thus H ( n ) is a subset of RN withH×2n elements ( including duplicate elements ) . Because of Theorem 4.1 , we call H ( n ) the fractal representation with depth n of notes .

Summary: This work proposes a new graph neural network architecture with modified rules for message passing, Iterated Graph Neural Network System (IGNNS). The paper then provides a theoretical analysis of the proposed architecture by connecting it with Iterated Function System (IFS), an important research field in fractal geometry. This paper further demonstrates empirically that the proposed architecture outperforms related models on citation network datasets.