Iterated graph neural network system

1 INTRODUCTION . GNN ( Scarselli et al. , 2009 ) has been proved to be effective in processing graph structured data , and has been widely used in natural language processing , computer vision , data mining , social network and biochemistry . In recent years , GNN has developed a variety of architectures , such as GCN ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) , DGI ( Veličković et al. , 2019 ) , GIN ( Xu et al. , 2019 ) , GCNII ( Ming Chen et al. , 2020 ) and GEN ( Li et al. , 2020 ) . These architectures have a common feature , that is , the representation of each node is updated using messages from its neighbors but without distinguishing the direction ( or angle ) of message passing between two nodes . Recent studies have shown that considering directed message passing between nodes can improve the performance of GNN and achieve success in related fields . For example , DimeNet ( Klicpera et al. , 2020 ) considers the spatial direction from one atom to another and can learn both molecular properties and atomic forces . R-GCN ( Schlichtkrull et al. , 2018 ) and Bi-GCN ( Marcheggiani & Titov , 2017 ; Fu et al. , 2019 ) are models for directed graph , applied in the field of natural language processing . We note that the above direction based model does not consider the bidirectional mixed passing of messages . But in real life , message passing is interactive in different directions . For example , node A obtains a message from node B . After processing the message , node A not only passes it to the next node C , but also feeds back to node B . Suppose there are only two directions for message passing , forward and backward , represented by 0 or 1 , respectively . The symbol space of the first generation message passing path is { 0 , 1 } = { 0 , 1 } 1 , and that of the second generation message passing path is { 00 , 01 , 10 , 11 } = { 0 , 1 } 2 . Generally , the symbol space of the n-th generation message passing path is { 0 , 1 } n and the size of the symbol space is 2n . This means that the scope of message passing spreads with exponent 2 . However , in Bi-GCN ( similar to Bi-LSTM ) and R-GCN architectures , the symbol space is { { 0 } n , { 1 } n } , and its size is 2 , which indicates that a lot of information will be lost in the process of message passing ( see Appendix A ) . How to characterize the above message passing patterns ? We use two mappings to represent message passing process in two directions . Then the interactive passing of messages in different directions is equivalent to the composite operation of corresponding mappings . In addition , the direction of message passing is often random , so we endow the two mappings with an adjoint probability vector to reflect the randomness . Because the symbol space of the iterative path of the Iterated Function System ( IFS ) with two mappings is also { 0 , 1 } n and the mapping is selected with a certain probability , the iterative process of IFS is similar to the message passing process . In other words , the above message passing pattern can be described perfectly by IFS with probabilities . We naturally present the Iterative Graph Neural Network System ( IGNNS ) , whose core layer is constructed by IFS . Figure 1 describes the differences in message passing patterns among GCN , Bi-GCN and IGNNS . At the same time , we regard undirected graph as a directed graph with equal probability of bidirectional message passing ( see Figure 1 ( a ) ) , so the IGNNS architecture can handle directed graph and undirected graph in a unified way . 2 PRELIMINARIES . A graph G = ( V , E ) is defined by its note set V = { v1 , v2 , ... , vN } and edge set E = { ( vi , vj ) |vi , vj ∈ V } . Let A ∈ RN denote the adjacency matrix of G , providing with relational information between nodes . A [ i , j ] denote i , jth element of A , A [ i , : ] means the ith row , and A [ : , j ] means the jth column . In this paper , we assume that all nodes of G are self adjacent , that is A [ i , i ] = 1 , i = 1 , 2 , ... , N . let D = diag ( d1 , d2 , ... , dN ) be the degree matrix of A , where di = ∑N j=1 A [ i , j ] . Neighborhood Normalization . There are two ways to normalize A . One approach is the following mean-pooling employed by Hamilton et al . ( 2017 ) and Veličković et al . ( 2019 ) for inductive learning : Amp = D −1A . Another approach is the following symmetric normalization employed by Kipf & Welling ( 2017 ) : Asym = D − 12AD− 1 2 . Iterated Function System . A mapping f : RN → RN is said to be a contractive mapping on RN if there exists a constant 0 < c < 1 such that ‖f ( x1 ) − f ( x2 ) ‖2 < c‖x1 − x2‖2 for all x1 , x2 ∈ RN . An iterated function system ( Hutchinson , 1981 ) is defined by IFS = { RN ; f1 , f2 , ... , fm ; p } , where each fi : RN → RN is a contractive mapping and p = ( p1 , p2 , ... , pm ) is an adjoint probability vector meaning that fi is selected by probability pi for each iteration . Hutchinson ( 1981 ) showed that there exists a unique nonempty compact set F such that F = m⋃ i=1 fi ( F ) . A = 1 1 0 01 1 0 10 1 1 1 0 1 1 1 , A0 = 1 2 1 2 0 0 0 12 0 1 2 0 0 12 1 2 0 0 0 1 and A1 = 1 0 0 0 1 2 1 2 0 0 0 12 1 2 0 0 13 1 3 1 3 . We call F the fractal set or invariant set of IFS . More conclusions on IFS can be found in the Appendix D. It is well known that there exists a unique probability measure µ with support F satisfying the equation µ = m∑ i=1 piµ ◦ f−1i . ( 1 ) The probability measure µ in ( 1 ) is called the self-similar measure of IFS with probability vector p . 3 IGNNS ARCHITECTURE . In this section , we will introduce the architecture of the IGNNS according to the input layer , IFS layer , representation layer and output layer , which is described in Figure 2 . 3.1 INPUT LAYER . Given a graph structure data X ∈ RN×F of G = ( V , E ) , called as the feature matrix of node set V . A row of X represents the F -dimensional feature vector of a node in V . Let W int ∈ RF×H be a learnable parameter matrix , where H is the dimension of the latent space . Then XW int ∈ RN×H . The output of input layer is defined by X int = σ ( XW int ) ∈ RN×H , ( 2 ) where σ ( · ) is the activation function . Generally , ReLU ( x ) = max ( 0 , x ) is used as the nonlinear activation function . Here , each column of X int is regarded as a point in RN , so X int is the set of H points in RN and arranged in a certain order . The vector composed of the ith component of these points ( the ith row of X int ) is a feature representation of the ith node of graph G . 3.2 IFS LAYER . Let A be the adjacency matrix of G. Let triu ( A ) denote the upper triangular matrix of A and tril ( A ) denote the lower triangular matrix of A . The symmetric normalization of triu ( A ) and tril ( A ) are A0 = D − 12 0 triu ( A ) D − 12 0 and A1 = D − 12 1 tril ( A ) D − 12 1 , where D0 and D1 are degree matrices of triu ( A ) and tril ( A ) respectively . Sometimes , we use the mean-pooling of triu ( A ) and tril ( A ) , i.e . A0 = D−10 triu ( A ) , A1 = D −1 1 tril ( A ) . Let f0 , f1 be the two affine transformations on RN , induced by A0 , A1 respectively , defined as follows : f0 : x→ A0x+ b0 , x ∈ RN , b0 ∈ R , f1 : x→ A1x+ b1 , x ∈ RN , b1 ∈ R , where b0 and b1 are learnable biases , namely add constants b0 and b1 to each component of A0x and A1x respectively . Constructing iterated function system IFS = { RN ; f0 , f1 ; p } , where p = ( p0 , p1 ) is a learnable adjoint probability vector , satisfying p0 > 0 , p1 > 0 and p0+p1 = 1 . Using the symbol space Ωm = { 0 , 1 } m , then for each i = ( i1 , i2 , ... , im ) ∈ Ωm the length of i is m , denoted as |i| = m , and defining pi = pi1pi2 · · · pim and fi = fi1 ◦ fi2 ◦ · · · ◦ fim . Let n be the number of iterations of IFS . For IGNNS , n is a preset parameter . The iterative process of IFS is described as follows : The first iteration ( |i| = 1 ) . The result of the first iteration is denoted by H ( 1 ) = { f0 ( X int ) , f1 ( X int ) } = { Hi } |i|=1 , where Hi = fi ( X int ) , ∀i ∈ Ω1 . Since IFS selects the iteration branch fi with probability pi , the mathematical expectation of H ( 1 ) is computed by E1 = p0f0 ( X int ) + p1f1 ( X int ) = p0H0 + p1H1 = ∑ |i|=1 piHi . If choose to use bias in iterations , then H0 = A0X int + b0 , H1 = A1X int + b1 , where b0 and b1 are learnable H-dimensional vectors . The second iteration ( |i| = 2 ) . Using the results of the first iteration as the input of the second iteration , then the result of the second iteration is denoted by H ( 2 ) = { f0 ( f0 ( X int ) ) , f0 ( f1 ( X int ) ) , f1 ( f0 ( X int ) ) , f1 ( f1 ( X int ) ) } = { f00 ( X int ) , f01 ( X int ) , f10 ( X int ) , f11 ( X int ) } = { Hi } |i|=2 , where Hi = fi ( X int ) , ∀i ∈ Ω2 . Note that IFS selects the iteration path fi with probability pi , then the mathematical expectation of H ( 2 ) is computed by E2 = ∑ |i|=2 pifi ( X int ) = ∑ |i|=2 piHi . We expand the expression of E2 and perceive its powerful feature representation ability . First , H00 = f0 ( H0 ) = A0 ( A0X int + b0 ) + b0 , H01 = f0 ( H1 ) = A0 ( A1X int + b1 ) + b0 , H10 = f1 ( H0 ) = A1 ( A0X int + b0 ) + b1 , H11 = f1 ( H1 ) = A1 ( A1X int + b1 ) + b1 . Then E2 = p00H00 + p01H01 + p10H10 + p11H11 = ( p00A00 + p01A01 + p10A10 + p11A11 ) X int + ( p00A0b0 + p01A0b1 + p10A1b0 + p11A1b1 ) + ( p00b0 + p01b0 + p10b1 + p11b1 ) , where Ai = Ai1Ai2 , ∀i = ( i1 , i2 ) ∈ Ω2 . The n-th iteration ( |i| = n ) . Inductively , we have H ( n ) = { Hi } |i|=n , Hi = fi ( X int ) , En = ∑ |i|=n piHi . ( 3 ) Note that Hi ∈ RN×H and each column of Hi is a point in RN , so we regard it as a subset of RN withH elements . Thus H ( n ) is a subset of RN withH×2n elements ( including duplicate elements ) . Because of Theorem 4.1 , we call H ( n ) the fractal representation with depth n of notes .

Summary: This paper proposes a new framework of GNN which can deal with undirected and directed graphs in a unified way. The authors argue that the size of the symbol space for a message passing path with length n is 2^n, while previous architectures only have constant size. Motivated by this observation, the authors borrow ideas from Iterated Function System to augment the symbol space.