Directional graph networks

1 INTRODUCTION . One of the most important distinctions between convolutional neural networks ( CNNs ) and graph neural networks ( GNNs ) is that CNNs allow for any convolutional kernel , while most GNN methods are limited to symmetric kernels ( also called isotropic kernels in the literature ) ( Kipf & Welling , 2016 ; Xu et al. , 2018a ; Gilmer et al. , 2017 ) . There are some implementation of asymmetric kernels using gated mechanisms ( Bresson & Laurent , 2017 ; Veličković et al. , 2017 ) , motif attention ( Peng et al. , 2019 ) , edge features ( Gilmer et al. , 2017 ) or by using the 3D structure of molecules for message passing ( Klicpera et al. , 2019 ) . However , to the best of our knowledge , there are currently no methods that allow asymmetric graph kernels that are dependent on the full graph structure or directional flows . They either depend on local structures or local features . This is in opposition to images which exhibit canonical directions : the horizontal and vertical axes . The absence of an analogous concept in graphs makes it difficult to define directional message passing and to produce an analogue of the directional frequency filters ( or Gabor filters ) widely present in image processing ( Olah et al. , 2020 ) . We propose a novel idea for GNNs : use vector fields in the graph to define directions for the propagation of information , with an overview of the paper presented in 1 . Hence , the aggregation or message passing will be projected onto these directions so that the contribution of each neighbouring node nv will be weighted by its alignment with the vector fields at the receiving node nu . This enables our method to propagate information via directional derivatives or smoothing of the features . We also explore using the gradients of the low-frequency eigenvectors of the Laplacian of the graph φk , since they exhibit interesting properties ( Bronstein et al. , 2017 ; Chung et al. , 1997 ) . In particular , they can be used to define optimal partitions of the nodes in a graph , to give a natural ordering ( Levy , 2006 ) , and to find the dominant directions of the graph diffusion process ( Chung & Yau , 2000 ) . Further , we show that they generalize the horizontal and vertical directional flows in a grid ( see figure 2 ) , allowing them to guide the aggregation and mimic the asymmetric and directional kernels present in computer vision . In fact , we demonstrate mathematically that our work generalizes CNNs by reproducing all convolutional kernels of radius R in an n-dimensional grid , while also bringing the powerful data augmentation capabilities of reflection , rotation or distortion of the directions . We further show that our directional graph network ( DGN ) model theoretically and empirically allows for efficient message passing across distant communities , which reduces the well-known problem of over-smoothing , and aligns well with the need of independent aggregation rules ( Corso et al. , 2020 ) . Alternative methods reduce the impact of over-smoothing by using skip connections ( Luan et al. , 2019 ) , global pooling ( Alon & Yahav , 2020 ) , or randomly dropping edges during training time ( Rong et al. , 2020 ) , but without solving the underlying problem . In fact , we also prove that DGN is more discriminative than standard GNNs in regards to the Weisfeiler-Lehman 1-WL test , showing that the reduction of over-smoothing is accompanied by an increase of expressiveness . Our method distinguishes itself from other spectral GNNs since the literature usually uses the low frequencies to estimate local Fourier transforms in the graph ( Levie et al. , 2018 ; Xu et al. , 2019 ) . Instead , we do not try to approximate the Fourier transform , but only to define a directional flow at each node and guide the aggregation . 2 THEORETICAL DEVELOPMENT . 2.1 INTUITIVE OVERVIEW . One of the biggest limitations of current GNN methods compared to CNNs is the inability to do message passing in a specific direction such as the horizontal one in a grid graph . In fact , it is difficult to define directions or coordinates based solely on the shape of the graph . The lack of directions strongly limits the discriminative abilities of GNNs to understand local structures and simple feature transformations . Most GNNs are invariant to the permutation of the neighbours ’ features , so the nodes ’ received signal is not influenced by swapping the features of 2 neighbours . Therefore , several layers in a deep network will be employed to understand these simple changes instead of being used for higher level features , thus over-squashing the message sent between 2 distant nodes ( Alon & Yahav , 2020 ) . In this work , one of the main contributions is the realisation that low-frequency eigenvectors of the Laplacian can overcome this limitation by providing a variety of intuitive directional flows . As a first example , taking a grid-shaped graph of sizeN×M with N2 < M < N , we find that the eigenvector associated to the smallest non-zero eigenvalue increases in the direction of the width N and the second one increases in the direction of the height M . This property generalizes to n-dimensional grids and motivated the use of gradients of eigenvectors as preferred directions for general graphs . We validated this intuition by looking at the flow of the gradient of the eigenvectors for a variety of graphs , as shown in figure 2 . For example , in the Minnesota map , the first 3 non-constant eigenvectors produce logical directions , namely South/North , suburb/city , and West/East . Another important contribution also noted in figure 2 is the ability to define any kind of direction based on prior knowledge of the problem . Hence , instead of relying on eigenvectors to find directions in a map , we can simply use the cardinal directions or the rush-hour traffic flow . 2.2 VECTOR FIELDS IN A GRAPH . Based on a recent review from Bronstein et al . ( 2017 ) , this section presents the ideas of differential geometry applied to graphs , with the goal of finding proper definitions of scalar products , gradients and directional derivatives . Let G = ( V , E ) be a graph with V the set of vertices and E ⊂ V × V the set of edges . The graph is undirected meaning that ( i , j ) ∈ E iff ( j , i ) ∈ E. Define the vector spaces L2 ( V ) and L2 ( E ) as the set of maps V → R and E → R with x , y ∈ L2 ( V ) and F , H ∈ L2 ( E ) and scalar products 〈x , y〉L2 ( V ) : = ∑ i∈V xiyi , 〈F , H〉L2 ( E ) : = ∑ ( i , j ) ∈E F ( i , j ) H ( i , j ) ( 1 ) Think of E as the “ tangent space ” to V and of L2 ( E ) as the set of “ vector fields ” on the space V with each row Fi , : representing a vector at the i-th node . Define the pointwise scalar product as the map L2 ( E ) ×L2 ( E ) → L2 ( V ) taking 2 vector fields and returning their inner product at each point of V , at the node i is defined by equation 2 . 〈F , H〉i : = ∑ j : ( i , j ) ∈E Fi , jHi , j ( 2 ) In equation 3 , we define the gradient ∇ as a mapping L2 ( V ) → L2 ( E ) and the divergence div as a mapping L2 ( E ) → L2 ( V ) , thus leading to an analogue of the directional derivative in equation 4 . ( ∇x ) ( i , j ) : = x ( j ) − x ( i ) , ( divF ) i : = ∑ j : ( i , j ) ∈E F ( i , j ) ( 3 ) Definition 1 . The directional derivative of the function x on the graph G in the direction of the vector field F̂ where each vector is of unit-norm is DF̂x ( i ) : = 〈∇x , F̂ 〉i = ∑ j : ( i , j ) ∈E ( x ( j ) − x ( i ) ) F̂i , j ( 4 ) |F | will denote the absolute value of F and ||Fi , :||Lp the Lp-norm of the i-th row of F . We also define the forward/backward directions as the positive/negative parts of the field F± . 2.3 DIRECTIONAL SMOOTHING AND DERIVATIVES . Next , we show how the vector field F is used to guide the graph aggregation by projecting the incoming messages . Specifically , we define the weighted aggregation matrices Bav and Bdx that allow to compute the directional smoothing and directional derivative of the node features . The directional average matrixBav is the weighted aggregation matrix such that all weights are positives and all rows have an L1-norm equal to 1 , as shown in equation 5 and theorem 2.1 , with a proof in the appendix C.1 . Bav ( F ) i , : = |Fi , :| ||Fi , :||L1 + ( 5 ) The variable is an arbitrarily small positive number used to avoid floating-point errors . The L1norm denominator is a local row-wise normalization . The aggregator works by assigning a large weight to the elements in the forward or backward direction of the field , while assigning a small weight to the other elements , with a total weight of 1 . Theorem 2.1 ( Directional smoothing ) . The operation y = Bavx is the directional average of x , in the sense that yu is the mean of xv , weighted by the direction and amplitude of F . The directional derivative matrix Bdx is defined in ( 6 ) and theorem 2.2 , with the proof in appendix C.2 . Again , the denominator is a local row-wise normalization but can be replaced by a global normalization . diag ( a ) is a square , diagonal matrix with diagonal entries given by a . The aggregator works by subtracting the projected forward message by the backward message ( similar to a center derivative ) , with an additional diagonal term to balance both directions . Bdx ( F ) i , : = F̂i , : − diag ( ∑ j F̂ : ,j ) i , : F̂i , : = ( Fi , : ||Fi , :||L1 + ) ( 6 ) Theorem 2.2 ( Directional derivative ) . Suppose F̂ have rows of unit L1 norm . The operation y = Bdx ( F̂ ) x is the centered directional derivative of x in the direction of F , in the sense of equation 4 , i.e . y = DF̂x = ( F̂ − diag ( ∑ j F̂ : ,j ) ) x These aggregators are directional , interpretable and complementary , making them ideal choices for GNNs . We discuss the choice of aggregators in more details in appendix A , while also providing alternative aggregation matrices such as the center-balanced smoothing , the forward-copy , the phantom zero-padding , and the hardening of the aggregators using softmax/argmax on the field . We further provide a visual interpretation of theBav andBdx aggregators in figure 3 . Interestingly , we also note in appendix A.1 thatBav andBdx yield respectively the mean and Laplacian aggregations when F is a vector field such that all entries are constant Fij = ±C . 𝑭𝑣 , 𝑢3 𝑭𝑣 , 𝑢1 𝑭𝑣 , 𝑢2 Direc�onal smoothing aggrega�on 𝑩𝑎𝑣 𝑭 𝒙 Direc�onal deriva�ve aggrega�on 𝑩𝑑𝑥 𝑭 𝒙Graph features focused on the neighbourhood of 𝒏𝒗 𝑣 : Node receiving the message𝑢1,2,3 : Neighbouring node𝒙𝑢 : Feature at node 𝑢𝑭𝑣 , 𝑢 : Direc�onal vector field between the node 𝑣 and 𝑢 Weighted forward deriva�ve with 𝑢1 Weighted backwardderiva�ve with 𝑢2 Weighted backwardderiva�ve with 𝑢3+ +Sum of the absolute weights Figure 3 : Illustration of how the directional aggregation works at a node nv , with the arrows representing the direction and intensity of the field F .

Summary: The authors propose a convolution as a message passing of node features over edges where messages are aggregated weighted by a "direction" edge field. Furthermore, the authors propose to use the gradients of Laplace eigenfunctions as direction fields. Presumably, the aggregation is done with different direction fields derived from the Laplace eigenfunctions with lowest eigenvalues, which are then linearly combined with learnable parameters. Doing so allows their graph network to behave more like a conventional CNN, in which the kernels have different parameters for signals from different directions. The authors achieve good results on several benchmarks. Furthermore, the authors prove that their method reduces to a conventional CNN on a rectangular grid and have theoretical results that suggest that their method suffers less from the "over-smoothing" and "over-squashing" problems.