Learning Graph Normalization for Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have shown great popularity due to their efficiency in learning on graphs for various application areas , such as natural language processing ( Yao et al. , 2019 ; Zhang et al. , 2018 ) , computer vision ( Li et al. , 2020 ; Cheng et al. , 2020 ) , point cloud ( Shi & Rajkumar , 2020 ) , drug discovery ( Lim et al. , 2019 ) , citation networks ( Kipf & Welling , 2016 ) , and social networks ( Chen et al. , 2018 ) . A graph consists of nodes and edges , where nodes represent individual objects and edges represent relationships among those objects . In the GNN framework , the node or edge representations are alternately updated by propagating information along the edges of a graph via non-linear transformation and aggregation functions ( Wu et al. , 2020 ; Zhang et al. , 2018 ) . GNN captures long-range node dependencies via stacking multiple message-passing layers , allowing the information to propagate over multiple-hops ( Xu et al. , 2018 ) . In essence , GNN is a new kind of neural networks which exploits neural network operations over graph structure . Among the numerous kinds of GNNs ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Maron et al. , 2019 ; Xu et al. , 2019 ) , message-passing GNNs ( Scarselli et al. , 2009 ; Li et al. , 2016 ; Kipf & Welling , 2016 ; Velickovic et al. , 2018 ; Bresson & Laurent , 2017 ) have been the most widely used due to their ability to leverage the basic building blocks of deep learning such as batching , normalization and residual connections . To update the feature representation of a node , many approaches are designed . For example , Graph ConvNet ( GCN ) ( Kipf & Welling , 2016 ) employs an averaging operation over the neighborhood node with the same weight value for each of its neighbors ; GraphSage ( Hamilton et al. , 2017 ) samples a fixed-size neighborhood of each node and performs mean aggregator or LSTM-based aggregator over the neighbors ; Graph Attention Network ( GAT ) ( Velickovic et al. , 2018 ) incorporates an attention mechanism into the propagation step , which updates the feature representation of each code via a weighted sum of adjacent node representations ; MoNet ( Monti et al. , 2017 ) designs a Gaussian kernel with learnable parameters to assign different weights to neighbors ; GatedGCN ( Bresson & Laurent , 2017 ) explicitly introduces edge features at each layer and updates edge features by considering the feature representations of these two con- It is well known that one of the critical ingredients to effectively train deep neural networks is normalization technique , e.g. , Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) is widely used to accelerate the deep neural networks training . Other than BN , several normalization methods have been developed from different perspectives , e.g. , Layer Normalization ( LN ) ( Ba et al. , 2016 ) and Group Normalization ( Wu & He , 2018 ) which operate along the channel dimension , Instance Normalization ( Ulyanov et al. , 2016 ) which performs a BN-like normalization for each sample , Switchable Normalization ( Luo et al. , 2019 ) which utilizes three distinct scopes—including channel , layer , and minibatch—to compute the first order and second order statistics . Each normalization method has its advantages and is suitable for some particular tasks . For instance , BN has achieved perfect performance in computer vision whereas LN outperforms BN in natural language processing ( Vaswani et al. , 2017 ) . As an analogue , in Dwivedi et al . ( 2020 ) , BN is utilized for each graph propagation layer during training GNNs . In Zhao & Akoglu ( 2020 ) , a novel normalization layer , denoted as PAIRNORM , is introduced to mitigate the over-smoothing problem and prevent all node representations from homogenization by differentiating the distances between different node pairs . Although these methods mentioned above have been demonstrated being useful in training GNNs , the local structure and global structure of the graph are ignored in these existing methods . Moreover , in previous work , only one of the mentioned normalization methods is selected and it is used for all normalization layers . This may limit the potential performance improvement of the normalization method and it is also hard to decide which normalization method is suitable to a specific task . Graph data contains rich structural information . By considering the structure information in the graph , in this paper , we propose two graph-aware normalization methods at different scales : a ) adjacency-wise normalization , and b ) graph-wise normalization . Unlike BN and LN , the adjacencywise normalization takes into account the local structure in the graph whereas the graph-wise normalization takes into account the global structure in the graph . On other hand , while multiple normalization methods are available for training GNNs and it is still hard to know in advance which normalization method is the most suitable to a specific task . To tackle with this deficiency , we further propose to learn attentive graph normalization by optimizing a weighted combination of multiple normalization methods . By optimizing the combination weights , we can select the best or the best combination of multiple normalization methods for training GNNs at a specific task automatically . The contributions of the paper are highlighted as follows . • We propose two graph-aware normalization methods : adjacency-wise normalization and graphwise normalization . To the best of our knowledge , it is for the first time that the graph-aware normalization method is proposed for training GNNs . • We present to learn attentive graph normalization by optimizing a weighted combination of different normalization methods . By learning the combination weights , we can automatically select the best normalization method or the best combination of multiple normalization methods for training GNNs at a specific task . • We conduct extensive experiments on benchmark datasets for different tasks and confirm that the graph-aware normalization methods leads to promising results and that the learned weights suggest the more appropriate normalization methods for specific task . 2 GRAPH-AWARE NORMALIZATION AT DIFFERENT SCALES . Suppose that we haveN graphs G1 , G2 , ... , GN in a mini-batch . Let Gk = ( Vk , Ek ) be the k-th graph , where Vk is the set of nodes and Ek is the set of edges . We use vk , i to denote the i-th node of graph Gk and use ek , i , j to denote the edge between nodes vk , i and vk , j of graph Gk . Moreover , we use hvk , i ∈ Rd to represent the feature of node vk , i and hjvk , i to represent the j-th element of hvk , i . We use N ( vk , i ) to represent the neighbors of node vk , i ( including node vk , i itself ) . For clarity , we formulate the normalization methods for training GNNs from different scales , as illustrated in Figure 1 ( a ) - ( d ) , including node-wise normalization , adjacency-wise normalization , graph-wise normalization and batch-wise normalization . Node-wise Normalization . Node-wise normalization on graph , denoted as GNn , considers to normalize the feature vector hvk , i of each node vk , i and compute the first and the second statistics over the d entries of the feature vector hvk , i as follows : ĥ ( n ) vk , i = hvk , i − µ ( n ) k , i 1 σ ( n ) k , i , µ ( n ) k , i = 1 d d∑ j=1 hjvk , i , σ ( n ) k , i = √√√√1 d d∑ j=1 ( hjvk , i − µ ( n ) k , i ) 2 , ( 1 ) where µ ( n ) k , i and σ ( n ) k , i are the mean and the standard deviation along the feature dimension for node vk , i , and 1 ∈ Rd represents a d-dimension vector of all 1 . Note that node-wise normalization is equivalent to applying LN to each node of the graph to reduce the “ covariate shift ” problem1 . Adjacency-wise Normalization . Each node in a graph has its neighbors . However , node-wise normalization performs normalization on each node individually and ignores the local structure in the graph . Here , we propose to take into account the adjacency structure in the graph and normalize the node features of the adjacent neighbors . We term it as adjacency-wise normalization on graph , denoted as GNa . For each node vk , i in graph Gk , we consider its adjacent nodesN ( vk , i ) , as illustrated in Figure 1 ( b ) . Specifically , the adjacency-wise normalization for node vk , i is defined as follows : ĥ ( a ) vk , i = hvk , i − µ ( a ) k , i1 σ ( a ) k , i , ( 2 ) µ ( a ) k , i = 1 |N ( vk , i ) | × d ∑ j′∈N ( vk , i ) d∑ j=1 hjvk , j′ , ( 3 ) σ ( a ) k , i = √√√√ 1 |N ( vk , i ) | × d ∑ j′∈N ( vk , i ) d∑ j=1 ( hjvk , j′ − µ ( a ) k , i ) 2 , ( 4 ) where µ ( a ) k , i and σ ( a ) k , i are the first order and second order statistics over the adjacent nodes . 2 Graph-wise Normalization . Note that nodes belonging to graph Gk naturally form a group . In order to preserve the global structure of a graph , we propose to normalize the node feature based on the first and the second order statistics computed over graph Gk . Specifically , we define a graph-wise 1The node-wise normalization method in Equation ( 1 ) can also be used to normalize the feature at each edge , as illustrated in Figure 1 ( e ) . 2For the edge ek , i , j , as Figure 1 ( f ) , the adjacent edgesN ( ek , i ) can be considered in a similar way . normalization on graph , denoted as GNg , for node vk , i as follows : ĥ ( g ) vk , i = ( hvk , i − µ ( g ) k ) Λ −1 k , ( 5 ) µ ( g ) k = 1 |Gk| ∑ vk , i∈Gk hvk , i , ( 6 ) where µ ( g ) k and Λk are the first order and the second order statistics in graph Gk in which Λk is a diagonal matrix with diagonal entry Λjjk is defined as Λjjk = √√√√ 1 |Gk| ∑ vk , i∈Gk ( hjvk , i − µ ( g ) , j k ) 2 . ( 7 ) If the task has only a single graph , then graph-wise normalization is similar to BN . However , unlike in BN , graph-wise normalization does not use a smoothing average updater.3 Batch-wise Normalization . To keep training stable , BN is one of the most critical components . For a mini-batch , there are N graphs . We compute the mean and standard deviation across over the graphs of a mini-batch , then each node feature hvk , i is normalized as follows : ĥ ( b ) vk , i = ( hvk , i − µ ( b ) ) Λ−1 , ( 8 ) µ ( b ) = 1 T N∑ k=1 |Gk|∑ i=1 hvk , i , ( 9 ) where T = ∑N k=1 |Gk| means the total number of the nodes in the N graphs and Λ is a diagonal matrix to keep the standard deviation of the note features over N graphs in which the diagonal entry Λjj is defined Λjj = √√√√ 1 T N∑ k=1 |Gk|∑ i=1 ( hjvk , i − µ ( b ) , j ) 2 . ( 10 ) Note that batch-wise normalization on graph , named as GNb , is effectively BN ( Ioffe & Szegedy , 2015 ) , which performs normalization over all nodes of the N graphs in a mini-batch . The normalization methods applying to node features hvk , i can also be extended to edge features hek , i , j where hek , i , j denotes the feature of edge ei , j in graph Gk , as illustrated in Figure 1 ( e ) - ( h ) . Remark . The properties of the four normalization methods are summarized as follows . • Node-wise normalization only considers to normalize the feature of each node individually but ignores the adjacency structure and the whole graph structures . It is equivalent to LN ( Ba et al. , 2016 ) in operation . • Adjacency-wise normalization takes the adjacent nodes into account , whereas graph-wise normalization takes into account the features of all nodes in a graph . • Batch-wise normalization is the same as the standard batch normalization ( Ioffe & Szegedy , 2015 ) . If the task only involves a single graph , then the batch-wise normalization is similar to the graph normalization except that momentum average used in batch-wise normalization is not used in the graph-wise normalization .

Summary: This paper proposes and evaluates different normalization techniques for graph neural networks. Also, the authors argue that the best normalization technique is task dependent, so they propose to use a weighted average of different normalizations that is learned during training, called AGN. In the paper they propose 4 different normalizations some of which are structure-dependent, and compare the performance of GCN, GAT and GatedGCN with and without these normalizations, and the learned combination of all of them.