PairNorm: Tackling Oversmoothing in GNNs

The performance of graph neural nets ( GNNs ) is known to gradually decrease with increasing number of layers . This decay is partly attributed to oversmoothing , where repeated graph convolutions eventually make node embeddings indistinguishable . We take a closer look at two different interpretations , aiming to quantify oversmoothing . Our main contribution is PAIRNORM , a novel normalization layer that is based on a careful analysis of the graph convolution operator , which prevents all node embeddings from becoming too similar . What is more , PAIRNORM is fast , easy to implement without any change to network architecture nor any additional parameters , and is broadly applicable to any GNN . Experiments on real-world graphs demonstrate that PAIRNORM makes deeper GCN , GAT , and SGC models more robust against oversmoothing , and significantly boosts performance for a new problem setting that benefits from deeper GNNs . Code is available at https : //github.com/LingxiaoShawn/PairNorm . 1 INTRODUCTION . Graph neural networks ( GNNs ) is a family of neural networks that can learn from graph structured data . Starting with the success of GCN ( Kipf & Welling , 2017 ) on achieving state-of-the-art performance on semi-supervised classification , several variants of GNNs have been developed for this task ; including GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Velickovic et al. , 2018 ) , SGC ( Wu et al. , 2019 ) , and GMNN ( Qu et al. , 2019 ) to name a few most recent ones . A key issue with GNNs is their depth limitations . It has been observed that deeply stacking the layers often results in significant drops in performance for GNNs , such as GCN and GAT , even beyond just a few ( 2–4 ) layers . This drop is associated with a number of factors ; including the vanishing gradients in back-propagation , overfitting due to the increasing number of parameters , as well as the phenomenon called oversmoothing . Li et al . ( 2018 ) was the first to call attention to the oversmoothing problem . Having shown that the graph convolution is a type of Laplacian smoothing , they proved that after repeatedly applying Laplacian smoothing many times , the features of the nodes in the ( connected ) graph would converge to similar values—the issue coined as “ oversmoothing ” . In effect , oversmoothing hurts classification performance by causing the node representations to be indistinguishable across different classes . Later , several others have alluded to the same problem ( Xu et al. , 2018 ; Klicpera et al. , 2019 ; Rong et al. , 2019 ; Li et al. , 2019 ) ( See §5 Related Work ) . In this work , we address the oversmoothing problem in deep GNNs . Specifically , we propose ( to the best of our knowledge ) the first normalization layer for GNNs that is applied in-between intermediate layers during training . Our normalization has the effect of preventing the output features of distant nodes to be too similar or indistinguishable , while at the same time allowing those of connected nodes in the same cluster become more similar . We summarize our main contributions as follows . • Normalization to Tackle Oversmoothing in GNNs : We introduce a normalization scheme , called PAIRNORM , that makes GNNs significantly more robust to oversmoothing and as a result enables the training of deeper models without sacrificing performance . Our proposed scheme capitalizes on the understanding that most GNNs perform a special form of Laplacian smoothing , which makes node features more similar to one another . The key idea is to ensure that the total pairwise feature distances remains a constant across layers , which in turn leads to distant pairs having less similar features , preventing feature mixing across clusters . • Speed and Generality : PAIRNORM is very straightforward to implement and introduces no additional parameters . It is simply applied to the output features of each layer ( except the last one ) consisting of simple operations , in particular centering and scaling , that are linear in the input size . Being a simple normalization step between layers , PAIRNORM is not specific to any particular GNN but rather applies broadly . • Use Case for Deeper GNNs : While PAIRNORM prevents performance from dropping significantly with increasing number of layers , it does not necessarily yield increased performance in absolute terms . We find that this is because shallow architectures with no more than 2–4 layers is sufficient for the often-used benchmark datasets in the literature . In response , we motivate a real-world scenario wherein a notable portion of the nodes have no feature vectors . In such settings , nodes benefit from a larger range ( i.e. , neighborhood , hence a deeper GNN ) to “ recover ” effective feature representations . Through extensive experiments , we show that GNNs employing our PAIRNORM significantly outperform the ‘ vanilla ’ GNNs when deeper models are beneficial to the classification task . 2 UNDERSTANDING OVERSMOOTHING . In this work , we consider the semi-supervised node classification ( SSNC ) problem on a graph . In the general setting , a graph G = ( V , E , X ) is given in which each node i ∈ V is associated with a feature vector xi ∈ Rd where X = [ x1 , . . . , xn ] T denotes the feature matrix , and a subset Vl ⊂ V of the nodes are labeled , i.e . yi ∈ { 1 , . . . , c } for each i ∈ Vl where c is the number of classes . Let A ∈ Rn×n be the adjacency matrix and D = diag ( deg1 , . . . , degn ) ∈ Rn×n be the degree matrix of G. Let Ã = A + I and D̃ = D + I denote the augmented adjacency and degree matrices with added self-loops on all nodes , respectively . Let Ãsym = D̃−1/2ÃD̃−1/2 and Ãrw = D̃−1Ã denote symmetrically and nonsymmetrically normalized adjacency matrices with self-loops . The task is to learn a hypothesis that predicts yi from xi that generalizes to the unlabeled nodes Vu = V\Vl . In Section 3.2 , we introduce a variant of this setting where only a subset F ⊂ V of the nodes have feature vectors and the rest are missing . 2.1 THE OVERSMOOTHING PROBLEM . Although GNNs like GCN and GAT achieve state-of-the-art results in a variety of graph-based tasks , these models are not very well-understood , especially why they work for the SSNC problem where only a small amount of training data is available . The success appears to be limited to shallow GNNs , where the performance gradually decreases with the increasing number of layers . This decrease is often attributed to three contributing factors : ( 1 ) overfitting due to increasing number of parameters , ( 2 ) difficulty of training due to vanishing gradients , and ( 3 ) oversmoothing due to many graph convolutions . Among these , perhaps the least understood one is oversmoothing , which indeed lacks a formal definition . In their analysis of GCN ’ s working mechanism , Li et al . ( 2018 ) showed that the graph convolution of GCN is a special form of Laplacian smoothing . The standard form being ( I−γI ) X+ γÃrwX , the graph convolution lets γ = 1 and uses the symmetrically normalized Laplacian to obtain X̃ = ÃsymX , where the new features x̃ of a node is the weighted average of its own and its neighbors ’ features . This smoothing allows the node representations within the same cluster become more similar , and in turn helps improve SSNC performance under the cluster assumption ( Chapelle et al. , 2006 ) . However when GCN goes deep , the performance can suffer from oversmoothing where node representations from different clusters become mixed up . Let us refer to this issue of node representations becoming too similar as node-wise oversmoothing . Another way of thinking about oversmoothing is as follows . Repeatedly applying Laplacian smoothing too many times would drive node features to a stationary point , washing away all the information from these features . Let x·j ∈ Rn denote the j-th column of X . Then , for any x·j ∈ Rn : lim k→∞ Ãksymx·j = πj and πj ‖πj‖1 = π , ( 1 ) where the normalized solution π ∈ Rn satisfies πi = √ degi∑ i √ degi for all i ∈ [ n ] . Notice that π is independent of the values x·j of the input feature and is only a function of the graph structure ( i.e. , degree ) . In other words , ( Laplacian ) oversmoothing washes away the signal from all the features , making them indistinguishable . We will refer to this viewpoint as feature-wise oversmoothing . To this end we propose two measures , row-diff and col-diff , to quantify these two types of oversmoothing . Let H ( k ) ∈ Rn×d be the representation matrix after k graph convolutions , i.e . H ( k ) = ÃksymX . Let h ( k ) i ∈ Rd be the i-th row of H ( k ) and h ( k ) ·i ∈ Rn be the i-th column of H ( k ) . Then we define row-diff ( H ( k ) ) and col-diff ( H ( k ) ) as follows . row-diff ( H ( k ) ) = 1 n2 ∑ i , j∈ [ n ] ∥∥∥h ( k ) i − h ( k ) j ∥∥∥ 2 ( 2 ) col-diff ( H ( k ) ) = 1 d2 ∑ i , j∈ [ d ] ∥∥∥h ( k ) ·i /‖h ( k ) ·i ‖1 − h ( k ) ·j /‖h ( k ) ·j ‖1∥∥∥ 2 ( 3 ) The row-diff measure is the average of all pairwise distances between the node features ( i.e. , rows of the representation matrix ) and quantifies node-wise oversmoothing , whereas col-diff is the average of pairwise distances between ( L1-normalized1 ) columns of the representation matrix and quantifies feature-wise oversmoothing . 2.2 STUDYING OVERSMOOTHING WITH SGC . Although oversmoothing can be a cause of performance drop with increasing number of layers in GCN , adding more layers also leads to more parameters ( due to learned linear projections W ( k ) at each layer k ) which magnify the potential of overfitting . Furthermore , deeper models also make the training harder as backpropagation suffers from vanishing gradients . In order to decouple the effect of oversmoothing from these other two factors , we study the oversmoothing problem using the SGC model ( Wu et al. , 2019 ) . ( Results on other GNNs are presented in §4 . ) SGC is simplified from GCN by removing all projection parameters of graph convolution layers and all nonlinear activations between layers . The estimation of SGC is simply written as : Ŷ = softmax ( ÃKsym X W ) ( 4 ) where K is the number of graph convolutions , and W ∈ Rd×c denote the learnable parameters of a logistic regression classifier . Note that SGC has a fixed number of parameters that does not depend on the number of graph convolutions ( i.e . layers ) . In effect , it is guarded against the influence of overfitting and vanishing gradient problem with more layers . This leaves us only with oversmoothing as a possible cause of performance degradation with increasing K. Interestingly , the simplicity of SGC does not seem to be a sacrifice ; it has been observed that it achieves similar or better accuracy in various relational classification tasks ( Wu et al. , 2019 ) . Dashed lines in Figure 1 illustrate the performance of SGC on the Cora dataset as we increase the number of layers ( K ) . The training ( cross-entropy ) loss monotonically increases with larger K , potentially because graph convolution mixes node representations with their neighbors ’ and makes them less distinguishable ( training becomes harder ) . On the other hand , graph convolutions ( i.e. , smoothing ) improve generalization ability , reducing the gap between training and validation/test loss 1We normalize each column j as the Laplacian smoothing stationary point πj is not scale-free . See Eq . ( 1 ) . up to K = 4 , after which ( over ) smoothing begins to hurt performance . The row-diff and col-diff both continue decreasing monotonically with K , providing supporting evidence for oversmoothing .

Summary: It is known that GNNs are vulnerable to the oversmoothing problem, in which feature vectors on nodes get closer as we increase the number of (message passing type graph convolution layers). This paper proposed PairNorm, which is a normalization layer for GNNs to tackle this problem. The idea is to pull apart feature vectors on a pair of non-adjacent nodes (based on the interpretation of Laplace-type smoothing by NT and Maehara (2019)). To achieve this approximately with low computational complexity, PairNorm keeps the sum of distances of feature vectors on all node pairs approximately the same throughout layers. The paper conducted empirical studies to evaluate the effectiveness of the method. PairNorm improved the prediction performance and enabled make GNNs deep, especially when feature vectors are missing in the large portion of nodes (the SSNC-MV problem).