PairNorm: Tackling Oversmoothing in GNNs

The performance of graph neural nets ( GNNs ) is known to gradually decrease with increasing number of layers . This decay is partly attributed to oversmoothing , where repeated graph convolutions eventually make node embeddings indistinguishable . We take a closer look at two different interpretations , aiming to quantify oversmoothing . Our main contribution is PAIRNORM , a novel normalization layer that is based on a careful analysis of the graph convolution operator , which prevents all node embeddings from becoming too similar . What is more , PAIRNORM is fast , easy to implement without any change to network architecture nor any additional parameters , and is broadly applicable to any GNN . Experiments on real-world graphs demonstrate that PAIRNORM makes deeper GCN , GAT , and SGC models more robust against oversmoothing , and significantly boosts performance for a new problem setting that benefits from deeper GNNs . Code is available at https : //github.com/LingxiaoShawn/PairNorm . 1 INTRODUCTION . Graph neural networks ( GNNs ) is a family of neural networks that can learn from graph structured data . Starting with the success of GCN ( Kipf & Welling , 2017 ) on achieving state-of-the-art performance on semi-supervised classification , several variants of GNNs have been developed for this task ; including GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Velickovic et al. , 2018 ) , SGC ( Wu et al. , 2019 ) , and GMNN ( Qu et al. , 2019 ) to name a few most recent ones . A key issue with GNNs is their depth limitations . It has been observed that deeply stacking the layers often results in significant drops in performance for GNNs , such as GCN and GAT , even beyond just a few ( 2–4 ) layers . This drop is associated with a number of factors ; including the vanishing gradients in back-propagation , overfitting due to the increasing number of parameters , as well as the phenomenon called oversmoothing . Li et al . ( 2018 ) was the first to call attention to the oversmoothing problem . Having shown that the graph convolution is a type of Laplacian smoothing , they proved that after repeatedly applying Laplacian smoothing many times , the features of the nodes in the ( connected ) graph would converge to similar values—the issue coined as “ oversmoothing ” . In effect , oversmoothing hurts classification performance by causing the node representations to be indistinguishable across different classes . Later , several others have alluded to the same problem ( Xu et al. , 2018 ; Klicpera et al. , 2019 ; Rong et al. , 2019 ; Li et al. , 2019 ) ( See §5 Related Work ) . In this work , we address the oversmoothing problem in deep GNNs . Specifically , we propose ( to the best of our knowledge ) the first normalization layer for GNNs that is applied in-between intermediate layers during training . Our normalization has the effect of preventing the output features of distant nodes to be too similar or indistinguishable , while at the same time allowing those of connected nodes in the same cluster become more similar . We summarize our main contributions as follows . • Normalization to Tackle Oversmoothing in GNNs : We introduce a normalization scheme , called PAIRNORM , that makes GNNs significantly more robust to oversmoothing and as a result enables the training of deeper models without sacrificing performance . Our proposed scheme capitalizes on the understanding that most GNNs perform a special form of Laplacian smoothing , which makes node features more similar to one another . The key idea is to ensure that the total pairwise feature distances remains a constant across layers , which in turn leads to distant pairs having less similar features , preventing feature mixing across clusters . • Speed and Generality : PAIRNORM is very straightforward to implement and introduces no additional parameters . It is simply applied to the output features of each layer ( except the last one ) consisting of simple operations , in particular centering and scaling , that are linear in the input size . Being a simple normalization step between layers , PAIRNORM is not specific to any particular GNN but rather applies broadly . • Use Case for Deeper GNNs : While PAIRNORM prevents performance from dropping significantly with increasing number of layers , it does not necessarily yield increased performance in absolute terms . We find that this is because shallow architectures with no more than 2–4 layers is sufficient for the often-used benchmark datasets in the literature . In response , we motivate a real-world scenario wherein a notable portion of the nodes have no feature vectors . In such settings , nodes benefit from a larger range ( i.e. , neighborhood , hence a deeper GNN ) to “ recover ” effective feature representations . Through extensive experiments , we show that GNNs employing our PAIRNORM significantly outperform the ‘ vanilla ’ GNNs when deeper models are beneficial to the classification task . 2 UNDERSTANDING OVERSMOOTHING . In this work , we consider the semi-supervised node classification ( SSNC ) problem on a graph . In the general setting , a graph G = ( V , E , X ) is given in which each node i ∈ V is associated with a feature vector xi ∈ Rd where X = [ x1 , . . . , xn ] T denotes the feature matrix , and a subset Vl ⊂ V of the nodes are labeled , i.e . yi ∈ { 1 , . . . , c } for each i ∈ Vl where c is the number of classes . Let A ∈ Rn×n be the adjacency matrix and D = diag ( deg1 , . . . , degn ) ∈ Rn×n be the degree matrix of G. Let Ã = A + I and D̃ = D + I denote the augmented adjacency and degree matrices with added self-loops on all nodes , respectively . Let Ãsym = D̃−1/2ÃD̃−1/2 and Ãrw = D̃−1Ã denote symmetrically and nonsymmetrically normalized adjacency matrices with self-loops . The task is to learn a hypothesis that predicts yi from xi that generalizes to the unlabeled nodes Vu = V\Vl . In Section 3.2 , we introduce a variant of this setting where only a subset F ⊂ V of the nodes have feature vectors and the rest are missing . 2.1 THE OVERSMOOTHING PROBLEM . Although GNNs like GCN and GAT achieve state-of-the-art results in a variety of graph-based tasks , these models are not very well-understood , especially why they work for the SSNC problem where only a small amount of training data is available . The success appears to be limited to shallow GNNs , where the performance gradually decreases with the increasing number of layers . This decrease is often attributed to three contributing factors : ( 1 ) overfitting due to increasing number of parameters , ( 2 ) difficulty of training due to vanishing gradients , and ( 3 ) oversmoothing due to many graph convolutions . Among these , perhaps the least understood one is oversmoothing , which indeed lacks a formal definition . In their analysis of GCN ’ s working mechanism , Li et al . ( 2018 ) showed that the graph convolution of GCN is a special form of Laplacian smoothing . The standard form being ( I−γI ) X+ γÃrwX , the graph convolution lets γ = 1 and uses the symmetrically normalized Laplacian to obtain X̃ = ÃsymX , where the new features x̃ of a node is the weighted average of its own and its neighbors ’ features . This smoothing allows the node representations within the same cluster become more similar , and in turn helps improve SSNC performance under the cluster assumption ( Chapelle et al. , 2006 ) . However when GCN goes deep , the performance can suffer from oversmoothing where node representations from different clusters become mixed up . Let us refer to this issue of node representations becoming too similar as node-wise oversmoothing . Another way of thinking about oversmoothing is as follows . Repeatedly applying Laplacian smoothing too many times would drive node features to a stationary point , washing away all the information from these features . Let x·j ∈ Rn denote the j-th column of X . Then , for any x·j ∈ Rn : lim k→∞ Ãksymx·j = πj and πj ‖πj‖1 = π , ( 1 ) where the normalized solution π ∈ Rn satisfies πi = √ degi∑ i √ degi for all i ∈ [ n ] . Notice that π is independent of the values x·j of the input feature and is only a function of the graph structure ( i.e. , degree ) . In other words , ( Laplacian ) oversmoothing washes away the signal from all the features , making them indistinguishable . We will refer to this viewpoint as feature-wise oversmoothing . To this end we propose two measures , row-diff and col-diff , to quantify these two types of oversmoothing . Let H ( k ) ∈ Rn×d be the representation matrix after k graph convolutions , i.e . H ( k ) = ÃksymX . Let h ( k ) i ∈ Rd be the i-th row of H ( k ) and h ( k ) ·i ∈ Rn be the i-th column of H ( k ) . Then we define row-diff ( H ( k ) ) and col-diff ( H ( k ) ) as follows . row-diff ( H ( k ) ) = 1 n2 ∑ i , j∈ [ n ] ∥∥∥h ( k ) i − h ( k ) j ∥∥∥ 2 ( 2 ) col-diff ( H ( k ) ) = 1 d2 ∑ i , j∈ [ d ] ∥∥∥h ( k ) ·i /‖h ( k ) ·i ‖1 − h ( k ) ·j /‖h ( k ) ·j ‖1∥∥∥ 2 ( 3 ) The row-diff measure is the average of all pairwise distances between the node features ( i.e. , rows of the representation matrix ) and quantifies node-wise oversmoothing , whereas col-diff is the average of pairwise distances between ( L1-normalized1 ) columns of the representation matrix and quantifies feature-wise oversmoothing . 2.2 STUDYING OVERSMOOTHING WITH SGC . Although oversmoothing can be a cause of performance drop with increasing number of layers in GCN , adding more layers also leads to more parameters ( due to learned linear projections W ( k ) at each layer k ) which magnify the potential of overfitting . Furthermore , deeper models also make the training harder as backpropagation suffers from vanishing gradients . In order to decouple the effect of oversmoothing from these other two factors , we study the oversmoothing problem using the SGC model ( Wu et al. , 2019 ) . ( Results on other GNNs are presented in §4 . ) SGC is simplified from GCN by removing all projection parameters of graph convolution layers and all nonlinear activations between layers . The estimation of SGC is simply written as : Ŷ = softmax ( ÃKsym X W ) ( 4 ) where K is the number of graph convolutions , and W ∈ Rd×c denote the learnable parameters of a logistic regression classifier . Note that SGC has a fixed number of parameters that does not depend on the number of graph convolutions ( i.e . layers ) . In effect , it is guarded against the influence of overfitting and vanishing gradient problem with more layers . This leaves us only with oversmoothing as a possible cause of performance degradation with increasing K. Interestingly , the simplicity of SGC does not seem to be a sacrifice ; it has been observed that it achieves similar or better accuracy in various relational classification tasks ( Wu et al. , 2019 ) . Dashed lines in Figure 1 illustrate the performance of SGC on the Cora dataset as we increase the number of layers ( K ) . The training ( cross-entropy ) loss monotonically increases with larger K , potentially because graph convolution mixes node representations with their neighbors ’ and makes them less distinguishable ( training becomes harder ) . On the other hand , graph convolutions ( i.e. , smoothing ) improve generalization ability , reducing the gap between training and validation/test loss 1We normalize each column j as the Laplacian smoothing stationary point πj is not scale-free . See Eq . ( 1 ) . up to K = 4 , after which ( over ) smoothing begins to hurt performance . The row-diff and col-diff both continue decreasing monotonically with K , providing supporting evidence for oversmoothing .

Summary: The article "PairNorm: Tackling Oversmoothing in GNNs" considers the interesting phenomenon of performance degradation of graph neural network when the depth of the network increases beyond the values of 2-4. The authors argue that one of the reasons for such behavior is so-called "oversmoothing", when intermediate representations become similar for all the nodes in the graph. The authors propose the special NN layer "PairNorm", which aims to battle with this issue.