Graph Neural Networks with Adaptive Residual

1 Introduction . Recent years have witnessed the great success of graph neural networks ( GNNs ) in representation learning for graph structure data [ 1 ] . Essentially , GNNs generalize deep neural networks ( DNNs ) from regular grids , such as image , video and text , to irregular data such as social , energy , transportation , citation , and biological networks . Such data can be naturally represented as graphs with nodes and edges . The key building block for such generalization is the neural message passing framework [ 2 ] : x ( k+1 ) u = UPDATE ( k ) ( x ( k ) u , m ( k ) N ( u ) ) ( 1 ) where x ( k ) u ∈ Rd denotes the feature vector of node u in the k-th iteration of message passing , and m ( k ) N ( u ) is the message aggregated from u ’ s neighborhood N ( u ) . The specific design of message passing scheme can be motivated from spectral domain [ 3 , 4 ] or spatial domain [ 5 , 6 , 7 , 2 ] . It usually linearly smooths the features in a local neighborhood on the graph . GNNs have achieved superior performance in a large number of benchmark datasets [ 8 ] where the node features are assumed to be complete and informative . However , in real-world applications , some node features could be abnormal from various aspects . For instance , in social networks , new users might not have complete profile before they make connections with others , leading to missing user features . In transportation networks , node features can be noisy since there exist certain 1The implementation is available at https : //github.com/lxiaorui/AirGNN . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . uncertainty and dynamics in the observation of the traffic information . What is worse , node features can be adversarially chosen by the attacker to maliciously manipulate the prediction made by GNNs . Therefore , it is greatly desired to design GNN models with stronger resilience to abnormal node features . In this work , we first perform empirical investigations on how representative GNN models behave on graphs with abnormal features . Specifically , based upon standard benchmark datasets , we simulate the abnormal features by replacing the features of randomly selected nodes with random Gaussian noise . Then the performance of node classification on abnormal features and normal features are examined separately . From our preliminary study in Section 2 , we reveal two interesting observations : ( 1 ) Feature aggregation can boost the resilience to abnormal features , but too many aggregations could hurt the performance on both normal and abnormal features ; and ( 2 ) Residual connection helps GNNs benefit from more layers for normal features , while making GNNs more fragile to abnormal features . We then provide possible explanations to understand these observed phenomena from the perspective of graph Laplacian smoothing . Our analyses imply that there might exist an intrinsic tension between feature aggregation and residual connection , which results in a performance tradeoff between normal features and abnormal features . Motivated by these findings and understandings , we aim to design new GNNs with stronger resilience to abnormal features while largely maintaining the performance on normal features . Our contributions can be summarized as follows : • We discover an intrinsic tension between feature aggregation and residual connection in GNNs , and the corresponding performance tradeoff between abnormal and normal features . We also analyze possible reasons to explain and understand these findings . • We propose a simple , efficient , principled and adaptive message passing scheme , which leads to a novel GNN model with adaptive residual , named as AirGNN . • Extensive experiments under various abnormal feature scenarios demonstrate the superiority of the proposed algorithm . The ablation study demonstrates how the adaptive residuals mitigate the impact of abnormal features . 2 Preliminary . Before introducing the preliminary study , we first define the notations used throughout the paper . Notations . We use bold upper-case letters such as X to denote matrices . Given a matrix X ∈ Rn×d , we use Xi to denote its i-th row and Xij to denote its element in i-th row and j-th column . The Frobenius norm and ` 21 norm of a matrix X are defined as ‖X‖F = √∑ ijX 2 ij and ‖X‖21 =∑ i ‖Xi‖2 = ∑ i √∑ jX 2 ij , respectively . We define ‖X‖2 = σmax ( X ) where σmax ( X ) is the largest singular value of X . Let G = { V , E } be a graph with the node set V = { v1 , . . . , vn } and the undirected edge set E = { e1 , . . . , em } . We use N ( vi ) to denote the neighboring nodes of node vi , including vi itself . Suppose that each node is associated with a d-dimensional feature vector , and the features for all nodes are denoted as Xfea ∈ Rn×d . The graph structure G can be represented as an adjacent matrix A ∈ Rn×n , where Aij = 1 when there exists an edge between nodes vi and vj , and Aij = 0 otherwise . The graph Laplacian matrix is defined as L = D − A , where D is the diagonal degree matrix . Let us denote the commonly used feature aggregation matrix in GNNs [ 3 ] as Ã = D̂− 1 2 ÂD̂− 1 2 where Â = A+ I is the adjacent matrix with self-loop and its degree matrix is D̂ . The corresponding Laplacian matrix is defined as L̃ = I− Ã . In this work , we focus on the setting where a subset of nodes in the graph contain abnormal features , while the remaining nodes have normal features . In the remaining of this paper , we use abnormal/normal features to denote nodes with abnormal/normal features , for simplicity . 2.1 Preliminary Study . Experimental setup . To investigate how GNNs behave on abnormal and normal node features , we design semi-supervised node classification experiments on three common datasets ( i.e. , Cora , CiteSeer and PubMed ) , following the data splits in the work [ 3 ] . Moreover , we simulate the abnormal features by assigning 10 % of the nodes with random features sampled from a standard Gaussian distribution . The experiments are performed on representative GNN models covering coupled and decoupled architectures , including GCN [ 3 ] , GCNII [ 9 ] , APPNP [ 10 ] , and their variants with or without residual connections in feature aggregations , denoted as w/Res and wo/Res . All methods follow the hyperparameter settings in their original papers . We examine how these models perform when the number of layers increases . Note that for the decoupled architectures such as APPNP , we fix the 2-layer MLP and increase the number of propagation layers . While for the coupled architectures such as GCN and GCNII , we increase the number of feature transformation and propagation layers simultaneously . We report the average performance over 10 times of random selection of the noise node sets . The node classification accuracy ( mean and standard variance ) on nodes with abnormal and normal features is illustrated in Figure 1 and Figure . 2 , separately . Observations . From Figure 1 and Figure 2 , we can make the following observations : ( 1 ) Without residual connection , more layers ( e.g. , > 2 for GCN and GCNII , > 10 for APPNP ) hurt the accuracy on nodes with normal features . However , more layers boost the accuracy on nodes with abnormal features significantly , before finally starting to decrease ; ( 2 ) With residual connection , the accuracy on nodes with normal features keeps increasing with more layers2 . However , the accuracy on nodes with abnormal features only increases marginally when stacking more layers , and then starts to decrease . While we only present the experiments on Cora , we defer the results on other datasets to Appendix C , which provide similar observations . To conclude , we can summarize these observations into two major findings : • Finding I : Feature aggregation can boost the resilience to abnormal features , but too many aggregations could hurt the performance on both normal and abnormal nodes ; • Finding II : Residual connection helps GNNs benefit from more layers for nodes with normal features , while making GNNs more fragile to abnormal features . 2GCN w/Res is an exception because its residual is not appropriate , which is consistent with the experiments in the work [ 3 ] . 2.2 Understandings . In this subsection , we provide the understanding and explanation for aforementioned findings , from the perspective of graph Laplacian smoothing . Understanding Finding I : Feature aggregation as Laplacian smoothing The message passing in GCN [ 3 ] , GCNII wo/ residual and APPNP wo/ residual ( as well as many popular GNN models ) , follows the feature aggregation Xout = ÃXin , ( 2 ) where Xin and Xout represent the features before and after message passing layer , respectively . It can be interpreted as one gradient descent step for the Laplacian smoothing problem [ 11 ] argmin X∈Rn×d L1 ( X ) : = 1 2 tr ( X > ( I− Ã ) X ) = 1 2 ∑ ( vi , vj ) ∈E ‖ Xi√ di + 1 − Xj√ dj + 1 ‖22 , ( 3 ) where di is the node degree of node vi . Eq . ( 2 ) can be derived from Xout = Xin− ( I−Ã ) Xin = ÃXin , with the initialization X = Xin and stepsize γ = 1 . The Laplacian smoothing problem penalizes the feature difference between neighboring nodes . To reduce this penalty , the feature aggregation in Eq . ( 2 ) smooths the node features by taking the average of local neighbors , and thus can be considered as low-pass filter which gradually filters out high-frequency signals [ 12 , 13 ] . Therefore , it increases the resilience to abnormal features which are likely to be high-frequency signals . In other words , the local neighboring nodes help to correct the abnormal features . Unfortunately , if applied too many times , these low-pass filters could overly smooth the features ( well-known as oversmoothing [ 14 , 15 ] ) such that nodes are not distinguishable enough , providing an explanation to the degraded performance on both abnormal and normal features when stacking too many layers . Understanding Finding II : Residual connection maintains feature proximity To adjust the feature smoothness for better performance , APPNP [ 10 ] utilizes residual connections in message passing as follows Xk+1 = ( 1− α ) ÃXk + αXin , ( 4 ) where X0 = Xin . It can be considered as an iterative solution for the regularized Laplacian smoothing problem [ 11 ] argmin X∈Rn×d L2 ( X ) : = α 2 ( 1− α ) ‖X−Xin‖2F + 1 2 tr ( X > ( I− Ã ) X ) , ( 5 ) with initialization X = Xin and stepsize γ = 1− α due to Xk+1 = Xk − ( 1− α ) ( α 1− α ( Xk −Xin ) + ( I− Ã ) Xk ) = ( 1− α ) ÃXk + αXin . GCNII [ 9 ] adopts a similar message passing but further combines a feature transformation layer in each message passing step , which leads to a coupled architecture , as contrast to the decoupled architecture of APPNP . The residual connection naturally arises when regularizing the proximity between input and output features , as showed in the first term of L2 ( X ) . Such proximity can help avoid the trivial solution for the problem in Eq . ( 3 ) , i.e. , totally oversmoothed features only depending on node degrees , and consequently mitigates the oversmoothing issue . More intuitively , residual connections in GNNs provide direct information flows between layers that can preserve some necessary high-frequency signals for better discrimination between classes . More layers with residual provide a more accurate solution to Eq . ( 5 ) , which explains the performance gain from deeper GNNs . Unfortunately , these residual connections also undesirably carry on abnormal features which are detrimental , leading to the inferior performance on abnormal features .

Summary:  This paper observed interesting phenomenons: feature aggregation helps smooth out abnormal features but would lead to over-smoothing for normal features, residual link helps adjust feature smoothness for normal features but may hurt the performance when we have abnormal features. Therefore, it points out that we need to tradeoff between normal and abnormal features while designing GNNs. Then, based on the observation, this work proposed the AMP method and the AirGNN model, which are shown by experiments to be effective under various abnormal feature scenarios.