Tackling Oversmoothing of GNNs with Contrastive Learning

1 INTRODUCTION . Combining the graph data comprehensive relations with the neural network models representation learning ability , graph neural networks ( GNNs ) achieve state-of-the-art performances in many realworld applications , such as document classification , natural language processing , computer vision , and recommender systems ( Zhang et al. , 2019 ) . GNNs consist of many variant neural network models with different message-passing mechanisms , to name a few , such as GCN ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Velickovic et al. , 2018 ) , GIN ( Xu et al. , 2019 ) , and GMNN ( Qu et al. , 2019 ) . In the complex real-world settings of applying GNNs , not every node is lucky enough to have node labels and/or node features . Hence , increasing the depth ( i.e. , the number of layers ) of GNNs is a viable solution to capture more latent knowledge to reduce the uncertainty caused by missing values ( Zhao & Akoglu , 2020 ) . However , as the number of layers increases , the performance of GNN will decrease to a large degree ( Kipf & Welling , 2017 ) . The reasons may come from many aspects after involving more parameters like vanishing gradient , overfitting , and oversmoothing . Compared with the first two reasons , oversmoothing of GNNs is recently introduced ( Li et al. , 2018 ; Oono & Suzuki , 2020 ) and widely discussed ( Chen et al. , 2020a ; Zhao & Akoglu , 2020 ; Rong et al. , 2020 ; Chen et al. , 2020b ; Liu et al. , 2020 ; Zhou et al. , 2020 ) . It is the phenomenon that the learned node representations become indistinguishable as the number of the hidden layers increases , thus hurting the performance of down-streaming tasks like node classification and link prediction . To tackle the oversmoothing problem of GNNs , some nascent research works are proposed ( Klicpera et al. , 2019 ; Chen et al. , 2020a ; Zhao & Akoglu , 2020 ; Rong et al. , 2020 ; Chen et al. , 2020b ; Liu et al. , 2020 ; Zhou et al. , 2020 ) . They share the same logic ( i.e. , keeping the divergence between nodes ) but differ in specific methodologies ( i.e. , rescaling divergences of learned representa- tions ( Zhao & Akoglu , 2020 ) , adding the divergence regularizer in the learning process ( Chen et al. , 2020a ; Zhou et al. , 2020 ) , changing input graph structures ( Chen et al. , 2020a ; Rong et al. , 2020 ; Chen et al. , 2020b ) , or personalizing the information aggregation for each specific node ( Klicpera et al. , 2019 ; Liu et al. , 2020 ) ) . Despite of good performance , some drawbacks still exist in those mentioned solutions . By surveying these SOTA de-oversmoothing strategies , we summarize three major metrics to evaluate a de-oversmoothing strategy : 1 ) constant divergence indicator , 2 ) easyto-determine divergence indicator , and 3 ) model-agnostic de-oversmoothing strategy . ( The detailed discussion could be found in Section 2 ) . We find that no prevalent de-oversmoothing methods for GNNs could maintain all of them . To bridge this gap , we propose a Topology-guided Graph Contrastive Layer ( TGCL ) with the inspiration from the contrastive learning concept ( van den Oord et al. , 2018 ) , where we contrast node topological information to obtain discriminative node representations after many GNN layers . TGCL is the first de-oversmoothing strategy attempting to maintain all three mentioned metrics . Specifically , we set a constant and easy-to-determine divergence indicator between nodes , which is purely based on the topology of the input graph . With this divergence indicator , we aim to guide latent representations of neighbor node pairs closer and non-neighbor node pairs farther apart to mitigate the oversmoothing of GNNs . Last but not least , the proposed TGCL is model-agnostic , which means TGCL could be incorporated in multiple GNN models . With theoretical proof and empirical analysis , we show that the proposed TGCL could alleviate the oversmoothing problem of GNNs to a large extent . Our contributions can be summarized as follows : • We survey current de-oversmoothing methods by analyzing the advantages and the disadvantages of each method and summarize three metrics to evaluate a de-oversmoothing method for GNNs . • We propose a topology-guided graph contrastive layer named TGCL to tackle the oversmoothing problem of GNNs , which enjoys all three metrics simultaneously . • We show the effectiveness of the proposed TGCL from the theoretical proof and the empirical aspect with extensive experiments . The rest of this paper is organized as follows . After a brief survey of de-oversmoothing methods in Section 2 , we introduce the proposed TGCL in Section 3 . The empirical evaluation of the proposed TGCL on real-world datasets is presented in Section 4 . Then , we review the related work in Section 5 before we conclude the paper in Section 6 . 2 BACKGROUND . As mentioned above , de-oversmoothing methods ( Klicpera et al. , 2019 ; Chen et al. , 2020a ; Zhao & Akoglu , 2020 ; Rong et al. , 2020 ; Chen et al. , 2020b ; Liu et al. , 2020 ; Zhou et al. , 2020 ) share the same logic of keeping the divergence between node representations but differ in specific methodologies focusing on different merits . By taking the union of the metrics used in different state-of-thearts , we get three metrics to evaluate a de-oversmoothing algorithm comprehensively . There are three metrics as shown in Table 1 , including constant divergence indicator , easy-todetermine divergence indicator , and model-agnostic de-oversmoothing strategy . Divergence indicator is indispensable for guiding the final node representation similarity based on the specified distance measurement . Several de-oversmoothing methods like ( Klicpera et al. , 2019 ; Zhao & Akoglu , 2020 ; Chen et al. , 2020b ; Liu et al. , 2020 ) achieve the constant divergence indicator , which means the guidance is much more robust and not dependent on the training process of GNNs . However , to guide the node representation similarity reasonably , the divergence indicator is not that easy to be determined . For example , PairNorm ( Zhao & Akoglu , 2020 ) is proposed as a normalization layer to keep the divergence of node representation against the original node feature . Instead of adding this regularizer directly to the learning objective of GNN models , PairNorm takes an alternative by rescaling the learned node representations with a constant hyperparameter to keep the original node feature divergence . PairNorm achieves two metrics : constant divergence indicator ( i.e. , the constant hyperparameter ) and model-agnostic strategy ( i.e. , PairNorm can be added on different GNN models as a layer ) . However , the selection of that constant hyperparameter heavily depends on the prior knowledge of the input graph data , which is hard to determine . ( The discussion of other de-oversmoothing methods can be found in Section 5 . ) As shown in Table 1 , PairNorm is an effective de-oversmoothing method that maintains two metrics but needs prior knowledge to scale divergence between node pairs . While our proposed TGCL transfers this hard-to-acquire prior knowledge into the topology information of the input graph , where the divergence guidance between nodes is constant and easy to be determined . To be specific , our TGCL is the first de-oversmoothing method attempting to maintain these three metrics at the same time . In the next section , we formally introduce the proposed TGCL with theoretical proof for the model effectiveness . Moreover , we prove that the objective of PairNorm is just a special case of our TGCL , which shows the effectiveness of our TGCL from another perspective . 3 PROPOSED METHOD . In this section , we begin with the notions used in this paper . Then , we prove that the objective of the de-oversmoothing model PairNorm ( Zhao & Akoglu , 2020 ) is a just special case of our Topologyguided Graph Contrastive Layer ( TGCL ) . After analyzing the limitations of PairNorm , we formally introduce our proposed TGCL and show why it could better alleviate the oversmoothing issue with the contrastive learning manner . 3.1 NOTION . Throughout this paper , we use regular letters to denote scalars ( e.g. , α ) , boldface lowercase letters to denote vectors ( e.g. , v ) , and boldface uppercase letters to denote matrices ( e.g. , A ) . We formalize the graph mining problem in the context of undirected graph G = ( V , E , X ) , where V consists of n vertices , E consists of m edges , X ∈ Rn×d denote the feature matrix and d is the feature dimension . We let A ∈ Rn×n denote the adjacency matrix , D ∈ Rn×n denote the diagonal matrix of vertex degrees , and I ∈ Rn×n denote the identity matrix . For ease explanation , we denote vi as node i , xi as the input feature of node i , zi as the embedding of node i by any type of GNNs and Ai as the adjacency vector for node i. Ni is a set that contains the neighbors of node i and N̄i is the complement of Ni , which contains the non-neighbor of node i . 3.2 PRELIMINARY . Each graph convolutional layer can be understood as a smoothing operation ( Li et al. , 2018 ) but stacking many layers renders the final representation of a node indistinguishable from others . Therefore , how to recover the divergence between node representations but preserving the shared information becomes a vital problem in graph mining . In PairNorm ( Zhao & Akoglu , 2020 ) , the divergence between node pairs is based on a hyper-parameter , which requires prior knowledge of the input graph data and is hard to acquire . More specifically , PairNorm is proposed as a novel normalization layer to prevent all node embeddings from becoming too similar by minimizing the following objective : Lp = ∑ vi∈V ‖zi − xi‖2 + ∑ ( i , j ) ∈E ‖zi − zj‖2 − ∑ ( i , k ) 6∈E ‖zi − zk‖2 ( 1 ) where zi is the node embedding vector of node vi and xi is the original feature vector of node vi . In the equation above , the first term is the reconstruction error , the second term is responsible for min- imizing the difference between two representations of a neighbor node pair , and the last term aims to maximize the difference between two representations of a remote node pair . By reformulating Eq . 1 , we could derive a upper bound of Lp in the form of contrastive learning loss term as follows : Lp = ∑ vi∈V ‖zi − xi‖2 + ∑ vi∈V ∑ vj∈Ni ‖zi − zj‖2 − ∑ vi∼V ∑ vk /∈Ni ‖zi − zk‖2 = ∑ vi∼V ‖zi − xi‖2 − ∑ vi∼V ∑ vj∈Ni log ( e−‖zi−zj‖ 2 ) + ∑ vi∼V ∑ vk /∈Ni log ( e−‖zi−zk‖ 2 ) ( 2 ) ≤ ∑ vi∼V ‖zi − xi‖2 − ∑ vi∼V ∑ vj∈Ni log ( e−‖zi−zj‖ 2 ) + ∑ vi∼V log ( ∑ vk /∈Ni e−‖zi−zk‖ 2 ) ( 3 ) ≤ ∑ vi∼V ‖zi − xi‖2 − ∑ vi∼V ∑ vj∈Ni log ( e−‖zi−zj‖ 2 ) + ∑ vi∼V ∑ vj∈Ni log ( ∑ vk /∈Ni e−‖zi−zk‖ 2 ) = ∑ vi∼V ‖zi − xi‖2 + ∑ vi∼V ∑ vj∈Ni log ( ∑ vk /∈Ni e −‖zi−zk‖2 e−‖zi−zj‖ 2 ) ≤ ∑ vi∼V ‖zi − xi‖2 + ∑ vi∼V ∑ vj∈Ni [ log ( 1 + ∑ vk /∈Ni e −‖zi−zk‖2 e−‖zi−zj‖ 2 ) ] = ∑ vi∼V ‖zi − xi‖2 − ∑ vi∼V ∑ vj∈Ni [ log ( e−‖zi−zj‖ 2 e−‖zi−zj‖ 2 + ∑ vk /∈Ni e −‖zi−zk‖2 ) ] = ∑ vi∼V ‖zi − xi‖2 − ∑ vi∼V ∑ vj∈Ni [ log ( f ( zi , zj ) f ( zi , zj ) + ∑ vk /∈Ni f ( zi , zk ) ) ] = L1 ( 4 ) where f ( zi , zk ) = e−‖zi−zk‖ 2 . Here , we apply Jensen ’ s inequality to derive Eq . 3 as a upper bound of Eq.2 since log ( · ) is concave . We observe that L1 is a upper bound of PairNorm and we could interpret two regularization terms ‖zi − zj‖2 and ‖zi − zk‖2 of PairNorm as a special case of a contrastive learning loss term in L1 by setting the similarity measurement function f ( zi , zk ) to be e−‖zi−zk‖ 2 . However , both PairNorm ( Eq . 1 ) and the upper bound of PairNorm ( Eq . 4 ) only consider the firstorder neighbor information but neglect the K-hop neighbors information . For example , in a realworld scenario , we are given a remote pair ( vk , vi ) . It is highly possible that vk and vi have the similar representations , if they share the same label information . However , simply minimizing the third term of PairNorm ( i.e. , −‖zi − zk‖2 ) will push zi away from zk , resulting in sub-optimal solution . In addition , if we are given two remote pairs ( vk1 , vi ) and ( vk2 , vi ) such that node vk1 is far from node vi and node vk2 is near node vi ( e.g. , 2-hop neighbor ) , the weight imposed on these two remote pairs should be different as we expect that zk1 should be more different from zi than zk2 due to the topological information in the graph . However , PairNorm and L1 ( Eq . 4 ) assume that all unconnected node pairs ( zi and zk ) have the same weight by setting the weights to be 1 for neighbor pairs and remote pairs . Therefore , if the K-hop neighbors of zi share the same topological structure of zi or the same label information , pushing zi away from the representation of its K-hop neighbors ( K > 1 ) and ignoring the different weights for different remote pairs will result in a sub-optimal solution . Motivated by these , we propose to utilize the similarity of two adjacency vectors of each node pair and embed the global topological structure information into the representation of each node such that GNNs can derive better discriminative representations for all nodes .

Summary: This paper proposes a method named TGCL which aims to deal with the over smoothing problem in GNN when a number of layers go deeper. Different from existing works, the proposed TGCL considers three different concepts which are Constant Divergence Indicator, Easy-to-Determine Divergence Indicator, and Model-Agnostic Strategy. Most of the existing works are only considered two of them. The proposed method can be recognized as an improvement of the PairNorm.