Resisting Over-Smoothing in Graph Neural Networks via Dual-Dimensional Decoupling

Authors: mang  ye
wenke  huang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:16:24

Abstract: Graph Neural Networks (GNNs) are widely employed to derive meaningful node representations from graphs. Despite their success, deep GNNs frequently grapple with the oversmoothing issue, where node representations become highly indistinguishable due to repeated aggregations. In this work, we consider the oversmoothing issue from two aspects of the node embedding space: dimension and instance. Specifically, while existing methods primarily concentrate on instance-level node relations to mitigate oversmoothing, we propose to mitigate oversmoothing at dimension level. We reveal the heightened information redundancy between dimensions which diminishes information diversity and impairs node differentiation in GNNs. Motivated by this insight, we propose Dimension-Level Decoupling (DLD) to reduce dimension redundancy, enhancing dimensional-level node differentiation. Besides, at the instance level, the neglect of class differences leads to vague classification boundaries. Hence, we introduce Instance-Level Class-Difference Decoupling (ICDD) that repels inter-class nodes and attracts intra-class nodes, improving the instance-level node discrimination with clear classification boundaries. Additionally, we introduce a novel evaluation metric that considers the impact of class differences on node distances, facilitating precise oversmoothing measurement. Extensive experiments demonstrate the effectiveness of our method Dual-Dimensional Class-Difference Decoupling (DDCD) across diverse scenarios.

Main paper: 
 oversmoothing: 
 Oversmoothing [2, 30, 38, 46, 66] is first defined by Li et al. [38] , where the node representations become highly indistinguishable in deep GNNs. Li et al. [38] attributed this issue to the multiple Laplacian smoothing steps and suggested that oversmoothing was a common phenomenon in GNNs as the Laplace operator is a ubiquitous component, hindering the superposition of GNN layers. Recent research [2, 30, 62, 67] has explored oversmoothing in-depth, emphasizing its significance in developing deep GNNs to enhance node representations [7, 11, 77, 80] . Several works alleviate oversmoothing from the perspective of model architecture designs [18, 34, 35, 44, 80] . However, the designs on model architecture are inflexible and introduce additional network parameters, hindering their applications. Some approaches involve dropping nodes or edges to avoid redundant aggregations [15, 42, 50] , but inadvertently disrupting the node and structural information of the graph, leading to the loss of original graph information and performance degradation. Other methods address oversmoothing through layer-wise normalization [27, 78, 79] , but the ignorance of class differences results in vague class boundaries and ultimately hinders performance. In this work, we present a flexible regularization method addressing oversmoothing with class differences.Dimensional Information Redundancy [41, 74] indicates a high degree of redundant information encoded across these dimensions. It restricts the diversity of dimension information, thereby impeding the representation capability of feature embeddings. This phenomenon is prevalent and has been extensively investigated in self-supervised learning [1, 25, 28, 37, 39, 75] . In this work, we first propose to alleviate oversmoothing at the dimension level. We reveal the heightened dimension information redundancy in GNNs, which inspires us to enhance dimension diversity for node differentiation by reducing the redundancy between dimensions.Contrastive Learning is widely applied [5, 70, 72] , which maximizes the agreement between positive samples while minimizing the agreement between negative samples. It utilizes augmentations to construct contrastive samples and is often applied in self-supervised learning for label-scarce scenes [4, 9, 20, 63, 73] or for pre-training networks [8, 10, 31, 47, 56, 70] to boost downstream tasks. In this work, we utilize class differences to construct contrastive samples, suppressing oversmoothing with clear class boundaries. 
methodology 3.1 prelimilaries: 
 We adopted the semi-supervised node classification (SSNC) setting proposed by Kipf and Welling [32] for our experiments. Specifically, in an undirected graph G = (V, E, X), each node 𝑣 𝑖 in V is associated with a feature vector 𝑥 𝑖 ∈ R 𝑑 , where X = [𝑥 1 , ..., 𝑥 𝑛 ] represents the initial feature matrix. E ⊆ V × V denotes the edge set between nodes. The subset V 𝑠 ⊆ V of nodes are labeled with 𝑦 𝑖 ∈ {0, ..., 𝑐 -1} for 𝑣 𝑖 ∈ V 𝑠 , where 𝑐 is the number of classes. The adjacency matrix A ∈ {0, 1} 𝑁 ×𝑁 encodes the relations between nodes, where A 𝑖 𝑗 = 1 indicates edge connections between node 𝑣 𝑖 and 𝑣 𝑗 , otherwise A 𝑖 𝑗 = 0.The task of SSNC is to predict the label of 𝑥 𝑖 ∉ V 𝑠 using the labeled nodes and graph structure. In GNNs, multiple network layers are utilized for propagation and transformation, with a linear layer for predicting the labels of nodes. Denoting the transformation and propagation as F , P, the forward process of multi-layer GNNs can be formulated as:H (𝑙 ) = F (P (H (𝑙 -1) , A)).(1) The number of GNN layers is 𝑛, and H (𝑙 ) represents the output of 𝑙-th GNN layer where 𝑙 ∈ {1, ...𝑛}.When deepening GNNs by repeatedly aggregating neighbor node representations, the node representations tend to become highly indistinguishable, i.e., the oversmoothing issue. A robust evaluation mechanism is critical for investigating this issue. Existing works [6, 23] measure the degree of oversmoothing based on the distance between node pairs. However, the pair-wise distances are closely associated with graph information (e.g., the class labels). For the node classification task, nodes in different classes should exhibit distinct pair-wise distances, while nodes in the same class should have closer pair-wise distances. Without the consideration of class information, the pair-wise distances are affected by class labels and fail to accurately assess oversmoothing. In this case, we propose a new metric: Relative Inter-Class Distance, which considers the impacts of class differences to accurately measure oversmoothing.Relative Inter-Class Distance (RICD). Denote the class set in the graph is M. For class 𝑎 ∈ M, the node set is denoted by N 𝑎 , and the node feature vector in N 𝑎 is denoted as 𝑛 𝑎 . Denote | • | is the set cardinality. The cosine similarity of node feature vectors 𝑛 𝑎 and 𝑛 𝑏 is 𝑐𝑜𝑠 (𝑛 𝑎 , 𝑛 𝑏 ). Firstly, we calculate the mean inter-class similarity as 𝑆 𝑃 :𝑆 𝑃 = 2 |M| (|M| -1) 𝑎≠𝑏 ∑︁ 𝑎,𝑏 ∈ M 1 |N 𝑎 | |N 𝑏 | 𝑛 𝑏 ∈ N 𝑏 ∑︁ 𝑛 𝑎 ∈ N 𝑎 𝑐𝑜𝑠 (𝑛 𝑎 , 𝑛 𝑏 ). (2)The mean intra-class similarity is formulated by:EQUATIONThe formulation of RICD (𝐷 𝐶 ) is as follows:EQUATIONThe high value of 𝐷 𝐶 means differentiated nodes and a low degree of oversmoothing. The proposed metric takes class differences into consideration, focusing on the relative discrepancy of interclass nodes compared to intra-class ones, accurately measuring the degree of oversmoothing.In the following two sections, we elaborate on the proposed framework: Dual-Dimensional Class-Difference Decoupling (DDCD). It is divided into two components: (1) Dimension-Level Decoupling (DLD in Section 3.3) minimizes covariances between dimensions, reducing the dimensional information redundancy to enhance node differentiation and alleviate oversmoothing. (2) Instance-Level Class-Difference Decoupling (ICDD in Section 3.4), which adaptively attracts or repels intra/inter-class nodes, alleviating oversmoothing while establishing clear class boundaries.Motivation. Existing methods primarily concentrate on instancelevel cues related to node relations for mitigating oversmoothing. In the context of self-supervised learning, the dimensional relations have been proven significant in enhancing the representation capacity of the embedding space in prior works [25, 28, 75] . Likewise, we posit that dimensional relations can also enhance the node representation in GNNs to alleviate oversmoothing. Furthermore, in deep GNNs, we observe that the multiple GNN layers lead to a substantially increased information redundancy at the dimensional level, resulting in limited node differentiation and aggravated oversmoothing. Specifically, denote C as the covariance matrix, which indicates the dimension redundancy of the embedding z:EQUATION)where 𝑖 and 𝑗 index the dimensions of z, and z 𝑖 represents the mean of z 𝑖 . As depicted in Figure 3 , we compute the mean redundancy across all dimensions of final embeddings. It tends to increase with deeper GNN layers, suggesting a high degree of dimension redundancy. It implies that less information is encoded within the learned dimensions, consequently limiting the representation capacity of node embeddings and the differentiating ability of nodes.The aforementioned insights motivate us to alleviate oversmoothing in deep GNNs by reducing the dimensional information redundancy, thereby boosting the diversity dimension information and enhancing the node differentiation ability. To extenuate dimensional information redundancy for well-differentiated nodes, we introduce a layer-wise decoupling module, DLD (Dimension-Level Decoupling). Concretely, in a 𝐾-layer GNN, assume H (𝑙 ) , 𝑙 ∈ 𝐾 is the 𝑙-th layer output, H (𝑙 ) 𝑖 represents the i-th row of H (𝑙 ) . We standardize the covariance matrix of the node representation matrix H (𝑙 ) and the objective of DLD is defined as follows:EQUATIONwhere 𝑖, 𝑗 indexes dimensions of node embedding. C (𝑙 ) is the autocovariance matrix of H (𝑙 ) , which is formulated as: :,𝑖 . To obtain well-differentiated final embeddings, we apply ℓ 𝐷𝑖𝑚 in each layer, and the loss function L 𝐷𝑖𝑚 can be defined as follows:EQUATIONEQUATIONMinimizing L 𝐷𝑖𝑚 encourages the dimensions independent in each layer, diversifying the dimension information, and ultimately improving the node differentiation for alleviating oversmoothing.Motivation. At the instance level, existing methods alleviate oversmoothing without taking class differences into account, which results in vague classification boundaries. For example, Zhou et al. [79] introduced Differentiable Group Normalization (DGN), a normalization method that assigns group probabilities using a learnable linear layer and normalizes node representations based on group relations. However, nodes from different classes can be grouped together, leading to fuzzy class boundaries and degraded performance. To effectively leverage class differences and establish distinct classification boundaries for alleviating oversmoothing, an ideal approach should encourage inter-class nodes to diverge while preserving the similarity of intra-class nodes. Through instancelevel decoupling that incorporates class differences, it significantly alleviates the oversmoothing with explicit classification boundaries, enhancing robust node representations. Inspired by the above discussions, we propose a supervised contrastive learning method Instance-Level Class-Difference Decoupling (ICDD). It leverages class differences to enhance the discriminative capability of nodes at the instance level. The fundamental concept is to minimize the similarity between node representations from different classes while simultaneously preserving the similarity between node representations from the same class. Consequently, the output representations within each layer can be distinctly separated by class, effectively mitigating oversmoothing at the instance level. Denote N as all node row indexes of the training set, and S 𝑖 ⊆ N, 𝑖 ∈ N represents node indexes with the same class of i. Denote D 𝑖 = N -{𝑖}, for loss function ℓ 𝐼𝑛𝑠,𝑖 of a single node 𝑖 in H (𝑙 ) , we have:EQUATION)nodes in the training set, the formulation is as follows:EQUATIONApplying ℓ 𝐼𝑛𝑠 in each GNN layer, L 𝐼𝑛𝑠 is as follows:EQUATIONMinimizing L 𝐼𝑛𝑠 prompts the representations of each layer to be discriminative while maintaining the similarity of nodes from the same class, thereby mitigating the oversmoothing issue with explicit classification boundaries.The dimensional decoupling loss in Equation ( 8) diversifies dimension information and enhances the dimension-level node differentiation, while the instance-level decoupling loss in Equation ( 11 loss, and ℓ 𝑐𝑒 (𝑣 𝑖 , 𝑦 𝑖 ) as calculating the cross-entropy loss with the representation of node 𝑣 𝑖 and its class labels 𝑦 𝑖 . The classification loss can be formulated as follows:EQUATIONCombine with Equation ( 8), Equation (11) , Equation ( 12), the overall objective function can be stated as (where 𝛼 > 0 is a coefficient that balances the trade-off of dimension decoupling):EQUATIONCompared to existing methods that primarily focus on instancelevel relations, we reveal the intensified dimension redundancy in deep GNNs and introduce a novel approach to eliminate the redundancy for enhancing node differentiation. Additionally, we propose utilizing class differences in the instance-level decoupling, alleviating oversmoothing with explicit classification boundaries. The dual-dimensional decoupling strategies complement each other and further enhance the overall performance. 
experiment: 
 Datasets. We conducted experiments on five well-known benchmarks: Cora, Citeseer, Pubmed [69] , CoauthorCS [53] and ogbnarxiv [24] . For the experiments on Cora/Citeseer/Pubmed, we followed the semi-supervised setting proposed by Kipf et al. [32] , with 20 nodes per class for training. For CoauthorCS, we adopted the same setting in [79] , using 40 nodes per class for training, 150 nodes per class for validation, and the remaining for testing. Besides, we also conduct experiments on large-scale datasets such as ogbn-arxiv [24] , to demonstrate the effectiveness of DDCD to data scalability. The detailed statistics of datasets can be found in Table 1 .Baselines. We conducted experiments on three GNN backbones: GCN [32] , GAT [57] , and ChebyNet [12] . We compared three regularization baseline methods: BatchNorm (BN) [27] , PairNorm (PN) [78] , DGN [79] , as well as three dropout methods: DropEdge (DE) [50] , SkipNode (SN) [42] , DropMessage (DM) [15] . To prove the effectiveness of DDCD in boosting deep GNNs, we experimented on two deep GNNs: GCNII [44] and EGNN [80] .Hyperparameters. We adopted the same experimental setting as DGN [79] . For Cora, Citeseer, and CoauthorCS, we used a dropout rate of 0.6, L2 regularization of 5 • 10 -4 , and a learning rate of 5 • 10 -3 . For Pubmed, we used a dropout rate of 0.6, L2 regularization of 1 • 10 -3 , and a learning rate of 1 • 10 -2 . We conducted each experiment five times and reported the average results. For BatchNorm, PairNorm, and DGN, we followed the results reported by [50] , SN: SkipNode [42] , DM: DropMessage [15] , BN: BatchNorm [27] , PN: PairNorm [78] , DGN: Differentiable Group Normalization [79] . DDCD: Our method. Bold represents the highest accuracy. Underline represents the second-highest accuracy. See details in Section 4.1. al. [79] on GCN and GAT. To investigate the effects of hyperparameters, we adjusted the hyperparameters 𝛼 in {0.002, 0.004, 0.006, 0.008, 0.01} and 𝜏 in {0.01, 0.03, 0.05, 0.07, 0.09} on the Cora dataset, reported the best performance with 𝛼 = 0.006 and 𝜏 = 0.05. Then we fixed 𝜏, adjusted 𝛼 in {6 • 10 -6 , 6 • 10 -5 , 6 • 10 -4 , 6 • 10 -3 } on Citeseer, Pubmed, and CoauthorCS, reported the best performance and recorded the values of 𝛼 respectively: 6 • 10 -5 on Citeseer, 6 • 10 -4 on Pubmed and CoauthorCS.We provide comparison results on both regularization and dropout methods with different backbones and layer numbers. Firstly, we compare the baseline methods with 2, 15, and 30 layers on different GNNs. As shown in quantitatively assess the performance of alleviating oversmoothing, we reported the Relative Inter-Class Distance (RICD) on the Cora dataset with various layers on the GCN model. Our method consistently exhibited the largest RICD at all layer cases, indicating the lowest degree of oversmoothing. All results confirm that our method achieves superior mitigation compared to baseline methods on different GNN models and various network layers.Firstly, we investigated the influence of two components, Dimension-Level Decoupling (in Section 3.3) and Instance-Level Class-Difference Decoupling (in Section 3.4), on the performance of the proposed method. We empirically demonstrate that both components have a positive impact on alleviating oversmoothing and can complementarily boost performance. Next, we investigate the effects of hyperparameters, demonstrating the stability with different parameters. Besides, we provide comparisons between Relative Inter-Class Distance (RICD) and existing metrics for oversmoothing, demonstrating the superiority of the proposed measurement.Effect of Each Component. To thoroughly assess the effectiveness of DDCD, we conducted extensive ablation experiments on the Cora dataset with varying numbers of GCN layers. Based on the results presented in Table 3 and Figure 5 , two main conclusions can be drawn. Firstly, both components have a positive impact on mitigating oversmoothing and enhancing performance. As depicted in Figure 5 , both DLD and ICDD result in a larger RICD compared to the baseline, signifying the effective mitigation of oversmoothing; Additionally, as shown in Table 3 , the utilization of DLD and ICDD results in an average performance improvement of 18.26% and 33.36% respectively, further indicating the successful alleviation of oversmoothing, subsequently enhancing classification performance across various layer scenarios. Secondly, the combination of these two components can complement each other and further enhance the performance. As illustrated in Table 3 , the combination of DLD and ICDD outperforms the baseline by an average of 45.30%, underscoring the collaborative effect of both DLD and ICDD.Effect of Hyperparameters. To investigate the impact of hyperparameters 𝜏 and 𝛼 for the proposed method, we conducted two ablation experiments on the Cora dataset with 15-layer GCN. Specifically, we performed the following experiments with DDCD and reported the node classification accuracy: (1) we changed the value of 𝛼 while keeping 𝜏 fixed at 0.05, and (2) changed the value of temperature 𝜏 while keeping the value of 𝛼 fixed at 0.006.In Experiment (1), as illustrated in Figure 6 (b), the performance of DDCD remains stable with changes in 𝛼, and is further boosted with Dimension-Level Decoupling equipped; In Experiment (2), as shown in Figure 6 (a), we observed that the performance of DDCD remains stable with different values of 𝜏, and is further improved with ICDD equipped. Specifically, on the Cora dataset, our proposed method achieves the best performance with 𝜏 = 0.05. We then apply the same value of 𝜏 on the Citeseer, Pubmed, and CoauthorCS datasets, and obtain similarly strong results. Both Experiment (1) and (2) show the stability of our method to hyperparameters, and the complementary of two modules.Pair-wise Distance v.s. Relative Inter-Class Distance. The pair-wise distance measures the degree of oversmoothing based on the distance between node pairs [6, 23, 40] . However, the pair-wise Failure to account for class information in the computation of pairwise distances results in an impact from class labels, leading to an inaccurate assessment of oversmoothing. Specifically, we provide the comparisons of Relative Inter-Class Distance and the existing metric SMV from [40] on the GCN model. As depicted in Figure 7 , SMV fails to accurately reflect the superiority of different methods. Notably, PairNorm and BatchNorm cannot be distinguished in SMV, and DGN demonstrates better oversmoothing alleviation but lower SMV than PN and BN. In contrast, the proposed RICD accurately reflects the superiority of different methods and the degree of oversmoothing alleviation.We conducted comprehensive experiments to prove the generalization of our method in various scenarios. First, we conducted experiments with SOTA deep GNN models, to emphasize the effectiveness of our approach in enhancing deep GNNs as a regularization technique. Additionally, we conducted experiments to demonstrate the robustness of our method in different settings and layer numbers. Moreover, we visualized the learned node representations, which depicts that DDCD is feasible to learn discriminative features for different classes.Boosting Deep GNN Models. Our proposed DDCD can be treated as a regularization for addressing oversmoothing, which can be seamlessly incorporated into different deep GNN models. To demonstrate the effectiveness of our method in enhancing deep GNNs, we apply DDCD to two deep models [44, 80] for graph data, with various network layers on the Cora dataset. As demonstrated in Table 4 , our method significantly enhances the performance of deep GNNs by mitigating the oversmoothing issue.Generalization to Large-Scale Graph. To demonstrate the generalization of the proposed DDCD to the graph scalability, we conduct experiments on the large-scale graph: ogbn-arxiv. To reduce the computational complexity, we randomly sample𝑁 nodes to apply DDCD in each GNN layer. As shown in Table 5 , we perform experiments on GCN and GAT with different layers, and the performance is consistently improved across all cases, demonstrating the robustness of DDCD to dataset scalability.EGNN Generalization to Imbalanced Classes. We leveraged class differences to address oversmoothing, and the performance of our method may potentially be influenced by the imbalanced numbers of nodes among different classes. To investigate the generalization of DDCD to the imbalanced classes, we re-sampled the Cora dataset, where one of the classes has three times as many nodes as the other classes. As depicted in Table 6 , our method consistently and effectively mitigates oversmoothing in all layer cases, indicating the robustness of DDCD to imbalanced classes.Different Numbers of Layers with GAT/Cheby. To demonstrate the generalization of DDCD with various layer numbers, we provided experimental results for the GAT and Cheby models with different numbers of layers. As depicted in Figure 8 , we experimented from 2 to 30 layers, and the incorporation of DDCD leads to a larger RICD and consistently improves classification accuracy in all layer scenarios, signifying the effective alleviation of oversmoothing and consequently boosting performance.Node Representation Visualization. We visualized the node representations of vanilla GCN, DGN [79] , and DDCD with a 30layer model on the Cora dataset. As depicted in Figure 9 (a), when with deep network architecture (30 layers), the node representations become mixed and cannot be clearly distinguished, indicating severe oversmoothing and leading to a decline in classification performance. Upon the integration of DGN [79] , as illustrated in Figure 9 (b), nodes can be roughly categorized by class, but the inappropriate grouping strategy blurs the nodes at the classification boundaries. However, as shown in Figure 9 (c), the node representations exhibit well discrimination and explicit classification boundaries equipped with DDCD. 
conclusion: 
 In deep GNNs, node representations become highly indistinguishable due to repeated aggregations, i.e., the oversmoothing issue. In this paper, we construct a simple and effective dual-dimensional regularization method to address the oversmoothing issue. Specifically, we consider the oversmoothing issue from two aspects of the node embedding space: dimension and instance. At the dimension level, we first reveal the dimensional causes for oversmoothing and propose to alleviate oversmoothing by minimizing covariances between dimensions to enhance node differentiation. Besides, at the instance level, we propose to utilize class differences, alleviating oversmoothing with explicit class boundaries. This research provides valuable insights for tacking oversmoothing in deep GNNs and will facilitate potential real-world applications. In future work, it is significant to generalize our instance-level method for tasks lacking node labels, e.g., link prediction tasks, where linked/unlinked nodes can be regarded as intra/inter-class samples and instance-level decoupling helps obtain discriminative node representations. 
: 
 1Graph data encodes information on both individual entities and the relationships between those entities, finding wide applications in practical scenarios such as social recommendation [14, 49, 81] , node classification [26, 29, 32, 59] , biology design [45, 51] , and so on [43, 48, 58, 60, 68, 71] . To extract meaningful information from graph data, Graph Neural Networks (GNNs) [12, 32, 57, 61, 65, 82] are widely adopted and have shown promising results in generating effective node representations. Inspired by the success of deepening Convolutional Neural Network (CNN) layers [21, 33, 54, 55] , recent works have attempted to improve GNN performance by increasing network depth to mine the multi-hop relation of graph data. However, deeper GNN layers often suffer from performance degradation, which has been the subject of extensive discussions in recent years. Li et al. [38] first revealed that repeated information aggregations in deep GNNs lead to highly indistinguishable node representations and hampers performance. This phenomenon is widely recognized as oversmoothing issue [2, 30, 38, 46, 66] .Numerous efforts have been made to alleviate oversmoothing [27, 35, 50, 78, 79] . Several works tackle oversmoothing by modifying the model architecture. Li et al. [35, 36] proposed to apply residual/dense connections and achieve better performance with deeper GNNs. Chen et al. [44] proposed to utilize identity mapping to improve GNN capabilities. Zhou et al. [80] investigated oversmoothing by the Dirichlet energy and developed a generalized principle to train deep GNNs. However, the designs on model architecture are inflexible and introduce additional network parameters, limiting their applications. Besides, several works attempt to address oversmoothing from the graph structure and node embeddings, which can be appropriately applied to existing deep GNNs: Dropout-based methods [15, 42, 50] randomly drop node features or graph edges to avoid redundant aggregation but can inadvertently disrupt the node and structural information of the graph, losing original graph information; Normalization-based methods [27, 78, 79] address oversmoothing by normalizing node representations after each layer to preserve node distinctiveness, are versatile and can be readily applied to various GNN models. However, they mainly focus on instance-level cues and the dimension-level relations for enhancing node differentiation remain unexplored.Beyond the aforementioned limitations, we consider the oversmoothing issue from two perspectives of the node embedding space: dimension and instance. In self-supervised learning, dimensional relations have been demonstrated to improve the representation capacity of the embedding space [1, 37, 39, 75] . Likewise, we assume that dimensional relations can also enhance the node representation in GNNs. We empirically uncovered that the pronounced information redundancy between dimensions hinders the capability of embeddings and impacts the node differentiation ability in GNNs (detailed in Section 3.3). Drawing inspiration from these insights, we propose to enhance node differentiation at the dimension level by reducing the information redundancy between dimensions of node embeddings in deep GNNs. To achieve this, we introduce the Dimension-Level Decoupling (DLD) that minimizes the covariance between dimensions in each GNN layer. It reduces the redundant information within node embedding dimensions, which effectively promotes diversity dimension information and enhances the differentiation ability of node embeddings.Besides, existing instance-level methods normalize node representations without taking class differences into account, resulting in vague classification boundaries and performance degradation. For instance, Zhou et al. [79] introduced Differentiable Group Normalization (DGN) that assigns group probabilities and normalizes node representations based on group relations. However, nodes from different classes can be grouped together, leading to fuzzy class boundaries. To utilize class differences for explicit classification boundaries, we introduce the Instance-Level Class-Difference Decoupling (ICDD) to alleviate the oversmoothing issue, which can be applied to various GNN architectures. The basic idea is to repel the inter-class nodes and attract intra-class nodes after each GNN layer, dynamically encouraging inter-class nodes to separate while maintaining similarity among intra-class node features. By decoupling node representations based on class differences, this strategy substantially enhances instance-level differentiation with explicit classification boundaries.Additionally, existing metrics [6, 23] assess oversmoothing based on the distances between node pairs. However, the pair-wise distances are associated with class information: nodes in different classes should exhibit distinct pair-wise distances while having closer pair-wise distances in the same class. Without considering class differences, the pair-wise distances are affected by class information and fail to accurately assess oversmoothing. In this case, we introduce a new metric: Relative Inter-Class Distance (RICD), which calculates the similarity difference between inter-class nodes and intra-class nodes. It focuses on the relative difference between inter-class nodes and intra-class nodes, accounting for the influence of class information on the accurate assessment of oversmoothing. In summary, our primary contributions can be outlined as follows:• We propose a dual-dimensional decoupling method for alleviating oversmoothing, which can be readily applied to various deep GNNs. Firstly, at the dimension level, we propose to reduce dimension redundancy by minimizing dimension covariances, diversifying the dimension information and ultimately promoting well-differentiated nodes for mitigating oversmoothing. Graph Neural Networks (GNNs) [13, 17, 52, 76] are proposed to generate meaningful node representations by propagation and transformation over multiple network layers. They can be broadly categorized into spectral-based [3, 12, 22, 32] and spatial-based [16, 19, 57] models. Spectral methods perform convolution operations in the graph domain using spectral filters [3] and their simplified variants. In spatial GNNs, convolution operations are carried out by propagating and aggregating local information along the edges of a graph. Different aggregation functions are employed in spatial GNNs to learn node representations, including mean/max pooling [19] , self-attention [57] , and summation [64] . 
 
 Appendix:  


Decision: This paper addresses the over-smoothing issue in Graph Neural Networks. The overall score is positive (5 4 4 3). In the rebuttal phase, reviewer scqz raised the score from 2 to 3 but still has concerns about over-smoothing which has been solved by other works. 

To conclude, this paper is a good one even though there exist several limitations.

Reviews: review: paper_summary: paper_summary: This work addresses the over-smoothing issue in Graph Neural Networks (GNNs), where deep GNNs suffer from indistinguishable node representations due to repeated aggregations. By approaching the problem from two perspectives: the dimension and the instance level. And introducing Dimension-Level Decoupling (DLD) to reduce dimensional information redundancy, thereby enhancing node differentiation. Additionally, the proposed Instance-Level Class-Difference Decoupling (ICDD) to improve instance-level node discrimination by repelling inter-class nodes and attracting intra-class nodes, resulting in clearer classification boundaries. This work also presents a novel evaluation metric that considers class differences' impact on node distances, facilitating precise over-smoothing measurement. Extensive experiments demonstrate the effectiveness of the proposed Dual-Dimensional Class-Difference Decoupling (DDCD) method across various scenarios.
strength_weakness: strength_weakness: 1. This work introduces a novel dual-dimensional decoupling approach to address the over-smoothing problem in deep Graph Neural Networks (GNNs). This method is adaptable to various GNN architectures. 
2. The work incorporates class differences at the instance level to alleviate over-smoothing by establishing explicit class boundaries.
3. The work introduces a new metric to measure over-smoothing. This metric quantifies the similarity differences between inter-class and intra-class nodes, taking into account the impact of class differences, thereby providing an accurate measure of over-smoothing extent.
4. The paper is well-organized and clearly written, it’s easy to follow.
limitations: limitations: 1. Effectiveness of Deep GNNs: The necessity of deep GNNs is questionable. The results in Table 2 indicate that the performance of DDCD is not as effective as a 2-layer GNN, raising concerns about the benefits of using deeper architectures.
2. Misalignment with Contrastive Learning: Although the RELATED WORK section highlights a focus on contrastive learning on graphs, the paper lacks a detailed analysis of how the proposed method aligns with contrastive learning principles. Additionally, the experimental section does not include baselines from graph contrastive learning, which is a significant omission.
3. Advanced Baselines: The paper compares DDCD with several baselines, but these baselines are not recent or advanced methods. It is crucial to include more current and sophisticated baselines, such as UAG[1], to provide a more comprehensive evaluation of the proposed method's effectiveness.

[1]Kong H, Kim S, Kim H J, et al. Unknown-Aware Graph Regularization for Robust Semi-supervised Learning from Uncurated Data[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12): 13265-13273.
paper_summary: paper_summary: The paper introduces a novel dual-dimensional decoupling mechanism, which focuses on the over-smoothing problem in graph neural networks. It tackles the main challenge: most existing methods alleviate the over-smoothing problem at the instance level while ignoring the dimension-level node differentiation. The authors adopt a dual-dimensional decoupling mechanism to reduce the redundant information within node embedding dimensions. Besides, a novel metric to measure over-smoothing is designed. The experimental results on several real-world datasets demonstrate the effectiveness of the proposed model.
strength_weakness: strength_weakness: 1. The paper is well-motivated since over-smoothing is a significant problem in graph representation learning. Besides, dimension-oriented mechanism to reduce the redundant information is interesting and worth exploring.

2. The paper provides a well-structured overview, making it easy to follow for readers.

3. It encompasses all the essential components required in tackling the over-smoothing problem, offering a highly comprehensive coverage.

4. The runnable code is provided, making readers easy for reproduction of experimental results.

5. A vast set of experiments were conducted in several benchmark datasets, showing the superiority of the proposed method.
limitations: limitations: 1. There seem to lack analysis about how previous researches addressed the over-smoothing problem with dimensional redundant.

2. There seem to lack some SOTA compared methods with decoupling, such as [1].

[1] Zhang, Hongyuan, Yanan Zhu, and Xuelong Li. "Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

3. There is no definition of decoupling, and it is not clear why the measurement with covariance matrix can be seen as a decoupled way.

4. I am not sure whether this kind of work for single graph is consistent with the multimedia theme of ACM MM 2024.
paper_summary: paper_summary: This study addresses the oversmoothing issue in GNNs by proposing a approach with two modules. Dimension-Level Decoupling (DLD) targets redundancy in node dimensions, improving differentiation, while Instance-Level Class-Difference Decoupling (ICDD) focuses on enhancing classification clarity by managing inter-class and intra-class node distinctions. Extensive experiments validate the effectiveness of the proposed Dual-Dimensional Class-Difference Decoupling (DDCD) across various scenarios, demonstrating its potential to significantly enhance GNN performance.
strength_weakness: strength_weakness: - This paper is well-organized, narrating from the background and existing problems to their solutions.
- The paper features excellent images and well-formatted tables, experiments are comprehensive.
- I appreciate the authors for providing code, which is easy to read.
limitations: limitations: The main concerns revolve around the novelty and rationality of the proposed solutions:
- The proposed method serves as a normalization approach to tackle the over-smoothing problem. However, simpler methods such as initial residual [1, 2], combining hidden representations from multiple hops/layers [3, 4], and decoupling aggregation and transformation [5] have already addressed this issue using straightforward strategies, enabling GNNs to stack hundreds of layers without performance degradation. Notably, the experiments incorporate several backbones but yield minimal gains; a 2-layer GCN often achieves the best performance in most situations. Therefore, the necessity of normalization tricks is questionable.
- DLD aims to minimize the non-diagonal elements of the covariance matrix in each layer. However, this could be achieved with traditional methods like PCA. Using it as a loss function may be ok but somewhat weird, at least a further disscusion is needed.
- The use of contrastive loss to over-smoothing has been previously explored [6]. The authors fail to discuss how their approach differs from this. Additionally, there is a significant discrepancy between the results reported in their paper and those in ContraNorm [6]; could the authors clarify this?
In summary, while the paper exhibits excellent research skills, the problems it addresses and the solutions it proposes lack significant impact.

[1] Predict then propagate: Graph neural networks meet personalized pagerank

[2] Simple and deep graph convolutional networks

[3] Adagcn: Adaboosting graph convolutional networks into deep models

[4] Representation learning on graphs with jumping knowledge networks

[5] Dissecting the diffusion process in linear graph convolutional networks

[6] Contranorm: A contrastive learning perspective on oversmoothing and beyond
paper_summary: paper_summary: This paper proposed Dual-Dimensional Class-Difference Decoupling (DDCD), which addresses the oversmoothing issue in Graph Neural Networks (GNNs) by reducing dimension redundancy with Dimension-Level Decoupling (DLD) and enhancing node differentiation by considering class differences with Instance-Level Class-Difference Decoupling (ICDD), supported by a novel evaluation metric.
strength_weakness: strength_weakness: 1. It's interesting and holds novelty in tracking over-smoothing problems from the node feature dimension.
2. The motivation for using "Dimenstion-level Decoupling" and "instance-level class-difference Decoupling" is clear and convincing.
3. A new metric for similarity between nodes is proposed.
4. The organization and writing are good and easy to follow, and every figure supports the claim of the authors.
5. The detailed and comprehensive experiments show the effectiveness of the proposed method.
limitations: limitations: 1. It's better to compare with some SOTA GCN methods, especially after 2023, to show the effectiveness of the proposed method.
2. For the proposed metric, Relative Inter-Class Distance (RICD), if one sets the group number the same as the class number for DGN, what is the difference between it and RICD?
3. Some recent research about over-smoothing problems is missing, please add some discussion:
   - Duan, W., J. Xuan, M. Qiao, and J. Lu. “Learning from the Dark: Boosting Graph Convolutional Neural Networks With Diverse Negative Samples”. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 6, June 2022, pp. 6550-8.
   - Wei Duan, Jie Lu, Yu Guang Wang, & Junyu Xuan (2024). Layer-diverse Negative Sampling for Graph Neural Networks. Transactions on Machine Learning Research.