GraphLearner: Graph Node Clustering with Fully Learnable Augmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 14:14:48

Abstract: Contrastive deep graph clustering (CDGC) leverages the power of contrastive learning to group nodes into different clusters.  The quality of contrastive samples is crucial for achieving better performance, making augmentation techniques a key factor in the process. However, the augmentation samples in existing methods are always predefined by human experiences, and agnostic from the downstream task clustering, thus leading to high human resource costs and poor performance. To overcome these limitations, we propose a Graph Node Clustering with Fully Learnable Augmentation, termed GraphLearner. It introduces learnable augmentors to generate high-quality and task-specific augmented samples for CDGC. GraphLearner incorporates two learnable augmentors specifically designed for capturing attribute and structural information. Moreover, we introduce two refinement matrices, including the high-confidence pseudo-label matrix and the cross-view sample similarity matrix, to enhance the reliability of the learned affinity matrix. During the training procedure, we notice the distinct optimization goals for training learnable augmentors and contrastive learning networks. In other words, we should both guarantee the consistency of the embeddings as well as the diversity of the augmented samples. To address this challenge, we propose an adversarial learning mechanism within our method. Besides, we leverage a two-stage training strategy to refine the high-confidence matrices. Extensive experimental results on six benchmark datasets validate the effectiveness of GraphLearner.

Main paper: 
 method: 
 In this section, we propose a novel attribute graph contrastive clustering method with fully learnable augmentation (GraphLearner). The overall framework of GraphLearner is shown in Fig. 1 . The main components of the proposed method include the fully learnable augmentation module and the dual refinement module. We will detail the proposed GraphLearner in the following subsections.For an undirected graph G = {X, A}. X ∈ R 𝑁 ×𝐷 is the attribute matrix, and A ∈ R 𝑁 ×𝑁 represents the original adjacency matrix. D = 𝑑𝑖𝑎𝑔(𝑑 1 , 𝑑 2 , . . . , 𝑑 𝑁 ) ∈ R 𝑁 ×𝑁 is denoted as the degree matrix, where 𝑑 𝑖 = (𝑣 𝑖 ,𝑣 𝑗 ) ∈ E 𝑎 𝑖 𝑗 . We define the graph Laplacian matrix as L = D -A. With the help of renormalization trick A = A + I, the normalized graph Laplacian matrix is denoted asL = D -1 2 L D -12 . Moreover, we define 𝑠𝑖𝑚(•) as a non-parametric metric function to calculate pair-wise similarity, e.g., cosine similarity function. Aug S and Aug X represent the augmented structure and attribute matrix, respectively. The basic notations are summarized in Table 2 . ∈ R 𝑁 ×𝐷 Attribute matrix A ∈ R 𝑁 ×𝑁 Original adjacency matrix D ∈ R 𝑁 ×𝑁 Degree matrix F 𝑣 𝑘 ∈ R 𝑁 ×𝑑 Node embeddings in 𝑘-th view 𝑠𝑖𝑚 (•) Non-parametric metric function S ∈ R 𝑁 ×𝑁 Similarity sample matrix Z ∈ R 𝑁 ×𝑁High-confidence pseudo label matrix Aug 𝑋 ∈ R 𝑁 ×𝑑 Augmented attribute matrix Aug 𝑆 ∈ R 𝑁 ×𝑁 Augmented structure matrixDifferent from previously augmentation method, in this subsection, we propose a fully learnable graph augmentation strategy in both structure and attribute level. To be specific, we design the structure augmentor and attribute augmentor to dynamically learn the structure and attribute, respectively. In the following, we will introduce these augmentors in detail.Augmentor. We design the structure augmentor to obtain the structure of the augmented view. Specifically, we propose three types structure augmentor, i.e., MLP-based structure augmentor, GCN-based structure augmentor, and Attention-based augmentor.MLP-based Structure Augmentor. We use Multi-Layer Perception (MLPs) to generate the structure Aug S . This procedure can be represented as follows:EQUATIONwhere F ∈ R 𝑁 ×𝐷 is the embedding of the original adjacency. We adopt the cosine similarity function as 𝑠𝑖𝑚(•) to calculate the similarity of F. The calculated similarity matrix can be regarded as the learned structure matrix A 𝑀𝐿𝑃 . GCN-based Structure Augmentor is the second type of our designed structure generator, which embeds the attribute matrix X and original adjacency matrix A into embeddings in the latent space. For simplicity, we define the GCN-based structure augmentor as:EQUATIONwhere F is embedding extracted by the GCN-based network, for example, GCN [12] , GCN-Cheby [6] , etc. 𝜎 (•) is a non-linear operation.Attention-based Structure Augmentor. Inspire by GAT [27] , we design an attentive network to capture the important structure of the input graph G. To be specific, the normalized attention coefficient matrix A 𝑎𝑡𝑡 𝑖 𝑗 between node 𝑥 𝑖 and 𝑥 𝑗 could be computed as:EQUATIONwhere ì n and W is the learnable weight vector and weight matrix, respectively. || is the concatenation operation between the weight Figure 1 : Illustration of the fully learnable augmentation algorithm for attribute graph contrastive clustering. In our proposed algorithm, we design the learnable augmentors to to dynamically learn the structure and attribute information. Besides, we optimize the structure of the augmented view with two aspects, i.e., high-confidence clustering pseudo label matrix and cross-view similarity matrix, which integrates the clustering task and the augmentation learning into the unified framework. Moreover, we propose an adversarial learning mechanism to keep cross-view consistency in the latent space while ensuring the diversity of augmented views. Lastly, a two-stage training strategy is designed to obtain high-confidence refinement matrices, thus improving the reliability of the learned graph structure. matrix W 𝑥 𝑖 and W 𝑥 𝑗 , and N 𝑖 represents the indices the neighbors of node 𝑥 𝑖 . By this setting, the model could preserve important topological and semantic graph patterns via the attention mechanism.To make the augmented view in a fully learnable manner, we design the attribute augmentor to dynamically learn the original attribute. Specifically, we design two types attribute augmentor, i.e., MLP-based attribute augmentor and attention-based attribute augmentor.MLP-based Attribute Augmentor. Similar to the MLP-based structure augmentor, we utilize the Multi-Layer Perception (MLP) as the network to learn the original attribute matrix X. The learned attribute matrix Aug X ∈ R 𝑁 ×𝐷 can be presented as:EQUATION)where 𝑀𝐿𝑃 (•) is the MLP network to learn the attribute. Attention-based Attribute Augmentor. To guide the network to take more attention to the important node attributes, we design an attention-based attribute augmentor. Specifically, we map the node attributes into three different latent spaces:Q = W 𝑞 X ⊤ K = W 𝑘 X ⊤ V = W 𝑣 X ⊤ (5)where W 𝑞 ∈ R 𝐷 ×𝐷 , W 𝑘 ∈ R 𝐷 ×𝐷 , W 𝑣 ∈ R 𝐷 ×𝐷 are the learnable parameter matrices. And Q ∈ R 𝐷 ×𝑁 , K ∈ R 𝐷 ×𝑁 and V ∈ R 𝐷 ×𝑁 denotes the query matrix, key matrix and value matrix, respectively.The attention-based attribute matrix Aug X can be calculated by:EQUATIONAfter the structure augmentor and attribute augmentor, we could obtain the augmented view G ′ = (Aug S , Aug X ). The , which is fully learnable.In this subsection, we propose a dual refinement module to optimize the learned graph structure. To be specific, the cross-view sample similarity matrix and the high-confidence matrix are generated to improve the quality of the structure in augmented view. Firstly, we use encoder network F (•) to obtain the embeddings of the original view G and the augmented view G ′ with ℓ 2 -norm as follows:EQUATIONIn the following, we fuse the two views of the node embeddings as follows: EQUATIONThen we perform K-means [9] on F and obtain the clustering results. After that, we will refine the learned view in two manners, i.e., similarity matrix and pseudo labels matrix refinement.Similarity Matrix Refinement. Through F (•), we could obtain the embeddings of each view. Subsequently, the similarity matrix S represents the similarity between 𝑖-th sample in the first view and 𝑗-th sample in the second view as formulated:EQUATION)where S is the cross-view similarity matrix, and ⟨•⟩ is the function to calculate similarity. Here, we adopt cosine similarity. The proposed similarity matrix S measures the similarity between samples by comprehensively considering attribute and structure information. The connected relationships between different nodes could be reflected by S. Therefore, we utilize S to refine the structure in augmented view with Hadamard product:EQUATIONPseudo Labels Matrix Refinement. To further improve the reliability of the learned structure matrix, we extract reliable clustering information to construct the matrix to further refine the structure in augmented view. Concretely, we utilize the top 𝜏 highconfidence pseudo labels p to construct the matrix as follows:EQUATIONwhere Z 𝑖 𝑗 denotes the category relation between 𝑖-th and 𝑗-th samples. In detail, when Z 𝑖 𝑗 = 1, two samples have the same pseudo label. While Z 𝑖 𝑗 = 0 implies that two samples have different pseudo labels. The pseudo-label matrix is constructed by the high-confidence category information. Therefore, the adjacency relation in the graph could be well reflected, leading to optimizing the structure of the learned structure in the augmented view. The pseudo labels matrix refines the learned structure with Hadamard product as:EQUATIONIn summary, in this subsection, we propose two strategies to refine the structure of the augmented view. Firstly, S is calculated by the cross-view similarity. The value of S represents the probability of connection relationships of the nodes. The structure of Aug S is optimized by S in the training process. Besides, we utilize the high-confidence clustering pseudo labels to construct the reliable node connection, which is constructed when the node belongs to the same category. By those settings, the learned structure is regularized by the similarity matrix and the high-confidence matrix, thus improving the reliability of the structure in the augmented view. Moreover, the connection between augmentation and the clustering task is enhanced.Input: The input graph G = {X, A}; The iteration number 𝐼 ; Hyper-parameters 𝜏, 𝛼. Output: The clustering result R.1: for 𝑖 = 1 to 𝐼 do 2:Obtain the learned structure matrix Aug S and attribute matrix Aug X with our augmentors.Encode the node with the network F (•) to obtain the node embeddings F 𝑣 1 and F 𝑣 2 .Fuse F 𝑣 1 and F 𝑣 2 to obtain F with Eq. ( 8).Perform K-means on F to obtain the clustering result.Calculate the similarity matrix of F 𝑣 1 and F 𝑣 2 .Obtain high-confidence pseudo label matrix.Refine the learned structure matrix Aug S with Eq.( 10) and Eq. ( 12). 9:Calculate the learnable augmentation loss L 𝑎 with Eq. ( 13).10:Calculate the contrastive loss L 𝑐 with Eq. ( 14).Update the whole network by minimizing L in Eq. ( 15). 12: end for 13: Perform K-means on F to obtain the final clustering result R.The proposed GraphLearner framework follows the common contrastive learning paradigm, where the model maximizes the agreement of the cross-view [45] . In detail, GraphLearner jointly optimizes two loss functions, including the learnable augmentation loss L 𝑎 and the contrastive loss L 𝑐 .To be specific, L 𝑎 is the Mean Squared Error (MSE) loss between the original graph G = {X, A} and the learnable graph G ′ = {Aug X , Aug S }, which can be formulated as:EQUATIONIn GraphLearner, we utilize the normalized temperature-scaled cross-entropy loss (NT-Xent) to pull close the positive samples, while pushing the negative samples away. The contrastive loss L 𝑐 is defined as:EQUATIONwhere temp is a temperature parameter. sim(•) denotes the function to calculate the similarity, e.g., inner product. The total loss of GraphLearner is calculated as follows:EQUATION)where 𝛼 is the trade-off between L 𝑎 and L 𝑐 . The first term in Eq.( 15) encourages the network to generate the augmented view with distinct semantics to ensure the diversity in input space, while the second term is the contrastive paradigm to learn the consistency of two views in latent space. The discriminative capacity of the network could be improved by minimizing the total loss function. The network is optimized by Eq.( 15) during the whole training process. The detailed learning process of GraphLearner is shown in Algorithm 1. The memory cost of L is acceptable. We utilize 𝐵 to denote the batch size. The dimension of the embeddings is 𝐷. The time complexity of L is O (𝐵 2 𝐷). And the space complexity of L is O (𝐵 2 ) due to matrix multiplication. The detailed experiments are shown in section 3.3. Besides, we design a two-stage training strategy to improve the confidence of the clustering pseudo labels during the overall training procedure. To be specific, the discriminative capacity of the network is improved by the first training stage. Then, in the second stage, we refine the learned structure Aug S in the augmented view with the more reliable similarity matrix and the pseudo labels matrix. 
experiment: 
 In this section, we implement experiments to verify GraphLearner. The downstream task of our method is graph node clustering, which is implemented in unsupervised scenario. Therefore, the compared methods do not include the graph classification method, e.g., JOAO [39] , AutoGCL [38] , AD-GCL [23] . We have already described the differences in the Introduction and Related work. The effectiveness and superiority of our proposed GraphLearner can be illustrated by answering the following questions: RQ1: How effective is GraphLearner for attribute node clustering? RQ2: How about the efficiency about GraphLearner? RQ3: How does the proposed module influence the performance of GraphLearner? RQ4: How do the hyper-parameters impact the performance of GraphLearner? RQ5: What is the clustering structure revealed by GraphLearner?Benchmark Datasets The experiments are implemented on six widely-used benchmark datasets, including CORA [5] , BAT [19] , EAT [19] , AMAP [15] , CITESEER1 , and UAT [19] . The summarized information is shown in Table 4 . Detailed description of the datasets are shown in Section 1 of Appendix.Training Details The experiments are conducted on the Py-Torch deep learning platform with the Intel Core i7-7820x CPU, one NVIDIA GeForce RTX 3080Ti GPU, 64GB RAM. The max training epoch number is set to 400. For fairness, we conduct ten runs for all methods. For the baselines, we adopt their source with original settings and reproduce the results.Evaluation Metrics The clustering performance is evaluated by four metrics, including Accuracy (ACC), Normalized Mutual Information (NMI), Average Rand Index (ARI), and macro F1-score (F1).Parameter Setting In our model, the learning rate is set to 1e-3 for UAT, 1e-4 for CORA/CITESEER, 1e-5 for AMAP/BAT, and 1e-7 for EAT, respectively. The threshold 𝜏 is set to 0.95 for all datasets. The epoch to begin the second training stage is set to 200. The tradeoff 𝛼 is set to 0.5. Due to the limited space, The hyper-parameter settings are summarized in Table . 1 of the Appendix.In this subsection, to verify the superiority of GraphLearner, we compare the clustering performance of our proposed algorithm with 10 baselines on six datasets with four metrics. We divide these methods into four categories, i.e., classical deep clustering methods (DAEGC [28] , SDCN [1]), contrastive deep graph clustering methods (DCRN [15] , AGC-DRR [7] , CCGC [36] , CONVERT [37] ), graph structure learning methods (SUBLIME [17] ), and graph augmentation methods (GCA [45] , AFGRL [13] , AutoSSL [11] ). Moreover, due to the limitation of the space, we conduct additional comparison experiments with 5 baselines. These results are shown in Table . 2 of the Appendix. The results could also demonstrate the superiority of GraphLearner.Here, we adopt the attention structure augmentor and the MLP attribute augmentor to generate the augmented view in a learnable way. The results are shown in Table .3 , we observe and analyze as follows.1) GraphLearner obtains better performance compared with classical deep clustering methods. The reason is that they rarely design a specifically contrastive learning strategy to capture the supervision information implicitly.2) Contrastive deep graph clustering methods achieve sub-optimal performance compared with ours. We conjecture that the discriminative capacity of our GraphLearner is improved with fully learnable augmentation and optimization strategies.3) The classical graph augmentation methods achieve unsatisfied clustering performance. This is because they merely consider the learnable of the structure, while neglecting the attribute. Moreover, most of those methods can not optimize with the downstream tasks. 4) It could be observed that the graph structure learning methods are not comparable with ours. We analyze the reason is that those methods refine the structure with the unreliable strategy at the beginning of the training. In summary, our method outperforms most of other algorithms on six datasets with four metrics. Taking the result on CORA dataset for example, GraphLearner exceeds the runner-up by 1.70%, 2.38%, 2.37%, 2.00% with respect to ACC, NMI, ARI, and F1.Moreover, we implement experiments on other augmentors. The results are shown in Table . 7, we conclude that our proposed augmentors could achieve better performance on most metrics compared with other graph clustering algorithms.In this subsection, we implement time and memory cost experiments to demonstrate the effectiveness of the proposed GraphLearner.Specifically, we test the training time of GraphLearner with six baselines on six datasets. For fairness, we train all algorithms with 400 epochs. From the results in Table 6 , we observe that the training time of GraphLearner is comparable with other seven algorithms. The reason we analyze is as follows: instead of using GCN as the encoder network, we adopt the graph filter to smooth the feature. This operation effectively reduces time consumption.Moreover, we conduct experiments to test GPU memory costs of our proposed GraphLearner. Due to the limited space, we conduct the memory experiments on Fig. 1 of the Appendix. The results demonstrate that the memory costs of our GraphLearner are also comparable with other algorithms.In this section, we first conduct ablation studies to verify the effectiveness of the proposed modules. Due to the limited space, we conduct experiments about the effectiveness of the the similarity and pseudo-label matrix refinement strategies on Fig. 2 in Appendix.3.4.1 Effectiveness of the Structure and Attribute Augmentor. To verify the effect of the proposed structure and attribute augmentor, we conduct extensive experiments as shown in Table 5 . Here, we adopt "(w/o) Aug X ", "(w/o) Aug S " and "(w/o) Aug X & Aug S " to represent the reduced models by removing the structure augmentor, the attribute augmentor, and both, respectively. From the observations, it is apparent that the performance will decrease without any of our proposed augmentors, revealing that both augmentors make essential contributions to boosting the performance. Taking the result on the CORA dataset for example, the model performance is improved substantially by utilizing the attribute augmentor.To avoid the existing and predefined augmentations on graphs, we design a novel fully learnable augmentation method for graph clustering.In this part, we compare our view construction method with other classical graph data augmentations, including mask feature [43] , drop edges [34] , add edges [34] , and graph diffusion [25] . Concretely, in Table 5 , we adopt the data augmentation as randomly dropping 20% edges ("Drop Edges"), or randomly adding 20% edges ("Add Edges"), or graph diffusion ("Diffusion") with 0.20 teleportation rate, or randomly masking 20% features ("Mask Feature"). From the results, we observe that the performance of commonly used graph augmentations is not comparable with ours. In summary, extensive experiments have demonstrated the effectiveness of the proposed learnable augmentation.We verify the sensitivity of 𝛼. The experimental results are shown in Fig. 3 . The performance will not fluctuate greatly when 𝛼 ∈ [0.1, 0.9]. From these results, we observe that our GraphLearner is insensitive to 𝛼 when 𝛼 ∈ [0.1, 0.9]. Due to the limited space, we investigate the influence of the hyper-parameter threshold 𝜏. Experimental evidence can be found in Fig. 3 in Appendix.In this subsection, we visualize the distribution of the learned embeddings to show the superiority of GraphLearner on CORA and AMAP datasets via 𝑡-SNE algorithm [26] . We implement experiments with DAEGC [28] , SDCN [1], GCA [45] , AutoSSL [11] , SUB-LIME [17] and ours. The results are shown in Fig. 2 . From the results, we can conclude that GraphLearner better reveals the intrinsic clustering structure. 
related work 4.1 contrastive deep graph clustering: 
 The existing deep graph clustering methods can be roughly categorized into three classes: generative methods [1, 4, 22, 25, 28, 29, 42] , adversarial methods [20, 21, 24] , and contrastive methods [5, 10, 15, 16, 34, 43] . In recent years, the contrastive learning has achieved great success in vision [2, 3, 8, 41] and graph [33, 40, 44] .In this paper, we focus on the data augmentation of the contrastive deep graph clustering methods. Concretely, a pioneer AGE [5] conducts contrastive learning by a designed adaptive encoder. Besides, MVGRL [10] generates two augmented graph views. Subsequently, DCRN [15] aims to alleviate the collapsed representation by reducing correlation in both sample and feature levels. Meanwhile, the positive and negative sample selection have attracted great attention of researchers. Concretely, GDCL [43] develops a debiased sampling strategy to correct the bias for negative samples. Although promising performance has been achieved, previous methods generate different graph views by adopting uniform data augmentations like graph diffusion, edge perturbation, and feature disturbation. Moreover, these augmentations are manually selected and can not be optimized by the network, thus limiting performance. To solve this problem, we propose a novel contrastive deep graph clustering framework with fully learnable graph data augmentations.Graph data augmentation [30, 31] is an important component of contrastive learning. The existing data augmentation methods in graph contrastive learning could rough be divided into three categories, i.e., augment-free methods [13] , adaptive augmentation methods [39, 45] , and learnable data augmentation methods [11, 23, 35, 38] .AFGRL [13] generates the alternative view by discovering nodes that have local and global information without augmentation. While the diversity of the constructed view is limited, leading to poor performance. Furthermore, to make graph augmentation adaptive to different tasks, JOAO [39] learns the sampling distribution of the predefined augmentation to automatically select data augmentation. GCA [45] proposed an adaptive augmentation with incorporating various priors for topological and semantic aspects of the graph. However, the augmentation is still not learnable in the adaptive augmentation methods. Besides, in the field of graph classification, AD-GCL [23] proposed a learnable augmentation for edge-level while neglecting the augmentations on the node level. More recently, AutoGCL [38] proposed a probability-based learnable augmentation. Although promising performance has been achieved, the previous methods still rely on the existing and predefined data augmentations. CONVERT [37] places a strong emphasis on the semantic reliability of augmented views by leveraging a reversible perturb-recover network to generate embeddings for these augmented views. However, it tends to overlook the significance of the underlying graph topology. In this work, we propose a fully learnable augmentation strategy specifically tailored for graph data.To highlight the uniqueness of our approach, we draw comparisons with existing graph data augmentation methodologies. First and foremost, GraphLearner stands out by introducing a fully learnable augmentation approach for graphs. This sets it apart from adaptive augmentation methods such as GCA [45] and JOAO [39] , which still rely on predefined augmentations. GraphLearner dynamically generates augmented views using a learnable process, considering both structural and attribute aspects of the graph. Moreover, while existing learnable augmentation methods design specific adaptable strategies, they often lack task-specific adaptability.Most of those methods are always graph classification, e.g., AD-GCL [23] and AutoGCL [38] . In contrast, GraphLearner addresses this limitation by tailoring the augmentation strategy to graph node clustering in an unsupervised scenario. This ensures that the augmentation is not only learnable but also responsive to downstream task results, enhancing its overall effectiveness. 
conclusion: 
 In this work, we propose a fully learnable augmentation method for graph contrastive clustering termed GraphLearner. To be specific, we design a fully learnable augmentation with the structure augmentor and the attribute augmentor to dynamically learn the structure and attribute information, respectively. Besides, an adversarial mechanism is designed to keep cross-view consistency in the latent space while ensuring the diversity of the augmented views. Meanwhile, we propose a two-stage training strategy to obtain more reliable clustering information during the model training. Benefiting from the clustering information, we refine the learned structure with the high-confidence pseudo-label matrix. Moreover, we refine the augmented view with the cross-view sample similarity matrix to further improve the discriminative capability of the learned structure. Extensive experiments on six datasets demonstrate the effectiveness of our proposed method. In the future, it is worth trying the augmentors designed in GraphLearner for other graph downstream tasks, e.g., node classification and link prediction. 
: 
 and divide them into disjoint clusters without manual labels, has become a hot auxiliary task in information systems.With the strong capability of capturing implicit supervision, contrastive learning has become an important technique in deep graph clustering. In general, the existing methods first generate augmented graph views by perturbing node connections or attributes, and then keep the same samples in different views consistent while enlarging the difference between distinct samples. Although verified effective, we find that the performance of the existing graph contrastive clustering methods [15, 43] heavily depends on the augmented view. However, the existing augmentation methods are usually predefined and selected with a cumbersome search. The connection of augmentation and the specific downstream task is deficient. To alleviate this problem, in graph classification, JOAO [39] selects a proper augmentation type among several predefined candidates. Although better performance is achieved, the specific augmentation process is still based on the predefined schemes and cannot be optimized by the network. To fill this gap, AD-GCL [23] proposes a learnable augmentation scheme to drop edges according to Bernoulli distribution, while neglecting augmentations on node attributes. More recently, AutoGCL [38] proposes an auto augmentation strategy to mask or drop nodes via learning a probability distribution. A large step is made by these algorithms by proposing learnable augmentation. However, these strategies only focus on exploring augmentation over affinity matrices while neglecting the learning of good attribute augmentations. Moreover, previous methods isolate the representation learning process with the specific downstream tasks, making the learned representation less suitable for the final learning task, degrading the algorithm performance.To solve this issue, we propose a fully learnable augmentation strategy for deep contrastive clustering, which generates more suitable augmented views. Specifically, we design the learnable augmenters to learn the structure and attribute information dynamically, thus avoiding the carefully selections of the existing and predefined augmentations. Besides, to improve the reliability of the learned structure, we refine that with the high-confidence clustering pseudo-label matrix and the cross-view sample similarity matrix. Moreover, an adversarial learning mechanism is proposed to learn the consistency of embeddings in latent space, while keeping the diversity of the augmented view. Lastly, during the model training, we present a two-stage training strategy to obtain highconfidence refinement matrices. We summarize the properties of the existing graph augmentation algorithms in Table . 1. From the results, we could observe that our proposed method offers a more comprehensive approach.By those settings, the augmentation strategies do not rely on tedious manual trial-and-error and repetitive attempts. Moreover, we enhance the connection between the augmentation and the clustering task and integrate the clustering task and the augmentation learning into the unified framework. Firstly, the high-quality Table 1 : An overview of graph augmentation methods. "S" and "A" denotes the structure augmentation and attribution augmentation, respectively. Besides, "Opt" means that the augmentation is optimized by the downstream task. The key contributions of this paper are listed as follows:• By designing the structure and attribute augmentor, we propose a fully learnable data augmentation framework for deep contrastive graph clustering termed GraphLearner to dynamically learn the structure and attribute information. • We refine the augmented graph structure with the crossview similarity matrix and high-confidence pseudo-label matrix to improve the reliability of the learned affinity matrix. Thus, the clustering task and the augmentation learning are integrated into the unified framework and promote each other.• Extensive experimental results have demonstrated that our method outperforms the existing state-of-the-art deep graph clustering competitors. 
 
 Appendix: http://citeseerx.ist.psu.edu/index 


Decision: According to all the review comments, rebuttals, discussions and final ratings, the majority of the reviewers gave positive ratings to this paper and the concerns were well addressed. I am happy to recommend to accept this paper. Please carefully revise the final manuscript according to the comments and discussions.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach to contrastive deep graph clustering (CDGC) by employing fully learnable augmentations for generating high-quality, task-specific samples. It incorporates two learnable augmentors for attribute and structural information, and introduces high-confidence pseudo-label and cross-view sample similarity matrices to refine the learned affinity matrix. An adversarial learning mechanism ensures consistency and diversity, while a two-stage training strategy further enhances clustering reliability. Extensive experiments on benchmark datasets show that GraphLearner outperforms existing methods, demonstrating superior clustering accuracy and robustness.
strength_weakness: strength_weakness: 1. The paper introduces fully learnable augmentations for contrastive deep graph clustering, which is a significant advancement over traditional predefined augmentations. This dynamic generation of high-quality, task-specific samples enhances clustering performance.

2. The integration of high-confidence pseudo-label and cross-view sample similarity matrices to refine the learned affinity matrix is a novel approach that improves the reliability and discriminative power of the clustering process.

3. Extensive experiments on multiple benchmark datasets demonstrate that GraphLearner significantly outperforms existing state-of-the-art methods. The results validate the effectiveness and efficiency of the proposed approach in various graph node clustering scenarios.

4. The paper is well-structured and clearly explains the proposed methods, including detailed descriptions of the learnable augmentors, adversarial learning mechanism, and two-stage training strategy. This clarity aids in the reproducibility and understanding of the research contributions.
limitations: limitations: 1. The proposed fully learnable augmentation strategy and adversarial learning mechanism add significant complexity to the implementation. This might make it challenging for practitioners to adopt and integrate the method into existing systems without substantial effort.

2. While the paper demonstrates effectiveness on benchmark datasets, it does not extensively discuss the scalability of the approach for very large graphs. The computational and memory requirements for the proposed method could be a limitation for large-scale applications.

3. Typos: Figure 1 caption: "we design the learnable augmentors to to dynamically learn the structure and attribute information." (Repeated "to")
paper_summary: paper_summary: CDGC methods utilize contrast learning to group nodes, and high-quality contrast samples are critical for performance. Existing methods' enhancement samples are predefined based on human experience and irrelevant to downstream clustering tasks, leading to high labor costs and poor performance. GraphLearner is proposed to introduce a learnable enhancer to generate high-quality, task-relevant enhancement samples. GraphLearner contains two learnable enhancers for capturing attribute and structural information. A high-confidence pseudo-labeling matrix and a cross-view sample similarity matrix are introduced to enhance the reliability of the learned affinity matrix. During the training process, the difference in optimization objectives between the learning enhancer and the comparison learning network is noted, which needs to ensure the consistency of the embedding and the diversity of the enhancement samples, for which the adversarial learning mechanism is proposed.
strength_weakness: strength_weakness: 1. It proposes the GraphLearner framework, a fully learning data enhancement method for deep comparative graph clustering that dynamically learns structural and attribute information.
2. Optimizes the augmented graph structure using a cross-view similarity matrix and a high-confidence pseudo-labeling matrix to improve the reliability of the learned affinity matrix.
3. integrate the clustering task and augmented learning into a unified framework that mutually reinforces each other.
limitations: limitations: 1. The experimental dataset is insufficient and should be tested on larger graphs, e.g. ogbn-papers 100M, ogbn-products.
2. The Structure Augmentor and Attribute Augmentor mentioned in the Fully Learnable Augmentation Module module mentioned in the paper are not very generalizable and must be adapted for different network frameworks. I can understand that different backbones require special handling, but the runtime logic between MLP-based, GCN-based, and Attention-based may not be uniform enough. Also, I doubt these three methods will cover all cases, such as APPNP.
3. The lack of robustness of the model may also be a problem, and in Figure 3, we see that the sensitivity of the hyper-parameter α is too strong.
paper_summary: paper_summary: This paper proposes a graph node clustering method, named GraphLearner, which enhances contrastive graph node clustering by introducing a learnable structure and attribute augmentor, as well as a dual refinement module. The dual refinement module refines the augmented graph using a cross-view similarity matrix and a pseudo-label matrix.
strength_weakness: strength_weakness: 1. GraphLearner introduces a learnable way to augment the structure of the augmented graph, significantly improving contrastive graph node clustering performance compared to baselines that use predefined augmentations or only augment attributes in a learnable manner.
2. GraphLearner refines the augmented graph, making it more suitable for downstream clustering tasks.
3. The manuscript is well-organized, making it clear and easy to understand.
4. The experiments are generally comprehensive and the results are good.
limitations: limitations: 1. The introduction highlights that previous learnable augmentation methods neglect the learning of effective attribute augmentations. However, an experimental comparison of CCGC and CONVERT, which only augment attributes in a learnable way, with the proposed GraphLearner (when the structure augmentation is ablated) in Table 5, reveals that the attribute augmentation performance of GraphLearner might not be satisfactory.
2. There is a lack of explanation regarding how the experiments ablating both structure and attribute augmentations were conducted. If neither attribute nor structure is augmented, how is the contrastive learning carried out?
3. The Related Works section of the manuscript needs improvement. For instance, the involved baseline AutoSSL is not introduced in the manuscript.
paper_summary: paper_summary: ## **1. Summary.**
The authors aim that many graph node clustering methods always use the augmentations predefined by human experiences, which lacks the flexibility for the node clustering task. To improve the augmentation, the authors designed a learnable graph augmentation called GraphLearner.
strength_weakness: strength_weakness: ## **2. Strengths.**
1. The motivation is well described. The authors design a learnable augmentation method to improve the node clustering performance.
2. The authors state the properties of existing augmentation methods. Different with the existing graph node clustering method, the authors propose the augmentors could learn the structure and attribute during the training.
3. The experiments are widely conducted. The experiments in this paper include the comparison experiments, hyper-parameter analysis, ablation experiments, and visualization experiments.
limitations: limitations: ## **3. Some minor issues.**
After carefully reading the entire paper, it can be seen that it has been carefully prepared. I only have a few small issues to provide to the authors for further clearing.

**About experiments:** In Line 795, the authors provide many ablation experiments. The ablation experiments in Subsection 3.4.1 contain three components, i.e., (w/o) Aug_X, (w/o) Aug_S, and (w/o) Aug_S & Aug_X. However, a detailed setting should provided about this experiment for w/o Aug_X, Aug_S and (w/o) Aug_S & Aug_X. The current version lacks information. Moreover, the authors should give details about the loss function used in the first stage and the second stage. Which stage uses the loss function defined in Eq.(15)? What is the loss function used for another training stage? It makes me a bit confused. This version lacks a clear description of the loss function.

**About future work:** The author should analyze the shortcomings of the design method and the directions for future improvement and exploration in the conclusion.  This paper designs a learnable augmentation method for graph node clustering task.  Designing the structure and attribute augmentors are the advantages of this paper.  The shortcomings are not described in the paper. 

**Other small details:** There are some typos in this paper, Line 868, augment-free methods can be polished as augment-free method. Typos will affect the overall readability of a paper. Therefore, the authors should carefully check the paper.

**Overall, this is a good work, motivation, method presentation, and experiments are all carefully performed. If the authors can clear some my issue to improve the quality of this paper, I am willing to increase my rating.**