Unified Graph Augmentations for Generalized Contrastive Learning on Graphs

Authors: jiaming  zhuo
yintong  lu
hui  ning
kun  fu
bingxin  niu
dongxiao  he
chuan  wang
yuanfang  guo
zhen  wang
xiaochun  cao
liang  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 10:41:05

Abstract: In real-world scenarios, networks (graphs) and their tasks possess unique characteristics, requiring the development of a versatile graph augmentation (GA) to meet the varied demands of network analysis. Unfortunately, most Graph Contrastive Learning (GCL) frameworks are hampered by the specificity, complexity, and incompleteness of their GA techniques. Firstly, GAs designed for specific scenarios may compromise the universality of models if mishandled. Secondly, the process of identifying and generating optimal augmentations generally involves substantial computational overhead. Thirdly, the effectiveness of the GCL, even the learnable ones, is constrained by the finite selection of GAs available. To overcome the above limitations, this paper introduces a novel unified GA module dubbed UGA after reinterpreting the mechanism of GAs in GCLs from a message-passing perspective. Theoretically, this module is capable of unifying any explicit GAs, including node, edge, attribute, and subgraph augmentations. Based on the proposed UGA, a novel generalized GCL framework dubbed Graph cOntrastive UnifieD Augmentations (GOUDA) is proposed. It seamlessly integrates widely adopted contrastive losses and an introduced independence loss to fulfill the common requirements of consistency and diversity of augmentation across diverse scenarios. Evaluations across various datasets and tasks demonstrate the generality and efficiency of the proposed GOUDA over existing state-of-the-art GCLs.

Main paper: 
 introduction: 
 Owing to their effectiveness and efficiency, Graph Neural Networks (GNNs) have become a standard toolkit for processing various graph tasks such as node classification and graph classification [17, 34, 41, 40] . They typically follow a message-passing paradigm [10] , where the representation of each node is updated by aggregating the representations of its adjacent nodes and subsequently combining the aggregated representations with itself. In general, to produce discriminative representations, GNNs need to resort to the task-relevant labels (i.e., supervised information) to guide the network training, which limits their applicability in the label scarcity scenarios [24, 37, 43] . To overcome this limitation, Graph Contrastive Learning (GCL), a typical graph self-supervised learning architecture, has been developed to provide training guidance by capturing the self-supervised information contained in the graph [53, 32, 50, 2, 21, 39, 20] .Inspired by the design philosophy of contrastive learning in Computer Vision (CV) [4, 12] , GCLs adopt the same architecture, which consists of three components: augmentation, encoder, and contrastive loss [53, 32] . Thus, GCLs inherit the merit of enabling learning representations invariant to augmentation, which is achieved by maximizing the agreement between embeddings from different perturbations of the same graph [51, 52] . To further improve the representation capacity of GCLs, great endeavors have been made to design augmentations for the original graph, i.e., Graph Augmentations (GAs), which target nodes, edges, attributes, and subgraphs. Based on how information is processed, GAs can be divided into two categories: heuristic [53, 35, 32, 14] and learnable methods [47, 31, 21, 56] . The heuristic GAs modify graphs through the combination of fixed, random rules, such as attribute masking [53] , edge removing [32, 2] , and graph diffusion [14] . They tend to neglect the subsequent steps, namely encoding and contrastive optimization, hence leading to suboptimal performances. In contrast, learnable GAs leverage prior knowledge and feedback during training to refine augmentations, already surpassing base augmentations on many tasks. Notable contributions include GAs based on spectral methods [21] , and adversarial training [31] .Given their inherent and distinct characteristics, various networks and tasks require the meticulous selection of optimal GAs to improve model performance pivotally. However, most GCLs face several limitations regarding the selection: (1) Specificity. GCLs are typically tailored with specific GAs to meet the needs of particular scenarios, resulting in a lack of generality across diverse scenarios. For instance, node dropping (specifically, removing nodes and their associated edges), widely applied in graph-level tasks [47, 48] , could significantly compromise the integrity of graphs [36] , rendering it less suitable for node-level tasks. (2) High complexity. Either way, identifying and generating the scene-specific GAs impose a considerable computational burden on the models. For example, the set sampling method necessitates a validation of all combinations [45, 38] . Furthermore, the adversarial attack method [47, 31] entails recalculating contrastive losses, which takes a quadratic complexity of O(n 2 ). Besides, the spectral method requires Laplacian matrix decomposition [5] , which has a cubic complexity of O(n 3 ). (3) Incompleteness. Despite the promise of existing learnable GAs in optimizing for specific scenarios, their efficacy is limited by the finite range of GAs at their disposal. This paper seeks to break these limitations by proposing a unified GA module for GCLs. Toward this end, the mechanisms of existing GAs in GCLs are systemically investigated and reinterpreted from a message-passing perspective [10] . The conclusion is that, from the message-passing perspective, GAs uniformly induce attribute modifications within the neighborhoods of nodes, even though they appear diverse from the spatial perspective, as depicted in Fig. 1 . Therefore, the essence of GCLs is to learn node representations invariant to such local augmentation. Drawing from this insight, a novel Unified GA (UGA) module with a simple yet effective design is presented. It strategically interpolates an appropriate amount of Augmentation-Centric (AC) vectors in a graph-structured manner [55, 8] , where AC vectors are treated as another type of node, as illustrated in Fig. 2 . In theory, UGA is able to simulate the impact of the above four explicit GAs on target nodes by aggregating features from the AC vectors that capture the attribute variations within the neighborhood of these nodes.Building upon the proposed UGA module, a generalized GCL framework dubbed Graph cOntrastive UnifieD Augmentations (GOUDA) is presented to overcome the above challenges in existing GCLs. This framework adopts a typical dual-channel architecture [4, 49, 53] , corresponding to two distinct augmented graphs (views) with their respective AC matrices, as shown in Fig. 2 . To realize general utility, GOUDA proposes to capture the consistency and diversity across augmentations (defined in Section 3.3), which are essential and shared goals for GCLs to be applicable across diverse scenarios. To be specific, the objective function of GOUDA is twofold: (1) maximizing Mutual Information (MI) between representations from these distinct views. (2) maximizing the distributional difference between the AC matrices. The former is a fundamental principle behind classic contrastive losses and inherently ensures consistency, while the latter is a constraint to modulate diversity. In practice, GOUDA is instantiated by leveraging widely employed contrastive losses alongside a Hilbert-Schmidt Independence Criterion (HSIC)-based distributional independence loss. This design makes GOUDA more effective and efficient than GCLs with learnable GAs.The main contributions of this work are summarized as follows:• We investigate the mechanism of GAs in GCLs through the lens of message-passing.• We propose a lightweight GA module named UGA to simulate the impacts of GAs on nodes.• We introduce GOUDA, an efficient and generalized GCL framework, which captures both consistency and diversity across augmentations.• Extensive experiments and in-depth analysis demonstrate that GOUDA outperforms stateof-the-art GCLs across various public benchmark datasets and tasks. 
preliminaries: 
 This section briefly introduces the notations utilized throughout the paper. Subsequently, it outlines the essential components of the Graph Contrastive Learning (GCL) framework.Matrices (e.g., Q) are in bold capital letters, vectors (e.g., q i,: , which denotes the i-th row of Q) are in bold lowercase letters, scalars (e.g., q i,j , which represents the entry of Q at the i-th row and the j-th column) are in lowercase letters, and sets (e.g., N ) are in calligraphic letters.For a general-purpose description, this paper considers an undirected attribute graph G(V, E), where V stands for the node-set containing n node instances {(x v , y v )} v∈V . And X ∈ R n×f and Y ∈ R n×c denote the attribute matrix and label matrix of node v, respectively, where f and c is the numbers of attirbutes and labels, respectively. Also, E = {e i } m-1 i=0 terms the edge set containing m edges. In general, the adjacency matrix A ∈ R n×n is employed to describe the graph topology, such that the matrix form of the graph can be expressed as G(A, X). Moreover, H ∈ R n×d terms the graph representation, where d terms the dimension of the representation.Graph Augmentations. Drawing on the successful experience of image augmentation in Computer Vision (CV) [4, 15] , Graph Augmentation (GA) [51] is introduced in graph learning to address the challenge of data scarcity. In the typical GCL frameworks, the input graph G(A, X) is processed through two separate perturbations (GA procedures), formulated as t i (G) : G(A, X) → G i (A (i) , X (i) ), to generate its two views (augmented graphs), denoted as G 1 (A (1) , X (1) ) and G 2 (A (2) , X (2) ). Based on the perturbed information, GAs can be broadly classified into four main categories: node augmentation [47, 48] , edge augmentation [53, 54, 32, 31] , attribute augmentation [19, 53, 44] , and subgraph augmentation [47, 13] . An overview of GAs can be found in Section B.Graph Encoders. For efficient processing and analysis, the graph encoders are leveraged to transform raw topology and attribute information of the input graph into low-dimensional vector representations. Most graph encoders in GCLs follow a message-passing paradigm [10] , which typically involves two primary processes: aggregation and combination. During these steps, each node iteratively updates its representations by aggregating and combining the node features from its neighborhoods, that isEQUATIONwhere h l v terms the representations of node v in the l-th layer and N v denotes the set of neighboring nodes of node v. In prevalent GCLs like GRACE [53] , a two-layer GCN [17] is adopted, where the Aggregation(•) and Combination(•) functions are implemented via average function. Thus, there isEQUATIONwhere σ(•) denotes the nonlinear activation functions, such as ReLU(•), and Â = A + I n stands for the adjacentcy matrix with self-loops, and D is the corresponding degree matrix, and W l represents the parameter matrix for the l-th layer. Therefore, for the two augmented graphs, their representations can be obtained by computing H (1) = GCN 2 (G 1 (A (1) , X (1) )) and H (2) = GCN 2 (G 2 (A (2) , X (2) )).Input graph 𝐺(𝐀, 𝐗) Contrastive losses. In line with the InfoMax principle [22] , various contrastive losses are incorporated in GCLs, guiding the training of graph encoders by maximizing the Mutual Information (MI) between the encoded representations on two augmented graphs. Specifically, given two representations H (1) and H (2) obtained from a shared encoder g Θ , the general objective of GCL is expressed as GCL: arg max Θ I(H (1) ; H (2) ), whereEQUATIONwhere I(X; Y ) represents the MI between X and Y . In general, the MI can be approximated using a lower bound estimator, i.e., the InfoNCE loss [33] , in the GCLs [53, 20] . This loss can be classified as a sample-level loss because it operates on the sample dimension of the representation matrix. In contrast, Barlow Twins loss [49] , another widely employed loss, is designed to remove redundancies among features and hence can be categorized as a feature-level loss. Both losses are used to implement the proposed framework. Section C provides detailed descriptions of these losses. 
motivations: 
 As previously mentioned, Contrastive Learning (CL) seeks to learn image representations invariant to augmentations by encouraging the agreement between embedding vectors from the different image distortions. Due to the employment of identical loss functions, typical Graph Contrastive Learning (GCL) inherits the above representation capabilities from CL. Nonetheless, GCLs should emphasize the local invariance owing to the application of graph encoders.The essence of the graph encoder is to explore the locality of graphs. To be specific, graph encoders in GCLs (generally a 2-layer GCN) follow the message-passing paradigm where node representations are updated in a local aggregation and combination manner, as detailed in Section 2. Given the localizing property of the graph encoder, GAs (i.e., node, edge, attribute, and subgraph augmentations), which are typically viewed as global operations in various forms, can be uniformly reinterpreted as attribute modifications in the neighborhoods of nodes, namely, local augmentations, as illustrated in Fig. 1 .1) Edge augmentation involves adding and removing perturbed edges in graphs, equivalent to inserting or masking the attributes of nodes connected by these edges in the neighborhood of impacted nodes. For example, the shown edge removing results in the complete attribute masking of partial 2-hop neighbors (nodes 1, 4, 7, and 8) of the target node 0 during message passing. 2) Attribute augmentation essentially replaces the attributes of perturbed nodes in the graph with new ones, which can be viewed as perturbing the neighborhoods of nodes that contain these perturbed nodes. The attribute masking example shows that during the aggregation phase, the attributes of neighbors (nodes 1, 3, 4, 5, 6, 7, and 8) on the 2-hop computation graphs of node 0 are masked. 3) Subgraph augmentation is to modify the specific subsets of the graph (including its edges and attributes), which also can be seen as the attribute perturbation in the neighborhoods of target nodes. For node 0, the shown subgraph cropping causes removing nodes 2, 4, 6, and 8 from the 2-hop neighborhood during the message-passing phase. Note that node augmentation is a specific case of subgraph augmentation, where the subset size is one. Thus, the above conclusion regarding subgraph augmentation applies to it.In short, the mechanism of GAs in GCLs is to induce attribute modification in the neighborhoods of nodes. Thus, the essence of GCLs is to learn representations invariant to such local augmentations.Motivated by the above insights, a unified graph augmentation module dubbed UGA is introduced to implement augmentation efficiently and flexibly. The primary idea is to introduce a collection of Augmentation-Centric (AC) vectors for nodes to simulate and exert the impact of GAs on these nodes, namely, attribute variations in the neighborhood of these nodes. A straightforward implementation of UGA is to align AC vectors one-to-one with nodes, match the size of their features, and then perform feature summation to achieve the desired augmentation.Given the input graph G, the above implementation can be formulated asEQUATIONwhere G * denotes an augmented graph derived from the UGA funtion t(G, Q), and Q ∈ R n×f terms the matrix of AC vectors q v ∈ R 1×f , i.e., AC matrix, and f is the dimension of node attributes.Fig. 2 (a) provides an illustrative example and explains the equivalence between the proposed UGA module and an explicit GA (edge removing). From this, it can be concluded that the UGA module can effectively substitute GAs as long as the combined AC vectors are the representations of cumulative attribute variations within the neighborhoods of nodes induced by these GAs.Theorem 3.1. Assuming any augmented graph G * (A ( * ) , X ( * ) ), where A ( * ) ∈ A and X ( * ) ∈ X, with A and X represent the candidate spaces for the augmented adjacency matrix and attribute matrix, respectively, in the proposed implementation of UGA (Eq. 4), there exists an AC matrix Q that meetsEQUATIONwhere g Θ stands for the graph encoder.This theorem suggests that the proposed UGA module can be equivalent to any existing GA (including node, edge, attribute, and subgraph operations), thereby demonstrating its unifying capability to GAs. Proofs for this theorem are presented in Section D.1. Furthermore, the UGA module possesses an attractive characteristic: adaptability, since the AC vectors are capable of dynamically capturing taskrelevant perturbation information throughout the training process. Nonetheless, this implementation introduces numerous parameters proportional to the network size, resulting in a significant increase in complexity and the risk of overfitting. To address this limitation, the proposed UGA is reimplemented in a graph-structured manner, where a modest parameter set is utilized, as shown in Fig. 2(b) .Shared AC vectors. In graphs, long-range dependencies signify the beyond-local interactions among nodes, represented by similar node attributes and neighborhood patterns [23, 40, 46] . Therefore, it is reasonable to assume that a group of interdependent nodes would benefit from the same optimal GAs. Thus, a shared AC matrix Q = [q i,: ] k-1 i=0 is introduced, where k ≪ n. Propagation mode. AC vectors propagate their features to nodes via a general attention mechanism, in which structural features (e.g., positional/structure encodings [1, 7] ) are employed to calculate the attention scores. Formally, the proposed UGA module can be reformulated asEQUATIONwhere b v,i stands for the propagation weight from i-th AC vector to node v within matrix B ∈ R n×k , and f([x v ||e v ]) ∈ R 1×f terms an integrated representation of node v, which concatenates the node attributes x v and structural features e v ∈ R t . This paper adopts t-steps random-walk encodings [7] as the structure features. f(•) : R f +t → R f denotes a projection layer. Input graph 𝐺(𝐀, 𝐗) Processing. Keeping a subset of the salient connections facilitates propagation and enhances computational efficiency. Thus, propagation weights below a predefined threshold are zeroed out, namelyShared g ! (, )EQUATIONwhere ε terms a threshold value. Then, the obtained weight B is applied in propagation, per Eq.6.Building upon the UGA module, a novel GCL framework named Graph cOntrastive UnifieD Augmentations (GOUDA) is proposed to achieve generality across diverse tasks and graphs. It utilizes the standard two-channel architecture (in Eq. 3), with each channel generating an augmented graph through the UGA module and its AC matrix, as depicted in Fig. 2 . Unlike traditional GCLs, GOUDA introduces a term to constrain the two AC matrices (denoted as Q and P).Specifically, GOUDA optimizes the following objective function:EQUATIONwhere I(X; Y ) stands for the Mutual Information (MI) between X and Y , and g Θ denotes a graph encoder shared between two views (or channels). G 1 = t (G, Q) and G 2 = t (G, P) represent two augmented graphs, and D (Q, P) denotes the constraint between Q and P.Definition 3.2. (Consistency across augmentations). Let H (i) and H (j) denote the representations of graphs G i , G j ∼ G ω , respectively, where j ̸ = i, and G ω denotes the family of graphs derived from a series of parametric graph augmentations. Consistency across augmentations for node v is defined asEQUATIONwhere S(X, Y ) terms the distributional similarity between X and Y .This consistency implies that augmentation should minimally impact the similarity between representations from different augmented graphs for the same nodes to preserve the intrinsic semantic integrity of the nodes. Note that the first term in the objective function of GOUDA, namely the MI maximization, essentially is a constraint for semantic consistency. Thus, the augmentation learned by the UGA module is capable of ensuring the desired property.Definition 3.3. (Diversity across augmentations). Given two augmented graphs G i (A (i) , X (i) ) and G j (A (j) , X (j) ) ∼ G ω , where j ̸ = i, let N k v represents the k-hop subgraph centered at node v, and let D(, ) stands for the measure of distributional difference. Diversity across augmentations is defined asEQUATIONwhereX (i) N k v ∈ R nv×fstands for the attribute matrix of nodes in the k-hop subgraph of node v, and n v is the number of neighbors for node v. COM(•) terms a combination function, such as sum(•).This definition is based on the conclusion in Section 3.1, namely, the mechanism of GAs in GCLs is to modify attributes within the node neighborhoods. Accordingly, another objective for augmentation is the minimization of the local attribute overlap between augmented graphs, ensuring the model does not overfocus on the specific features of a single distribution.In the proposed GOUDA framework, local attribute variations for each node are represented by AC vectors, e.g., q v and p v , with the augmentation being generated by aggregating features from these vectors. Therefore, the diversity across augmentations can be quantified using two AC matrices (Q and P) corresponding to two distinct views, which is supported by the following analysis.Theorem 3.4. Let D(X, Y ) = ∥X -Y ∥ 2F stands for the distributional difference, and let COM(•) = sum(•) terms the combination function. In GOUDA (in Eq. 8), the diversity across augmentations (in Eq. 10) can be approximated by the distributional difference between AC matrices Q and P, that isEQUATIONwhere qt,: = b q t,: Q denote the features propagated from AC matrices Q to node t. Theorem 3.4 shows that the diversity across augmentations can be controlled by imposing constraints on the AC matrices, particularly through the second term in the objective function of GOUDA. Refer to Section D.2 for the proofs. In brief, maintaining a balance between consistency and diversity across augmentations is crucial for the effectiveness of GCLs. Specifically, diversity encourages exploring and exploiting the local attribute variations while consistency anchors the learned representations to the original semantics.This subsection introduces a practical implementation of the proposed GOUDA framework (Eq.8). The overview of this framework is depicted in Fig. 2 , while the step-by-step procedure is detailed in Algorithm 1. The objective of GOUDA is to learn the discriminative and robust representations. To achieve this, it seeks to train the graph encoder g Θ to maximize the Mutual Information (MI) between representations from two augmented graphs G 1 = t(G, Q) and G 2 = t(G, P), while simultaneously maintaining consistency and diversity in the augmentation process.The first term of GOUDA is implemented utilizing the sample-level InfoNCE loss (in Eq. 22), which serves as a lower bound estimator for MI, and the feature-level Barlow Twins loss (in Eq. 24). This term is denoted as contrastive loss L contrast . Owing to limited space, the above losses are introduced in Section C.A distributional independence loss is introduced to instantiate the second term of GOUDA. Specifically, the Hilbert-Schmidt Independence Criterion (HSIC) [11] is adopted to measure the statistical dependence between two augmentation distributions. Furthermore, the Gram matrices derived from HSIC are constrained to minimize their off-diagonal elements. To be concrete, the independence loss is formulated asEQUATIONwhere K and L stand for the Gram matrices of Q and P, respectively, defined by k i,j = κ(q i,: , q j,: ) and l i,j = κ(p i,: , p j,: ). In practice, the kernel function κ(•) is defined as the linear kernel, specifically k i,j = q i,: q T j,: . Additionally, R = I n -1 n 11 ⊤ represents the centering matrix, where I ∈ R n×n and 1 n ∈ R n×1 denote the identity matrix and all-one column vector, respectively. β 1 and β 2 represent two hyper-parameters. Minimizing this term serves two purposes: on the one hand, it enhances the diversity across augmentations by amplifying the differences between two distributions, and on the other hand, it avoids trivial solutions by increasing the differences among the augmentation elements (q i,: ) within each distribution.Objective. The overall objective function of GOUDA is a weighted sum of these two terms, that isEQUATIONwhere γ denotes a hyperparameter used to trade off two terms. This subsection evaluates the complexity of the proposed GOUDA framework in comparison to the baseline GCLs configured with learnable GAs, including SPAN, JOAO, and AD-GCL. As illustrated in Tab. 1, GOUDA introduces lighter computational overhead compared to these baselines. For a detailed description of the complexity, refer to Section E.4.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have included complete and executable code within the supplemental material, ensuring the reproducibility of our results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 This section evaluates the effectiveness and generality of the proposed GOUDA through a comprehensive comparison against multiple baselines across tasks at both the node-level (node classification and node clustering) and the graph-level (graph classification) tasks. Furthermore, it conducts several additional experiments to deepen the understanding of this framework. For an exhaustive account of datasets, baselines, configurations, and hyper-parameters, refer to Section E.Datasets. The experiment utilizes ten benchmark datasets, namely: Cora [28] , CiteSeer [28] , PubMed [28] , Wiki-CS [25] , Photo [29] , Computers [29] , and Physics [29] for node-level tasks, and IMDB-B [42] , IMDB-M [42] , and COLLAB [42] for graph-level tasks. See Section E.1 for dataset descriptions.Baselines. The baseline models comprise two supervised graph neural networks (GCN [17] , GAT [34] ) and eleven self-supervised graph learning models (DGI [35] , GMI [26] , MVGRL [14] , GRACE [53] , GCA [54] , BGRL [32] , CCA-SSG [50] , GBT [2] , SPAN [21] , DSSL [39] , HomoGCL [20] ) for node-level tasks. Four self-supervised learning models are compared (InfoGraph [30] , GraphCL [48] , JOAO [47] , AD-GCL [31] ) for graph-level tasks. Refer to Section E.3 for model introductions.Node Classification. It can be observed from Tab. 2, which exhibits the results of node classification tasks, that the proposed GOUDA outperforms the unsupervised baselines in six of the seven datasets. This demonstrates the superiority of GOUDA. Furthermore, on the CiteSeer dataset, notable performance improvements are observed with both models, GOUDA-IF and GOUDA-BT, surpassing the baselines GRACE and GBT. Specifically, the accuracy of GOUDA-IF surpasses that of GRACE by 3.14%, and similarly, the accuracy of GOUDA-BT exceeds that of GBT by 1.90%. Note that the baselines GRACE and GBT adopt identical encoders and contrastive losses as GOUDA-IF and GOUDA-BT, respectively. Therefore, the observed performance improvement can be attributed to the adaptive modeling capacity for augmentations of the proposed GOUDA.Node Clustering. One can draw two conclusions from the observations in Tab. 3. Firstly, it is evident that GOUDA consistently surpasses all baselines (e.g., GRACE, and GBT) across all datasets, which illustrates the superior representation capacity of GOUDA. This can be attributed to the selfadaptive learning ability of the proposed module UGA. Secondly, GOUDA-IF consistently outperforms GOUDA-BT, suggesting that within the proposed GOUDA framework, the InfoNCE loss more effectively captures local information for clustering than the BarlowTwins loss. Graph Classification. The results of this experiment are presented in Tab. 4 and Fig. 3 . Firstly, it can be observed from Tab. 4 that GOUDA outperforms the baselines regarding classification performance, which illustrates the general validity of GOUDA. In particular, GOUDA-IF and GOUDA-BT surpass the second-place MVGRL by 1.02% and 2.6%, respectively, on the IMDB-B dataset, which highlights the superiority of GOUDA. Moreover, GOUDA exceeds the GCLs employing learnable GAs, i.e., JOAO, and AD-GCL. This can be due to the unified ability of UGA to integrate diverse GAs, which provides GOUDA with broader augmentation options than the baseline. Secondly, as illustrated in Fig. 3 , GOUDA achieves superior performance and consumes less time than the baselines. Specifically, two triangles, indicating the proposed GOUDA, are superior left and above the other shape in the figure. This implies GOUDA is lightweight, aligning with conclusions in Section 3.5. Besides, UGA introduces modest memory usage, which promises the scalability of GOUDA. Robustness Analysis. This experiment aims to evaluate the robustness of GOUDA against topology attacks (adding edges) and attribute attacks (flipping attributes). According to results in Fig. 4 and Fig. 5 , several conclusions can be derived. Firstly, compared to the baselines using the same contrastive losses, GOUDA consistently achieves performance gains at all perturbation rates. It demonstrates the robustness of GOUDA against both topology and attribute attacks. This is attributed to the greater adaptability of the UGA module, stemming from its integration of augmentation and contrastive updating over random GAs. Secondly, attribute attacks cause more severe performance degradation than topology attacks, even for our proposed GOUDA. This could be because the node attributes, rich with class-discriminative information, are erased essential identification info during attacks.Ablation Study. This experiment aims to evaluate the contribution of individual components. To be specific, it introduces two variants: one without the structure features (in Eq. 6) and another without the independence loss (in Eq. 13). From Fig. 6 , it is observable that the performance declined in both model variants compared to the complete model, which illustrates that the efficacy of GOUDA stems from the collective contribution of all components. Besides lacking Independence loss, GOUDA-IF performs inferior to GOUDA-BT, implying that InfoNCE might drive the model toward excessive consistency, diminishing its discriminative power. This highlights the critical role of independence loss in preserving diversity across augmentations. Parameter Sensitivity Analysis. These experiments are performed to offer an intuitive understanding of hyper-parameter selection. Firstly, as depicted in Fig. 7 , which illustrates performance variance for varying k, GOUDA achieves consistently stable performances across {5, 10, 20}. Notably, the performance variation on these datasets remains minimal, staying within a 2% margin. Thus, GOUDA has low sensitivity to parameter k. Moreover, this parameter requires no significant value for GOUDA to perform well; a setting as low as 5 suffices. However, a value of 1 is inadequate due to the absence of augmentation diversity. The analysis of other hyper-parameters is given in Section E.6. 
conclusions: 
 In this paper, we present UGA, a unified Graph Augmentation (GA) module that addresses the issues in existing GAs, including specificity, complexity, and incompleteness. Motivated by the local attribute-modifying characteristics of GAs, UGA introduces a moderate number of Augmentation-Centric (AC) vectors to simulate GA impact on nodes. We further propose GOUDA, a generalized Graph Contrastive Learning (GCL) framework built on UGA. GOUDA promotes both consistency and diversity across augmentations by employing a contrastive loss and an independence loss, respectively. Extensive evaluations demonstrate the generality and efficiency of GOUDA. However, the robustness analysis suggests a scope for enhancement in its robustness against attribute attacks. Future research could explore multi-modal learning methods that fuse diverse structural features into node attributes, aiming to better preserve discriminative information and thus enhance robustness. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided detailed descriptions of our experimental setup in Section E, including data splits, hyperparameters, and optimizer, etc.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have reported error bars representing the standard deviation of our experimental results (e.g., Fig. 6 ).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have provided the computational resources used for all experiments in Section E.5.2.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Due to the nature of this work, there may be no potential negative social impact that is easily predictable.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not involve releasing data or models that pose a high risk for misuse, so no specific safeguards are necessary.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have accurately credited the sources and provided URLs in Section E.1 and Section E.3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The supplemental material includes the zip file of our code. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This paper does not involve crowdsourcing or research with human subjects, so this information is not applicable. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: This research does not involve human subjects, so IRB approvals or equivalent reviews are not required. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To demonstrate the broad applicability of the proposed framework GOUDA, this paper implements two models, namely GOUDA-IF and GOUDA-BT. Specifically, GOUDA-IF employs a prevalent node-level contrastive loss ( i.e., InfoNCE), while GOUDA-BT utilizes a feature-level contrastive loss (i.e., BarlowTwins). GOUDA-IF is illustrated as an example to describe the entire algorithm, as presented in Algorithm 1. Accordingly, GOUDA-BT can be described by substituting the contrastive loss L contrast (, ) in line 3 of this algorithm.Algorithm 1: GOUDA-IF Input: graph G(A, X), hyperparameters γ, β 1 , β 2 and τ .Output: node representations H (1) ∈ R n×d and H (2) ∈ R n×d . Initialization: graph encoder g Θ (, ), projection heads g(•), and the matrices of Augmentation-Centric (AC) vectors Q ∈ R k×f and P ∈ R k×f .while not converged do % Augmentation % 1. G 1 ← t(G, Q) and G 2 ← t(G, P) via Eq. 6 and Eq. 7; % Encoding % 2. H (1) ← g Θ (G 1 ) and H (2) ← g Θ (G 2 ) via Eq. 2; % Calculating loss % 3. L ← L contrast (H (1) , H (2) ) + γL indep (Q, P) via Eq. 13; % Optimizing % 4. Θ ← Adam(L, Θ); end return H (1) ∈ R n×d , H (2) ∈ R n×d and g Θ (, );Categorized by the type of graph information they manipulate, Graph Augmentations (GAs) can be broadly divided into four categories: node augmentation, edge augmentation, attribute augmentation, and subgraph augmentation. The detailed introduction and examples are given below.Node Augmentation. This augmentation generally creates the new graph by dropping or adding the perturbated nodes and the edges that connect to these perturbated nodes of the input graph, as shown in Fig. 1(a) . Employing the adjacent matrix to formally represent the graph topology, the edge augmentation can be denoted by G n (A ( * ) , X ( * ) ). Therefore, these can be formulated as Node Dropping :EQUATIONNode Adding :EQUATIONwhere V denotes the perturbated node set, X stands for the attributes of these nodes, and Ē terms the set of edges connected to these nodes. The operator is widely used on graph classification [48, 47] .Edge Augmentation. Unlike node augmentation, the edge augmentation, denoted by G e (A ( * ) , X), exclusively operate on edges. It involves either removing perturbated edges from or adding perturbated edges to the input graph, as indicated in Fig. 1(b ). These can be expressed as Edge Removing :EQUATIONEdge Adding :EQUATIONwhere Ē represents the set of perturbated edges, which is randomly determined [53, 54, 32, 20] or adaptively learned [31] during each training epoch.Attribute Augmentation. Typically, the attribute augmentation (expressed by G a (A, X ( * ) )) generates the new graph by masking (in Fig. 1(c )) or corrupting the raw node attributes. These can be described as Attribute Masking :EQUATIONAttribute Corrupting :EQUATIONwhere M ∈ R n×f stands for the mask matrix and δ v terms the noise vector for node v, which is updated iteratively by adversarial training [19, 44] .As typical graph-level operators, the subgraph augmentation crops out subgraphs (in Fig. 1(d) ) or inserts additional subgraphs to create new graphs, as followsEQUATIONSubgraph Inserting :EQUATIONwhere Ḡ( Ā, X) stands for the perturbated subgraph. The subgraph augmentation is mostly used for graph-level tasks [47, 13] .The contrastive loss serves as a crucial technique that enhances data representation through discrimination. Generally, it operates on two levels of representation matrices: the sample level [4, 53] , where it aligns the representations of positive samples and uniformly distributes all representations, and the feature level [49, 2] , where it targets reducing redundancy between features.Sample-level contrastive losses. In the nascent stages of research, the designs of contrastive losses were inspired by the success of contrastive learning (CL) in computer vision (CV) [4] . To be specific, this type of contrastive loss aims to minimize the distance between the anchor sample and positive samples while maximizing the distance between the anchor sample and negative samples [33] . As a typically sample-level contrastive loss, InfoNCE loss [33] classifies the embeddings of the same node from different views as positive samples while treating the embeddings from all other nodes as negative samples. This loss can be formulated asEQUATIONwhere ℓ h (1) v , hEQUATIONEQUATIONv ,hEQUATIONv ,hEQUATIONwhere Φ(h v , h u ) = s (g(h v ), g(h u )) stands for the feature similarity function, and g(•) represents the projection heads [4] , and s(•) terms the consine similarity. τ denotes the temperature coefficient.Feature-level contrastive losses. This type of loss is designed to directly optimize contrastive losses in the feature space, bypassing the need for defining explicit positive and negative samples, thus overcoming sample selection challenges. Barlow Twins (BT) [49] and CCA-SSG [50] are two notable approaches aimed at improving feature representation learning by minimizing redundancy between feature dimensions. BT enforces the mutual information matrix computed on the features from two different views to approximate the identity matrix, ensuring that the learned representations are free from redundant information. This can be formulated asEQUATIONwhere λ denotes a hyperparameter to tradeoff two terms.CCA-SSG not only pushes the covariance matrices from each view to approach an identity matrix but also enhances feature consistency across views to learn informative representations of both unique and shared data characteristics.D.1 Proofs for Theorem 3.1For the sake of clarity, let us briefly describe the whole process. Firstly, the proof proceeds from the premise of a one-layer graph encoder without activation functions. For node-level tasks, the graph encoder g Θ (, ) is configured as a one-layer GCN. For graph-level tasks, g Θ (, ) is set as a one-layer GIN with a sum pooling sum(•). Subsequently, the obtained conclusions are generalized to the case of multi-layer encoders.Specifically, for the one-layer graph encoders, the encoding process for node representations (H) and graph representations (h) can be expressed as Node representations:EQUATION)where Ã = D-1 2 Â D-1 2 denotes the normalized adjacency matrix and ϵ terms an learnable parameter. In the proposed UGA implementation, both node and graph representations can be decoupled into two terms: the original representations (represented as H and h), which can be regarded as directly calculated from Eq. 25 and Eq. 26, respectively, and the augmented representations (denoted as △H q and △h q ). To be specific, two terms of the node representations can be expressed asH q = GCN (A, X + Q) = Ã • (X + Q) • W = Ã • X • W + Ã • Q • W = H + △H q . (27)Similarly, two terms of the graph representations can be described asEQUATIONNext, to establish Theorem 3.1, Lemma D.1 is introduced. Lemma D.1. For any GA t(•) applied to the input graph G(A, X), it can be decoupled to a series of three augmentations: attribute augmentation (t a (X)), edge augmentation (t e (A)), and subgraph augmentation (t s (A, X)) containing node augmentation [8] .Accordingly, the candidate spaces (i.e., A and X) can be created through these three augmentations. Theorem 3.1 can be proven based on this lemma by establishing the following three propositions. Proposition D.2. From the message-passing perspective of the graph encoder g Θ (, ), the proposed implementation of UGA, which is expressed as Eq. 4, can be equivalent to any attribute augmentation t a (X), that is gEQUATIONwhere X ( * ) = t a (X) stands for the augmented node attirbutes.Proof. Firstly, let us discuss the equivalence for node representations. To establish Eq. 29, the goal is to identify Q such that H q = H a . Let △X = X -X ( * ) denote the variance in attributes resulting from attribute augmentation, H a can be decomposed into two terms: the original representations H and the augmented representations △H a . This decomposition can be expressed asEQUATIONTherefore, the proof shifts from establishing H q = H a to demonstrating ∆H q = ∆H a . It becomes evident that this equivalence holds true under the condition q i,j = ∆x i,j , thus ensuring that ÃQW = Ã∆XW.Moreover, this derivation highlights that the conclusion remains consistent regardless of the encoder chosen. Hence, the solution q i,j = △x i,j holds for the node representations encoded using GIN.Besides, the conclusions drawn at the node level, unaffected by the choice of readout functions (e.g., mean and sum), are equally applicable to the graph level. This insight extends our solution to graph representations h, thereby completing the proof.Proposition D.3. From the message-passing perspective of the graph encoder g Θ (, ), the proposed implementation of UGA, which is expressed as Eq. 4, can be equivalent to any edge augmentation t e (A), that isEQUATIONwhere A ( * ) = t e (A) denotes the adjacency matrix obtained from edge augmentation.Proof. Consistent with the method adopted in the above proofs, the edge-augmented representations (H e and h e ) are first computed. Next, the equivalence of these representations with UGA counterparts (H q and h q ), respectively, is demonstrated.Let △A = A -A ( * ) denotes the topology variance caused by edge augmentation, H e can be decoupled into two terms as follows.H e = GCN (A + △A, X)EQUATION)where △ Ã = Ã -Ã( * ) terms the topology variance between the adjacency matrix and its augmented version, both of which are normalized. Therefore, this demonstration solely necessitates establishing the equivalence between △H q and △H e , particularlyEQUATIONThrough the utilization of Cayley-Hamilton theorem [6] , which asserts that every matrix adheres to its own characteristic polynomial, that isEQUATIONwhere {c i } n i = 1 stands for the set of polynomial coefficients. Thus, the inverse of matrix Ã can be expressed asEQUATIONBased on it, the equivalent between the proposed UGA and edge augmentation (denoted as Eq. 33) can be demonstrated if it holds thatEQUATIONFurthermore, the solution derived for node representations is directly applicable to graph representations as well, with the sole modification being the substitution of Ã with A + (1 + ϵI). In light of the above analysis, this proposition is proven.Proposition D.4. From the message-passing perspective of the graph encoder g Θ (, ), the proposed implementation of UGA, which is expressed as Eq. 4, can be equivalent to any subgraph augmentation t s (A, X), that is gEQUATIONwhere A ( * ) , X ( * ) = t s (A, X) stand for the adjacency matrix and attribute matrix obtained from subgraph augmentation.Proof. Note that the subgraph augmentation is typically tailored for graph-level tasks. For node-level tasks, specific subgraph augmentation (where the number of nodes remains unchanged) can be regarded as a type of edge or attribute augmentation, such as masking all attributes of nodes in the perturbated subgraph. Hence, based on Proposition D.2 and Proposition D.3, it is not hard to conclude that this proposition holds.Recall that within our UGA, the graph representation h q ∈ R 1×d is articulated as h q = h + △h q . This can be formulated asEQUATION)Furthermore, let us assume that the original graph encompasses k subgraphs. The corresponding subgraph-augmented graph, in turn, encompasses m -1 subgraphs, which can be formulated asEQUATIONwhere A i denotes the adjacency matrix of the i-th subgraph and X i terms the corresponding attribute matrix. Thus, for the augmented subgraphs {G(A t , X t )} m-1 i=0 , the graph representation h s can be formulated asEQUATIONLet △h = h sh denotes the representation variance caused by subgraph augmentation, there isEQUATIONwhere Ω denotes an indicator vector. If m -1 < k, the subgraph augmentation typically refers to the subgraph corrupting (denoted as Eq. 20). Thus, for the t-th perturbated subgraph, there is Ω t = -1.And if m -1 > k, is generally understood as the subgraph inserting (formulated as Eq. 21), Ω t = 1. Additionally, if the subgraph is not changed, Ω t = 0.Next, we aim to identify a solution for q that satisfies the following conditions:EQUATIONThis can be further formulated asEQUATIONwhere d = [d 0 + 1 + ϵ, . . . , d n-1 + 1 + ϵ] ∈ R n stands for a degree vector.In the general case, assuming the absence of isolated nodes within the graph, the degree vector d of the input graph is devoid of zero elements. Consequently, a solution for Q can be formulated asEQUATIONwhereH = m-1 t=0 Ω t • sum((A t + (1 + ϵ)I) • X t) and D + = 1 dd ⊤ d denotes the Moore-Penrose pseudoinverse of d. Thus, equivalence can be established if each element q i,j in AC matrix Q satisfies the following condition:EQUATIONTherefore, the proof ends.Extension to multi-layer graph encoders. Following the discussion on single-layer graph encoders, solutions for multi-layer graph encoders are identified. Initially, as discussed in [18] , various GNNs can be formulated asEQUATIONwhere S denotes the diffusion matrix, exemplified by Ã for GCN and A + (1 + ϵ)I for GIN. In addition, W represents the projection layer, such as the linear projection utilized in both GCN and GIN. Here, the nonlinear activation function between layers is not considered. With this architecture, the node representations at the k-th layer can be formulated asEQUATIONwhere S = l i=0 S l terms the product of adjacency matrices. Similarly, W = l i=0 W l represents the product of linear projection matrices. Given that the aforementioned conclusions are independent of the forms of these two matrices, it is not hard to prove that by substituing A and W with S and W in Eq. 25 and Eq. 26, respectively, Proposition D.2, D.3, and D.4 still hold true.Proof. Firstly, note that two augmented graphs are from the same input graph G(A, X) without loss of the information in this graph (especially the edges and attributes). Therefore, there is A (1) = A (2) and X (1) = X (2) . Accordingly, the diversity can be transformed intoEQUATIONGiven the conditions b q t,: = σ(xt,: Q ⊤ ) where σ denotes the softmax function, we can calculate thatEQUATIONt,:EQUATIONt,:EQUATIONEQUATIONt,:EQUATIONFor clarity, x t,: is employed to represent both xt,: and x(2) t,: . In light of the consistency constraint within the GOUDA framework, the difference between Q and P is minimal, which can be interpreted as a minor perturbation ∆, such that Q = P + ∆. Therefore, Eq. 53 can be further reformulated asEQUATION)Given that ∆ is considered to be small, the terms ∆ 2 and x t,: ∆ ⊤ can be neglected. Moreover, it can be assumed that the product of x t,: ∆ ⊤ with P and ∆ is insignificant in comparison to the other terms. Hence, the above formulation can be simplified asEQUATIONSince t∈Nv∪v x t,: P ⊤ is a constant matrix, it can be factored out, resulting in:EQUATIONHence, taking into account that t∈Nv∪v x t,: P ⊤ 2 serves as the proportionality constant, which depends on the values of x t,: and P, we can deduce thatEQUATION)Based on the above analysis, we conclude the proof.Datasets for node-level tasks.• Citation networks [28] : Cora, Citeseer, PubMed. Each node in these networks represents a scholarly article, with edges indicating citation relationships. Nodes are defined by attributes such as abstracts, keywords, full-text content, and derived features like TF-IDF vectors. Labels to which nodes belong, typically corresponding to research areas or topics. • Reference network [25] : Wiki-CS. This network represents a collection of Wikipedia articles in computer science. Each node corresponds to an article, characterized by its text and hyperlinks, while the edges depict hyperlinked references between articles. Node labels denote specific subfields of computer science covered by the articles. • Co-purchase networks [29] : Amazon Photo (Photo for short), Amazon Computers (Computers for short). Nodes represent products available for purchase, with attributes such as features, prices, and customer reviews. Node labels correspond to product types or brands, while edges indicate co-purchase relationships, reflecting the frequency of items commonly bought together by customers. • Co-author network [29] : Coauthor Physics (Physics for short). Nodes represent physicists, each described by their publication record, research interests, and affiliations. Labels indicate distinct areas or subfields within physics. Edges between nodes stand for collaborations, typically formed through joint publications or co-authorship of scientific papers.Datasets for graph-level tasks.• Collaborative movie networks [42] : IMDB-BINARY (IMDB-B for short) and IMDB-MULTI (IMDB-M for short). Nodes denote actors or actresses, and an edge exists between two nodes if the individuals have co-starred in the same film. • Scholarly collaboration network [42] : COLLAB. Researchers as nodes and edges indicate partnerships between them.It is important to mention that node attributes are absent in the three datasets for graph-level tasks, making one-hot encoding of the degree a typical approach.We source these datasets from the public repository PyTorch Geometric (PyG) [9] . The datasets can be accessed through the URLs listed below: For the seven benchmark datasets utilized for node classification tasks (Cora, Citeseer, PubMed, Wiki-CS, Computers, Photo, and Physics), the dataset is divided into training, validation, and testing sets in the ratio of 1:1:8. For the three benchmark datasets employed for graph classification tasks, namely IMDB-B, IMDB-M, and COLLAB, a 10-fold cross-validation approach is adopted to partition. Baselines for node-level tasks. Details on the baselines for node-level tasks are outlined below.GCN [17] : It is a representative Graph Neural Network (GNN) that utilizes spectral and spatial strategies to perform graph convolutional operations. It makes each node aggregate information from its neighbor nodes by integrating the graph topology and node attributes.GAT [34] : It introduces an attention mechanism into the GNN, enabling each node to weigh the importance of its neighbor nodes during the aggregation process.Unsupervised baselines are detailed below.DGI [35] : An Infomax principle-based GCL augments the graph via row-wise shuffling of the attribute matrix and maximizes the mutual information between global and local representations.GMI [26] : It is a variant model of DGI that maximizes a comprehensive graphical mutual information metric, including features and edges between nodes in both the input and reconstructed output graphs.MVGRL [14] : It is a variant model of DGI, employing contrastive learning between various structural views of graphs, including first-order adjacency and graph diffusions.GRACE [53] : It is a GCL model that generates node embeddings by corrupting both graph structure (via random edge removing) and attributes (via random attribute masking) to create diverse views and maximize their agreement.GCA [54] : It is a variant model of GRACE, which incorporates adaptive augmentation strategies based on node centrality, aiming to enhance the flexibility of the model.BGRL [32] : It is a GCL model that augments the graph through random edge removing and employs bootstrapping to update the parameters of the online encoder.GBT [2] : A feature-level GCL model leverages the validated Barlow Twins loss for training, aiming to reduce redundant information between two views obtained through random edge removing.CCA-SSG [50] : It is a GCL model utilizing feature-level contrast derived from Canonical Correlation Analysis (CCA) to learn the node representations.SPAN [21] : It introduces a spectral augmentation scheme for topology augmentation by perturbing the graph spectrum, and aiming to maintain spectral invariance to sensitive structures, and minimizing graph spectrum changes.DSSL [39] : It is a graph self-supervised model that presents an approach that disentangles the varied neighborhood contexts of a node, aiming to model multifaceted information within the graph.HomoGCL [20] : It is a localized variant model of GRACE, incorporating k-mean-based saliency values to weigh the importance of neighbor nodes.Baselines for graph-level tasks. Introductions of the baselines for graph-level tasks are given below.InfoGraph [30] : It is a variant of DGI, which maximizes mutual information between graph-level representations and substructures at different scales, such as nodes, edges, and triangles.GraphCL [48] : It is a GCL framework that learns graph representations by employing various augmentation techniques on the local subgraphs of nodes.JOAO [47] : It is a variant of GraphCL, which utilizes the min-max optimization to automatically select the most effective GAs during the contrastive learning process.AD-GCL [31] : It is a GCL framework based on adversarial training, introducing an attack process to modify the edges.For the baseline implementations, we utilize PyG to implement GCN and GAT. In addition, for the self-supervised baselines, we employ their source codes. The sources are listed below:• GCN, GAT: https://github.com/pyg-team/pytorch_geometric/tree/master/torch_geometric /nn/conv. • DGI: https://github.com/PetarV-/DGI. • GMI: https://github.com/zpeng27/GMI. • MVGRL: https://github.com/kavehhassani/mvgrl. • GRACE: https://github.com/CRIPAC-DIG/GRACE. • GCA: https://github.com/CRIPAC-DIG/GCA/.• BGRL: https://github.com/nerdslab/bgrl. • GBT: https://github.com/pbielak/graph-barlow-twins.• CCA-SSG: https://github.com/hengruizhang98/CCA-SSG.• SPAN: https://github.com/haonan3/spgcl. • DSSL: https://papers.nips.cc/paper_files/paper/2022/file/040c816286b3844fd78f2124eec 75f2e-Supplemental-Conference.zip. • HomoGCL: https://github.com/wenzhilics/HomoGCL • InfoGraph: https://github.com/sunfanyunn/InfoGraph • GraphCL: https://github.com/Shen-Lab/GraphCL. • JOAO: https://github.com/Shen-Lab/GraphCL_Automated. • AD-GCL: https://github.com/susheels/adgcl.This subsection analyzes the complexity of GOUDA in comparison with the baseline GCL equipped with learnable GAs (i.e., SPAN, JOAO, and AD-GCL). Note that since the updates of these GCLs can utilize the same graph encoder (e.g., GCN) and contrastive loss (e.g., InfoNCE loss), this discussion focuses solely on the time complexity of the augmentation phase.To enhance clarity, let us define our terms: n represents the number of nodes corresponding to the network size; m signifies the number of edges; f refers to the dimension of attributes; and d denotes the dimension of the hidden layers.SPAN [21] : The time complexity for augmentations in SPAN is O(n 2 tk), where t denotes the time of iterations and k is the number of eigenvalues to be selected. Specifically, SPAN necessitates an iterative optimization of the augmentation through eigen-decomposition, which demands a O(tn 3 ) complexity. Nonetheless, this complexity can be reduced to O(n 2 tk) by employing selective eigendecomposition on the k lowest-and highest-eigenvalues via the Lanczos Algorithm.JOAO [47] : The time complexity for augmentations in JOAO is O(n 2 d). Specifically, JOAO employs a min-max optimization strategy to refine the parameters that are used for selecting augmentations from an option pool. This process involves maximizing the contrastive loss, which inherently requires a O(n 2 d) complexity. [31] : The time complexity for augmentations in AD-GCL is O(n 2 d). Similarly, AD-GCL employs a min-max optimization strategy, but its objective is to modify edges. Therefore, this process also entails maximizing the contrastive loss, which carries a O(n 2 d) complexity. In addition, encoding edges is required, which introduces a complexity of O(md 2 ).GOUDA : The time complexity for augmentations in the proposed GOUDA is O(nkf ). GOUDA introduces k AC vectors to nodes and utilizes the consistency-diversity balance principle to update these AC vectors, where k ≪ n. Firstly, nodes are updated by aggregating the features of AC vectors, which incurs a complexity of O(nkf ). Besides, GOUDA maintains consistency by minimizing the contrastive loss, a process inherent to the training model and thus does not introduce additional complexity; to ensure diversity, it calculates the independence loss, which brings in a complexity of O(k 2 f ). Therefore, the overall complexity is O(nkf ).In summary, GOUDA presents a more computationally efficient approach than the baselines, which is supported by the evidence presented in Section 4.The experiments leverage the linear evaluation method [26] , where models are firstly trained in an unsupervised manner, and then, the obtained embeddings are utilized for downstream tasks. For the node-level tasks, the graph encoder g Θ is configured as a two-layer GCN [17] , while for the graph classification tasks, it is set as a five-layer GIN [40] . In the evaluation phase, we utilize a single-layer linear classifier [27] for node classification [26] , apply K-means [16] to node embeddings for node clustering, and train an SVM classifier [3] for graph classification. The results for node-level tasks are the average of ten random runs, while those for graph-level tasks are based on five runs.All experiments are conducted on two Linux machines as shown in Tab. 6. GOUDA is implemented as two models: GOUDA-IF, which utilizes InfoNCE loss, and GOUDA-BT, which employs BarlowTwins loss. For the node-level tasks, both models are trained using an Adam optimizer with a learning rate of 1e -3 and the weight decay rate from {0, 5e -5 , 5e -4 }. The dimensions d of node embeddings are selected from {256, 512, 1024, 2048}, and their impact is analyzed in Section 4.2. The hyperparameters β 1 and β 2 of independence loss are chosen from {1e -4 , 1e -3 }, while the related hyperparameter γ is selected among {1e -2 , 1e -1 , 1, 10}. Additionally, for GOUDA-IF, the temperature coefficient τ is selected from {0.2, 0.4, 0.6, 0.8}, while for GOUDA-BT, the hyperparameter λ is set to 1 d . For the graph-level task, the configuration follows GraphCL [48] , where the hidden dimension is fixed to 128 and the penalty parameter of SVM is selected from {1e -3 , 1e -2 , 1e -1 , 1, 1e 2 , 1e 3 }. The choice of threshold ϵ is given in Section E.6.2.Tab. 7 presents the comprehensive results of the node clustering experiments. The analysis of these results can be found in Section 4.Embedding Dimension. This experiment aims to shed light on the selection of hyperparameter d. As depicted in Fig. 8 , the proposed GOUDA shows improved performance with an increased embedding dimension. Notably, GOUDA-IF and GOUDA-BT exhibit reduced performance at an embedding dimension of 256 compared to higher dimensions. This indicates that a large embedding dimension is essential for contrastive learning models to capture robust representations. Moreover, it is observable that there is a slight decrease in the performance of GOUDA-IF with large dimensions. This can be due to overfitting to the self-supervised signal, which may hinder its generalization capability. Weight of Independence Several insights are yielded from observations in Fig. 9 . Firstly, the proposed GOUDA shows stability regardless of parameter γ changes. Secondly, the framework maintains robust performance even at a low value of 0.01 and 0.1. However, the omission of the proposed Independence loss does have a detrimental effect, as demonstrated by the Ablation Study (in Section 4.2). Lastly, while the tested parameter range is {1, 10}, future research should consider broader or more detailed ranges.Threshold for sparsification. The performance changes resulting from varying the hyperparameter ϵ are detailed in Tab. 8. To eliminate bias due to network size, ϵ is not freely tuned. Instead, it is set as the output of the selection function ϵ = selection(B, s), which estimates this threshold. B denotes the matrix of propagation weights from AC vectors to nodes, and s stands for the proportion of the largest elements retained. The value of s is chosen from the set {0.2, 0.4, 0.6, 0.8} in the experiments. From Tab. 8, it can be observed that the threshold does not significantly affect the model performance. Specifically, within the range of selection, the variation in model performance does not exceed 2%. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Our research adheres to the NeurIPS Code of Ethics, and we have ensured that all aspects of our work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper addresses potential limitations in existing Graph Contrastive Learning (GCL) frameworks, specifically the issues of specificity, complexity, and incompleteness of Graph Augmentation (GA) techniques. The authors introduce UGA, a method with theoretical guarantees, alongside GOUDA, a framework that can be integrated with various contrastive losses. Experimental results demonstrate the effectiveness of GOUDA in enhancing GCL performance.

The paper has received predominantly positive feedback both before and after the rebuttal period, with final scores of (8, 7, 7, 5). Please carefully consider the questions and concerns raised by the reviewers, particularly those from reviewer i3ih, when the authors preparing the final version of the paper, e.g., some implementation details and more comparisons.

Reviews: review: paper_summary: paper_summary: The paper introduces GOUDA, a versatile framework for Graph Contrastive Learning that addresses the limitations of existing graph augmentation techniques. GOUDA proposes a unified graph augmentation module capable of simulating various explicit graph augmentations, enhancing the generality and efficiency of GCL models. By incorporating both widely-adopted contrastive losses and a novel independence loss, GOUDA ensures consistency and diversity across different augmentations.
strength_weakness: strength_weakness: 1. Some of the mathematical derivations and formulas are complex and may be difficult for readers to follow. Although the notations are defined, the paper could provide more intuitive explanations and step-by-step derivations to enhance understanding.
2. While authors have compared with many GCL methods, the comparison with GA techniques like simple node augmentation is missing.
questions: questions: 1. How to interpolate the learned AC vectors? 
2. How is k selected in line 181?
3. Graph augmentation is based on the assumption that the augmentation does not change the original class label, how does UGA guarantee this?
limitations: limitations: N/A
paper_summary: paper_summary: The paper reconsiders the formulation of graph augmentations in graph contrastive learning, introducing a novel perspective on GA through message passing. The proposed UGA framework interprets graph augmentations as mechanisms for aggregation and propagation between nodes, highlighting the significance of local aggregation and propagation within GA. The effectiveness of the proposed GOUDA framework is demonstrated in experimental evaluations.
strength_weakness: strength_weakness: - The paper lacks presentation on the interpretability of learned graph augmentation vectors. Given that graph augmentation can occur at nodes, edges, attributes, and sub-graphs, an explanation of how these learned vectors are interpreted is necessary.
- The paper should include a comparison of time consumption during experiments.
questions: questions: - Can the proposed GOUDA, especially the Graph Augmentation Vectors (GAVs), be integrated with other Graph Contrastive Learning (GCL) methods?
- How to assess the contribution of the proposed GOUDA framework to Graph Contrastive Learning (GCL)?
limitations: limitations: The paper requires interpretability and should clarify the contribution to GCL.
paper_summary: paper_summary: This paper explores the characteristics of local attribute modification in current graph contrastive learning methods. It then integrates diverse augmentation strategies with attribute learning into a unified framework. This unified approach introduces a novel and straightforward graph contrastive learning framework that establishes consistency between embeddings and ensures diversity through augmentation. Specifically, diversity is enforced using the HSIC term. The proposed framework offers two main advantages: 1) high efficiency and 2) universally learnable augmentation. Its effectiveness is validated across graph-level and node-level tasks, demonstrating its superiority.
strength_weakness: strength_weakness: - The symbols $\gets$ used in many equations, such as Eq. (4), should be changed to $\to$ for convenience.

- Some experiments require further explanation. It's unclear why the proposed framework demonstrates robustness to topology and attribute noises.

- Some hyper-parameters remain unverified. It's uncertain whether the parameter $\varepsilon$ in Eq. (7) significantly affects performance.

- As far as I know, the HSIC term is computationally intensive. Does this affect efficiency?

- Descriptions from Lines 143 to 153 are difficult to understand.

- Reference [33] is based on adversarial training rather than adversarial attack.

- Some commas are missing after equations, for instance, in Eq. (9).
questions: questions: See above.
limitations: limitations: N/A
paper_summary: paper_summary: This paper investigates the generality of graph augmentations across various types of graphs and tasks. Firstly, the paper presents a new interpretation that unifies the various graph augmentations into local attribute modifications of each node from a basic message-passing perspective. Then, the paper develops a novel graph augmentation module that alters node attributes to simulate the augmentation effects. Utilizing this module, the paper presents a novel framework for graph contrastive learning, where the balance between consistency and diversity across different augmented views is carefully considered. Finally, it provides a theoretical understanding of the generality of the proposed framework. Extensive evaluations demonstrate its superior performance on diverse datasets and tasks.
strength_weakness: strength_weakness: 1) There are a few typos and grammar mistakes in the paper that need fixing. 
2) The theoretical analysis requires intuitive explanations. While the authors offer the proof of generality, they should elucidate how the theorem relates to the superior performance. 
3) The concepts of consistency and diversity deserve a more comprehensive explanation. Additionally, how do the definitions of consistency and diversity across augmentations in self-supervised learning scenarios differ from those in semi-supervised learning scenarios? 
4) The connection between the number of AC vectors and model performance is not clear.
questions: questions: Please refer to Weaknesses.
limitations: limitations: N/A

Comments: title: Response to authors
Discussion period instructions
comment: Thank you for your response. My concerns have been addressed, so I have decided to maintain my original score.
Thank you for the response, which solves most of my concerns, and I will maintain my score on this paper.
Thank you for the authors' detailed response. I am satisfied with the authors' response and revised manuscript, so I would like to increase my score from 7 to 8.
Thanks for your feedback. We hope that our response appropriately answers your questions. We are willing to provide further clarification if you have any additional concerns.
Thanks for authors' rebuttal, I have adjusted my score based on the response.
Dear Reviewers,

The authors have provided comprehensive rebuttals and tried to address the concerns raised in your reviews. Please take the time to review their responses carefully. If you have any further questions or require additional clarification, please engage in a discussion with the authors. Thank you for your continued efforts.

AC