Exploitation of a Latent Mechanism in Graph Contrastive Learning: Representation Scattering
Authors: Dongxiao He
Lianze Shan
Jitao Zhao
Hengrui Zhang
Zhen Wang
Weixiong Zhang
Venue: NeurIPS 2024 oral (2024)
Keywords: Graph Contrastive Learning
Primary area: graph_neural_networks

Abstract: Graph Contrastive Learning (GCL) has emerged as a powerful approach for generating graph representations without the need for manual annotation. Most advanced GCL methods fall into three main frameworks: node discrimination, group discrimination, and bootstrapping schemes, all of which achieve comparable performance. However, the underlying mechanisms and factors that contribute to their effectiveness are not yet fully understood. In this paper, we revisit these frameworks and reveal a common mechanism—representation scattering—that significantly enhances their performance. Our discovery highlights an essential feature of GCL and unifies these seemingly disparate methods under the concept of representation scattering. To leverage this insight, we introduce Scattering Graph Representation Learning (SGRL), a novel framework that incorporates a new representation scattering mechanism designed to enhance representation diversity through a center-away strategy. Additionally, consider the interconnected nature of graphs, we develop a topology-based constraint  mechanism that integrates graph structural properties with representation scattering to prevent excessive scattering. We extensively evaluate SGRL across various downstream tasks on benchmark datasets, demonstrating its efficacy and superiority over existing GCL methods. Our findings underscore the significance of representation scattering in GCL and provide a structured framework for harnessing this mechanism to advance graph representation learning. The code of SGRL is at https://github.com/hedongxiao-tju/SGRL.

Exploitation of a Latent Mechanism in Graph Contrastive Learning: Representation Scattering
Dongxiao He', Lianze Shan', Jitao Zhao', Hengrui Zhang', Zhen Wang?; Weixiong Zhang*
CCollege of Intelligence and Computing, Tianjin University, Tianjin, China Department of Computer Science, University of Illinois at Chicago, Chicago, II,, United States
School of Cybersecurity, Northwestern Polytechnical University, Xi'an, China
Department of Computing, Department of Health Technology and Informatics,
The Hong Kong Polytechnic University, Kowloon, Hong Kong (hedongxiao, shanlz21 19, zjtao] @tju.edu.en, hhzhan55 @ @uic.edu, ?w-zhen @ @wwpu.ddu.cn "weixiong zaaionng..znggeeee @ polyu.edu.hk
Abstract
Graph Contrastive Learning (GCL) has emerged as a powerful approach for gener- ating graph representations without the need for manual annotation. Most advanced GCL methods fall into three main frameworks: node discrimination, group discrim- ination, and bootstrapping schemes, all of which achieve comparable performance. However, the underlying mechanisms and factors that contribute to their effec- tiveness are not yet fully understood. In this paper, we revisit these frameworks and reveal a common mechanism mechanism--represennie representation scattering-than significantly enhances their performance. Our discovery highlights an essential feature GCL and unifies these seemingly disparate methods under the concept of representation scattering. To leverage this insight, we introduce Scattering Graph Representation Learning (SGRL), novel framework that incorporates new representation scatter- ing mechanism designed to enhance representation diversity through a center-away strategy. Additionally, consider the interconnected nature of graphs, we develop a topology-based constraint mechanism that integrates graph structural properties with representation scattering to prevent excessive scattering. We extensively eval- uate SGRL across various downstream tasks on benchmark datasets, demonstrating its efficacy and superiority over existing GCL methods. Our findings underscore the significance of representation scattering in GCL and provide a structured frame- work for harnessing this mechanism to advance graph representation learning. The httpaaa..oommeeddoiioooiiii////e
code of SGRL
1 Introduction
Graph Neural Networks (GNNs) have shown impressive performance across various fields, including social networks [1 (2] bioinformatics B and fraud detection [5]. However, training GNNs typically requires large datasets with manually annotated labels, which can be both costly and labor- intensive [6]. This limitation restricts their broader application. To address this challenge, Graph Contrastive Learning (GCL) has attracted significant attention 77 9 10), focusing on creating proxy tasks from the data itself enable self-supervised training of GNN encoders [11]. Currently, most GCL research 66
13 is concentrated on enhancing one of the three main frameworks: InfoNCE- based (i.e., node discrimination) bootstrapping schemes) [10].
12
[14) DGI-like (i.e., group discrimination) (7). and BGRL-like (i.e.,
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The three mainstream graph contrastive learning frameworks differ significantly, particularly in their approach to node-level tasks. InfoNCE-based methods prioritize node-level discrimination [8] g [2]. treating an anchor node as a reference while considering all other nodes as negative samples to enhance the distinctiveness of each node's representation. In contrast, DGI-like methods adopt group discrimination paradigm 6 7 15], viewing all nodes as positive samples from the same
distribution and differentiating them from a noise distribution. BGRL-like methods [00 [3 [6]
employ bootstrapping scheme that eliminates the need for negative samples during training, focusing instead on aligning positive samples. Despite these differences, all three frameworks demonstrate comparable performance, leading us to conjecture that they may share common mechanism. This idea is partially inspired by previous research in visual contrastive learning, which identified uniformity key factor in many contrastive learning methods [17]. However, uniformity, which focuses solely on the discriminative properties of node instances, cannot explain the shared underlying factors across these GCL frameworks, as further discussed in Appendix A
We carried out a detailed analysis of the three GCL. frameworks and discovered that representation scattering is a crucial common factor in their success (in Section In Section we provide formal definition of representation scattering and examine how each framework achieves it. For the DGI framework, we analyze the distributions of original and noise data, proving that after GNN message passing [18 19], the noise data distribution aligns with the mean distribution of the original data. This indicates that DGI's objective can be interpreted as distinguishing between the local semantics of nodes within the original graph and its mean, which correlates with representation scattering (Section 3.I. In the InfoNCE framework, existing work has demonstrated that the mechanism of negative sampling facilitates a process of achieving uniformity [7]. We extend this further by theoretically proving that the InfoNCE loss serves as upper bound for representation scattering loss (Section 3.2). Regarding the BGRL framework, we explore the role of Batch Normalization (20) and its connection to representation scattering, showing that it acts as specific instance of this concept. Notably, removing Batch Normalization significantly degrades BGRL's performance (Figure 2 Our findings confirm that representation scattering is key mechanism present across all three mainstream GCL frameworks.
However, the existing GCL frameworks have not fully leveraged the latent mechanisms inherent in their designs and neglected the interconnected nature of graphs when implementing representation scattering. This oversight results in inefficiencies and reduced robustness. Firstly, generating augmented views or computing similarities between node pairs for representation scattering 7 g incurs significant computational and memory overhead. Secondly, manually defined negative samples may result in the generation of numerous false negative samples, introducing noise [12] that can hinder model training. Lastly, those frameworks that rely solely on positive samples [TO] implement representation scattering indirectly through Batch Normalization, which lacks explicit guidance for scattering and can lead suboptimal performance. Addressing these challenges is crucial for enhancing the effectiveness of GCL methods.
To fully and effectively utilize representation scattering, propose Scattering Graph Representation Learning (SGRL). Our approach introduces a Representation Scattering Mechanism (RSM) that embeds node representations into designated hypersphere, positioning them away from the mean center. This method offers a direct algorithm for representation scattering compared to the existing GCL frameworks, eliminating biases introduced by manually defined negative samples. Additionally, we introduce a Topology-based Constraint Mechanism (TCM) that considers the interconnected nature of graphs. TCM aligns representations derived from structural information with scattered representations, thereby preserving topology information while facilitating scattering. Through these innovations, we aim to enhance the efficiency and robustness of GCL methods. We made the following contributions in this paper:
We discovered common representation scattering mechanism in GCLs and showed that the three mainstream GCL frameworks implicitly utilize this mechanism, and importantly can be unified under the concept to representation scattering.
We showed that the existing methods do not fully exploit the inherent mechanism of representation scattering. We introduced the novel SGRL framework to integrate RSM and TCM, producing an adaptive scattering approach for model training.
We experimented with various downstream tasks on benchmark datasets and demonstrated the effectiveness and efficiency of SGRL.

---

2 Preliminary
Graph Data. We define a graph as G (V,8), where V= {V1, U2,.... uN denotes the set of nodes, and & V V represents the set of edges. The node feature matrix is denoted by X E RRNx where is the number of nodes and D the feature dimension. In addition, the adjacency matrix is indicated by A E R RNNN,, formulated such that Aij 1 when edge (V;, (1), exists within the set E, or Aij = 0, otherwise. The degree matrix is denoted as D = diag d2..... d2 ), where each element d, İjev Aiy. The degree- normalzedd adjacency matrix with self-loops is represented as Asym = D (A + I)D
Graph Contrastive Learning, Given a graph's attributes X and adjacency matrix A, the objective of GCL is to train an encoder f()) in a self-supervised fashion. The learned encoder f()) can generate representations H f(X, A), H RRNK which are both topologies decoupled and dense. These representations can be applied to many downstream tasks.
3 Representation Scattering in GCL
We now examine the shared elements that contribute to the effectiveness of popular Graph Contrastive Learning (GCL) frameworks. Upon revisiting three widely used baseline GCL frameworks, find that they all inherently utilize the mechanism of representation scattering, which plays a crucial role in their success. Here, formally define Representation Scattering:
Definition 1. (Representation Scattering) In a d-dimensional embedding space IR comprising vectors organized into a matrix V E Rnxd consider a subspace S (1 S k S d) of R and a scattrr center Representation scattering is process satisfying constraints, (i) Center-Away Construint: Node representations are encouraged to be distant from the scatlered center c, and (ii) Uniformity Constraint: Node representations uniformly distributed over the subspace Sk.
According to Definition 1 achieving representation scattering requires identifying a scattered center e within the subspace S and simultaneously satisfying the Center-Away and Uniformity Constraints.
We will study its relationship with the popular GCL frameworks in the following section.
3.1 DGI-like methods
DGI-like methods generate negative samples through random permutation of nodes. They employ a mutual information discriminator D(-), which maximizes the mutual information between nodes and their source graphs to train the model 7 Here, we show that the objective function of DGI is special case of representation scattering. To facilitate the proof, we present the following assumption:
Assumption 1. (a) The normalized propagation matrix À is defined as À = D- where A + I. (b) DGI generates the corrupted graph by randomly shuffling the entities in the feature matrix X. while keeping the adjacency matrix A unchanged. (c) The original data are class baas-balanced, i.e., for any classes and j. num(k ()) num(j).
The following results do not strictly require Assumption to be satisfied. Assumption [rrrtppiiionnneeeeeeeeee represeetss common scenario and serves to simplify the proof. A discussion on the validity of Assumption and proof's of the subsequent results without Assumption I can be found in Appendix C
Theorem 1. At the node level, minimizing the DGI loss is equivalent maximizing the Jensen- Shannon (JS) divergence between the local semantic distribution in the original graph and its average distribution.
Proof. Let Piata denote the distribution of all nodes in the original graph, characterized by a mean and variance 2². To analyze the local distribution of node after its embedding aggregation by GNN encoders, we define P; as the distribution of node v; and its first-order neighbors with mean μ4; and variance of. We make use of a conclusion in [6] and introduce the following lemma:
Lemma 2. Minimizing the DGI loss, denoted as LOGL, equals to maximizing the -haanonn (JS) divergence between the distribution of the original graph G and the corrupted graph G, i.e., Min(C +* Max(JSiG G))
3

---

We discuss Lemma 2 in Appendix D.I. Lemma establishes the relationship between the DGI loss and the distributions of the original and corrupted graphs. To investigate the distribution of representations in original and corrupted graphs, we focus on the case of a single-layer GNN [21 and have the following formulation:
H = =NGNN(,,X) - AXW. h, = EaSx, W = Σ JEN, di
Xj W
JEN.
where W <(W), W RDX with & being an activation function like ReLU [22] for ease of understanding, .N; denotes the set of first-order neighbors of node Us inclusive of U, itself. In Eq. [ Nj E N, ~ pi (f) For node vú, subsequent to GNN message passing, we compute the mean and variance of the aggregated representation h, as follows:
[h. =
Σ (JEN())
0syX, W
= E
Osyy B(y)) W
Σ JEN(1)
W
JEN())
(2)
Var(h:) Var
Σ (EV())
Xxjy, W
Σ JEN(()
any Var(x;) W = of Σ dij W
JEN(1)
Given that is a normalized propagation matrix with EjEN(a) = for the aggregated repre- sentation h, in the original graph, the mean of representation h, 's distribution equals W' We then analyze the distribution of the aggregated representation h; in the corrupted graph. Based on Assumption T the adjacency matrix A is unaltered and the feature vector x; selected randomly from the feature matrix X of the original graph, following the distribution Pdata (11, a²). According to Eq the mean of the distribution of h; is J W', and the variance is o² EjEN()) oiyy w
Consequently, we have h, (14 W of 2 and h, ~ Paana (1 W o² ?),, which directly prove that maximizing the JS divergence between the original and corrupted graph distributions at the node level maximizes, in effect, the JS divergence between the local semantic distribution of node i and the mean distribution of the nodes in the original graph.
IT
Corollary 3. Taking the mean of the original graph as the center e, and the original representation space as subspace St the objective of DGI can be described as follows: within the subspace Sk DGI increases the distance between the nodes of the original graph and its center e, achieving the objective of representation scattering.
Corollary 3 reveals, for the first time, that the primary goal of DGI-like methods position node represen- tations away from central point to encourage a uniform distribution of the nodes. To illustrate these theo- retical insights, we performed a vi- sualization experiment. As shown in (a) and (b), both a randomly initialized GNN and trained single- layer GNN demonstrated that their representations are distanced from the mean of the original graph nodes. Ad- ditionally, Figure (b) and (c) provide clearer evidence that minimizing the LDGI equivalent to maximizing the -hannnon (JS) divergence between the positive and negative samples, i.c., maximizing the JS divergence between the local semantic distribution in the original graph and its mean distribution. However, the row-wise shuffling mechanism may introduce potential disturbances. In a graph with n nodes, each node retains its local semantic distribution with probability of 1/k during the shuffling process, where l denotes the number of distinct node types, assuming class balance. Despite this, the non-discriminatory nature of the perturbation means that these unchanged nodes could still be mistakenly classified as negative samples. Consequently, genuine node rep- resentations may be incorrectly labeled as negatives, leading to bias in the learning process. The overlapping nodes depicted in Figure intuitively support this perspective.
(a) Random Init
(b) GNN-Layer
(c) GNN-Layer #2
Figure I: points represent the embeddings of the perturbed negative samples, and the red points denote that of positive nodes. As can be seen in Figures (a) and (b) of model random initialization and the layer of the trained encoder, the DGI-like methods essentially maximize the JS divergence between node embedding and embedding mean.
t-SN embedding of DGI Co.CS dataset. The blue
Figure (a)

---

3.2 InfoNCE-baseed methods
We now show that the mechanism of negative sampling in the InfoNCE-based methods equivalent to representation scattering.
Theorem 4. Let h 1/n Σi= h, and sim(.) be the cosine similarity function (here, h, represents the encoded representation of node v;). For node vi, the lower bound of the InfoNCE loss LinfoNCE (h;) exists: (mgaNce(h;) 2 sim(h,, h) In(2n).
A detailed proof is given in Appendix D.3 Theorem Theorem[inniceen Hindicates that when minimizing CInfeNCE (h.). the similarity between node v; and mean node 6, i.c. Sim(h,, h) is also minimized. Taking the mean the subspace S the mechanism of
of all nodes as the scattered center c, and the hypersphere negative sampling is equívalent to representation scattering. However, the InfoNCE loss function inefficient for representation scattering as needs to compute and reduce the similarity of each negative pair. Moreover, it indiscriminately treats all negative samples, ignoring the distinctions among them, which leads to inappropriate scattering of negative samples and potential bias from false negatives. Consequently, many recent methods have incurred additional computational overhead by manually and intuitively defining positive and negative samples [2] 23
3.3 BGRL-like methods
All BGRL-like methods incorporate a key component: Batch Normalization (BN). For feature vector x; and its corresponding batch statistics, mean μ and variance o², the BN is applied as follows: -/)//////// /)/Vo= + + 3. where y and 3 are learnable parameters that scale and shift the normalized value, and is small constant added for numerical stability.
(N(x.) >(x; -
Theorem 5. The process data normalization by batch normalization can be seen as special case of representation scaltering.
detailed proof can be found Appendix D.2 To empirically assess the impact of batch normalization on BGRL, we con- ducted experiments comparing both BGRL and BGRL. w/o BN across four benchmark datasets. The results, shown in Figure 2] reveal significant drop in accuracy for BGRL across all datasets without batch normalization, highlighting its critical role in the bootstrapping framework. While BGRL incorpo- rates representation scattering through batch normalization, its training process lacks explicit guidance or a dedicated mech- anism to efficiently manage this scattering. The absence of direct supervision during the representation scattering phase lead to unoptimized distribution of representation within the embedding space, resulting in suboptimal performance.
52
F1-scoess
54-
Campanu
Figure 2: The impact of Batch Normal- ization in BGRL.
In summary, analysis of the three mainstream GCL frameworks theoretically proved that they all inherently utilize representation scattering but also fail to fully utilize this effective mechanism. This analytic result has motivated to design more effective representation scattering method tailored for learning on graphs.
4 Methodology
The proceeding sections have revealed the importance of representation scattering in GCLs. Based on the findings, have designed novel method, namely Scattering Graph Representation Learning, short-handed as SGRL (Figure 3). We introduce the components of SGRL and provide discussion in the following sub-sections.
4.1 Representation Seattering Mechanism (RSM)
To address the shortcomings in the application of representation scattering within the three mainstream graph contrastive learning frameworks, we design RSM to explicitly guide the target encoder in learning scattered representations. Following Definition I
introduce subspace S* and scattered
5

---

Topology-Basee Based Constraint Mechanism
Alignment Loss Function
wwm.mmm iin
One-HHop bused
Constraint
Mult-Hlpp Cunstraint
(H A)
Piccaonnl IR (Rgegen))
EMA
H Stop Gradient
w fo
Representation Scattering Mechanism
Seuttering Loss Funetion
Trandie
Looitering
Meau
[Elpugge A)
Representation
Hypersphere Center
Regresentation
Figure 3: The overview of SGRL. Consider graph & processed using two distinct encoders (online encoder and larget encoder): fa(-) with parameters and fo(-) with ø, aimed at generating node representations Honiin and Hurget, respectively. For Harget the mean representation of all nodes calculated to serve as the scattered center c. The parameters of fo(-) are updated via RSM that encourages node representations to diverge from C. Honlime is processed through TCM to incorporate topology information, resulting in Hoopciogy Subsequently, Hi is embedded through a predictor qe to predict Harget and the parameters in fa(+) )) is updated through back-propagation while stopping the gradient of (.). Both channels are trained simultaneously. At the end of cach epoch, we employ an Exponential Moving Average (EMA) to update parameters 6. Finally, the representations generated by f-(-) are employed across various downstream tasks.
center effectively perform representation scattering. For the subspace S* transformation function T-ans(-) is introduced to transform representations from the original space Rd into S Specifically, we apply & normalization to each row vector h, the matrix Hurget:
h,
h, Transga (h,) Max (2)
s* = (h; |la =].
where h; is representation for node eú E V. generated by the target encoder, |lh, = -E²= and is small value to avoid division by zero. defined by Eq. 3 the representations of all nodes distributed on hypersphere $ This mapping prevents arbitrary scattering of representations in the space, avoiding instability and optimization difficulties during training.
Next, we define the scattered center e and introduce representation scattering loss function Lecattering in s to push node representations away from the center as formulated as follows:
71 Σ" h cl2
-= h
Ewattering
(4)
Through Eq. 4 SGRL achieves uniformity of representations globally across the entire dataset, with- out emphasizing local uniformity. Specifically, RSM enables representations of different semantics to be globally scattered across the hypersphere while accommodating representations of the same semantics to aggregate locally.
Discussion of RSM. We now provide a theoretical analysis demonstrating that the proposed repre- sentation scattering mechanism outperforms the three graph contrastive learning frameworks. RSM achieves representation scattering more effectively by encouraging distances between node represen- tations and the scattered center, eliminating the dependence on manually designed negative samples. Most traditional methods 7
rely on negative samples to indirectly promote representation scattering, which is inefficient and introduces biases. DGI-like methods, as discussed in Section 3.1 aim to maximize the -hasennn (JS) divergence between the distribution of the original graph and its mean distribution. Based on this, generating additional noise graph inefficient. Moreover, some negative samples generated through random shuffling may align with the distribution of positive samples, resulting false negative samples that hinder model training. InfoNCE- based methods consider all nodes as negative samples, except the matching ones in two augmented views. While pushing nodes away from each other ensures the discriminability of each node, it also results in significant computational overhead. Moreover, due the potential conflict between the encoder's gressage-passing mechanism and the InfoNCE loss function, many negative samples can not effec- tively distance themselves from each other 24] Therefore, by employing center-away strategy,
8
9 15

---

RSM effectively reduces the additional computational overhead and mitigates the biases caused by manually designed negative samples.
4.2
Topology-based Constraint Mechanism(TC.M)
After obtaining the scattered representations Hanget = fo (A. through the target encoder, it is necessary to consider the differences in the degree of scattering of different node representations.
Consider the interconnected nature of graphs, the representations of two nodes connected topologically should be closer in space St. Specifically, Vus, 1/ V. h, h3 S*, given threshold d: l h < d, if (i E V and |h. 1 > d, if (i, & V. Intuitively, to this end, simple way is replace the individually scattered representations with the aggregated representations H Harget from first-order neighbors: Hurger Hurget However, attempting to consider the topology information and achieve representation scattering through the same encoder may lead conflicts.
To address this issue, we propose a Topology-based Constraint Mechanism (TCM). Specifically, separate the process of constraint from the process of scattering by letting the online encoder generate topologically aggregated representations instead of the target. The online encoder enhances its representations by summing the original representations Honline with the topologically aggregated representations of its k-order neighbors Á HHallne which can be described as:
(5) where represents the order of neighbors and Å = A + I is the adjacency matrix with self- loops. By separating scattering and constraints. SGRL effectively achieve a balance between representation scattering and topology aggregation adaptively, rather than setting the scattering distance empirically. Next, the topology representations H lupology are fed into a predictor (o(-) ()) to online
H topology
Å A'Honline + Honline:
online ). Our objective is align the predicted topology representations Zunline closely to the scattered representations Hurges. enhancing the model's effectiveness in capturing the essential semantic details of the graph. Based this, the alignment loss Lalignment is defined as follows:
generate the predicted representations Zunline qo [H topology
N
Z (online,x) H (targot,))
Calignment
(6)
N
7 (online. .)
H (aaget,)))
where Zoaline and Harget represent the predicted and scattered representations, respectively. During this process, the online encoder's parameters 0 are updated and the target encoder's parameters stop gradient propagation. Compared to directly aligning constrained and scattered representations. this predictive objective can act as buffer, allowing the online encoder to adaptively learn scattered representations and topology information. Furthermore, to make the target encoder consider topologi- cal semantic information into the process of representation scattering, instead of solely focusing on scattering, we employ an Exponential Moving Average at the end of each training epoch:
Φ + TØ + (1 T)0,
(7)
where 7 is target decay rate and [0, 1]. Eq. 7 effectively mitigates the adversarial interactions between RSM and TCM, while also facilitating the integration of topology information into the representation scattering process. Moreover, RSM enhances the discriminability of representations, while TCM incorporates topology information into the representations. The interaction between these two mechanisms effectively mimics the role of data augmentation, i.e., train encoders to learn the invariance of data to perturbations. Consequently, SGRL obviates the need for explicitly designing data augmentation strategies, which leads additional computational overhead and heavy reliance on the choice of augmentation techniques.
5 Experiment
We evaluated SGRL on the five of the most widely used benchmark datasets, including Amazon- Photo (Photo) and Amazon-Computers (Computers) 255 WikiCS (26), Coauthor-CS (Co.CS) and
Coauthor- Physics (Co. Physies) (2]] Detailed information of these datasets in Appendix B.2
We compared SGRL with four types of methods: (I) Three mainstream basclines: GRACE 8
7

---

DGI
7
and BGRL 10
(2)Six recently advanced algorithms: GCA 19, ProGCL [2], AFGRL
13
iGCL (28]
2B MVGRL [3]. (3)Two classic graph representation learning methods:
Node2vec 30 and Deepwalk [3I. (4)The semi-supervised training baseline GCN 18] We utilized
the representations generated by the online encoder for downstream tasks. For node classification, we followed the evaluate scheme from [TO]. We trained a simple linear model using only the representations from logistic regression loss with an le regularization and no backpropagation of any gradient to the graph encoder network. Specifically, we trained the downstream classifier using 10% the data and tested the classifier on the remaining 90%. We ran SGRL 20 times and report here the average and standard deviation of the Fl-score. For node clustering, adopted the evaluation method from [3.. The testing was conducted on the learned representations at each epoch, and the best performance is reported below. More details of the experiments can be found in Appendix B
Table I:
Performance node classification. OOM signifies out-of-memory on 24GB RTX 3090. X, A, denote the node attributes, adjacency matrix, and labels the datasets. Optimal results shown bold.
Method
Available Data
WikiCS
Computers
Photo
Co.CS
Co.Physics
Raw Features
X
71.98 = 0.00
73.81 = 0.00
78.53 = 0.00
90.37 = 0.00
93.58 = 0.00
Node2vec
A
71.79 + 0.05
84.39 + 0.08
89.67 = 0.12
85.08 + 0.03
91.19 + 0.04
DeepWalk
A
74.35 0.06
85.68 = 0.06
89.44
84.61 = 0.22
91.77 = 0.15
GRACE
X.A
77.97 # 0.63
86.50 0.33
92.46
9..17
0OM
DGI
X.A
75.35 0.14
83.95 = 0.47
91.61 = 0.22
12.15
94.51 + 0.52
BGRL
X.A
76,86 0.74
89.69 0.37
93.07 + 0.38
92.59 2..59 0.14
95.48 + 0.08
GBT
A,
76.65 + 0.62
88. + 0.33
92.63 + 0.44
92.95 0.17
95.07 + 0.17
MVGRL
X.A
77,52 0.08
87.52 0.08
92.11 9.074
95.33 + 0.03
GCA
X.A
0.53 0.67
0.33
0.15
OOM
ProGCL
X.A
0915
03.6
93.67
0OM
AFGRL
X.A
7
0.38
93.2220.28
93.27
95.69 0.10
iGCL
X.A
78.83
0.08
8.06
93.02
0.05
+ 0.20
(GrL(Ours)
X.A
79.40 0.10
90.23 0.03
9..15 0.09
96.23 0.01
Supervised GCN
X,AYY
77.19
86.51 + 0.54
92.42
93.03 + 0.31
95.65 + 0.16
5.1 Performance Analysis
Overall evaluation. Table the aver-
Table 2: Performance on Clustering in terms of NMI and homogeneity, Optimal results are shown in bold and suboptimal results are underlined.
ages and standard deviations of the Fl-scores for all methods on the node classification task. Some statistics for the existing methods are reported from either their original papers or [33. As shown in the table, the proposed SGRL exhibits superior performance across five datasets tested, achieving the highest accu- racy. Compared to the three baselines, BGRL, GRACE, and DGI, our new method outperforms them by 1.23%. 2. 2.1,, and 3.26%, respec- tively, showing SGRL's effectiveness in more effectively exploiting representation scattering. Moreover, observe that SGRL. outperforms methods like GCA, AFGRL and iGCL. which focus on improving upon data augmentation. This demonstrates the validity of employing a graph contrastive framework based on representation scattering and topology aggregation, instead of using data augmentation. SGRL also outperforms the advanced negative sampling method, ProGCL. We attribute this to SGRL's representation scattering, which does not explicitly define negative samples, achieving better performance than manually defined negative samples. When evaluated on node clustering tasks using the scheme in [13), it can be observed that SGRL achieves the best or second-best accuracy on most datasets. Although SGRL attempts to make node representations scattered, which seems unsuitable for clustering tasks, it still manages to deliver competitive performance. This demonstrates that TCM can preserve the topology information of the original graph in the process of representation scattering, ensuring that while
GRACE
DG1
BGRL
SGRL
NMI
0.4282
0.4312
0.3969
0.4188
WiRiCS
Hom.
0.4423
0.4498
0.4156
0.4369
0.4793
0.5364
Amazon Compuuters
Hom.
0.5222
0.4836
0,5869
0.5705
0.6513
0.5487
0.6841
0.6788
Amazon-Phato
Hom.
0.6657
0.5557
0.7004
0.6786
Co-CS
NMI
0.7562
0.7162
0,7732
0.7961
Co-Physics
NMI
0OM
0.6540
0.5568
0.7232
0.6018
8

---

(a) GRACE
(b) DGI
(c) BGRL
(d) SGRL
Figure 4: t-SNE embeddings of nodes in CS dataset.
performing representation scattering, it maintains the local structure and global semantic consisteney of the nodes.
Visualization. To more intuitively show the advantages of the representations learned by SGRL, we employ t-SNE [2] for visualizing the learned representations from the Co-CS dataset. Each point represents node, with the color indicating the node's s label. As shown in Figure 4 SGRI. shows clearer inter-class boundaries compared to the other three methods, indicating that RSM achieves superior scattering. Moreover, we observe better intra-class clustering in SGRL. This highlights that SGRL, which does not require manually defined negative samples, prevents the inappropriate distancing of intra-class nodes. It achieves both global scattering and local semantic aggregation, demonstrating the effectiveness of adaptive scattering methods based on topology constraints.
Table 3: Ablation study on node classification. Optimal results are shown in bold.
Variant
WikiCS
Amazon-Computee
Amazon-Phott
Co-CS
Co-Physics
SGRL w/o RSM and TCM
76.86 + 0.74
89.69 = 0.37
93.07 = 0.38
92.59 0.14
95.48 = 0.08
SGRL TCM
78.55 0.08
89.54
93.58 0.05
94.08 0.03
96. # 0.04
SGRL w/o EMA
79.36 + 0.08
90.03 0.07
93.92 0.02
93.89 0.07
96.16 0.07
SGRL (Ours)
79.40 = 0.13
90.23 = 0.03
93.95 + 0.03
94.15 = 0.04
96.23 = 0.01
5.2 Model Analysis
Hyperparamter Analysis. In this subsection, investigate the sensitivity of the hyperparameter in SGRL, as shown in Eq. [5] The parameter repre- sents the order of neighbors aggregated, which di- rectly influences the strength of the topology con- straints. In our experiments, we adjust k in the range 0, 2,
Accuracy (86 3xa
36.5
Photo
evaluate the impact of different con- straint strengths on SGRL. The results are shown in in Figure 5 when the lack of topology constraints leads to a decrease in model performance. As increases, the constraint ability of TCM gradually strengthens, and the model performance exhibits unimodal shape with respect to changes in k. This highlights that weak topology cannot preserve adequate topological information for adaptive representation scattering, whereas strong topology may excessively restrict the scattering of representations, which aligns with the perspectives proposed in Sections 3 and 4
0, i.e., "SGRL w/o TCM",
98.9
Wikics
Computers
The Order of Neighbors (k)
The Order Neighbors (k)
Figure S: Hyperparameter Analysis on k
Ablation Studies. To verify the effectiveness of each component of SGRL, we conducted ablation studies on five datasets. As shown in Table "SGRL w/o TCM and TCM" demonstrates significant performance improvements on four datasets with the addition of RSM alone, i.e.. "SGRL w/o TCM". confirming its effectiveness. We observe that there is slight performance decrease on the Computers dataset, which we attribute to over-scattering resulting from absence of constraints during representation scattering. Additionally, comparison between SGRL and "SGRL w/o TCM" reveals that SGRL achieves improved accuracy across all five datasets, particularly on the Computers dataset. This indicates that the TCM effectively constrain scattering, achieving adaptive representation

---

scattering, verifying our view in Section 4 the potential adversarial relationship between RSM and TCM, we employed EMA balance this effect. The experimental results in Table g highlight the necessity of incorporating EMA to mitigate the adversarial interaction between RSM and TCM.
Finally, we performed an ablation study on EMA. Given
6 Conclusion
In this paper, made two significant contributions to Graph Contrastive Learning (GCL), an actively researched subject with numerous applications across diverse domains. First, through comprehensive analysis of the three popular GCL frameworks, we discovered common latent mechanism representation scattering that underlies these distinct contrastive methods. This discovery highlights essential feature of GCL and unifies these seemingly disparate frameworks under the concept of representation scattering. Despite their popularity, the existing methods have not fully leveraged this mechanism, leaving the potential of representation scattering largely untaped. However, applying representation scattering directly to GCL poses technical challenges. Our second contribution addressed this issue by introducing the Representation Scattering Mechanism (RSM) and the Topology-base Constraint Mechanism (TCM), which we integrate into a novel GCL approach named Scattering Graph Representation Learning (SGRL). In SGRL, we effectively balance the adversarial relationship between RSM and TCM using an Exponential Moving Average (EMA) strategy. Extensive experimental results benchmark datasets validate the effectiveness of proposed method. Future work will further explore the broader implications of representation scattering beyond GCL, discussed Appendices G
Acknowledgement
This work was supported by the National Natural Science Foundation of China [No. 62422210 10, No. U22B2036, No. 62276187), the National Science Fund for Distinguished Young Scholarship (No. 62025602), the XPLORER PRIZE. the Hong Kong RGC theme- based Strategic Target Grant (STGI/M-501/23NN) the Hong Kong Global STEM Professorship Scheme, and the Hong Kong Jockey Club Charity Trust.
References
[1] Sanjay Kumar, Abhishek Mallik, Anavi Khetarpal, and Bhawani Sankar Panda. Influence maximization in social networks using graph embedding and graph neural network. Inf. Sci., 607:1617-1636, 2022.
[2] Liangwei Yang, Zhiwei Liu, Yingtong Dou, Jing Ma, and Philip S Yu. Consisrec: Enhancing gnn for social recommendation via consistent neighbor aggregation. In Proceedings of the 44th international ACM SIGIR conference on Research and development in information retrieval, pages 2141-2144 2021.
[3] Xiao-Meng Zhang, Li Liang, Lin Liu, and Ming-Jing Tang. Graph neural networks and their current applications in bioinformatics. Frontiers in genetics, 12:690049, 2021.
[4] Hai-Cheng Yi, Zhu-Hong You, De-Shuang Huang, and Chee Keong Kwoh. Graph representation learning in bioinformatics: trends, methods and applications. Briefings in Bioinformatics, 23(1):bbab340, 2022.
[5] Zhiwei Liu, Yingtong Dou, Philip Yu, Yutong Deng, and Hao Peng. Alleviating the inconsis- tency problem of applying graph neural network to fraud detection. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 1 1572, 2020.
[6] Yizhen Zheng, Shirui Pan, Vincent Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. Advances in Neural Information Processing Systems, 35:10809-10877 2022.
10

---

[7] Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[8] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, foss/2006.0413/ 2020.
[9] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 2069-2080. ACM IW3C2, 2021.
[10] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning graphs. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning, 2021.
[11] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Li.. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering. 35(4):4216-4235, 2021.
[12] Jun Xia, Lirong Wu, Ge Wang, Jintao Chen, and Stan Z. Li. Progel: Rethinking hard negative mining in graph contrastive learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24332-24346. PMLR, 2022.
[13] Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 7372-7380. AAAI Press, 2022.
[14] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807 03744, 2018
[15] Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020, Viriual Event, volume 119 of Proceedings of Machine Learning Research, pages 41 16 4126. PMLR, 2020.
[16] Wangbin Sun, Jintang Li, Liang Chen, Bingzhe Wu, Yatao Bian, and Zibin Zheng. Rethinking and simplifying bootstrapped graph latents. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 665-673, 2024.
[17] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages
PMLR, 2020.
[18] Thomas N. Kipf and Max Welling. Semi-supervien classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[19] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[20] Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[21] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst., 22(1):4-44 2021.
11

---

[22] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807-814, 2010.
[23] Hengrui Zhang, Qitian Wu, Yu Wang, Shaofeng Zhang, Junchi Yan, and Philip S Yu. Localized contrastive learning on graphs. arXiv preprint arXiv:2212.04660 2022.
[24] Dongxiao He, Jitao Zhao, Cuiying Huo, Yongqi Huang, Yuxiao Huang, and Zhiyong Feng. A new mechanism for eliminating implicit conflict in graph contrastive learning. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014. February 20-27, 2024, Vancouver, Canada, pages 1-3400 AAAI Press, 2024.
[25] Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago. Chile, August 9-13, 2015, pages 43 52. ACM, 2015.
[26] Peter Mernyei and C Wiki-CS Cangea. A wikipedia-basee benchmark for graph neural networks. arxiv 2020. arXiv preprint arXiv:2007.02901 2007.
[27] Arna Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Fide, Bo-June Paul Hsu, and Kuansan Wang. An overview of microsoft academic service (MAS) and applications. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015- Companion Volume, pages 243 233-246. ACM, 2015.
(28) Haifeng Li, Jun Cao, Jiawei Zhu, Qinyao Luo, Silu He, and Xuying Wang. Augmentation-ffee graph contrastive learning of invariantdiisrriinative representations. IEEE Trans. Neural Networks Learn. Syst. 35(8):
2024.
[29] Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V. Chawla. Graph barlow twins: A self- supervised representation learning framework for graphs. Knowl. Based Syst., 256: 256:109631, 2022.
[30] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. pages 855-864. ACM, 2016.
[31] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA August 24- 27, 2014, pages 701-710. ACM, 2014.
[32] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[34] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: htt::/903.000 1903.02428. 2019.
[35] Fenxiao Chen, Yun-Cheng Wang, Bin Wang, and C-C Jay Kuo. Graph representation learning:
a survey, APSIPA Transactions on Signal and Information Processing, 9:e15, 2020.
[36] Yuefeng Zhang. An introduction to matrix factorization and factorization machines in recom- mendation system, and beyond. arXiv preprint 2rXiv:2203. 1006,, 2022.
[37] Gregory Lawler and Vlada Limic. Random walk: a modern introduction, volume 123. Cambridge University Press, 2010.
12

---

[38] Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vee: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 385-394, 2017.
[39] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, pages 1024-1034, 2017.
[40] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[41] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[42] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21288 2020.
13

---

A Discussion of Uniformity
In visual contrastive learning. uniformity recognized as a critical factor contributing the effec- tiveness of contrastive methods. In [17]. uniformity is defined as the condition where feature vectors should be roughly uniformly distributed on the unit hypersphere, preserving as much information of the data possible. This concept is based on the InfoNCE loss. Due to the similarities in implemen- tation mechanisms between SimCLR framework and the Deep InfoMax (DIM) framework, DIM can be considered a special case of SimCLR when the number of negative samples within a batch is limited to one. Consequently, this concept can be seen as key factor contributing to the success of most visual contrastive learning methods.
However, this concept fails to adequately unify the frameworks within graph contrastive learning, particularly for DGI framework. In DGI, all nodes of the original graph are treated single distribution, whereas the distribution generated from a randomly shuffling noise graph is treated another. The encoder is trained by maximizing the Jensen-Shannon (JS) divergence between these two distributions. First, the representations
DGI do not conform to distribution on the unit hypersphere. Moreover, considering all nodes of the original graph as one distribution more closely aligns with process of aggregation rather than facilitating a uniform distribution of nodes throughout the space.
Furthermore, while the negative sampling mechanism of InfoNCE-based methods implements uniformity, the interconnected nature of graphs can lead to conflicts during message passing within the encoder, which hinders the effective distancing of negative samples [24.. Therefore, we conducted comprehensive theoretical analysis of three mainstream graph contrastive frameworks (Section 3) and introduced the concept of representation scattering. Specifically, the center-away constraint within representation scattering provides more detailed elucidation of the operational mechanisms underlying the DGI framework than uniformity. Moreover, this constraint effectively mitigates previously discussed confficts, thereby facilitating the integration of representations and topological information and enabling adaptive representation scattering.
B Experiment Details
B.1
Hyperparameters Settings
Table 4: Detailed hyperparamcters of SGRL..
Datasct
Hidden dim
online learning rate
target learning rate
Training epochs
Activation
momentum
WikiCS
1024
0.00001
PRel.U
Maazon-Compaters
1024
0.001
0.001
700
PRcLU
0.99
Amazon-Photo
1024
0.001
0.001
700
PRcLU
0.99
Coauthor CS
1024
0.001
0.001
700
PReLU
0.99
Coauthor Physies
1024
0.0001
0.00001
1000
PReL.
0.99
B.2 Datasets
We use Wiki-CS, Amazon-Computers (Computers), Amazon-Photo (Photo), Coauthor-CS and Coauthor-Physee Physscs to evaluate the performance SGRL. The detailed statistics all the used datasets are in Table 5
Table 5: Statistics of datasets used in this paper.
Dataset
Type
Nodes
Edges
Autributes
Classes
Wiki-CS
reference
11,701
216,123
300
10
Amazon- MaazoonCoompters
co-purchase
13,381
245,778
767
Amazon-Phann
co-purchase
7,487
119,043
745
8
Coauthor-CS
co-author
18.333
81,894
6,805
15
Coauthor-Physice
co-author
34.493
247.962
8,415
5
Wiki-CS (6 is a directed graph dataset sourced from Wikipedia. comprises nodes that represent articles in the field of computer science, with edges corresponding the hyperlinks interconnecting
14

---

these articles. Each article is categorized into one of 10 associated subfields. The attributes of the nodes are derived as the average of the text embeddings for the respective articles, making it the sole dataset characterized by dense attributes.
25 are networks that map out the co-purchase rela- tionships among products on the Amazon platform. In these networks, nodes represent products, and edges are established between products that are frequently bought together. Products within each network are categorized into 10 and 8 distinct classes respectively, according to their product categories, and node features are represented using bag-of-word. model from the product reviews.
Amazon-Computtrs and Amazon-Photo
Coauthor-CS and Coauthor-Physice [27] represent two academic networks based on the Microsoft Academic Graph, illustrating the co-authorship links among academics. In these networks, nodes correspond to individual authors, while edges reflect collaborative authorship between them. Authors in each network categorized into 15 and 5 research fields respectively, with node features encapsulated as a bag-of-words representation derived from the keywords their publications.
B.3 Environment Configurations
All experiments were carried out NVIDIA GeForce GTX 3090 GPU, which comes equipped with 24GB of memory. For model development, we utilized Py Torch version 1.13.1 (3] along with PyTorch Geometric version 2.3.0 34], which also served the source for all the datasets used in study. The detailed hyperparameters for node classification and clustering are shown in Table 4
B.4 Implementation Details
Our method employs GCN the encoder. The encoding process be described as follows:
H (1)
GCN k-1: A),
(8)
where H represents the node embedding matrix at the 1-th layer during the k-th epoch of encoding and H is initialized by graph feature matrix X.and A is the adjacency matrix without self-loops. In addition, the encoder structure is defined as:
GCN (H A) o 1/2 - 1/2
(9)
where À = A+ is the adjacency matrix with self-loops, (-) is the PReLU activation function, D is the degree matrix of Å, and (())) is the trainable weight matrix for the I-th layer. In our experiments, we set | 1.
C Detailed Discussion on Assumption
1
Assumption 1.(a) In assumption 1.(a), we define the normalized propagation matrix as Å = D-1 A. When the encoder is GCN 18], assumption 1. 1.(a) holds. Next, we will discuss whether assumption 1 1.(a) holds in Corollary 3 when the encoder is GAT [19. The message-passe paaeeeppssnnggggg formula for GAT is similar to that GCN. For node f;, h, = GAT(A, X) Eneighhor j Xij W, but the method for obtaining the propagation matrix different:
exp(LeakyReLU(aa Wx Wx;)) Oij REV(() exp(LeakyReLU(a' [Wx;]] [Xx+))
(10)
where a is the weight vector used to compute the attention scores, || denotes the concatenation operation, W is the weight matrix, and LeakyRel.U is the activation function. In the original graph, node v and its first-order neighbors follow distribution with mean μ4 and variance of, i.c. Vj E Ni, zj (p(,; Subsequent to GAT aggregation, the representation of node v, i... h, follows local semantic distribution with a mean given by EE (> acyyaay) = Σ a,jii; and a variance of of a. This is consistent with the conclusion in Section 3.1 In the perturbed graph, the method for calculating the aggregated distribution differs from that of GCN. For GCN, the propagation matrix Oij = 1/d, is independent of the distribution of node v/ and its neighbors. contrast, for GAT, as shown in Eq. 10 an reflects the correlation between node and its neighbor uy. Therefore, in
15

---

the perturbed graph, there are two distributions present among node vs and its first-order neighbors: the local semantic distribution pt and the original data distribution Pdata- Let = Σ asks where Ty ~ [1,, of). Let b Σ oas, where J. ~ Pdata (1, (1, o²). After aggregation through GAT, the mean the distribution that node v; follows a - 44 + b j.. this case, minimizing the DGI loss equívalent to maximizing the local semantic distribution of node with mean of a 14, + b μ and
a variance of of Σ ai + o² Σ az
In our problem,
aim find subspace S and scattered center achieve representation scattering, and this goal can be accomplished with either GCN or GAT. To this end, Corollary also holds when the encoder is GAT.
Assumption L.(b) For assumption I.(b), there are two ways to generate a corrupted graph in DGI 77 one is by randomly shuffling the feature matrix X. and the other is by keeping X. unchanged and shuffling the adjacency matrix A. Here, we discuss whether Theorem still holds when the assumption is the latter.
We have already demonstrated Section 3.1 that, in the original graph G. the aggregated features of node follow distribution with mean and variance of ΣjeN()) oty. In the corrupted graph G. randomly shuffling the adjacency matrix A is equivalent to randomly re-wiring n nodes, where E: Ei= A.j = Ei= Ej=1 Aij. The first-order neighbors of node 2 still derive their features from the original graph distribution, i.e., fig. Pdam (1, o²), and in the corrupted graph, the new topology still satisfies [(as) = 1. Therefore, the aggregated representation follows a distribution with mean μ and variance o² EjEN(e) ozyy which is consistent with the conclusion derived from assumption (b).
Assumption 1.(c) In our theoretical analysis, we assume class balance, which is common in many theoretical proofs 77]. When the classes are imbalanced, the conclusion similar to what we explained in assumption 1. (a): the scattered center changes from the mean to f4i + b (s, i.e., it leans towards the classes with a larger number of instances. This often occurs long-tail distribution problems. We will further explore this aspect in our future work.
D Proof
D.1
Proof of Lemma
Proof. Here, we rewrite the proof from work 66 for ease of understanding Lemma 2 summary vector s with an all-ones vector. By removing the weight parameters W in the discriminator
We replace the
D. we obtain:
N log D(h;, s) + +log(1 D(h,, s))
LPGI =
2N
1
log(h; 8) + log(1 h; .s)
(11)
2N
1
log(sum(ha)) log(1 sum(h.)))
2N
where sum(-) is the summation function for every dimension in vector h, As described by Eq I1 minimizing DGI loss can viewed as binary classification problem distinguishing between sum(h; from the original graph and sum(h;) from the perturbed graph. Therefore, minimizing DGI loss is equivalent to maximizing the JS divergence between the original and perturbed graphs. O
D.2 Proof of Theorem S
Proof. Consider the output of single-layer Graph Neural Network (GNN) as H E N where N represents the batch size and K denotes the dimension of representations. Fach column H, of H represents all samples for particular feature, with mean fy and standard deviation d,.
16

---

Batch Normalization (BN) processes each feature dimension j independently by calculating the mean Mj and variance of. and then applying the following transformation:
HJ BN = H, MJ 03 +
Mj
H 5
(12)
where is a small value to prevent division by zero. The initial representations H are distributed on an arbitrary space characterized by mean center μ = {pI..... ........,, space with zero mean and unit variance. This transformation effectively constitutes a center-away displacement, i.e., each feature vector shifted away from the center of the original distribution. Moreover, Batch Normalization ensures that each feature dimension exhibits a variance of unity, thereby facilitating a more uniform distribution across the transformed representations. Although the primary objective of batch normal- ization is not representation scattering, it can be inferred from the aforementioned analysis that BN's adjustment of data distribution indirectly facilitates a more uniform scattering of representations across space.
the space of the transformed representations H' is standardized Mxc). J. After batch normalization,
D.3 Proof of Theorem 4
Proof. It is well-known that the InfoNCE loss (with /2 norm) is enforcing the embeddings to be uniformly distributed in hypersphere. is exactly representation scattering. We begin with analysing the formula of the InfoNCE [1]
loss:
(b..h b()
LInFONCE (h.) log(( (b.,, + ( (hh(b) + (hb
(13)
where h, denotes the representation node ve, obtained from one augmented view of the original graph, and h represents the representation of the corresponding node ef from the other augmented view. Here, (h,, h²)= = Sim(h,, ht)/T, where Sim(.) denotes the cosine similarity between node representations. Specifically, Sim Sim(h,, hτ) is computed the normalized dot product: Sim(h, ht) (h, (ll/ Eq. 13 states that minimizing the loss function requires the maximization of similarity among positive samples and the simultaneous minimization of similarity among negative samples.
To simplify the proof, define (b, h() as the cosine similarity between node pairs, omitting the hyper-parameter т. Furthermore, we replace the logarithm with the natural logarithm In in Equation (l3) yielding the expression below:
LIntaNCE (h,) -h,, h() + Σラ=
( (e + b())
(14)
To facilitate a clearer analysis of Equation [44 and given the convex nature of the exponential function, we simplify the expression using Jensen's 's Inequality:
2n
(15)
2n
where 8, represents the cosine similarity between the anchor node v and other nodes that form negative pairs. According to Eq. (15), the sum of exponentials can be transformed into the exponential of a sum, which can then cancel out with the logarithm, thereby simplifying the equation:
2n-1
LInGONCE (h+) 2
Σ 0, + In(m).
(16)
2m
In Eq. (16), it is demonstrated that the InfoNCE loss associated with the anchor node ;; serves as an upper bound to the sum of cosine similarities between and all negative samples. To elucidate the relationship between representation scattering and InfoNCE loss more clearly, the following lemma is presented:
Lemma 6. Let 0; be the anchor node, and let {01, U2 The average of the sum of cosine similarities between and all other nodes in the set is equivalent to the cosine similarity between the eg- fo-ormmalieed vector of us and the Ez-normalized mean vector obtained from all other nodes.
035 ;0) be the set of nodes, excluding vs-
17

---

The proof is provided in Appendix D.4 Lemma demonstrates that the minimization of Eq. (33 for the anchor node v entails a concurrent reduction in the cosine similarity between the nn-norallieed vector of vj and the 'z-normalized centroid of all other nodes. This process can be viewed as transforming the set of nodes onto a hypersphere with a radius of and distancing the anchor node from the centroid of other nodes. We consider this hypersphere as subspace St and the centroid as the scattered center c, which aligns with our definition of representation scattering, Thus, minimizing the InfoNCE loss effectively promotes representation scattering.
D.4 Proof of Lemma 6
Proof. Given two vectors Vi and Vj, their cosine similarity is defined of the angle between them
v.V, |
which is the cosine
2n-1
2n
Σ
B,j
ViVj
(17)
|
jmljýıi
For Ez-nomallieed vectors v;
V: M this is equivalent their dot product: Vi Vy:
2n-1
2x-1
Vivj
Σ V.
V.
(18)
V
j=1.jıi
j=1.jæx
Since the dot product of vectors possesses the distributive property of multiplication:
2n
2n-1
V
Vj
= Vi
(19)
j= 1,
Finally, we compute the mean of Eq. 19
O.j
V
Σ
vJ
Vi E evss
(20)
2n
2n
J=1,
J=1,
As a consequence, the mean the sum of cosine similarity between the anchor node v; and other nodes can be regarded as the expected similarity of the En-nomallzed o with all other &g normalized points.
E Related Work
Graph Representation Learning (GRL) has attracted substantial attention in recent years due the broader application of graph data across various real-world scenarios (35). The primary goal of GRL embed graph data, which is inherently high-ddmensional, sparse, and Euclldeid,,,,, into lower- dimensional, dense, Euclidean space to facilitate easier processing in downstream tasks. Early GRL works rely on Matrix Factorization 36 and Random Walk (37], failing simultaneously utilize the structural and attribute information of graphs [38 In recent years, Graph Neural Networks (GNNs) have achieved great success, such as GCN [88. GAT [9,, GraphSAGE 39, which effectively extract graph attribute and structural information simultaneously through message passing and aggregation mechanisms. However, the training GNNs requires large amount of manually annotated labels, which is expensive and labor-intensive 6 To overcome this limitation, Graph Contrastive Learnings (GCLs) have emerged as a promising and increasingly popular learning paradigm for handling unlabeled graph data.
Currently, there are three mainstream graph contrastive frameworks, each inspired by methods from visual contrastive learning. DGI (7]. inspired by Deep InfoMax (DIM) 40), generates noise distribution through random shuffling and learns node representations by maximizing the Jensen- Shannon (JS) divergence between the semantic distribution of the original graph and the noise distribution. GRACE R a representative method based on InfoNCE loss and inspired by SimCLR 41. generates two correlated graph views through augmentation techniques. This framework employs the InfoNCE loss to enhance the similarity of identical nodes across the two views while minimizing the similarity among different nodes. BGRL [10), inspired by BYOL 42), utilizes two
18

---

distinct encoders: an online encoder and a target encoder. The online encoder is trained to predict the output of the target encoder, while the target encoder is updated by Exponential Moving Average (EMA)..
Although the three mainstream graph contrastive learning frameworks: GRACE, DGI, and BGRL, have achieved notable success in addressing the challenges of unlabeled graph data, the underlying mechanisms driving their success have not been fully explored. Current research predominantly focuses on the implementation and optimization of these frameworks 6 b 12 [3] with insufficient understanding of their theoretical foundations and intrinsic principles. In visual contrastive learning. studies have revealed that "uniformity", which refers to the uniform distribution of representations in hypersphere, is a key factor contributing to improved generalization and discriminative power of models [17]. However, due to the non non-Euclidean structure of graphs and the complex dependencies among nodes, this theory cannot be directly applied to graph contrastive learning. The unique char- acteristics of graphs necessitate a reevaluation and adaptation of the uniformity concept to develop a corresponding theoretical framework suitable for graphs. In this paper, we reanalyze the three mechanism: representation
mainstream graph contrastive learning frameworks and propose scattering, which is prevalent and important in these frameworks. Furthermore, considering the inter- connected nature of graphs, we design a constraint mechanism to achieve adaptive representation scattering.
F
A Detailed Discussion on the Limitations
Application of Heterogeneous Datasets. In this work, we focused exclusively on ho- mogeneous datasets and achieved satisfactory results. Our experiments demonstrated the effectiveness of the proposed topology-based constraint under the assumption of homogene- ity. However, this constraint may face challenges when applied to heterogeneous datasets. Heterogeneous datasets often exhibit varying structures and characteristics, which could undermine the performance of our method. Consequently, addressing these challenges and adapting our approach handle heterogeneous datasets effectively is crucial direction for future research. We plan to investigate and refine our method to ensure its robustness and applicability across diverse and complex datasets.
The Methods of Balancing RSM and TCM. RSM aims for the representations of nodes to be scattered from each other, while TCM prevents excessive scattering among nodes with the same semantics. These two mechanisms inherently have an adversarial relationship. We use EMA to balance this adversarial relationship, which is just one of many possible methods.
G Broader Impacts of SGRL
As preliminary exploration in the field of GCL, SGRL has the following impacts:
Uncovering potential success factors in GCL: SGRL explores three different mainstream graph contrastive methods and identifies their common underlying factors, providing new perspective for future GCL research.
Applications in many fields: Compared to existing methods, SGRL offers higher accuracy and can more efficiently process graph data. It can be widely applied in various fields such as social networks, biological networks, and recommendation systems.
Negative social impacts: There are no negative social impacts foreseen.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction in this paper accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification: In this paper, all proofs of theorems are provided and all assumptions are clearly stated or referenced in the statement of any theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the results in this paper can be reproduced.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
publ icllic//ee deeuidee//oeeeee
ss Lonoolic cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ ctc.pp.ccppubbic
CooeeoodeSSSa Submisbbbsss ss swssoooolicyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report eтror bars suitably and correctly detined or other appropriate information about the statistical significance of the experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, the paper provide sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification; The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "yJephZ5t7M", "reviewer": "Reviewer_HnqZ", "summary": "The authors attempt to propose a universal theory of graph contrastive learning which may benefit this field. Most existing GCLs directly inherit from other fields. While existing GCLs have achieved similar success, there are intuitive differences and even conflicts in the operations. By analyzing three representative GCL frameworks from the unique perspectives of topology and message passing, the authors of this work find a key factor behind GCLs, which is defined as representation scattering. To better achieve node representations scattering, they mine the natural linking characteristic of graphs and propose a new contrastive concept. It involves contrasting nodes with embedding centers and implementing aggregation constraints based on graph topology, which replaces the traditional inefficient augmentation and sampling-based GCL proxy tasks. The proposed method is compared to existing GCL methods across multiple downstream scenarios to demonstrate the superiority, confirming it can learn high-quality node representations in a self-supervised manner.", "questions": "The analysis of DGI is based on the assumption that the perturbation is randomly shuffling the node features. While the authors have provided discussions in Theorem 1 when this assumption fails, I wonder whether the discussion on the shortcomings of DGI-like methods remains valid when this assumption fails.", "limitations": "NA", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors propose a new theory that unifies existing GCLs, which is insightful and may have implications for a broad field. \n\n2. The presentation of the paper is clear and easy to understand. \n\n3. The theoretical analysis is comprehensive and rigorous. This reveals the underlying mechanism of the success of existing GCLs, and gives strong theoretical support to the proposed method. \n\n4. The empirical evaluation is sufficient, showing powerful performances across various datasets. \n\nOverall, I think this paper makes a significant contribution to the field of graph representation learning.", "weaknesses": "1. The proposed SGRL is an augmentation-free framework, avoiding manual bias and reducing training overhead in augmentation. To my knowledge, there are also some augmentation-free contrastive methods available, such as [1, 2]. While data augmentation is not the main focus of this paper, it may be beneficial if the authors provide a comparison between SGRL and this type of method.\n\n2. In SGRL, an existing component, EMA [3, 4], is given new functionality to balance the adversarial effects of two constraints (RSM and TCM) and achieve satisfactory results. I find this innovation interesting. However, it would be better if the authors provided further discussions and experiments on the chosen EMA’s hyper-parameters, since different hyper-parameters of EMA may make the two adversarial branches achieve different equilibrium.\n\n3. The provided theory primarily discusses the impact of representation scattering mechanisms on various GCLs, encompassing multiple aspects such as data augmentation, GNN encoders, and negative contrasting. However, the authors seem to have omitted positive contrasts. While topological aggregation constraints can be seen as a more effective form of positive sample contrast, I still hope the authors can incorporate positive contrast into the proposed theory.\n\nSome minor points:\n\n1. p. 2 line 72 - I am not sure what is meant by \"learning invariance post-disturbance.\"\n\n2. p. 3 line 149 - Figure 1 is mentioned too frequently within a single section.\n\n3. Table 1: Some names of the datasets used have prefixes (Co.CS and Co.Physics) while others do not (Computers and Photo). It would be better to maintain a consistent format.\n\n[1] Lee N, Lee J, et al., Augmentation-free self-supervised learning on graphs, AAAI, 2022.\n\n[2] Yu J, Yin H, et al., Are graph augmentations necessary? simple graph contrastive learning for recommendation, SIGIR, 2022.\n\n[3] Thakoor S, Tallec C, et al., Large-scale representation learning on graphs via bootstrapping, ICLR, 2022.\n\n[4] Grill J B, Strub F, et al., Bootstrap your own latent-a new approach to self-supervised learning, NeurIPS, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time in reading our paper and giving valuable suggestions. To address your concerns, we respond below.\n\n> The proposed SGRL is an augmentation-free framework, avoiding manual bias and reducing training overhead in augmentation. To my knowledge, there are also some augmentation-free contrastive methods available, such as [1, 2]. While data augmentation is not the main focus of this paper, it may be beneficial if the authors provide a comparison between SGRL and this type of method.\n\nThanks for your suggestion. We will provide a comparison between SGRL and [1, 2] regarding data augmentation. \n\n- [1] generates positive samples of original nodes through k-NN search and filters these samples from both local (whether nodes are connected) and global perspectives (whether they belong to the same cluster), which overemphasizes structure information. \n- [2] generates different views by introducing random, uniform noise into the original representations, neglecting the impact of the structure information. \n- In contrast, SGRL considers both structure and attribute information. Specifically, SGRL employs an asymmetric dual-channel design, generating views from a structure perspective with topological semantics ($H_{online}^{topology}$) and an attribute perspective with scattered representations ($H_{target}$). Therefore, our augmentation-free method supports a more comprehensive utilization of the graph's information.\n\n> In SGRL, an existing component, EMA [3, 4], is given new functionality to balance the adversarial effects of two constraints (RSM and TCM) and achieve satisfactory results. I find this innovation interesting. However, it would be better if the authors provided further discussions and experiments on the chosen EMA’s hyper-parameters, since different hyper-parameters of EMA may make the two adversarial branches achieve different equilibrium.\n\nThank you for suggesting improving the experiment. To investigate the impact of EMA on the two branches, we evaluated the performance of SGRL by changing the hyper-parameter $\\tau$. We set the value of $\\tau$ to 0.99 (align with the value in our paper), 0.95, 0.90, 0.80, 0.50, 0.00 (SGRL-EMA), and the experimental results are shown in Table Re-HnqZ-1. \n\n**Table Re-HnqZ-1: The Impact of EMA.**\n|| 0.99 | 0.95 | 0.90 | 0.80 | 0.50 | 0.00 |\n|-|-|-|-|-|-|-|\n| Co.CS | 94.15±0.04 | 94.08±0.02 | 94.11±0.02 | 94.09±0.05 | 94.04±0.01 |\t93.89±0.07 |\n| Co.Physics | 96.23±0.01 | 96.18±0.05 | 96.16±0.03 | 96.16±0.01 | 96.17±0.02 |96.16±0.07 |\n| Wiki-CS | 79.40±0.13 | 79.40±0.08 | 79.38±0.08 | 79.39±0.09 | 79.38±0.06 | 79.36±0.08 |\n\nIt is evident from the result that SGRL achieves the best performance when $\\tau = 0.99$. This is because the topological semantic information is gradually fed into the representation scattering mechanism at each epoch, which moderates the adversarial interaction between the two branches. In addition, when the topological semantic information excessively influences the representation scattering process (as $\\tau$ decreases), SGRL’s performance slightly declines but remains better than SGRL-EMA ($\\tau = 0.00$). This result indicates that balancing the two adversarial branches is necessary, which is consistent with our conclusions in Section 5.2. We will explore more effective balancing methods in future work.\n\n> The provided theory primarily discusses the impact of representation scattering mechanisms on various GCLs, encompassing multiple aspects such as data augmentation, GNN encoders, and negative contrasting. However, the authors seem to have omitted positive contrasts. While topological aggregation constraints can be seen as a more effective form of positive sample contrast, I still hope the authors can incorporate positive contrast into the proposed theory.\n\nSGRL is different from traditional contrastive learning based on positive and negative sampling. In SGRL, we do not focus on defining positive and negative samples. Instead, we propose two mechanisms that train the encoder in an adversarial-like manner. In addition, positive contrasting is usually used to train the encoder to learn consistent representations across different views, whereas TCM aims to regulate the scattering of representations in space. \n\n> p.2 line 72.\n\nIt indicates that TCM can enhance the robustness of SGRL. We explain this in Section 4.3.\n\n> p.3 line 149 and Table 1.\n\nThanks for your suggestion. We will make corrections in future version.\n\n> The analysis of DGI is based on the assumption that the perturbation is randomly shuffling the node features. While the authors have provided discussions in Theorem 1 when this assumption fails, I wonder whether the discussion on the shortcomings of DGI-like methods remains valid when this assumption fails.\n\nSorry for the confusion. We follow the setup described in Section 3.1: Node $v_i$ and its first-order neighbors follow the distribution $ p_i(\\cdot)$ over $\\mathbb{R}^{M+1}$. In DGI, they design the other corruption function $C$ by sampling, i.i.d., a switch parameter $\\Sigma_{ij}$ to determine whether to corrupt the adjacency matrix at position $(i,j)$. Given a corruption rate $\\rho$, they get perturbed graph by $\\hat{A} = A \\oplus \\Sigma$, where $\\oplus$ is the XOR (exclusive OR) operation and $\\Sigma_{ij} \\sim \\text{Bernoulli}(\\rho)$. In this case, node $v_i $ has a probability of $1/2k$ to connect with node $v_j$, where $v_i, v_j \\sim p_i(\\cdot)$. However, DGI indiscriminately maximizes the Jensen-Shannon divergence between the original graph and the perturbed graph, treating such cases as negative samples, which leads to additional bias. Although the probability changes (from $1/k$ to $1/2k$), this does not affect the analysis of the shortcomings of DGI-like methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swYIWc9jBr", "reviewer": "Reviewer_ErfB", "summary": "This paper provides an insightful perspective of representation scattering to unify various GCL frameworks, and proposes an effective framework called SGRL. Specifically, the contributions are as follows: 1) Theoretically, with a well-defined representation scattering concept, the authors provide a universal theoretical explanation for the success of existing GCL frameworks. 2) They propose SGRL, which employs a unique adversarial approach to effectively utilize this mechanism. Specifically, SGRL integrates topological aggregation with representation scattering, with two adversarial objectives smoothed by EMA. 3) The proposed SGRL achieves powerful performances in extensive experiments across multiple tasks.", "questions": "See the above weakness.", "limitations": "Yes, they have.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The theoretical foundation is solid. The authors provide a new concept of representation scattering with a clear mathematical definition. All theoretical claims have formal and rigorous proofs. In addition, appropriate motivation experiments are provided to support their theorems.\n2. The authors design a well-motivated and novel framework. Through a comprehensive exploration of representation scattering, SGRL seems break through the limitations of existing methods and can be regarded as a new branch of GCL. Besides, SGRL integrates topological aggregation with representation scattering via adversarial strategy, which is interesting and technically reasonable.\n3. SGRL shows strong performance across various tasks on multiple datasets.", "weaknesses": "1. Although the results in Figure 5 clearly demonstrate how model performance varies with different strengths of topological constraints, I believe it would be beneficial for the authors to present more results from additional datasets. To my knowledge, the preprocessing method used for Wiki-CS often differs from that of the other four datasets, which may influence the choice of $k$. Therefore, I would appreciate it if the authors could provide further experiments to enrich their analysis.\n2. In Definition 1, there are two constraints in representation scattering, the center-away constraint and the uniformity constraint. When considering only the goal of making representations scattered enough, the center-away constraint seems somewhat redundant, as satisfying the uniformity constraint alone can ensure the discreteness of representations. The authors should provide additional explanations regarding the role of the center-away constraint in the process of representation scattering.\n3. I suggest that the authors provide more detailed explanations of some symbols, even though this is common in graph representation learning. For example, the $\\alpha_{ij}$ and $d_i$ in Equation 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time in reading and reviewing our submission. We respond below.\n\n> Although the results in Figure 5 clearly demonstrate how model performance varies with different strengths of topological constraints, I believe it would be beneficial for the authors to present more results from additional datasets. To my knowledge, the preprocessing method used for Wiki-CS often differs from that of the other four datasets, which may influence the choice of \n. Therefore, I would appreciate it if the authors could provide further experiments to enrich their analysis.\n\nThank you for the valuable suggestion. In Section 5.2, we conduct a sensitivity analysis of $k$ and show the results in Figure 5, showing the impact of topological constraint. Your comments have made us realize that we need to include more results from datasets like Wiki-CS, which utilize a unique pre-processing method, to help better understand this mechanism. To enhance the readability of our paper, we present additional experimental results from other datasets in Table Re-ErfB-1. Admittedly, different pre-processing methods may influence the peak of SGRL, but they do not affect the analysis presented in our paper.\n\n**Table Re-ErfB-1: Additional Hyper-parameter Analysis on $k$.**\n|| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| Wiki-CS | 78.56±0.05 | 79.40±0.10 | 79.48±0.01 | **79.54±0.03** | 79.53±0.01 | 79.45±0.04 | 79.37±0.03 | 79.28±0.06 | 79.21±0.06 | 79.20±0.04 | 79.11±0.05 |\n| Amazon.Photo | 93.54±0.05 | **93.95±0.03** | 93.84±0.02 | 93.83±0.02 | 93.80±0.01 | 93.76±0.02 | 93.73±0.01| 93.73±0.02 | 93.69±0.05 | 93.72±0.05 | 93.71±0.05 |\n| Co.Physics | 96.16±0.03 | **96.23±0.01** | 96.18±0.02 | 96.18±0.03 | 96.18±0.03 | 96.17±0.02 | 96.17±0.01 | 96.17±0.01| 96.16±0.02 | 96.18±0.01 | 96.18±0.01 |\n\nThe results are still consistent with the analysis in the paper: considering the differences in the degree of scattering of different node representations is necessary and the hyper-parameter $k$ exhibits an unimodal effect on SGRL's performance. As $k$ increases, the model performance initially improves, indicating that topological constraint is effective. With the continuous increase of $k$, exceeding its peak will lead to a decline in performance. This is due to excessive constraints intensifying the antagonism between TCM and RSM, leading to RSM failure. Overall, although the pre-processing methods of different datasets may be different, this does not affect our conclusion. We will include this discussion in future version.\n\n> In Definition 1, there are two constraints in representation scattering, the center-away constraint and the uniformity constraint. When considering only the goal of making representations scattered enough, the center-away constraint seems somewhat redundant, as satisfying the uniformity constraint alone can ensure the discreteness of representations. The authors should provide additional explanations regarding the role of the center-away constraint in the process of representation scattering.\n\nIn fact, the center-away constraint is a vital component of the representation scattering mechanism. The reasons are as follows.\n- **Clarification of Scattering Mechanism:** With this constraint, Definition 1 delineates more precisely the scattering mechanism inherent in three frameworks (InfoNCE, DGI, BGRL), dictating how node representations achieve scattering within the embedding space. Despite their diverse approaches, all these frameworks incorporate a principle of moving away from the center.\n\n- **Expressiveness of Representations:** If node representations are close to the center, the expressiveness of representations will be weakened. For instance, in BGRL-like methods, the lack of center-away constraint will result in a concentration of the representations near the center, which diminishes the informativeness of the embeddings and reduces their distinctiveness. This is also a drawback of Batch Normalization (BN). \n\nTherefore, we introduce center-away constraint to ensure the completeness of the representation scattering theory. We will provide more explanations on this aspect in the appendix to complete the theorem.\n\n> I suggest that the authors provide more detailed explanations of some symbols, even though this is common in graph representation learning. For example, the $\\alpha_{ij}$ and $d_i$ in Equation 1.\n\nIn Equation 1, $\\alpha_{ij}$ represents the normalized connection weight between node $i$ and its neighboring node $j$, typically calculated as the element $A_{ij}$ from the adjacency matrix $A$ divided by the degree $d_i$ of node $i$, i.e., $\\alpha_{ij} = \\frac{A_{ij}}{d_i}$. We will supplement this explanation in future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R69YSgYIka", "reviewer": "Reviewer_dtE3", "summary": "In this paper, the authors provide an interesting discovery: the successes with mainstream GCL paradigms essentially come from implicitly scattering representations. They point out that the bottleneck of current GCLs lies in ignoring this, and they provide detailed theoretical proofs. Furthermore, they propose a new method to fully utilize representation scattering. They propose an asymmetric framework that, through carefully designed central discrete loss, enhances the distinctiveness of representations, resulting in favorable performance improvements.", "questions": "see weaknesses", "limitations": "see above", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The authors observe an interesting phenomenon: intuitively, DGI-based methods and InfoNCE-based methods seem conflicting on node-level proxy tasks, yet both achieve good performance. Previous work has overlooked the connection between them, but this paper unifies these paradigms, potentially providing insights for the future development of graph representation learning.\n3. Exploring and formalizing the definition of representation scattering is highly valuable. Previously, the notion of uniformity—intuitively understood as diversity in encoding negative samples—was only mentioned in graph studies based on InfoNCE. However, this paper provides a clear definition and broadens its application across all GCL paradigms.", "weaknesses": "1. As I mentioned in the summary, the authors have designed an asymmetric contrastive framework with two opposing types of losses, which may lead to misunderstandings of the training process. Although the description in the paper is given, including an algorithm flowchart would be better.\n2. In Figure 1, the authors plot the t-SNE visualization of DGI on Co.CS . When the number of encoder layers changes, two distinctly different results are produced. I hope the authors provide a detailed explanation of this phenomenon.\n3. Is the proposed central discrete loss effective in all scenarios? In extreme cases, if all node representations converge at a single point, can this loss achieve dispersion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and review our submission. We have provided our responses below.\n\n> As I mentioned in the summary, the authors have designed an asymmetric contrastive framework with two opposing types of losses, which may lead to misunderstandings of the training process. Although the description in the paper is given, including an algorithm flowchart would be better.\n\nThank you for your valuable suggestions. We apologize for any misunderstanding. To facilitate reader understanding, we will provide an algorithm flowchart in future version. Here, please allow us to clarify the structure of the SGRL algorithm.\n\nGiven a graph $\\mathcal{G}$, two different encoders $f_\\theta(\\cdot)$ and $f_\\phi(\\cdot)$ generate node embeddings $H_{online}$  and $H_{target}$ , respectively. \n\n- **Topological Aggregation:** $H_{online}$ is processed through TCM to obtain topologically aggregated representations $H_{online}^{topology}$, without updating the parameters of $f_\\theta(\\cdot)$ and $f_\\phi(\\cdot)$.\n\n- **Representation Scattering:** For $H_{target}$, we use RSM to encourage node representations to diverge from the center $c$, and update the parameters of $f_\\phi(\\cdot)$.\n\n- **Prediction and Parameter Update:** $H_{online}^{topology}$ is used to predict $H_{target}$ using the predictor $q_{\\theta}$. During this process, the parameters of $f_\\theta(\\cdot)$ are updated while the gradient of $f_\\phi(\\cdot)$ is stopped. Finally, we gradually update the parameters of $f_\\phi(\\cdot)$ using Exponential Moving Average (EMA).\n\nTo facilitate reader understanding, we will provide an algorithm flowchart in future version.\n\n> In Figure 1, the authors plot the t-SNE visualization of DGI on Co.CS . When the number of encoder layers changes, two distinctly different results are produced. I hope the authors provide a detailed explanation of this phenomenon.\n\nWe have provided explanations in the caption of Figure 1, but your comments made us realize that further clarification is needed.\n\n**Figure 1(a):** This figure shows the t-SNE embedding of randomly initialized GNN on Co.CS, where blue points represent the representation distribution of the perturbed graph and red points depict that of the original graph. It can be clearly observed that the representation distribution of the perturbed graph approximates the center of the original graph's representation distribution, which is consistent with the formulation provided in Theorem 1.\n\n**Figure 1(b):** This figure illustrates the t-SNE embedding after training on a one-layer GNN. Compared to Figure 1(a), there is a more pronounced separation between the specific semantic distributions of the original graph (red points) and the center, while the distribution of the perturbed graph (blue points) has converged more towards the center. Additionally, the intra-class boundaries within the original graph have become more distinct. The transition from Figure 1(a) to Figure 1(b) intuitively shows the outcome of DGI training, where the specific semantic distribution of the original graph diverges from the center. According to Corollary 3, this process is a specific case of representation scattering.\n\n**Figure 1(c):** This figure presents the result after training on a two-layer GNN. The incorporation of nonlinear activation functions makes Figure 1(c) align closely with the objective of DGI, which is to maximize the Jensen-Shannon (JS) divergence between the distributions of the original and perturbed graphs. Based on Figures 1(b) and 1(c), we can conclude that the training objective of DGI maximizes the JS divergence between the specific semantic distribution of the original graph and its mean distribution.\n\n> Is the proposed central discrete loss effective in all scenarios? In extreme cases, if all node representations converge at a single point, can this loss achieve dispersion?\n\nThe extreme scenario you mentioned, where all node representations converge to a single point, is indeed meaningless as it would result in indistinguishable node representations. We have considered a scenario similar yet meaningful to the one you described.\nConsider an embedding subspace $\\mathbb{E}$ containing a set of points $\\mathcal{V}$ in $\\mathbb{R}^n$. In this subspace $\\mathbb{E}$, for any $v_i, v_j \\in \\mathcal{V}$, their representations satisfy $||h_i - h_j||_2^2 < \\epsilon$. In this case, the center-away loss plays a significant role. It emphasizes distancing nodes from a relative mean center rather than a fixed absolute center. Initially, even if all nodes are clustered within the subspace $\\mathbb{E}$, by promoting their distance from the dynamic scattered center $\\mathbf{c}$, SGRL is still effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NgoBkZiWtC", "reviewer": "Reviewer_6M1t", "summary": "The authors focus on exploring an essential mechanism existing in different contrastive strategies. They first define a concept in embedding space, which contains a center $\\mathbf{c}$, a subspace $\\mathbb{S}$ and two constraints, called representation scattering. They then investigate the relationships between this concept and the mainstream GCL frameworks, through intuitive experiments and rigorous formula derivation. These discoveries motivate the authors to develop a new GCL framework aligned with this concept. The authors also present a model, namely SGRL, which includes a contrastive loss that directly makes representations away from the mean center. Experiments validate that SGRL has a better performance.", "questions": "In this work, the theories and methods proposed by the authors are mainly based on node-level tasks. So, the reviewer wonders whether this approach can achieve the same powerful performance in graph-level tasks.", "limitations": "The authors have discussed the limitations of their work adequately. I have no further concerns.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* As the paper states, existing work typically lacks a deeper insight into the mechanism behind various graph contrastive frameworks, which is a very important issue in GCL. This paper makes a positive contribution towards achieving this goal. \n* The proposed method SGRL is technically sound. The authors introduce the representation scattering mechanism and design SGRL following this new mechanism. Each section is supported by some convincing theorems and derivations.\n* The writing of this paper is clear and this work is comprehensive. The authors provide extensive experiments and appendices to substantiate their claims.", "weaknesses": "* The authors demonstrate that the representation scattering mechanism exists in several graph contrastive frameworks and argue that these methods do not fully utilize this mechanism. But the reviewer only observes discussions about the shortcomings of several baselines in the paper, lacking a deeper discussion on how these methods do not fully utilize representation scattering.\n*  In the viewpoints of the reviewer, the constraint based on topological aggregation seems unnecessary. Since the encoder already aggregates information from neighbors, adding an additional loss may lack a sufficient justification.\n* The SGRL employs two encoders with non-shared parameters, while the authors provide little explanation on this point. Based on the experience of the reviewer, allowing the two encoders to share parameters might perform better, as it may prevent divergences in the learned patterns between the models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions and comments. We respond below.\n\n> The authors demonstrate that the representation scattering mechanism exists in several graph contrastive frameworks and argue that these methods do not fully utilize this mechanism. But the reviewer only observes discussions about the shortcomings of several baselines in the paper, lacking a deeper discussion on how these methods do not fully utilize representation scattering.\n\nThank you for your comment. We have addressed your concern in Section 3. Here, we provide further explanation. First, most existing GCLs are mainly improved based on one of the following framework: the InfoNCE-based framework, the DGI framework, and the BGRL framework. As described in Definition 1, Representation Scattering is a simple and effective mechanism. However, the three frameworks and the methods based on them have not recognized for its importance, resulting in limited performance and low efficiency. \n\n**InfoNCE-based methods:** InfoNCE loss function indirectly achieves representation scattering by separating node pairs apart. It treats all negative samples indiscriminately and doesn’t differentiate their respective contributions to the loss. Moreover, it evaluates similarities for every possible pair within the batch, giving rise to $O(n^2)$ complexity. \n\n**DGI-like methods:** The objective of DGI-like methods is to maximize the Jensen-Shannon divergence between the original graph and the perturbed graph, which is a special case of representation scattering. However, the perturbed graph is unnecessary, leading to extra memory overhead and potential bias in negative samples. \n\n**BGRL-like methods:** The key component of BGRL-like methods, Batch Normalization (BN), is a special case of representation scattering. However, BN is not used in training; it is merely used to adjust the distribution in the embedding space. This could result in sub-optimal performance.\n\n> In the viewpoints of the reviewer, the constraint based on topological aggregation seems unnecessary. Since the encoder already aggregates information from neighbors, adding an additional loss may lack a sufficient justification.\n\nThanks for the comment, but we disagree respectfully. Topology-based Constraint Mechanism (TCM) is very important because it enables representations to be scattered more precisely in the embedding space, significantly improving performance. \n1) GNNs primarily aggregate local information and fail to effectively capture common features among local nodes (especially with fewer layers). In contrast, TCM enables node representations to be scattered globally while being aggregated locally in the embedding space. \n\n2) The introduction of TCM significantly enhances the capability of SGRL to process graphs. Different from image and text data, the linked nature of nodes results in connected nodes often being closer in the embedding space. The lack of TCM could lead to semantically similar nodes being distant and semantically dissimilar nodes being close. \n\n3) The ablation study of TCM is presented in Table 3 of our paper. The accuracy of SGRL-TCM decreases by 0.5% on average, with a notable reduction of approximately 1% on Wiki-CS. This further validates the significance of TCM. \n\nWe will add this discussion in future version.\n\n> The SGRL employs two encoders with non-shared parameters, while the authors provide little explanation on this point. Based on the experience of the reviewer, allowing the two encoders to share parameters might perform better, as it may prevent divergences in the learned patterns between the models.\n\nAdmittedly, in many GCL frameworks, such as GRACE and DGI, shared-parameter encoders are employed to maintain consistency in node representations across different views. However, this approach does not apply to SGRL. To achieve adaptive representation scattering, we have designed two distinct mechanisms, RSM and TCM. It is important to note that the objectives of these two mechanisms can be somewhat conflicting. Shared-parameter encoders might lead to decreased performance and efficiency due to the following reasons:\n\n**Conflicting Objectives:** The encoders must satisfy two potentially conflicting objectives simultaneously during training, which can lead to unstable outcomes.\n\n**Increased Iterations:** More iterations may be necessary to find a set of parameters that effectively balances both objectives.\n\nTo address these challenges, we utilize two separate encoders for the distinct mechanisms and balance the potentially conflicting objectives through Exponential Moving Average (EMA). This approach is more efficient and results in smoother performance.\n\n> In this work, the theories and methods proposed by the authors are mainly based on node-level tasks. So, the reviewer wonders whether this approach can achieve the same powerful performance in graph-level tasks.\n\nThanks for this thought-provoking question. In this paper, we revisit GCL frameworks that focus on node-level tasks. There are some similarities between graph-level and node-level tasks in terms of achieving representation scattering:\n\n**Scattered Center Definition:** In node-level tasks, the scattered center is defined by computing the mean of all node representations. Similarly, for graph-level tasks, the mean of all sub-graph representations can be used to define the graph-level scattered center. This approach allows us to achieve representation scattering through a center-away loss.\n\n**Application of TCM:**  Although we can't constrain the graph representations as we do with nodes, the similarity between different graphs can be evaluated by designing appropriate graph kernel functions. Therefore, we can still design a graph-level \"TCM\" to constrain the representations.\n\nWe plan to further explore the application of representation scattering in graph-level tasks in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper discovers a common mechanism hidden behind main-stream GCL frameworks such as Info-NCE-based methods, DGI, and BGRL. Specifically, the authors prove that the mainstream baselines implicitly employ the representation scattering law (a combination of the center away and uniformity constraint). 

This is an important contribution to the GCL community, and reviews are consistently positive. I thus recommend accepting this paper.
Decision: Accept (oral)