Title: Graph Contrastive Learning with Stable and Scalable Spectral Encoding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Contrastive Learning

Spectral Embedding
Abstract: Graph contrastive learning (GCL) aims to learn representations by capturing the agreements between different graph views. Traditional GCL methods generate views in the spatial domain, but it has been recently discovered that the spectral domain also plays a vital role in complementing spatial views. However, existing spectral-based graph views either ignore the eigenvectors that encode valuable positional information or suffer from high complexity when trying to address the instability of spectral features. To tackle these challenges, we first design an informative, stable, and scalable spectral encoder, termed EigenMLP, to learn effective representations from the spectral features. Theoretically, EigenMLP is invariant to the rotation and reflection transformations on eigenvectors and robust against perturbations. Then, we propose a spatial-spectral contrastive framework (Sp$^{2}$GCL) to capture the consistency between the spatial information encoded by graph neural networks and the spectral information learned by EigenMLP, thus effectively fusing these two graph views. Experiments on the node- and graph-level datasets show that our method not only learns effective graph representations but also achieves a 2--10x speedup over other spectral-based methods.

Graph Contrastive Learning with Stable and Scalable Spectral Encoding

Abstract

Graph contrastive learning (GCL) aims to learn representations by capturing the agreements between different graph views. Traditional GCL methods generate views in the spatial domain, but it has been recently discovered that the spectral domain also plays a vital role in complementing spatial views. However, existing spectral-based graph views either ignore the eigenvectors that encode valuable positional information, or suffer from high complexity when trying to address the instability of spectral features. To tackle these challenges, we first design an informative, stable, and scalable spectral encoder, termed EigenMLP, to learn effective representations from the spectral features. Theoretically, EigenMLP is invariant to the rotation and reflection transformations on eigenvectors and robust against perturbations. Then, we propose a spatial-spectral contrastive framework (Sp 2 GCL) to capture the consistency between the spatial information encoded by graph neural networks and the spectral information learned by EigenMLP, thus effectively fusing these two graph views. Experiments on the node- and graph-level datasets show that our method not only learns effective graph representations but also achieves a 2-10x speedup over other spectral-based methods.

1 Introduction

Graph neural networks (GNNs) have become the de facto framework to encode graph-structured data [14, 35, 39]. However, training high-quality GNNs usually requires a large number of domainspecific labels, which is not feasible in many real-world applications. Therefore, as a paradigm of self-supervised learning, graph contrastive learning (GCL) is proposed to learn node or graph representations without using labels [20, 19, 12, 38].

Typically, GCL methods first generate different views of a graph, and then contrast the positive views against the negative ones. By minimizing a contrastive loss, GCL methods can learn invariant information from different views for various downstream tasks. Therefore, how to generate ideal graph views is crucial to GCL. Most graph views are obtained by augmenting graphs in the spatial domain, such as dropping nodes and edges, heuristically [49, 44, 8] or adversarially [27, 40, 41]. Nevertheless, recent studies [18, 17] argue that spatial perturbations ignore the structural properties, and propose to perturb graph spectrum in the spectral domain.

Generally, the spatial and spectral views represent different information of the graph. The spatial domain captures the feature information and learns local graph representations by propagating the node features along local topology, i.e ., k -hop subgraphs. In contrast, the spectral domain covers the global structural information. The eigenvalues and eigenvectors encode the global shapes [13] and

node absolute positions [25]. Leveraging the agreements between the spatial and spectral views can significantly improve the expressive power and generalization ability of GNNs [37, 47]. However, several reasons hinder the study of spectral view: First, the spectral features are unstable. Previous work [24] shows that randomly flipping the signs or rotating the coordinates of eigenvectors also satisfies the eigenvalue decomposition, a.k.a ., the sign and basis ambiguity issues, implying that spectral features are not unique and hard to transfer [15, 37]. Besides, perturbing spectral features is time-consuming. Existing spectral-based methods [18, 17] need to decompose and reconstruct the adjacency matrix, in which the complexity is cubic and quadratic in the number of nodes, respectively.

Therefore, to realize the spatial-spectral contrast, it is natural to ask: How to encode the spectral view of a graph effectively and efficiently? To answer this question, we first propose a spectral encoder, named EigenMLP, which not only inherits the scalability advantage of multilayer perceptrons (MLP), but also adheres to two key design principles. First, EigenMLP resolves the sign ambiguity issue by taking both positive and negative eigenvectors as input. Second, to address the basis ambiguity issue, the weights of EigenMLP are generated by a learnable mapping of the eigenvalues, making them equivariant to the coordinates of eigenvectors. To integrate the representations learned from the spatial and spectral views, we propose a Spatial-Spectral GCL framework (Sp 2 GCL) to maximize the agreements between views and learn effective representations for downstream tasks.

The contributions of our paper are as follows. (1) We propose EigenMLP, a novel spectral encoder to effectively and efficiently learn sign- and basis-invariant representations from spectral features. (2) We theoretically prove that EigenMLP is permutation-equivariant, rotation- and reflection-invariant, and can learn more stable representations against structural perturbations. (3) We propose Sp 2 GCL, a spatial-spectral GCL framework that utilizes the cross-domain contrasts to effectively fuse the feature and structural information learned by spatial GNNs and EigenMLP. (4) Extensive experiments on both node-level and graph-level tasks demonstrate the effectiveness of the proposed framework Sp 2 GCL, and verify the scalability and stability aspects of EigenMLP.

2 Related Work

Graph Contrastive Learning. Most GCL methods aim to learn invariant information by maximizing the mutual information between different graph views [1, 36]. There are many ways to generate graph views and we broadly categorize them into the spatial and spectral approaches. In the spatial domain, graph views are usually generated by augmenting the original graphs. For example, GRACE [48], GCA [49] and CCA-SSG [44] propose to augment graphs by randomly dropping edges and nodes, AD-GCL [27] leverages adversarial training to filter unimportant edges, and JOAO [41] combines different augmentation strategies automatically. While in the spectral domain, the views are generated by perturbing graph spectrum. For example, MVGRL [8] heuristically uses a graph diffusion as augmentation, which acts as a low-pass filter, SpCo [18] proposes to preserve low-frequency components and perturbs high-frequency ones, and SPAN [17] generates augmentations by maximizing the spectral change. Besides, SFA [46] analyzes the spectrum of node representations, which is out of the scope of this work. In general, spectral views of graphs may have better performance and interpretability than spatial views but also suffer from high complexity.

Spectral Encoder. Perturbing the graph spectrum is time-consuming. Another approach is to encode the eigenvalues and eigenvectors instead of the eigenspaces. However, eigenvectors suffer from sign and basis ambiguity issues, and using these features directly will affect the stability of the model. Therefore, some spectral encoders are proposed to learn invariant representations from the non-unique spectral features. For example, SAN [15] uses a Transformer-based [34] encoder, which is invariant to the order of the bases. BasisNet [16] uses IGN [21] to learn permutation-invariant representations from the eigenspaces. PEG [37] leverages the distance between eigenvectors to reweigh the graph structure and avoids sign and basis ambiguity. However, the complexity of SAN and BasisNet is quadratic, which is difficult to scale to large graphs.

3 Preliminaries

̸

Assume that G = ( V , E ) is a graph, where V is the node set with |V| = N and E is the edge set with |E| = E . Let A ∈ { 0 , 1 } N × N be the adjacency matrix, and X ∈ R N × d be the node feature matrix on G . The normalized graph Laplacian L of G is defined as L = I n -D -1 2 AD -1 2 , where I n is the N × N identity matrix and D is the degree matrix with D ii = ∑ j A ij for i ∈ V and D ij = 0 for i = j . The eigenvalue decomposition (EVD) of graph Laplacian is defined as L = UΛU ⊤ , where Λ

is a diagonal matrix whose diagonal entries 0 ≤ λ 1 ≤ · · · ≤ λ N ≤ 2 are the eigenvalues of L , and U = [ u 1 , · · · , u N ] are the corresponding eigenvectors.

It is worth noting that in some cases, randomly flipping the signs and rotating the coordinates of eigenvectors may also satisfy EVD [24], which we refer to as sign and basis ambiguity.

Sign ambiguity. Given a pair of eigenvalue and eigenvector ( λ i , u i ) , it satisfies Lu i = λ i u i , and λ i = u ⊤ i Lu i = ∑ ( v,v ′ ) ∈E ( u iv -u iv ′ ) 2 . Therefore, if u i is an eigenvector of L , then -u i also satisfies EVD, i.e ., u ⊤ i Lu i = ( -u i ) ⊤ L ( -u i ) .

Basis ambiguity. If there is high multiplicity in the eigenvalues, i.e ., λ p +1 = · · · = λ p + q for some q > 1 , then the corresponding eigenvectors [ u p +1 , · · · , u p + q ] lie in an orthogonal group O( q ) = { Q ∈ R q × q | Q ⊤ Q = QQ ⊤ = I q } . Therefore, for any Q ∈ O( q ), replacing [ u p +1 , · · · , u p + q ] with [ u p +1 , · · · , u p + q ] Q also satisfies EVD, i.e ., Lu i = λ j u i , p +1 ≤ i, j ≤ p + q .

The above two facts state that the eigenvectors of graph Laplacian are not unique. Therefore, the model should consider how to learn sign- and basis-invariant representations from spectral features for better stability and generalization [37, 16, 15].

4 Proposed Framework: Sp 2 GCL

In this section, we present the proposed spatial-spectral GCL framework called Sp 2 GCL. We first give a high-level overview on how to represent and contrast the spatial and spectral views of a graph. We then introduce the proposed invariant and equivariant spectral view encoder in detail. Finally, we briefly describe the preprocessing, training, and inference processes.

4.1 Overview of Sp 2 GCL

Views refer to different perspectives of the same data [30]. Unlike previous GCL methods that generate graph views in a single domain, we propose to model the spatial and spectral views separately and further utilize cross-domain contrasts to capture invariant information. Here we first describe how to represent the spatial and spectral views of graphs.

Spatial View represents the explicit connectivity of nodes, which can be denoted as V a = ( A , X ) . Through propagating node features along graph substructures, the spatial view can naturally fuse the topology and content information and learn local smooth representations of a graph.

Spectral View indicates the implicit relationships between nodes, which is expressed as V e = ( Λ , U ) . The eigenvalues and eigenvectors encode the geometric information and node positions of the graph topology, which can be seen as global structural information of a graph.

Since graphs are non-Euclidean data, it is difficult to directly contrast the spatial and spectral views. Therefore, we need to design suitable encoders to learn different view representations for GCL:

where f and g are the encoders of the spatial and spectral views, respectively, and H a , H e ∈ R N × d are the spatial and spectral representation matrices, respectively. These representations can then be used in contrastive learning to learn invariant information across both domains.

The fundamental idea of contrastive learning is to define the positive and negative pairs, from which the model can capture the self-supervised signals. In our framework, we define the spatial and spectral representations of the same node or graph as positive pairs, and those of different nodes or graphs as negative pairs. For graph-level contrasts, we additionally use a readout function to learn the graph-level representations. Then, two projection heads φ a , φ e : R d → R d are used to transform the representations into the contrastive space:

Subsequently, we employ InfoNCE [33], a classical contrastive objective function, to maximize the agreements between spatial and spectral representations:

̸

̸

EigenMLP

𝜌(𝜆

!

)

𝑢

!

"

-𝑢

!

"

⋮

𝑢

#

"

-𝑢

#

"

𝜌(𝜆

#

)

Σ|𝜎

Σ|𝜎

⋮

𝑤

!

𝑢

!

"

𝑢

$

"

⋮

𝑢

#

"

𝑤

#

Σ|𝜎

Σ|𝜎

⋮

⋮

MLP

Note : Here k is the number of eigenvectors, T is the period of polynomial, and d is the hidden dimension.

where ⟨· , ·⟩ represents the cosine similarity and i is the index of nodes or graphs.

Finally, to materialize our contrastive framework Sp 2 GCL, we must choose or design the encoders f and g for the two views. For the spatial encoder f , we directly employ a standard message-passing neural network (MPNN), e.g ., GraphSAGE [7] or GIN [39], which is widely adopted in previous GCL methods. For the spectral encoder g , which is the emphasis of this work, we propose EigenMLP in the next part (Section 4.2) based on several key properties. Note that, on the one hand, although MPNNs can learn useful representations, their expressive power is bounded by the 1-Weisfeiler-Lehman (WL) test [39]. On the other hand, the spectral view encodes the global information, which can help overcome the limitation of the 1-WL test [5]. Hence, from the perspective of expressive power, the spectral view is also complementary to the spatial view.

4.2 Proposed Spectral Encoder: EigenMLP

The spectral encoder is designed to learn stable representations from spectral features. A desirable spectral encoder should have three properties: (1) It can encode both the information of eigenvalues and eigenvectors, which represent different structural information [15]; (2) It is free of the sign and basis ambiguity issues in spectral features, toward learning stable representations [37]; (3) It is scalable to large graphs and has linear or sublinear time complexity. These three properties pose great challenges to the design of spectral encoders. Here we propose EigenMLP, which adopts an MLP-based architecture for scalability and addresses the sign and basis ambiguity problems for stability. Table 1 summarizes the differences between the proposed EigenMLP and existing spectral encoders. Specifically, EigenMLP satisfies all of the three properties above, as we elaborate below.

Scalability. In general, MLPs have demonstrated good scalability and have been widely used in learning graph representations [45, 11]. Since EigenMLP employs an MLP-based architecture, it inherits the low complexity and is linear to the number of nodes N .

Stability. MLPs are sensitive to input data, making the outputs vary with respect to the flipping of signs and rotation of coordinates of eigenvectors. This motivates us to increase the stability of MLP while maintaining high efficiency. We adopt two design principles that enable MLPs to learn signand basis-invariant representations. Figure 1 illustrates the difference between EigenMLP and MLP.

To address the sign ambiguity issue, we follow the design of SignNet [16], taking both positive and negative eigenvectors as inputs:

where ψ and ϕ are neural networks, [ · ] is the concatenation operator, and ˜ U is the sign-invariant eigenvectors. In practice, the sign-invariant neural networks may slow down model convergence. In this case, we can resort to some heuristics to determine the sign of the eigenvectors [47].

To solve the basis ambiguity issue, we first observe that each eigenvector has a corresponding eigenvalue, and when the coordinates of the eigenvectors are rotated, the positions of the eigenvalues are also displaced. Therefore, if we replace the weights of MLP with eigenvalues λ = [ λ 1 , λ 2 , · · · , λ N ] , the model will be invariant to the rotation of eigenvectors, i.e ., UQ ( λ Q ) ⊤ = UQQ ⊤ λ ⊤ = U λ ⊤ . However, directly replacing the learnable weights with fixed eigenvalues is trivial and will greatly

limit the expressive power of the model. Therefore, we first extend the scalar eigenvalues to their high-dimensional Fourier features [28]. The weights of EigenMLP are then decoded from the Fourier features using a learnable matrix:

where T is the period and W ρ ∈ R 2 T × d is a learnable matrix. Here ρ ( λ ) can be seen as a graph filter and the filtered eigenvalues are equivariant to the coordinates of eigenvectors [2], which can be used to learn powerful and basis-invariant spectral representations:

where ρ ( λ ) = [ ρ ( λ 1 ) , ρ ( λ 2 ) , · · · , ρ ( λ N )] ⊤ ∈ R N × d represents all the filtered eigenvalues. The detailed matrix form of Equation (6) is provided in Appendix C. Note that the learnable matrix W ρ is shared between different eigenvalues. Therefore, the size of W ρ is independent of the number of eigenvalues but depends on the period T , which reduces the number of parameters in EigenMLP. More discussions can be found in Section 5.2.

Information. Notably, the invariant layer, i.e ., Equation (6), in EigenMLP incorporates both eigenvalues and eigenvectors, which can capture both geometric and positional information.

4.3 Preprocessing, Training, and Inference

It is worth noting that running EVD for full eigenvectors has the complexity O ( N 3 ) , which is unacceptable for large graphs. Therefore, in the preprocessing, we use the eigenvectors with smallestk eigenvalues as a substitute and reduce the complexity to O ( N 2 k ) . Besides, we can pre-calculate the rotation-invariant spectral features V · ρ ( λ k ) . Therefore, in the training and inference, EigenMLP has the implementation of MLP. Note that the decomposition is only computed once in the training. Therefore, the overhead of the preprocessing should be amortized by each training epoch.

In training, traditional GCL methods need to compute the message-passing twice for different graph views, whose complexity is O ( E ) . While in our framework, only the spatial view needs to calculate the message-passing, and the complexity of spectral view is O ( N ) . Therefore, our framework is faster than others in the training processing. In inference, because we need to calculate the spatial and spectral representations separately, the inference speed is slightly lower than traditional methods. The overheads of training and inference are provided in Section 6.6.

5 Deeper Insights

In this section, we provide deeper insights into EigenMLP to understand its effectiveness. Specifically, we prove that EigenMLP is invariant, equivariant, and stable, and can generalize existing spectral augmentations. Besides, we also comment on the connections to previous work.

5.1 Theoretical Results

Theorem 1. EigenMLP is equivariant to permutation, and invariant to rotation, and reflection.

Proof. (Permutation) Assume that there are two matrices L (1) , L (2) ∈ R N × N , and L (1) = PL (2) P ⊤ . We have L (1) = PL (2) P ⊤ = ( PU (2) ) Λ ( PU (2) ) ⊤ such that U (1) ρ ( λ ) = PU (2) ρ ( λ ) .

(Rotation) For any rotation matrix Q ∈ O ( N ) , the eigenvectors are rotated as UQ , and the corresponding Fourier matrix are permuted as Q ⊤ ρ ( λ ) . Therefore, we have UQ · Q ⊤ ρ ( λ ) = U ρ ( λ ) .

(Reflection) Following Proposition 1 in [16], a continuous function is sign-invariant iff it satisfies f ( v ) = h ( v ) + h ( -v ) . Hence, EigenMLP is invariant to reflection on the signs of eigenvectors.

Lemma 1. (Lemma 3.4 in [37]) For any positive semidefinite L without multiple eigenvalues, set V as the eigenvectors corresponding to the smallest k eigenvalues and sorted as 0 < λ 1 < · · · < λ k . For any sufficiently small ϵ > 0 , there exists a perturbation ∆ L , ∥ ∆ L ∥ F < ϵ such that

Theorem 2. For any sufficiently small ϵ > 0 , there exists a perturbation ∆ L , ∥ ∆ L ∥ F < ϵ such that

Lemma 1 states that a small structural perturbation will produce changes unbounded from above in non-equivariant spectral features V if there is a small spectral gap. Theorem 2 shows that there is a finite upper bound on the changes of equivariant spectral features V ρ ( λ k ) . Therefore, EigenMLP can learn more stable representations against structural perturbations. We present the proof of Theorem 2 in Appendix A. Experiments in Section 6.5 further give an empirical justification of Theorem 2.

Proposition 1. EigenMLP generalizes existing spectral augmentations.

Proof. Existing spectral augmentations, such as SpCo [18] and SPAN [17], generate graph views by perturbing the graph spectrum. We assume that there is a continuous univariate function δ ( · ) between the original eigenvalues λ and the perturbed eigenvalues λ ′ , such that λ ′ i = δ ( λ i ) , and the perturbed eigenvalues are still non-negative. Then these spectral augmentations can be represented as:

Note that Equation (5) is a Fourier series and can approximate any continuous functions. Therefore, the function δ ( · ) is a special case of ρ ( · ) , and EigenMLP can approximate the simplex geometry [32] of the augmentations, i.e ., U δ ( Λ ) 1 2 , thus generalizing existing spectral augmentations.

5.2 Connections to Existing Work

Hypernetworks [6] are a class of neural networks used to generate parameters for another network. EigenMLP can be seen as a special case of Hypernetworks, which takes eigenvalues as input and generates equivariant parameters for spectral features. Because Hypernetworks can generate a set of non-shared weights from shared parameters, it can greatly reduce the number of trainable parameters. EigenMLP inherits this advantage, and its number of parameters is only related to the period of the Fourier series, rather than the number of eigenvectors. Therefore, EigenMLP is more efficient than other spectral encoders, including vanilla MLP.

Spectral GNNs aim to combine the eigenspaces with filtered eigenvalues, i.e ., ∑ N i =1 δ ( λ i ) u i u ⊤ i , and EigenMLP is used to combine the eigenvectors with filtered eigenvalues, i.e ., ∑ N i =1 ρ ( λ i ) u i . Therefore, both methods choose to use eigenvalue mappings as weights to guarantee the equivariance. However, the calculation of eigenspaces has the complexity of O ( N 2 ) , which is not suitable for large graphs, whereas EigenMLP is more scalable.

6 Experiments

In this section, we conduct three types of experiments, including unsupervised node classification, unsupervised graph prediction, and transfer learning, to verify the effectiveness of Sp 2 GCL. Besides, we also test the stability and time overhead of the proposed method.

6.1 Unsupervised Node Classification

Datasets. In the node classification task, we consider using graphs with different scales to evaluate both the effectiveness and scalability of GCL methods. Specifically, for the small graphs (< 50,000), we use Pubmed [14], Wiki-CS [22], and Facebook [23] datasets. For the large graphs (> 50,000), we use Flickr [43], arXiv [9], and PPI [7] datasets. Additional statistics are provided in Appendix B.

Baselines and Setting. We compare our model against a wide range of baselines, including semisupervised GNNs, e.g ., GCN [14] and GAT [35], graph self-supervised learning methods, e.g ., DGI [36] and BGRL [29], GCL with spatial augmentations, e.g ., MVGRL [8], GRACE [48], and CCASSG [44], and GCL with spectral augmentations, e.g ., SpCo [18] and SPAN [17]. For the Facebook dataset, we randomly split the nodes into train/validation/test data with a ratio of 1:1:8. For other datasets, we use the public splits for a fair comparison. We use a two-layer GCN as the encoder for

all datasets and set the hidden dimension d = 512 for all methods. For our model, the spatial encoder is the same as baselines, and we additionally use EigenMLP to learn the spectral representation. In the evaluation, we use a linear classifier to evaluate the performance of all methods, as suggested by [27]. We run all the models 10 times and report the mean accuracy and standard deviation. More details, e.g ., optimizers, and hyperparameters, are provided in Appendix B.

Results. From Table 2, we can find that Sp 2 GCL consistently outperforms state-of-the-art baselines on 5 out of 6 datasets, which validates the effectiveness of the proposed spatial-spectral contrastive framework. Meanwhile, spectral-based methods are proven to be more effective than spatial-based methods, suggesting that integrating spectral information into GCL can help models learn better representations. However, it is worth noting that neither SpCo nor SPAN works for large graphs, implying that perturbing graph spectrum cannot be scalable to large-scale datasets. Therefore, the application scenarios of these two graph augmentations are limited. On the contrary, our method can be used for large graphs and can be easily trained in a mini-batch manner, which is more scalable than other spectral-based methods. Additionally, We find that Sp 2 GCL does not perform well in the Wiki-CS dataset. The reason is that the node features dominate the classification results while the graph structure contributes less. Therefore, the spectral view cannot complement the spatial view.

6.2 Unsupervised Graph Prediction

Setup. We benchmark our model on the OGB graph property prediction task [9], which contains three regression datasets and five classification datasets. We choose a series of competitive GCL methods as baselines, including InfoGraph [26], GraphCL [42], MVGRL [8], JOAO [41], AD-GCL [27], and SPAN [17]. It is worth noting that SpCo [18] is not designed for graph-level contrastive learning, so we do not compare with it. We use a five-layer GIN [39] with a graph pooling layer as the encoder for all methods. Similarly, we additionally use EigenMLP to encode the spectral view for Sp 2 GCL. We use a linear downstream classifier, e.g ., logistic regression model, to evaluate the performance of different GCL methods, as suggested by [27].

Results. Table 3 summarizes the graph prediction performance. Sp 2 GCL has 4 best and 3 runner-up performances and makes significant improvements on molfreesolv and molbace. Compared with GraphCL and JOAO, which employ multiple spatial augmentations, e.g ., edge and node dropping, and feature masking, Sp 2 GCL outperforms them by only using the spectral domain information. This demonstrates the effectiveness of spectral information. Compared with AD-GCL and SPAN, which use adversarial learning to find the near-optimal invariant information [31], Sp 2 GCL only uses the traditional InfoMax principle [1] and achieves competitive performance. This shows that the position information of spectral features is crucial to graph representation learning. Besides, in Table 9, we show that the efficiency and scalability of Sp 2 GCL significantly outperform adversarial learning.

6.3 Transfer Learning Scenario

Setup. The transfer learning experiments are used to evaluate the generalization ability of GCL methods. We follow the experimental setup described in [10], which involves pre-training the GCL methods on a large-scale dataset and subsequently fine-tuning the model on downstream datasets to evaluate their out-of-distribution performance We use the same graph encoder and baseline as in the unsupervised graph prediction task. Further details are provided in Appendix B.

Results. According to Table 4, Sp 2 GCL outperforms baselines in 4 out of 8 datasets and achieves an average rank of 1.7 across these datasets. Notably, the performance of different methods on different downstream datasets varies greatly, suggesting that the downstream tasks require distinct information for prediction. Nevertheless, spectral information proves to be crucial in the majority of tasks. Furthermore, Sp 2 GCL achieves comparable performance to SPAN, which employs a spectrumbased augmentation for learning graph representations. This observation indicates that Sp 2 GCL can learn similar spectrums as SPAN. However, our model falls short of InfoGraph, GraphCL, and JOAO on certain datasets due to the limited availability of feature information in the spectral view.

6.4 Ablation Studies

We conducted two ablation studies to validate the effectiveness of EigenMLP and the proposed framework, Sp 2 GCL. Initially, we replace EigenMLP with different spectral encoders and evaluate their performance on PubMed, Facebook, and molbace datasets. The results, presented in Tables 5, demonstrate that EigenMLP consistently outperforms other spectral encoders, thus confirming its effectiveness in learning spectral representations. Moreover, we aim to justify the superior ability of spatial-spectral contrast in fusing spatial and spectral representations. To accomplish this, we concatenate node features with eigenvectors and eigenvalues and subsequently feed them into GRACE. As shown in Table 6, we observe that incorporating spectral features improves the performance in

GRACE. Nevertheless, the performance is still inferior to that of Sp 2 GCL, thus providing further evidence of the effectiveness of spatial-spectral contrast.

6.5 Stability of EigenMLP and MLP

We conduct stability experiments to evaluate whether EigenMLP and MLP can learn stable representations against perturbations. We consider two types of perturbations: 1) Synthetic perturbation, which applies random reflection Π and rotations Q on the eigenvectors. 2) Practical perturbation, which decomposes the Laplacian matrix with different tolerances ( 10 -3 , 10 -4 , 10 -5 ). The corresponding eigenvectors are expressed as U 3 , U 4 , and U 5 . Note that synthetic perturbation only changes the signs and coordinates of eigenvectors while practical perturbation is more challenging because it perturbs the values. For clearer results, we only evaluate the performance of spectral representations.

For each type of perturbation, we construct three instances, i.e ., ( U , UΠ , UQ ) or ( U 3 , U 4 , U 5 ). The models are trained on one instance and tested on the other two instances on PubMed. We report the results of EigenMLP and show the performance change after switching to MLP in the brackets. From Table 7, we can find that the synthetic perturbation can hardly change the representations learned by EigenMLP but has a great influence on MLP. Additionally, Table 8 shows that practical perturbations can affect both EigenMLP and MLP. Nevertheless, EigenMLP still consistently outperforms MLP across different tolerances, which verifies the stability of EigenMLP.

6.6 Time Overhead

To assess the efficiency, we compare the time overhead of Sp 2 GCL with other GCL methods as well as compare EigenMLP with different spectral encoders. The time costs associated with preprocessing, training, and inference are shown in Table 9. Previous spectrum-based methods have significant time costs in the preprocessing stage, e.g ., graph diffusion (GD) or Sinkhorn's Iteration (SI), whereas Sp 2 GCL exhibits minimal overhead. In the training stage, Sp 2 GCL outperforms other methods due to the efficient spectral encoding, but it also introduces an additional burden during inference. Table 10 illustrates the overhead imposed by various spectral encoders within the Sp 2 GCL framework. Both MLP and EigenMLP exhibit remarkable efficiency. Conversely, SAN and BasisNet entail excessive time costs due to their quadratic time complexity.

7 Conclusion

In this study, we introduce Sp 2 GCL, a novel spatial-spectral GCL framework that learns the consistency between the spatial and spectral views of graphs. To effectively and efficiently learn the spectral view information, we propose EigenMLP, a scalable spectral encoder to learn stable spectral representations from the non-unique spectral features. Extensive experiments on various graph-related tasks demonstrate the effectiveness, efficiency, and stability of the proposed method.

Limitation and Broader Impact Currently, we focus on encoding spectral features while ignoring node features. A promising future direction is to unify these two kinds of information to learn effective graph representations. Our work reveals the superiority of integrating spectral information into GCL and may inspire the community to pay more attention to the spectral view of graphs.

Acknowledgments and Disclosure of Funding

This work is supported in part by the National Natural Science Foundation of China (No. U20B2045, 62192784, U22B2038, 62002029, 62172052). This work is also partially supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Proposal ID: T2EP20122-0041). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.

References

Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS , pages 15509-15519, 2019.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS , pages 3837-3845, 2016.

Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. CoRR , abs/2012.09699, 2020.

Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. CoRR , abs/2003.00982, 2020.

Or Feldman, Amit Boyarski, Shai Feldman, Dani Kogan, Avi Mendelson, and Chaim Baskin. Weisfeiler and leman go infinite: Spectral and combinatorial pre-colorings. CoRR , abs/2201.13410, 2022.

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In ICLR (Poster) . OpenReview.net, 2017.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS , pages 1024-1034, 2017.

Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In ICML , volume 119 of Proceedings of Machine Learning Research , pages 41164126. PMLR, 2020.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS , 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR . OpenReview.net, 2020.

Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: Node classification without message passing in graph. CoRR , abs/2106.04051, 2021.

Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. CoRR , abs/2006.10141, 2020.

Mark Kac. Can one hear the shape of a drum? The american mathematical monthly , 73(4P2):123, 1966.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In NeurIPS , pages 21618-21629, 2021.

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess E. Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. CoRR , abs/2202.13013, 2022.

Lu Lin, Jinghui Chen, and Hongning Wang. Spectral augmentation for self-supervised learning on graphs. In ICLR , 2023.

Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. In NeurIPS , 2022.

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Selfsupervised learning: Generative or contrastive. IEEE Trans. Knowl. Data Eng. , 35(1):857-876, 2023.

Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey. CoRR , abs/2103.00111, 2021.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In ICLR (Poster) . OpenReview.net, 2019.

Péter Mernyei and Catalina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. CoRR , abs/2007.02901, 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. J. Complex Networks , 9(2), 2021.

Daniel A. Spielman. Spectral graph theory and its applications. In FOCS , pages 29-38. IEEE Computer Society, 2007.

Balasubramaniam Srinivasan and Bruno Ribeiro. On the equivalence between positional node embeddings and structural graph representations. In ICLR . OpenReview.net, 2020.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR . OpenReview.net, 2020.

Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In NeurIPS , pages 15920-15933, 2021.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS , 2020.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, Rémi Munos, Petar Velickovic, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR . OpenReview.net, 2022.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV , pages 776-794. Springer, 2020.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurIPS , 2020.

Leo Torres, Kevin S. Chan, and Tina Eliassi-Rad. GLEE: geometric laplacian eigenmap embedding. J. Complex Networks , 8(2), 2020.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR (Poster) . OpenReview.net, 2019.

Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. In ICLR . OpenReview.net, 2022.

Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z. Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering , 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR . OpenReview.net, 2019.

Longqi Yang, Liangliang Zhang, and Wenjing Yang. Graph adversarial self-supervised learning. In NeurIPS , pages 14887-14899, 2021.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML , volume 139 of Proceedings of Machine Learning Research , pages 12121-12132. PMLR, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , 2020.

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In ICLR . OpenReview.net, 2020.

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. From canonical correlation analysis to self-supervised graph neural networks. In NeurIPS , pages 76-89, 2021.

Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In ICLR . OpenReview.net, 2022.

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In AAAI , 2023.

Ziwei Zhang, Peng Cui, Jian Pei, Xin Wang, and Wenwu Zhu. Eigen-gnn: A graph structure preserving plug-in for gnns. IEEE Trans. Knowl. Data Eng. , 35(3):2544-2555, 2023.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR , abs/2006.04131, 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW , pages 2069-2080. ACM / IW3C2, 2021.

Appendices

A Proof of Theorem 2

We use i to indicate the index of λ -1 = max i ∈ N | λ i +1 -λ i | -1 and perturb graph Laplacian L by perturbing the eigenvectors. Specifically, we set

Note that || u ′ i || = || u ′ i +1 || = 1 and u ′ i ⊤ u ′ i +1 = 0 . Therefore, replacing u i and u i +1 with u ′ i and u ′ i +1 still satisfies EVD. We denote V ′ = [ u 1 , · · · , u ′ i , u ′ i +1 , · · · , u k ] . Then the perturbation can be represented as ∆ L = V ′ ΛV ′ ⊤ -VΛV ⊤ = λ i ( u ′ i u ′ i ⊤ -u i u ⊤ i ) + λ i +1 ( u ′ i +1 u ′ i +1 ⊤ -u i +1 u ⊤ i +1 ) .

For sufficient small ϵ > 0 , we have:

For Fourier features with period T 2 , we have:

Next, we characterize || ∆ L || F :

Combining Equations (2) and (4), we have the lower bound of the changes of non-equivariant spectral features under small perturbations:

which concludes the Lemma 1, i.e ., Lemma 3.4 in [37].

Combining Equations (3) and (4), we have the upper bound of the changes of equivariant spectral features under small perturbations:

which concludes the Theorem 2.

B Detailed Experimental Setup

In this section, we report the details of our experiments. Specifically, we first introduce some general settings in all experiments. Then we introduce the detailed setup of each experiment one by one.

B.1 General Settings

Optimizer. For all experiments, we use the Adam optimizer.

Environment. The environment in which we run experiments is:

Linux version: 5.19.0-38-generic

Operating system: Ubuntu 22.04.2

CPU information: AMD EPYC 7313P 16-Core Processor

GPU information: GeForce RTX 3090 (24 GB)

Resources. The addresses and licenses of all datasets are as follows:

PubMed: https://github.com/tkipf/pygcn (MIT License)

Wiki-CS: https://github.com/pmernyei/wiki-cs-dataset (MIT License)

Facebook: https://github.com/benedekrozemberczki/MUSAE (GPL-3.0 license)

arXiv: https://github.com/snap-stanford/ogb (MIT license)

Flickr: https://github.com/GraphSAINT/GraphSAINT (MIT license)

PPI: https://github.com/mims-harvard/ohmnet (MIT license)

OGB-graph: https://github.com/snap-stanford/ogb (MIT license)

ZINC-2M: https://github.com/snap-stanford/pretrain-gnns (MIT license)

Reproducibility. Our code is attached in the supplementary material.

B.2 Unsupervised Node Classification

Evaluation protocol. In the unsupervised node classification task, all methods are first trained with the corresponding self-supervised learning objectives. Then the learned representations are evaluated with a Logistic classifier with l 2 normalization. We evaluate the method every 10 epochs and the maximum epoch is set to 1000. For the mini-batch training, we set the batch size to 1024. The detailed statistics are shown in Table 1 and the hyperparameters are shown in Table 2.

B.3 Unsupervised Graph Prediction

Evaluation protocol. In the unsupervised graph prediction task, we use the stand encoder, provided by OGB 4 , as the spatial encoder of Sp 2 GCL, which is a 5-layer GIN with hidden dimension d = 300 . We use add pooling to learn graph-level representations and set the batch size to 32. For the spectral

encoder, due to the relatively small sizes of the molecular graphs, we use all eigenvectors as the spectral features. We set the learning rate to 0.001 and the period to 10 for all datasets, and the number of training epochs is chosen among {20, 50, 80, 100, 150} using the validation set, as suggested by AD-GCL [27]. For the downstream evaluator, we use a Riger regressor for the regression tasks and a Logistic classifier for the binary classification tasks. The strength of l 2 normalization is grid searched among {0.001, 0.01, 0.1, 1, 10, 100, 1000} on the validation set for each dataset. The detailed statistics of the datasets are shown in Table 3.

B.4 Transfer Learning

Evaluation protocol. For the transfer learning task, we use the same GIN encoder as [10]. In the pre-training stage, the learning rate is set to 0.001 and the number of training epochs is chosen from {20, 50, 80, 100} based on the validation set. Similarly, we use all eigenvalues and eigenvectors as the spectral features, and the period is set to 10. In the fine-tuning stage, we remove the self-supervised learning objective, and an additional linear projection layer is used on the output of the encoder for classification. The hyperparameters are the same as in the pre-training stage. The detailed statistics of the datasets are shown in Table 4.

B.5 Stability Experiment

We use the PyTorch-style pseudo code to explain how we generate the synthetic perturbations (Figure 2) and practical perturbations (Figure 3).

C Matrix Form of EigenMLP

We give a detailed matrix form of Equation 6, from which we can see that the Fourier features of eigenvalues give different weights to the eigenvectors, thus making the model invariant to the rotation of coordinates and preserving good fitting ability.

D Pseudo Algorithm

In order to better demonstrate our algorithm, here we provide the pseudo algorithms of EigenMLP (Figure 4) and Sp 2 GCL (Figure 5).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for Graph Contrastive Learning (GCL) by contrasting the spatial and spectral views ($Sp^2GCL$). The spatial view is obtained using a message passing GNN. For the spectral view the authors propose an equivariant model called EigenMLP. EigenMLP precomputes the k smallest eigenvalues and corresponding eigenvectors for a graph. The eigenvectors are made sign invariant using positive and negative eigenvectors as in SignNet. Permutation/Basis equivariance is obtained by learning MLP weights from fourier features of the eigenvalues. The method uses the same node/graph representations as positive views and from different graph/nodes as negative views. The InfoNCE contrastive function is used as the objective. After the self supervised training, a linear classifier is used on the downstream task of node/graph classification/prediction.\n\nSTRENGTHS:\n1) Paper proposes an important direction of combining spatial and spectral views\n2) The method obtains competitive results compared with baselines\n\nWEAKNESSES:\nRegarding the views for a node/graph, the spectral views are obtained from the eigendecomposition that are made equivariant to sign and basis and the spatial views are obtained from an MPNN. This would give a fixed view for every node/graph and the concern is for a node/graph how to obtain multiple positive views? In the absence of multiple views the learning may be limited to fixed representations and may not scale well for larger models.\n\n**Minor Typos:**\n1. The eigenvalues and eigenvectors **encoder** the global shapes [13] and node absolute positions $\\rightarrow$ The eigenvalues and eigenvectors **encode** the global shapes [13] and node absolute positions\n2. It can **encoder** both the information of eigenvalues and eigenvectors $\\rightarrow$ It can **encode** both the information of eigenvalues and eigenvectors\n3. In practice, the sign-invariant neural networks may slow down model **converge** $\\rightarrow$ In practice, the sign-invariant neural networks may slow down model **convergence**\n4. Therefore, EigenMLP can learn more stable representations **again** structural perturbations $\\rightarrow$ Therefore, EigenMLP can learn more stable representations **against** structural perturbations\n\nQUESTIONS:\n1. Regarding the views for a node/graph, the spectral views are obtained from the eigendecomposition that are made equivariant to sign and basis and the spatial views are obtained from an MPNN. This would give a fixed view for every node/graph and the concern is for a node/graph how to obtain multiple positive views? In the absence of multiple views the learning may be limited to fixed representations and may not scale well for larger models.\n2. As pointed out by the authors, the spatial method learns local features and the spectral method learns global properties. Is it always the right approach to contrast them in the proposed manner? In which cases would it work and how do practitioners make a decision?\n\nLIMITATIONS:\nSome of the limitations have been addressed in the paper. Please refer to the Weakness and Questions section to address further limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

comments: SUMMARY:
The authors present a novel approach called Sp2GCL that combines spatial and spectral views of graphs using EigenMLP, an informative and stable spectral encoder. The proposed method shows promising results in learning effective graph representations and outperforms other spectral-based methods in terms of both performance and efficiency.

STRENGTHS:
This work proposes contrasting two views in the spectral domain and spatial domain and introduces a novel encoder called EnigenMLP to encode spectral domain information, which has not been done by former work.

WEAKNESSES:
- The contribution of the article is considered limited as the traditional Graph Neural Network (like GCN), which can be understood as spectral filtering in the spectral domain, is similar to the EnigenMLP proposed in this work. 

- Simply contrasting these two representations might not lead to significant improvements, as indicated in the results where there is no clear enhancement and suspicion of cherry-picking.

- The article suggests that the proposed method may not offer significant improvement over Graph Convolutional Network methods, and the results indicate a lack of noticeable enhancement. This raises concerns about the effectiveness of the proposed approach.
The article mentions that the contribution of the work is relatively limited, indicating that the proposed method may not introduce significant advancements beyond existing approaches.

QUESTIONS:
- How are the scalar eigenvalues extended to high-dimensional Fourier features in EigenMLP?
- What are the overheads of training and inference in EigenMLP?
- How does EigenMLP handle the sign and basis ambiguity issues in spectral features?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

comments: SUMMARY:
In this paper, the authors propose eigenMLP,an informative, stable, and scalable spectral encoder, which is invariant to the rotation and reflection transformations on eigenvectors and robust against perturbations. Based on eigenMLP,  spatial-spectral contrastive framework is proposed to capture the consistency between the spatial information and spectral information.

STRENGTHS:
1. The eigenMLP is very motivated. The motivation and the theoretical analysis are convincing.

2. Based on eigenMLP, the spectral augmentation is bounded by pertubation delta L.

WEAKNESSES:
There are some major issues:
1. I think the eigenMLP is the main contribution but I don't know why the authors try to highlight that the spatial-spectral contrastive framework is a contribution. In my opinion, it is not the first work using spectral-based GNN and spatial-based GNN as two different views for GCL.

2. Because of the sparsity, eigenvalues usually obtain by randomized SVD. The computational cost is nk^2 rather than n^2k usually. I guess you do not use SVD designed for dense matrices. If you do you can check randomized SVD that may help you.

3. The novelty of eigenMLP: the eigenMLP looks like SignNet + position embedding.

QUESTIONS:
1. What is the T in Eq.8?

2. EigenMLP is able to outperform other spectral-based GNNs with the usual semi-supervised setting? 

3. Why the MLP is so close to EIgenMLP in Table 5?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a graph contrastive learning model with spatial and spectral augmentations, with a novel spectral encoder EigenMLP that could address the stability issue from eigen-decomposition. To exploit the strength of spatial and spectral domains, SP2GCL deploys two augmentation views for the contrastive framework; in the spectral view, it introduces tricks to alleviate the ambiguity of signs and basis, in order to stabilize the training process. The performances of node-level and graph-level experiments, as well as the transfer learning tasks, show the advantages of the proposed framework.\n\nSTRENGTHS:\n1. Contrastive learning for graph data is a prosperous and promising field, especially for the label-sparse settings and for the exploration of properties of non-Euclidean data.\n2. The spectral part in the proposed framework attempts to address the stability and overhead issues, which are the main obstacle in the spectral methods for graphs.\n3. Bridging among the spatial and spectral views is an interesting attempt, which could also make full use of their complementary properties.\n\nWEAKNESSES:\n1. As the authors emphasize the stability of their method, more theoretical and empirical analysis are expected to validate it.\n2. I feel confused about the introduction of Fourier features in Line 169; the reasons and influences of them could be more detailed.\n3. The performances are not always competitive in the experiment section; it may help to explain based on the properties of data sets.\n\nQUESTIONS:\nIn the 3rd paragraph of Introduction, it's stated that spatial methods capture local features and so as spectral ones to global features, does it still true for the augmentations? Is it possible that some spectral augmentations merely perturb some kind of edges (such as low-degree or deviant ones)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}