Graph Contrastive Learning Under Heterophily: Utilizing Graph Filters to Generate Graph Views
Authors: Wenhan Yang
Baharan Mirzasoleiman
Venue: Submitted to ICLR 2023 (2023)
Keywords: GNN
Contrastive learning
Heterophily
Graph Representation Learning

Abstract: Graph Neural Networks have achieved tremendous success in (semi-)supervised tasks for which task-specific node labels are available. However, obtaining labels is expensive in many domains, specially as the graphs grow larger in size. Hence, there has been a growing interest in the application of self-supervised techniques, in particular contrastive learning (CL), to graph data. In general, CL methods work by maximizing the agreement between encoded augmentations of the same example, and minimizing agreement between encoded augmentations of different examples. However, we show that existing graph CL methods perform very poorly on graphs with heterophily, in which connected nodes tend to belong to different classes. First, we show that this is attributed to the ineffectiveness of existing graph augmentation methods. Then, we leverage graph filters to directly generate augmented graph views for graph CL under heterophily. In particular, instead of explicitly augmenting the graph topology and encoding the augmentations, we use a high-pass filter in the encoder to generate node representations only based on high-frequency graph signals. Then, we contrast the high-pass filtered representations with their low-pass counterparts produced by the same encoder, to generate representations. Our experimental results confirm that our proposed method, HLCL, outperforms state-of-the-art CL methods on benchmark graphs with heterophily, by up to 10%.

Under review conference paper at ICLR 2023
GRAPH CONTRASTIVE LEARNING UNDER HETEROPHILY: UTILIZING GRAPH FILTERS TO GENERATE GRAPH VIEWS
Anonymous authors Paper under double-blind review
ABSTRACT
Graph Neural Networks have achieved tremendous success in seuperssue tasks for which task-specific node labels are available. However, obtaining labels is expensive in many domains, specially as the graphs grow larger in size. Hence, there has been a growing interest in the application of self-supprvised techniques, in particular contrastive learning (CL), to graph data, In general, CL methods work by maximizing the agreement between encoded augmentations of the same example, and minimizing agreement between encoded augmentations of different examples. However, we show that existing graph CL methods perform very poorly on graphs with heterophily, in which connected nodes tend to belong to different classes. First, we show that this is attributed to the ineffectiveness of existing graph augmentation methods. Then, we leverage graph filters to directly generate aug- mented graph views for graph CL under heterophily. In particular, instead of explicitly augmenting the graph topology and encoding the augmentations, we use high-pass filter in the encoder to generate node representations only based on high-frequency graph sig- nals. Then, we contrast the high-pass filtered representations with their low-pass counter- parts produced by the same encoder, to generate representations. Our experimental results confirm that our proposed method, HLCL, outperforms state-of-the-art CL methods on benchmark graphs with heterophily, by up to 10%.
INTRODUCTION
Graph neural networks (GNNs) are powerful tools for learning graph-structured data in various domains, including social networks, biological compound structures, and citation networks (Kipf & Welling 2016 Hamilton al. 2017] Veličković al. 2017). In general, GNNs leverage the graph's adjacency matrix update the node representations by aggregating information from their neighbors. This can be seen as low- pass filter that smooths the graph signals and produces similar node representations <Nt & Maehara, 2019) GNNs have achieved great success in supervised and semi-supervised learning, where task-specific labels are available. However, obtaining high-quality labels very expensive in many domains, specially as graphs grow larger in size. This has motivated a recent body of work on self sulf-super learning on graphs that learn the representations in an unsupervised manner (Velickovic et al., 20199] Peng et al., 2020 Qiu et al., 2020 Hassani & Khasahmadi] 2020 Zhu et al. (2200b). Among self self-supervised methods, Contrastive Learning (CL) has shown a great success by achieving comparable performance with its supervised counterparts (Chen ct al. 2020). Contrastive learning obtains representations by maximizing the mutual information between different augmented views of the same example, and minimizing agreement between differently augmented views of different examples. Despite being successful graphs with homophily, where neighboring nodes tend to share the same label, existing graph CL methods cannot learn high-quality representations for graphs with heterophily, where connected nodes often belong to different classes Zhu et al. (0200)).
State-of-the-an graph CL methods work by contrasting the encoded node representations in two explicitly augmented graph views, generated by altering the graph topology or node features (Zhu et al. 2020c 2021b

---

Under review conference paper at ICLR 2023
Shared encoder ft.
High-pass augmentation channel
Projection head g()
Optional graph augmentation
FиР
Ha
X
Hi
Maximize agreement
Wi
W/
W
Hi
H
FLP
Low-pass encoder (GNN)
Figure 1: Our proposed HLCL framework leverages high-pass graph filter in the encoder to generate a non-smooth graph view, and contrast with its smooth counterparts generated by the same encoder.
Velickovic et al. 2019 Thakoor et al.) 2021
Specifically, topology augmentation methods alter the graph structure by removing or adding nodes or edges. Feature augmentation methods alter the node features by masking particular columns, dropping features at random,
randomly shuffling the node features. The choice of the augmentation strategy has been shown to have a great influence on the performance of graph CL methods (Zhang & Ma 2022 Zhu et al., 202Tb). In particular, topology augmentations such edge dropping that produce sparser graphs are the most effective for graphs with homophily (Zhu et al. 2021b).
In this work, first study the effect of existing graph augmentation methods on the performance of graph CL under heterophily. We show that topology and feature augmentation methods that can effectively boost the performance of graph CL under homophily do not provide any considerable advantage under heterophily. In particular, we experimentally confirm that for graphs with homophily, topology augmentation effectively decreases the homophily ratio, and feature augmentation changes the variance of features in neighborhood. Hence, explicit graph augmentation techniques enable the low-pass GNN encoder to better distinguish the smooth node representations- that are similar neighborhood However, such methods cannot effectively diversify and alter the homophily ratio variance of features in graphs with heterophily. Indeed, for such graphs, incorporating the graph structure is crucial for generating high-quality representations that can distinguish dissimilar nodes from their neighborhood.
adding more diversity to graph.
Next, we address the above shortcoming by proposing an effective augmentation strategy for CL on graphs with heterophily, while ensuring high-quality representation learning graphs with homophily. The key idea of our proposed method, HLCL, is to leverage a high-pass graph filter in a typical GNN encoder directly generate non-smooth node representations, and contrast them with their smooth counterparts generated by the same encoder. More specifically, we use the normalized Lapcalian matrix as the high-pass filter in the GNN encoder to aggregate the node representations. The Laplacian matrix magnifies the differences in the node features in a neighborhood and makes the representations distinct. This is crucial contrast, a typical GNN encoder uses
for learning high-quality representation under heterophily. the normalized adjacency matrix as low-pass filter to aggregate the node features with those of their neighbors, and enforce similar representations for the nodes in the same neighborhood. Maximizing the mutual information between the high-pass and low-pass filtered representations enables the encoder to learn rich representations on graphs with heterophily. Importantly, our method is complementary to explicit graph augmentation techniques, and such methods can be applied further boost the performance, particularly on graphs with homophily. This allows our framework to achieve state-of-the-ard under heterophily and comparable performance under homophily
confirm by our experiments.
We show the effectiveness of our HLCL framework through extensive experiments on graphs with het- erophily and homophily for unsupervised representations learning under the linear evaluation protocol. Our
2

---

Under review conference paper at ICLR 2023
results demonstrate that, on five popular public benchmark datasets, our framework can outperform existing graph CL methods by up to 10% for representation learning on graphs with heterophily, while ensuring a comparable performance to state-of-the-an graph CL methods on graphs with homophily.
2 RELATED WORK
-Seeii-supervised learning on graphs. In recent years, GNNs have become one of the most prominent tools for processing graph-structured data. In general, GNNs utilize the adjacency matrix to learn the node representations, by aggregating information within every node's 's neighborhood (Defferrard et al. 2016:] Kipf & Welling, 2016). Existing variants, including GraphSAGE (Hamilton et al. (007)). Graph Attention (GAT) (Velčkkvvić et al. 2017). MixHop (Abu-El-Haija et al.) 2019), SGC (Nt & Maehara 2019), GAT (Velickovic jtt al.. 2019). and GIN (Xu al. 2018), learn a more general class of neighborhood mixing relationships, by aggregating weighted information within multi-hop neighborhood of every node. GNNs can be generally seen as applying a fix, or a parametric and learnable (e.g. GAT) low-pass graph filter to graph signals. Those with trainable parameters can adapt to a wider range of frequency levels different graphs. However, they still have a higher emphasis on sweer-freyency signals and discard the feee-frryeency signals in a graph. While the aggregation operation makes GNNs powerful tools for sami-supervised learning, it often leads to over-smoothing issue, i.e., the learned node representations become indistinguishable in a neighborhood (Ni & Maehara 2019). As result, typical GNNs and their variants have been long criticized their inability to capture graph heterophily and poor generalization performance on heterogeneous dataset (Balcilar et al. 2020). where most connected nodes are from different classes.
(Semi- supi-vsupe learning under heterophily. To address over-smoothing issue of GNNs, recent meth- ods propose to use other types of aggregation that better fit graphs with heterophily. Geom-GCN uses geo- metric aggregation in place of the typical aggregation (Pei et al. 2020). H₂GCN uses several special model designs including separate aggregation and nrigher-neihhootie
aggregation to train the model for handling graphs with heterophily, and CPGNN trains compatibility matrix to model the heterophily level (Zhu et al., 2020a). More recently. Wang et al. (2019) proposed to learn an aggregation filter for every graph from a set of based filters designed based on different ways of normalizing the adjacency matrix. Most recently, GGCN introduced degree corrections and signed message passing GCN to address both oversmoothing problems and the model's poor performances on heterophily graphs (Yan et al.] 202T). Zhu et al. (2021a) analyzed and designed a uniform framework for GNNs propagations and proposed GNN-LF and GNN-HF HF that preserve information of different frequency separately by using different filtering kernels with Icarnable weights. FAGCN (Bo et al., 2021) and FBGNN (Luan al. 2020) also leveraged the Laplacian matrix capture the non-smooth signals discarded in typical GNNs. Two separate encoders are trained to capture the high-pass and low-pass graph signals separately, and the two outputs are combined with learnable parame- ters to balance their importance. Such methods achieve a superior performance under heterophily. However, learning how to combine the encoder outputs is highly sensitive having high-quality labels. This makes such methods highly impractical for unsupervised contrastive learning.
Contrastive learning on graphs. Self-supervised contrastive learning methods learn representations of data points by maximizing the mutual information between different views of the same data point, and minimizing agreement between differently augmented views of different examples (Bachman et al.) 2019 Ye et al. 2019 Wu et al. 2018 Chen et al., 2020; |Sohn al., 2020). For graphs, global graph-leve and local nove- data are augmented and contrasted in different ways. DGI (Velickovic et al., 2019) and GMI (Peng et al. 2020) contrast graph and node representations within one augmented view of the original graph. More recent methods contrast global and local representations in two augmented views. GRAPHCL generates graph augmentations by subgraph sampling, node dropping, and edge perturbation and contrasts the augmented graph representations. GCC samples and contrasts subgraphs of the original graph (Qiu jet al. 2020). MVGRL leverages node diffusion to augment the graph and contrasts the node representations (Hassani & Khasahmadi 2020). More recently, contrasting the local node representations has been shown to
3

---

Under review conference paper at ICLR 2023
achieve state-of-the-art. GRACE contrasts the node representations in two graph views augmented with fea- ture masking and edge removal (Zhu et al. 2020c). GCA extends this by dropping the less important edges and features, based on node centrality and feature importance metries (Zhu al. all] (021c)) A thorough empir- ical study on the combinatorial effect of different augmentations has been conducted by ZZuu et al. (2021b). Due to the complexity of collecting negative samples in graph data, regative-samppls-free contrastive ob- jectives have been also studied. Among existing methods, BGRL that uses the Bootstrapping Latent loss (Thakoor et al. (022), J, and GBT uses Barlow Twins loss (Bielak et al. (021] are the most successful. Exist- ing graph CL methods explicitly augment the input graph and contrast the augmented graph representations obtained with low-pass GNN-based encoders. Hence, they perform poorly on graphs with heterophily. In contrast, we leverage low-pass and high-pass graph filters directly obtain different graph views, without relying on explicit graph augmentation. This allows achieving state-of-the-ar under heterophily.
3 CONTRASTIVE LEARNING VIA GRAPH FILTERS
Here, we introduce proposed contrastive learning pipeline, HLCL,. The key idea of our method is to gen- erate a non-smooth augmented representation and contrast it with the smooth representation generated using the same encoder. typical GNN filters smooth graph frequencies by aggregating the node representations with those of their neighbors. Hence, it results in similar representations for the nodes in a neighborhood. In contrast, the high-pass filter only preserves the high-pass frequencies. In doing so, it magnifies the dissim- ilarities between the nodes and diversifies the representations of nodes in a neighborhood. Contrasting the two filtered representations results learning rich representations, particularly under heterophily.
3.1 PRELIMINARIES
We denote by 9 = EV,E) an undirected graph, where = {1i, u2,...,,y represents the node set, and E " V x V represents the edge set. We denote by A E 1) J XN
the symmetric adjacency matrix of graph G. That is, A.j = if and only if (vi, 02) E E, and A.j = otherwise. We also denote the feature matrix by X, where X, E Rm is the feature vector of the jth node, and x E RR is a column of the matrix and represents a graph signal. D is the degree matrix of the graph, with D -,, Aij. and N = (j A.j = 1) is the neighborhood of node i. L is the Laplacian matrix of the graph, defined as L D-A. A. The normalized Laplacian matrix is denoted by L Lym D LD 3, and the normalized adjacency matrix is defined in similar fashion: Asym D AD- 3. Here, we use the renormalized version of the adjacency matrix A = D AD as introduced in (Kipf & Welling 2016). where À = A + II, = D ++I. I. Similarly, the renormalized Laplacian matrix is defined as Lsym I A.gm- Lagm is real symmetric matrix, with orthonormal eigenvectors { E R", and corresponding eigenvalues A; E [0,2) (Chung 1997). For
Asgm we have >; (A (-1,
High-pass and Low-pass graph filters. Multiplication of Laplacian with graph signal E Σ, Auuuu x, acts as a filtering operation over z, adjusting the scale of the components of x in the frequency domain. The entries of every eigenvector, aligns with a cluster of connected nodes in the graph. For the Laplacian matrix, a smaller eigenvalue X; corresponds to a smoother eigenvector u, and corresponds to a larger cluster of connected nodes. On the other hand, a larger X; corresponds to non-smooth eigenvectors u, which identify smaller clusters of closely connected nodes in the graph. A Laplacian filter magnifies the part of the signal that aligns well with basis functions corresponding large cigenvalues X (1,2) and suppresses the part the signal that aligns with basis functions corresponding to small eigenvalues X E [0, 1]. That means, for the small clusters of nodes that have large alignment with u, corresponding to X; 1, the projection X u zamplifies x within the cluster and consequently magnifies the difference in x among the nodes within that cluster. On the other hand, for the larger clusters that align well with u, corresponding
4

---

Under review conference paper at ICLR 2023
A < the projection Auu,u x suppresses z within the cluster and reduces the differences in z among the nodes within that cluster. Hence the Laplacian matrices can be generally regarded as high-pass filters (Ekambaram) 2014), that enlarge the differences in node features over small clusters, and smooths out the differences over larger clusters in the graph. On the other hand, affinity matrices, such as the normalized adjacency matrix, can be treated as low-pass filters (Nt Maehara 2019), which suppress and filter out non-smooth components of the signals. This is because all of the eigenvalues of the affinity matrices are smaller than I, ie.. X (-1,1).
On the node level, left multiplying Lmmm and Aoym filters with x can be understood as diversification and aggregation operations, respectively (Luan et al. 2020):
(Leg = Σ
x x,), (Asymz); = E D
DD
JEN; D 2j
(1)
JEN;
High-pass filters highlight the differences between the nodes and their neighbors, making them distinguish- able. On the other hand, low-pass filters aggregate the features of the nodes in neighborhood and make them similar. Since Lygm Asym = I, both filters capture complementary information and their combina- tion allow learning richer representations.
Homophily Ratio. To quantify how likely nodes with similar labels are connected in the graph, homophily ratio, B, is proposed (Pei et al. 2020). as follows:
Number of v's neighbors who have the same label as v Number of v's neighbors
VEV
For larger values of B, it is more likely that nodes with the same labels are connected together. As the nodes belonging to different classes often have different features, graphs with higher are generally smoother.
3.2 GENERATING GRAPH VIEWS VIA GRAPH FILTERS
As discussed, the key idea of our HLCL framework is to leverages a high-pass filter to generate a diverse set of node representations, and contrast them with smooth and similar node representations generated by a low-pass filter. To do so, we leverage the normalized Laplacian matrix Lyy and adjacency matrix A Aym as the diversification and aggregation operations in Eq. (1. We note that other types of high-pass and low-pass filters can be used in a similar way in our framework.
More specifically, we input the graph X into graph encoder, and apply a high-pass filter Fur = i Lsym and a low-pass filter F FLP Asym to the encoder to generate high-pass node representations H и and low-pass node representations H as follows:
H = (F (F ''''), H = HF
H
H X.
(2)
where Hi and H # are the low-pass filtered and high-pass filtered representations at layer of the encoder, W' E gdixd-1 is the weight matrix in layer / the encoder, and a is the activation function.
The high-pass filter F filters out the low-frequency signals and preserves the high-frequency signals. In doing so, it captures the difference between features of every node with the features of the nodes in its neighborhood. Using high-pass encoder within multi-layer encoder iteratively captures the difference between features of the nodes in a multi-hop neighborhood of nodes that have different features from their neighbors distinct in their multi-hop neighhorhood.
node. Hence, makes the representations
On the other hand, the low-pass filter, Fz. resembles a typical GNN, which only preserves the wwefrreuency signals by aggregating every node's features with the features of nodes in its immediate neighborhood.
5

---

Under review conference paper at ICLR 2023
Algorithm 1 Contrastive Learning with Graph Filters (HLCL)
1: for epoch=1,2.3.... do
2: Input graph 9 into the shared graph Encoder f(-)
3: Generate high-frequency-sige
node embeddings: H =0 (
4: Generate www-frequency-sieee node embeddings: Hi o o L (wi-))
5: Compute the contrastive objective L with Eq. (5)
6: Update parameters by applying stochastic gradient ascent to maximize
7: end for
Using the low-pass filter within a multi-layer graph encoder, iteratively aggregates features multi-hop neighborhood of every node to learn its representation. Hence, it smooths out the node representations and produces similar representations for the nodes within the same multi-hop neighborhood.
While the low-pass filter is essential for learning good representations in smooth graphs, it cannot produce distinguished representations for graphs that are non-smooth, specially under heterophily. For such graphs, the high-pass filter is crucial to provide distinct representations. The combinations of both filters provides complementary information and allows learning both smooth and non-smooth components the graphs simultaneously. Hence, it enables learning richer representations, particularly under heterophily.
The high-pass and low-pass filtered representations can be obtained through message passing, according to Eq. (I). In particular, the high-pass filtered representations can be obtained by iteratively differentiating the representations of node and those of its neighbors, and the low-pass filtered representations can be obtained by aggregating the node's representation with those of its neighbors. Formally:
h = oW"''hh ))-)), (h() = EEE(NGO(O) (h) + +h)), (h)) = ESE -h)).
(3)
Explicit graph augmentation. We note that the graph X can be optionally explicitly augmented before entering the encoder. While explicit augmentation do not provide considerable benefit and can even be harmful under heterophily, it is crucial for graphs with homophily. We will study in details the effect of explicit augmentation under homophily and heterophily, our experiments.
Our HLCL framework is demonstrated in Fig. In Appendix, study the effect of filters on toy example.
3.3
THE CONTRASTIVE LEARNING FRAMEWORK
After generating the low low-pass and high-pase pass filtered graph views, we pass them through a shared non non-linear projection head and employ contrastive objective that enforces the filtered representations of each node in the two views to agree with each other and can be discriminated from representations of other nodes.
Non-linear projection head. Before contrasting the high-pass and low-pass filtered representations, we pass them to a non-linear projection head g(-), which maps the filtered views to another latent space where the contrastive loss is calculated. This has been shown to improve the performance (Chen et al.. 2020 You jet al. 2020 Zhu et al.) 2220cc a). and we also confirm its effect our experiment. a two-layer perceptron network to obtain during the contrastive training, and use the low-past filtered representations H & in the downstream task.
our setting, we use g H #) and = g r.).. We use the projection head only
Contrastive loss. For every node i, its projected representation generated by the low-pass filter, z/. is treated as the anchor, and the one generated by the high-pass filter, zj. is treated as the corresponding positive sample. The other projected representations in the two views treated the negative samples.
6

---

Under review conference paper at ICLR 2023
For the contrastive loss function, we use InfoNCE proposed in (Van den Oord et al. 2018). Formally:
sim( s/,+
21) log e e5im(2) + EKEIN
e sim (2im()) (3) + EKEIN (i/ kgi
(4)
kfi
in which sim(2),z)) is the cosine similarity between 2 and z),, and T is a temperature parameter. The second term in the denominator, Exμt (aim(s),a), represents the inter-view negative pairs, i.e., between the low-pass view zi of node and the high-pass views of all the other nodes, and Exzr e(im(z) represents the intra-view negative pairs within the low-pass view. Since two views are symmetric, the loss for another view, [(2), zf), is defined in a similar fashion. The overall objective to be maximized is then defined as the average over all positive pairs. Formally, maximize:
N
Σ ((2) + 1(2), z1)1-
(5)
2N
Effectively, maximizing the agreement between the low-pass and high-pass views pulls away the represen- tation of nodes with different features from their neighborhood, and allows them to be distinguished from their neighbors. The pseudocode is illustrated in Alg.
4 EXPERIMENTS
In this section, we first study and show the ineffectiveness of existing graph augmentation methods under heterophily. Then, we evaluate our proposed framework for node representation learning under linear probe.
Datasets. We conduct experiments on five widely-used public benchmark datasets with different levels of homophily ratios, B. For graphs with homophily, we use the citation networks including Cora and Citeseer (Yang et al. (2016). For graphs with heterophily, we use the Wikipideia network and the web page networks including Chameleon, Squirrel, and Texas (Rozemberezki et al. 202T Pei et al. 2020). Table[4shows, 3, and the number of nodes, edges, and classes for the above datasets. We also report our results on a large-scale social network, namely Penn94 (Lim et al., 202I). Details can be found in Tables and 5
of the Appendix.
HLCL setup. We employ two-layer GCN (Kipf & Welling 2016) as our low-pass encoder, and consider additional high-pass channel with Lyym the message passing filter to capture the high-frequency signals from the graphs. The high-pass channel generates augmented node representations to he contrasted with those encoded by the GCN. The high-pass channel shares the same weight parameters with the GCN, but generate embedding with different filters. The final embedding is generated by the GCN encoder.
4.1 GRAPH AUGMENTATION UNDER HETEROPHILY
First, we study existing data augmentation methods and their effecctivnness for graph CL under heterophily. We use InfoNCE loss defined in Eq. (5) to contrast node representations in augmented graphs, using a stan- dard 2-layer GCN as the graph encoder. We consider topology augmentation methods including Edge Re- moving (ER), Node Dropping (ND), Edge Adding (EA), Subgraph with Random Walk (RWS), and Personal- ized RageRank (PPR), as well as feature augmentation method including Feature Masking (FM) and Feature Dropping (FD) for generating different graph augmentations. We compare these methods with HLCL. Ex- cept for the proposed high-pas pass augmentation methods, we do not use any other explicit graph augmentation. We train the model 10 times with storlpst. and report the average accuracy the final result. For each run, we randomly select 10% nodes for training, 10% nodes for validation, 80% nodes for testing.
Table I shows that method achieves remarkable boost of up 10% compared the best graph aug- mentation method. This confirms the necessity of utilizing the non-smooth graph components under het- erophily. Note that even though topology and feature augmentation methods benefit the homophily graphs,
7

---

Under review conference paper at ICLR 2023
Table 1: Existing graph augmentation methods are highly ineffective for graph CL under heterophily. In
contrast, our HLCLL method achieves up to 10% improvement over the best graph augmentation method.
Chameleon
Squirre
Texas
Ours
HLCL (no-aug)
48.03 +3.1
33,44 11.
64.21 =11.2
ER
34.23 13.88
25.14 =2.3
52.63 =10.7
ND
36,02 1.9
26.70 =1.8
00,53=223
Topo
EA
38.20 =3.1
26.92 =1.6
54.73 #9.7
RWS
35.63 +3.33
24.59 +13.5
56.84 =11.7
PPR
34.14 4:3.5
24.43 4:2.
50.00 +6.7
FM
37.55 13.6
25.58 #1.1
58.42 11422
Feat
FD
36.81
+339
25.75
#2.7
50.52 +6.7
they are highly ineffective for graph CL under heterophily. This can be explained from the perspective of Homophily ratio (3) and graph smoothness, which is represented here as neighborhood feature variance (a)).As shown in figure augmentation methods that remove or add edges (ER, EA) decrease the graphs under homophily. On Cora with homophily, 3 drops rapidly given its initial high value, so it generate a more diverse graph structures for aggrega- tions among nodes that are more beneficial for the contrastive framework. However, for Chameleon with heterophily, B does not con- siderably change, diminishing the benefits of explicit augmentation. Similarly, for Feature augmentation methods that zero out feature entries, a decreases for both heterophily and homophily graphs, smoothing out the graphs. For homophily graphs that have smooth node neighborhoods, the change is not rapid, and the masked features serve as dropout on the input layer, which improves the performance accordingly Zhu et al.] (2021b). On the other hand, feature masking decreases a greatly on Chameleon with heterophily.It smoothes out its non-smooth graph features in deteriorated manner, hence outweights its benefits dropout layer.
Tesplogy Augmentation
Feature Augmentation
0.8
Chamelson
3 30.00000
0.66
Chamaleon
0.00005
0.4
Cora
0.00000
0.2
0.00005
0.0
0,00000
0.0
0.2
0.6
0.5
0.55
1.0
0.0
0.
0.4
0.6
0.8
1.0
Retto
Ratio
Figure 2: Left: Homophily ratio drops with the in- creasing edge removing ratio and edge adding ratio with Cora under homophily and stays approximately the same with Chameleon under heterophily. Right: neigh- borhood feature variance drops with the increasing fea- ture masking ratio, with both Cora under homophily and Chameleon under heterophily, with the decrease more rapid on Chameleon.
4.2 SELF-SUPERVISED REPRESENTATION LEARNING
Next, we evaluate HLCL for self-supervised learning under linear probe We follow the evaluation protocol used in (Zhu et al.] 2020c). Models are first trained in a self-supervised manner without labels. Then, we fed the final node embeddings into lg-regularized logistic regression classifier to fit the labeled data. We train the model 10 times with carly-stopping, and report the average accuracy as the final result. For each run, we randomly select 10% nodes for training. 10% nodes for validation, and 80% nodes for testing. We consider traditional representation learning methods like DeepWalk (Perozzi et al., 2014), deep learning methods including DGI (Velickovic al.) 2019), BGRL (Thakoor et al.) 2021), and GRACE (Zhu al.] (020c),
and other popular representation learning baselines like Raw Features and DeepWalk + Raw Features as baselines. We also include MixHop HAbj-al-Haija et al. 2019) in our experiments as the supervised learning baseline. Table Mabble/compace the performance of HLCL when used on its own, HLCL (no-aug), and when it is applied to the graphs that are augmented with Edge Removing (ER) on the graphs with homophily and Feature Masking (FM) the graphs with heterophily. We see that HLCL shows a significant boost on graphs with heterophily and a comparable performance on the graphs with homophily. Even without explicit augmentation, HLCL surpasses other baselines on graphs with heterophily, which confirms the effectiveness
8

---

Under review conference paper at ICLR 2023
Table 2: HLCL achieves stae-oo-tie of-e-of-theear under heterophily, and comparable performance under homophily when combined with explicit graph augmentation.
Homophily
Heterophily
Cora
CiteSeer
Chameleon
Squirrel
Texas
HLCL
82.34 +22.7
71.35 +1.4 64.34
49.78 +1.
34.11
+1.9
65.26 77.8
HL MT.CC.(mO (no-aug)
77.09 +1.7
+1.5
48.04
+.1
33.44
+1.7
64.21
+11.2
Raw feature
64.8
64.6
31.61
24.43
52.63
DeepWalk
75.7 +1.7
50.5 11.6
27.51
+3.1
25.47
+2,4
40.00
+15.9
DeepWalk features
73.1 +223
47.6 11.5
36.21
+4.6
26.23
+1.8
53.68
+00.2
DGI
82.60 10.4
68.80 100.7
40.48
11.9
27.95
10.9
60.00 110.8
BGRL
74.67 10.6
64.25
1.2.4
38.20
11.4
26.03
10.9
61.05 +11.1
GRACE
83,30 10.4
72.10
10.5
37.37
10.6
28.67
10.9
62.6 17.2
MixHop
85.34 10.4
73.23
10.5
46.84
13.5
36.42
13.4
62.15 -2.5
of the high-pass channel. Importantly, applying HLCL to graphs that are explicitly augmented achieves a comparable performance under homophily. Thus, the high-pass filter does not harm the performance on such graphs. Comparing to supervised learning model such as MixHop trained in an end-to-end manner, HLCL achieves a comparable or superior performances. This, demonstrates the effectiveness of our design.
4.3
ABLATION STUDIES: HLCL WITH EXPLICIT AUGMENTATION
Table Combining HLCL with different graph view augmentations. LP represents augmentation on low- pass channel only, HL represents augmentation both channels. Both topology and feature augmentations are included. The full table/6 presented in the Appendix. Worst results are highlighted in gray.
Homophily
Heterophily
Cora
CiteSeer
Chameleon
Squirrel
Texas
HLCL (no-aug)
77.09 1.7
64.34
1.5
48.03
13.1
33.44 11.7
64.21
+11.2
FM
78.46 22.4
11.7 66.68
49.78
11.8
34.11
11.9 33.65
65.26
1.7.8
LP Feat
FD
78.42 2.7
67.75
11.7
47.46
13.2
11.7
63.15
17.4
ER
82.34 22.7
71.35 11.4
37.29
+2.6
25.60
1.6
57.37
+8.3
HI Topo
ND
76.10 226
65.90 12266
35.81
11.8
25.39
11.5
61.58
+8.8
EA
81.02 2.5
70.02 13.6
37.46
1..
26.12
11.7
60.53
17.9
FM
78.60 2.3
66.08
12.0
49.52
13.6
33.90
11.5
63.68
18.9
HL Feat
FD
77.50
330
66.59
11.7
48.42
14.0
33.49
11.8
63.68
1-6.8
To investigate the interactions between traditional graph augmentations and HLCL on graphs with different 3, we perform ablation studies on explicit graph augmentation methods applied to both high-pass and the typical low-pass GCN channels. We include Edge Removing (ER), Node Dropping (ND), Edge Adding (EA), Feature Masking (FM), and Feature Dropout (FD) as graph augmentation methods. Table 3 shows that combined with HLCL, feature and topology augmentations considerably benefit the performance on graphs with homophily. Topology augmentation, however, may not be helpful under heterophily.
5 CONCLUSION
We proposed HLCL, contrastive learning framework that leverages a high-pass graph filter to generate non- smooth augmented representations be contrasted with their smooth counterparts, generated using the same encoder. This is particularly beneficial for graphs with heterophily, where existing graph CL methods are highly ineffective. Through extensive experiments, up to 10% boost in the performance for representation learning under the linear probe, for various graphs under heterophily, At the same time, it provides comparable performance graphs with homophily.
demonstrated that our proposed framework achieves
9

---

Under review conference paper at ICLR 2023
REFERENCES
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyun- yan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference machine learning, pp. 21-29. PMLR, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in spectral perspective. In International Conference on Learning Representations, 2020.
Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V Chawla. Graph barlow twins: A self-supervised represen- tation learning framework for graphs. arXiv preprint arXiv:2106.02444 2021.
Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph con- volutional networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 3950-3957, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In International conference machine learning, pp. 1597- 1607. PMLR, 2020.
Fan RK Chung, Spectral graph theory, volume 92. American Mathematical Soc., 1997.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
Venkatesan Nallampatti Ekambaram. Graph-structured data viewed through Fourier lens. University of California, Berkeley, 2014.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pp. 4116-4126. PMLR, 2020.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv
preprint arXiv: 02907, 2016.
Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-honee non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems, 34:20887-20902, 2021.
Sitao Luan, Mingde Zhao, Chenqing Hua, Xiao-Wen Chang, and Doina Precup. Complete the missing half: Augmenting aggregation filtering with diversification for graph convolutional networks. arXiv preprint arXiv:2008.0888 08844. 2020.
Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters, arXiv preprint weXiv:/905.09555 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint 27Xiv:2002.055287, 2020.
10

---

Under review conference paper at ICLR 2023
Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong. Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pp. 259-270, 2020.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 701-710, 2014.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1150-1600, 2020.
Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Nerworks, 9()xcnab014, 2021.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
Shantanu
Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva Dyer, Remi Munos, Petar Veličković, and Michal Valko. Large-scale representation learning on graphs via boot- strapping. arXiv preprint arXiv:2102.06514, 2021.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10003 2017.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liō, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Yewen Wang, Ziniu Hu, Yusong Ye, and Yizhou Sun. Demystifying graph neural network via graph filter assessment, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via parametrre instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 3733-37422 2018
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 18100008000 2018
Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing graph convolutional neural networks. arXiv preprint arXiv:2102.06462. 2021.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pp. 40-48. PMLR, 2016.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 2210-6219, 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5822 2020.
11

---

Under review conference paper at ICLR 2023
Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16650-16659, 2022.
Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen Ahmed, and Danai Koutra. Graph neural networks with heterophily. arXiv preprint arXiv:2009. pp. 11168-11176 2020a.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond ho- mophily in graph neural networks: Current limitations and effective designs. Advances in Neural Infor- mation Processing Systems, 33:7793-7804, 2020b.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pp. 1215-1226, 2021a.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive represen- tation learning. arXiv preprint 2oXiv:0006.0413.. 2020c.
Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. arXiv preprint arXiv:2109.01 01 116, 2021b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pp. 2021c
12

---

Under review conference paper at ICLR 2023
A APPENDIX
A.1 BENCHIMARK DATASETS
We conduct experiments five widely-used public benchmark datasets with different level of ho- mophily ratios, B. We shows the specific statistics of the datasets including 3, the number of nodes, edges, and classes, in table 4 In addition, to illustrate the scalability of our model, dataset like Penn94, proposed by Lim et al. (202T). We train the model 3 times with early-stopping, and report the average accuracy the final result. For each run, we randomly select 10% nodes for training, 10% nodes for validation, and 80% nodes for testing. We present our result in table As shown, our method works better in scage-scale dataset than other self--uuer learning methods as well.
our models realistic
Table 4: Statistics for our datasets
Datasets
Cora
CiteSeer
Chameleon
Squirrel
Texas
Penn94
Hom. ratio (9)
0.83
0.71
0.25
0.22
0.06
0.47
Nodes
2708
3327
2277
5201
183
41,554
Edges
5278
4676
31421
198493
295
1,362,229
Classes
7
5
2
Table 5: HLCL achieves a noticeably higher accuracy on realistic large-scale dataset comparing to other GCL methods. HLCLL outperforms SOTA by 8%.
Penn94
HLCL
70.97 +0.5
DGI
62.85 10.3
BGRL
56.57 #0.8
GRACE
62.53 +0.1
A.2 ABLATION STUDIES: HLCL WITH EXPLICIT AUGMENTATION
To investigate the interactions between traditional graph augmentations and HLCL graphs with different 3. we perform ablation studies on explicit graph augmentation methods applied to both high-pass and the typical low-pass GCN channels. We include Edge Removing (ER), Node Dropping (ND), Edge Adding (EA), Feature Masking (FM), and Feature Dropout (FD) as graph augmentation methods. We present our result in 6
13

---

Under review conference paper at ICLR 2023
Table 6: The result on combining HLCL with different graph view augmentations. HP represents aug- mentation on high-pass channel only, LP represents augmentation on low-pass channel only, HI represents augmentation on both channels. Both topology and feature augmentations are included in the experiments.
Homophily
Hcterophily
Cora
CiteSeer
Chameleon
Squirrel
Texas
HLCL (no-aug)
77.09 +1.7
04.34#1.3
48.03 #3.1
33.44 #1.7
64.21 #11.2
ER
80.80 +:3.0
71.20
+1.3
36.11
#4.1
26.26 +2.3
57.36
4.8.9
HP Topo
ND
78.42 +2.0
70.24 +2.0
35,80 22.7
26.79 =1.
61.05 =12.
EA
74.04 12.9
65.75
#3.2
35.81 33.3
26.79 =2.0
60.53 #7.9
FM
77.13 12.9
11. 64.67
49.08 13.8
33.80
=1.7
62.63 19.8
HP Feat
FD
77.94 1:2.6
11.8 66.05
47.07 14.0
32.71
+2.0
61.57 16.2
ER
77.94 13311
65.18
11.9
37.07 13.4
25.36
12.0
57.37
17.6
LP Topo
ND
75.66 11.6
61.28 12.6
36.15 1200
26.53
11.6
59.47 18.5
EA
80.73 12.2
12.0 68.35
39.08
12.0
26.75
11.5
61.05 1118
FM
78.46 12244
66.68
11.77
49.78
118
34.11
11.9
65,26 17.88
LP Feat
FD
78.42 #2.7
67.75 +1.7
47.46
#3.2
33.65
11.
63.15 #7.4
ER
82.34 1:2.7
71.35
+1.4
37.29
226
25.60
116
57.37
+8.3
HL Topo
ND
76.10 +2266
65.90
+2.6
35.81
11.
25.39 11.5
61.58 4888
EA
81.02 +2.5
70.02
+3.6
37.46
1.8
26.12 117
60.53 77.9
FM
78.60 +2.3
66.08
+2.0
49.52 3.6
33,90
11.
63.68 88.
HI. Feat
FD
77.50
+3.0
66.59 117
48.42 4.0
33.49 11.
63.68 16.
A.3 ABLATION STUDIES: OUTPUT ENCODER WITH DIFFERENT GRAPH FILTERS
our experiments, we use low-pass graph filter in our GNN encoder as the final representation encoder. It is important to note that the encoder that is trained with contrasting with the high-pass view does not suffer from the over-smoothing problem, even without using the high-pass filter during the inference. This because the parameters of the encoder are learned by contrasting the low-pass representation with high-pass representations. Hence, we do not need to rely on the high-pass filter during the inference. To illustrate the effectiveness of our structural design, we present an ablation study using different filters combinations of filters in our final encoder. The result is shown in table 7 We see that on most datasets low-pass encoder achieves the highest accuracy with a large gap, and variation involving the high-pass encoder has worse performances across all datasets except Texas, which has a very low homophily ratio. This demonstrates the ineffectiveness of using high-pass filter directly in the representation encoder.
Table 7: HLCL with different graph filters in the final encoder. HP represents high-pass filter; LP represents low-pass filter; HP Concat IPP represents using both high-pass filters and concatenate the output representa- tions; HP Aggre LP represents using both high-pac pass filters and aggregate the output representations
Homophily
Heterophily
Cora
CiteSeer
Chamelcon
Squirrel
Texas
LP
77.09 +1.7
64.34 +1.5
48.04 +3.1
33.44 +1.7
64.21 +11.2
HP
51.94 +2.9
36.22
+2.5
35.58
+3.1
29.83 +1.
65.26 +10.3
HP Concat LP
66.47 +3.6
55.96 +2.6
41.09 #3.6
30.86
+1.9
65.21 +9.4
HP Aggre LP
65.33
+.2.8
53.47 43.1
38.77 +3.4
28.00 +1.5
63.15 +11.7
14

---

Under review conference paper at ICLR 2023
Table 8: Swapping low-pass encoders in GCL methods to high-pass encoders. Significant performance drops are highlighted in gray.
Homophily
Heterophily
Cora
CiteSeer
Chameleon
Squirel
Texas
HLCL
82.34 +2.7
71.35
11.
49.78 +1.8
34.11 +1.9
65.26 87.8
DGLllw
82.60 +0.4
68.80
=0.7
40.48 11.9
27.95
10.9
60.00 10.88
DGIhiggh
31.95
+2.8
30.54
=1.7
38.38 #3.5
29.57 #1.2
62.11 #10.2
BGRL:low
74.67
1:0.6
64.25
1224
38.20 1144
26.03 10.9
61.05 111.1
BGRL:high
29.63
12.8
24.99
#3.1
35.41 13.4
29.57 =1.1
65.26 18.2
GRACE:ow
83.30
+0.4
72.10
+0.5
37.37 +0.6
28.67
+:09
62.6 #7.2
GRACE:high
32.46
+2.0
26.55
=3.1
39.65 12.9
33.05 12.1
72.63 19.3
A.4 ABLATION STUDIES: OTHER GCL METHODS WITH DIFFRRENT ENCODERS
In this section, illustrate the importance of our contrastive structure. We show that the main accuracy gains on heterophily datasets are from contrasting the low- and high-pass filtered graph views, instead of the introduction of the high-pass filter alone. The high-pass filter alone cannot provide high-quality repre- sentations under heterophily. For different graphs, different combinations of low-pass and high-pass filters are required to provide good representations. Having access to labels, the best combination can be learned. However, this is indeed not possible without labels. We have shown that contrasting the low- and high- filtered graph views can provide high-quality representations in the unsupervised scenario. Here, We conduct a new ablation study by swapping the encoders from other graph CL methods with high-pass encoder. The result is presented in By replacing low-pass filter with high-pass filter, grph CL. methods are able to achieve an better accuracy some heterophily datasets like Squirrel and Texas. However, their performances de- teriorate significantly under homophily cases, where aggregation among neighborhoods are desired. Our results show that when using high-pass low-pass alone, the model cannot achieve good performance.
A.5 ILLUSTRATION EXAMPLES ON HP AND LP FILTER UNDER DIFFERENT HOMOPHILY RATIOS
To explain the effect of high-pass and low-pass filters and motivates using both filters in contrastive way, we construct a simple graph with seven nodes. Nodes are separated into two classes, labeled as red and green shown in the figure. First, generate features vectors with 200 dimensions for nodes from red and green classes based on two Gaussian distributions, centered at -5 and 5 respectively with a standard deviation of 1, Then, we do graph aggregation or diversification with low- and high-pass filters separately, following Eq. T For clearer illustration, we do not project the feature vectors into other dimensions.
We considered three different homophily ratios and ran the node features through high-pass and low-pass filters for few iterations. We see that: (1) for high homophily ratio graph (Fig. 33, the low-pass filter reduces the variance of the features and pulls different classes towards each other, while the high-pass filter increases their variance and eventually pulls the classes away from each other. Hence, contrasting the two filtered representations prevents smoo-sngg The figure illustration is shown at Fig. 4 (2) for lower homophily ratio graph, where a part of the graph has homogeneous labels and the other part has heterogeneous labels (Fig. (5). the low-pass filter mainly reduces the variance of the features for the habo--homogeneuus part, while the high-pass filter unnecessarily mixes the class features. At the same time, the high-pass filter increases the variance of features for the label-heterogeneouu part and pushes them away from the other class, while the low-pass filter unnecessarily mixes the class features. In this case, contrasting the two filters is crucial for learning good-quality representations. The figure illustration is shown at Fig. (3) Finally, for very low
15

---

Under review conference paper at ICLR 2023
homophily ratio graph (Fig. 7). the low-pass filter reduces the variance of the distributions and mixes the distributions of the features in the two classes, making them indistinguishabl. On the other hand, the high- pass filter pushes away the distributions of the features in different classes. By contrasting the high-pass and low-pass wwwpass filtered features, HLCL is able to distinguish the feature distributions under heterophily. The figure illustration is shown at Fig. g
Figure 3: Case 1: High homophily ratio graph
Green festure
025
Creen festure
Grean festure
Green LP
Green LP
Groon LP
0.2
Green HP
Green HP
Green HP
feature
Red feature
0.15
2:16
2015
6.10
0.56
10
-10
10
20
Figure 4: The change distributions of node (green class) Fig.
at iterations 1, 3, and 6.
Figure 5: Case 2: A mixture of homophily and heterophily components
16

---

Under review conference paper at ICLR 2023
Green teature
Green feature
Green teature
Red LP Red HP
5.20
Red LP
Red LP
Red HP
Reed HP
Stature
Red foature
Dengi
40.0
0/10
1.45
10
20
10
0.20
Greon
0.00
Green
Green
Green
0.15
Red feature
0.15
Red feature
Red foature
Dondtt
2.10
D.10
0.50
0.00
vall
Figure 6: The change of distributions of node 2 (top row) and node 4 (bottom row) in Fig, iterations
3, and 6. Node 2 is at a heterogeneous neighborhoods, while node at a homogeneous neighborhoods
Figure 7: Case 3: Very low homophily ratio
Green festure
Creen fexture
Grean fealure
Red LP
2.20
0.20
HP
Ocneit
0.19
0.10
2.10
0.55
0.00
-10
5.00
-20
-10
10
000
-40
-20
Figure The change of distributions of node 2 (green class) in Fig.
7 at iterations 1, 3, and
17


Reviews:
[{"review_id": "xEHsnD7L--6", "reviewer": "Reviewer_TBxC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "They argue that current graph contrastive learning methods are inadequate for graphs with ‘heterophily’ in which nodes have diverse signals. They argue this is partially due to the nature of GNNs over-smoothing node signals in neighborhoods. They propose a method of view generation for contrastive learning on graphs in which a typical GNN-encoded graph is contrasted with a graph in which differences between nodes are magnified. They do this by using a ‘laplacian matrix’ or the degree diagonal matrix minus the adjacency matrix in place of the typical augmented adjacency matrix when ‘aggregating’ signals in the typical GNN way. The laplacian matrix is used in one branch and the other uses the adjacency matrix as a typical GNN, they are in my opinion analogous to a sharpening kernel v.s. a blurring kernel. The results of these two branches are used for contrastive learning as a positive pair, with other samples being negative. They show results against benchmarks on datasets with homophily (similar nodes) v.s. heterophily, and have particularly impressive results with the latter, and studies on how augmentations compare with their method in graphs with heterophily, and how their method changes with augmentation schemes. ", "summary_of_the_review": "I believe the paper is clear and concise. The technical contribution / deviation from related work is very simple, and it is hard to say it’s entirely novel, but is put to good use, and I believe it is novel in its application to graph contrastive learning at the least. The results are impressive. ", "strength_and_weaknesses": "The strengths of the paper are that it is very clear and the results seem impressive. I don’t have particular actionable criticism that I can think of. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i6At54ZqsY", "reviewer": "Reviewer_vWUj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for graph contrastive learning, specifically optimized for heterophily graphs. The idea of HLCL is simple: to generate contrastive pairs, HLCL uses two branches of encoders. One encoder is the vanilla GCN which performs low-pass filtering (smoothing); the other encoder is uses the Laplacian matrix as the propagation matrix so that each layer preserves high-pass filtering. The low-pass embeddings are contrasted with the high-pass ones with a standard contrastive loss (InfoNCE). The additional high-pass encoder helps preserve information for heterophilous graph. Experiments on both homophilous and heterophilous graphs show that HLCL achieves significant accuracy improvements compared with various baselines. ", "summary_of_the_review": "Overall, the paper presents a simple constrastive learning method that shows some accuracy gains on small heterophilous datasets. The paper shows some potential from its evaluation. However, at its current stage, I think the paper is still below the bar of acceptance due to its lack of in-depth analysis, limited novelty and lack of evaluation on realistic, large graphs. ", "strength_and_weaknesses": "## Strengths\n\n+ The paper is well-written and very easy to read. \n+ The idea to use high-pass filters for high heterophily is reasonable. \n+ Experiments have been performed extensively to demonstrate the accuracy gains as well as ablations. \n\n## Weaknesses\n\n- The novelty of the design is limited. There have been many existing works combining low-pass and high-pass filtering in GNN layers. Using high-pass filtering to address heterophily is also known in the literature. The designs of the constrastive learning architecture and loss function follow a standard protocol.  \n- The overall design is simple. It would be better to include more analysis to deepen the understanding. \n- It is mentioned that the \"high-pass-low-pass\" encoder pair is used only in contrastive learning. Only the low-pass encoder is used during inference. It is not clear how such a design can handle heterophily during inference: regardless of the training process, the inference architecture only performs smoothing and is unable to preserve high-frequency components. \n- Experiments would be more convincing if larger graph are used. The currently included benchmarks are all very small. However, one of the main motivations for CL is that labels are expensive to obtain on *large* graphs. Achieving accuracy gain on small graphs do not fully justify the proposed CL method. \n- Experiments should follow a standard setup. For example, each dataset should come with a standard data split commonly accepted by the community. However, the authors \"randomly select 10% nodes for training, 10% nodes for validation and 80% nodes for testing\". As a result, some accuracy number of the baselines do not match those in the original papers (e.g., DGI on CiteSeer). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hQRi4cTD5O", "reviewer": "Reviewer_32Ub", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a graph contrastive learning method called HLCL, which performs better than SOTA methods on heterophily graph datasets. HLCL uses a high-pass and a low-pass graph filter to generate two graph representations and then contrasts the two representations. The high-pass filter uses renormalized Laplacian matrix left, multiplying the product of the node embeddings matrix and weight matrix, while the low-pass filter uses a renormalized version of the adjacency matrix. Some experiments analyze the influence of different graph augmentation methods.", "summary_of_the_review": "Although this paper is not so novel in specific GNN, it still proposed a contrastive learning frame for heterophily graphs, and also discussed the influence of different graph augmentation methods in its frame.", "strength_and_weaknesses": "Strengths:\n1. In the experiment part, this paper discussed the effect of different graph augmentation methods very thoroughly.\n\n2. The problem of heterophily is worth exploring, and there are not many works analyzing this problem very deeply. This paper provides a contrastive learning method for this problem, which has great potential.\n\nWeaknesses:\n1. Lack of explanation about the influence of the low homophily ratio on the low-pass filter, and about why HLCL works in high homophily ratio graphs. Since this paper claims that HLCL can obtain good performance, there should be some explicit explanation about why HLCL is supposed to do better on heterophily graphs. Explanation with only words is not convincing. Some toy examples or math descriptions of the influence of the graph’s homophily ratio and high- and low-pass filters are strongly suggested.\n\n2. Confusing math notations. In 3.1 PRELIMINARIES  High-pass and Low-pass graph filters, several transpose donations are missed for vector ui.\n\n3. Outdated SOTA result. SOTA results in Table 2 are not up to date. For example, Chameleon, Squirrel, and Texas datasets. For example, GGCN in Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks (accepted in ICDM 2022), whose code is open-sourced, outperforms HLCL on both Chameleon and Squirrel datasets, two datasets with heterophily graph. Only comparing HLCL with Contrastive learning methods cannot prove its high accuracy. \n\n4. The datasets used in the experiments are small and outdated. There are only two types of networks, i.e., small citation networks and small web page networks. This makes the results not sufficiently convincing.\n\n5. Lake of novelty. In Interpreting and Unifying Graph Neural Networks with An Optimization Framework (www2021), GNN-LF (GNN with Low-pass Filtering Kernel) and GNN-HF (GNN with High-pass Filtering Kernel) are proposed, GNN-HF-closed has SOTA performance on Cora and CiteSeer. This paper uses contrastive learning with these two GNNs, so the novelty is marginal. Besides, GNN-LF contains a linear combination of normalized adjacent matrix and identity matrix, while GNN-HF contains the sum of normalized Laplacian matrix and identity matrix. The details of the high- and low-pass filters are very similar to this paper.\n\n6. Bad algorithm expression. The pseudocode illustrated in Alg. 1 is meaningless. It is better to add some unique details of HLCL in Alg. 1. The current Alg. 1 seems to fit every graph contrastive learning algorithm.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "F4ljMJ-Dpy", "reviewer": "Reviewer_KBX6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes taking leverage of graph filters to directly generate augmented graph views for graph contrastive learning. The model tries to maximize the agreement between a low- and high-pass encoder of the same graph. The experiments show that this approach improves the graph CL results in heterophily graphs. ", "summary_of_the_review": "Most of the points are mentioned in the strength and weaknesses section but in summary, to the best of my knowledge, this is the first paper that addresses graph CL for heterophily graphs.\\\nThe paper is clear and the method is straightforward.\\\nHowever, I believe the scope is very limited since the significant gains are seen only in heterophily graphs. Also, calculating Laplacian for very large graphs can be troublesome. Besides, we don't know the effect of just replacing the encoders in the current graphCL methods with a high-pass one.", "strength_and_weaknesses": "**Strength**:\n- The paper is well-written and clear.\n- Heterophily graphs have gained more attention recently, and the problem they are trying to tackle is relatively novel.  \n- The method is straightforward and the results are promising.\n\n**Weaknesses**:\n- My main question is how much of the reported performance gains in heterophily graphs are coming from the contrastive part and how much from the high-pass filter. What would happen if we just replace the encoder in the other graph CL methods in table 2, with a high-pass encoder?\n- Minor:\\\ntypo: \"We follows\" in line 1, section 4.2 \\\nPage 6, the paragraph above equation 3 has 2 repetitive sentences. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject