Title: Simple and Asymmetric Graph Contrastive Learning without Augmentations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Contrastive Learning

Graph Representation Learning
Abstract: Graph Contrastive Learning (GCL) has shown superior performance in representation learning in graph-structured data. Despite their success, most existing GCL methods rely on prefabricated graph augmentation and homophily assumptions. Thus, they fail to generalize well to heterophilic graphs where connected nodes may have different class labels and dissimilar features. In this paper, we study the problem of conducting contrastive learning on homophilic and heterophilic graphs. We find that we can achieve promising performance simply by considering an asymmetric view of the neighboring nodes. The resulting simple algorithm, Asymmetric Contrastive Learning for Graphs (GraphACL), is easy to implement and does not rely on graph augmentations and homophily assumptions. We provide theoretical and empirical evidence that GraphACL can capture one-hop local neighborhood information and two-hop monophily similarity, which are both important for modeling heterophilic graphs. Experimental results show that the simple GraphACL significantly outperforms state-of-the-art graph contrastive learning and self-supervised learning methods on  homophilic and heterophilic graphs. The code of GraphACL is available at https://github.com/tengxiao1/GraphACL.

Simple and Asymmetric Graph Contrastive Learning without Augmentations

Abstract

Graph Contrastive Learning (GCL) has shown superior performance in representation learning in graph-structured data. Despite their success, most existing GCL methods rely on prefabricated graph augmentation and homophily assumptions. Thus, they fail to generalize well to heterophilic graphs where connected nodes may have different class labels and dissimilar features. In this paper, we study the problem of conducting contrastive learning on homophilic and heterophilic graphs. We find that we can achieve promising performance simply by considering an asymmetric view of the neighboring nodes. The resulting simple algorithm, Asymmetric Contrastive Learning for Graphs (GraphACL), is easy to implement and does not rely on graph augmentations and homophily assumptions. We provide theoretical and empirical evidence that GraphACL can capture one-hop local neighborhood information and two-hop monophily similarity, which are both important for modeling heterophilic graphs. Experimental results show that the simple GraphACL significantly outperforms state-of-the-art graph contrastive learning and self-supervised learning methods on homophilic and heterophilic graphs. The code of GraphACL is available at https://github.com/tengxiao1/GraphACL .

1 Introduction

Contrastive learning has emerged as a promising regime for unsupervised vision representation learning without using annotated labeled data [1]. Recently, graph contrastive learning (GCL) has also been introduced to graph-structured data due to the lack of task-specific node labels [2, 3, 4, 5, 6, 7]. GCL has achieved competitive (or even better) performance on many downstream tasks on graphs compared to its counterparts trained with annotated ground-truth labels [7, 8].

Generally, existing GCL methods can be categorized into two categories. The first contrastive scheme [9, 10, 11, 12] reconstructs the local network structure (i.e., observed edges) to align with traditional network-embedding objectives [13, 14]. Specifically, this scheme treats one-hop or random-walk local neighboring nodes of the target node as positive examples and non-neighboring nodes as negative samples. It then employs contrastive loss functions to minimize the representation distance of positive pairs and maximize the distance of negative pairs [10]. The key motivation behind this scheme is the explicit homophily assumption that semantically similar nodes are more likely to be linked than dissimilar ones. Thus, connected nodes should have similar representations in the latent space. However, in heterophilic graphs, connected nodes are not necessarily from the same semantic class [15, 16], and should not be simply pulled together in the latent space. This scheme empirically faces issues with performance in graphs exhibiting heterophily [17, 18].

The second graph contrastive scheme involves graph augmentations [3, 4, 5, 6, 7]. Specifically, it constructs two views through stochastic graph augmentation and then learns representations by contrasting these views based on the information maximization principle [19]. A positive node

pair consists of two views resulting from stochastic data augmentation of the same node, while a negative pair might consist of two views of different nodes [8]. This scheme is built based on the idea that augmentations can preserve the semantic nature of samples, i.e. augmented nodes have consistent semantic labels with the same original nodes. However, recent work has shown that graph augmentation methods struggle to achieve good performance on heterophilic graphs [18] as they still implicitly rely on homophily [20]. Studies have shown that stochastic augmentation primarily captures the common low-frequency information between two views, while neglecting the high-frequency one [20] (also see Appendix C.4). The latter is known to be more crucial for heterophilic graphs [21, 22]. Given that heterophilic graphs are prevalent across various real-world domains [16, 23], a fundamental and open question naturally arises: What kind of contrastive learning objectives can learn robust node representations on both homophilic and heterophilic graphs?

This work is the first to address the aforementioned question without relying on augmentations and homophily assumptions. Instead, we take into account two more generalized insights, as depicted in Figure 1. (i) Even in heterophilic graphs, two nodes of the same semantic class often share a similar one-hop neighborhood context [24, 18]. Thus, capturing this heterophilic one-hop neighborhood context can lead to more discriminative node representations. (ii) While homophily might be minimal or non-existent in heterophilic graphs, another frequently observed

(a) Homophily

(b) Heterophily

phenomenon in real-world graphs is monophily [25]. Monophily describes situations in real-world graphs where the attributes of a node's friends are likely to be similar to the attributes of that node's other friends [25]. Put another way, 'monophily' essentially induces similarities between two-hop neighbors. As noted by [25, 26], the two-hop similarities brought about by monophily can persist even in the total absence of any one-hop similarities that might be suggested by homophily.

To support our motivations, we provide statistics of homophily, two-hop monophily ratios, and neighborhood similarities of various graphs in the real-world in Appendix C. As shown in these statistics, homophilic and heterophilic graphs in the real-world exhibit relatively strong neighborhood similarity calculated based on one-hop neighborhoods compared to homophily. In cases where homophily is weak or non-existent, monophily has been shown to still hold in real-world graphs.

In this work, we focus on exploiting the above insights to design new objectives for better node representations on both homophilic and heterophilic graphs. We propose a simple yet effective framework termed as graph asymmetric contrastive learning (GraphACL) for better node representation learning. Essentially, we are faced with the challenge of simultaneously capturing the one-hop neighborhood context and monophily in the contrastive objective. To solve this challenge, we consider each node to play two roles: the node itself (identity representation) and specific 'neighbors' of other nodes (context representation), and thus should be treated differently. GraphACL trains the node identity representation by predicting the context representation of one-hop neighbors through an asymmetric predictor. Intuitively, by enforcing identity representations of two-hop neighbors to reconstruct the same context representation of the same central nodes, GraphACL implicitly makes representations of two-hop neighbors similar and captures the one-hop neighborhood context (Figure 2).

Our primary technical contributions are: (1) We propose a simple , effective , and intuitive graph contrastive learning approach which captures one-hop neighborhood context and two-hop monophily similarities in a simple asymmetric learning framework. (2) We theoretically analyze the learning behavior and prove that GraphACL is guaranteed to yield good node representations for both homophilic and heterophilic graphs. (3) Empirically, we corroborate the effectiveness of GraphACL on 15 graph benchmarks. The results demonstrate that GraphACL can significantly outperform previous GCL methods. The surprising effectiveness of GraphACL shows that in the context of GCL, simpler methods have been underexplored in favor of more elaborate algorithmic contributions.

2 Related Work

Graph Contrastive Learning. Contrastive methods are central to traditional network-embedding methods [27, 13, 14], but have recently been applied in graph self-supervised learning [9, 28, 10].

similar

predict

(a) Graph

(b) Homophily

(d) Monophily

similar

predict

similar

(c) Context

predict

predict

similar

The key motivation behind them is the explicit homophily assumption that connected nodes belong to the same class and, thus, should be treated as positive pairs in contrastive learning. However, real-world graphs do not always obey the homophily assumption [16], limiting their applicability to heterophilic graphs [18]. Recently, many GCL algorithms with augmentations [2, 3, 4, 5, 6, 29] have been proposed. However, [20] recently proved that GCL with augmentations attempts to manipulate the encoder to capture low-frequency information instead of the high-frequency part and suffers performance degradation in heterophilic graphs [18]. In contrast, we propose a simple asymmetric contrastive learning framework for graphs without augmentations and homophily assumption.

Heterophilic Graphs. There are many heterophilic graphs in the real world that exhibit nonhomophilic properties, such as transaction [30, 31], ecological food [32] and molecular networks [15], where the linked nodes have different characteristics and different class labels. Various graph neural networks (GNNs) [33, 15, 21, 34, 35, 23, 36, 37] have been proposed to achieve higher performance in low homophily settings. They focus on designing advanced GNN architectures and consider the semi-supervised setting with labels. In contrast, we focus on designing the contrastive learning algorithm without labels, not on specific GNN architectures. Recently, DSSL [18] and HGRL [29] have been proposed to conduct self-supervised learning on nonhomophilous graphs by capturing global and high-order information. Specifically, HGRL is based on graph augmentation, and DSSL works by assuming a graph generation process, which may not always hold true for real-world graphs. In contrast, our work provides a novel perspective on graph contrastive learning, which does not make any assumptions about the generation process or rely on any augmentations.

3 Preliminaries

Notations and Problem. Let G = ( V , E ) be a input graph, where V = { v 1 , . . . , v |V| } is the set of |V| nodes and E is the set of edges. Let X =[ x 1 , x 2 , · · · , x |V| ] ∈ R |V|× D x be the node attribute matrix, where x i is the D x -dimensional feature vector of v i . Each edge e i,j ∈ E denotes a link between node v i and v j . The graph structure can be denoted by an adjacency matrix A ∈ [0 , 1] |V|×|V| with A i,j = 1 if e i,j ∈ E , otherwise A i,j = 0 . We denote the normalized adjacency matrix ˜ A = D -1 / 2 AD -1 / 2 where D is the diagonal degree matrix. Let L = I -˜ A be the symmetric normalized graph Laplacian matrix. We also define the unweighted two-hop graph G 2 , whose adjacency matrix is A 2 . ( A 2 ) ik = 1 if there exists j such that e ij ∈ E and e jk ∈ E . The input graph G can be denoted as a tuple of matrices G = ( X , A ) . Our goal is to learn a GNN encoder f θ parameterized by θ , such that the representation of node v : v = f θ ( G )[ v ] ∈ R D , can perform well for downstream tasks.

GCL with Representation Smoothing. This paradigm [13, 9, 28, 12] ensures that local neighboring nodes have similar representations and has also been shown to be equivalent to factorizing graph proximity [38]. Despite minor differences, the objective for this paradigm can be expressed as:

Homophily Ratio. Typically, the graph homophily ratio [33, 15] is defined as the fraction of edges connecting nodes with the same labels, i.e., H ( G ) = | { ( u, v ) : ( u, v ) ∈ E ∧ y u = y v } | / |E| . Homophily Graphs have high edge homophily ratio H ( G ) → 1 , while heterophilic graphs (i.e., low/weak homophily) correspond to small edge homophily ratio H ( G ) → 0 [15, 16].

Here v , u and v -are the projected node representations by f θ ( G ) of nodes v , u and v -, respectively. τ is the temperature hyper-parameter. Typically, N ( v ) is the positive sample set containing one-hop

<latexi sh

1_b

64="Wk3jdEVm

Rog9

GCy

r

KF

>A

B

H

c

SwN

+

q

I

8

fY

v

Q

U

O/

MP

Z

u

7

n

D

0

T

5

p

<latexi sh

1_b

64="

R

97+E0v

JG

U

X

fV

oq5Y

>A

C

H

cj

L

S

N

FD2

r

p

Zdugk

w

I

B

mn

y

M

/

8

T

z

O

K

W

3

P

Q

<latexi sh

1_b

64="v y803Mmun5wc

N

V

g

2

d

E

>A

B

X

DLS

O

r

fUY9

oP

K

I

WG

J

k

Z

R

C

/

HT

q

j

7

F

Q

+

p

z

<latexi sh

1_b

64="v y803Mmun5wc

N

V

g

2

d

E

>A

B

X

DLS

O

r

fUY9

oP

K

I

WG

J

k

Z

R

C

/

HT

q

j

7

F

Q

+

p

z

<latexi sh

1_b

64="K

Sk

Nw

p

o

f

VF

g

I

T3j

Y

>A

B8X

c

DL

M

9Z

W

Rn

y

Pr

d

0

m

O

E

G

/

u

C

Q

q7

v

2

5

U

H

J

+

z

<latexi sh

1_b

64="K

Sk

Nw

p

o

f

VF

g

I

T3j

Y

>A

B8X

c

DL

M

9Z

W

Rn

y

Pr

d

0

m

O

E

G

/

u

C

Q

q7

v

2

5

U

H

J

+

z

<latexi sh

1_b

64="

A

wNq

LDmy

FGn

p

8+

uoB

>

X

c

V

SgM

T

7W

d

Iv r2

j

K9

H

U

J

3

Z

k

C

/

0

Y

QO

z

5f

R

E

P

<latexi sh

1_b

64="j nTF

Z+

9

G8

B

ug

>A

/3

c

VDLS

N

J

rPU

y

CR

E

7

o

I

f

O

z

Y

X

5

0

wM

H

Q

K

m

k

W

2

q

<latexi sh

1_b

64="Wk3jdEVm

Rog9

GCy

r

KF

>A

B

H

c

SwN

+

q

I

8

fY

v

Q

U

O/

MP

Z

u

7

n

D

0

T

5

p

<latexi sh

1_b

64="Wk3jdEVm

Rog9

GCy

r

KF

>A

B

H

c

SwN

+

q

I

8

fY

v

Q

U

O/

MP

Z

u

7

n

D

0

T

5

p

<latexi sh

1_b

64="v y803Mmun5wc

N

V

g

2

d

E

>A

B

X

DLS

O

r

fUY9

oP

K

I

WG

J

k

Z

R

C

/

HT

q

j

7

F

Q

+

p

z

<latexi sh

1_b

64="OU

RHQ

Cnf

B

zZoM

>A

7

c

VDLSgN

y

r

Y9

P

K

2

8

I

G

J

k

m

X

/w

3

T

q

vp

5

u

+

F

0

j

W

<latexi sh

1_b

64="

XyK

VC

Mu2G

UP

Ydock

w

>A

B9

DLSg

F

3

W

r

JN

z

H

O

Z

m

8

E

f/

0

Q

5

R

+

v

7

n

j

I

T

p

q

<latexi sh

1_b

64="oJwO0KX8D

7A

N

vRQL

pHP

>

B9

c

V

S

F

2

r

f

Zdu gq

T

mU

+

Y

WE

/

k

y

3

j

z

G

5

n

C

M

I

<latexi sh

1_b

64="IHNY

/

pGMk

7

d

Qw nS

J

>A

C0X

cjVL

FD2

r

f

Z

ug

R

y

Pq

5o

9

+

B

m

z

3

8

W

v

T

OE

K

U

<latexi sh

1_b

64="uo

O23z

y

mB

V5U

c

+Y

H

T0

>A

/

DLS

N

FJ

vq

G

W

RX

I

P

K

M

8

r

7j

g

d

w

Z

p

k

n

9

Q

<latexi sh

1_b

64="Wk3jdEVm

Rog9

GCy

r

KF

>A

B

H

c

SwN

+

q

I

8

fY

v

Q

U

O/

MP

Z

u

7

n

D

0

T

5

p

<latexi sh

1_b

64="

R

97+E0v

JG

U

X

fV

oq5Y

>A

C

H

cj

L

S

N

FD2

r

p

Zdugk

w

I

B

mn

y

M

/

8

T

z

O

K

W

3

P

Q

<latexi sh

1_b

64="v y803Mmun5wc

N

V

g

2

d

E

>A

B

X

DLS

O

r

fUY9

oP

K

I

WG

J

k

Z

R

C

/

HT

q

j

7

F

Q

+

p

z

<latexi sh

1_b

64="OU

RHQ

Cnf

B

zZoM

>A

7

c

VDLSgN

y

r

Y9

P

K

2

8

I

G

J

k

m

X

/w

3

T

q

vp

5

u

+

F

0

j

W

<latexi sh

1_b

64="

A

wNq

LDmy

FGn

p

8+

uoB

>

X

c

V

SgM

T

7W

d

Iv r2

j

K9

H

U

J

3

Z

k

C

/

0

Y

QO

z

5f

R

E

P

<latexi sh

1_b

64="K

Sk

Nw

p

o

f

VF

g

I

T3j

Y

>A

B8X

c

DL

M

9Z

W

Rn

y

Pr

d

0

m

O

E

G

/

u

C

Q

q7

v

2

5

U

H

J

+

z

<latexi sh

1_b

64="3QrI

VRJ

2

EHp

UC

Y

9

AkG

>

B8X

c

DLSgM

F

T

7W+q

d

v

y

j

K

o

NZ

/

0

m

O

zw

5f

n

P

u

<latexi sh

1_b

64="2ZcvAwNR

juO

HdK

o

WGDTVE

>

B/3

LS

FJ

rPU

y

C

n

7

I

f

z

Y

+

X

9

5

0

M

Q

m

kp

q

g

8

Representation

Representation

Representation

Representation

v

v

f

θ

f

θ

v

u

L

S

L

A

v

v

u

p

f

θ

f

ξ

u

u

Representation

Representation

v

L

V

f

θ

v

1

v

2

v

z

1

f

ξ

local neighborhoods of node v , and V -is the negative sample set that can be randomly sampled from the node space V or sampled proportionally to the power 3 / 4 of the degree of the node [14]. Figure 3 (a) illustrates this contrastive learning scheme with this explicit homophily assumption.

GCL with Augmented Views. This paradigm [4, 20, 39, 3] learns representations by contrasting views based on stochastic augmentations. Specifically, for node v , its representation in one augmented view is learned to be close to the representation of the same node v from the other augmented view and far away from the representations of negative samples from other nodes. Given two augmentations G 1 and G 2 extracted from original graph G in a predefined way, the contrastive objective is as follows:

Here v 1 = f θ ( G 1 )[ v ] and v 2 = f θ ( G 2 )[ v ] are projected representations of node v from two augmented views. V -is the set of negative samples of v from the inter- or intra-view [39]. Inspired by BYOL [40], non-contrastive BGRL [5, 41] predicts the target augmented view of the nodes: L V = -1 / |V| ∑ v ∈V z 1 · v 2 / ∥ z 1 ∥∥ v 2 ∥ , where z 1 = g ϕ ( v 1 ) is the prediction of the representation from the target augmented view. An illustration for this scheme is presented in Figure 3 (b).

4 Simple Asymmetric Contrastive Learning of Graphs

In this section, we elaborate on our GraphACL. The key idea behind GraphACL is encouraging the encoder to learn representations by simultaneously capturing one-hop neighborhood context and two-hop monophily, which generalizes the homophily assumption for modeling both homophilic and heterophilic graphs. Specifically, GraphACL introduces an additional predictor g ϕ which maps the graph G to node representations that can predict the one-hop neighborhood context from the representation of the central node as shown in Figure 3 (c). Importantly, unlike previous contrastive schemes, which force neighboring nodes to have similar representations, GraphACL directly predicts the original neighborhood signal of each node through an asymmetric design induced by a predictor. This simple asymmetric design allows neighboring nodes to have different representations but still can capture the one-hop neighborhood context and two-hop monophily as illustrated in Figure 2.

4.1 Graph Asymmetric Contrastive Learning

Based on the above motivation, for each node v , we first propose to learn its representation by capturing its one-hop neighborhood signal. A natural idea of capturing the neighborhood signal is learning the representations of v that can well predict the original features of v 's neighbors, i.e., x u of neighbor u . However, the original features are typically noisy and high-dimensional [42]. To solve this issue, we cast the prediction problem in the representation space, i.e., the representation of the central node should be predictive of representations of its neighbors. Specifically, we adopt the following simple prediction loss induced by an asymmetric predictor on neighbors:

where v = f θ ( G )[ v ] and u = f θ ( G )[ u ] are representations, and g ϕ ( · ) is a introduced predictor which maps the latent representation v to reconstruct the neighbor's representation u . Intuitively, in this case, each node is treated as a specific neighbor 'context', and nodes with similar distributions over the neighbor "context" are assumed to be similar. Here, we utilize only one single simple g ϕ to predict representations of all v 's neighbors and empirically find that it works very well.

This simple prediction objective in Equation (3) can preserve local neighborhood distribution in representations without homophily assumption since we do not directly enforce v and u to be similar to each other. To prevent g ϕ from degenerating to the identity function, we consider the representations of central node v and the one-hop neighbor u come from two decoupled encoders: v = f θ ( G )[ v ] and u = f ξ ( G )[ u ] , where f θ is the online identity encoder and f ξ is the preference target encoder. Importantly, the gradient of the loss in Equation (5) is only used to update the online encoder f θ , while being blocked in the target encoder f ξ as shown in Figure 3 (c). The weights of the target network ξ are updated using the exponential moving average of online network weights θ : ξ ← λξ +(1 -λ ) θ where λ ∈ [0 , 1] is the target decay rate. An intuitive illustration for using two encoders is that each node typically plays two roles: the node itself and a specific context of other nodes, corresponding to separated notions of node identity and node preference in real-world graphs. The key principle of decoupled encoders is to allow the identity representation of a node to be more possibly different from the identity representation with which the node prefers to link.

The simple objective in Equation (3) can capture the one-hop neighborhood context without relying on the homophily assumption or requiring graph augmentation as shown in Figure 2 (c). This matches our first motivation in the introduction part. Another remaining problem is why optimizing this simple objective helps capturing 2-hop node similarity, i.e., monophily. Consider a pair of 2-hop neighbors v and u 2 which both neighbor on the same node u . Intuitively, by enforcing v and u 2 to reconstruct the same context representation of neighborhood u , we implicitly make their representations similar. Thus, the 2-hop neighbors serve as positive pairs that will be implicitly aligned as shown in Figure 2 (d). We formally prove this intuition in Theorem 2 in our following theoretical analysis.

We note that some contrastive objectives based on augmentations such as BGRL [5] and BYOL [40] also utilize a predictor as shown in Figure 2 (b). However, the idea behind our loss in Equation (3) differs significantly from them. Specifically, they rely on data augmentations, and are built based on the idea that the augmentation can preserve the semantic nature of samples, i.e., the augmented samples have consistent semantic labels with the original ones. Nevertheless, unlike images, it is theoretically difficult to design graph augmentations without changing the semantic nature of nodes [7] and augmentations tend to capture homophily [20, 18]. In contrast, the simple loss in Equation (3) does not rely on augmentations but directly considers modeling one-hop neighborhood distribution. We theoretically prove that our GraphACL maximizes the mutual information between representations and one-hop neighborhood context, and captures two-hop monophily in § 5.

Although this simple neighborhood prediction objective can capture both one-hop neighborhood pattern and two-hop monophily, it may result in a collapsed and trivial encoder: all node representations degenerate to a the same single vector on the hypersphere. The main reason is that the prediction loss operates in a fully flexible latent space, and it can be minimized when the encoder produces a constant representation for all nodes. To address this issue, we introduce an explicit uniformity regularization to further enhance the representation diversity. Specifically, we add the following explicit regularization on representation uniformity into Equation (3):

where v = f θ ( G )[ v ] and v -= f θ ( G )[ v -] . Here, we consider the representations of negative samples v -coming from the same online encoder as central sample. This uniformity loss is typically approximated by randomly sampling K negative samples v -. Intuitively, minimizing this term will push all node representations away from each other and alleviate the representation collapse issue.

Graph Asymmetric Contrastive Loss . Directly combining Equations (3) and (4) arrives at a loss function: L COM = L PRE + L UNI. However, minimizing this combination loss is an ill-posed problem, it approaches -∞ as we can simply scale the norm of representations to reduce the loss. Even if the representations are normalized, minimizing L UNI still leads to a relatively poor performance as shown in our experiments. To address this issue, we instead minimize an upper bound of L COM, which results in the following simple objective of GraphACL (see Appendix A.1 for details):

Here prediction p = g ϕ ( v ) with v = f θ ( G )[ v ] , representations u = f ξ ( G )[ u ] and v -= f θ ( G )[ v -] . L A is a simple generalization of the graph contrastive loss with representation smoothing in Equation (1) from symmetric view to an asymmetric view. Note that when the predictor g ϕ becomes the

identity function, L A degenerates to the GCL loss L S . We demonstrate in extensive experiments that such asymmetric framework via a simple predictor helps achieve better downstream performance on both homophilic and heterophilic graphs than many GCL methods with prefabricated augmentations.

5 Theoretical Analysis

In this section, we provide theoretical understandings of GraphACL. We show that our simple GraphACL can simultaneously capture one-hop neighborhood context and two-hop monophily, which are important for heterophilic graphs. We also establish theoretical guarantees for the downstream performance of the learned representations. All proofs can be found in Appendix B.

Notations. We denote the random variable of node representations as V , and define the mean representations from the one-hop neighborhoods of node v as z = 1 N ( v ) ∑ u ∈N ( v ) u . This means that the representation z profiles the one-hop neighborhood pattern of node v . Since two nodes of the same semantic class tend to share similar neighborhood patterns in real-worlds, z can be viewed sampled from Z | Y ∼ N ( z Y , I ) where Y is the latent semantic class indicating the one-hop pattern of v . Given the above, we first demonstrate the rationality of GraphACL in capturing one-hop patterns:

Theorem 1. Minimizing GraphACL's objective in Equation (5) with exponential moving average is equivalent to maximizing mutual information between representation V and the one-hop pattern Y :

Theorem 1 indicates that minimizing GraphACL loss in Equation (5) promotes maximizing the mutual information I ( V ; Y ) between representations and one-hop neighborhood context. Next, we theoretically verify that our GraphACL can capture intuition about the two-hop monophily similarity.

Theorem 2. Let N 2 ( v ) denote the set of two-hop neighbors of v . Minimizing the GraphACL objective in Equation (5) is approximately minimizing the following alignment loss between two-hop neighbors:

where L is the L -bi-Lipschitz constant of g ϕ : ∀ ( v 1 , v 2 ) , 1 /L ∥ v 1 -v 2 ∥ 2 2 ≤ ∥ g ϕ ( v 1 ) -g ϕ ( v 2 ) ∥ 2 .

The above theorem shows that minimizing the GraphACL objective implies a small alignment loss of two-hop neighbors. Thus, GraphACL not only captures the one-hop neighborhood pattern, but also the two-hop monophily implicitly, i.e., encouraging a large similarity between two-hop neighbors.

Next, we theoretically show that the learned representations can achieve better downstream performance. We evaluate the representation by its performance on a multi-class classification task using the mean classifier [43, 44, 45]. Specifically, we consider the classifier p W ( v ) = arg max Wv , where v = f θ ( G )[ v ] and W ∈ R C × D is the weight of the linear classification head. The y th row of W is the mean µ y of representations of nodes with the label y : µ y = E v | y [ v ] . For simplicity, we assume that the node classes are balanced but our results can be easily extended to unbalanced settings by considering a label shift term as shown in the domain adaptation literature [46].

Theorem 3. Let ˆ h 2 = H ( G 2 ) be the homophily ratio of the two-hop graph G 2 . Suppose that the downstream task is the M-categorical linear classification and the class is balanced. Then, ∀ q ∈ the hypothesis class with q = g ◦ f , the upper bound for the classification error on the optimal v ∗ is :

̸

This theorem says that the downstream error on representations is bounded by GraphACL loss and homophily ratio ˆ h 2 of the two-hop graph G 2 , i.e, monophily. Specifically, a larger monophily ratio ˆ h 2 would provably imply a smaller downstream classification error. Prior work [15] and the statistics of real-world graphs in Appendix C show that although the one-hop graph may be heterophily-dominant, the two-hop graph will always be homophily-dominant ( ˆ h 2 is large). This theorem reveals why the simple GraphACL enjoys good performance on both homophilic and heterophilic graphs.

6 Experiments

6.1 Experimental Settings

Datasets and Splits. We conduct experiments on both homophilic and heterophilic graphs. For heterophilic graphs, we adopt Wisconsin, Cornell, Texas [33], Actor, Squirrel, Crocodile, and

Chameleon [33, 47]. We also use two large heterophilic graphs proposed recently: Roman-empire (Roman) [48] and arXiv-year [16] ( ∼ 170k) nodes. For homophilic graphs, we adopt three citation graphs: Cora, Citeseer and Pubmed [49, 50], and two co-purchase graphs: Computer and Photo [5, 51]. We further include a large-scale homophilic graph Ogbn-Arxiv (Arxiv) [52]. For all datasets, we use the public and standard splits used by cited papers. Detailed descriptions, splits, and one-hop homophily and two-hop monophily statistics of datasets are given in the Appendix C.1.

Baselines. We compare GraphACL with a traditional network embedding method: LINE [14], and the recent state-of-the-art self-supervised learning methods: VGAE [9], DGI [2], GCA [4], Local (L)-GCL [12], HGRL [29], BGRL [5], CCA-SSG [6], SP-GCL [53], and DSSL [18]. The descriptions and implementation details of baselines are given in Appendix C.2.

Evaluation Protocol. We utilize node and graph classification and node clustering to evaluate the quality of the representation. For classification, we follow the standard linear evaluation protocol [2]. We train a linear classifier on top of the frozen representation and report the test accuracy. For node clustering, we perform k-means clustering on the obtained representations and set the number of clusters to the number of ground truth classes and report the normalized mutual information [54].

Setup. For all methods, we use a standard GCN model [49] as the encoder. We randomly initialize the model parameters and use the Adam optimizer to train the encoder. We run experiments with ten random seeds and report the average performance and standard deviation. For fair comparison, for all methods, we select the best configuration of hyperparameters only based on accuracy on the validation set. For baselines that did not report results in part of datasets or do not use standard public data splits [18, 29], we reproduce the results using the official code of the authors. More details on the implementation and the hyperparameter search space can be found in Appendix C.3.

6.2 Overall Performance Comparison

Table 1 reports the average node classification accuracy on both heterophilic and homophilic graphs. We provide the graph classification and node clustering results in Appenxi C.6 and C.5, respectively, showing that GraphACL can also effectively adapt to various downstream tasks. Surprisingly, among all methods, our simple GraphACL achieves the best performance in 14 of 15 data sets with various homophily ratios, as shown in Table 1. Specifically, GraphACL achieves significant improvements on most of the heterophilic datasets and comparable performance on homophilic graphs compared with the second-best method. Specifically, GraphACL achieves 4.3% (Roman), 11.6% (Cornell), 14.4% (Texas), 5.1% (Crocodile), and 3.1% (Arxiv-year) relative improvement over the second-best method. We can observe that contrastive strategies (CCA-SSG, GCA, and BGRL) with augmentations can not work well on heterophilic graphs compared to homophilic graphs. This verifies that they still implicitly leverage homophily. In contrast, GraphACL does not need augmentations, and we attribute our significant improvement to modeling the one-hop neighborhood context and two-hop monophily.

6.3 Ablation Study and Sensitivity Analysis

Ablation Study. We conduct an ablation study in Table 2 to validate our motivation and design, with the following three ablations: (i) directly minimize the combination loss L COM, (ii) Removing the

<latexi sh

1_b

64="nKmfocFW39

CTNBvMU

0

SRL

>A

H

VD

g

EOy

r

Y

P

2

Q

I

j

7

Z

X

k

8

q

+

u

G

d

/

w

z

5

J

p

<latexi sh

1_b

64="nKmfocFW39

CTNBvMU

0

SRL

>A

H

VD

g

EOy

r

Y

P

2

Q

I

j

7

Z

X

k

8

q

+

u

G

d

/

w

z

5

J

p

(a) Sensitivity of representation dimension

D

D

asymmetric encoder architecture, (iii) Removing the uniformity loss with negative samples. We also test the combination (ii) & (iii) by removing both asymmetric architecture and uniformity loss. First, minimizing the ill-posed combination loss is a valid baseline, but can not achieve better performance and is unstable with large standard deviations. We also find that the model without uniformity loss (ablation ( ii )) does not achieve the best in homophily graphs, although it does still serve as a strong ablation. When each component is individually applied, the asymmetric architecture alone achieves the very best performance in Cornell and arXiv-year. This confirms our motivation that the asymmetric architecture is of more importance for modeling neighbors in heterophilic graphs where connected nodes have different classes. The full model (last row) achieves the best performance, demonstrating that our designed components are complementary to each other.

Representation Dimension. In Figure 4 (a), we analyze the effect of the dimension of node representations. We can observe that having a large dimension can generally lead to better performance for both homophilic and heterophilic graphs. Moreover, we find that the dimension effect is more obvious in heterophilic graphs

compared to it in homophily graphs. This could be justified by our Theorems 3, which give the lower bound and upper bound of GraphACL loss in terms of the homophily ratio. Specifically, since the two-hop homophily ratio is still smaller in heterophilic graphs compared to homophilic graphs, and a small downstream error requires a small loss. Thus, a large dimension can effectively reduce the lower bound of the training loss and benefit the learning on heterophilic graphs. Training with extremely large dimensions for some graphs may lead to a slight drop of performance as GraphACL may suffer from the over-fitting issue, limiting its generalization performance.

Other Hyper-parameters. Figure 5 shows the performance with various decay rate λ . We observe that (i) λ plays an important role in GraphACL. Having a large λ typically improves the model performance; (ii) Instantaneously updating the target network, i.e., when λ = 0 , destabilizes training and leads to poor performance. Thus, there is a trade-off between updating the target too often and updating it too slowly. The temperature τ and the number of negative samples K are not included here since they are not directly relevant to our key motivation and have been employed and evaluated in the recent works [55, 56]. Thus, we provide corresponding results in Appendices C.8 and C.9.

6.4 Qualitative Analysis and Case Study

Similarity Visualization. Figures 4 (b) and (c) plot the distribution of pair-wise cosine similarities of randomly sampled nodes, one-hop neighbor, and two-hop neighbor pairs based on learned representations. We can observe that, for the homophilic graph, that is, Cora, nodes are forced to have representations similar to those of their neighbors. GraphACL enlarges the similarities between neighbor nodes compared to randomly sampled node pairs, demonstrating that GraphACL can well preserve one-hop neighborhood contexts. Compared to the homophilic graph Cora, GraphACL seeks to further pull the two-hop neighbor nodes together on the heterophilic graph, i.e., Squirrel. This observation matches our motivation and analysis that GraphACL can effectively discover the underlying two-hop monophily, which benefits the learning on heterophilic graphs. Figures 6 (a) and (b) show the cosine similarity between the v and prediction g ϕ ( v ) for each node. We find that the cosine similarities are typically smaller than 1, showing that the predictor g ϕ is not an identity matrix after convergence. Moreover, we can observe that, compared to the homophilic graph Cora, the

Node ID 2329

Node ID 258

similarities on the heterophilic graph Squirrel are smaller. This verifies that GraphACL automatically differentiates node identity and context representations, which is important for heterophilic graphs.

Case Study. In Figures 6 (c) and (d), we randomly sample the two-hop subgraph of a central node and calculate the cosine similarity based on learned representations between the central node and neighbor nodes. We can observe that GraphACL can successfully identify the node whose local neighborhood patterns are similar to the central node on both homophilic and heterophilic graphs. Additionally, our GraphACL can pull two-hop neighbor nodes, which share a similar semantic class to the central nodes, instead of favoring nearby one-hop neighbor nodes. These observations endorse our intuition that heterophilic graphs can not benefit much from one-hop neighbor nodes, and our GraphACL can effectively capture the latent semantic information related to two-hop monophily.

7 Conclusions

We propose a simple contrastive learning framework named GraphACL for homophilic and heterophilic graphs. The key idea of GraphACL is to capture both a local neighborhood context of one hop and a monophily similarity of two hops in one single objective. Specifically, we present a simple asymmetric contrastive objective for node representation learning. We provide a theoretical understanding of GraphACL, which shows that GraphACL can explicitly maximize mutual information between representations and one-hop neighborhood patterns. We show that GraphACL also implicitly aligns the two-hop neighbors and enjoys a good downstream performance. Extensive experiments on 15 graph benchmarks show that GraphACL significantly outperforms state-of-the-art methods. We hope that the straightforward nature of our approach serves as a reminder to the community to reevaluate simpler alternatives that may have been overlooked, thus inspiring future research.

Acknowledgments

This work supported by, or in part by, the National Science Foundation (NSF) under grant number IIS-1909702, the Army Research Office (ARO) under grant number W911NF21-1-0198, Department of Homeland Security (DNS) CINA under grant number E205949D, and Cisco Faculty Research Award. The findings in this paper do not necessarily reflect the view of the funding agencies.

References

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Petar Veliˇ ckovi´ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR , 2018.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurIPS , pages 5812-5823, 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In The Web Conference , pages 2069-2080, 2021.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veliˇ ckovi´ c, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR , 2021.

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. NeurIPS , 2021.

Puja Trivedi, Ekdeep S Lubana, Mark Heimann, Danai Koutra, and Jayaraman Thiagarajan. Analyzing data-centric properties for graph contrastive learning. NeurIPS , 2022.

Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. arXiv preprint arXiv:2109.01116 , 2021.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. ArXiv , 2016.

William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584 , 2017.

Hao Zhu, Ke Sun, and Peter Koniusz. Contrastive laplacian eigenmaps. NeurIPS , 2021.

Hengrui Zhang, Qitian Wu, Yu Wang, Shaofeng Zhang, Junchi Yan, and Philip S Yu. Localized contrastive learning on graphs. arXiv preprint arXiv:2212.04604 , 2022.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD , pages 855-864, 2016.

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In WWW , pages 1067-1077, 2015.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. NeurIPS , pages 7793-7804, 2020.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. NeurIPS , pages 20887-20902, 2021.

Mingyue Tang, Pan Li, and Carl Yang. Graph auto-encoder via neighborhood wasserstein reconstruction. In ICLR , 2021.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self-supervised learning for non-homophilous graphs. arXiv preprint arXiv:2206.03601 , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. In NeurIPS , 2022.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In ICLR , 2020.

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In AAAI , 2021.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In ICLR , 2021.

Kristen M Altenburger and Johan Ugander. Monophily in social networks introduces similarity among friends-of-friends. Nature human behaviour , pages 284-290, 2018.

Alex Chin, Yatong Chen, Kristen M. Altenburger, and Johan Ugander. Decoupled smoothing on graphs. In WWW , 2019.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In KDD , pages 701-710, 2014.

William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS , 2017.

Jingfan Chen, Guanghui Zhu, Yifan Qi, Chunfeng Yuan, and Yihua Huang. Towards selfsupervised learning on graphs with heterophily. In CIKM , pages 201-211, 2022.

Shashank Pandit, Duen Horng Chau, Samuel Wang, and Christos Faloutsos. Netprobe: a fast and scalable system for fraud detection in online auction networks. In WWW , 2007.

Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. In AAAI , 2021.

Susheel Suresh, Vinith Budde, Jennifer Neville, Pan Li, and Jianzhu Ma. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. In KDD , 2021.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In ICLR , 2019.

Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In KDD , pages 1894-1903, 2021.

Huaisheng Zhu, Guoji Fu, Zhimeng Guo, Zhiwei Zhang, Teng Xiao, and Suhang Wang. Fairnessaware message passing for graph neural networks. arXiv preprint arXiv:2306.11132 , 2023.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. In NeurIPS , 2022.

Zhimeng Guo, Teng Xiao, Charu Aggarwal, Hui Liu, and Suhang Wang. Counterfactual learning on graphs: A survey. arXiv preprint arXiv:2304.01391 , 2023.

Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In WSDM , 2018.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 , 2020.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS , 2020.

William Shiao, Zhichun Guo, Tong Zhao, Evangelos E Papalexakis, Yozen Liu, and Neil Shah. Link prediction with non-contrastive learning. arXiv preprint arXiv:2211.14394 , 2022.

Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In CIKM , pages 1202-1211, 2021.

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In ICML , pages 9904-9923, 2019.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. Arxiv , 2022.

Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. NeurIPS , 35, 2022.

Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. In NeurIPS , 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 2021.

Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? In ICLR , 2023.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , pages 93-93, 2008.

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In SIGIR , pages 43-52, 2015.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS , pages 22118-22133, 2020.

Haonan Wang, Jieyu Zhang, Qi Zhu, and Wei Huang. Can single-pass contrastive learning work for both homophilic and heterophilic graph? arXiv preprint arXiv:2211.10890 , 2022.

Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In ICML , 2009.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

Jun Xia, Lirong Wu, Ge Wang, Jintao Chen, and Stan Z Li. Progcl: Rethinking hard negative mining in graph contrastive learning. In ICML , pages 24332-24346, 2022.

Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV , pages 548-564, 2020.

Meihong Wang and Fei Sha. Information theoretical clustering via semidefinite programming. In AISTATS , pages 761-769, 2011.

Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NeurIPS , 2022.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. NeurIPS , 34:5000-5011, 2021.

Minjie Yu Wang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop on representation learning on graphs and manifolds , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , pages 15750-15758, 2021.

A The Omitted Details of GraphACL

A.1 The Loss of Graph Asymmetric Contrastive Learning

We provide details of the derivations of GraphACL in Equation (5) and show that minimizing GraphACL is approximately to minimize the combination of prediction and uniformity losses.

where the symbol c = indicates equality up to a multiplicative and/or additive constant. Here, we utilize Jensen's inequality in Equation (10). Equation (11) holds because p and u are both ℓ 2 -normalized. Given the above, we can conclude that minimizing GraphACL objective is approximately to minimizing the combination of the prediction and uniformity losses.

B Proofs in Section 5

B.1 Proof of Theorem 1

Theorem 1. Minimizing GraphACL's objective in Equation (5) with exponential moving average is equivalent to maximizing mutual information between representation V and the one-hop pattern Y :

Proof. According to the derivations in Appendix A.1, we have:

where z = 1 N ( v ) ∑ u ∈N ( v ) u is the mean representations from one-hop neighborhoods of node v . Equation (13) holds because that gradient of u is being blocked in the target encoder with exponential moving average; thus it can be viewed as a constant that depends only on the target encoder but not the variable f θ . Since two nodes of the same semantic class tend to share similar one-hop neighborhood patterns even in non-homophilous graphs, thus we can view the neighborhood representations z is

sampled from the conditional distribution given the pseudo label Y which indicates the one-hop neighborhood pattern: Z | Y ∼ N ( z Y , I ) . As demonstrated in [57], we can interpret the first term as a conditional cross-entropy between V and another random variable Z whose conditional distribution given the pseudo-label Y indicates the one-hop neighborhood pattern: Z | Y ∼ N ( z Y , I ) :

where H ( · ) and D KL ( · ) denote the entropy and KL-divergence, respectively. Minimizing the first term in Equation (14) is approximately minimizing H ( V | Y ) . We then inspect the second term in Equation (14). As shown in [58, 57], the second term is close to the differential entropy estimator:

As a result, minimizing the loss of GraphACL can be seen as a proxy for maximizing the mutual information between the representations V and the pseudo-labels Y which indicates the neighborhood patterns of one hop nodes. Thus, the proof is completed.

B.2 Proof of Theorem 2

Theorem 2 Let N 2 ( v ) denote the set of two-hop neighbors of v . Minimizing the GraphACL objective in Equation (5) is approximately minimizing the following alignment loss between two-hop neighbors:

where L is the L -bi-Lipschitz constant of g ϕ : ∀ ( v 1 , v 2 ) , 1 /L ∥ v 1 -v 2 ∥ 2 2 ≤ ∥ g ϕ ( v 1 ) -g ϕ ( v 2 ) ∥ 2 .

Proof. Given the derivations in Appendix A.1, we have:

where the last line holds because v ∈ V : ∑ v -∈V exp ( v ⊤ v -/τ ) > exp( v ⊤ v ) = e > 1 . To conduct the proof, we first formulate the degree-related predictions and representations as two matrices: ˜ P and ˜ U . Here, ˜ U v 1 = √ d u u is the u -th row of the matrix ˜ U ∈ R N × D and ˜ P v = √ d v · g ϕ ( v ) is the v -th row of the matrix ˜ P ∈ R N × D . Recall that the normalized adjacency matrix ˜ A = D -1 / 2 AD -1 / 2 where D is the diagonal degree matrix. With the above and Equation (18), we have:

where d m is the maximum degree of nodes and tr ( · ) denotes the matrix trace. In Equation (19), we utilize the inequality: tr( PQ ) ≤ 1 2 ( ∥ P ∥ 2 2 + ∥ Q ∥ 2 2 ) for any two matrices P ∈ R m × n and Q ∈ R n × t . Equation (20) holds as u is normalized. Since the trace is the sum of elements on the main diagonal

of the matrix, the v -th diagonal value of ( ˜ P ˜ P ⊤ ˜ A ⊤ ˜ A ) is:

where Equation (21) holds is because the decoder is L -bi-Lipschitz: ∀ ( x 1 , x 2 ) , 1 /L ∥ x 1 -x 2 ∥ 2 ≤ ∥ ∥ g ϕ ( x 1 ) -g ϕ ( x 2 ) ∥ ∥ 2 . Combining Equations (18), (20) and (21), we have:

where min θ f 1 ⇒ min θ f 2 represents that minimizing function f 1 with respect to θ is equivalent to maximizing function f 2 with respect to θ . Thus, the proof is completed.

B.3 Proof of Theorem 3

Theorem 3 Let ˆ h 2 = H ( G 2 ) be the homophily ratio of the two-hop graph G 2 . Suppose that the downstream task is the M-categorical linear classification and the class is balanced. Then, ∀ q ∈ the hypothesis class with q = g ◦ f , the upper bound for the classification error on the optimal v ∗ is:

̸

Proof. We define the adjacency matrix of the two-hop graph of G as A 2 , i.e., ( A 2 ) ik = 1 if there exists node j such that e ij ∈ E and e jk ∈ E , otherwise ( A 2 ) ik = 0 . The normalized form of A 2 is ˆ A 2 = ˆ D -1 / 2 A 2 ˆ D 2 where ˆ D 2 is the diagonal degree matrix of the two-hop graph. N 2 ( v ) is the set of the neighbors of v on the two-hop graph. We also define the label matrix of all nodes as Y ∈ R N × M and the representation matrix V ∈ R M × D with the v -th node representation as V v = v . Let c i be the number of nodes belonging to class i . For any class i , we have c i = c = |V| M since we assume the ideal balanced setting for simplicity. However, our theoretical results can be easily extended to unbalanced settings by explicitly considering the shift of label distributions [46]. Given the above definitions, we have the following:

where C v,i = c -1 i ✶ y v = i and ✶ is the indicator function. y v is the label index of node v and y v is the one-hot vector of label y v . The last line holds because ∥ P + Q ∥ 2 2 < 2 ∥ P ∥ 2 2 +2 ∥ Q ∥ 2 2 for any

two matrices P and Q of the same dimension. The ( v, i ) -th elements of matrices Y and A 2 C are: Y v,i = ✶ y v = i and ( A 2 C ) v,i = ∑ u 2 ∈N 2 ( v ) c -1 i y u 2 = i . If y v = i , we have:

Similarly, for y v = i , we have:

̸

Given the above, we further have:

̸

̸

For the second term ∥ A 2 C -VW ⊤ ∥ 2 2 in Equation (24), we have:

where Equation (29) holds because the i th row of W is the mean of representations of nodes with label i : ∥ ( V ⊤ C -W ⊤ ) ∥ 2 2 = 0 . Combining Equations (24), (27), and (30), we have:

̸

̸

̸

̸

̸

̸

where ˆ h 2 = H ( G 2 ) is the homophily ratio of the two-hop graph. Based on the derivations in Theorem 2 and Equation (9), we have:

We can find the last line is exactly the SimCLR-style loss [55] over two-hop graphs. Recent work [59] proves that finding the global optimum of the un-normalized SimCLR-style objective L SimCLR ( V ) is equivalent to solving the matrix factorization problem: min V ∥ A 2 -VV T ∥ 2 2 . Given Equations (31) and (33), we further have:

̸

Here, we convert the mean-squared error bound to classification error bound in Equation (34) as shown by Claim B.9 in [60]. The α is the constant indicating the gap between the optimal loss value of SimCLR-style and matrix factorization losses [59] and β = α +4 .

C Experimental Details

C.1 Datasets Details and Statistics

C.1.1 One-hop Homophily Levels

We utilize the following edge homophily ratio [15] to measure the one-hop neighbor homophily of the graph. Specifically, the edge homophily ratio H ( G ) is the proportion of edges that connect two nodes of the same class:

C.1.2 Neighborhood Context Similarity

To justify our first intuition that two nodes of the same semantic class tend to share similar one-hop neighborhood patterns even in heterophilic graphs, we consider if the same label shares similar one-hop neighborhood distributions of labels in the neighborhoods regardless of the homophily. To measure this property, we calculate the class neighborhood similarity [24] defined as follows:

where M is the number of classes, V m is the set of nodes with class m , and d ( u ) is the empirical label histogram of node u 's neighbors over M classes. cos ( · ) represents the cosine similarity function. This cross-class neighborhood similarity measures the neighborhood distributions between different classes. When m = m ′ , s ( m,m ′ ) calculates the intra-class similarity. To measure the neighborhood similarity, we average the intra-class similarities of all classes:

If nodes with the same label share the same neighborhood distributions, the class neighborhood similarity H s ( G ) is high.

C.1.3 Two-hop Monophily Levels

To further verify our second intuition in the introduction: two-hop similarities (monophily) still exist even without the one-hop homophily assumption, we also consider monophily properties of two-hop neighborhoods. Following [16], we use the following two-hop homophily to measure the monophily, where the neighborhood of each node is defined to be the nodes of exactly two hops away.

where N 2 ( v ) denotes the set of two-hop neighborhoods of v .

The statistics of datasets, including their one-hop and two-hop homophily levels and neighborhood similarities, are given in Table 3. We can observe both homophilic and heterophilic graphs exhibit strong neighborhood similarity calculated based on one-hop neighborhoods. In addition, the two-hop homophily level is generally higher than the one-hop homophily level in heterophilic graphs. This observation confirms our intuition that two-hop similarities still exist without a strong one-hop homophily level. The descriptions of the datasets are given below:

Cora, Citeseer, and Pubmed [49]. They are among the most widely used node classification benchmarks. Each dataset is a citation and high-homophily graph, where nodes are documents, and edges are citation relationships from one to another. The class label of each node is based on the research field. A bag of words of its abstracts is used as the features of nodes. The public split is used for these datasets, where each class has fixed 20 nodes for training, another fixed 500 nodes and 1000 nodes for validation/test, respectively for evaluation.

Computer and Photo [5, 51]. They are co-purchase graphs from Amazon, where nodes represent products and frequently bought products are connected by edges. Node features represent product reviews, and class labels indicate the product category. Following the experiment settings from [12], we randomly split the nodes into train/validation/test (10%/10%/80%) sets.

For these homophilic datasets, we utilized the process version of them provided by Deep Graph Library [61]. These datasets can be found in https://docs.dgl.ai/en/0.6.x/api/python/ dgl.data.html .

Ogbn-arxiv (Arxiv) [52]. It is a citation network between all Computer Science (CS) arXiv papers. Each node represents one paper and each edge indicates the citation relationships between two papers. The feature of each node is obtained by a 128-dimensional feature vector through averaging the embeddings of words in its title and abstract. The embeddings of words are obtained by running the skip-gram over the MAG corpus. Following [12], we also use 10%/10%/80% split for this dataset.

Texas, Wisconsin and Cornell [33]. They are webpage networks collected from computer science departments of different universities by Carnegie Mellon University. For each network, nodes are

web pages and edges indicate hyperlinks between web pages. Node features are bag-of-words representations of web pages. The task is to classify nodes into five categories: student, project, course, staff, and faculty.

Chameleon, Crocodile and Squirrel [47]. They are Wikipedia networks, where nodes represent web pages and edges represent hyperlinks between them. Features of nodes are several informative nouns on Wikipedia pages. Labels of nodes are based on the average daily traffic of the web page.

Actor [33]. It is an actor-only induced subgraph of the film-director-actor-writer network. Nodes correspond to actors and edges represent the co-occurrence of two nodes on the same Wikipedia page. Node features are keywords in Wikipedia pages. Labels are assigned five categories in terms of words on the actor's Wikipedia.

For Texas, Wisconsin, Cornell, Chameleon, Crocodile, Squirrel and Actor , we use the raw data provided by the Geom-GCN [33] with the standard fixed 10-fold split for our experiment. These datasets can be downloaded from: https://github.com/graphdml-uiuc-jlu/geom-gcn .

Roman-empire (Roman) [48] is a heterophilous graph based on the Roman Empire article in English Wikipedia. Each node in the graph corresponds to one (non-unique) word in the text. The node features are from word embeddings. The class of a node is its syntactic role (the 17 most frequent roles as unique classes and all the other roles are grouped into the 18th class). Following [48], we utilize the fix 10 random 50%/25%/25% train/validation/test splits. This dataset can be found in https://github.com/yandex-research/heterophilous-graphs .

Arxiv-year [16] is a citation network from a subset of the Microsoft Academic Graph, where the task focuses on predicting the year that a paper is posted. Nodes are papers, and edges are relevant to citations. The node features correspond to the average of word embeddings of the title and abstract of the papers. Following [16], 50%/25%/25% train/val/test split is utilized for this dataset. This dataset can be found in https://github.com/CUAI/Non-Homophily-Large-Scale .

C.2 Baselines

LINE [14]: This network embedding model proposed an objective function to preserve both the first-order and second-order proximities.

VGAE [9]: VGAE is a generative model based on variational autoencoder for node representation learning by directly reconstructing adjacency matrix.

DGI [2]: It is a self-supervised learning method that maximizes the mutual information between node representations and graph summary.

GCA [4]: This is a graph contrastive representation learning method with adaptive augmentation that incorporates various priors for topological and semantic aspects of the graph.

BGRL [5]: This model is a graph contrastive learning method, where the alternative augmentations of the graph are predicted to learn representations of nodes.

CCA-SSG [6]: CCA-SSG is a graph contrastive learning model, which encourages the learned representations of nodes by reducing the correlation between different views.

L-GCL [12]: It is a self-supervised node representation learning method, which samples positive samples from the first order neighborhoods and kernelizes negative loss to reduce the training time.

SP-GCL [53]: This is a single-pass graph contrastive learning method based on the concentration property of node representations.

HGRL [29]: This is a self-supervised representation learning framework on graphs with heterophily, which leverages the node original features and the high-order information.

DSSL [18]: This method introduces a representation learning framework by decoupling the diverse neighborhood context of a node in an unsupervised manner.

C.3 Setup and Hyper-parameter Settings

We use official implementation publicly released by the authors on Github of the baselines. For fair comparison, we used grid search to find the best hyperparameters of the baselines. We run

experiments on a machine with a NVIDIA RTX A6000 GPU with 49GB of GPU memory. In all experiments, we use the Adam optimizer [62]. A small grid search is used to select the best hyperparameter for all methods. In particular, for our GraphACL, we search λ from {0, 0.90, 0.95, 0.97, 0.99, 0.999, 1}, D from {256, 512, 1024, 2048, 4096, 8192}, τ from {0.25, 0.5, 0.75, 0.99, 1}, and K from {1, 5, 10} when we utilize the negative sampling. We tune the learning rate over {0.001, 0.0005, 0.0001} and weight decay over {0, 0.0001, 0.0003, 0.000001}. We select the best configuration of hyper-parameters based on average accuracy only based on the validation set.

C.4 The Spectrum Changes in Augmented Views

In Figure 7, we categorize the eigenvalues (spectrum) of the symmetric normalized graph Laplacian into five distinct frequency bands, ranging from low to high. Subsequently, we computed the average spectrum changes across these frequency bands for two augmented views. The perturbations resulting from augmentations reveal a consistent trend in the graph spectrum. After augmenting the graph, we can observe that the perturbations in the low-frequency components (far left) are comparatively smaller than those observed in the high-frequency components (far right) for both homophilic (PubMed) and heterophilic (Squirrel) graphs. This observation confirms that mainstream GCL methods can not work well on heterophilic graphs, particularly when the perturbation in high-frequency signals is more significant.

C.5 Node Clustering Performance

We also conduct node clustering to evaluate the quality of the learned node representations. Specifically, we obtain node representation with GraphACL, then perform k-means clustering on the obtained representations and set the number of clusters to the number of ground truth classes. The experiment is conducted five times. We report the average normalized mutual information (NMI) for clustering in Table 4 on both homophilic and heterophilic graphs. From the table, we can observe that our GraphACL can consistently improve node clustering performance compared to the state-of-the-art self-supervised learning baselines on eight datasets. This observation, along with the results of the node classification, demonstrates the effectiveness of GraphACL in learning more expressive and robust node representations for various downstream tasks. These results further validate that

modeling one-hop neighborhood pattern and two-hop monophily similarity benefit downstream tasks on real-world graphs with various homophily ratios.

C.6 Graph Classification Performance

Existing works for heterophilic graphs typically focus on node-level tasks (node classification and node clustering). Thus, an empirical comparison of the graph-level task is relatively not straightforward. Nevertheless, for graph classification, we can use a non-parameterized graph pooling (readout) function, e.g., MeanPooling, to obtain the graph-level representation. For graph classification, we conduct experiments on three graph classification benchmarks: MUTAG and IMDB-B, and follow the same setting in GraphCL [3]. The results are shown in Table 5. From the table, we can observe that our ACL framework can also work well on the graph classification task and still can achieve better (or competitive) performance compared to baselines.

C.7 The Effect of Target Decay Rate

Figures 8 and 9 show the results of node classification with varying decay rate λ on four other homophilic and four heterophilic graphs, respectively. We can still observe that having large values of λ improves the overall performance. We observe that GraphACL obtains a competitive but not the best result when λ = 0 , which confirms that slowly updating the target network is crucial in obtaining superior performance. We also notice that we can achieve good performance even with a random target encoder, i.e., λ = 1 on some datasets, which can be explained by the recent work [63] that shows only using the simple stop-gradient operation can sometimes prevent collapsing.

C.8 The Effect of Temperature

Figures 10 and 11 show the results of node classification with variable temperature τ in homophilic and heterophilic graphs. We can observe that our GraphACL is not very sensitive to temperature τ on heterophilic graphs, while moderate hardness of the softmax (large τ ) produces the best result. For homophilic graphs, the large or small temperature can lead to poor performance.

C.9 The Effect of the Number of Negative Samples

We run a sweep over the size of negative samples K to study how it affects performance. We vary K as { 1 , 5 , 10 } . For each K , we first learn node representation and then use the learned node representation for node classification. Figure 12 shows the results with varying K . From the figure, we can observe that a small number of negative samples (e.g., K = 5 ) is enough to achieve good performance on all graphs. For homophilic graphs, we can observe that large K can promote the

Citeseer (id 2483)

Computer (id 2330)

Texas (id 108)

Cornell (id 106)

Crocodile (id 7849)

Wisconsin (id 83)

Pubmed (id 19803)

performance of GraphACL. In contrast, for heterophilic graphs, training with large K will lead to a slight drop in performance. A possible reason is that the randomly sampled negative samples can not represent the whole node-set, given the heterogeneous and diverse patterns of heterophilic graphs.

C.10 More Similarity Histograms of Node Pairs

Figure 13 shows the additional results on the representation similarity. The observations are generally similar to the results in the main body of the paper. As shown by the figure, we can observe that the randomly sampled node pairs are easier to be distinguished from one-hop and two-hop neighbors based on the representation similarity, which demonstrates that our GraphACL indeed captures the semantic meaning of node and will desirably push away the semantically dissimilar nodes. Moreover, the two-hop similarities in heterophilic graphs are much larger than that in homophilic graphs. These phenomena provide explanations for why GraphACL achieves good performance by capturing two-hop monophily structure information. Figure 14 shows the cosine similarity between the v and prediction g ϕ ( v ) for each node on other graphs. We can find that the cosine similarities are typically smaller than 1, which shows that the predictor g ϕ is not an identity matrix after convergence. Moreover, we can observe that, compared to the homophilic graph Cora, the similarities on the heterophilic graph Squirrel are smaller. Since the objective of GraphACL will pull p = g ϕ ( v ) and u together, thus GraphACL can automatically differ identity representation v and preference representation u , which is important for modeling heterophilic graphs. For the homophilic graph

Cora, the node identity representation v and the preference representation u should be similar, which is also captured by our simple contrastive objective in GraphACL.

C.11 Additional Case Study Results

In Figure 15, we provide the additional case studies and visualization results on other graphs. We can find that, in most cases, nodes sharing the same semantic classes with the central nodes have larger similarities to the central nodes. This observation interprets the reason why GraphACL can achieve good performance. We observe that GraphACL can effectively capture the local neighborhood pattern and two-hop monophily for both homophilic and heterophilic graphs, which empirically verifies our theoretical analysis given in the main body of the paper.

D Societal Impacts and Limitations

There are numerous graphs in the real world that exhibit heterophilic properties, such as transaction networks, ecological food networks, and molecular networks, in which the connected nodes possess dissimilar features and distinct class labels. Given the successful deployment of Graph Neural Networks (GNNs) in various human-related real-world applications, including social networks, knowledge graphs, and molecular property prediction, it is crucial to propose unsupervised representation learning techniques for GNNs that can effectively handle both homophilic and heterophilic graphs, thereby potentially yielding direct social impacts.

The collection of labeled data is often expensive and impractical, particularly in domains requiring specialized knowledge, such as medicine and chemistry. Considering the potential positive impact, we believe that our work can assist researchers and practitioners in devising solutions that alleviate the reliance on labeled data. However, it is important to acknowledge the potential negative consequences, as these learned representations could also be exploited for malicious purposes, such as adversarial attacks on GNNs that exploit systematic biases. Nonetheless, we believe that our simple framework and theoretical insights derived from this work contribute as a small step towards advancing the simplicity and generalizability of graph contrastive learning models within the research community.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an asymmetric contrastive learning framework for the homophilic and heterophilic graphs, which does not rely on graph augmentations and homophily assumptions. The theoretical analysis and empirical results further support the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper is easy to understand and the motivation is clear and reasonable.\n2. The proposed method is simple yet effective. The authors further demonstrate the effectiveness of the proposed method from the information theory and downstream tasks perspectives. Such results give a theoretical understanding of this work.\n3. Extensive experimental results including comparison experiments, ablation study, visualization, and case study empirically verify the effectiveness of the proposed method. Moreover, the proposed method shows superior performance on both the  homophilic and heterophilic graph datasets.\n\nWEAKNESSES:\n1. This paper is somewhat over-claim and some important references are missed. In lines 45-47, the authors claim that this work makes the first attempt to design contrastive learning objectives by neither relying on explicit nor implicit homophily assumptions. Actually, there are several works already done. For example,\n[1] Wang H, Zhang J, Zhu Q, et al. Augmentation-free graph contrastive learning. arXiv preprint arXiv:2204.04874, 2022.\n\n2. The proposed objective function seems like a variant of the GCL with representation smoothing diagram and the GCL with augmented views diagram. Actually, the difference between them and the proposed method is the definition of the positive pairs. \n\n3. In this work, the authors claim that the proposed method does not rely on the homophily assumption and conduct a theoretical analysis to further support the proposed method. Are these theorems hold without any assumptions?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

comments: SUMMARY:
This work first points out that existing GCL can fail to generalize to heterophilic graphs, then develops a new framework called GraphACL based on an encoder capturing one-hop neighbourhood context and two-hop monophyly. Experiments validate the effectiveness of the proposed method.

STRENGTHS:
The paper is well organized. Each part has a clear goal and supports the authors' claim well.

The proposed method, Graph Asymmetric Contrastive Learning, is not only explained in detail but also justified by solid theoretical analysis. The authors adequately answered how and why it works.

In the empirical study, the choice of baselines has good coverage. The selection of datasets respects the theme of each subsection, and the experiment setup is reasonable. The experiments are well explained, making it easy to reproduce them.

WEAKNESSES:
There are no significant weaknesses in this paper. The application scenario of the proposed method may be a little bit limited, but this is completely acceptable.

QUESTIONS:
Does GADC have the potential to be applied to graph tasks other than node classification shown in the paper?

LIMITATIONS:
The authors adequately addressed their work's limitations and potential societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents GraphACL, which aims to tackle limitations of other graph contrastive learning works which have implicit or explicit homophily assumptions, and suffer in learning effective representations for heterophilic tasks.  The approach is designed to leverage the principle of monophily, and the authors evaluate their work on several homophilic and heterophilic datasets, achieving at-par performance on the former and improved performance on the latter, on downstream node classification tasks.\n\nSTRENGTHS:\n- The approach presented here is intuitive once the monophilic principle is understood, and the implementation seems straightforward.\n\n- Results suggest the approach is strongly effective (e.g. Table 1), and hte improvement on heterophilic datasets without any material performance loss / delta on homophilic datasets is compelling.\n\n- The paper is well-written, and the figures help clarify the principles motivating the design.\n\nWEAKNESSES:\n- Lines 69-80 could greatly benefit from a toy example with nodes illustrated; it is confusing to understand the principles of monophily and the design of GraphACL without such an example early in the paper.\n\n- \"we are faced with the challenge of simultaneously capturing the one-hop neighborhood context and monophily in the contrastive objective\" (line 74) -- it's not clear why this is a particular challenge; is there something about capturing these two together that is difficult to reconcile in an objective?  If so, I didn't understand this from the text.\n\n- Lines 40 and 97 rely quite a bit on the (in paper) reference [22] to reference issues with contrasting view-based GCL approaches on heterophilic graphs; it would be helpful to include some more formal claims or concrete examples in this work to make it more self-contained, since the work strongly seems to draw motivation from [22]'s findings.  Without this, it is difficult to fully appreciate the supposed homophilic biases in those GCL approaches.\n\n- The section in Related Work from line 137 onwards seems to categorize BGRL as a contrastive method.  BGRL is not conventionally contrastive due to its lack of utilizing negative examples.  There are other such non-contrastive methods (several mentioned in [1]) which are used in graph SSL which should probably be mentioned in the related work in this paper.  In fact, I believe the authors' intent to pursue a graph SSL approach which does not use augmentations but instead uses induced asymmetry is actually the objective of many \"non-contrastive\" methods.  I would encourage they look into the literature and evaluate whether they indeed think their method is contrastive or not, given it's lack of need of negative samples.\n\n- The words \"context\" and \"preference\" are used quite often throughout the paper (I think sometimes to mean the same thing).  It would be helpful to formalize these words and just use the same word frequently, as it can be confusing to disambiguate which representation is which.\n\n- Can the authors clarify if/why the loss in Eqn 4 is required?  Methods like SimSiam, BYOL, BGRL etc. which use this kind of predictor structure and stop-gradient show they avoid collapse through the asymmetric predictor and the EMA procedure on target weight update.  Does GraphACL not naturally avoid collapse for the same reasons?  Does it require this uniformity regularization?\n\n- Figure 3 could go earlier to help introduce monophily and the intuition behind what existing GCL methods prioritize and what this method prioritizes.\n\n- There are a few very strong self-supervised GNN approaches missing from baselines, e.g. [2] or the approach mentioned in [1].\n\n- There are opportunities to better evaluate the quality of self-supervised representations using link-level tasks (e.g. link prediction), or several of the tasks mentioned in [2], which may offer improved understanding beyond node classification and node clustering (referenced in Sec 6.1)\n\n- Evaluation on some larger datasets with greater data diversity and lower homogeneity would have been more compelling, e.g. ogbn-arxiv or ogbn-products.  Flickr might also be a useful moderate-size heterophilic dataset to try out.\n\n- The node clustering results should be included in the main paper in my opinion -- the evaluation on just 1 downstream SSL task is fairly \"light\" and could use greater support to evidence that GraphACL is a generally strong SSL method, rather than \"SSL for node classification\" method.  This is one of the biggest gripes in this paper -- the results demonstrating that GraphACL is a very effective SSL method are sparse, with only 1 demonstrated task in the main (evaluated) paper in terms of node classification. Other graph SSL work typically considers several downstream tasks and evaluates across them.\n\n[1] Link Prediction with Non-Contrastive Learning (Shiao et al, ICLR'23)\n[2] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization (Ju et al, ICRL'23)\n\nQUESTIONS:\nPlease see \"Weaknesses\" for general comments / concerns which would be helpful during rebuttal.\n\nLIMITATIONS:\nThese are addressed in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a simple and effective contrastive learning framework named GraphACL for both homophilic and heterophilic graphs. In particular, GraphACL can capture both one-hop local neighborhood context and two-hop monophily similarties in one single objective. The authors theoretically analyze the learning process of the GraphACL and show that GraphACL can explicitly maximize the mutual information between representations and one-hop neighborhood patterns. The experiments and theoretical analysis show the effectiveness of the proposed approach.\n\nSTRENGTHS:\n-This paper is well written and well structured. The authors first analyze the common limitations of existing GCL frameworks, and then propose some ideas and validate them on many datasets, and finally solve these limitations through the proposed methods and theorems, making this paper well understandable.\n-The experiments and theories are sufficient to demonstrate the effectiveness.\n-This paper is an early investigation of GCL on heterophilic graphs, which provides a new direction for the subsequent development of GCL.\n\nWEAKNESSES:\n-It is common to know about the graph contrastive learning with graph augmentations mentioned in paragraph 3 of the introduction. For the first contrastive scheme mentioned in paragraph 2, the authors should give more details.\n-The authors should make a more specific presentation and explanation of the high-frequency and low-frequency information mentioned in the introduction and give a specific instance that high-frequency information is more beneficial for heterophilic graphs.\n-Some nodes mentioned in the description of the monophily property of the graphs in lines 190 to 192 can be corresponded to the subgraph d in Figure 3, thus making it less difficult for the reader to read.\n-The experimental setup is inconsistent. In the 296 line the authors introduce experiments on five random seeds, while in section 6.2 it is changed to ten runs.\n\nQUESTIONS:\n1.The 184 lines of \"node identity and node preference in real-world graphs\" is not very understandable. Is this a metaphor of your own?\n2.In a general GCL methods (such as GraphCL), the representations are usually mapped into the contrastive space via a projection head before the contrastive loss is calculated. Is this operation similar to the one you mentioned in this paper.\nSome typos:\n-In Eq.9, $\\mathcal{L}_{A C L}(q)$ -> $ \\mathcal{L}_{A}(q) $\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents GraphACL, a contrastive learning method for graph representation learning. It aims to address the limitations of current methods that only consider the homophily property of graphs or rely heavily on graph augmentation methods. The authors propose an asymmetric predictor approach where the one-hop local neighborhood context and the two-hop monophily similarity are captured.\n\nSTRENGTHS:\n- The premise of the need for a new contrastive learning method that does not overly rely on graph augmentation methods or the homophily property of graphs is reasonably argued and valid. Evidence from other studies or more data to support this argument could further its strength.\n\nWEAKNESSES:\n- The explanation of how GraphACL works and how it captures both the one-hop local neighborhood context and the two-hop monophily similarity might be difficult to comprehend for readers lacking technical background. The use of concrete examples or analogies could strengthen this argument.\n\n- The argument strength could be improved through more detailed explanation of the theoretical analysis of GraphACL, giving more justifications or linking it to established theoretical or empirical studies in the field of contrastive learning.\n\n- The paper's argument that GraphACL is a superior tool when compared to current state-of-the-art graph contrastive learning methods is backed by empirical evidence on both homophilic and heterophilic graphs. However, some rencent proposed method are missing (i.e., [1] and [2])\n\n[1] Spectral Augmentation for Self-Supervised Learning on Graphs\n\n[2] Spectral Feature Augmentation for Graph Contrastive Learning and Beyond\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nOverall, the paper presents an interesting and valid argument. However, to enhance its strength, the author could provide more empirical data, concrete examples, and clearer explanation of the concepts and methodology.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}