SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Teng Xiao $^{1*}$ , Huaisheng Zhu $^{1*}$ , Zhengyu Chen $^{2}$ , Suhang Wang $^{1}$ $^{1}$ The Pennsylvania State University,  $^{2}$ Zhejiang University, {tengxiao,hvz5312,szw494}@psu.edu, chenzhengyu@zju.edu.cn
Graph Contrastive Learning (GCL) has shown superior performance in representation learning in graph-structured data. Despite their success, most existing GCL methods rely on prefabricated graph augmentation and homophily assumptions. Thus, they fail to generalize well to heterophilic graphs where connected nodes may have different class labels and dissimilar features. In this paper, we study the problem of conducting contrastive learning on homophilic and heterophilic graphs. We find that we can achieve promising performance simply by considering an asymmetric view of the neighboring nodes. The resulting simple algorithm, Asymmetric Contrastive Learning for Graphs (GraphACL), is easy to implement and does not rely on graph augmentations and homophily assumptions. We provide theoretical and empirical evidence that GraphACL can capture one-hop local neighborhood information and two-hop monophily similarity, which are both important for modeling heterophilic graphs. Experimental results show that the simple GraphACL significantly outperforms state-of-the-art graph contrastive learning and self-supervised learning methods on homophilic and heterophilic graphs. The code of GraphACL is available at https://github.com/tengxiao1/GraphACL.
Contrastive learning has emerged as a promising regime for unsupervised vision representation learning without using annotated labeled data [1]. Recently, graph contrastive learning (GCL) has also been introduced to graph-structured data due to the lack of task-specific node labels [2, 3, 4, 5, 6, 7]. GCL has achieved competitive (or even better) performance on many downstream tasks on graphs compared to its counterparts trained with annotated ground-truth labels [7, 8].
Generally, existing GCL methods can be categorized into two categories. The first contrastive scheme [9, 10, 11, 12] reconstructs the local network structure (i.e., observed edges) to align with traditional network-embedding objectives [13, 14]. Specifically, this scheme treats one-hop or random-walk local neighboring nodes of the target node as positive examples and non-neighboring nodes as negative samples. It then employs contrastive loss functions to minimize the representation distance of positive pairs and maximize the distance of negative pairs [10]. The key motivation behind this scheme is the explicit homophily assumption that semantically similar nodes are more likely to be linked than dissimilar ones. Thus, connected nodes should have similar representations in the latent space. However, in heterophilic graphs, connected nodes are not necessarily from the same semantic class [15, 16], and should not be simply pulled together in the latent space. This scheme empirically faces issues with performance in graphs exhibiting heterophily [17, 18].
The second graph contrastive scheme involves graph augmentations [3, 4, 5, 6, 7]. Specifically, it constructs two views through stochastic graph augmentation and then learns representations by contrasting these views based on the information maximization principle [19]. A positive node
pair consists of two views resulting from stochastic data augmentation of the same node, while a negative pair might consist of two views of different nodes [8]. This scheme is built based on the idea that augmentations can preserve the semantic nature of samples, i.e. augmented nodes have consistent semantic labels with the same original nodes. However, recent work has shown that graph augmentation methods struggle to achieve good performance on heterophilic graphs [18] as they still implicitly rely on homophily [20]. Studies have shown that stochastic augmentation primarily captures the common low-frequency information between two views, while neglecting the high-frequency one [20] (also see Appendix C.4). The latter is known to be more crucial for heterophilic graphs [21, 22]. Given that heterophilic graphs are prevalent across various real-world domains [16, 23], a fundamental and open question naturally arises: What kind of contrastive learning objectives can learn robust node representations on both homophilic and heterophilic graphs?
This work is the first to address the aforementioned question without relying on augmentations and homophily assumptions. Instead, we take into account two more generalized insights, as depicted in Figure 1. (i) Even in heterophilic graphs, two nodes of the same semantic class often share a similar one-hop neighborhood context [24, 18]. Thus, capturing this heterophilic one-hop neighborhood context can lead to more discriminative node representations. (ii) While homophily might be minimal or non-existent in heterophilic graphs, another frequently observed
(a) Homophily
Figure 1: The graphs of pure homophily and pure heterophily, where color denotes the semantic class. For both graphs, nodes with a similar one-hop neighborhood context have similar semantic classes and two-hop similarities still exist even without the one-hop homophily.
(b) Heterophily
phenomenon in real-world graphs is monophily [25]. Monophily describes situations in real-world graphs where the attributes of a node's friends are likely to be similar to the attributes of that node's other friends [25]. Put another way, "monophily" essentially induces similarities between two-hop neighbors. As noted by [25, 26], the two-hop similarities brought about by monophily can persist even in the total absence of any one-hop similarities that might be suggested by homophily.
To support our motivations, we provide statistics of homophily, two-hop monophily ratios, and neighborhood similarities of various graphs in the real-world in Appendix C. As shown in these statistics, homophilic and heterophilic graphs in the real-world exhibit relatively strong neighborhood similarity calculated based on one-hop neighborhoods compared to homophily. In cases where homophily is weak or non-existent, monophily has been shown to still hold in real-world graphs.
In this work, we focus on exploiting the above insights to design new objectives for better node representations on both homophilic and heterophilic graphs. We propose a simple yet effective framework termed as graph asymmetric contrastive learning (GraphACL) for better node representation learning. Essentially, we are faced with the challenge of simultaneously capturing the one-hop neighborhood context and monophily in the contrastive objective. To solve this challenge, we consider each node to play two roles: the node itself (identity representation) and specific "neighbors" of other nodes (context representation), and thus should be treated differently. GraphACL trains the node identity representation by predicting the context representation of one-hop neighbors through an asymmetric predictor. Intuitively, by enforcing identity representations of two-hop neighbors to reconstruct the same context representation of the same central nodes, GraphACL implicitly makes representations of two-hop neighbors similar and captures the one-hop neighborhood context (Figure 2).
Our primary technical contributions are: (1) We propose a simple, effective, and intuitive graph contrastive learning approach which captures one-hop neighborhood context and two-hop monophily similarities in a simple asymmetric learning framework. (2) We theoretically analyze the learning behavior and prove that GraphACL is guaranteed to yield good node representations for both homophilic and heterophilic graphs. (3) Empirically, we corroborate the effectiveness of GraphACL on 15 graph benchmarks. The results demonstrate that GraphACL can significantly outperform previous GCL methods. The surprising effectiveness of GraphACL shows that in the context of GCL, simpler methods have been underexplored in favor of more elaborate algorithmic contributions.
Graph Contrastive Learning. Contrastive methods are central to traditional network-embedding methods [27, 13, 14], but have recently been applied in graph self-supervised learning [9, 28, 10].
(b) Homophily
(c) Context
Figure 2: An illustration of various design motivations. (a) The heterophilic graph where the color denotes node's semantic class. (b) Contrastive objectives with the homophily assumption encourage one-hop neighbors to have similar representations. GraphACL simply encourages the node to predict its neighbors, which can implicitly capture neighborhood context (c) and two-hop monophily (d).
(d) Monophily
The key motivation behind them is the explicit homophily assumption that connected nodes belong to the same class and, thus, should be treated as positive pairs in contrastive learning. However, real-world graphs do not always obey the homophily assumption [16], limiting their applicability to heterophilic graphs [18]. Recently, many GCL algorithms with augmentations [2, 3, 4, 5, 6, 29] have been proposed. However, [20] recently proved that GCL with augmentations attempts to manipulate the encoder to capture low-frequency information instead of the high-frequency part and suffers performance degradation in heterophilic graphs [18]. In contrast, we propose a simple asymmetric contrastive learning framework for graphs without augmentations and homophily assumption.
Heterophilic Graphs. There are many heterophilic graphs in the real world that exhibit nonhomophilic properties, such as transaction [30, 31], ecological food [32] and molecular networks [15], where the linked nodes have different characteristics and different class labels. Various graph neural networks (GNNs) [33, 15, 21, 34, 35, 23, 36, 37] have been proposed to achieve higher performance in low homophily settings. They focus on designing advanced GNN architectures and consider the semi-supervised setting with labels. In contrast, we focus on designing the contrastive learning algorithm without labels, not on specific GNN architectures. Recently, DSSL [18] and HGRL [29] have been proposed to conduct self-supervised learning on nonhomophilous graphs by capturing global and high-order information. Specifically, HGRL is based on graph augmentation, and DSSL works by assuming a graph generation process, which may not always hold true for real-world graphs. In contrast, our work provides a novel perspective on graph contrastive learning, which does not make any assumptions about the generation process or rely on any augmentations.
Notations and Problem. Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be a input graph, where  $\mathcal{V} = \{v_{1},\dots ,v_{|\mathcal{V}|}\}$  is the set of  $|\mathcal{V}|$  nodes and  $\mathcal{E}$  is the set of edges. Let  $\mathbf{X} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_{|\mathcal{V}|}]\in \mathbb{R}^{|\mathcal{V}|\times D_x}$  be the node attribute matrix, where  $\mathbf{x}_i$  is the  $D_{x}$ -dimensional feature vector of  $v_{i}$ . Each edge  $e_{i,j}\in \mathcal{E}$  denotes a link between node  $v_{i}$  and  $v_{j}$ . The graph structure can be denoted by an adjacency matrix  $\mathbf{A}\in [0,1]^{|\mathcal{V}|\times |\mathcal{V}|}$  with  $\mathbf{A}_{i,j} = 1$  if  $e_{i,j}\in \mathcal{E}$ , otherwise  $\mathbf{A}_{i,j} = 0$ . We denote the normalized adjacency matrix  $\tilde{\mathbf{A}} = \mathbf{D}^{-1 / 2}\mathbf{A}\mathbf{D}^{-1 / 2}$  where  $\mathbf{D}$  is the diagonal degree matrix. Let  $\mathbf{L} = \mathbf{I} - \tilde{\mathbf{A}}$  be the symmetric normalized graph Laplacian matrix. We also define the unweighted two-hop graph  $\mathcal{G}_2$ , whose adjacency matrix is  $\mathbf{A}_2$ .  $(\mathbf{A}_2)_{ik} = 1$  if there exists  $j$  such that  $e_{ij}\in \mathcal{E}$  and  $e_{jk}\in \mathcal{E}$ . The input graph  $\mathcal{G}$  can be denoted as a tuple of matrices  $G = (\mathbf{X},\mathbf{A})$ . Our goal is to learn a GNN encoder  $f_{\theta}$  parameterized by  $\theta$ , such that the representation of node  $v$ :  $\mathbf{v} = f_{\theta}(G)[v]\in \mathbb{R}^{D}$ , can perform well for downstream tasks.
Homophily Ratio. Typically, the graph homophily ratio [33, 15] is defined as the fraction of edges connecting nodes with the same labels, i.e.,  $\mathcal{H}(\mathcal{G}) = |\{(u,v):(u,v)\in \mathcal{E}\wedge y_u = y_v\} | / |\mathcal{E}|.$  Homophily Graphs have high edge homophily ratio  $\mathcal{H}(\mathcal{G})\to 1$  , while heterophilic graphs (i.e., low/weak homophily) correspond to small edge homophily ratio  $\mathcal{H}(\mathcal{G})\to 0$  [15, 16].
GCL with Representation Smoothing. This paradigm [13, 9, 28, 12] ensures that local neighboring nodes have similar representations and has also been shown to be equivalent to factorizing graph proximity [38]. Despite minor differences, the objective for this paradigm can be expressed as:
$$
\mathcal {L} _ {\mathrm {S}} = - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \log \frac {\exp \left(\mathbf {v} ^ {\top} \mathbf {u} / \tau\right)}{\exp \left(\mathbf {v} ^ {\top} \mathbf {u} / \tau\right) + \sum_ {v _ {-} \in \mathcal {V} _ {-}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)}. \tag {1}
$$
Here  $\mathbf{v}$ ,  $\mathbf{u}$  and  $\mathbf{v}_{-}$  are the projected node representations by  $f_{\theta}(G)$  of nodes  $v$ ,  $u$  and  $v_{-}$ , respectively.  $\tau$  is the temperature hyper-parameter. Typically,  $\mathcal{N}(v)$  is the positive sample set containing one-hop
(a) GCL with Representation Smoothing
(b) GCL with Augmented Views
(c) Graph Asymmetric Contrastive Learning
Figure 3: Illustration of existing contrastive schemes and GraphACL. (a) forces neighboring nodes to have similar representations based on the homophily assumption. (b) augments the graph and learns the augment-invariant representations of the same node. Our GraphACL in (c) simply reconstructs the neighborhood signal of each node based on an asymmetric predictor without relying on the homophily assumption and augmentation.
local neighborhoods of node  $v$ , and  $\mathcal{V}_{-}$  is the negative sample set that can be randomly sampled from the node space  $\mathcal{V}$  or sampled proportionally to the power  $3/4$  of the degree of the node [14]. Figure 3 (a) illustrates this contrastive learning scheme with this explicit homophily assumption.
GCL with Augmented Views. This paradigm [4, 20, 39, 3] learns representations by contrasting views based on stochastic augmentations. Specifically, for node  $v$ , its representation in one augmented view is learned to be close to the representation of the same node  $v$  from the other augmented view and far away from the representations of negative samples from other nodes. Given two augmentations  $G_{1}$  and  $G_{2}$  extracted from original graph  $\bar{G}$  in a predefined way, the contrastive objective is as follows:
$$
\mathcal {L} _ {\mathrm {V}} = - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \log \frac {\exp \left(\mathbf {v} ^ {1} \cdot \mathbf {v} ^ {2} / \tau\right)}{\exp \left(\mathbf {v} ^ {1} \cdot \mathbf {v} ^ {2} / \tau\right) + \sum_ {v _ {-} \in \mathcal {V} _ {-}} \exp \left(\mathbf {v} ^ {1} \cdot \mathbf {v} _ {-} / \tau\right)}. \tag {2}
$$
Here  $\mathbf{v}^1 = f_{\theta}(G_1)[v]$  and  $\mathbf{v}^2 = f_{\theta}(G_2)[v]$  are projected representations of node  $v$  from two augmented views.  $\mathcal{V}_{-}$  is the set of negative samples of  $v$  from the inter- or intra-view [39]. Inspired by BYOL [40], non-contrastive BGRL [5, 41] predicts the target augmented view of the nodes:  $\mathcal{L}_V = -1 / |\mathcal{V}|\sum_{v\in \mathcal{V}}\mathbf{z}^1\cdot \mathbf{v}^2 /\| \mathbf{z}^1\| \| \mathbf{v}^2\|$ , where  $\mathbf{z}^1 = g_\phi (\mathbf{v}^1)$  is the prediction of the representation from the target augmented view. An illustration for this scheme is presented in Figure 3 (b).
4 Simple Asymmetric Contrastive Learning of Graphs
In this section, we elaborate on our GraphACL. The key idea behind GraphACL is encouraging the encoder to learn representations by simultaneously capturing one-hop neighborhood context and two-hop monophily, which generalizes the homophily assumption for modeling both homophilic and heterophilic graphs. Specifically, GraphACL introduces an additional predictor  $g_{\phi}$  which maps the graph  $G$  to node representations that can predict the one-hop neighborhood context from the representation of the central node as shown in Figure 3 (c). Importantly, unlike previous contrastive schemes, which force neighboring nodes to have similar representations, GraphACL directly predicts the original neighborhood signal of each node through an asymmetric design induced by a predictor. This simple asymmetric design allows neighboring nodes to have different representations but still can capture the one-hop neighborhood context and two-hop monophily as illustrated in Figure 2.
4.1 Graph Asymmetric Contrastive Learning
Based on the above motivation, for each node  $v$ , we first propose to learn its representation by capturing its one-hop neighborhood signal. A natural idea of capturing the neighborhood signal is learning the representations of  $v$  that can well predict the original features of  $v$ 's neighbors, i.e.,  $\mathbf{x}_u$  of neighbor  $u$ . However, the original features are typically noisy and high-dimensional [42]. To solve this issue, we cast the prediction problem in the representation space, i.e., the representation of the central node should be predictive of representations of its neighbors. Specifically, we adopt the following simple prediction loss induced by an asymmetric predictor on neighbors:
$$
\mathcal {L} _ {\mathrm {P R E}} = \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \| g _ {\phi} (\mathbf {v}) - \mathbf {u} \| _ {2} ^ {2}, \tag {3}
$$
where  $\mathbf{v} = f_{\theta}(G)[v]$  and  $\mathbf{u} = f_{\theta}(G)[u]$  are representations, and  $g_{\phi}(\cdot)$  is a introduced predictor which maps the latent representation  $\mathbf{v}$  to reconstruct the neighbor's representation  $\mathbf{u}$ . Intuitively, in this case, each node is treated as a specific neighbor "context", and nodes with similar distributions over the neighbor "context" are assumed to be similar. Here, we utilize only one single simple  $g_{\phi}$  to predict representations of all  $v$ 's neighbors and empirically find that it works very well.
This simple prediction objective in Equation (3) can preserve local neighborhood distribution in representations without homophily assumption since we do not directly enforce  $\mathbf{v}$  and  $\mathbf{u}$  to be similar to each other. To prevent  $g_{\phi}$  from degenerating to the identity function, we consider the representations of central node  $v$  and the one-hop neighbor  $u$  come from two decoupled encoders:  $\mathbf{v} = f_{\theta}(G)[v]$  and  $\mathbf{u} = f_{\xi}(G)[u]$ , where  $f_{\theta}$  is the online identity encoder and  $f_{\xi}$  is the preference target encoder. Importantly, the gradient of the loss in Equation (5) is only used to update the online encoder  $f_{\theta}$ , while being blocked in the target encoder  $f_{\xi}$  as shown in Figure 3 (c). The weights of the target network  $\xi$  are updated using the exponential moving average of online network weights  $\theta$ :  $\xi \gets \lambda \xi + (1 - \lambda)\theta$  where  $\lambda \in [0,1]$  is the target decay rate. An intuitive illustration for using two encoders is that each node typically plays two roles: the node itself and a specific context of other nodes, corresponding to separated notions of node identity and node preference in real-world graphs. The key principle of decoupled encoders is to allow the identity representation of a node to be more possibly different from the identity representation with which the node prefers to link.
The simple objective in Equation (3) can capture the one-hop neighborhood context without relying on the homophily assumption or requiring graph augmentation as shown in Figure 2 (c). This matches our first motivation in the introduction part. Another remaining problem is why optimizing this simple objective helps capturing 2-hop node similarity, i.e., monophily. Consider a pair of 2-hop neighbors  $v$  and  $u_{2}$  which both neighbor on the same node  $u$ . Intuitively, by enforcing  $\mathbf{v}$  and  $\mathbf{u}_{2}$  to reconstruct the same context representation of neighborhood  $\mathbf{u}$ , we implicitly make their representations similar. Thus, the 2-hop neighbors serve as positive pairs that will be implicitly aligned as shown in Figure 2 (d). We formally prove this intuition in Theorem 2 in our following theoretical analysis.
We note that some contrastive objectives based on augmentations such as BGRL [5] and BYOL [40] also utilize a predictor as shown in Figure 2 (b). However, the idea behind our loss in Equation (3) differs significantly from them. Specifically, they rely on data augmentations, and are built based on the idea that the augmentation can preserve the semantic nature of samples, i.e., the augmented samples have consistent semantic labels with the original ones. Nevertheless, unlike images, it is theoretically difficult to design graph augmentations without changing the semantic nature of nodes [7] and augmentations tend to capture homophily [20, 18]. In contrast, the simple loss in Equation (3) does not rely on augmentations but directly considers modeling one-hop neighborhood distribution. We theoretically prove that our GraphACL maximizes the mutual information between representations and one-hop neighborhood context, and captures two-hop monophily in § 5.
Although this simple neighborhood prediction objective can capture both one-hop neighborhood pattern and two-hop monophily, it may result in a collapsed and trivial encoder: all node representations degenerate to a the same single vector on the hypersphere. The main reason is that the prediction loss operates in a fully flexible latent space, and it can be minimized when the encoder produces a constant representation for all nodes. To address this issue, we introduce an explicit uniformity regularization to further enhance the representation diversity. Specifically, we add the following explicit regularization on representation uniformity into Equation (3):
$$
\mathcal {L} _ {\mathrm {U N I}} = - \frac {1}{| \mathcal {V} |} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {v _ {-} \in \mathcal {V}} \| \mathbf {v} - \mathbf {v} _ {-} \| _ {2} ^ {2}, \tag {4}
$$
where  $\mathbf{v} = f_{\theta}(G)[v]$  and  $\mathbf{v}_{-} = f_{\theta}(G)[v_{-}]$ . Here, we consider the representations of negative samples  $\mathbf{v}_{-}$  coming from the same online encoder as central sample. This uniformity loss is typically approximated by randomly sampling  $K$  negative samples  $v_{-}$ . Intuitively, minimizing this term will push all node representations away from each other and alleviate the representation collapse issue.
Graph Asymmetric Contraitive Loss. Directly combining Equations (3) and (4) arrives at a loss function:  $\mathcal{L}_{\mathrm{COM}} = \mathcal{L}_{\mathrm{PRE}} + \mathcal{L}_{\mathrm{UNI}}$ . However, minimizing this combination loss is an ill-posed problem, it approaches  $-\infty$  as we can simply scale the norm of representations to reduce the loss. Even if the representations are normalized, minimizing  $\mathcal{L}_{\mathrm{UNI}}$  still leads to a relatively poor performance as shown in our experiments. To address this issue, we instead minimize an upper bound of  $\mathcal{L}_{\mathrm{COM}}$ , which results in the following simple objective of GraphACL (see Appendix A.1 for details):
$$
\mathcal {L} _ {\mathrm {A}} = - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \log \frac {\exp \left(\mathbf {p} ^ {\top} \mathbf {u} / \tau\right)}{\exp \left(\mathbf {p} ^ {\top} \mathbf {u} / \tau\right) + \sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)}. \tag {5}
$$