SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To further verify our second intuition in the introduction: two-hop similarities (monophily) still exist even without the one-hop homophily assumption, we also consider monophily properties of two-hop neighborhoods. Following [16], we use the following two-hop homophily to measure the monophily, where the neighborhood of each node is defined to be the nodes of exactly two hops away.
$$
\mathcal {H} \left(\mathcal {G} _ {2}\right) = \frac {1}{| \mathcal {V} |} \sum_ {v} \frac {1}{| N _ {2} (v) |} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \mathbb {1} _ {y _ {v _ {2}} = y _ {v}}, \tag {39}
$$
where  $\mathcal{N}_2(v)$  denotes the set of two-hop neighborhoods of  $v$ .
The statistics of datasets, including their one-hop and two-hop homophily levels and neighborhood similarities, are given in Table 3. We can observe both homophilic and heterophilic graphs exhibit strong neighborhood similarity calculated based on one-hop neighborhoods. In addition, the two-hop homophily level is generally higher than the one-hop homophily level in heterophilic graphs. This observation confirms our intuition that two-hop similarities still exist without a strong one-hop homophily level. The descriptions of the datasets are given below:
Cora, Citeseer, and Pubmed [49]. They are among the most widely used node classification benchmarks. Each dataset is a citation and high-homophily graph, where nodes are documents, and edges are citation relationships from one to another. The class label of each node is based on the research field. A bag of words of its abstracts is used as the features of nodes. The public split is used for these datasets, where each class has fixed 20 nodes for training, another fixed 500 nodes and 1000 nodes for validation/test, respectively for evaluation.
Computer and Photo [5, 51]. They are co-purchase graphs from Amazon, where nodes represent products and frequently bought products are connected by edges. Node features represent product reviews, and class labels indicate the product category. Following the experiment settings from [12], we randomly split the nodes into train/validation/test (10%/10%/80%) sets.
For these homophilic datasets, we utilized the process version of them provided by Deep Graph Library [61]. These datasets can be found in https://docs.dgl.ai/en/0.6.x/api/python/dgl.data.html.
Ogbn-arxiv (Arxiv) [52]. It is a citation network between all Computer Science (CS) arXiv papers. Each node represents one paper and each edge indicates the citation relationships between two papers. The feature of each node is obtained by a 128-dimensional feature vector through averaging the embeddings of words in its title and abstract. The embeddings of words are obtained by running the skip-gram over the MAG corpus. Following [12], we also use  $10\% / 10\% / 80\%$  split for this dataset.
Texas, Wisconsin and Cornell [33]. They are webpage networks collected from computer science departments of different universities by Carnegie Mellon University. For each network, nodes are
web pages and edges indicate hyperlinks between web pages. Node features are bag-of-words representations of web pages. The task is to classify nodes into five categories: student, project, course, staff, and faculty.
Chameleon, Crocodile and Squirrel [47]. They are Wikipedia networks, where nodes represent web pages and edges represent hyperlinks between them. Features of nodes are several informative nouns on Wikipedia pages. Labels of nodes are based on the average daily traffic of the web page.
Actor [33]. It is an actor-only induced subgraph of the film-director-actor-writer network. Nodes correspond to actors and edges represent the co-occurrence of two nodes on the same Wikipedia page. Node features are keywords in Wikipedia pages. Labels are assigned five categories in terms of words on the actor's Wikipedia.
For Texas, Wisconsin, Cornell, Chameleon, Crocodile, Squirrel and Actor, we use the raw data provided by the Geom-GCN [33] with the standard fixed 10-fold split for our experiment. These datasets can be downloaded from: https://github.com/graphhdluiuc-jlu/geom-gcn.
Roman-empire (Roman) [48] is a heterophilous graph based on the Roman Empire article in English Wikipedia. Each node in the graph corresponds to one (non-unique) word in the text. The node features are from word embeddings. The class of a node is its syntactic role (the 17 most frequent roles as unique classes and all the other roles are grouped into the 18th class). Following [48], we utilize the fix 10 random  $50\% /25\% /25\%$  train/validation/test splits. This dataset can be found in https://github.com/yandex-research/heterophilous-graphs.
Arxiv-year [16] is a citation network from a subset of the Microsoft Academic Graph, where the task focuses on predicting the year that a paper is posted. Nodes are papers, and edges are relevant to citations. The node features correspond to the average of word embeddings of the title and abstract of the papers. Following [16],  $50\% / 25\% / 25\%$  train/val/test split is utilized for this dataset. This dataset can be found in https://github.com/CUAI/Non-Homophily-Large-Scale.
LINE [14]: This network embedding model proposed an objective function to preserve both the first-order and second-order proximities.
VGAE [9]: VGAE is a generative model based on variational autoencoder for node representation learning by directly reconstructing adjacency matrix.
DGI [2]: It is a self-supervised learning method that maximizes the mutual information between node representations and graph summary.
GCA [4]: This is a graph contrastive representation learning method with adaptive augmentation that incorporates various priors for topological and semantic aspects of the graph.
BGRL [5]: This model is a graph contrastive learning method, where the alternative augmentations of the graph are predicted to learn representations of nodes.
CCA-SSG [6]: CCA-SSG is a graph contrastive learning model, which encourages the learned representations of nodes by reducing the correlation between different views.
L-GCL [12]: It is a self-supervised node representation learning method, which samples positive samples from the first order neighborhoods and kernelizes negative loss to reduce the training time.
SP-GCL [53]: This is a single-pass graph contrastive learning method based on the concentration property of node representations.
HGRL [29]: This is a self-supervised representation learning framework on graphs with heterophily, which leverages the node original features and the high-order information.
DSSL [18]: This method introduces a representation learning framework by decoupling the diverse neighborhood context of a node in an unsupervised manner.
C.3 Setup and Hyper-parameter Settings
We use official implementation publicly released by the authors on Github of the baselines. For fair comparison, we used grid search to find the best hyperparameters of the baselines. We run
Figure 7: The average spectrum changes across these frequency bands for two augmented views. The perturbations resulting from augmentations reveal a consistent trend in the graph spectrum.
experiments on a machine with a NVIDIA RTX A6000 GPU with 49GB of GPU memory. In all experiments, we use the Adam optimizer [62]. A small grid search is used to select the best hyperparameter for all methods. In particular, for our GraphACL, we search  $\lambda$  from  $\{0, 0.90, 0.95, 0.97, 0.99, 0.999, 1\}$ ,  $D$  from  $\{256, 512, 1024, 2048, 4096, 8192\}$ ,  $\tau$  from  $\{0.25, 0.5, 0.75, 0.99, 1\}$ , and  $K$  from  $\{1, 5, 10\}$  when we utilize the negative sampling. We tune the learning rate over  $\{0.001, 0.0005, 0.0001\}$  and weight decay over  $\{0, 0.0001, 0.0003, 0.000001\}$ . We select the best configuration of hyper-parameters based on average accuracy only based on the validation set.
Table 4: Node clustering performance in terms of NMI (%) on homophilic and heterophilic graphs
Method	CiteSeer	Computer	Photo	Arxiv	Texas	Cornell	Squirrel	Arxiv-year
VGAE	36.40±0.01	22.30±0.00	53.00±0.04	31.10±0.01	27.75±0.16	17.87±0.13	10.83±0.09	25.64±0.05
DGI	43.90±0.00	31.80±0.02	47.60±0.03	31.90±0.01	34.17±0.07	15.92±0.15	8.49±0.13	24.35±0.08
BGRL	45.38±0.04	36.20±0.07	54.61±0.08	33.71±0.09	33.59±0.15	19.74±0.14	15.13±0.09	25.31±0.05
DSSL	45.91±0.06	36.82±0.08	54.99±0.05	32.98±0.06	38.22±0.15	20.36±0.08	19.85±0.13	24.45±0.07
L-GCL	46.52±0.08	37.51±0.09	55.25±0.01	32.77±0.05	37.92±0.10	19.25±0.05	18.15±0.11	26.87±0.09
GraphACL	46.82±0.07	39.51±0.09	55.97±0.04	34.15±0.06	43.31±0.08	22.79±0.05	23.96±0.09	29.84±0.08
C.4 The Spectrum Changes in Augmented Views
In Figure 7, we categorize the eigenvalues (spectrum) of the symmetric normalized graph Laplacian into five distinct frequency bands, ranging from low to high. Subsequently, we computed the average spectrum changes across these frequency bands for two augmented views. The perturbations resulting from augmentations reveal a consistent trend in the graph spectrum. After augmenting the graph, we can observe that the perturbations in the low-frequency components (far left) are comparatively smaller than those observed in the high-frequency components (far right) for both homophilic (PubMed) and heterophilic (Squirrel) graphs. This observation confirms that mainstream GCL methods can not work well on heterophilic graphs, particularly when the perturbation in high-frequency signals is more significant.
C.5 Node Clustering Performance
We also conduct node clustering to evaluate the quality of the learned node representations. Specifically, we obtain node representation with GraphACL, then perform k-means clustering on the obtained representations and set the number of clusters to the number of ground truth classes. The experiment is conducted five times. We report the average normalized mutual information (NMI) for clustering in Table 4 on both homophilic and heterophilic graphs. From the table, we can observe that our GraphACL can consistently improve node clustering performance compared to the state-of-the-art self-supervised learning baselines on eight datasets. This observation, along with the results of the node classification, demonstrates the effectiveness of GraphACL in learning more expressive and robust node representations for various downstream tasks. These results further validate that
Table 5: Experimental results (%) on the graph classification results.
Method	LINE	VGAE	GraphCL	L-GCL	DSSL	GraphACL
MUTAG	75.6±2.3	84.4±0.6	86.8±1.3	85.3±0.5	87.2±1.5	89.4±2.0
PROTEINS	61.1±1.7	74.0±0.5	74.4±0.5	72.9±0.6	73.5±0.7	75.3±0.5
modeling one-hop neighborhood pattern and two-hop monophily similarity benefit downstream tasks on real-world graphs with various homophily ratios.
C.6 Graph Classification Performance
Existing works for heterophilic graphs typically focus on node-level tasks (node classification and node clustering). Thus, an empirical comparison of the graph-level task is relatively not straightforward. Nevertheless, for graph classification, we can use a non-parameterized graph pooling (readout) function, e.g., MeanPooling, to obtain the graph-level representation. For graph classification, we conduct experiments on three graph classification benchmarks: MUTAG and IMDB-B, and follow the same setting in GraphCL [3]. The results are shown in Table 5. From the table, we can observe that our ACL framework can also work well on the graph classification task and still can achieve better (or competitive) performance compared to baselines.
Figure 8: The classification performance curves with varying decay rate  $\lambda$  on homophilic graphs.
Figure 9: The node classification performance with varying decay rate  $\lambda$  on other heterophilic graphs.
C.7 The Effect of Target Decay Rate
Figures 8 and 9 show the results of node classification with varying decay rate  $\lambda$  on four other homophilic and four heterophilic graphs, respectively. We can still observe that having large values of  $\lambda$  improves the overall performance. We observe that GraphACL obtains a competitive but not the best result when  $\lambda = 0$ , which confirms that slowly updating the target network is crucial in obtaining superior performance. We also notice that we can achieve good performance even with a random target encoder, i.e.,  $\lambda = 1$  on some datasets, which can be explained by the recent work [63] that shows only using the simple stop-gradient operation can sometimes prevent collapsing.
C.8 The Effect of Temperature
Figures 10 and 11 show the results of node classification with variable temperature  $\tau$  in homophilic and heterophilic graphs. We can observe that our GraphACL is not very sensitive to temperature  $\tau$  on heterophilic graphs, while moderate hardness of the softmax (large  $\tau$ ) produces the best result. For homophilic graphs, the large or small temperature can lead to poor performance.
Figure 10: The effect of temperature  $\tau$  on five homophilic graphs.
Figure 11: The effect of temperature  $\tau$  on five heterophilic graphs.
Figure 12: The effect of the number of negative samples  $K$ .
Figure 13: The distribution of pair-wise cosine similarity calculated by learned representations on randomly sampled node pairs, one-hop neighbors and two-hop neighbors.
C.9 The Effect of the Number of Negative Samples
We run a sweep over the size of negative samples  $K$  to study how it affects performance. We vary  $K$  as  $\{1, 5, 10\}$ . For each  $K$ , we first learn node representation and then use the learned node representation for node classification. Figure 12 shows the results with varying  $K$ . From the figure, we can observe that a small number of negative samples (e.g.,  $K = 5$ ) is enough to achieve good performance on all graphs. For homophilic graphs, we can observe that large  $K$  can promote the
Figure 14: Similarity (cosine similarity between representation  $\mathbf{v}$  and prediction  $\mathbf{p} = g_{\phi}(\mathbf{v})$ ) histogram on different graphs.
Pubmed (id 19803)
CiteSeer (id 2483)
Computer (id 2330)
Cornell (id 106)
Texas (id 108)
Crocodile (id 7849)
Wisconsin (id 83)
Chameleon (id 117)
Figure 15: Case studies. Node colors denote the semantic labels of nodes. The size of the node is proportional to its similarity to the central node denoted as the star.
performance of GraphACL. In contrast, for heterophilic graphs, training with large  $K$  will lead to a slight drop in performance. A possible reason is that the randomly sampled negative samples can not represent the whole node-set, given the heterogeneous and diverse patterns of heterophilic graphs.
C.10 More Similarity Histograms of Node Pairs
Figure 13 shows the additional results on the representation similarity. The observations are generally similar to the results in the main body of the paper. As shown by the figure, we can observe that the randomly sampled node pairs are easier to be distinguished from one-hop and two-hop neighbors based on the representation similarity, which demonstrates that our GraphACL indeed captures the semantic meaning of node and will desirably push away the semantically dissimilar nodes. Moreover, the two-hop similarities in heterophilic graphs are much larger than that in homophilic graphs. These phenomena provide explanations for why GraphACL achieves good performance by capturing two-hop monophily structure information. Figure 14 shows the cosine similarity between the v and prediction  $g_{\phi}(\mathbf{v})$  for each node on other graphs. We can find that the cosine similarities are typically smaller than 1, which shows that the predictor  $g_{\phi}$  is not an identity matrix after convergence. Moreover, we can observe that, compared to the homophilic graph Cora, the similarities on the heterophilic graph Squirrel are smaller. Since the objective of GraphACL will pull  $\mathbf{p} = g_{\phi}(\mathbf{v})$  and  $\mathbf{u}$  together, thus GraphACL can automatically differ identity representation  $\mathbf{v}$  and preference representation  $\mathbf{u}$ , which is important for modeling heterophilic graphs. For the homophilic graph
Cora, the node identity representation  $\mathbf{v}$  and the preference representation  $\mathbf{u}$  should be similar, which is also captured by our simple contrastive objective in GraphACL.
C.11 Additional Case Study Results
In Figure 15, we provide the additional case studies and visualization results on other graphs. We can find that, in most cases, nodes sharing the same semantic classes with the central nodes have larger similarities to the central nodes. This observation interprets the reason why GraphACL can achieve good performance. We observe that GraphACL can effectively capture the local neighborhood pattern and two-hop monophily for both homophilic and heterophilic graphs, which empirically verifies our theoretical analysis given in the main body of the paper.
D Societal Impacts and Limitations
There are numerous graphs in the real world that exhibit heterophilic properties, such as transaction networks, ecological food networks, and molecular networks, in which the connected nodes possess dissimilar features and distinct class labels. Given the successful deployment of Graph Neural Networks (GNNs) in various human-related real-world applications, including social networks, knowledge graphs, and molecular property prediction, it is crucial to propose unsupervised representation learning techniques for GNNs that can effectively handle both homophilic and heterophilic graphs, thereby potentially yielding direct social impacts.
The collection of labeled data is often expensive and impractical, particularly in domains requiring specialized knowledge, such as medicine and chemistry. Considering the potential positive impact, we believe that our work can assist researchers and practitioners in devising solutions that alleviate the reliance on labeled data. However, it is important to acknowledge the potential negative consequences, as these learned representations could also be exploited for malicious purposes, such as adversarial attacks on GNNs that exploit systematic biases. Nonetheless, we believe that our simple framework and theoretical insights derived from this work contribute as a small step towards advancing the simplicity and generalizability of graph contrastive learning models within the research community.