SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here prediction  $\mathbf{p} = g_{\phi}(\mathbf{v})$  with  $\mathbf{v} = f_{\theta}(G)[v]$ , representations  $\mathbf{u} = f_{\xi}(G)[u]$  and  $\mathbf{v}_{-} = f_{\theta}(G)[v_{-}]$ .  $\mathcal{L}_A$  is a simple generalization of the graph contrastive loss with representation smoothing in Equation (1) from symmetric view to an asymmetric view. Note that when the predictor  $g_{\phi}$  becomes the
identity function,  $\mathcal{L}_A$  degenerates to the GCL loss  $\mathcal{L}_S$ . We demonstrate in extensive experiments that such asymmetric framework via a simple predictor helps achieve better downstream performance on both homophilic and heterophilic graphs than many GCL methods with prefabricated augmentations.
In this section, we provide theoretical understandings of GraphACL. We show that our simple GraphACL can simultaneously capture one-hop neighborhood context and two-hop monophily, which are important for heterophilic graphs. We also establish theoretical guarantees for the downstream performance of the learned representations. All proofs can be found in Appendix B.
Notations. We denote the random variable of node representations as  $V$ , and define the mean representations from the one-hop neighborhoods of node  $v$  as  $\mathbf{z} = \frac{1}{\mathcal{N}(v)} \sum_{u \in \mathcal{N}(v)} \mathbf{u}$ . This means that the representation  $\mathbf{z}$  profiles the one-hop neighborhood pattern of node  $v$ . Since two nodes of the same semantic class tend to share similar neighborhood patterns in real-worlds,  $\mathbf{z}$  can be viewed sampled from  $Z|Y \sim \mathcal{N}(\mathbf{z}_Y, I)$  where  $Y$  is the latent semantic class indicating the one-hop pattern of  $v$ . Given the above, we first demonstrate the rationality of GraphACL in capturing one-hop patterns:
Theorem 1. Minimizing GraphACL's objective in Equation (5) with exponential moving average is equivalent to maximizing mutual information between representation  $V$  and the one-hop pattern  $Y$ :
$$
\mathcal {L} _ {A} \geq H (V | Y) - H (V) = - I (V; Y). \tag {6}
$$
Theorem 1 indicates that minimizing GraphACL loss in Equation (5) promotes maximizing the mutual information  $I(V;Y)$  between representations and one-hop neighborhood context. Next, we theoretically verify that our GraphACL can capture intuition about the two-hop monophily similarity.
Theorem 2. Let  $\mathcal{N}_2(v)$  denote the set of two-hop neighbors of  $v$ . Minimizing the GraphACL objective in Equation (5) is approximately minimizing the following alignment loss between two-hop neighbors:
$$
\mathcal {L} _ {\text {t w o - h o p}} = \frac {1}{| \mathcal {V} |} \frac {1}{2 L} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} _ {2} (v) |} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2}, \tag {7}
$$
where  $L$  is the  $L$ -bi-Lipschitz constant of  $g_{\phi}$ :  $\forall (\mathbf{v}_1,\mathbf{v}_2), 1 / L\|\mathbf{v}_1 - \mathbf{v}_2\|_2^2 \leq \|g_{\phi}(\mathbf{v}_1) - g_{\phi}(\mathbf{v}_2)\|^2$ .
The above theorem shows that minimizing the GraphACL objective implies a small alignment loss of two-hop neighbors. Thus, GraphACL not only captures the one-hop neighborhood pattern, but also the two-hop monophily implicitly, i.e., encouraging a large similarity between two-hop neighbors.
Next, we theoretically show that the learned representations can achieve better downstream performance. We evaluate the representation by its performance on a multi-class classification task using the mean classifier [43, 44, 45]. Specifically, we consider the classifier  $p_W(v) = \arg \max_{\mathbf{W}\mathbf{v}}$  where  $\mathbf{v} = f_{\theta}(G)[v]$  and  $\mathbf{W} \in \mathbb{R}^{C \times D}$  is the weight of the linear classification head. The  $y_{\mathrm{th}}$  row of  $\mathbf{W}$  is the mean  $\pmb{\mu}_{y}$  of representations of nodes with the label  $y$ :  $\pmb{\mu}_{y} = \mathbb{E}_{v|y}[\mathbf{v}]$ . For simplicity, we assume that the node classes are balanced but our results can be easily extended to unbalanced settings by considering a label shift term as shown in the domain adaptation literature [46].
Theorem 3. Let  $\hat{h}_2 = \mathcal{H}(\mathcal{G}_2)$  be the homophily ratio of the two-hop graph  $\mathcal{G}_2$ . Suppose that the downstream task is the  $M$ -categorical linear classification and the class is balanced. Then,  $\forall q \in$  the hypothesis class with  $q = g \circ f$ , the upper bound for the classification error on the optimal  $\mathbf{v}^*$  is:
$$
P \left(y _ {v} \neq p _ {W} \left(\mathbf {v} ^ {*}\right)\right) \leq 4 M ^ {2} \left(4 L \mathcal {L} _ {A} (q) + \left(1 - \hat {h} _ {2}\right)\right) + c o n s t. \tag {8}
$$
This theorem says that the downstream error on representations is bounded by GraphACL loss and homophily ratio  $\hat{h}_2$  of the two-hop graph  $\mathcal{G}_2$ , i.e., monophily. Specifically, a larger monophily ratio  $\hat{h}_2$  would provably imply a smaller downstream classification error. Prior work [15] and the statistics of real-world graphs in Appendix C show that although the one-hop graph may be heterophily-dominant, the two-hop graph will always be homophily-dominant ( $\hat{h}_2$  is large). This theorem reveals why the simple GraphACL enjoys good performance on both homophilic and heterophilic graphs.
6.1 Experimental Settings
Datasets and Splits. We conduct experiments on both homophilic and heterophilic graphs. For heterophilic graphs, we adopt Wisconsin, Cornell, Texas [33], Actor, Squirrel, Crocodile, and
Table 1: Node classification accuracy (%) on heterophilic and homophilic graphs. The best and second best performance under each dataset are marked with boldface and underline, respectively.
Method	LINE	VGAE	DGI	GCA	CCA-SSG	BGRL	L-GCL	HGRL	DSSL	SP-GCL	GraphACL
Squirrel	38.92±1.58	29.13±1.16	26.44±1.12	48.09±0.21	46.76±0.36	36.22±1.97	52.94±0.88	48.31±0.65	40.51±0.38	52.10±0.67	54.05±0.13
Chameleon	48.59±1.17	42.65±1.27	60.27±0.70	63.66±0.32	62.41±0.22	64.86±0.63	68.74±0.49	65.82±0.61	66.15±0.32	65.28±0.53	69.12±0.24
Crocodile	42.21±1.12	45.72±1.53	51.25±0.51	60.73±0.28	56.77±0.39	53.87±0.65	60.18±0.43	61.87±0.45	62.98±0.51	61.72±0.21	66.17±0.24
Actor	27.55±0.32	26.99±1.56	28.30±0.76	28.77±0.29	27.82±0.60	28.80±0.54	32.55±1.18	27.95±0.30	28.15±0.31	28.94±0.69	30.03±0.13
Wisconsin	37.45±2.51	55.67±1.37	55.21±1.02	59.55±0.81	58.46±0.96	51.23±1.17	65.28±0.52	63.90±0.58	62.25±0.55	60.12±0.39	69.22±0.40
Cornell	43.68±2.17	48.73±4.19	45.33±6.11	52.31±1.09	52.17±1.04	50.33±2.29	52.11±2.37	51.78±1.03	53.15±1.28	52.29±1.21	59.33±1.48
Texas	48.69±1.39	50.27±2.21	58.53±2.98	52.92±0.46	59.89±0.78	52.77±1.98	60.68±1.18	61.83±0.71	62.11±1.53	59.81±1.33	71.08±0.34
Roman	55.42±0.87	50.89±0.96	63.71±0.63	65.79±0.75	67.35±0.61	68.66±0.39	69.74±0.53	71.84±0.41	71.70±0.54	70.88±0.35	74.91±0.28
Arxiv-year	33.21±0.13	35.11±0.25	39.26±0.72	42.96±0.39	37.38±0.41	43.02±0.62	43.92±0.52	43.71±0.54	45.80±0.57	44.11±0.35	47.21±0.39
Cora	68.25±0.31	76.30±0.21	82.30±0.60	82.93±0.42	84.00±0.40	82.70±0.60	84.00±0.35	82.52±0.31	83.51±0.42	83.16±0.13	84.20±0.31
CiteSeer	43.92±0.51	66.80±0.23	71.80±0.70	72.19±0.31	73.10±0.30	71.10±0.80	73.26±0.50	71.05±0.49	73.20±0.51	71.96±0.42	73.63±0.22
Pubmed	66.29±0.60	75.80±0.40	76.80±0.60	80.79±0.45	81.00±0.40	79.60±0.50	81.82±0.50	79.83±0.31	81.25±0.31	79.16±0.73	82.02±0.15
Computer	86.50±0.21	85.80±0.31	83.95±0.47	87.85±0.31	88.74±0.28	89.69±0.37	88.72±0.42	88.53±0.18	89.24±0.23	89.68±0.19	89.80±0.25
Photo	89.82±0.20	91.50±0.20	91.61±0.22	91.70±0.10	93.14±0.14	92.90±0.30	93.15±0.47	92.85±0.38	93.10±0.32	92.49±0.31	93.31±0.19
Arxiv	65.22±0.30	66.40±0.20	70.32±0.25	69.37±0.20	71.21±0.20	71.64±0.24	71.33±0.20	68.55±0.38	69.87±0.47	68.25±0.22	71.72±0.26
Chameleon [33, 47]. We also use two large heterophilic graphs proposed recently: Roman-empire (Roman) [48] and arXiv-year [16] ( $\sim$ 170k) nodes. For homophilic graphs, we adopt three citation graphs: Cora, Citeseer and Pubmed [49, 50], and two co-purchase graphs: Computer and Photo [5, 51]. We further include a large-scale homophilic graph Ogbn-Arxiv (Arxiv) [52]. For all datasets, we use the public and standard splits used by cited papers. Detailed descriptions, splits, and one-hop homophily and two-hop monophily statistics of datasets are given in the Appendix C.1.
Baselines. We compare GraphACL with a traditional network embedding method: LINE [14], and the recent state-of-the-art self-supervised learning methods: VGAE [9], DGI [2], GCA [4], Local (L)-GCL [12], HGRL [29], BGRL [5], CCA-SSG [6], SP-GCL [53], and DSSL [18]. The descriptions and implementation details of baselines are given in Appendix C.2.
Evaluation Protocol. We utilize node and graph classification and node clustering to evaluate the quality of the representation. For classification, we follow the standard linear evaluation protocol [2]. We train a linear classifier on top of the frozen representation and report the test accuracy. For node clustering, we perform k-means clustering on the obtained representations and set the number of clusters to the number of ground truth classes and report the normalized mutual information [54].
Setup. For all methods, we use a standard GCN model [49] as the encoder. We randomly initialize the model parameters and use the Adam optimizer to train the encoder. We run experiments with ten random seeds and report the average performance and standard deviation. For fair comparison, for all methods, we select the best configuration of hyperparameters only based on accuracy on the validation set. For baselines that did not report results in part of datasets or do not use standard public data splits [18, 29], we reproduce the results using the official code of the authors. More details on the implementation and the hyperparameter search space can be found in Appendix C.3.
6.2 Overall Performance Comparison
Table 1 reports the average node classification accuracy on both heterophilic and homophilic graphs. We provide the graph classification and node clustering results in Appendix C.6 and C.5, respectively, showing that GraphACL can also effectively adapt to various downstream tasks. Surprisingly, among all methods, our simple GraphACL achieves the best performance in 14 of 15 data sets with various homophily ratios, as shown in Table 1. Specifically, GraphACL achieves significant improvements on most of the heterophilic datasets and comparable performance on homophilic graphs compared with the second-best method. Specifically, GraphACL achieves  $4.3\%$  (Roman),  $11.6\%$  (Cornell),  $14.4\%$  (Texas),  $5.1\%$  (Crocodile), and  $3.1\%$  (Arxiv-year) relative improvement over the second-best method. We can observe that contrastive strategies (CCA-SSG, GCA, and BGRL) with augmentations can not work well on heterophilic graphs compared to homophilic graphs. This verifies that they still implicitly leverage homophily. In contrast, GraphACL does not need augmentations, and we attribute our significant improvement to modeling the one-hop neighborhood context and two-hop monophily.
6.3 Ablation Study and Sensitivity Analysis
Ablation Study. We conduct an ablation study in Table 2 to validate our motivation and design, with the following three ablations: (i) directly minimize the combination loss  $\mathcal{L}_{\mathrm{COM}}$ , (ii) Removing the
(a) Sensitivity of representation dimension  $D$
Figure 4: The effect of representation dimension, and the pair-wise similarities of randomly sampled node pairs, one-hop and two-hop neighbors. More results are provided in Appendix C.10.
(b) Cora
(c) Squirrel
asymmetric encoder architecture, (iii) Removing the uniformity loss with negative samples. We also test the combination (ii) & (iii) by removing both asymmetric architecture and uniformity loss. First, minimizing the ill-posed combination loss is a valid baseline, but can not achieve better performance and is unstable with large standard deviations. We also find that the model without uniformity loss (ablation (ii)) does not achieve the best in homophily graphs, although it does still serve as a strong ablation. When each component is individually applied, the asymmetric architecture alone achieves the very best performance in Cornell and arXiv-year. This confirms our motivation that the asymmetric architecture is of more importance for modeling neighbors in heterophilic graphs where connected nodes have different classes. The full model (last row) achieves the best performance, demonstrating that our designed components are complementary to each other.
Representation Dimension. In Figure 4 (a), we analyze the effect of the dimension of node representations. We can observe that having a large dimension can generally lead to better performance for both homophilic and heterophilic graphs. Moreover, we find that the dimension effect is more obvious in heterophilic graphs
compared to it in homophily graphs. This could be justified by our Theorems 3, which give the lower bound and upper bound of GraphACL loss in terms of the homophily ratio. Specifically, since the two-hop homophily ratio is still smaller in heterophilic graphs compared to homophilic graphs, and a small downstream error requires a small loss. Thus, a large dimension can effectively reduce the lower bound of the training loss and benefit the learning on heterophilic graphs. Training with extremely large dimensions for some graphs may lead to a slight drop of performance as GraphACL may suffer from the over-fitting issue, limiting its generalization performance.
Other Hyper-parameters. Figure 5 shows the performance with various decay rate  $\lambda$ . We observe that (i)  $\lambda$  plays an important role in GraphACL. Having a large  $\lambda$  typically improves the model performance; (ii) Instantaneously updating the target network, i.e., when  $\lambda = 0$ , destabilizes training and leads to poor performance. Thus, there is a trade-off between updating the target too often and updating it too slowly. The temperature  $\tau$  and the number of negative samples  $K$  are not included here since they are not directly relevant to our key motivation and have been employed and evaluated in the recent works [55, 56]. Thus, we provide corresponding results in Appendices C.8 and C.9.
6.4 Qualitative Analysis and Case Study
Similarity Visualization. Figures 4 (b) and (c) plot the distribution of pair-wise cosine similarities of randomly sampled nodes, one-hop neighbor, and two-hop neighbor pairs based on learned representations. We can observe that, for the homophilic graph, that is, Cora, nodes are forced to have representations similar to those of their neighbors. GraphACL enlarges the similarities between neighbor nodes compared to randomly sampled node pairs, demonstrating that GraphACL can well preserve one-hop neighborhood contexts. Compared to the homophilic graph Cora, GraphACL seeks to further pull the two-hop neighbor nodes together on the heterophilic graph, i.e., Squirrel. This observation matches our motivation and analysis that GraphACL can effectively discover the underlying two-hop monophily, which benefits the learning on heterophilic graphs. Figures 6 (a) and (b) show the cosine similarity between the v and prediction  $g_{\phi}(\mathbf{v})$  for each node. We find that the cosine similarities are typically smaller than 1, showing that the predictor  $g_{\phi}$  is not an identity matrix after convergence. Moreover, we can observe that, compared to the homophilic graph Cora, the
Table 2: Ablation studies on the node classification task.
Baseline	Cora	Pubmed	Cornell	arXiv-year
(i) minimize LCOM	83.08±0.75	81.31±0.92	58.10±2.89	45.79±0.62
(ii) w/o asymmetric encoder	82.21±0.60	81.05±0.75	57.13±0.13	45.33±0.22
(iii) w/o uniformity loss	81.39±0.20	80.56±0.35	57.87±0.27	46.54±0.28
(ii) & (iii) w/o both	80.85±0.65	77.05±0.15	42.03±1.21	42.71±0.20
GraphACL	84.20±0.31	82.02±0.15	59.33±1.48	47.21±0.39
Figure 5: The testing performance varying decay rate  $\lambda$ . More results are given in Appendix C.7.
(b) Squirrel
(c) Cora
Figure 6: (a) and (b) show the similarity between representation  $\mathbf{v}$  and prediction  $\mathbf{p}$ . (c) and (d) show case studies, where we randomly pick a node with the drastic neighborhood variations and visualize its neighborhood. Node colors denote ground-truth labels. The size of the node is proportional to its representation similarity to the central node denoted as star. See Appendix C.11 for more results.
similarities on the heterophilic graph Squirrel are smaller. This verifies that GraphACL automatically differentiates node identity and context representations, which is important for heterophilic graphs.
Case Study. In Figures 6 (c) and (d), we randomly sample the two-hop subgraph of a central node and calculate the cosine similarity based on learned representations between the central node and neighbor nodes. We can observe that GraphACL can successfully identify the node whose local neighborhood patterns are similar to the central node on both homophilic and heterophilic graphs. Additionally, our GraphACL can pull two-hop neighbor nodes, which share a similar semantic class to the central nodes, instead of favoring nearby one-hop neighbor nodes. These observations endorse our intuition that heterophilic graphs can not benefit much from one-hop neighbor nodes, and our GraphACL can effectively capture the latent semantic information related to two-hop monophily.
We propose a simple contrastive learning framework named GraphACL for homophilic and heterophilic graphs. The key idea of GraphACL is to capture both a local neighborhood context of one hop and a monophily similarity of two hops in one single objective. Specifically, we present a simple asymmetric contrastive objective for node representation learning. We provide a theoretical understanding of GraphACL, which shows that GraphACL can explicitly maximize mutual information between representations and one-hop neighborhood patterns. We show that GraphACL also implicitly aligns the two-hop neighbors and enjoys a good downstream performance. Extensive experiments on 15 graph benchmarks show that GraphACL significantly outperforms state-of-the-art methods. We hope that the straightforward nature of our approach serves as a reminder to the community to reevaluate simpler alternatives that may have been overlooked, thus inspiring future research.
This work supported by, or in part by, the National Science Foundation (NSF) under grant number IIS-1909702, the Army Research Office (ARO) under grant number W911NF21-1-0198, Department of Homeland Security (DNS) CINA under grant number E205949D, and Cisco Faculty Research Award. The findings in this paper do not necessarily reflect the view of the funding agencies.
A The Omitted Details of GraphACL
A.1 The Loss of Graph Asymmetric Contrastive Learning
We provide details of the derivations of GraphACL in Equation (5) and show that minimizing GraphACL is approximately to minimize the combination of prediction and uniformity losses.
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {A}} = - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \log \frac {\exp (\mathbf {p} ^ {\top} \mathbf {u} / \tau)}{\exp (\mathbf {p} ^ {\top} \mathbf {u} / \tau) + \sum_ {v _ {-} \in \mathcal {V}} \exp (\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau)} \\ = \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log \left(\exp \left(\mathbf {p} ^ {\top} \mathbf {u} / \tau\right) + \sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)\right) \\ \geq \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log \sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right) (9) \\ = \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log \sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right) \\ = \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log \sum_ {v _ {-} \in \mathcal {V}} \frac {\exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)}{| \mathcal {V} |} \cdot | \mathcal {V} | \\ \geq \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log | \mathcal {V} | + \sum_ {v _ {-} \in \mathcal {V}} \log \frac {\exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)}{| \mathcal {V} |} (10) \\ \stackrel {{\mathrm {c}}} {{=}} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \frac {1}{| \mathcal {V} |} \sum_ {v _ {-} \in \mathcal {V}} \log \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right) \\ \stackrel {{c}} {=} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \mathbf {p} ^ {\top} \mathbf {u} + \frac {1}{| \mathcal {V} |} \sum_ {v _ {-} \in \mathcal {V}} \mathbf {v} ^ {\top} \mathbf {v} _ {-} \\ \stackrel {\mathrm {c}} {=} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \| \mathbf {p} - \mathbf {u} \| _ {2} ^ {2} - \frac {1}{| \mathcal {V} |} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {v _ {-} \in \mathcal {V}} \| \mathbf {v} - \mathbf {v} _ {-} \| _ {2} ^ {2}, (11) \\ \end{array}
$$
where the symbol  $\stackrel{\mathrm{c}}{=}^{\mathrm{c}}$  indicates equality up to a multiplicative and/or additive constant. Here, we utilize Jensen's inequality in Equation (10). Equation (11) holds because  $\mathbf{p}$  and  $\mathbf{u}$  are both  $\ell_2$ -normalized. Given the above, we can conclude that minimizing GraphACL objective is approximately to minimizing the combination of the prediction and uniformity losses.
B Proofs in Section 5
$$
\mathcal {L} _ {A} \geq H (V | Y) - H (V) = - I (V; Y). \tag {12}
$$
Proof. According to the derivations in Appendix A.1, we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {A}} \geq \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \mathbf {p} ^ {\top} \mathbf {u} + \frac {1}{| \mathcal {V} |} \sum_ {v _ {-} \in \mathcal {V}} \mathbf {v} ^ {\top} \mathbf {v} _ {-} \\ \stackrel {c} {=} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \| \mathbf {p} - \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \mathbf {u} \| _ {2} ^ {2} - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {v _ {-} \in \mathcal {V}} \| \mathbf {v} - \mathbf {v} _ {-} \| _ {2} ^ {2} (13) \\ = \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \| \mathbf {p} - \mathbf {z} \| _ {2} ^ {2} - \frac {1}{| \mathcal {V} |} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {v _ {-} \in \mathcal {V}} \| \mathbf {v} - \mathbf {v} _ {-} \| _ {2} ^ {2}, (14) \\ \end{array}
$$
where  $\mathbf{z} = \frac{1}{\mathcal{N}(v)}\sum_{u\in \mathcal{N}(v)}\mathbf{u}$  is the mean representations from one-hop neighborhoods of node  $v$ . Equation (13) holds because that gradient of  $\mathbf{u}$  is being blocked in the target encoder with exponential moving average; thus it can be viewed as a constant that depends only on the target encoder but not the variable  $f_{\theta}$ . Since two nodes of the same semantic class tend to share similar one-hop neighborhood patterns even in non-homophilous graphs, thus we can view the neighborhood representations  $z$  is