SlowGuess/NeurIPS_2023_OCR huggingface dataset:
sampled from the conditional distribution given the pseudo label  $Y$  which indicates the one-hop neighborhood pattern:  $Z|Y \sim \mathcal{N}(\mathbf{z}_Y, I)$ . As demonstrated in [57], we can interpret the first term as a conditional cross-entropy between  $V$  and another random variable  $Z$  whose conditional distribution given the pseudo-label  $Y$  indicates the one-hop neighborhood pattern:  $Z|Y \sim \mathcal{N}(\mathbf{z}_Y, I)$ :
$$
\frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \| \mathbf {p} - \mathbf {z} \| _ {2} ^ {2} \stackrel {\mathrm {c}} {=} H (V; Z | Y) = H (V | Y) + \mathcal {D} _ {K L} (V | | Z | Y) \geq H (V | Y), \tag {15}
$$
where  $H(\cdot)$  and  $\mathcal{D}_{KL}(\cdot)$  denote the entropy and KL-divergence, respectively. Minimizing the first term in Equation (14) is approximately minimizing  $H(V|Y)$ . We then inspect the second term in Equation (14). As shown in [58, 57], the second term is close to the differential entropy estimator:
$$
\frac {1}{| \mathcal {V} |} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {v _ {-} \in \mathcal {V}} \| \mathbf {v} - \mathbf {v} _ {-} \| _ {2} ^ {2} \propto H (V). \tag {16}
$$
As a result, minimizing the loss of GraphACL can be seen as a proxy for maximizing the mutual information between the representations  $V$  and the pseudo-labels  $Y$  which indicates the neighborhood patterns of one hop nodes. Thus, the proof is completed.
Theorem 2 Let  $\mathcal{N}_2(v)$  denote the set of two-hop neighbors of  $v$ . Minimizing the GraphACL objective in Equation (5) is approximately minimizing the following alignment loss between two-hop neighbors:
$$
\mathcal {L} _ {\text {t w o - h o p}} = \frac {1}{| \mathcal {V} |} \frac {1}{2 L} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} _ {2} (v) |} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2}, \tag {17}
$$
where  $L$  is the  $L$ -bi-Lipschitz constant of  $g_{\phi}$ :  $\forall (\mathbf{v}_1,\mathbf{v}_2)$ ,  $1 / L\left\| \mathbf{v}_1 - \mathbf{v}_2\right\| _2^2\leq \| g_\phi (\mathbf{v}_1) - g_\phi (\mathbf{v}_2)\|^2$ .
Proof. Given the derivations in Appendix A.1, we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {A}} \stackrel {{\mathrm {c}}} {{=}} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} - \frac {\mathbf {p} ^ {\top} \mathbf {u}}{\tau} + \log \left(\exp \left(\mathbf {p} ^ {\top} \mathbf {u} / \tau\right) + \sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)\right) \\ \geq - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \frac {\mathbf {u} ^ {\top} g _ {\phi} (\mathbf {v})}{\tau} \overset {\mathrm {c}} {=} - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \mathbf {u} ^ {\top} g _ {\phi} (\mathbf {v}), \tag {18} \\ \end{array}
$$
where the last line holds because  $v \in \mathcal{V}$ :  $\sum_{v_- \in \mathcal{V}} \exp(\mathbf{v}^\top \mathbf{v}_- / \tau) > \exp(\mathbf{v}^\top \mathbf{v}) = e > 1$ . To conduct the proof, we first formulate the degree-related predictions and representations as two matrices:  $\tilde{\mathbf{P}}$  and  $\tilde{\mathbf{U}}$ . Here,  $\tilde{\mathbf{U}}_{v_1} = \sqrt{d_u} \mathbf{u}$  is the  $u$ -th row of the matrix  $\tilde{\mathbf{U}} \in \mathbb{R}^{N \times D}$  and  $\tilde{\mathbf{P}}_v = \sqrt{d_v} \cdot g_\phi(\mathbf{v})$  is the  $v$ -th row of the matrix  $\tilde{\mathbf{P}} \in \mathbb{R}^{N \times D}$ . Recall that the normalized adjacency matrix  $\tilde{\mathbf{A}} = \mathbf{D}^{-1/2} \mathbf{A} \mathbf{D}^{-1/2}$  where  $\mathbf{D}$  is the diagonal degree matrix. With the above and Equation (18), we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {A}} \geq - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} g _ {\phi} (\mathbf {v}) ^ {\top} \mathbf {u} \\ = - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \frac {1}{\sqrt {d _ {v}} \sqrt {d _ {u}}} \sqrt {d _ {v}} g _ {\phi} (\mathbf {v}) \cdot \sqrt {d _ {u}} \mathbf {u} \\ \stackrel {\mathrm {c}} {=} - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {d _ {\mathrm {m}}}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \frac {1}{\sqrt {d _ {v}} \sqrt {d _ {u}}} \sqrt {d _ {v}} g _ {\phi} (\mathbf {v}) \cdot \sqrt {d _ {u}} \mathbf {u} \\ \geq - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {u \in \mathcal {N} (v)} \frac {1}{\sqrt {d _ {v}} \sqrt {d _ {u}}} \sqrt {d _ {v}} g _ {\phi} (\mathbf {v}) \cdot \sqrt {d _ {u}} \mathbf {u} \\ = - \frac {1}{| \mathcal {V} |} \operatorname {t r} \left(\tilde {\mathbf {A}} \tilde {\mathbf {P}} \tilde {\mathbf {U}} ^ {\top}\right) \geq - \frac {1}{| \mathcal {V} |} \frac {1}{2} \left(\left\| \tilde {\mathbf {A}} \tilde {\mathbf {P}} \right\| _ {2} ^ {2} + \left\| \tilde {\mathbf {U}} ^ {\top} \right\| _ {2} ^ {2}\right) (19) \\ = - \frac {1}{| \mathcal {V} |} \frac {1}{2} \operatorname {t r} \left(\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top} \tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}\right) - \frac {1}{| \mathcal {V} |} \frac {1}{2} \sum_ {u \in \mathcal {V}} d _ {u} \| \mathbf {u} \| _ {2} ^ {2} \stackrel {{\mathrm {c}}} {=} - \frac {1}{| \mathcal {V} |} \frac {1}{2} \operatorname {t r} \left(\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top} \tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}\right), (20) \\ \end{array}
$$
where  $d_{\mathrm{m}}$  is the maximum degree of nodes and  $\operatorname{tr}(\cdot)$  denotes the matrix trace. In Equation (19), we utilize the inequality:  $\operatorname{tr}(\mathbf{PQ}) \leq \frac{1}{2}\left(\|\mathbf{P}\|_2^2 + \|\mathbf{Q}\|_2^2\right)$  for any two matrices  $\mathbf{P} \in \mathbb{R}^{m \times n}$  and  $\mathbf{Q} \in \mathbb{R}^{n \times t}$ . Equation (20) holds as  $\mathbf{u}$  is normalized. Since the trace is the sum of elements on the main diagonal
of the matrix,  $\mathbf{A}$ , is:
$$
\begin{array}{l} (\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top} \tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}) _ {v, v} = \sum_ {u _ {2}} (\tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}) _ {v, u _ {2}} (\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top}) _ {u _ {2}, v} = \sum_ {u _ {2} \in \mathcal {V}} \sum_ {u \in \mathcal {N} (v) \cap \mathcal {N} (u _ {2})} \tilde {\mathbf {A}} _ {v, u} ^ {T} \tilde {\mathbf {A}} _ {u, u _ {2}} \sqrt {d _ {v}} \sqrt {d _ {u _ {2}}} \mathbf {p} _ {v} ^ {\top} \mathbf {p} _ {u _ {2}} \\ = \sum_ {u \in \mathcal {N} (v)} \sum_ {u _ {2} \in \mathcal {N} (u)} \frac {1}{d _ {u}} \frac {1}{\sqrt {d _ {v}} \sqrt {d _ {u _ {2}}}} \sqrt {d _ {v}} \sqrt {d _ {u _ {2}}} \mathbf {p} _ {v} ^ {\top} \mathbf {p} _ {u _ {2}} \\ = \sum_ {u \in \mathcal {N} (v)} \sum_ {u _ {2} \in \mathcal {N} (u)} \frac {1}{| \mathcal {N} (u) |} \mathbf {p} _ {v} ^ {\top} \mathbf {p} _ {u _ {2}} \stackrel {{\mathrm {c}}} {{=}} - \sum_ {u \in \mathcal {N} (v)} \sum_ {u _ {2} \in \mathcal {N} (u)} \frac {1}{| \mathcal {N} (u) |} \| \mathbf {p} _ {v} - \mathbf {p} _ {u _ {2}} \| _ {2} ^ {2} \\ = - \sum_ {u \in \mathcal {N} (v)} \sum_ {u _ {2} \in \mathcal {N} (u)} \frac {1}{| \mathcal {N} (u) |} \| g _ {\phi} (\mathbf {v}) - g _ {\phi} (\mathbf {u} _ {2}) \| _ {2} ^ {2} \\ \leq - \frac {1}{L} \sum_ {u _ {2} \in \mathcal {N} (u)} \sum_ {u \in \mathcal {N} (v)} \frac {1}{| \mathcal {N} (u) |} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2}, \tag {21} \\ \end{array}
$$
where Equation (21) holds is because the decoder is  $L$ -bi-Lipschitz:  $\forall (\mathbf{x}_1,\mathbf{x}_2), 1 / L\| \mathbf{x}_1 - \mathbf{x}_2\|^2 \leq \left\| g_{\phi}(\mathbf{x}_1) - g_{\phi}(\mathbf{x}_2)\right\|^2$ . Combining Equations (18), (20) and (21), we have:
$$
\begin{array}{l} \min _ {\theta , \phi} \mathcal {L} _ {\mathrm {A}} \Rightarrow \min _ {\theta , \phi} - \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} (v) |} \sum_ {u \in \mathcal {N} (v)} \mathbf {u} ^ {\top} g _ {\phi} (\mathbf {v}) \\ \Rightarrow \min  _ {\theta , \phi} - \frac {1}{| \mathcal {V} |} \frac {1}{2} \operatorname {t r} \left(\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top} \tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}\right) \Leftrightarrow \min  _ {\theta , \phi} - \frac {1}{| \mathcal {V} |} \frac {1}{2} \sum_ {v \in \mathcal {V}} \left(\tilde {\mathbf {P}} \tilde {\mathbf {P}} ^ {\top} \tilde {\mathbf {A}} ^ {\top} \tilde {\mathbf {A}}\right) _ {v, v} \\ \Rightarrow \min _ {\theta , \phi} \frac {1}{2 L} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {u \in \mathcal {N} (v)} \frac {1}{| \mathcal {N} (u) |} \sum_ {u _ {2} \in \mathcal {N} (u)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2} \\ \Leftrightarrow \min  _ {\theta , \phi} \frac {1}{2 L} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {u \in \mathcal {N} (v)} \frac {d _ {\mathrm {m}}}{| \mathcal {N} (u) |} \sum_ {u _ {2} \in \mathcal {N} (u)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2} \\ \Rightarrow \min  _ {\theta , \phi} \frac {1}{2 L} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2} \\ \Rightarrow \min  _ {\theta , \phi} \frac {1}{2 L} \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \frac {1}{\left| \mathcal {N} _ {2} (v) \right|} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \| \mathbf {v} - \mathbf {u} _ {2} \| _ {2} ^ {2}, \tag {22} \\ \end{array}
$$
where  $\min_{\theta}f_1\Rightarrow \min_{\theta}f_2$  represents that minimizing function  $f_{1}$  with respect to  $\theta$  is equivalent to maximizing function  $f_{2}$  with respect to  $\theta$ . Thus, the proof is completed.
Theorem 3 Let  $\hat{h}_2 = \mathcal{H}(\mathcal{G}_2)$  be the homophily ratio of the two-hop graph  $\mathcal{G}_2$ . Suppose that the downstream task is the  $M$ -categorical linear classification and the class is balanced. Then,  $\forall q \in$  the hypothesis class with  $q = g \circ f$ , the upper bound for the classification error on the optimal  $\mathbf{v}^*$  is:
$$
P \left(y _ {v} \neq p _ {W} \left(\mathbf {v} ^ {*}\right)\right) \leq 4 M ^ {2} \left(4 L \mathcal {L} _ {A} (q) + \left(1 - \hat {h} _ {2}\right)\right) + c o n s t, \tag {23}
$$
Proof. We define the adjacency matrix of the two-hop graph of  $G$  as  $\mathbf{A}_2$ , i.e.,  $(\mathbf{A}_2)_{ik} = 1$  if there exists node  $j$  such that  $e_{ij} \in \mathcal{E}$  and  $e_{jk} \in \mathcal{E}$ , otherwise  $(\mathbf{A}_2)_{ik} = 0$ . The normalized form of  $\mathbf{A}_2$  is  $\hat{\mathbf{A}}_2 = \hat{\mathbf{D}}^{-1/2}\mathbf{A}_2\hat{\mathbf{D}}_2$  where  $\hat{\mathbf{D}}_2$  is the diagonal degree matrix of the two-hop graph.  $\mathcal{N}_2(v)$  is the set of the neighbors of  $v$  on the two-hop graph. We also define the label matrix of all nodes as  $\mathbf{Y} \in \mathbb{R}^{N \times M}$  and the representation matrix  $\mathbf{V} \in \mathbb{R}^{M \times D}$  with the  $v$ -th node representation as  $\mathbf{V}_v = \mathbf{v}$ . Let  $c_i$  be the number of nodes belonging to class  $i$ . For any class  $i$ , we have  $c_i = c = \frac{|V|}{M}$  since we assume the ideal balanced setting for simplicity. However, our theoretical results can be easily extended to unbalanced settings by explicitly considering the shift of label distributions [46]. Given the above definitions, we have the following:
$$
\begin{array}{l} \mathbb {E} _ {v, \mathbf {y} _ {v}} \| \mathbf {y} _ {v} - \mathbf {W v} \| _ {2} ^ {2} \simeq \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \| \mathbf {y} _ {v} - \mathbf {W v} \| _ {2} ^ {2} = \frac {1}{| \mathcal {V} |} \| \mathbf {Y} - \mathbf {W W} ^ {\top} \| _ {2} ^ {2}, \\ = \frac {1}{| \mathcal {V} |} \| \mathbf {Y} - \mathbf {A} _ {2} \mathbf {C} + \mathbf {A} _ {2} \mathbf {C} - \mathbf {V} \mathbf {W} ^ {\top} \| _ {2} ^ {2}, \\ \leq \frac {1}{| \mathcal {V} |} \left(2 \| \mathbf {Y} - \mathbf {A} _ {2} \mathbf {C} \| _ {2} ^ {2} + 2 \| \mathbf {A} _ {2} \mathbf {C} - \mathbf {V} \mathbf {W} ^ {\top} \| _ {2} ^ {2}\right), \tag {24} \\ \end{array}
$$
where  $\mathbf{C}_{v,i} = c_i^{-1}\mathbb{1}_{y_v = i}$  and  $\mathbb{1}$  is the indicator function.  $y_{v}$  is the label index of node  $v$  and  $\mathbf{y}_v$  is the one-hot vector of label  $y_{v}$ . The last line holds because  $\| \mathbf{P} + \mathbf{Q}\| _2^2 < 2\| \mathbf{P}\| _2^2 +2\| \mathbf{Q}\| _2^2$  for any
two matrices  $\mathbf{P}$  and  $\mathbf{Q}$  of the same dimension. The  $(v,i)$ -th elements of matrices  $\mathbf{Y}$  and  $\mathbf{A}_2\mathbf{C}$  are:  $\mathbf{Y}_{v,i} = \mathbb{1}_{y_v = i}$  and  $(\mathbf{A}_2\mathbf{C})_{v,i} = \sum_{u_2\in \mathcal{N}_2(v)}c_i^{-1}\mathbb{1}_{y_{u_2} = i}$ . If  $y_{v} = i$ , we have:
$$
\left. \left(\mathbf {Y} - \mathbf {A} _ {2} \mathbf {C}\right) _ {v, i} = \mathbb {1} _ {y _ {v} = i} - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i} = 1 - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}, \right. \tag {25}
$$
Similarly, for  $y_{v}\neq i$  , we have:
$$
\left(\mathbf {Y} - \mathbf {A} _ {2} \mathbf {C}\right) _ {v, i} = \mathbb {1} _ {y _ {v} = i} - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i} = - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}. \tag {26}
$$
Given the above, we further have:
$$
\begin{array}{l} \| \mathbf {Y} - \mathbf {A} _ {2} \mathbf {C} \| _ {2} ^ {2} = \sum_ {v} \| (\mathbf {Y} - \mathbf {A} _ {2} \mathbf {C}) _ {v} \| _ {2} ^ {2} = \sum_ {v} \sum_ {i = 1} ^ {M} (\mathbf {Y} - \mathbf {A} _ {2} \mathbf {C}) _ {v, i} ^ {2} \\ = \sum_ {v} \sum_ {i = 1} ^ {M} \mathbb {1} _ {y _ {v} = i} (1 - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}) ^ {2} + (1 - \mathbb {1} _ {y _ {v} = i}) (- \sum_ {v _ {u} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}) ^ {2} \\ = \sum_ {v} (1 - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{C _ {i}} \mathbb {1} _ {y _ {u _ {2}} = y _ {v}}) ^ {2} + \sum_ {i = 1} ^ {M} \mathbb {1} _ {y _ {v} \neq i} (\sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{C _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}) ^ {2} \\ \leq \sum_ {v} (1 - \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{C _ {i}} \mathbb {1} _ {y _ {u _ {2}} = y _ {v}}) ^ {2} + (\sum_ {i = 1} ^ {M} \mathbb {1} _ {y _ {v} \neq i} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{C _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}) ^ {2} \\ = \sum_ {v} \left(1 - \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = y _ {v}}\right) ^ {2} + \left(\sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \left(\sum_ {i = 1} ^ {M} \mathbb {1} _ {y _ {v} \neq i} \frac {1}{c _ {i}} \mathbb {1} _ {y _ {u _ {2}} = i}\right)\right) ^ {2} \\ = \sum_ {v} 1 + \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \left(\sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \left(\sum_ {i = 1} ^ {M} \mathbb {1} _ {y _ {v} \neq i} \mathbb {1} _ {y _ {u _ {2}} = i}\right)\right) ^ {2} \\ = \sum_ {v} 1 + \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \left(\sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \mathbb {1} _ {y _ {u _ {2}} \neq y _ {v}}\right) ^ {2} \leq | \mathcal {V} | + \frac {M M}{| \mathcal {V} |} \sum_ {v} \sum_ {u _ {2} \in \mathcal {N} _ {2} (v)} \mathbb {1} _ {y _ {u _ {2}} \neq y _ {v}}. \tag {27} \\ \end{array}
$$
For the second term  $\| \mathbf{A}_2\mathbf{C} - \mathbf{V}\mathbf{W}^\top \|_2^2$  in Equation (24), we have:
$$
\begin{array}{l} \left\| \mathbf {A} _ {2} \mathbf {C} - \mathbf {V} \mathbf {W} ^ {\top} \right\| _ {2} ^ {2} = \left\| \left(\mathbf {A} _ {2} - \mathbf {V} \mathbf {V} ^ {\top} + \mathbf {V} \mathbf {V} ^ {\top}\right) \mathbf {C} - \mathbf {V} \mathbf {W} ^ {\top} \right\| _ {2} ^ {2} (28) \\ = \left\| \left(\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}\right) \mathbf {C} + \mathbf {V} \left(\mathbf {V} ^ {\top} \mathbf {C} - \mathbf {W} ^ {\top}\right) \right\| _ {2} ^ {2} \\ \leq 2 \| \left(\mathbf {A} _ {2} - \mathbf {V} \mathbf {V} ^ {\top}\right) \mathbf {C} \| _ {2} ^ {2} + 2 \| \mathbf {V} \left(\mathbf {V} ^ {\top} \mathbf {C} - \mathbf {W} ^ {\top}\right) \| _ {2} ^ {2} \\ \leq 2 \| \left(\mathbf {A} _ {2} - \mathbf {V} \mathbf {V} ^ {\top}\right) \| _ {2} ^ {2} \| \mathbf {C} \| _ {2} ^ {2} + 2 \| \mathbf {V} \| _ {2} ^ {2} \| \left(\mathbf {V} ^ {\top} \mathbf {C} - \mathbf {W} ^ {\top}\right) \| _ {2} ^ {2} \\ \leq 2 \| \left(\mathbf {A} _ {2} - \mathbf {V} \mathbf {V} ^ {\top}\right) \| _ {2} ^ {2} \| \mathbf {C} \| _ {2} ^ {2} + 2 \| \mathbf {V} \| _ {2} ^ {2} \| \left(\mathbf {V} ^ {\top} \mathbf {C} - \mathbf {W} ^ {\top}\right) \| _ {2} ^ {2} \\ = 2 \| \left(\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}\right) \| _ {2} ^ {2} \| \mathbf {C} \| _ {2} ^ {2} (29) \\ = 2 \frac {M M}{| \mathcal {V} |} \| \left(\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}\right) \| _ {2} ^ {2}, (30) \\ \end{array}
$$
where Equation (29) holds because the  $i_{\mathrm{th}}$  row of  $\mathbf{W}$  is the mean of representations of nodes with label  $i$ :  $\| (\mathbf{V}^{\top}\mathbf{C} - \mathbf{W}^{\top})\| _2^2 = 0$ . Combining Equations (24), (27), and (30), we have:
$$
\begin{array}{l} \mathbb {E} _ {v, \mathbf {y} _ {v}} \| \mathbf {y} _ {v} - \mathbf {W v} \| _ {2} ^ {2} \leq \frac {1}{| \mathcal {V} |} \left(2 \| \mathbf {Y} - \mathbf {A} _ {2} \mathbf {C} \| _ {2} ^ {2} + 2 \| \mathbf {A} _ {2} \mathbf {C} - \mathbf {V W} ^ {\top} \| _ {2} ^ {2}\right) \\ \leq 4 \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \| (\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}) \| _ {2} ^ {2} + 2 \left(1 + \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \sum_ {v} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \mathbb {1} _ {y _ {v _ {2}} \neq y _ {v}}\right) \\ \leq 4 \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \| (\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}) \| _ {2} ^ {2} + 2 (1 + \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \sum_ {v} \frac {| \mathcal {V} |}{| \mathcal {N} _ {2} (v) |} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \mathbb {1} _ {y _ {v _ {2}} \neq y _ {v}}), \\ \leq 4 \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \| (\mathbf {A} _ {2} - \mathbf {V V} ^ {\top}) \| _ {2} ^ {2} + 2 M ^ {2} (1 - \hat {h} _ {2}) + 2, \tag {31} \\ \end{array}
$$
where  $\hat{h}_2 = \mathcal{H}(\mathcal{G}_2)$  is the homophily ratio of the two-hop graph. Based on the derivations in Theorem 2 and Equation (9), we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {A}} (q) \geq \frac {1}{| \mathcal {V} |} \frac {1}{2 L} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} _ {2} (v) |} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \| \mathbf {v} - \mathbf {v} _ {2} \| _ {2} ^ {2} + \log \left(\sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)\right) \\ \geq - \frac {1}{| \mathcal {V} |} \frac {1}{2 L} \sum_ {v \in \mathcal {V}} \frac {1}{| \mathcal {N} _ {2} (v) |} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \log \frac {\exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {2} / \tau\right)}{\sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)} (32) \\ \geq - \frac {1}{| \mathcal {V} | | \mathcal {V} |} \frac {1}{2 L} \sum_ {v \in \mathcal {V}} \sum_ {v _ {2} \in \mathcal {N} _ {2} (v)} \log \frac {\exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {2} / \tau\right)}{\sum_ {v _ {-} \in \mathcal {V}} \exp \left(\mathbf {v} ^ {\top} \mathbf {v} _ {-} / \tau\right)} \triangleq \frac {1}{| \mathcal {V} | | \mathcal {V} |} \frac {1}{2 L} \mathcal {L} _ {\text {S i m C L R}} (\mathbf {V}). (33) \\ \end{array}
$$
We can find the last line is exactly the SimCLR-style loss [55] over two-hop graphs. Recent work [59] proves that finding the global optimum of the un-normalized SimCLR-style objective  $\mathcal{L}_{\mathrm{SimCLR}}(\mathbf{V})$  is equivalent to solving the matrix factorization problem:  $\min_{\mathbf{V}}\| \mathbf{A}_2 - \mathbf{V}\mathbf{V}^T\| _2^2$ . Given Equations (31) and (33), we further have:
$$
\begin{array}{l} P \left(y _ {v} \neq p _ {W} \left(\mathbf {v} ^ {*}\right)\right) \leq 2 \mathbb {E} _ {v, \mathbf {y} _ {v}} \| \mathbf {y} _ {v} - \mathbf {W} \mathbf {v} ^ {*} \| _ {2} ^ {2} (34) \\ \leq 8 \frac {M M}{| \mathcal {V} | | \mathcal {V} |} \| (\mathbf {A} _ {2} - \mathbf {V} ^ {*} \mathbf {V} ^ {* ^ {\top}}) \| _ {2} ^ {2} + 4 M ^ {2} (1 - \hat {h} _ {2}) + 4 \\ = 8 \frac {M ^ {2}}{| \mathcal {V} | | \mathcal {V} |} \mathcal {L} _ {\text {S i m C L R}} \left(\mathbf {V} ^ {*}\right) + 4 M ^ {2} \left(1 - \hat {h} _ {2}\right) + 4 + \alpha \\ \leq 1 6 M ^ {2} L \mathcal {L} _ {\mathrm {A}} \left(q ^ {*}\right) + 4 M ^ {2} \left(1 - \hat {h} _ {2}\right) + \beta \\ \leq 4 M ^ {2} \left(4 L \mathcal {L} _ {\mathrm {A}} (q) + \left(1 - \hat {h} _ {2}\right)\right) + \beta . (35) \\ \end{array}
$$
Here, we convert the mean-squared error bound to classification error bound in Equation (34) as shown by Claim B.9 in [60]. The  $\alpha$  is the constant indicating the gap between the optimal loss value of SimCLR-style and matrix factorization losses [59] and  $\beta = \alpha + 4$ .
C.1 Datasets Details and Statistics
Table 3: Statistics of used homophilic and heterophilic graph datasets in this paper.
Dataset	#Nodes	#Edges	#Classes	#Features	H(G)	H(G2)	S(G)
Cora	2,708	5,278	7	1,433	0.81	0.71	0.89
Citeseer	3,327	4,552	6	3,703	0.74	0.56	0.81
Pubmed	19,717	44,324	3	500	0.80	0.74	0.87
Photo	7,650	119,081	8	745	0.83	0.66	0.91
Computer	13,752	574,418	10	767	0.78	0.55	0.90
Arxiv	169,343	2,332,386	40	128	0.66	0.61	0.79
Texas	183	309	5	1,793	0.11	0.54	0.79
Cornell	183	295	5	1,703	0.30	0.40	0.40
Wisconsin	251	466	5	1,703	0.21	0.42	0.42
Chameleon	2,277	36,101	5	2,325	0.23	0.35	0.67
Squirrel	5,201	216,933	5	2,089	0.22	0.22	0.73
Crocodile	11,631	360,040	5	2,089	0.25	0.30	0.71
Actor	7,600	33,544	5	931	0.22	0.21	0.68
Roman	22,662	32,927	18	300	0.05	0.67	0.59
Arxiv-year	169,343	1,166,243	5	128	0.22	0.58	0.77
C.1.1 One-hop Homophily Levels
We utilize the following edge homophily ratio [15] to measure the one-hop neighbor homophily of the graph. Specifically, the edge homophily ratio  $\mathcal{H}(\mathcal{G})$  is the proportion of edges that connect two nodes of the same class:
$$
\mathcal {H} (\mathcal {G}) = \frac {\left| \left\{(u , v): (u , v) \in \mathcal {E} \wedge y _ {u} = y _ {v} \right\} \right|}{| \mathcal {E} |} \tag {36}
$$
C.1.2 Neighborhood Context Similarity
To justify our first intuition that two nodes of the same semantic class tend to share similar one-hop neighborhood patterns even in heterophilic graphs, we consider if the same label shares similar one-hop neighborhood distributions of labels in the neighborhoods regardless of the homophily. To measure this property, we calculate the class neighborhood similarity [24] defined as follows:
$$
s \left(m, m ^ {\prime}\right) = \frac {1}{\left| \mathcal {V} _ {m} \| \mathcal {V} _ {m ^ {\prime}} \right|} \sum_ {u \in \mathcal {V} _ {m}, v \in \mathcal {V} _ {m ^ {\prime}}} \cos (d (u), d (v)), \tag {37}
$$
where  $M$  is the number of classes,  $\mathcal{V}_m$  is the set of nodes with class  $m$ , and  $d(u)$  is the empirical label histogram of node  $u$ 's neighbors over  $M$  classes.  $\cos(\cdot)$  represents the cosine similarity function. This cross-class neighborhood similarity measures the neighborhood distributions between different classes. When  $m = m'$ ,  $s(m, m')$  calculates the intra-class similarity. To measure the neighborhood similarity, we average the intra-class similarities of all classes:
$$
\mathcal {S} (\mathcal {G}) = \sum_ {m = 1} ^ {M} \frac {1}{M} s (m, m). \tag {38}
$$
If nodes with the same label share the same neighborhood distributions, the class neighborhood similarity  $\mathcal{H}_s(\mathcal{G})$  is high.
C.1.3 Two-hop Monophily Levels