SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where (8) follows from Lemma 4.1 with probability  $1 - o(1)$ , by our choice of  $k$  (as Lemma 4.1 guarantees that each of  $\| \widehat{X} + \widehat{Y} \|^2 = (1 \pm \epsilon) \| X + Y \|^2$  and  $\| \widehat{X} - \widehat{Y} \|^2 = (1 \pm \epsilon) \| X - Y \|^2$  holds with probability  $1 - o(1)$ , and one can take a union bound over the two events).
Lemma 4.3.  $H_{u,v} = 2M\langle \mathbf{r}_v - \mathbf{r}_u,\mathbf{r}_v - \mathbf{p}\rangle$  where  $\mathbf{p} := \sum_u\pi_uy$ .
Proof. Consider the following expression of hitting times in terms of commute times by [43].
$$
H _ {u, v} = \frac {1}{2} \left[ K _ {u, v} + \sum_ {i} \pi_ {i} \left(K _ {v, i} - K _ {u, i}\right) \right]. \tag {9}
$$
Dividing both sides of eq. (9) and using the relation  $K_{u,v} = 2M\mathrm{Res}(u,v)$ , we see that:
$$
\begin{array}{l} \frac {H _ {u , v}}{2 M} = \frac {1}{2} \left[ \operatorname {R e s} (u, v) + \sum_ {i} \pi_ {i} \left(\operatorname {R e s} (v, i) - \operatorname {R e s} (u, i)\right) \right] \\ = \frac {1}{2} \left[ \| \mathbf {r} _ {u} - \mathbf {r} _ {v} \| ^ {2} + \sum_ {i} \pi_ {i} \left(\| \mathbf {r} _ {v} - \mathbf {r} _ {i} \| ^ {2} - \| \mathbf {r} _ {u} - \mathbf {r} _ {i} \| ^ {2}\right) \right]. \tag {10} \\ \end{array}
$$
Let's focus on the inner summation. After expanding out the squared norms, we see that:
$$
\begin{array}{l} \sum_ {i} \pi_ {i} \left(\| \mathbf {r} _ {v} - \mathbf {r} _ {i} \| ^ {2} - \| \mathbf {r} _ {u} - \mathbf {r} _ {i} \| ^ {2}\right) = \sum_ {i} \pi_ {i} \left(\| \mathbf {r} _ {v} \| ^ {2} - \| \mathbf {r} _ {u} \| ^ {2}\right) \\ - 2 \sum_ {i} \pi_ {i} \left\langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {r} _ {i} \right\rangle \\ = \left(\| \mathbf {r} _ {v} \| ^ {2} - \| \mathbf {r} _ {u} \| ^ {2}\right) \\ - 2 \left\langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \sum_ {i} \pi_ {i} \mathbf {r} _ {i} \right\rangle \\ = \left(\| \mathbf {r} _ {v} \| ^ {2} - \| \mathbf {r} _ {u} \| ^ {2}\right) - 2 \langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {p} \rangle . \\ \end{array}
$$
Substituting this back into eq. (10), we can express  $\frac{1}{2M} H_{u,v}$  as:
$$
\begin{array}{l} \frac {1}{2} \left(\| \mathbf {r} _ {v} - \mathbf {r} _ {u} \| ^ {2} + \| \mathbf {r} _ {v} \| ^ {2} - \| \mathbf {r} _ {u} \| ^ {2} - 2 \langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {p} \rangle\right) \\ = \left\| \mathbf {r} _ {v} \right\| ^ {2} - \langle \mathbf {r} _ {u}, \mathbf {r} _ {v} \rangle - \langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {p} \rangle = \langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {r} _ {v} - \mathbf {p} \rangle . \\ \end{array}
$$
Proof. Using Lemma 4.3, we see that
$$
\begin{array}{l} \left| \widehat {H} _ {u, v} - H _ {u, v} \right| = 2 M \left| \left\langle \widehat {\mathbf {r}} _ {v} - \widehat {\mathbf {r}} _ {u}, \widehat {\mathbf {r}} _ {v} - \widehat {\mathbf {p}} \right\rangle - \left\langle \mathbf {r} _ {v} - \mathbf {r} _ {u}, \mathbf {r} _ {v} - \mathbf {p} \right\rangle \right| \\ \leq \epsilon M \left(\| \mathbf {r} _ {v} - \mathbf {r} _ {u} \| ^ {2} + \| \mathbf {r} _ {v} - \mathbf {p} \| ^ {2}\right) \\ \leq 3 \epsilon H _ {\max }, \\ \end{array}
$$
where we used Corollary 4.2 in the first inequality and Definition 3.4 in the last inequality.
D Comparison: Effective Resistances vs. Shortest Path Distances
Given that effective resistance (ER) captures times associated with random walks in a graph, it is tempting to ask how effective resistances compare to shortest path distances (SPDs) between nodes in a graph. Indeed, for some simple graphs, e.g., trees, shortest path distances and effective resistances turn out to be identical. However, in general, effective resistances and shortest path distances behave quite differently.
Nevertheless, it is tempting to ask how effective resistance features compare to SPD features in GNNs, especially as there have been a number of recent model architectures that make use of SPD features (e.g., Graphormer [49], Position-Aware GNNs [52], DE-GNN [29]). We first note that the most natural direct comparison of our ER-based MPNNs with SPD-based networks does not quite make sense. The reason is that the analogous comparison would be to determine the effect of replace ERs with SPDs as features in our MPNNs. However, since our networks only use ER features along edges of the given graph, the corresponding SPD features would then be trivial (as the SPD between two nodes directly connected by an edge in the graph is 1, resulting in a constant feature on every edge)!
As a result, graph learning architectures that use SPDs typically either (a.) use a densely-connected network (e.g., Graphormer [49], which uses a densely-connected attention mechanism) that incurs  $O(n^{2})$  overhead, or (b.) pick a small set of anchor nodes or landmark nodes to which SPDs from all other nodes are computed and incorporated as node features (e.g., Position-Aware GNNs [52], DE-GNN [29]). We stress that the former approach generally modifies the graph (by connecting all pairs of nodes) and therefore does not fall within the standard MPNN approach, while the latter includes architectures that fall within the MPNN paradigm.
Furthermore, we note that DE-GNNs are arguably one of the closest proposals to ours, as they compute distance-encoded features. These features can be at least as powerful as our proposed affinity-based features if polynomially many powers of the adjacency matrix are used. However, for all but the smallest graphs, using this many powers will be impractical—in fact, [29] only use powers of  $A$  up to 3, which would not be able to reliably approximate affinity-based features. We also
observe that the DE-GNN paper is concerned with learning representations of small sets of nodes (e.g., node-, link-, and triangle-prediction) and does not show how to handle graph prediction tasks, which the authors mention as possible future work. This makes a direct comparison of our methods with DE-GNNs difficult.
D.1 Empirical Results
In an effort to empirically compare the expressivity of ER features with that of SPD features, we once again perform experiments on the PNA dataset, picking the following baselines that make use of SPD features:
- The first baseline is roughly an MPNN with Graphomer-based features. More precisely, it is a densely-connected MPNN with SPDs from the original graph as edge features. In order to retain the structure of the original graph, we also use additional edge features to indicate whether or not an edge in the dense (complete) graph is a true edge of the original graph. We also explore the use of the centrality encoding (in-degree and out-degree embeddings) from Graphomer as additional node features.
- The second baseline is the Position-Aware GNN (P-GNN), which makes use of "anchor sets" of nodes and encodes distances to these nodes.
The results of these baselines are shown in Table 7. In particular, we note that our ER-based MPNNs outperform all aforementioned baselines.
Table 7: Results on the PNA dataset for MPNNs with Graphormer-based features (yellow) as well as SPD-based P-GNNs (orange). Here, CE refers to the centrality encoding, which is incorporated in the relevant MPNNs as additional node features. Similarly, SPD refers to shortest path distance features — in the relevant MPNNs, shortest path distances between all pairs of nodes in the graph are incorporated as edge features, along with an additional edge feature indicating whether an edge exists in the input graph. Therefore, the MPNN baselines are all variants of the same model with additional node/edge features. Similarly, P-GNN [52] uses SPD features with respect to a set of chosen anchor nodes. The average score metric is, as before, the average of the  $\log(MSE)$  metric over all six tasks, as in Table 1.
Model	Average score
*MPNN + CE	-2.728
*MPNN (dense) + SPD	-2.157
*MPNN (dense) + CE + SPD	-2.107
*P-GNN	-2.650
MPNN w/ resistive (edge) embeddings	-2.789
MPNN w/ all affinity measure features	-3.106
D.2 Theory: ER vs. SPD
In addition to experimental results, we would like to provide some theory for why effective resistances can capture structure in GNNs that SPDs are unable to.
We will call an initialization function  $u \mapsto \mathbf{h}_u^{(0)}$  on nodes of a graph node-based if it assigns values that are independent of the edges of the graph. Such an initialization is, however, allowed to depend on node identities (e.g., for the single-source shortest path problem from a source  $s$ , one might find it natural to define  $\mathbf{h}_s^{(0)} = 0$  and  $\mathbf{h}_u^{(0)} = +\infty$  for all  $u \neq s$ ).
Consider the task of computing "single-source effective resistances," i.e., the effective resistance from a particular node to every other node. We show that a GNN with a limited number of message passing steps cannot possibly learn single-source effective resistances, even to nearby nodes.
Theorem D.1. Suppose we fix  $k > 0$ . Then, given any node-based initialization function  $\mathbf{h}_u^{(0)}$ , it is impossible for a GNN to compute single-source effective resistances from a given node  $w$  to any nodes within a  $k$ -hop neighborhood.
More specifically, for any update rule
$$
\mathbf {m} _ {u v} ^ {(t + 1)} = \psi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, \mathbf {h} _ {v} ^ {(t)}, f _ {e} (\mathbf {x} _ {u v})\right) \tag {11}
$$
$$
\mathbf {h} _ {u} ^ {(t + 1)} = \phi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, f \left(\left\{\mathbf {m} _ {u v}: v \in \mathcal {N} (u) \right\}\right)\right) ^ {\prime}
$$
there exists a graph  $G = (V, E)$  and  $u \in V$  such that after  $k$  rounds of message passing,  $h_v^{(k)} \neq \operatorname{Res}(u, v)$  for some  $v \neq u$  within a  $k$ -hop neighborhood of  $u$ .
On the other hand, there exists an initialization with respect to which  $k$  rounds of message passing will compute the correct shortest path distances to all nodes within  $k$ -hop neighborhood.
Note that the assumption on the initialization function in the above theorem is reasonable because enabling the use of arbitrary, unrestricted functions would allow for the possibility of precomputing effective resistances in the graph and trivially incorporating them as node features, which would defeat the purpose of computing them using message-passing.
Proof. Consider the following set of graphs, each on  $4k + 1$  nodes:
Figure 2: Both of the above graphs are on  $4k + 1$  vertices, labeled  $v_{0}, v_{1}, \ldots, v_{4k}$ . The only difference is a single edge, i.e., the graph on the left has an edge between  $v_{2k}$  and  $v_{2k + 1}$ , while the one on the right does not have this edge.
Let  $V = \{v_{0}, v_{1}, \ldots, v_{4k}\}$ . The first graph  $G = (V, E)$  is a cycle, while the second graph  $G' = (V, E')$  is a path, obtained by removing a single edge from the first graph (namely, the one between  $v_{k}$  and  $v_{k+1}$ ). Suppose the edge weights are all 1 in the above graphs.
Let  $w = v_{0}$  be the source and let  $\{\mathbf{h}_v^{(0)} : v \in V\}$  be a "local" node feature initialization. Note that for any GNN (i.e., update and aggregation rules in (11), add the formal update rule somewhere), the computation tree after  $k$  rounds of message passing is identical for nodes  $v_{0}, v_{1}, \ldots, v_{k}, v_{3k+1}, v_{3k+2}, \ldots, v_{4k}$  (i.e., the nodes within the  $k$ -hop neighborhood of  $v_{0}$ ) in both  $G$  and  $G'$ . This is because the only difference between  $G$  and  $G'$  is the existence of the edge between  $v_{2k}$  and  $v_{2k+1}$ , and this edge is beyond a  $k$ -hop neighborhood centered at any one of the aforementioned nodes. Therefore, we will necessarily have that  $\mathbf{h}_{v_i}^{(k)}$  is identical in both  $G$  and  $G'$  for  $i = 1, \ldots, k, 3k + 1, 3k + 2, \ldots, 4k$ .
However, it is easy to calculate the effective resistances in both graphs. In  $G$ , we have  $\operatorname{Res}_G(v_0, v_i) = \frac{i(4k + 1 - i)}{4k + 1}$ , while in  $G'$ , we have  $\operatorname{Res}_{G'}(v_0, v_i) = \min \{i, 4k + 1 - i\}$ . Therefore,  $\operatorname{Res}_G(v_0, v_i) \neq \operatorname{Res}_{G'}(v_0, v_i)$  for all  $i = 1, 2, \dots, k, 3k + 1, 3k + 2, \dots, 4k$ .
It follows that for any  $i = 1,2,\ldots ,k,3k + 1,3k + 2,\ldots ,4k$ , the execution of  $k$  message passing steps of a GNN cannot result in  $\mathbf{h}_{v_i}^{(k)} = \operatorname {Res}(v_0,v_i)$  for both  $G$  and  $G^{\prime}$ , which proves the first claim of the theorem.
For the second part (regarding single-source shortest paths), observe that single-source shortest path distances can, indeed, be realized via aggregation and update rules for a message passing network. In particular, for  $k$  rounds of message passing, it is possible to learn shortest path distances of all nodes within a  $k$ -hop neighborhood. Specifically, for a source  $w$ , we can use the following setup: Take  $\mathbf{h}_w = 0$  and  $\mathbf{h}_u = \infty$  for all  $u \neq w$ . Moreover, for any edge  $(u,v)$ , let the edge feature  $\mathbf{x}_{uv} \in \mathbb{R}$  simply be the weight of  $(u,v)$  in the graph. Then, take the update rule (11) with  $f_e, \psi_{t+1}$  as identity functions and
$$
f _ {e} (\mathbf {x} _ {u v}) = \mathbf {x} _ {u v}
$$
$$
\psi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, \mathbf {h} _ {v} ^ {(t)}, f _ {e} (\mathbf {x} _ {u v})\right) = \mathbf {h} _ {u} ^ {(t)} + \mathbf {x} _ {u v}
$$
$$
f (S) = \min  _ {S} \{s \in S \}
$$
$$
\phi_ {t + 1} (a, b) = \min  \{a, b \}.
$$
It is clear that the above update rule simply simulates the execution of an iteration of the Bellman-Ford algorithm. Therefore,  $k$  message passing steps will simulate  $k$  iterations of Bellman-Ford, resulting in correct shortest path distances from the source  $w$  for every node within a  $k$ -hop neighborhood.