SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 4.2. For any fixed vectors  $\alpha, \beta \in \mathbb{R}^n$ , if we let  $X := \sum_{i} \alpha_i x_i$ ,  $\widehat{X} := \sum_{i} \alpha_i \widehat{x}_i$  and similarly  $Y := \sum_{i} \beta_i x_i$ ,  $\widehat{Y} := \sum_{i} \beta_i \widehat{x}_i$ ; then:
$$
\left| \langle X, Y \rangle - \langle \widehat {X}, \widehat {Y} \rangle \right| \leq \frac {\epsilon}{2} \left(\| X \| ^ {2} + \| Y \| ^ {2}\right).
$$
Therefore, we can choose a desired  $\epsilon > 0$  and  $k = O(\log(n) / \epsilon^2)$  and instead use  $\widehat{\mathbf{r}}: V \to \mathbb{R}^k$  as the embedding, where
$$
\widehat {\mathbf {r}} _ {v} = \frac {1}{\sqrt {k}} \Pi B L ^ {\dagger} e _ {v} \tag {1}
$$
for a randomly chosen  $k \times d$  matrix  $\Pi$  whose entries are i.i.d. Gaussians from  $\mathcal{N}(0,1)$ . Then, by lemma 3.2 and lemma 4.1, we have that for every edge  $(u,v) \in E$ ,
$$
\left\| \widehat {\mathbf {r}} _ {u} - \widehat {\mathbf {r}} _ {v} \right\| _ {2} ^ {2} = (1 \pm 3 \epsilon) \operatorname {R e s} (u, v)
$$
with probability at least  $1 - \frac{1}{n^2}$ .
So the computation of random embeddings,  $\widehat{\mathbf{r}}$ , requires solving  $O((n + m)\log n / \epsilon^2)$  many Laplacian linear systems. By using one of the nearly linear time Laplacian solvers [26], we can compute the random embeddings in the near-linear time. Hence the total running time becomes  $O\left((n + m)\log^{3 / 2}npoly\log \log n / \epsilon^2\right)$ .
4.2 Fast Computation of Hitting Times
Note that it is not clear whether there is a fast method for computing hitting times similar to commute times / effective resistances. The naive approach involves solving a linear system for each edge, resulting in a running time of at least  $\Omega (nm)$ , which is prohibitive. One of our technical contributions in this paper is a novel method for fast computation of hitting times. In particular, we will show how to use the approximate effective resistance embeddings,  $\widehat{\mathbf{r}}$ , to obtain an estimate for hitting times with additive error. Proofs can be found in Appendix C.
Let  $\widehat{\mathbf{p}} \coloneqq \sum_{u} \pi_u \widehat{\mathbf{r}}_u$ . Just like  $\widehat{\mathbf{r}}$  being an approximation of  $\mathbf{r}$ ,  $\widehat{\mathbf{p}}$  is an approximation of  $\mathbf{p}$ . Consider the following quantity:
$$
\widehat {H} _ {u, v} = 2 M \left\langle \widehat {\mathbf {r}} _ {v} - \widehat {\mathbf {r}} _ {u}, \widehat {\mathbf {r}} _ {v} - \widehat {\mathbf {p}} \right\rangle . \tag {2}
$$
We will use this quantity as an approximation of  $H_{u,v}$ . In the following part, we will bound the difference between  $H_{u,v}$  and  $\widehat{H}_{u,v}$ . Our starting point will be expressing  $H_{u,v}$  in terms of the effective resistance embeddings.
Lemma 4.3.  $H_{u,v} = 2M\langle \mathbf{r}_v - \mathbf{r}_u,\mathbf{r}_v - \mathbf{p}\rangle$  where  $\mathbf{p} := \sum_u\pi_ura_v$
Given Lemma 4.3, we can establish the desired additive approximation property of  $\widehat{H}_{u,v}$ .
Lemma 4.4.  $|\widehat{H}_{u,v} - H_{u,v}| \leq 3\epsilon H_{\max}$ .
As previously discussed, our empirical evaluation seeks to show benefits from endowing standard expressive GNNs with additional affinity-based features. All architectures we experiment with will therefore conform to the MPNN blueprint [15], which we describe in Appendix B. When relevant, we may also note results that a particular strong baseline (e.g., DGN [4], Graphormer [48]) achieves on a dataset of interest. Note that these baselines modulate the message passing procedure rather than appending features and are hence a different category to our method—their performance is provided for indicative reasons only. Where appropriate, we use "DGN (features)" to refer to an MPNN that uses eigenvector flows as additional edge features, without modulating the mechanism. We also use "ER (GNN)" to refer to the use of scalar effective resistance features.
5.1 PNA dataset
The first dataset we explore is the PNA dataset [9], which captures a multimodal setting. This consists of a collection of node tasks, i.e. (1) Single-source shortest paths, (2) Eccentricity and (3) Laplacian features, as well as graph tasks, i.e. (4) Connectivity, (5) Diameter and (6) Spectral radius. PNA dataset is a set of structured graph tasks, and it complements our other datasets. As we can see in Table 1, even adding a single feature, effective resistance (ER GNN), yields the best average score compared to other models. Using hitting times as edge features improve upon effective resistances. However once we combine all ER features, which include effective resistances, hitting times as well as node and edge embeddings, we get the best scores. On these structured tasks, we can see that the affinity based measures provide a significant advantage.
We also note that incorporating affinity measures as additional features to the "DGN (features)" MPNN model also provides improvement over the standard "DGN (features)" baseline. We have accordingly included the results of a model that uses (both node and edge) ER embeddings and hitting time features along with DGN features.
5.2 Small molecule classification: ogbg-molhiv
The ogbg-molhiv dataset is a molecular property prediction dataset comprised of molecular graphs without spatial information (such as atom coordinates). Each graph corresponds to a molecule, with nodes representing atoms and edges representing chemical bonds. Each node has an associated 9-dimensional feature, containing atomic number and chirality, as well as other additional atom features such as formal charge and whether the atom is in the ring or not. The goal is to predict whether a molecule inhibits HIV virus replication or not. Our results are given in Table 2.
On this dataset, effective resistances provide an improvement over the standard MPNN. We achieve the best performance using ER node embeddings and hitting times with random rotations. With these features, our network achieves  $79.13\% \pm 0.358$  test accuracy, which is close to DGN.
Note that we provide GSN [6], HIMP [13], and GRWNN [35] as baselines. HIMP and GSN exploit structural/topological properties; the former is specifically designed for learning on molecular graphs, while the latter incorporates graph substructures. In addition to the comparison to GSN as a baseline, we additionally train an MPNN with substructure counts as additional features, namely counts of  $k$ -cycles ( $k = 3, 4$ ) and  $k$ -paths ( $k = 3$ ) as motifs (in the same vein as the approach of [6], obtaining a test ROC-AUC of  $76.41\% \pm 0.42$  (note that this improves on the base MPNN test ROC-AUC of  $74.67\% \pm 0.19$ ). Furthermore, if we also add affinity measures (in addition to the aforementioned substructure counts), we obtain a test ROC-AUC of  $78.50\% \pm 0.51$ . This suggests that affinity measures provide a performance or expressivity lift beyond that of substructure counts.
Table 2: Test % AUC-ROC on the Molt-HIV dataset. Our results are averaged over five seeds.
MolHIV
Test % ROC-AUC
GCN	76.06 ± 0.97
GIN	75.58 ± 1.40
MPNN	74.67 ± 0.19
DGN	79.70 ± 0.97
GSN (GIN + VN)	77.99 ± 1.00
HIMP	78.80 ± 0.82
GRWNN	78.38 ± 0.99
ER GNN	77.75 ± 0.426
ER (node) embed. + HT	79.13 ± 0.358
(with random rotations)
5.3 Multi-task molecular classification: ogbg-molpcba
The ogbg-molpcba dataset comprises molecular graphs without spatial information (such as atom coordinates). The aim is to classify them across 128 different biological activities. We follow the baseline MPNN architecture from [17], including the use of Noisy Nodes.
Mirroring the evaluation protocol of [17], Table 3 compares the performance of incorporating ER and hitting time (HT) features into the baseline MPNN models with Noisy Nodes, at various depths. What can be noticed is that models utilising affinity-based features are capable of reaching as well as exceeding peak test performance (in terms of mean average precision). However, what's important is the effect of these features at lower depths: it is possible to achieve comparable or better levels of performance with half the layers, when utilising ER or HT features. This result illustrates the potential benefit affinity-based computations can have on molecular benchmarks, especially when no spatial geometry is provided as input.
Table 3: ogbg-molpcba performance for various model depths. Best performance across all models is underlined. NN refers to Noisy Nodes.
Model	Test Mean Average Precision
4 layers	8 layers	16 layers
MPNN [17]	27.75% ± 0.20	27.91% ± 0.22	27.64% ± 0.25
MPNN + NN	27.92% ± 0.11	28.07% ± 0.14	28.29% ± 0.13
MPNN + NN + ER (ours)	28.11% ± 0.19	28.27% ± 0.17	28.28% ± 0.14
MPNN + NN + HT (ours)	28.03% ± 0.15	28.32% ± 0.13	28.20% ± 0.19
Table 4: ogbg-molpcba performance compared to other baselines.
Model	Test Mean AP
GCN (w/ virtual node)	24.24 ± 0.34
GIN (w/ virtual node)	27.03 ± 0.23
DeeperGCN	27.81 ± 0.38
HIMP	27.39 ± 0.17
MPNN + NN + HT (ours)	28.32% ± 0.13
5.4 Scaling to larger graphs
Next, we present results on ogbn-arxiv and ogbn-mag, transductive datasets with large graphs.
5.4.1 Large citation network: ogbn-arxiv
Most expressive GNNs that rely on computation of structural features have not been scaled beyond small molecular datasets (such as the ones discussed in prior sections). This is due to the fact that computing them requires (time or storage) complexity which is at least quadratic in the graph size—making them inapplicable even for modest-sized graphs. This is, however, not the case for our proposed affinity-based metrics. We demonstrate this by scalably computing them on a larger-scale node classification benchmark, ogbn-arxiv (a citation network with the goal of predicting the arXiv category of each paper). ogbn-arxiv has 169,343 nodes and 1,166,243 edges, making quadratic approaches infeasible. Using the combinatorial multigrid preconditioner [27, 25], we constructed the effective resistances on this graph in an hour on a standard MacBook Pro 2019 laptop.
As MPNN models overfit this transductive dataset quite easily, the dominant approach to tackling it are graph attention networks (GATs) [44]. Accordingly, we trained a simple four-layer GAT on this dataset, achieving  $72.02\% \pm 0.05$  test accuracy. This compares with  $71.97\% \pm 0.24$  reported for a related attentional baseline on the leaderboard [54], indicating that our baseline performance is relevant.
ER embeddings on ogbn-arxiv need to be exceptionally high-dimensional to achieve accurate ER estimates ( $\sim$ 11,000 dimensions), hence we were unable to use them here. However, incorporating ER scalar features into our GAT model yielded a statistically-significant improvement of  $72.14\% \pm 0.03$  test accuracy. Hitting time features improve this result further to  $72.25\% \pm 0.04$  test accuracy. This demonstrates that our affinity-based metrics can yield useful improvements even on larger scale graphs, which are traditionally out of reach for methods like DGN [4] due to computational complexity limitations.
Reliable global leaderboarding with respect to ogbn-arxiv is difficult, as state-of-the-art approaches rely either on privileged information (such as raw text of the paper abstracts), incorporating node labels as features [46], post-processing the predictions [22], or various related tricks [46]. With that in mind, we report for convenience that the current state-of-the-art performance for ogbn-arxiv without using raw text is  $76.11\% \pm 0.09$  test accuracy, achieved by GIANT-XRT+DRGAT.
5.4.2 Heterogeneous citation network: ogbn-mag
We additionally present experiments on an even larger-scale network, ogbn-mag, a heterogeneous network derived from the Microsoft Academic Graph whose nodes consist of four types of entities (papers, authors, institutions, fields of study) and whose edges capture directed relations (e.g., a paper cites another paper, an author is affiliated with an institution, etc.). The task is to predict the venues of papers. The ogbn-mag dataset consists of 111,059,956 nodes and 1,615,685,872 edges.
As in the case of ogbn-arxiv, ER embeddings require an exceptionally high number of dimensions, given the large size of the network. Hence, we incorporated ER scalar features. As a baseline, we used a GraphSAINT [53] model that uses R-GCN aggregation [40], which reported a test accuracy of  $47.51\% \pm 0.22$  in a relevant leaderboard entry. After incorporating ER scalar features into the same model, we obtained a statistically-significant improvement of  $47.99\% \pm 0.23$  in test accuracy.
5.5 Large scale graph regression: OGB-LSC PCQM4Mv1
We finally include experimental results for one of the largest-scale publicly available graph regression tasks: the PCQM4Mv1 dataset from the OGB Large Scale Challenge [20]. PCQM4M is a quantum chemistry dataset spanning 4 million small molecules, with a task to predict the HOMO-LUMO gap, an important quantum-chemical property. It is anticipated that structural features such as ER could be of great help on this task, as the v1 version of it is provided without any structural information, and the molecule's geometry is assumed critical for predicting the gap. We report the single-model validation performance on this dataset, in line with previous works [17, 48, 1].
PCQM4Mv1 comprises molecular graphs which consist of bonds and atom types, and no 3D or 2D coordinates. We reuse the experimental setup and architecture from [17], with only one difference: appending the effective resistance to the edge features. Additionally, we compare against an equivalent model
Table 5: Single-model OGB-LSC PCQM4Mv1 results.
Model	#Layers	Noisy Nodes	Validation MAE
MPNN [17]	16	Yes	0.1249 ± 0.0003
MPNN [17]	50	No	0.1236 ± 0.0001
Graphorner [48]	-	-	0.1234
MPNN [17]	50	Yes	0.1218 ± 0.0001
MPNN + Conformers [1]	32	Yes	0.1212 ± 0.0001
MPNN + ER (ours)	32	Yes	0.1197 ± 0.0002
which uses molecular conformations estimated by RDKit as an additional feature. This gives us a baseline which leverages an explicit estimate of the molecular geometry.
Our results are summarised in Table 5. We once again see a powerful synergy of effective resistance-endowed GNNs and Noisy Nodes [17], allowing us to significantly reduce the number of layers (to 32) and outperform the 50-layer MPNN result in [17]. Further, we improve on the single-model performance of both the Graphormer [48] (which won the original PCQM4M contest after assembling), and an equivalent model to ours which uses molecular conformers from RDKit. This illustrates how ER features can be competitive in geometry-relevant tasks even against features that inherently encode an estimate of the molecule's spatial geometry.
Lastly, we remark that, to the best of our knowledge, our result is the best published single-model result on the large-scale PCQM4M-v1 benchmark to date, and the only single model result with validation MAE under 0.120. We hope this will inspire future investigation on affinity-related GNNs for molecular tasks, especially in settings where spatial geometry is not reliably available.
In this paper, we proposed a message passing network based on random walk based affinity measures.
We believe that the comprehensive theoretical and practical results presented in our paper have solidified affinity-based computations as a strong component of a graph representation learner's toolbox. Our proposal carefully balances theoretical expressive power, empirical performance, and scalability to large graphs. While adding affinity measures as node/edge features provides the most direct route to incorporating them in a graph network, an interesting future direction would be to explore variants of GNN message functions that explicitly make use of affinity-based computations.
We would like to specially thank Jonathan Godwin for his extensive support in setting up the codebase for large-scale GNNs, and Gabriele Corso for his support in setting up the PNA dataset. Additionally, we would like to thank Beatrice Bevilacqua, Wilfried Bounsi, Larisa Markeeva and Pete Battaglia for reviewing the paper prior to submission.
This research was funded by Google Research and Google DeepMind.
[1] Ravichandra Addanki, Peter W Battaglia, David Budden, Andreea Deac, Jonathan Godwin, Thomas Keck, Wai Lok Sibon Li, Alvaro Sanchez-Gonzalez, Jacklynn Stott, Shantanu Thakoor, et al. Large-scale graph representation learning with very deep gnns and self-supervision. arXiv preprint arXiv:2107.09422, 2021.
[2] Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in a spectral perspective. In International Conference on Learning Representations, 2021.
[3] Victor Bapst, Thomas Keck, A Grabska-Barwińska, Craig Donner, Ekin Dogus Cubuk, Samuel S Schoenholz, Annette Obika, Alexander WR Nelson, Trevor Back, Demis Hassabis, et al. Unveiling the predictive power of static structure in glassy systems. Nature Physics, 16(4):448-454, 2020.
[4] Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference on Machine Learning (ICML), pages 748-758. PMLR, 2021.
[5] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yu Guang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34, 2021.
[6] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. arXiv preprint arXiv:2006.09252, 2020.
[7] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Velicković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[8] Siheng Chen, Sufeng Niu, Tian Lan, and Baoan Liu. Pct: Large-scale 3d point cloud representations via graph inception networks with applications to autonomous driving. In 2019 IEEE international conference on image processing (ICIP), pages 4395-4399. IEEE, 2019.
A Hyperparameters for PNA dataset.
In this section we provide the hyperparameters used for the different models on the PNA multitask benchmark. We train all models for 2000 steps and with 3 layers. The remaining hyperparameters for hidden size of each layer, learning rate, number of message passing steps (only valid for MPNN models), number of rotation matrices and same example frequency (when relevant) are provided in Table 6.
Table 6: Training hyperparameters for PNA dataset
Model	#Hidden size	L. rate	#MP steps	#Rotation matrices	#Same Examples
GAT	64	10-4	-	-	-
GCN	64	10-4	-	-	-
DGN	256	10-3	-	-	-
MPNN	256	10-3	2	-	-
ER GNN	128	10-3	2	-	-
ER (node) embed.	64	10-3	1	-	-
ER (edge) embed.	256	10-3	2	-	-
ER (edge) embed.	256	10-3	2	-	-
All ER features	256	10-4	2	23	9
HT + ER (rand rot)	512	10-4	2	23	4
B Details of MPNN Framework
As discussed previously, the architectures used in the experiments conform to the MPNN framework, which allows affinity measures to be added as additional node and edge features. We describe the details here for completeness.
Assume that our input graph,  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , has node features  $\mathbf{x}_u\in \mathbb{R}^n$ , edge features  $\mathbf{x}_{uv}\in \mathbb{R}^{m}$  and graph-level features  $\mathbf{x}_{\mathcal{G}}\in \mathbb{R}^l$ , for nodes  $u,v\in \mathcal{V}$  and edges  $(u,v)\in \mathcal{E}$ . We provide encoders  $f_{n}:\mathbb{R}^{n}\to \mathbb{R}^{k}$ ,  $f_{e}:\mathbb{R}^{m}\to \mathbb{R}^{k}$  and  $f_{g}:\mathbb{R}^{l}\to \mathbb{R}^{k}$  that transform these inputs into a latent space:
$$
\mathbf {h} _ {u} ^ {(0)} = f _ {n} \left(\mathbf {x} _ {u}\right) \quad \mathbf {h} _ {u v} ^ {(0)} = f _ {e} \left(\mathbf {x} _ {u v}\right) \quad \mathbf {h} _ {\mathcal {G}} ^ {(0)} = f _ {g} \left(\mathbf {x} _ {\mathcal {G}}\right) \tag {3}
$$
Our MPNN then performs several message passing steps:
$$
\mathbf {H} ^ {(t + 1)} = P _ {t + 1} \left(\mathbf {H} ^ {(t)}\right) \tag {4}
$$
where  $\mathbf{H}^{(t)} = \left(\left\{\mathbf{h}_u^{(t)}\right\}_{u\in \mathcal{V}},\left\{\mathbf{h}_{uv}^{(t)}\right\}_{(u,v)\in \mathcal{E}},\mathbf{h}_{\mathcal{G}}^{(t)}\right)$  contains all of the latents at a particular processing step  $t\geq 0$
This process is iterated for  $T$  steps, recovering final latents  $\mathbf{H}^{(T)}$ . These can then be decoded into node-, edge-, and graph-level predictions (as required), using analogous decoder functions  $g_{n}$ ,  $g_{e}$  and  $g_{q}$ :
$$
\mathbf {y} _ {u} = g _ {n} \left(\mathbf {h} _ {u} ^ {(T)}\right) \quad \mathbf {y} _ {u v} = g _ {e} \left(\mathbf {h} _ {u v} ^ {(T)}\right) \quad \mathbf {y} _ {\mathcal {G}} = g _ {g} \left(\mathbf {h} _ {\mathcal {G}} ^ {(T)}\right) \tag {5}
$$
Generally,  $f$  and  $g$  are simple MLPs, whereas we use the MPNN update rule for  $P$ . It computes message vectors,  $\mathbf{m}_{uv}^{(t)}$ , to be sent across the edge  $(u,v)$ , and then aggregates them in the receiver nodes as follows:
$$
\mathbf {m} _ {u v} ^ {(t + 1)} = \psi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, \mathbf {h} _ {v} ^ {(t)}, \mathbf {h} _ {u v} ^ {(0)}\right), \quad \mathbf {h} _ {u} ^ {(t + 1)} = \phi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, \sum_ {u \in \mathcal {N} _ {v}} \mathbf {m} _ {v u} ^ {(t + 1)}\right) \tag {6}
$$
The message function  $\psi_{t + 1}$  and the update function  $\phi_{t + 1}$  are both MLPs. All of our models have been implemented using the graph library [16].
We incorporate edge-based affinity features (e.g., effective resistances and hitting times) in  $f_{e}$  and node-based affinity features (e.g., resistive embeddings) in  $f_{n}$ . Note that node-based affinity
features may also naturally be incorporated as edge features by concatenating the node features at the endpoints.
Occasionally, the dataset in question will be easy to overfit with the most general form of message function (see (6)). In these cases, we resort to assuming that  $\psi$  factorises into an attention mechanism:
$$
\mathbf {m} _ {u v} ^ {(t + 1)} = a _ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}, \mathbf {h} _ {v} ^ {(t)}, \mathbf {h} _ {u v} ^ {(0)}\right) \psi_ {t + 1} \left(\mathbf {h} _ {u} ^ {(t)}\right) \tag {7}
$$
where the attention function  $a$  is scalar-valued. We will refer to this particular MPNN baseline as a graph attention network (GAT) [44].
C Omitted Proofs
Lemma 3.2. For any pair of nodes  $u, v$ , we have  $\|\mathbf{r}_u - \mathbf{r}_v\|_2^2 = \operatorname{Res}(u, v)$ .
$$
\begin{array}{l} \left\| \mathbf {r} _ {u} - \mathbf {r} _ {v} \right\| _ {2} ^ {2} = \left\| C ^ {1 / 2} B L _ {G} ^ {- 1} \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) \right\| _ {2} ^ {2} \\ = \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) ^ {T} L ^ {\dagger} \left(B ^ {T} C B\right) L ^ {\dagger} \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) \\ = \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) ^ {T} L ^ {\dagger} L L ^ {\dagger} \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) \\ = \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) ^ {T} L ^ {\dagger} \left(\mathbf {1} _ {u} - \mathbf {1} _ {v}\right) = \operatorname {R e s} (u, v). \\ \end{array}
$$
Proof. Since  $\langle X,Y\rangle = \frac{1}{4}\left(\| X + Y\| ^2 -\| X - Y\| ^2\right)$ , we can bound  $A = \big|\langle X,Y\rangle -\langle \widehat{X},\widehat{Y}\rangle \big|$  from above as:
$$
\begin{array}{l} A = \left| \frac {1}{4} \left(\| X + Y \| ^ {2} - \| \hat {X} + \hat {Y} \| ^ {2} - \| X - Y \| ^ {2} + \| \hat {X} - \hat {Y} \| ^ {2}\right) \right| \\ \leq \frac {1}{4} \left(\left| \| \widehat {X} + \widehat {Y} \| ^ {2} - \| X + Y \| ^ {2} \right| + \left| \| \widehat {X} - \widehat {Y} \| ^ {2} - \| X - Y \| ^ {2} \right|\right) \\ \leq \frac {1}{4} \left(\epsilon \cdot \| X + Y \| ^ {2} + \epsilon \cdot \| X - Y \| ^ {2}\right) \tag {8} \\ = \frac {\epsilon}{4} \left(\| X + Y \| ^ {2} + \| X - Y \| ^ {2}\right) \\ = \frac {\epsilon}{2} \left(\| X \| ^ {2} + \| Y \| ^ {2}\right), \\ \end{array}
$$