SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 SAMPLE
1:	Input: set S of {yi}	10: S1 = S2 = ∅
set X of {xi}	11: for yi ∈ S do
2:	Output: E = {(yi, xj) for some i and j}	12: r ~ Unif[0, 1]
13: if r ≤ gX1(yi)/(gX1(yi) + gX2(yi)) then
3:	if |X| = 1 then
4:	return S × X	14: S1 = S1 ∪ {yi}
5:	else	15: else
6:	X1 = {xj : j < |X|/2}	16: S2 = S2 ∪ {yi}
7:	X2 = {xj : j ≥ |X|/2}	17: end if
8:	Compute gX1(yi) for all i with a KDE al-
algorithm	18: end for
19: return SAMPLE(S1, X1)∪SAMPLE(S2, X2)
9:	Compute gX2(yi) for all i with a KDE al-
algorithm	20: end if
where  $\widehat{p}(i,j) \triangleq \widehat{p}_i(j) + \widehat{p}_j(i) - \widehat{p}_i(j) \cdot \widehat{p}_j(i)$  is an estimate of the sampling probability of edge  $v_i \sim v_j$ , and
$$
\widehat {p} _ {i} (j) \triangleq \min  \left\{6 C \cdot \log n \cdot \frac {k (x _ {i} , x _ {j})}{g _ {[ 1 , n ]} (x _ {i})}, 1 \right\}
$$
for some constant  $C \in \mathbb{R}^+$ ; see Algorithm 2 for the formal description.
4.2 Algorithm Analysis
Now we analyse Algorithm 2, and sketch the proof of Theorem 2. We first analyse the running time of Algorithm 2. Since it involves  $O(\log n)$  executions of Algorithm 1 in total, it is sufficient to examine the running time of Algorithm 1.
We visualise the recursion of Algorithm 1 with respect to  $S$  and  $X$  in Figure 4. Notice that, although the number of nodes doubles at each level of the recursion tree, the total number of samples  $S$  and data points  $X$  remain constant for each level of the tree: it holds for any  $i$  that  $\bigcup_{j=1}^{2^i} S_{i,j} = S_0$  and
Figure 4: The recursion tree for Algorithm 1.
Algorithm 2 FASTSIMILARITYGRAPH
1: Input: data point set  $X = \{x_{1},\ldots ,x_{n}\}$
2: Output: similarity graph G
3:  $E = \emptyset$ ,  $L = 6C \cdot \log(n) / \lambda_{k+1}$
4: for  $\ell \in [1,L]$  do
5:  $E = E\cup \mathrm{SAMPLE}(X,X)$
6: end for
7: Compute  $g_{[1,n]}(x_i)$  for each  $i$  with a KDE algorithm
8: for  $(v_{i}, v_{j}) \in E$  do
9:  $\widehat{p}_i(j) = \min \left\{L\cdot k(x_i,x_j) / g_{[1,n]}(x_i),1\right\}$
10:  $\widehat{p}_j(i) = \min \left\{L\cdot k(x_i,x_j) / g_{[1,n]}(x_j),1\right\}$
11:  $\widehat{p}(i,j) = \widehat{p}_i(j) + \widehat{p}_j(i) - \widehat{p}_i(j) \cdot \widehat{p}_j(i)$
12: Set  $w_{\mathsf{G}}(v_i, v_j) = k(x_i, x_j) / \widehat{p}(i, j)$
13: end for
14: return graph  $\mathsf{G} = (X, E, w_{\mathsf{G}})$
$\bigcup_{j=1}^{2^i} X_{i,j} = X_0$ . Since the running time of the KDE algorithm is superadditive, the combined running time of all nodes at level  $i$  of the tree is
$$
T _ {i} = \sum_ {j = 1} ^ {2 ^ {i}} T _ {\mathsf {K D E}} (| S _ {i, j} |, | X _ {i, j} |, \epsilon) \leq T _ {\mathsf {K D E}} \left(\sum_ {j = 1} ^ {2 ^ {i}} | S _ {i, j} |, \sum_ {j = 1} ^ {2 ^ {i}} | X _ {i, j} |, \epsilon\right) = T _ {\mathsf {K D E}} (n, n, \epsilon).
$$
Hence, the total running time of Algorithm 2 is  $\widetilde{O}(T_{\mathsf{KDE}}(n,n,\epsilon))$  as the tree has  $\log_2(n)$  levels. Moreover, when applying the Fast Gauss Transform (FGT) [10] as the KDE algorithm, the total running time of Algorithm 1 is  $\widetilde{O}(n)$  when  $d = O(1)$ .
Finally, we prove that the output of Algorithm 2 is a cluster-preserving sparsifier of a fully connected similarity graph, and has  $\widetilde{O}(n)$  edges. Notice that, comparing with the sampling probabilities  $p_u(v)$  and  $p_v(u)$  used in the SZ algorithm, Algorithm 2 samples each edge through  $O(\log n)$  recursive executions of a KDE algorithm, each of which introduces a  $(1 + \epsilon)$ -multiplicative error. We carefully examine these  $(1 + \epsilon)$ -multiplicative errors and prove that the actual sampling probability  $\widetilde{p}(i,j)$  used in Algorithm 2 is an  $O(1)$ -approximation of  $p_e$  for every  $e = \{v_i,v_j\}$ . As such the output of Algorithm 2 is a cluster-preserving sparsifier of a fully connected similarity graph, and satisfies the two properties of Theorem 2. The total number of edges in G follows from the sampling scheme. We refer the reader to Appendix A for the complete proof of Theorem 2.
In this section, we empirically evaluate the performance of spectral clustering with our new algorithm for constructing similarity graphs. We compare our algorithm with the algorithms for similarity graph construction provided by the scikit-learn library [20] and the FAISS library [11] which are commonly used machine learning libraries for Python. In the remainder of this section, we compare the following six spectral clustering methods.
1. SKLEARN GK: spectral clustering with the fully connected Gaussian kernel similarity graph constructed with the scikit-learn library.
2. SKLEARN  $k$ -NN: spectral clustering with the  $k$ -nearest neighbour similarity graph constructed with the scikit-learn library.
3. FAISS EXACT: spectral clustering with the exact  $k$ -nearest neighbour graph constructed with the FAISS library.
4. FAISS HNSW: spectral clustering with the approximate  $k$ -nearest neighbour graph constructed with the "Hierarchical Navigable Small World" method [18].
5. FAISS IVF: spectral clustering with the approximate  $k$ -nearest neighbour graph constructed with the "Invertex File Index" method [3].
6. OUR ALGORITHM: spectral clustering with the Gaussian kernel similarity graph constructed by Algorithm 2.
We implement Algorithm 2 in  $\mathbf{C} + +$ , using the implementation of the Fast Gauss Transform provided by Yang et al. [33], and use the Python SciPy [30] and stag [17] libraries for eigenvector computation and graph operations respectively. The scikit-learn and FAISS libraries are well-optimised
and use C, C++, and FORTRAN for efficient implementation of core algorithms. We first employ classical synthetic clustering datasets to clearly compare how the running time of different algorithms is affected by the number of data points. This experiment highlights the nearly-linear time complexity of our algorithm. Next we demonstrate the applicability of our new algorithm for image segmentation on the Berkeley Image Segmentation Dataset (BSDS) [2].
For each experiment, we set  $k = 10$  for the approximate nearest neighbour algorithms. For the synthetic datasets, we set the  $\sigma$  value of the Gaussian kernel used by SKLEARN GK and OUR ALGORITHM to be 0.1, and for the BSDS experiment we set  $\sigma = 0.2$ . This choice follows the heuristic suggested by von Luxburg [31] to choose  $\sigma$  to approximate the distance from a point to its  $k$ th nearest neighbour. All experiments are performed on an HP ZBook laptop with an 11th Gen Intel(R) Core(TM)i7-11800H @ 2.30GHz processor and 32 GB RAM. The code to reproduce our results is available at https://github.com/pmacg/kde-similarity-graph.
5.1 Synthetic Dataset
In this experiment we use the scikit-learn library to generate 15,000 data points in  $\mathbb{R}^2$  from a variety of classical synthetic clustering datasets. The data is generated with the make_circles, make moons, and make_blobs methods of the scikit-learn library with a noise parameter of 0.05. The linear transformation  $\{\{0.6, -0.6\}, \{-0.4, 0.8\}\}$  is applied to the blobs data to create asymmetric clusters. As shown in Figure 1, all of the algorithms return approximately the same clustering, but our algorithm runs much faster than the ones from scikit-learn and FAISS.
We further compare the speedup of our algorithm against the five competitors on the two moons dataset with an increasing number of data points, and our result is reported in Figure 5. Notice that the running time of the scikit-learn and FAISS algorithms scales roughly quadratically with the size of the input, while the running time of our algorithm is nearly linear. Furthermore, we note that on a laptop with 32 GB RAM, the SKLEARN GK algorithm could not scale beyond 20,000 data points due to its quadratic memory requirement, while our new algorithm can cluster 1,000,000 data points in 240 seconds under the same memory constraint.
Figure 5: Comparison of the running time of spectral clustering on the two moons dataset. In every case, all algorithms return the correct clusters.
5.2 BSDS Image Segmentation Dataset
Finally we study the application of spectral clustering for image segmentation on the BSDS dataset. The dataset contains 500 images with several ground-truth segmentations for each image. Given an input image, we consider each pixel to be a point  $(r,g,b,x,y)^{\top} \in \mathbb{R}^{5}$  where  $r,g,b$  are the red, green, blue pixel values and  $x,y$  are the coordinates of the pixel in the image. Then, we construct a similarity graph based on these points, and apply spectral clustering, for which we set the number of clusters to be the median number of clusters in the provided ground truth segmentations. Our experimental result is reported in Table 1, where the "Downsampled" version is employed to reduce the resolution of the image to 20,000 pixels in order to run the SKLEARN GK and the "Full Resolution" one is to apply the original image of over 150,000 pixels as input. This set of experiments demonstrates our algorithm produces better clustering results with respect to the average Rand Index [23].
Table 1: The average Rand Index of the output from the six algorithms. Due to the quadratic space complexity constraint, the SKLEARN GK cannot be applied to the full resolution image.
Algorithm	Downsampled	Full Resolution
SKLEARN GK	0.681	-
SKLEARN k-NN	0.675	0.682
FAISS EXACT	0.675	0.680
FAISS HNSW	0.679	0.677
FAISS IVF	0.675	0.680
OUR ALGORITHM	0.680	0.702
In this paper we develop a new technique that constructs a similarity graph, and show that an approximation algorithm for the KDE can be employed to construct a similarity graph with proven approximation guarantee. Applying the publicly available implementations of the KDE as a black-box, our algorithm outperforms five competing ones from the well-known scikit-learn and FAISS libraries for the classical datasets of low dimensions. We believe that our work will motivate more research on faster algorithms for the KDE in higher dimensions and their efficient implementation, as well as more efficient constructions of similarity graphs.
We would like to thank the anonymous reviewers for their valuable comments on the paper. This work is supported by an EPSRC Early Career Fellowship (EP/T00729X/1).
[1] Josh Alman, Timothy Chu, Aaron Schild, and Zhao Song. Algorithms and hardness for linear algebra on geometric graphs. In 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS'20), pages 541-552, 2020.
[2] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898-916, 2011.
[3] Artem Babenko and Victor Lempitsky. The inverted multi-index. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(6):1247-1260, 2014.
[4] Arturs Backurs, Piotr Indyk, and Tal Wagner. Space and time efficient kernel density estimation in high dimensions. In 33rd Advances in Neural Information Processing Systems (NeurIPS'19), pages 15773-15782, 2019.
[5] Ainesh Bakshi, Piotr Indyk, Praneeth Kacham, Sandeep Silwal, and Samson Zhou. Subquadratic algorithms for kernel matrices via kernel density estimation. In 11th International Conference on Learning Representations (ICLR '23), 2023.
[6] Moses Charikar, Michael Kapralov, Navid Nouri, and Paris Siminelakis. Kernel density estimation through density constrained near neighbor search. In 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS'20), pages 172-183, 2020.
[7] Moses Charikar and Paris Siminelakis. Hashing-based-estimators for kernel density in high dimensions. In 58th Annual IEEE Symposium on Foundations of Computer Science (FOCS'17), pages 1032–1043, 2017.
[8] Fan Chung and Linyuan Lu. Concentration inequalities and martingale inequalities: a survey. _Internet mathematics_, 3(1):79-127, 2006.
[9] Wei Dong, Moses Charikar, and Kai Li. Efficient  $k$ -nearest neighbor graph construction for generic similarity measures. In 20th International Conference on World Wide Web (WWW '11), pages 577-586, 2011.
van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
[31] Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395-416, 2007.
[32] Yiqiu Wang, Anshumali Shrivastava, Jonathan Wang, and Junghee Ryu. Randomized algorithms accelerated over CPU-GPU for ultra-high dimensional similarity search. In 2018 International Conference on Management of Data (SIGMOD '18), pages 889-903, 2018.
[33] Changjiang Yang, Ramani Duraiswami, Nail A. Gumerov, and Larry Davis. Improved fast Gauss transform and efficient kernel density estimation. In 9th International Conference on Computer Vision (ICCV'03), pages 664-671, 2003.
This section presents the complete proof of Theorem 2. Let  $y_{i,1}, \ldots, y_{i,L}$  be random variables which are equal to the indices of the  $L$  points sampled for  $x_i$ . Recall that by the SZ algorithm, the "ideal" sampling probability for  $x_j$  from  $x_i$  is
$$
p _ {i} (j) \triangleq \min  \left\{C \cdot \frac {\log (n)}{\lambda_ {k + 1}} \cdot \frac {k \left(x _ {i} , x _ {j}\right)}{\deg_ {K} \left(v _ {i}\right)}, 1 \right\}.
$$
We denote the actual sampling probability that  $x_{j}$  is sampled from  $x_{i}$  under Algorithm 2 to be
$$
\widetilde {p} _ {i} (j) \triangleq \mathbb {P} \left[ x _ {j} \in \left\{y _ {i, 1}, \dots y _ {i, L} \right\} \right].
$$
Finally, for each added edge, Algorithm 2 also computes an estimate of  $p_i(x_j)$  which we denote
$$
\widehat {p} _ {i} (j) \triangleq \min \left\{6 C \cdot \frac {\log (n)}{\lambda_ {k + 1}} \cdot \frac {k (x _ {i} , x _ {j})}{g _ {[ 1 , n ]} (x _ {i})}, 1 \right\}.
$$
Similar with the definition of  $p_e$  in Section 3, we define
-  $p(i,j) = p_i(j) + p_j(i) - p_i(j) \cdot p_j(i),$
-  $\widetilde{p}(i,j) = \widetilde{p}_i(j) + \widetilde{p}_j(i) - \widetilde{p}_i(j) \cdot \widetilde{p}_j(i)$ , and
-  $\widehat{p}(i,j) = \widehat{p}_i(j) + \widehat{p}_j(i) - \widehat{p}_i(j) \cdot \widehat{p}_j(i)$ .
Notice that, following the convention of [27], we use  $p_i(j)$  to refer to the probability that a given edge is sampled from the vertex  $x_i$  and  $p(i,j)$  is the probability that the given edge  $\{v_i,v_j\}$  is sampled at all by the algorithm. We use the same convention for  $\widetilde{p}_i(j)$  and  $\widehat{p}_i(j)$ .
We first prove a sequence of lemmas showing that these probabilities are all within a constant factor of each other.
Lemma 1. For any point  $x_{i}$ , the probability that a given sampled neighbour  $y_{i,l}$  is equal to  $j$  is given by
$$
\frac {k (x _ {i} , x _ {j})}{2 \deg_ {\mathsf {K}} (v _ {i})} \leq \mathbb {P} [ y _ {i, l} = j ] \leq \frac {2 k (x _ {i} , x _ {j})}{\deg_ {\mathsf {K}} (v _ {i})}.
$$
Proof. Let  $X = \{x_{1},\ldots ,x_{n}\}$  be the input data points for Algorithm 2, and  $[n] = \{1,\dots n\}$  be the indices of the input data points. Furthermore, let  $[a,b] = \{a,\dots ,b\}$  be the set of indices between  $a$  and  $b$ . Then, in each recursive call to Algorithm 1, we are given a range of indices  $[a,b]$  as input and assign  $y_{i,l}$  to one half of it: either  $[a,\lfloor b / 2\rfloor ]$  or  $[\lfloor b / a\rfloor +1,b]$ . By Algorithm 1, we have that the probability of assigning  $y_{i,l}$  to  $[a,\lfloor b / 2\rfloor ]$  is
$$
\mathbb {P} \left[ y _ {i, l} \in [ a, \lfloor b / 2 \rfloor ] \mid y _ {i, l} \in [ a, b ] \right] = \frac {g _ {[ a , \lfloor b / 2 \rfloor ]} (x _ {i})}{g _ {[ a , b ]} (x _ {i})}.
$$
By the performance guarantee of the KDE algorithm, we have that  $g_{[a,b]}(x_i) \in (1 \pm \epsilon) \deg_{[a,b]}(v_i)$ , where we define
$$
\deg_ {[ a, b ]} (x _ {i}) \triangleq \sum_ {j = a} ^ {b} k (x _ {i}, x _ {j}).
$$
This gives
$$
\left(\frac {1 - \epsilon}{1 + \epsilon}\right) \frac {\deg_ {[ a , \lfloor b / 2 \rfloor ]} (v _ {i})}{\deg_ {[ a , b ]} (v _ {i})} \leq \mathbb {P} \left[ y _ {i, l} \in X _ {[ a, \lfloor b / 2 \rfloor ]} \mid y _ {i, l} \in X _ {[ a, b ]} \right] \leq \left(\frac {1 + \epsilon}{1 - \epsilon}\right) \frac {\deg_ {[ a , \lfloor b / 2 \rfloor ]} (v _ {i})}{\deg_ {[ a , b ]} (v _ {i})}. \tag {3}
$$
Next, notice that we can write for a sequence of intervals  $[a_1,b_1]\subset [a_2,b_2]\subset \ldots \subset [1,n]$  that
$$
\begin{array}{l} \mathbb {P} \left[ y _ {i, l} = j \right] = \mathbb {P} \left[ y _ {i, l} = j \mid y _ {i, l} \in \left[ a _ {1}, b _ {1} \right] \right] \times \mathbb {P} \left[ y _ {i, l} \in \left[ a _ {1}, b _ {1} \right] \mid y _ {i, l} \in \left[ a _ {2}, b _ {2} \right] \right] \\ \times \dots \times \mathbb {P} \left[ y _ {i, l} \in \left[ a _ {k}, b _ {k} \right] \mid y _ {i, l} \in [ 1, n ] \right], \\ \end{array}
$$
where each term corresponds to one level of recursion of Algorithm 1 and there are at most  $\lceil \log_2(n) \rceil$  terms. Then, by (3) and the fact that the denominator and numerator of adjacent terms cancel out, we have
$$
\left(\frac {1 - \epsilon}{1 + \epsilon}\right) ^ {\lceil \log_ {2} (n) \rceil} \frac {k (x _ {i} , x _ {j})}{\deg_ {K} (v _ {i})} \leq \mathbb {P} \left[ y _ {i, l} = j \right] \leq \left(\frac {1 + \epsilon}{1 - \epsilon}\right) ^ {\lceil \log_ {2} (n) \rceil} \frac {k (x _ {i} , x _ {j})}{\deg_ {K} (v _ {i})}
$$
since  $\deg_{[j,j]}(v_i) = k(x_i,x_j)$  and  $\deg_{[1,n]}(v_i) = \deg_K(v_i)$ .
$$
\left(\frac {1 - \epsilon}{1 + \epsilon}\right) ^ {\lceil \log_ {2} (n) \rceil} \geq (1 - 2 \epsilon) ^ {\lceil \log_ {2} (n) \rceil} \geq 1 - 3 \log_ {2} (n) \epsilon \geq 1 / 2,
$$
where the final inequality follows by the condition of  $\epsilon$  that  $\epsilon \leq 1 / (6\log_2(n))$ .
For the upper bound, we similarly have
$$
\left(\frac {1 + \epsilon}{1 - \epsilon}\right) ^ {\lceil \log_ {2} (n) \rceil} \leq (1 + 3 \epsilon) ^ {\lceil \log_ {2} (n) \rceil} \leq \exp \left(3 \lceil \log_ {2} (n) \rceil \epsilon\right) \leq \mathrm {e} ^ {2 / 3} \leq 2,
$$
where the first inequality follows since  $\epsilon < 1 / (6\log_2(n))$
The next lemma shows that the probability of sampling each edge  $\{v_i, v_j\}$  is approximately equal to the "ideal" sampling probability  $p_i(j)$ .
Lemma 2. For every  $i$  and  $j\neq i$  ,we have
$$
\frac {9}{1 0} \cdot p _ {i} (j) \leq \widetilde {p} _ {i} (j) \leq 1 2 \cdot p _ {i} (j).
$$
Proof. Let  $Y = \{y_{i,1},\dots ,y_{i,L}\}$  be the neighbours of  $x_{i}$  sampled by Algorithm 2, where  $L = 6C\log (n) / \lambda_{k + 1}$ . Then,
$$
\mathbb {P} [ j \in Y ] = 1 - \prod_ {l = 1} ^ {L} (1 - \mathbb {P} [ y _ {i, l} = j ]) \geq 1 - \left(1 - \frac {k (x _ {i} , x _ {j})}{2 \deg_ {\mathrm {K}} (v _ {i})}\right) ^ {L} \geq 1 - \exp \left(- L \cdot \frac {k (x _ {i} , x _ {j})}{2 \deg_ {\mathrm {K}} (v _ {i})}\right)
$$
The proof proceeds by case distinction.
Case 1:  $p_i(j) \leq 9 / 10$ . In this case, we have
$$
\mathbb {P} [ j \in Y ] \geq 1 - \exp (- 6 p _ {i} (j) / 2) \geq p _ {i} (j).
$$
Case 2:  $p_i(j) > 9 / 10$ . In this case, we have
$$
\mathbb {P} \left[ j \in Y \right] \geq 1 - \exp \left(- \frac {9 \cdot 6}{2 0}\right) \geq \frac {9}{1 0},
$$
which completes the proof on the lower bound of  $\widetilde{p}_i(j)$ .
$$
\mathbb {P} \left[ j \in Y \right] \leq 1 - \left(1 - \frac {2 k \left(x _ {i} , x _ {j}\right)}{\deg_ {\mathsf {K}} \left(v _ {i}\right)}\right) ^ {L} \leq \frac {2 k \left(x _ {i} , x _ {j}\right)}{\deg_ {\mathsf {K}} \left(v _ {i}\right)} \cdot L = 1 2 C \cdot \frac {\log (n)}{\lambda_ {k + 1}} \cdot \frac {k \left(x _ {i} , x _ {j}\right)}{\deg_ {\mathsf {K}} \left(v _ {i}\right)},
$$
from which the statement follows.
An immediate corollary of Lemma 2 is as follows.
Corollary 1. For all different  $i,j\in [n]$ , it holds that
$$
\frac {9}{1 0} \cdot p (i, j) \leq \widetilde {p} (i, j) \leq 1 2 \cdot p (i, j)
$$
$$
\frac {6}{7} \cdot p (i, j) \leq \widehat {p} (i, j) \leq \frac {3 6}{5} \cdot p (i, j).
$$
Proof. For the upper bound of the first statement, we can assume that  $p_i(j) \leq 1/12$  and  $p_j(i) \leq 1/12$ , since otherwise we have  $\widetilde{p}(i,j) \leq 1 \leq 12 \cdot p(i,j)$  and the statement holds trivially. Then, by Lemma 2, we have
$$
\begin{array}{l} \widetilde {p} (i, j) = \widetilde {p} _ {i} (j) + \widetilde {p} _ {j} (i) - \widetilde {p} _ {i} (j) \cdot \widetilde {p} _ {j} (i) \\ \leq 1 2 p _ {i} (j) + 1 2 p _ {j} (i) - 1 2 p _ {i} (j) \cdot 1 2 p _ {j} (i) \\ \leq 1 2 \left(p _ {i} (j) + p _ {j} (i) - p _ {i} (j) \cdot p _ {j} (i)\right) \\ = 1 2 \cdot p (i, j) \\ \end{array}
$$
$$
\begin{array}{l} \widetilde {p} (i, j) = \widetilde {p} _ {i} (j) + \widetilde {p} _ {j} (i) - \widetilde {p} _ {i} (j) \cdot \widetilde {p} _ {j} (i) \\ \geq \frac {9}{1 0} \cdot p _ {i} (j) + \frac {9}{1 0} \cdot p _ {j} (i) - \frac {9}{1 0} p _ {i} (j) \cdot \frac {9}{1 0} p _ {j} (i) \\ \geq \frac {9}{1 0} (p _ {i} (j) + p _ {j} (i) - p _ {i} (j) p _ {j} (i)) \\ = \frac {9}{1 0} \cdot p (i, j). \\ \end{array}
$$
For the second statement, notice that
$$
\begin{array}{l} \widehat {p} _ {i} (j) = \min  \left\{\frac {6 C \log (n)}{\lambda_ {k + 1}} \cdot \frac {k (i , j)}{g _ {[ 1 , n ]} (x _ {i})}, 1 \right\} \\ \geq \min  \left\{\frac {1}{1 + \varepsilon} \frac {C \log (n)}{\lambda_ {k + 1}} \cdot \frac {k (i , j)}{\deg_ {\mathbf {K}} (v _ {i})}, 1 \right\} \\ \geq \frac {1}{1 + \varepsilon} \cdot \min  \left\{\frac {C \log (n)}{\lambda_ {k + 1}} \cdot \frac {k (i , j)}{\deg_ {\mathsf {K}} (v _ {i})}, 1 \right\} \\ = \frac {1}{1 + \varepsilon} \cdot p _ {i} (j) \\ \geq \frac {6}{7} \cdot p _ {i} (j), \\ \end{array}
$$
since  $g_{[1,n]}(x_i)$  is a  $(1\pm \varepsilon)$  approximation of  $\deg_{\mathsf{K}}(v_i)$  and  $\varepsilon \leq 1 / 6$ . Similarly,
$$
\widehat {p} _ {i} (j) \leq \frac {6}{1 - \varepsilon} \cdot p _ {i} (j) \leq \frac {3 6}{5} \cdot p _ {i} (j).
$$
Then, for the upper bound of the second statement, we can assume that  $p_i(j) \leq 5/36$  and  $p_j(i) \leq 5/36$ , since otherwise  $\widehat{p}(i,j) \leq 1 \leq (36/5) \cdot \widetilde{p}(i,j)$  and the statement holds trivially. This implies that
$$
\begin{array}{l} \widehat {p} (i, j) = \widehat {p _ {i}} (j) + \widehat {p _ {j}} (i) - \widehat {p _ {i}} (j) \cdot \widehat {p _ {j}} (i) \\ \leq \frac {3 6}{5} p _ {i} (j) + \frac {3 6}{5} p _ {j} (i) - \frac {3 6}{5} p _ {i} (j) \cdot \frac {3 6}{5} p _ {j} (i) \\ \leq \frac {3 6}{5} \left(p _ {i} (j) + p _ {j} (i) - p _ {i} (j) \cdot p _ {j} (i)\right) \\ = \frac {3 6}{5} \cdot p (i, j) \\ \end{array}
$$
$$
\begin{array}{l} \widehat {p} (i, j) \geq \frac {6}{7} p _ {i} (j) + \frac {6}{7} p _ {j} (i) - \frac {6}{7} p _ {i} (j) \cdot \frac {6}{7} p _ {j} (i) \\ \geq \frac {6}{7} \left(p _ {i} (j) + p _ {j} (i) - p _ {i} (j) \cdot p _ {j} (i)\right) \\ = \frac {6}{7} \cdot p (i, j), \\ \end{array}
$$
We are now ready to prove Theorem 2. It is important to note that, although some of the proofs below are parallel to that of [27], our analysis needs to carefully take into account the approximation ratios introduced by the approximate KDE algorithm, which makes our analysis more involved. The following concentration inequalities will be used in our proof.
Lemma 3 (Bernstein's Inequality [8]). Let  $X_{1},\ldots ,X_{n}$  be independent random variables such that  $|X_{i}|\leq M$  for any  $i\in \{1,\dots ,n\}$ . Let  $X = \sum_{i = 1}^{n}X_{i}$ , and  $R = \sum_{i = 1}^{n}\mathbb{E}\left[X_i^2\right]$ . Then, it holds that
$$
\mathbb {P} \left[ | X - \mathbb {E} [ X ] | \geq t \right] \leq 2 \exp \left(- \frac {t ^ {2}}{2 (R + M t / 3)}\right).
$$
Lemma 4 (Matrix Chernoff Bound [28]). Consider a finite sequence  $\{X_i\}$  of independent, random, PSD matrices of dimension  $d$  that satisfy  $\| X_i \| \leq R$ . Let  $\mu_{\min} \triangleq \lambda_{\min}(\mathbb{E}[\sum_i X_i])$  and  $\mu_{\max} \triangleq \lambda_{\max}(\mathbb{E}[\sum_i X_i])$ . Then, it holds that
$$
\mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {i} X _ {i}\right) \leq (1 - \delta) \mu_ {\min } \right] \leq d \left(\frac {\mathrm {e} ^ {- \delta}}{(1 - \delta) ^ {1 - \delta}}\right) ^ {\mu_ {\min } / R}
$$
for  $\delta \in [0,1]$ , and
$$
\mathbb {P} \left[ \lambda_ {\max } \left(\sum_ {i} X _ {i}\right) \geq (1 + \delta) \mu_ {\max } \right] \leq d \left(\frac {\mathrm {e} ^ {\delta}}{(1 + \delta) ^ {1 + \delta}}\right) ^ {\mu_ {\max } / R}
$$
for  $\delta \geq 0$ .