SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peter Macgregor  
School of Informatics  
University of Edinburgh  
United Kingdom
He Sun  
School of Informatics  
University of Edinburgh  
United Kingdom
Constructing a similarity graph from a set  $X$  of data points in  $\mathbb{R}^d$  is the first step of many modern clustering algorithms. However, typical constructions of a similarity graph have high time complexity, and a quadratic space dependency with respect to  $|X|$ . We address this limitation and present a new algorithmic framework that constructs a sparse approximation of the fully connected similarity graph while preserving its cluster structure. Our presented algorithm is based on the kernel density estimation problem, and is applicable for arbitrary kernel functions. We compare our designed algorithm with the well-known implementations from the scikit-learn library and the FAISS library, and find that our method significantly outperforms the implementation from both libraries on a variety of datasets.
Given a set  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  of data points and a similarity function  $k:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}_{\geq 0}$  for any pair of data points  $x_{i}$  and  $x_{j}$ , the objective of clustering is to partition these  $n$  data points into clusters such that similar points are in the same cluster. As a fundamental data analysis technique, clustering has been extensively studied in different disciplines ranging from algorithms and machine learning, social network analysis, to data science and statistics.
One prominent approach for clustering data points in Euclidean space consists of two simple steps: the first step is to construct a similarity graph  $\mathsf{K} = (V, E, w)$  from  $X$ , where every vertex  $v_i$  of  $\mathsf{G}$  corresponds to  $x_i \in X$ , and different vertices  $v_i$  and  $v_j$  are connected by an edge with weight  $w(v_i, v_j)$  if their similarity  $k(x_i, x_j)$  is positive, or higher than some threshold. Secondly, we apply spectral clustering on  $\mathsf{G}$ , and its output naturally corresponds to some clustering on  $X$  [19]. Because of its out-performance over traditional clustering algorithms like  $k$ -means, this approach has become one of the most popular modern clustering algorithms.
On the other side, different constructions of similarity graphs have significant impact on the quality and time complexity of spectral clustering, which is clearly acknowledged and appropriately discussed by von Luxburg [31]. Generally speaking, there are two types of similarity graphs:
- the first one is the  $k$ -nearest neighbour graph ( $k$ -NN graph), in which every vertex  $v_i$  connects to  $v_j$  if  $v_j$  is among the  $k$ -nearest neighbours of  $v_i$ . A  $k$ -NN graph is sparse by construction, but loses some of the structural information in the dataset since  $k$  is usually small and the added edges are unweighted.
- the second one is the fully connected graph, in which different vertices  $v_{i}$  and  $v_{j}$  are connected with weight  $w(v_{i},v_{j}) = k(x_{i},x_{j})$ . While a fully connected graph maintains most of the distance information about  $X$ , this graph is dense and storing such graphs requires quadratic memory in  $n$ .
Taking the pros and cons of the two constructions into account, one would naturally ask the question:
Is it possible to directly construct a sparse graph that preserves the cluster structure of a fully connected similarity graph?
We answer this question affirmatively, and present a fast algorithm that constructs an approximation of the fully connected similarity graph. Our constructed graph consists of only  $\widetilde{O}(n)$  edges $^1$ , and preserves the cluster structure of the fully connected similarity graph.
1.1 Our Result
Given any set  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  and a kernel function  $k:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}_{\geq 0}$ , a fully connected similarity graph  $\mathsf{K} = (V,E,w)$  of  $X$  consists of  $n$  vertices, and every  $v_{i}\in V$  corresponds to  $x_{i}\in X$ ; we set  $w(v_{i},v_{j})\triangleq k(x_{i},x_{j})$  for any different  $v_{i}$  and  $v_{j}$ . We introduce an efficient algorithm that constructs a sparse graph  $G$  directly from  $X$ , such that  $K$  and  $G$  share the same cluster-structure, and the graph matrices for  $K$  and  $G$  have approximately the same eigen-gap. This ensures that spectral clustering from  $G$  and  $K$  return approximately the same result.
The design of our algorithm is based on a novel reduction from the approximate construction of similarity graphs to the problem of Kernel Density Estimation (KDE). This reduction shows that any algorithm for the KDE can be employed to construct a sparse representation of a fully connected similarity graph, while preserving the cluster-structure of the input data points. This is summarised as follows:
Theorem 1 (Informal Statement of Theorem 2). Given a set of data points  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  as input, there is a randomised algorithm that constructs a sparse graph  $G$  of  $X$ , such that it holds with probability at least 9/10 that
1. graph  $\mathsf{G}$  has  $\widetilde{O}(n)$  edges,
2. graph G has the same cluster structure as the fully connected similarity graph  $\mathsf{K}$  of  $X$ .
The algorithm uses an approximate KDE algorithm as a black-box, and has running time  $\widetilde{O}(T_{\mathrm{KDE}}(n,n,\epsilon))$  for  $\epsilon \leq 1/(6\log(n))$ , where  $T_{\mathrm{KDE}}(n,n,\epsilon)$  is the running time of solving the KDE problem for  $n$  data points up to a  $(1+\epsilon)$ -approximation.
This result builds a novel connection between the KDE and the fast construction of similarity graphs, and further represents a state-of-the-art algorithm for constructing similarity graphs. For instance, when employing the fast Gauss transform [10] as the KDE solver, Theorem 1 shows that a sparse representation of the fully connected similarity graph with the Gaussian kernel can be constructed in  $\widetilde{O}(n)$  time when  $d$  is constant. As such, in the case of low dimensions, spectral clustering runs as fast (up to a poly-logarithmic factor) as the time needed to read the input data points. Moreover, any improved algorithm for the KDE would result in a faster construction of approximate similarity graphs.
To demonstrate the significance of this work in practice, we compare the performance of our algorithm with five competing algorithms from the well-known scikit-learn library [20] and FAISS library [11]: the algorithm that constructs a fully connected Gaussian kernel graph, and four algorithms that construct different variants of  $k$ -nearest neighbour graphs. We apply spectral clustering on the six constructed similarity graphs, and compare the quality of the resulting clustering. For a typical input dataset of 15,000 points in  $\mathbb{R}^2$ , our algorithm runs in 4.7 seconds, in comparison with between 16.1 - 128.9 seconds for the five competing algorithms from scikit-learn and FAISS libraries. As shown in Figure 1, all the six algorithms return reasonable output.
We further compare the quality of the six algorithms on the BSDS image segmentation dataset [2], and our algorithm presents clear improvement over the other five algorithms based on the output's average Rand Index. In particular, due to its quadratic memory requirement in the input size, one would need to reduce the resolution of every image down to 20,000 pixels in order to construct the fully connected similarity graph with scikit-learn on a typical laptop. In contrast, our algorithm is able to segment the full-resolution image with over 150,000 pixels. Our experimental result on
<sup>1</sup>We use  $\widetilde{O}(n)$  to represent  $O\left(n \cdot \log^c(n)\right)$  for some constant  $c$ .
Figure 1: Output of spectral clustering with different similarity graph constructions.
the BSDS dataset is showcased in Figure 2 and demonstrates that, in comparison with SKLEARN GK, our algorithm identifies a more detailed pattern on the butterfly's wing. In contrast, none of the  $k$ -nearest neighbour based algorithms from the two libraries is able to identify the wings of the butterfly.
(a) Original Image
(b) SKLEARN GK
(c) OUR ALGORITHM
(d) SKLEARN  $k$ -NN
(e) FAISS EXACT
(f) FAISS HNSW
Figure 2: Comparison on the performance of spectral clustering with different similarity graph constructions. Here, SKLEARN GK is based on the fully connected similarity graph construction, and (d) - (g) are based on different  $k$ -nearest neighbour graph constructions from the two libraries.
(g) FAISS IVF
There are a number of works on efficient constructions of  $\varepsilon$ -neighbourhood graphs and  $k$ -NN graphs. For instance, Dong et al. [9] presents an algorithm for approximate  $k$ -NN graph construction, and their algorithm is based on local search. Wang et al. [32] presents an LSH-based algorithm for constructing an approximate  $k$ -NN graph, and employs several sampling and hashing techniques to reduce the computational and parallelisation cost. These two algorithms [9, 32] have shown to work very well in practice, but lack a theoretical guarantee on the performance.
Our work also relates to a large and growing number of KDE algorithms. Charikar and Siminelakis [7] study the KDE problem through LSH, and present a class of unbiased estimators for kernel density in high dimensions for a variety of commonly used kernels. Their work has been improved through the sketching technique [24], and a revised description and analysis of the original algorithm [4]. Charikar et al. [6] presents a data structure for the KDE problem, and their result essentially matches the query time and space complexity for most studied kernels in the literature. In addition, there are studies on designing efficient KDE algorithms based on interpolation of kernel density estimators [29], and coresets [12].
Our work further relates to efficient constructions of spectral sparsifiers for kernel graphs. Quan-rud [22] studies smooth kernel functions, and shows that an explicit  $(1 + \varepsilon)$ -approximate spectral approximation of the geometric graph with  $\widetilde{O}(n / \varepsilon^2)$  edges can be computed in  $\widetilde{O}(n / \varepsilon^2)$  time. Bakshi et al. [5] proves that, under the strong exponential time hypothesis, constructing an  $O(1)$ -approximate spectral sparsifier with  $O(n^{2 - \delta})$  edges for the Gaussian kernel graph requires  $\Omega\left(n \cdot 2^{\log(1 / \tau)^{0.32}}\right)$  time, where  $\delta < 0.01$  is a fixed universal constant and  $\tau$  is the minimum entry of the kernel matrix. Compared with their results, we show that, when the similarity graph with the Gaussian kernel presents a well-defined structure of clusters, an approximate construction of this similarity graph can be constructed in nearly-linear time.
Let  $\mathsf{G} = (V, E, w_{\mathsf{G}})$  be an undirected graph with weight function  $w_{\mathsf{G}}: E \to \mathbb{R}_{\geq 0}$ , and  $n \triangleq |V|$ . The degree of any vertex  $v$  is defined as  $\deg_{\mathsf{G}}(v) \triangleq \sum_{u \sim v} w_{\mathsf{G}}(u, v)$ , where we write  $u \sim v$  if  $\{u, v\} \in E(\mathsf{G})$ . For any  $S \subset V$ , the volume of  $S$  is defined by  $\mathrm{vol}_{\mathsf{G}}(S) \triangleq \sum_{v \in S} \deg_{\mathsf{G}}(v)$ , and the conductance of  $S$  is defined by
$$
\phi_ {\mathbf {G}} (S) \triangleq \frac {\partial_ {\mathbf {G}} (S)}{\operatorname {v o l} _ {\mathbf {G}} (S)},
$$
where  $\partial_{\mathsf{G}}(S)\triangleq \sum_{u\in S,v\notin S}w_{\mathsf{G}}(u,v)$ . For any  $k\geq 2$ , we call subsets of vertices  $A_{1},\ldots ,A_{k}$  a  $k$ -way partition if  $A_{i}\neq \emptyset$  for any  $1\leq i\leq k$ ,  $A_{i}\cap A_{j} = \emptyset$  for any  $i\neq j$ , and  $\bigcup_{i = 1}^{k}A_{i} = V$ . Moreover, we define the  $k$ -way expansion constant by
$$
\rho_ {\mathbb {G}} (k) \triangleq \min  _ {\text {p a r t i t i o n} A _ {1}, \ldots , A _ {k}} \max  _ {1 \leq i \leq k} \phi_ {\mathbb {G}} (A _ {i}).
$$
Note that a lower value of  $\rho_{\mathbf{G}}(k)$  ensures the existence of  $k$  clusters  $A_1, \ldots, A_k$  of low conductance, i.e.,  $\mathbf{G}$  has at least  $k$  clusters.
For any undirected graph  $G$ , the adjacency matrix  $\mathbf{A}_G$  of  $G$  is defined by  $\mathbf{A}_G(u, v) = w_G(u, v)$  if  $u \sim v$ , and  $\mathbf{A}_G(u, v) = 0$  otherwise. We write  $\mathbf{D}_G$  as the diagonal matrix defined by  $\mathbf{D}_G(v, v) = \deg_G(v)$ , and the normalised Laplacian of  $G$  is defined by  $\mathbf{N}_G \triangleq \mathbf{I} - \mathbf{D}_G^{-1/2} \mathbf{A}_G \mathbf{D}_G^{-1/2}$ . For any PSD matrix  $\mathbf{B} \in \mathbb{R}^{n \times n}$ , we write the eigenvalues of  $\mathbf{B}$  as  $\lambda_1(\mathbf{B}) \leq \ldots \leq \lambda_n(\mathbf{B})$ .
It is well-known that, while computing  $\rho_{\mathsf{G}}(k)$  exactly is NP-hard,  $\rho_{\mathsf{G}}(k)$  is closely related to  $\lambda_{k}$  through the higher-order Cheeger inequality [13]: it holds for any  $k$  that
$$
\lambda_ {k} \left(\mathrm {N} _ {\mathrm {G}}\right) / 2 \leq \rho_ {\mathrm {G}} (k) \leq O \left(k ^ {3}\right) \sqrt {\lambda_ {k} \left(\mathrm {N} _ {\mathrm {G}}\right)}.
$$
2.1 Fully Connected Similarity Graphs
We use  $X \triangleq \{x_1, \ldots, x_n\}$  to represent the set of input data points, where every  $x_i \in \mathbb{R}^d$ . Given  $X$  and some kernel function  $k: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}_{\geq 0}$ , we use  $\mathsf{K} = (V_{\mathsf{K}}, E_{\mathsf{K}}, w_{\mathsf{K}})$  to represent the fully connected similarity graph from  $X$ , which is constructed as follows: every  $v_i \in V_{\mathsf{K}}$  corresponds to  $x_i \in X$ , and any pair of different  $v_i$  and  $v_j$  is connected by an edge with weight  $w_{\mathsf{K}}(v_i, v_j) = k(x_i, x_j)$ . One of the most common kernels used for clustering is the Gaussian kernel, which is defined by
$$
k (x _ {i}, x _ {j}) = \exp \left(- \frac {\| x _ {i} - x _ {j} \| _ {2} ^ {2}}{\sigma^ {2}}\right)
$$
for some bandwidth parameter  $\sigma$ . Other popular kernels include the Laplacian kernel and the exponential kernel which use  $\|x_i - x_j\|_1$  and  $\|x_i - x_j\|_2$  in the exponent respectively.
2.2 Kernel Density Estimation
Our work is based on algorithms for kernel density estimation (KDE), which is defined as follows. Given a kernel function  $k: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}_{\geq 0}$  with  $n$  source points  $x_1, \ldots, x_n \in \mathbb{R}^d$  and  $m$  target points  $y_1, \ldots, y_m \in \mathbb{R}^d$ , the KDE problem is to compute  $g_{[1,n]}(y_1), \ldots, g_{[1,n]}(y_m)$ , where
$$
g _ {[ a, b ]} \left(y _ {i}\right) \triangleq \sum_ {j = a} ^ {b} k \left(y _ {i}, x _ {j}\right) \tag {1}
$$
for  $1 \leq i \leq m$ . While a direct computation of the  $m$  values  $g_{[1,n]}(y_1), \ldots, g_{[1,n]}(y_m)$  requires  $mn$  operations, there is substantial research to develop faster algorithms approximating these  $m$  quantities.
In this paper we are interested in the algorithms that approximately compute  $g_{[1,n]}(y_i)$  for all  $1 \leq i \leq m$  up to a  $(1 \pm \epsilon)$ -multiplicative error, and use  $T_{\mathsf{KDE}}(m,n,\epsilon)$  to denote the asymptotic complexity of such a KDE algorithm. We also require that  $T_{\mathsf{KDE}}(m,n,\epsilon)$  is superadditive in  $m$  and  $n$ ; that is, for  $m = m_1 + m_2$  and  $n = n_1 + n_2$ , we have
$$
T _ {\mathrm {K D E}} \left(m _ {1}, n _ {1}, \epsilon\right) + T _ {\mathrm {K D E}} \left(m _ {2}, n _ {2}, \epsilon\right) \leq T _ {\mathrm {K D E}} \left(m, n, \epsilon\right);
$$
it is known that such property holds for many KDE algorithms (e.g., [1, 6, 10]).
3 Cluster-Preserving Sparsifiers
A graph sparsifier is a sparse representation of an input graph that inherits certain properties of the original dense graph. The efficient construction of sparsifiers plays an important role in designing a number of nearly-linear time graph algorithms. However, most algorithms for constructing sparsifiers rely on the recursive decomposition of an input graph [26], sampling with respect to effective resistances [15, 25], or fast SDP solvers [14]; all of these need the explicit representation of an input graph, requiring  $\Omega(n^2)$  time and space complexity for a fully connected graph.
Sun and Zanetti [27] study a variant of graph sparsifiers that mainly preserve the cluster structure of an input graph, and introduce the notion of cluster-preserving sparsifier defined as follows:
Definition 1 (Cluster-preserving sparsifier). Let  $\mathsf{K} = (V,E,w_{\mathsf{K}})$  be any graph, and  $\{A_i\}_{i=1}^k$  the  $k$ -way partition of  $\mathsf{K}$  corresponding to  $\rho_{\mathsf{K}}(k)$ . We call a re-weighted subgraph  $\mathsf{G} = (V,F\subset E,w_{\mathsf{G}})$  a cluster-preserving sparsifier of  $\mathsf{K}$  if  $\phi_{\mathsf{G}}(A_i) = O(k\cdot \phi_{\mathsf{K}}(A_i))$  for  $1\leq i\leq k$ , and  $\lambda_{k+1}(\mathbf{N}_{\mathsf{G}}) = \Omega(\lambda_{k+1}(\mathbf{N}_{\mathsf{K}}))$ .
Notice that graph  $\mathsf{K}$  has exactly  $k$  clusters if (i)  $\mathsf{K}$  has  $k$  disjoint subsets  $A_{1},\ldots ,A_{k}$  of low conductance, and (ii) any  $(k + 1)$ -way partition of  $\mathsf{K}$  would include some  $A\subset V$  of high conductance, which would be implied by a lower bound on  $\lambda_{k + 1}(\mathbf{N}_{\mathsf{K}})$  due to the higher-order Cheeger inequality. Together with the well-known eigen-gap heuristic [13, 31] and theoretical analysis on spectral clustering [16, 21], the two properties in Definition 1 ensures that spectral clustering returns approximately the same output from  $\mathsf{K}$  and  $\mathsf{H}$ .<sup>2</sup>
Now we present the algorithm in [27] for constructing a cluster-preserving sparsifier, and we call it the SZ algorithm for simplicity. Given any input graph  $\mathsf{K} = (V,E,w_{\mathsf{K}})$  with weight function  $w_{\mathsf{K}}$ , the algorithm computes
$$
p _ {u} (v) \triangleq \min  \left\{C \cdot \frac {\log n}{\lambda_ {k + 1}} \cdot \frac {w _ {\mathsf {K}} (u , v)}{\deg_ {\mathsf {K}} (u)}, 1 \right\}, \quad \text {a n d} \quad p _ {v} (u) \triangleq \min  \left\{C \cdot \frac {\log n}{\lambda_ {k + 1}} \cdot \frac {w _ {\mathsf {K}} (v , u)}{\deg_ {\mathsf {K}} (v)}, 1 \right\},
$$
for every edge  $e = \{u, v\}$ , where  $C \in \mathbb{R}^+$  is some constant. Then, the algorithm samples every edge  $e = \{u, v\}$  with probability
$$
p _ {e} \triangleq p _ {u} (v) + p _ {v} (u) - p _ {u} (v) \cdot p _ {v} (u),
$$
and sets the weight of every sampled  $e = \{u, v\}$  in  $\mathsf{G}$  as  $w_{\mathsf{G}}(u, v) \triangleq w_{\mathsf{K}}(u, v) / p_e$ . By setting  $F$  as the set of the sampled edges, the algorithm returns  $\mathsf{G} = (V, F, w_{\mathsf{G}})$  as output. It is shown in [27] that, with high probability, the constructed  $\mathsf{G}$  has  $\widetilde{O}(n)$  edges and is a cluster-preserving sparsifier of  $\mathsf{K}$ .
The most interesting regime for this definition is  $k = \widetilde{O}(1)$  and  $\lambda_{k+1}(\mathbf{N}_{\mathsf{K}}) = \Omega(1)$ , and we assume this in the rest of the paper.
We remark that a cluster-preserving sparsifier is a much weaker notion than a spectral sparsifier, which approximately preserves all the cut values and the eigenvalues of the graph Laplacian matrices. On the other side, while a cluster-preserving sparsifier is sufficient for the task of graph clustering, the SZ algorithm runs in  $\Omega(n^2)$  time for a fully connected input graph, since it's based on the computation of the vertex degrees as well as the sampling probabilities  $p_u(v)$  for every pair of vertices  $u$  and  $v$ .
This section presents our algorithm that directly constructs an approximation of a fully connected similarity graph from  $X \subseteq \mathbb{R}^d$  with  $|X| = n$ . As the main theoretical contribution, we demonstrate that neither the quadratic space complexity for directly constructing a fully connected similarity graph nor the quadratic time complexity of the SZ algorithm is necessary when approximately constructing a fully connected similarity graph for the purpose of clustering. The performance of our algorithm is as follows:
Theorem 2 (Main Result). Given a set of data points  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  as input, there is a randomised algorithm that constructs a sparse graph  $G$  of  $X$ , such that it holds with probability at least 9/10 that
1. graph  $\mathsf{G}$  has  $\widetilde{O}(n)$  edges,
2. graph  $\mathsf{G}$  has the same cluster structure as the fully connected similarity graph  $\mathsf{K}$  of  $X$ ; that is, if  $\mathsf{K}$  has  $k$  well-defined clusters, then it holds that  $\rho_{\mathsf{G}}(k) = O(k \cdot \rho_{\mathsf{K}}(k))$  and  $\lambda_{k+1}(\mathbf{N}_{\mathsf{G}}) = \Omega(\lambda_{k+1}(\mathbf{N}_{\mathsf{K}}))$ .
The algorithm uses an approximate KDE algorithm as a black-box, and has running time  $\widetilde{O}(T_{\mathsf{KDE}}(n,n,\epsilon))$  for  $\epsilon \leq 1 / (6\log(n))$ .
At a very high level, our designed algorithm applies a KDE algorithm as a black-box, and constructs a cluster-preserving sparsifier by simultaneous sampling of the edges from a non-explicitly constructed fully connected graph. To explain our technique, we first claim that, for an arbitrary  $x_{i}$ , a random  $x_{j}$  can be sampled with probability  $k(x_{i},x_{j}) / \deg_{\mathsf{K}}(v_{i})$  through  $O(\log n)$  queries to a KDE algorithm. To see this, notice that we can apply a KDE algorithm to compute the probability that the sampled neighbour is in some set  $X_{1}\subset X$ , i.e.,
$$
\mathbb {P} \left[ z \in X _ {1} \right] = \sum_ {x _ {j} \in X _ {1}} \frac {k \left(x _ {i} , x _ {j}\right)}{\deg_ {\mathsf {K}} (v _ {i})} = \frac {g _ {X _ {1}} \left(x _ {i}\right)}{g _ {X} \left(x _ {i}\right)},
$$
where we use  $g_{X}(y_{i})$  to denote that the KDE is taken with respect to the set of source points  $X$ . Based on this, we recursively split the set of possible neighbours in half and choose between the two subsets with the correct probability. The sampling procedure is summarised as follows, and is illustrated in Figure 3. We remark that the method of sampling a random neighbour of a vertex in  $K$  through KDE and binary search also appears in Backurs et al. [5]
1. Set the feasible neighbours to be  $X = \{x_{1},\ldots ,x_{n}\}$
2. While  $|X| > 1$ :
- Split  $X$  into  $X_{1}$  and  $X_{2}$  with  $|X_{1}| = \lfloor |X| / 2 \rfloor$  and  $|X_{2}| = \lceil |X| / 2 \rceil$ .
- Compute  $g_{X}(x_{i})$  and  $g_{X_1}(x_i)$ ; set  $X\gets X_{1}$  with probability  $g_{X_1}(x_i) / g_X(x_i)$ , and  $X\gets X_{2}$  with probability  $1 - g_{X_1}(x_i) / g_X(x_i)$ .
3. Return the remaining element in  $X$  as the sampled neighbour.
Next we generalise this idea and show that, instead of sampling a neighbour of one vertex at a time, a KDE algorithm allows us to sample neighbours of every vertex in the graph "simultaneously". Our designed sampling procedure is formalised in Algorithm 1.
Finally, to construct a cluster-preserving sparsifier, we apply Algorithm 1 to sample  $O(\log n)$  neighbours for every vertex  $v_{i}$ , and set the weight of every sampled edge  $v_{i} \sim v_{j}$  as
$$
w _ {\mathbf {G}} (v _ {i}, v _ {j}) = \frac {k \left(x _ {i} , x _ {j}\right)}{\widehat {p} (i , j)}, \tag {2}
$$
Figure 3: The procedure of sampling a neighbour  $v_{j}$  of  $v_{i}$  with probability  $k(x_{i},x_{j}) / \deg_{\mathsf{K}}(v_{i})$ . Our algorithm performs a binary search to find the sampled neighbour. At each step, the value of two kernel density estimates are used to determine where the sample lies. Notice that the algorithm doesn't compute any edge weights directly until the last step.