Title: Fast Approximation of Similarity Graphs with Kernel Density Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: similarity graphs

spectral clustering
Abstract: Constructing a similarity graph from a set $X$ of data points in $ \mathbb{R}^d$ is the first step of many modern clustering algorithms. However, typical constructions of a similarity graph have high time complexity, and a quadratic space dependency with respect to $|X|$. We address this limitation and present a new algorithmic framework that constructs a sparse approximation of the fully connected similarity graph while preserving its cluster structure. Our presented algorithm is based on the kernel density estimation problem, and is applicable for arbitrary kernel functions. We compare our designed algorithm with the  well-known implementations from the scikit-learn library and the FAISS library,  and find that our method significantly outperforms the implementation from both libraries on a variety of datasets.

Fast Approximation of Similarity Graphs with Kernel Density Estimation

Abstract

Constructing a similarity graph from a set X of data points in R d is the first step of many modern clustering algorithms. However, typical constructions of a similarity graph have high time complexity, and a quadratic space dependency with respect to | X | . We address this limitation and present a new algorithmic framework that constructs a sparse approximation of the fully connected similarity graph while preserving its cluster structure. Our presented algorithm is based on the kernel density estimation problem, and is applicable for arbitrary kernel functions. We compare our designed algorithm with the well-known implementations from the scikit-learn library and the FAISS library, and find that our method significantly outperforms the implementation from both libraries on a variety of datasets.

1 Introduction

Given a set X = { x 1 , . . . , x n } ⊂ R d of data points and a similarity function k : R d × R d → R ≥ 0 for any pair of data points x i and x j , the objective of clustering is to partition these n data points into clusters such that similar points are in the same cluster. As a fundamental data analysis technique, clustering has been extensively studied in different disciplines ranging from algorithms and machine learning, social network analysis, to data science and statistics.

One prominent approach for clustering data points in Euclidean space consists of two simple steps: the first step is to construct a similarity graph K = ( V, E, w ) from X , where every vertex v i of G corresponds to x i ∈ X , and different vertices v i and v j are connected by an edge with weight w ( v i , v j ) if their similarity k ( x i , x j ) is positive, or higher than some threshold. Secondly, we apply spectral clustering on G , and its output naturally corresponds to some clustering on X [19]. Because of its out-performance over traditional clustering algorithms like k -means, this approach has become one of the most popular modern clustering algorithms.

On the other side, different constructions of similarity graphs have significant impact on the quality and time complexity of spectral clustering, which is clearly acknowledged and appropriately discussed by von Luxburg [31]. Generally speaking, there are two types of similarity graphs:

the first one is the k -nearest neighbour graph ( k -NN graph), in which every vertex v i connects to v j if v j is among the k -nearest neighbours of v i . A k -NN graph is sparse by construction, but loses some of the structural information in the dataset since k is usually small and the added edges are unweighted.

the second one is the fully connected graph, in which different vertices v i and v j are connected with weight w ( v i , v j ) = k ( x i , x j ) . While a fully connected graph maintains most of the distance information about X , this graph is dense and storing such graphs requires quadratic memory in n .

Taking the pros and cons of the two constructions into account, one would naturally ask the question:

Is it possible to directly construct a sparse graph that preserves the cluster structure of a fully connected similarity graph?

We answer this question affirmatively, and present a fast algorithm that constructs an approximation of the fully connected similarity graph. Our constructed graph consists of only ˜ O ( n ) edges 1 , and preserves the cluster structure of the fully connected similarity graph.

1.1 Our Result

Given any set X = { x 1 , . . . , x n } ⊂ R d and a kernel function k : R d × R d → R ≥ 0 , a fully connected similarity graph K = ( V, E, w ) of X consists of n vertices, and every v i ∈ V corresponds to x i ∈ X ; we set w ( v i , v j ) ≜ k ( x i , x j ) for any different v i and v j . We introduce an efficient algorithm that constructs a sparse graph G directly from X , such that K and G share the same cluster-structure, and the graph matrices for K and G have approximately the same eigen-gap. This ensures that spectral clustering from G and K return approximately the same result.

The design of our algorithm is based on a novel reduction from the approximate construction of similarity graphs to the problem of Kernel Density Estimation ( KDE ). This reduction shows that any algorithm for the KDE can be employed to construct a sparse representation of a fully connected similarity graph, while preserving the cluster-structure of the input data points. This is summarised as follows:

Theorem 1 (Informal Statement of Theorem 2) . Given a set of data points X = { x 1 , . . . , x n } ⊂ R d as input, there is a randomised algorithm that constructs a sparse graph G of X , such that it holds with probability at least 9 / 10 that

graph G has ˜ O ( n ) edges,

graph G has the same cluster structure as the fully connected similarity graph K of X .

The algorithm uses an approximate KDE algorithm as a black-box, and has running time ˜ O ( T KDE ( n, n, ϵ )) for ϵ ≤ 1 / (6 log( n )) , where T KDE ( n, n, ϵ ) is the running time of solving the KDE problem for n data points up to a (1 + ϵ ) -approximation.

This result builds a novel connection between the KDE and the fast construction of similarity graphs, and further represents a state-of-the-art algorithm for constructing similarity graphs. For instance, when employing the fast Gauss transform [10] as the KDE solver, Theorem 1 shows that a sparse representation of the fully connected similarity graph with the Gaussian kernel can be constructed in ˜ O ( n ) time when d is constant. As such, in the case of low dimensions, spectral clustering runs as fast (up to a poly-logarithmic factor) as the time needed to read the input data points. Moreover, any improved algorithm for the KDE would result in a faster construction of approximate similarity graphs.

To demonstrate the significance of this work in practice, we compare the performance of our algorithm with five competing algorithms from the well-known scikit-learn library [20] and FAISS library [11]: the algorithm that constructs a fully connected Gaussian kernel graph, and four algorithms that construct different variants of k -nearest neighbour graphs. We apply spectral clustering on the six constructed similarity graphs, and compare the quality of the resulting clustering. For a typical input dataset of 15,000 points in R 2 , our algorithm runs in 4.7 seconds, in comparison with between 16.1 - 128.9 seconds for the five competing algorithms from scikit-learn and FAISS libraries. As shown in Figure 1, all the six algorithms return reasonable output.

We further compare the quality of the six algorithms on the BSDS image segmentation dataset [2], and our algorithm presents clear improvement over the other five algorithms based on the output's average Rand Index. In particular, due to its quadratic memory requirement in the input size, one would need to reduce the resolution of every image down to 20,000 pixels in order to construct the fully connected similarity graph with scikit-learn on a typical laptop. In contrast, our algorithm is able to segment the full-resolution image with over 150,000 pixels. Our experimental result on

FAISS EXACT

FAISS HNSW

FAISS IVF

OURALGORITHM

95.91s

65.64s

48.40s

38.60s

36.76s

4.46s

128.88s

56.48s

73.34s

37.87s

47.10s

4.52s

29.13s

55.08s

31.70s

12.26s

16.07s

4.75s

the BSDS dataset is showcased in Figure 2 and demonstrates that, in comparison with SKLEARN GK, our algorithm identifies a more detailed pattern on the butterfly's wing. In contrast, none of the k -nearest neighbour based algorithms from the two libraries is able to identify the wings of the butterfly.

(a) Original Image

(b) SKLEARN GK

(c) OUR ALGORITHM

(d) SKLEARN

k

-NN

(e) FAISS EXACT

(f) FAISS HNSW

(g) FAISS IVF

1.2 Related Work

There are a number of works on efficient constructions of ε -neighbourhood graphs and k -NN graphs. For instance, Dong et al. [9] presents an algorithm for approximate k -NN graph construction, and their algorithm is based on local search. Wang et al. [32] presents an LSH -based algorithm for constructing an approximate k -NN graph, and employs several sampling and hashing techniques to reduce the computational and parallelisation cost. These two algorithms [9, 32] have shown to work very well in practice, but lack a theoretical guarantee on the performance.

Our work also relates to a large and growing number of KDE algorithms. Charikar and Siminelakis [7] study the KDE problem through LSH , and present a class of unbiased estimators for kernel density in high dimensions for a variety of commonly used kernels. Their work has been improved through the sketching technique [24], and a revised description and analysis of the original algorithm [4]. Charikar et al. [6] presents a data structure for the KDE problem, and their result essentially matches the query time and space complexity for most studied kernels in the literature. In addition, there are studies on designing efficient KDE algorithms based on interpolation of kernel density estimators [29], and coresets [12].

Our work further relates to efficient constructions of spectral sparsifiers for kernel graphs. Quanrud [22] studies smooth kernel functions, and shows that an explicit (1 + ε ) -approximate spectral approximation of the geometric graph with ˜ O ( n/ε 2 ) edges can be computed in ˜ O ( n/ε 2 ) time. Bakshi et al. [5] proves that, under the strong exponential time hypothesis, constructing an O (1) -approximate spectral sparsifier with O ( n 2 -δ ) edges for the Gaussian kernel graph requires Ω ( n · 2 log(1 /τ ) 0 . 32 )

time, where δ < 0 . 01 is a fixed universal constant and τ is the minimum entry of the kernel matrix. Compared with their results, we show that, when the similarity graph with the Gaussian kernel presents a well-defined structure of clusters, an approximate construction of this similarity graph can be constructed in nearly-linear time.

2 Preliminaries

Let G = ( V, E, w G ) be an undirected graph with weight function w G : E → R ≥ 0 , and n ≜ | V | . The degree of any vertex v is defined as deg G ( v ) ≜ ∑ u ∼ v w G ( u, v ) , where we write u ∼ v if { u, v } ∈ E ( G ) . For any S ⊂ V , the volume of S is defined by vol G ( S ) ≜ ∑ v ∈ S deg G ( v ) , and the conductance of S is defined by

where ∂ G ( S ) ≜ ∑ u ∈ S,v ̸∈ S w G ( u, v ) . For any k ≥ 2 , we call subsets of vertices A 1 , . . . , A k a k -way partition if A i = ∅ for any 1 ≤ i ≤ k , A i ∩ A j = ∅ for any i = j , and ⋃ k i =1 A i = V . Moreover, we define the k -way expansion constant by

̸

Note that a lower value of ρ G ( k ) ensures the existence of k clusters A 1 , . . . , A k of low conductance, i.e, G has at least k clusters.

For any undirected graph G , the adjacency matrix A G of G is defined by A G ( u, v ) = w G ( u, v ) if u ∼ v , and A G ( u, v ) = 0 otherwise. We write D G as the diagonal matrix defined by D G ( v, v ) = deg G ( v ) , and the normalised Laplacian of G is defined by N G ≜ I -D -1 / 2 G A G D -1 / 2 G . For any PSD matrix B ∈ R n × n , we write the eigenvalues of B as λ 1 ( B ) ≤ . . . ≤ λ n ( B ) .

It is well-known that, while computing ρ G ( k ) exactly is NP -hard, ρ G ( k ) is closely related to λ k through the higher-order Cheeger inequality [13]: it holds for any k that

2.1 Fully Connected Similarity Graphs

We use X ≜ { x 1 , . . . x n } to represent the set of input data points, where every x i ∈ R d . Given X and some kernel function k : R d × R d → R ≥ 0 , we use K = ( V K , E K , w K ) to represent the fully connected similarity graph from X , which is constructed as follows: every v i ∈ V K corresponds to x i ∈ X , and any pair of different v i and v j is connected by an edge with weight w K ( v i , v j ) = k ( x i , x j ) . One of the most common kernels used for clustering is the Gaussian kernel, which is defined by

for some bandwidth parameter σ . Other popular kernels include the Laplacian kernel and the exponential kernel which use ∥ x i -x j ∥ 1 and ∥ x i -x j ∥ 2 in the exponent respectively.

̸

2.2 Kernel Density Estimation

Our work is based on algorithms for kernel density estimation ( KDE ), which is defined as follows. Given a kernel function k : R d × R d → R ≥ 0 with n source points x 1 , . . . , x n ∈ R d and m target points y 1 , . . . , y m ∈ R d , the KDE problem is to compute g [1 ,n ] ( y 1 ) , . . . g [1 ,n ] ( y m ) , where

for 1 ≤ i ≤ m . While a direct computation of the m values g [1 ,n ] ( y 1 ) , . . . g [1 ,n ] ( y m ) requires mn operations, there is substantial research to develop faster algorithms approximating these m quantities.

In this paper we are interested in the algorithms that approximately compute g [1 ,n ] ( y i ) for all 1 ≤ i ≤ m up to a (1 ± ϵ ) -multiplicative error, and use T KDE ( m,n,ϵ ) to denote the asymptotic complexity of such a KDE algorithm. We also require that T KDE ( m,n,ϵ ) is superadditive in m and n ; that is, for m = m 1 + m 2 and n = n 1 + n 2 , we have

it is known that such property holds for many KDE algorithms (e.g., [1, 6, 10]).

3 Cluster-Preserving Sparsifiers

A graph sparsifier is a sparse representation of an input graph that inherits certain properties of the original dense graph. The efficient construction of sparsifiers plays an important role in designing a number of nearly-linear time graph algorithms. However, most algorithms for constructing sparsifiers rely on the recursive decomposition of an input graph [26], sampling with respect to effective resistances [15, 25], or fast SDP solvers [14]; all of these need the explicit representation of an input graph, requiring Ω( n 2 ) time and space complexity for a fully connected graph.

Sun and Zanetti [27] study a variant of graph sparsifiers that mainly preserve the cluster structure of an input graph, and introduce the notion of cluster-preserving sparsifier defined as follows:

Definition 1 (Cluster-preserving sparsifier) . Let K = ( V, E, w K ) be any graph, and { A i } k i =1 the k -way partition of K corresponding to ρ K ( k ) . We call a re-weighted subgraph G = ( V, F ⊂ E,w G ) a cluster-preserving sparsifier of K if ϕ G ( A i ) = O ( k · ϕ K ( A i )) for 1 ≤ i ≤ k , and λ k +1 ( N G ) = Ω( λ k +1 ( N K )) .

Notice that graph K has exactly k clusters if (i) K has k disjoint subsets A 1 , . . . , A k of low conductance, and (ii) any ( k + 1) -way partition of K would include some A ⊂ V of high conductance, which would be implied by a lower bound on λ k +1 ( N K ) due to the higher-order Cheeger inequality. Together with the well-known eigen-gap heuristic [13, 31] and theoretical analysis on spectral clustering [16, 21], the two properties in Definition 1 ensures that spectral clustering returns approximately the same output from K and H . 2

Now we present the algorithm in [27] for constructing a cluster-preserving sparsifier, and we call it the SZ algorithm for simplicity. Given any input graph K = ( V, E, w K ) with weight function w K , the algorithm computes

for every edge e = { u, v } , where C ∈ R + is some constant. Then, the algorithm samples every edge e = { u, v } with probability

and sets the weight of every sampled e = { u, v } in G as w G ( u, v ) ≜ w K ( u, v ) /p e . By setting F as the set of the sampled edges, the algorithm returns G = ( V, F, w G ) as output. It is shown in [27] that, with high probability, the constructed G has ˜ O ( n ) edges and is a cluster-preserving sparsifier of K .

We remark that a cluster-preserving sparsifier is a much weaker notion than a spectral sparsifier, which approximately preserves all the cut values and the eigenvalues of the graph Laplacian matrices. On the other side, while a cluster-preserving sparsifier is sufficient for the task of graph clustering, the SZ algorithm runs in Ω( n 2 ) time for a fully connected input graph, since it's based on the computation of the vertex degrees as well as the sampling probabilities p u ( v ) for every pair of vertices u and v .

4 Algorithm

This section presents our algorithm that directly constructs an approximation of a fully connected similarity graph from X ⊆ R d with | X | = n . As the main theoretical contribution, we demonstrate that neither the quadratic space complexity for directly constructing a fully connected similarity graph nor the quadratic time complexity of the SZ algorithm is necessary when approximately constructing a fully connected similarity graph for the purpose of clustering. The performance of our algorithm is as follows:

Theorem 2 (Main Result) . Given a set of data points X = { x 1 , . . . , x n } ⊂ R d as input, there is a randomised algorithm that constructs a sparse graph G of X , such that it holds with probability at least 9 / 10 that

graph G has ˜ O ( n ) edges,

graph G has the same cluster structure as the fully connected similarity graph K of X ; that is, if K has k well-defined clusters, then it holds that ρ G ( k ) = O ( k · ρ K ( k )) and λ k +1 ( N G ) = Ω( λ k +1 ( N K )) .

The algorithm uses an approximate KDE algorithm as a black-box, and has running time ˜ O ( T KDE ( n, n, ϵ )) for ϵ ≤ 1 / (6 log( n )) .

4.1 Algorithm Description

At a very high level, our designed algorithm applies a KDE algorithm as a black-box, and constructs a cluster-preserving sparsifier by simultaneous sampling of the edges from a non-explicitly constructed fully connected graph. To explain our technique, we first claim that, for an arbitrary x i , a random x j can be sampled with probability k ( x i , x j ) / deg K ( v i ) through O (log n ) queries to a KDE algorithm. To see this, notice that we can apply a KDE algorithm to compute the probability that the sampled neighbour is in some set X 1 ⊂ X , i.e.,

where we use g X ( y i ) to denote that the KDE is taken with respect to the set of source points X . Based on this, we recursively split the set of possible neighbours in half and choose between the two subsets with the correct probability. The sampling procedure is summarised as follows, and is illustrated in Figure 3. We remark that the method of sampling a random neighbour of a vertex in K through KDE and binary search also appears in Backurs et al. [5]

Set the feasible neighbours to be X = { x 1 , . . . , x n } .

While | X | > 1 :

Split X into X 1 and X 2 with | X 1 | = ⌊| X | / 2 ⌋ and | X 2 | = ⌈| X | / 2 ⌉ .

Compute g X ( x i ) and g X 1 ( x i ) ; set X ← X 1 with probability g X 1 ( x i ) /g X ( x i ) , and X ← X 2 with probability 1 -g X 1 ( x i ) /g X ( x i ) .

Return the remaining element in X as the sampled neighbour.

Next we generalise this idea and show that, instead of sampling a neighbour of one vertex at a time, a KDE algorithm allows us to sample neighbours of every vertex in the graph 'simultaneously'. Our designed sampling procedure is formalised in Algorithm 1.

Finally, to construct a cluster-preserving sparsifier, we apply Algorithm 1 to sample O (log n ) neighbours for every vertex v i , and set the weight of every sampled edge v i ∼ v j as

Step

1

r

∼

Unif[0

,

1]

︸

︷︷

︸

g

[0

,n/

2]

(

x

i

)

︸

︷︷

︸

g

[

n/

2

,n

]

(

x

i

)

Step

2

︸

︷︷

︸

g

[0

,n/

4]

(

x

i

)

︸

︷︷

︸

g

[

n/

4

,n/

2]

(

x

i

)

.

.

.

Step

log

2

(

n

)

︸

︷︷

︸

k

(

x

i

, x

j

)

︸

︷︷

︸

k

(

x

i

, x

j

+1

)

)

where ̂ p ( i, j ) ≜ ̂ p i ( j ) + ̂ p j ( i ) -̂ p i ( j ) · ̂ p j ( i ) is an estimate of the sampling probability of edge v i ∼ v j , and

for some constant C ∈ R + ; see Algorithm 2 for the formal description.

4.2 Algorithm Analysis

Now we analyse Algorithm 2, and sketch the proof of Theorem 2. We first analyse the running time of Algorithm 2. Since it involves O (log n ) executions of Algorithm 1 in total, it is sufficient to examine the running time of Algorithm 1.

We visualise the recursion of Algorithm 1 with respect to S and X in Figure 4. Notice that, although the number of nodes doubles at each level of the recursion tree, the total number of samples S and data points X remain constant for each level of the tree: it holds for any i that ⋃ 2 i j =1 S i,j = S 0 and

(

S

0

, X

0

)

(

S

1

,

1

, X

1

,

1

)

(

S

1

,

2

, X

1

,

2

)

(

S

2

,

1

, X

2

,

1

)

Algorithm 2 FASTSIMILARITYGRAPH

⋃ 2 i j =1 X i,j = X 0 . Since the running time of the KDE algorithm is superadditive, the combined running time of all nodes at level i of the tree is

Hence, the total running time of Algorithm 2 is ˜ O ( T KDE ( n, n, ϵ )) as the tree has log 2 ( n ) levels. Moreover, when applying the Fast Gauss Transform ( FGT ) [10] as the KDE algorithm, the total running time of Algorithm 1 is ˜ O ( n ) when d = O (1) .

Finally, we prove that the output of Algorithm 2 is a cluster preserving sparsifier of a fully connected similarity graph, and has ˜ O ( n ) edges. Notice that, comparing with the sampling probabilities p u ( v ) and p v ( u ) used in the SZ algorithm, Algorithm 2 samples each edge through O (log n ) recursive executions of a KDE algorithm, each of which introduces a (1 + ϵ ) -multiplicative error. We carefully examine these (1 + ϵ ) -multiplicative errors and prove that the actual sampling probability ˜ p ( i, j ) used in Algorithm 2 is an O (1) -approximation of p e for every e = { v i , v j } . As such the output of Algorithm 2 is a cluster-preserving sparsifier of a fully connected similarity graph, and satisfies the two properties of Theorem 2. The total number of edges in G follows from the sampling scheme. We refer the reader to Appendix A for the complete proof of Theorem 2.

5 Experiments

In this section, we empirically evaluate the performance of spectral clustering with our new algorithm for constructing similarity graphs. We compare our algorithm with the algorithms for similarity graph construction provided by the scikit-learn library [20] and the FAISS library [11] which are commonly used machine learning libraries for Python. In the remainder of this section, we compare the following six spectral clustering methods.

SKLEARN GK: spectral clustering with the fully connected Gaussian kernel similarity graph constructed with the scikit-learn library.

SKLEARN k -NN: spectral clustering with the k -nearest neighbour similarity graph constructed with the scikit-learn library.

FAISS EXACT: spectral clustering with the exact k -nearest neighbour graph constructed with the FAISS library.

FAISS HNSW: spectral clustering with the approximate k -nearest neighbour graph constructed with the 'Hierarchical Navigable Small World' method [18].

FAISS IVF: spectral clustering with the approximate k -nearest neighbour graph constructed with the 'Invertex File Index' method [3].

OUR ALGORITHM: spectral clustering with the Gaussian kernel similarity graph constructed by Algorithm 2.

Weimplement Algorithm 2 in C++, using the implementation of the Fast Gauss Transform provided by Yang et al. [33], and use the Python SciPy [30] and stag [17] libraries for eigenvector computation and graph operations respectively. The scikit-learn and FAISS libraries are well-optimised

and use C, C++, and FORTRAN for efficient implementation of core algorithms. We first employ classical synthetic clustering datasets to clearly compare how the running time of different algorithms is affected by the number of data points. This experiment highlights the nearly-linear time complexity of our algorithm. Next we demonstrate the applicability of our new algorithm for image segmentation on the Berkeley Image Segmentation Dataset (BSDS) [2].

For each experiment, we set k = 10 for the approximate nearest neighbour algorithms. For the synthetic datasets, we set the σ value of the Gaussian kernel used by SKLEARN GK and OUR ALGORITHM to be 0 . 1 , and for the BSDS experiment we set σ = 0 . 2 . This choice follows the heuristic suggested by von Luxburg [31] to choose σ to approximate the distance from a point to its k th nearest neighbour. All experiments are performed on an HP ZBook laptop with an 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz processor and 32 GB RAM. The code to reproduce our results is available at https://github.com/pmacg/kde-similarity-graph .

5.1 Synthetic Dataset

In this experiment we use the scikit-learn library to generate 15,000 data points in R 2 from a variety of classical synthetic clustering datasets. The data is generated with the make_circles , make_moons , and make_blobs methods of the scikit-learn library with a noise parameter of 0 . 05 . The linear transformation {{ 0 . 6 , -0 . 6 } , {-0 . 4 , 0 . 8 }} is applied to the blobs data to create asymmetric clusters. As shown in Figure 1, all of the algorithms return approximately the same clustering, but our algorithm runs much faster than the ones from scikit-learn and FAISS .

We further compare the speedup of our algorithm against the five competitors on the two_moons dataset with an increasing number of data points, and our result is reported in Figure 5. Notice that the running time of the scikit-learn and FAISS algorithms scales roughly quadratically with the size of the input, while the running time of our algorithm is nearly linear. Furthermore, we note that on a laptop with 32 GB RAM, the SKLEARN GK algorithm could not scale beyond 20,000 data points due to its quadratic memory requirement, while our new algorithm can cluster 1,000,000 data points in 240 seconds under the same memory constraint.

0

5,000

10,000

15,000

20,000

Number of data points

0

10

20

30

40

50

60

70

80

Running time (s)

0

50,000

100,000

Number of data points

0

10

20

30

40

50

60

70

80

Running time (s)

SKLEARN

k

-NN

FAISS EXACT

FAISS HNSW

FAISS IVF

OUR ALGORITHM

5.2 BSDS Image Segmentation Dataset

Finally we study the application of spectral clustering for image segmentation on the BSDS dataset. The dataset contains 500 images with several ground-truth segmentations for each image. Given an input image, we consider each pixel to be a point ( r, g, b, x, y ) ⊺ ∈ R 5 where r , g , b are the red, green, blue pixel values and x , y are the coordinates of the pixel in the image. Then, we construct a similarity graph based on these points, and apply spectral clustering, for which we set the number of clusters to be the median number of clusters in the provided ground truth segmentations. Our experimental result is reported in Table 1, where the 'Downsampled' version is employed to reduce the resolution of the image to 20,000 pixels in order to run the SKLEARN GK and the 'Full Resolution' one is to apply the original image of over 150,000 pixels as input. This set of experiments demonstrates our algorithm produces better clustering results with repsect to the average Rand Index [23].

6 Conclusion

In this paper we develop a new technique that constructs a similarity graph, and show that an approximation algorithm for the KDE can be employed to construct a similarity graph with proven approximation guarantee. Applying the publicly available implementations of the KDE as a blackbox, our algorithm outperforms five competing ones from the well-known scikit-learn and FAISS libraries for the classical datasets of low dimensions. We believe that our work will motivate more research on faster algorithms for the KDE in higher dimensions and their efficient implementation, as well as more efficient constructions of similarity graphs.

References

Josh Alman, Timothy Chu, Aaron Schild, and Zhao Song. Algorithms and hardness for linear algebra on geometric graphs. In 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS'20) , pages 541-552, 2020.

Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(5):898-916, 2011.

Artem Babenko and Victor Lempitsky. The inverted multi-index. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(6):1247-1260, 2014.

Arturs Backurs, Piotr Indyk, and Tal Wagner. Space and time efficient kernel density estimation in high dimensions. In 33rd Advances in Neural Information Processing Systems (NeurIPS'19) , pages 15773-15782, 2019.

Ainesh Bakshi, Piotr Indyk, Praneeth Kacham, Sandeep Silwal, and Samson Zhou. Subquadratic algorithms for kernel matrices via kernel density estimation. In 11th International Conference on Learning Representations (ICLR '23) , 2023.

Moses Charikar, Michael Kapralov, Navid Nouri, and Paris Siminelakis. Kernel density estimation through density constrained near neighbor search. In 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS'20) , pages 172-183, 2020.

Moses Charikar and Paris Siminelakis. Hashing-based-estimators for kernel density in high dimensions. In 58th Annual IEEE Symposium on Foundations of Computer Science (FOCS'17) , pages 1032-1043, 2017.

Fan Chung and Linyuan Lu. Concentration inequalities and martingale inequalities: a survey. Internet mathematics , 3(1):79-127, 2006.

Wei Dong, Moses Charikar, and Kai Li. Efficient k -nearest neighbor graph construction for generic similarity measures. In 20th International Conference on World Wide Web (WWW '11) , pages 577-586, 2011.

Leslie Greengard and John Strain. The fast gauss transform. SIAM Journal on Scientific & Statistical Computing , 12(1):79-94, 1991.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2021.

Zohar Karnin and Edo Liberty. Discrepancy, coresets, and sketches in machine learning. In 32nd Conference on Learning Theory (COLT '19) , pages 1975-1993, 2019.

James R. Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher-order Cheeger inequalities. Journal of the ACM , 61(6):37:1-37:30, 2014.

Yin Tat Lee and He Sun. An SDP-based algorithm for linear-sized spectral sparsification. In 49th Annual ACM Symposium on Theory of Computing (STOC'17) , pages 678-687, 2017.

Yin Tat Lee and He Sun. Constructing linear-sized spectral sparsification in almost-linear time. SIAM Journal on Computing , 47(6):2315-2336, 2018.

Peter Macgregor and He Sun. A tighter analysis of spectral clustering, and beyond. In 39th International Conference on Machine Learning (ICML'22) , pages 14717-14742, 2022.

Peter Macgregor and He Sun. Spectral toolkit of algorithms for graphs: Technical report (1). CoRR , abs/2304.03170, 2023.

Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on Pattern Analysis and Machine Intelligence , 42(4):824-836, 2018.

Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In 14th Advances in Neural Information Processing Systems (NeurIPS'01) , pages 849-856, 2001.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Richard Peng, He Sun, and Luca Zanetti. Partitioning well-clustered graphs: Spectral clustering works! SIAM Journal on Computing , 46(2):710-743, 2017.

Kent Quanrud. Spectral sparsification of metrics and kernels. In 32nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'21) , pages 1445-1464, 2021.

William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association , 66(336):846-850, 1971.

Paris Siminelakis, Kexin Rong, Peter Bailis, Moses Charikar, and Philip Levis. Rehashing kernel evaluation in high dimensions. In 36th International Conference on Machine Learning (ICML'19) , Proceedings of Machine Learning Research, pages 5789-5798, 2019.

Daniel A. Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing , 40(6):1913-1926, 2011.

Daniel A. Spielman and Shang-Hua Teng. Spectral sparsification of graphs. SIAM Journal on Computing , 40(4):981-1025, 2011.

He Sun and Luca Zanetti. Distributed graph clustering and sparsification. ACM Transactions on Parallel Computing , 6(3):17:1-17:23, 2019.

Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics , 12(4):389-434, 2012.

Paxton Turner, Jingbo Liu, and Philippe Rigollet. Efficient interpolation of density estimators. In 24th International Conference on Artificial Intelligence and Statistics (AISTATS '21) , pages 2503-2511, 2021.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul

van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17(4):395-416, 2007.

Yiqiu Wang, Anshumali Shrivastava, Jonathan Wang, and Junghee Ryu. Randomized algorithms accelerated over CPU-GPU for ultra-high dimensional similarity search. In 2018 International Conference on Management of Data (SIGMOD '18) , pages 889-903, 2018.

Changjiang Yang, Ramani Duraiswami, Nail A. Gumerov, and Larry Davis. Improved fast Gauss transform and efficient kernel density estimation. In 9th International Conference on Computer Vision (ICCV'03) , pages 664-671, 2003.

A Proof of Theorem 2

This section presents the complete proof of Theorem 2. Let y i, 1 , . . . , y i,L be random variables which are equal to the indices of the L points sampled for x i . Recall that by the SZ algorithm, the 'ideal' sampling probability for x j from x i is

We denote the actual sampling probability that x j is sampled from x i under Algorithm 2 to be

Finally, for each added edge, Algorithm 2 also computes an estimate of p i ( x j ) which we denote

Similar with the definition of p e in Section 3, we define

p ( i, j ) = p i ( j ) + p j ( i ) -p i ( j ) · p j ( i ) ,

˜ p ( i, j ) = ˜ p i ( j ) + ˜ p j ( i ) -˜ p i ( j ) · ˜ p j ( i ) , and

̂ p ( i, j ) = ̂ p i ( j ) + ̂ p j ( i ) -̂ p i ( j ) · ̂ p j ( i ) .

Notice that, following the convention of [27], we use p i ( j ) to refer to the probability that a given edge is sampled from the vertex x i and p ( i, j ) is the probability that the given edge { v i , v j } is sampled at all by the algorithm. We use the same convention for ˜ p i ( j ) and ̂ p i ( j ) .

We first prove a sequence of lemmas showing that these probabilities are all within a constant factor of each other.

Lemma 1. For any point x i , the probability that a given sampled neighbour y i,l is equal to j is given by

Proof. Let X = { x 1 , . . . , x n } be the input data points for Algorithm 2, and [ n ] = { 1 , . . . n } be the indices of the input data points. Furthermore, let [ a, b ] = { a, . . . , b } be the set of indices between a and b . Then, in each recursive call to Algorithm 1, we are given a range of indices [ a, b ] as input and assign y i,l to one half of it: either [ a, ⌊ b/ 2 ⌋ ] or [ ⌊ b/a ⌋ +1 , b ] . By Algorithm 1, we have that the probability of assigning y i,l to [ a, ⌊ b/ 2 ⌋ ] is

By the performance guarantee of the KDE algorithm, we have that g [ a,b ] ( x i ) ∈ (1 ± ϵ ) deg [ a,b ] ( v i ) , where we define

This gives

Next, notice that we can write for a sequence of intervals [ a 1 , b 1 ] ⊂ [ a 2 , b 2 ] ⊂ . . . ⊂ [1 , n ] that

where each term corresponds to one level of recursion of Algorithm 1 and there are at most ⌈ log 2 ( n ) ⌉ terms. Then, by (3) and the fact that the denominator and numerator of adjacent terms cancel out, we have

since deg [ j,j ] ( v i ) = k ( x i , x j ) and deg [1 ,n ] ( v i ) = deg K ( v i ) .

For the lower bound, we have that

where the final inequality follows by the condition of ϵ that ϵ ≤ 1 / (6 log 2 ( n )) .

For the upper bound, we similarly have

where the first inequality follows since ϵ < 1 / (6 log 2 ( n )) .

The next lemma shows that the probability of sampling each edge { v i , v j } is approximately equal to the 'ideal' sampling probability p i ( j ) .

̸

Lemma 2. For every i and j = i , we have

Proof. Let Y = { y i, 1 , . . . , y i,L } be the neighbours of x i sampled by Algorithm 2, where L = 6 C log( n ) /λ k +1 . Then,

The proof proceeds by case distinction.

Case 1: p i ( j ) ≤ 9 / 10 . In this case, we have

Case 2: p i ( j ) > 9 / 10 . In this case, we have

which completes the proof on the lower bound of ˜ p i ( j ) .

For the upper bound, we have

from which the statement follows.

An immediate corollary of Lemma 2 is as follows.

Corollary 1. For all different i, j ∈ [ n ] , it holds that

and

Proof. For the upper bound of the first statement, we can assume that p i ( j ) ≤ 1 / 12 and p j ( i ) ≤ 1 / 12 , since otherwise we have ˜ p ( i, j ) ≤ 1 ≤ 12 · p ( i, j ) and the statement holds trivially. Then, by Lemma 2, we have

and

For the second statement, notice that

since g [1 ,n ] ( x i ) is a (1 ± ε ) approximation of deg K ( v i ) and ε ≤ 1 / 6 . Similarly,

Then, for the upper bound of the second statement, we can assume that p i ( j ) ≤ 5 / 36 and p j ( i ) ≤ 5 / 36 , since otherwise ̂ p ( i, j ) ≤ 1 ≤ (36 / 5) · ˜ p ( i, j ) and the statement holds trivially. This implies that

which completes the proof.

and

We are now ready to prove Theorem 2. It is important to note that, although some of the proofs below are parallel to that of [27], our analysis needs to carefully take into account the approximation ratios introduced by the approximate KDE algorithm, which makes our analysis more involved. The following concentration inequalities will be used in our proof.

Lemma 3 (Bernstein's Inequality [8]) . Let X 1 , . . . , X n be independent random variables such that | X i | ≤ M for any i ∈ { 1 , . . . , n } . Let X = ∑ n i =1 X i , and R = ∑ n i =1 E [ X 2 i ] . Then, it holds that

Lemma 4 (Matrix Chernoff Bound [28]) . Consider a finite sequence { X i } of independent, random, PSD matrices of dimension d that satisfy ∥ X i ∥ ≤ R . Let µ min ≜ λ min ( E [ ∑ i X i ]) and µ max ≜ λ max ( E [ ∑ i X i ]) . Then, it holds that

for δ ∈ [0 , 1] , and

Proof of Theorem 2. We first show that the degrees of all the vertices in the similarity graph K are preserved with high probability in the sparsifier G . For any vertex v i , let y i, 1 , . . . , y i,L be the indices of the neighbours of v i sampled by Algorithm 2.

̸

For every pair of indices i = j , and for every 1 ≤ l ≤ L , we define the random variable Z i,j,l to be the weight of the sampled edge if j is the neighbour sampled from i at iteration l , and 0 otherwise:

Then, fixing an arbitrary vertex x i , we can write

̸

̸

We have

̸

By Lemmas 1 and 2 and Corollary 1, we have

̸

̸

̸

where the last inequality follows by the fact that ̂ p ( i, j ) ≤ (36 / 5) · p ( i, j ) ≤ (36 / 5) · ( p i ( j ) + p j ( i )) . Similarly, we have

̸

̸

where the inequality follows by the fact that

In order to prove a concentration bound on this degree estimate, we would like to apply the Bernstein inequality for which we need to bound

̸

̸

where the third inequality follows by the fact that

Then, by applying Bernstein's inequality we have for any constant C 2 that

where we use the fact that

̸

̸

̸

̸

̸

Therefore, by taking C to be sufficiently large and by the union bound, it holds with high probability that the degree of all the nodes in G are preserved up to a constant factor. For the remainder of the

proof, we assume that this is the case. Note in particular that this implies vol G ( S ) = Θ(vol K ( S )) for any subset S ⊆ V .

Next, we prove it holds for G that ϕ G ( S i ) = O ( k · ϕ K ( S i )) for any 1 ≤ i ≤ k , where S 1 , . . . , S k form an optimal clustering in K .

By the definition of Z i,j,l , it holds for any 1 ≤ i ≤ k that

where the last line follows by Corollary 1. By Markov's inequality and the union bound, with constant probability it holds for all i = 1 , . . . , k that

Therefore, it holds with constant probability that

Finally, we prove that λ k +1 ( N G ) = Ω( λ k +1 ( N K )) . Let N K be the projection of N K on its top n -k eigenspaces, and notice that N K can be written

where f 1 , . . . , f n are the eigenvectors of N K . Let N -1 / 2 K be the square root of the pseudoinverse of N K .

We prove that the top n -k eigenvalues of N K are preserved, which implies that λ k +1 ( N G ) = Θ( λ k +1 ( N K )) . To prove this, for each data point x i and sample 1 ≤ l ≤ L , we define a random matrix X i,l ∈ R n × n by

where j = y i,l , b e ≜ χ v i -χ v j is the edge indicator vector, and x v i ∈ R n is defined by

Notice that

where

is the Laplacian matrix of G normalised with respect to the degrees of the nodes in K . We prove that, with high probability, the top n -k eigenvectors of N ′ G and N K are approximately the same. Then, we show the same for N G and N ′ G which implies that λ k +1 ( N G ) = Ω( λ k +1 ( N K )) .

We begin by looking at the first moment of ∑ n i =1 ∑ L l =1 X i,l , and have that

̸

̸

where the last inequality follows by the fact that

̸

̸

Similarly,

where the last inequality follows by the fact that

Additionally, for any

Now, we apply the matrix Chernoff bound and have that

for some constant c . The other side of the matrix Chernoff bound gives us that

Combining these, with probability 1 -O (1 /n c ) it holds for any non-zero x ∈ R n in the space spanned by f k +1 , . . . , f n that

By setting y = N -1 / 2 K x , we can rewrite this as

Since dim(span { f k +1 , . . . , f n } ) = n -k , we have shown that there exist n -k orthogonal vectors whose Rayleigh quotient with respect to N ′ G is Ω( λ k +1 ( N K )) . By the Courant-Fischer Theorem, we have λ k +1 ( N ′ G ) = Ω( λ k +1 ( N K )) .

It only remains to show that λ k +1 ( N G ) = Ω( λ k +1 ( N ′ G )) , which implies that λ k +1 ( N G ) = Ω( λ k +1 ( N K )) . By the definition of N ′ G , we have that N G = D -1 / 2 G D 1 / 2 K N ′ G D 1 / 2 K D -1 / 2 G . Therefore, for any x ∈ R n and y = D 1 / 2 K D -1 / 2 G x , it holds that

where the final guarantee follows from the fact that the degrees in G are preserved up to a constant factor. The conclusion of the theorem follows by the Courant-Fischer Theorem.

Finally, we bound the running time of Algorithm 2 which is dominated by the recursive calls to Algorithm 1. We note that, although the number of nodes doubles at each level of the recursion tree (visualised in Figure 4), the total number of samples S and data points X remain constant for each level of the tree. Then, since the running time of the KDE algorithm is superadditive, the total running time of the KDE algorithms at level i of the tree is

Since there are O (log( n )) levels of the tree, the total running time of Algorithm 1 is ˜ O ( T KDE ( | S | , | X | , ϵ )) . This completes the proof.

B Additional Experimental Results

In this section, we include in Figures 6 and 7 some additional examples of the performance of the six spectral clustering algorithms on the BSDS image segmentation dataset. Due to its quadratic memory requirement, the SKLEARN GK algorithm cannot be used on the full-resolution image. Therefore, we present its results on each image downsampled to 20,000 pixels. For every other algorithm, we show the results on the full-resolution image. In every case, we find that our algorithm is able to identify more refined detail of the image when compared with the alternative algorithms.

(a) Original Image

(b) SKLEARN GK

(c) OUR ALGORITHM

(d) SKLEARN

k

-NN

(e) FAISS EXACT

(f) FAISS HNSW

(g) FAISS IVF

(h) Original Image

(i) SKLEARN GK

(j) OUR ALGORITHM

(k) SKLEARN

k

-NN

(l) FAISS EXACT

(m) FAISS HNSW

(n) FAISS IVF

(o) Original Image

(p) SKLEARN GK

(q) OUR ALGORITHM

(r) SKLEARN

k

-NN

(s) FAISS EXACT

(t) FAISS HNSW

(u) FAISS IVF

(a) Original Image

(b) SKLEARN GK

(c) OUR ALGORITHM

(d) SKLEARN

k

-NN

(e) FAISS EXACT

(f) FAISS HNSW

(g) FAISS IVF

(h) Original Image

(i) SKLEARN GK

(j) OUR ALGORITHM

(k) SKLEARN

k

-NN

(l) FAISS EXACT

(m) FAISS HNSW

(n) FAISS IVF

3001911

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a set of n points and a pairwise kernel k(x,y), the fully connected similarity graph K is the weighted graph that has an edge between every pair x,y with weight k(x,y). The paper presents an algorithm for computing a sparse graph G that approximates K in a certain sense that is useful for spectral clustering. The algorithm is based on black-box calls to an efficient kernel density estimation (KDE) algorithm. For some kernels such black-boxes are available and have been the focus of much work recently, and they lead to efficient graph construction in this paper. The result is given in the form of a formal theorem and empirical evaluation.\n\nSTRENGTHS:\nThe paper is generally clearly written and contains nice and convincing theoretical and empirical findings that contribute to the literature on the topic and could be useful in practice. It is a nice application of the recent progress on efficient KDE to graph clustering. Even though it may not be as novel and general and advertised (see below), I still find it above the bar for acceptance.\n\nWEAKNESSES:\n(1) Novelty is somewhat limited, and seems overstated. The intro (lines 61-62) claims \"a novel connection between the KDE and the fast construction of similarity graphs\". This connection is quite well-known already and has been the focus of some recent works, including [1], [6] and \"Spectral Sparsification of Metrics and Kernels\" (Quanrud, SODA 2021). Anyway this connection between efficient KDE and kernel similarity graphs is very natural and straightforward, and it is unsurprising that all these works appeared shortly after the recent progress on efficient KDE. These works instantiate the connection in possibly different ways, but the general idea is similar. The novelty in this work seems to me restricted to the specific technical application to the SZ clustering framework, and not so much a conceptual point about kernels and graphs. \n\n(2) A more specific concern about novelty is that the main sampling method at the heart of the algorithm (lines 195-204, Algorithm 1) seems to have appeared already in [6] (see their \"sample random neighbor\" primitive). The work is cited only for a certain conditional hardness result, while its very similar algorithmic ideas are not mentioned. How do these techniques relate to each other?\n\n(3) The generality of the result itself is also overstated, particularly in the abstract that the result is \"applicable to arbitrary kernels\". I suppose this alludes to the fact that the algorithm uses KDE as a black-box, this is rather misleading, since there relatively few and specialized instances of known black-boxes of the type needed for this algorithm, and for a relatively limited set of kernels. For arbitrary kernels the black-box does not exist and the result of the paper is not applicable.\n\nThis affects the writing in the intro. Lines 35-40 formulate the main question as \"constructing a sparse graph that preserves the cluster structure\", focusing on sparsity and eschewing running time, possibly in order to circumvent said limitations on generality. However, this renders the main question moot, since trivially the full graph can be constructed and then spectrally sparsified. It would probably have been preferable to promptly discuss running times with their inevitable limitations.\n\nQUESTIONS:\n1. Please clarify connection to sampling in [6] (see above)\n\n2. The paragraph about LSH in lines 90-94 (related work) sounds odd. It says it's unclear how to use LSH for approximating geometric similarity graphs. However, much of the work you cite for efficient KDE builds directly on LSH (and consequently, so do some instantiations of your algorithm, as well as similar graph algorithms from prior work mentioned earlier).\n\nLIMITATIONS:\nNo major concerns, though some limitations on the generality of the results may not be sufficiently discussed, as mentioned above in weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an algorithm framework for creating a (weighted) sparse similarity graph, which is an important object for a variety of ML problems. The algorithm samples pairs of points to include in the graph using a sampler based on kernel density estimation (KDE). The idea is to recursively split the data into subsets, compute the KDE on each subset, and select a subset proportional to the density. This process is done for every point in the dataset (node in the graph), resulting in a collection of edges in the sparse graph.\n\nA nice feature of this method is that it provably preserves the spectral clustering properties of the graph. To my knowledge, this is the only sub-quadratic (in space / time) algorithm that does this. The method is also agnostic to the choice of KDE solver, leading to interesting practical algorithms. When paired with a classical KDE method (the Fast Gauss Transform), the method outperforms a number of reasonable baselines for graph construction + spectral clustering in low dimensions.\n\nSTRENGTHS:\n**S1: Novel and interesting reduction,** from KDE to similarity graph construction. The sampling mechanism (Algorithm 1) is particularly nice, and it likely has further applications to other settings where we want to sample proportional to kernel sums. The proposed framework also benefits from and motivates the growing body of work on the KDE problem. This leads to exciting possibilities for practical algorithms, depending on which KDE method is used in the framework.\n\n**S2: Good theoretical results.** It is great to see rigorous guarantees on the quality of the subsequent partitioning of the similarity graph as well as the time / space complexity. Many studies only perform the latter, as it is difficult to prove this kind of result.\n\n**S3: Well-engineered implementation,** with reproducible experiments. I was able to fully reproduce all of the experiments in this paper. I think the implementation will be of use to the community.\n\nWEAKNESSES:\n**W1: Notation.** The notation in Section 2 is understandable but would benefit from some polishing. For example, $w_G(u, v)$ is the edge weight function over node inputs and is defined only for those $(u,v)$ that are in the graph. But there is a $w_G(S, V\\setminus S)$ (with a different-script $G$ and set-of-node inputs), which is the sum of outbound edge weights from within $S$ to other parts of the graph. There is also a $w(u,v)$ used in the definition of the node degree, which isn't defined anywhere. A graph is defined as $G = (V, E)$ (with a separately-specified weight function) but the similarity graph is given as $K = (V, E, \\mathrm{weights})$. The kernel density sum KDE is introduced with a subscript $g_{[a,b]}$, but the analysis doesn't use this notation and later introduces $g_X$ as the KDE sum over a set of points $X$. This is all a bit confusing.\n\nOne possible fix is to define everything in terms of a weighted graph $G = (V, E, w)$. Here, $w$ is the weighting function from $(u,v)\\to \\mathbb{R}^{+}$ where $w(u,v) = $ edge weight if $(u,v) \\in G$ and 0 otherwise. This would give a much simpler definition for the adjacency matrix ($A_{ij} = w(v_i, v_j)$), the degree ($d(v) = \\sum_{u \\in V} w(v, u)$), and the outbound edge weight sum, which can be written as a unary function $w_o(S) = \\sum_{u \\in S, v \\not \\in S} w(u,v)$ to further reduce confusion with $w(u,v)$. Also meanings of $\\rho_K$ and $\\lambda$ should also be introduced before they are used in the main theorem, even if the full definition is deferred until later.\n\n**W2: Literature positioning / context.** The flexibility of the framework is a major strength of the paper, so it would be nice to highlight the (growing) number of KDE algorithms. For example:\n- Hashing-Based Estimators (HBE): Uses LSH tables to do KDE, where we only compute the KDE over the points that collide in the LSH table. The original paper was in [[FOCS 2017]](https://arxiv.org/abs/1808.10530), with a practical version in [[ICML 2019]](http://proceedings.mlr.press/v97/siminelakis19a) and some incremental progress in [[NeurIPS 2019]](https://openreview.net/forum?id=H1xEABrgIH). (only provably works for some kernels, fast in high dimensions, good in practice).\n- Interpolation KDE: [[AISTAT 2021]](http://proceedings.mlr.press/v130/turner21a/turner21a.pdf) performs KDE via polynomial interpolation (for arbitrary smooth kernels).\n- RACE: [[WWW 2020]](https://dl.acm.org/doi/fullHtml/10.1145/3366423.3380244) performs fast, online KDE, using a few MB of space (only for a limited set of kernels, but extremely space / time efficient). \n- Discrepancy coresets: [[COLT 2019]](https://proceedings.mlr.press/v99/karnin19a.html) performs KDE using online coresets (a re-weighted sub-sample of the dataset). In practice, this requires slower preprocessing than other methods but is more general / can be more accurate.\n\nEach of these methods will present unique tradeoffs when incorporated into this framework. While a full comparison of various KDE subroutines is probably out-of-scope, it would be nice to discuss some of these tradeoffs / make recommendations. A few similarity graph construction methods have also been developed since the review paper by Luxburg in 2007, which should be mentioned:\n\n- NN Descent: [[WWW 2011]](https://dl.acm.org/doi/abs/10.1145/1963405.1963487) This is an iterative technique that progressively refines a random guess for the initial k-NN graph.\n- FLASH: [[SIGMOD 2018]](https://dl.acm.org/doi/abs/10.1145/3183713.3196925) This is an LSH-based method that constructs a similarity graph (i.e. finds all pairs $(u,v)$ such that $w(u,v)$ exceeds a threshold). This is much faster than other LSH methods because it uses clever counting tricks to avoid doing any explicit distance / kernel calculations.\n\nFinally, the method in the paper, \"Learning space partitions for nearest neighbor search\" [[ICLR 2020]](https://openreview.net/forum?id=rkenmREFDr), might be relevant. This paper provides a framework for approximate near neighbor search that starts with a similarity graph, then partitions the data via a balanced min-cut partitioning of the graph. These partitions are then used to do a partition-based (FAISS IVF-style) similarity search, with provable guarantees (first theory for this type of algorithm). From a practical perspective, the hard part of this framework is to get the initial similarity graph, so your algorithm might help.\n\n\n**W3: Experiments.** A couple of highly competitive baselines are not represented in the experimental evaluation, notably FLASH (which is $O(N \\mathrm{polylog}(N))$) and NN-Descent (which scales about $O(N^{1.15})$, based on empirical results). For example, the C++ implementation of FLASH can construct a similarity graph for the webspam dataset (N = 350k) in under 10 seconds and for the friendster dataset (N = 65 million) in 1578 seconds. This 3-10x faster than the result of extrapolating Figure 5, and these datasets are of much higher dimension than two moons.\n\nThe datasets in the experiments are also all low-dimensional: even the BSDS task is only 5-dimensional. Many clustering applications (e.g. those that handle text embeddings) have inputs in the hundreds of dimensions. It would be nice to understand how this method performs in the higher-dimensional setting.\n\nQUESTIONS:\n1. Does T_KDE include preprocessing + query time? Most of the recent work in fast KDE requires a preprocessing algorithm (typically $O(N)$) followed by a query algorithm (typically much faster than $O(N)$). It looks like this is the case based on the T_KDE reported for FGT and the LSH method, but it wasn't clear from the definition. This is important because Algorithm 1 calls the KDE subroutine on different subsets of X.\n\n2. What is the state of the art result (using e.g. neural network models) on the BSDS image segmentation dataset? Is it close to the result for k-NN / similarity graph algorithms or is it much better?\n\n**Minor items:**\nTo get this to compile under MacOS arm64 (M1 chips), I did the following.\n1. Pass an up-to-date clang compiler to cmake (via -D CMAKE_CXX_COMPILER).\n2. Add the flag \"-I/usr/local/include\" to the compile_args in setup.py.\n3. Break the function declarations in stag_lib/KMeansRex/mersenneTwister2002.c into a header.\n4. pip install stag (stag isn't in requirements.txt, for some reason)\n\nNitpick on Line 92: \"Despite extensive research on LSH-based algorithms, it remains unclear whether such techniques can be employed to approximately construct a similarity graph in nearly-linear time.\"\nThis has been done at scale (albeit without theoretical guarantees on the quality of the similarity graph), see section 4 of [the FLASH paper](https://arxiv.org/pdf/1709.01190.pdf).\n\nLIMITATIONS:\nI do not foresee any negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed an efficient approximation of the similarity graph using the Kernel Density Estimation (KDE) method as a black box. The proposed framework can preserve the clustering structure and can be run in nearly linear time. Lastly, experiments on synthetic and image segmentation datasets are provided to support these claims.\n\nSTRENGTHS:\n1. [Originality] The author proposed a novel algorithm by combining the ideas of subset/subspace dividing (somewhat related to LHS) with the help of probability measure estimated from KDE.\n1. [Clarity] An excellent written paper that provides clear insights into motivations and formulation. Great overview of how the algorithm works Figures 3-4 and Algorithms 1-2. The fast runtime is achieved by the $O(\\log(n))$ recursive call to the KDE as well as the superaddictive property of the $T_{KDE}$. \n1. [Quality] The authors compared their proposed algorithm with publicly available nearest neighbor implementations (FAISS, sklearn) and show drastic speedup in the large sample size regime. \n1. [Significance] I believe this paper makes a significant contribution in the unsupervised learning domains. The graph sparsification/Laplacian construction usually is done in a two-way approach, i.e., calculate pairwise distances and compute the similarity graph (with an additional sparsification step). The proposed method can do the aforementioned steps in a single step, thus drastically reducing the run time.\n\nWEAKNESSES:\n1. The authors have shown theoretically that the proposed algorithm is a {\\em cluster-preserving sparsifier}. This claim can be further supported by showing a comparison between the eigenvalues (or eigengaps) of the true, graph sparsifiers (e.g., from [13, 14, 19]), and the proposed method. \n1. There is another kind of similarity graph, called the $\\varepsilon$-radius graph. The graph $G(V, E)$ is constructed with $E = { (x_i, x_j) \\in V^2 :  \\|x_i - x_j\\| \\leq \\varepsilon}$. I think it will be informative to discuss this in the Introduction (L27-34).\n1. It is quite confusing to see Theorem 1 without the full definition, or even a high-level description, of what $\\rho$, $\\lambda$, $N_K$ are in Section 1. Readers can somewhat guess what they are based on notational convention, but to improve readability, I would suggest to either include the full definition or (even better) simplify it to make it less rigorous when it is first introduced in Section 1 (then provided a detailed version later in Section 4 or Appendix if no space).\n1. For the runtime of the FGT (L148), it might be more informative to write out all the terms inside the log rather than writing $\\widetilde O(m+n)$, because the error term $\\epsilon$ is hidden in the log.\n1. It will be interesting to see experiments on higher dimensional (real or synthetic) datasets and compare the runtime correspondingly. The experiments are run on (relatively) low-dimensional and straightforward examples, e.g., synthetic datasets (d=2) and image data (d=5). Some potential high-dimensional real datasets to consider e.g., [A,B].\n\n---\n[A] Mahmoud, Eman, Ali Takey, and Amin Shoukry. “Spectral Clustering for Optical Confirmation and Redshift Estimation of X-Ray Selected Galaxy Cluster Candidates in the SDSS Stripe 82.” Astronomy and Computing 16 (2016): 174–84.\n\n[B] Chmiela, Stefan, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof T. Schütt, and Klaus-Robert Müller. “Machine Learning of Accurate Energy-Conserving Molecular Force Fields.” Science Advances 3, no. 5 (2017): e1603015.\n\nQUESTIONS:\n1. Related to Weakness #2, in practice, people usually use exponentially decaying kernel $k$ for the fully connected graph. You can approximate the fully connected graph with $\\varepsilon$-radius graph by choosing the radius $\\varepsilon$ and the bandwidth $\\sigma$ appropriately, e.g., if using the exponential kernel as per L132, you can choose $\\varepsilon = 3\\cdot \\sigma$; that way the far enough pairs will have small kernel values, i.e., $=\\exp(-9) \\sim 10^{-4}$, thus is negligible. I am curious if your method can be extended to the $\\varepsilon$-radius graph under this scenario as well?\n1. I am curious how difficult it is to extend this framework to the scenario of higher-order (neighborhood) simplicial complex and/or k-Laplacian. There are several works trying to use this information in data analysis [A-D], as well as algorithms [E] to sparsify the higher-order simplicial complexes with higher-order Cheerger constant. The reason I am asking is because the Higher-order k-Laplacian has dependency on $O(n_k)$ where $n_k$ is the cardinality of the k-complex (e.g., size of edges is $n_1 = |E|$), so if we can successfully sparsify the simplicial complex and/or k-Laplacian it can reduce the runtime drastically for those studies. \n1. How can this framework be extended to the manifold learning or diffusion map setting? In this scenario, we care not only about the approximation of the null space but also the first few eigne-values. Can we have similar guarantees for these scenarios?\n1. [Minor language usage] L21-23 (“Thanks to its out-performance over traditional clustering algorithms like k-means, this approach has…”) might be able to be improved by something like “Due to its superior performance compared to conventional clustering algorithms such as k-means, this approach has….”\n1. [Minor notational consistency] For the equation between L194-195, you might be able to improve the clarity by changing the $z \\in X_1$ to $x_j \\in X_1$ and the $x_j$ in summation to another variable. This is because you used $x_i$ and $x_j$ in the discussion above (L192). Similar to L13 in Algorithm 1, you might want to reconsider the use of $y_i$ there to make the notation more consistent.\n\n---\n[A] Dey, Tamal K., Jian Sun, and Yusu Wang. “Approximating Loops in a Shortest Homology Basis from Point Data.” In Proceedings of the Twenty-Sixth Annual Symposium on Computational Geometry, 166–75, 2010.\n\n[B] Chazal, Frédéric, Brittany Fasy, Fabrizio Lecci, Bertrand Michel, Alessandro Rinaldo, Alessandro Rinaldo, and Larry Wasserman. “Robust Topological Inference: Distance to a Measure and Kernel Distance.” The Journal of Machine Learning Research 18, no. 1 (2017): 5845–84.\n\n[C] Chen, Yu-Chia, and Marina Meila. “The Decomposition of the Higher-Order Homology Embedding Constructed from the k-Laplacian.” Advances in Neural Information Processing Systems 34 (2021).\n\n[D] Keros, Alexandros D., Vidit Nanda, and Kartic Subr. “Dist2cycle: A Simplicial Neural Network for Homology Localization.” In Proceedings of the AAAI Conference on Artificial Intelligence, 36:7133–42, 2022.\n\n[E] Osting, Braxton, Sourabh Palande, and Bei Wang. “Spectral Sparsification of Simplicial Complexes for Clustering and Label Propagation.” ArXiv:1708.08436 [Cs], February 1, 2019. http://arxiv.org/abs/1708.08436.\n\nLIMITATIONS:\nI believe that the authors have covered all the limitations of the algorithms (e.g., it is only cluster-preserving not spectral sparsification, runtime will depend on the implementation of $T_KDE$, etc.) in the current version, but adding it and summarize it in a specific section will be beneficial.\n\nNegative social impact is not applicable in this case, as this work primarily constitutes a theoretical contribution.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Authors proposes a fast approximation method for constructing sparse similarity graphs from data for spectral clustering. They leverage Sun and Zanetti (2019)'s sampling-based approach to find "cluster-preserving sparsifier" for similarity graphs. This is combined with black box KDE and binary search algorithms to generate a fast & memory efficient approximation of SZ's algorithm. The experimental results are shown to be fast and accurate compared to Scikit-learn and FAISS's implementation.

STRENGTHS:
This paper combines existing ideas (e.g. SZ algorithm, KDE algorithm, binary search) to create a novel approach to kernel graph construction that has potential to be used by many researchers in the community. The experimental results including runtime comparison and image segmentation (e.g. Figure 2) are quite impressive. It was also well written and clearly presented.

WEAKNESSES:
Please see questions below.

QUESTIONS:
1. Given how it was used as a major motivation in this paper, I expected to see a comparison with the output of SZ algorithm in the experiments section. Have the authors compared SZ and proposed method experimentally for clustering?

2. What do the calculated probabilities and sampled sparse graphs look like? Can the authors show any examples on a small graph or using a synthetic dataset? And how does the graph actually compare to that from SZ?

3. It's missing a discussion on limitations of this work.

LIMITATIONS:
Not really.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07