Title: Fast and Simple Spectral Clustering in Theory and Practice
Venue: NeurIPS 2023
Accepted: yes
Keywords: spectral clustering

power method

spectral graph theory

graph algorithms
Abstract: Spectral clustering is a popular and effective algorithm designed to find $k$ clusters in a graph $G$.
In the classical spectral clustering algorithm, the vertices of $G$ are embedded into $\mathbb{R}^k$ using $k$ eigenvectors of the graph Laplacian matrix.
However, computing this embedding is computationally expensive and dominates the running time of the algorithm.
In this paper, we present a simple spectral clustering algorithm based on a vertex embedding with $O(\log(k))$ vectors computed by the power method.
The vertex embedding is computed in nearly-linear time with respect to the size of the graph, and
the algorithm provably recovers the ground truth clusters under natural assumptions on the input graph.
We evaluate the new algorithm on several synthetic and real-world datasets, finding that it is significantly faster than alternative clustering algorithms,
while producing results with approximately the same clustering accuracy.

Fast and Simple Spectral Clustering in Theory and Practice

Abstract

Spectral clustering is a popular and effective algorithm designed to find k clusters in a graph G . In the classical spectral clustering algorithm, the vertices of G are embedded into R k using k eigenvectors of the graph Laplacian matrix. However, computing this embedding is computationally expensive and dominates the running time of the algorithm. In this paper, we present a simple spectral clustering algorithm based on a vertex embedding with O (log( k )) vectors computed by the power method. The vertex embedding is computed in nearly-linear time with respect to the size of the graph, and the algorithm provably recovers the ground truth clusters under natural assumptions on the input graph. We evaluate the new algorithm on several synthetic and real-world datasets, finding that it is significantly faster than alternative clustering algorithms, while producing results with approximately the same clustering accuracy.

1 Introduction

Graph clustering is an important problem with numerous applications in machine learning, data science, and theoretical computer science. Spectral clustering is a popular graph clustering algorithm with strong theoretical guarantees and excellent empirical performance. Given a graph G with n vertices and k clusters, the classical spectral clustering algorithm consists of the following two high-level steps [26, 33].

Embed the vertices of G into R k according to k eigenvectors of the graph Laplacian matrix.

Apply a k -means clustering algorithm to partition the vertices into k clusters.

Recent work shows that if the graph has a well-defined cluster structure, then the clusters are wellseparated in the spectral embedding and the k -means algorithm will return clusters which are close to optimal [21, 28].

The main downside of this algorithm is the high computational cost of computing k eigenvectors of the graph Laplacian matrix. In this paper, we address this computational bottleneck and propose a new fast spectral clustering algorithm which avoids the need to compute eigenvectors while maintaining excellent theoretical guarantees. Moreover, our proposed algorithm is simple, fast, and effective in practice.

1.1 Sketch of Our Approach

In this section, we introduce the high-level idea of this paper, which is also illustrated in Figure 1. We begin by considering a recent result of Makarychev et al. [23] who show that a random projection of data into O (log( k )) dimensions preserves the k -means objective function for all partitions of the

Graph

Compute

k

Random

k

-means

k

-means

Power

Spectral

Embedding

Low-dim.

Embedding

Projection

Eigenvectors

Method

Clusters

data, with high probability. Since the final step of spectral clustering is to apply k -means, we might consider the following alternative spectral clustering algorithm which will produce roughly the same output as the classical algorithm.

Embed the vertices of G into R k according to k eigenvectors of the graph Laplacian matrix.

Randomly project the embedded points into O (log( k )) dimensions.

Apply a k -means clustering algorithm to partition the vertices into k clusters.

Of course, this does not avoid the expensive eigenvector computation and so it is not immediately clear that this random projection can be used to improve the spectral clustering algorithm.

The key technical element of our paper is a proof that we can efficiently approximate a random projection of the spectral embedding without computing the spectral embedding itself. For this, we use the power method, which is a well-known technique in numerical linear algebra for approximating the dominant eigenvalue of a matrix [11]. We propose the following simple algorithm (formally described in Algorithm 2).

Embed the vertices of G into O (log( k )) dimensions using O (log( k )) random vectors computed with the power method.

Apply a k -means clustering algorithm to partition the vertices into k clusters.

We prove that the projection obtained using the power method is approximately equivalent to a random projection of the spectral embedding. Then, by carefully applying the techniques developed by Makarychev et al. [23] and Macgregor and Sun [21], we obtain a theoretical bound on the number of vertices misclassified by our proposed algorithm. Moreover, the time complexity of step 1 is nearly linear in the size of the graph, and the algorithm is fast in practice. The formal theoretical guarantee is given in Theorem 3.1.

1.2 Related Work

This paper is closely related to a sequence of recent results which prove an upper bound on the number of vertices misclassified by the classical spectral clustering algorithm [15, 21, 24, 28]. While we will directly compare our result with these in a later section, our proposed algorithm has a much faster running time than the classical spectral clustering algorithm, and has similar theoretical guarantees.

Boutsidis et al. [5] also study spectral clustering using the power method. Our result improves on theirs in two respects. Firstly, our algorithm is faster since we compute O (log( k )) vectors rather than k vectors and their algorithm includes an additional singular value decomposition step. Secondly, we give a theoretical upper bound on the total number of vertices misclassified by our algorithm.

Makarychev et al. [23] generalise the well-known Johnson-Lindenstrauss lemma [13] to show that random projections of data into O (log( k )) dimensions preserves the k -means objective, and we make use of their result in our analysis.

Macgregor and Sun [21] show that for graphs with certain structures of clusters, spectral clustering with fewer than k eigenvectors performs better than using k eigenvectors. In this paper, we present the first proof that embedding with O (log( k )) vectors is sufficient to find k clusters with spectral clustering.

Other proposed methods for fast spectral clustering include the Nystrom method [6] and using a 'pre-clustering' step to reduce the number of data points to be clustered [35]. These methods lack rigorous theoretical guarantees on the accuracy of the returned clustering. Moreover, our proposed algorithm is significantly simpler to implement that the alternative methods.

2 Preliminaries

̸

Throughout the paper, we use big-O notation to hide constants. For example, we use l = O ( n ) to mean that there exists a universal constant c such that l ≤ cn . We sometimes use ˜ O ( n ) in place of O ( n log c ( n )) for some constant c . Following [21], we say that a partition { S i } k i =1 of V is almost-balanced if vol( S i ) = Θ(vol( V ) /k ) for all i ∈ [ k ] .

Let G = ( V, E, w ) be a graph with n = | V | and m = | E | . For any v ∈ V , the degree of v is given by d ( v ) = ∑ u = v w ( u, v ) . For any S ⊂ V , the volume of S is given by vol( S ) = ∑ u ∈ S d ( u ) . The Laplacian matrix of G is L = D -A where D is the diagonal matrix with D ( i, i ) = d ( i ) and A is the adjacency matrix of G . The normalised Laplacian is given by N = D -1 2 LD -1 2 . We always use λ 1 ≤ λ 2 ≤ . . . ≤ λ n to be the eigenvalues of N and the corresponding eigenvectors are f 1 , . . . , f n . For any graph, it holds that λ 1 = 0 and λ n ≤ 2 [7]. We will also use the signless Laplacian matrix 1 M = I -(1 / 2) N and will let γ 1 ≥ . . . ≥ γ n be the eigenvalues of M . Notice that by the definition of M , we have γ i = 1 -(1 / 2) λ i and the eigenvectors of M are also f 1 , . . . , f n . For an integer k , we let [ k ] = { 1 , . . . k } be the set of all positive integers less than or equal to k . Given two sets A and B , their symmetric difference is given by A △ B = ( A \ B ) ∪ ( B \ A ) . We call { S i } k i =1 a k -way partition of V if S i ∩ S j = ∅ for i = j and ⋃ k i =1 S i = V .

̸

2.1 Conductance and the Graph Spectrum

Given a graph G = ( V, E ) , and a cluster S ⊂ V , the conductance of S is given by where w ( S, S ) = ∑ u ∈ S ∑ v ∈ S w ( u, v ) . Then, the k -way expansion of G is defined to be

Notice that ρ ( k ) is small if and only if G can be partitioned into k clusters of low conductance. There is a close connection between the k -way expansion of G and the eigenvalues of the graph Laplacian matrix, as shown in the following higher-order Cheeger inequuality.

Lemma 2.1 (Higher-Order Cheeger Inequality, [19]) . For a graph G , let λ 1 ≤ . . . ≤ λ n be the eigenvalues of the normalised Laplacian matrix. Then, for any k ,

From this lemma, we can see that an upper bound on ρ ( k ) and a lower bound on λ k +1 are sufficient conditions to guarantee that G can be partitioned into k clusters of low conductance, and cannot be partitioned into k + 1 clusters. This condition is commonly used in the analysis of graph clustering [15, 21, 24, 28].

2.2 The k -means Objective

For any matrix B ∈ R n × d , the k -means cost of a k -way partition { A i } k i =1 of the data points is given by

where B ( u, :) is the u th row of B and µ i = (1 / | A i | ) ∑ u ∈ A i B ( u, :) is the mean of the points in A i . Although optimising the k -means objective is NP -hard, there is a polynomial-time constant factor approximation algorithm [14], and an approximation scheme which is polynomial in n and d with an exponential dependency on k [8, 16]. Lloyds algorithm [20], with k -means++ initialisation [3] is an O (log( n )) -approximation algorithm which is fast and effective in practice.

2.3 The Power Method

The power method is an algorithm which is most often used to approximate the dominant eigenvalue of a matrix [11, 25]. Given some matrix M ∈ R n × n , a vector x 0 ∈ R n , and a positive integer t , the power method computes the value of M t x 0 by repeated multiplication by M . The formal algorithm is given in Algorithm 1.

Algorithm 1: POWERMETHOD ( M ∈ R n × n , x 0 ∈ R n , t ∈ Z ≥ 0 ) 1 for i ∈ { 1 , . . . , t } do 2 x i = M x i -1 3 end 4 return x t

3 Algorithm Description and Analysis

In this section, we present our newly proposed algorithm and sketch the proof of our main result. Omitted proofs can be found in the Appendix. We first prove that if M has k eigenvalues γ 1 , . . . , γ k close to 1 , then the power method can be used to compute a random vector in the span of the eigenvectors corresponding to γ 1 , . . . , γ k .

We then apply this result to the signless Laplacian matrix of a graph to develop a fast spectral clustering algorithm and we bound the number of misclassified vertices when the algorithm is applied to a well-clustered graph.

3.1 Approximating a Random Vector with the Power Method

Suppose we are given some matrix M ∈ R n × n with eigenvalues 1 ≥ γ 1 ≥ . . . ≥ γ n ≥ 0 and corresponding eigenvectors f 1 , . . . , f n . Typically, the power method is used to approximate the dominant eigenvector, f 1 . In this section, we show that when γ k is sufficiently close to 1 , the power method can be used to compute a random vector in the space spanned by f 1 , . . . , f k .

Let x 0 ∈ R n be a random vector chosen according to the n -dimensional Gaussian distribution N( 0 , I ) . We can write x 0 as a linear combination of the eigenvectors:

where a i = ⟨ x 0 , f i ⟩ . Then, the vector x t computed by POWERMETHOD ( M , x 0 , t ) can be written as

Informally, if γ k ≥ 1 -c 1 and γ k +1 ≤ 1 -c 2 for sufficiently small c 1 and sufficiently large c 2 , then for a carefully chosen value of t we have γ t i ≈ 1 for i ≤ k and γ t i ≈ 0 for i ≥ k +1 . This implies that

and we observe that ( ∑ k i =1 f i f ⊺ i ) x 0 is a random vector distributed according to a k -dimensional Gaussian distribution in the space spanned by f 1 , . . . , f k . Wemake this intuition formal in Lemma 3.1 and specify the required conditions on γ k , γ k +1 and t .

Lemma 3.1. Let M ∈ R n × n be a matrix with eigenvalues 1 ≥ γ 1 ≥ . . . ≥ γ n ≥ 0 and corresponding eigenvectors f 1 , . . . f n . Let x 0 ∈ R n be drawn from the n -dimensional Gaussian distribution N ( 0 , I ) . Let x t = POWERMETHOD ( M , x 0 , t ) for t = Θ ( log( n/ϵ 2 k ) ) . If γ k ≥ 1 -O ( ϵ · log( n/ϵ 2 k ) -1 ) and γ k +1 ≤ 1 -Ω(1) , then with probability at least 1 -1 / (10 k ) , where P = ∑ k i =1 f i f ⊺ i is the projection onto the space spanned by the first k eigenvectors of M .

3.2 The Fast Spectral Clustering Algorithm

We now introduce the fast spectral clustering algorithm. The algorithm follows the pattern of the classical spectral clustering algorithm, with an important difference: rather than embedding the vertices according to k eigenvectors of the graph Laplacian, we embed the vertices with Θ ( log( k ) · ϵ -2 ) random vectors computed with the power method for the signless graph Laplacian M . Algorithm 2 formally specifies the algorithm, and the theoretical guarantees are given in Theorem 3.1.

Theorem 3.1. Let G be a graph with λ k +1 = Ω(1) and ρ ( k ) = O ( ϵ · log( n/ϵ ) -1 ) . Additionally, let { S i } k i =1 be the k -way partition corresponding to ρ ( k ) and suppose that { S i } k i =1 are almost balanced. Let { A i } k i =1 be the output of Algorithm 2. With probability at least 0 . 9 -ϵ , there exists a permutation σ : [ k ] - → [ k ] such that

Moreover, the running time of Algorithm 2 is

where m is the number of edges in G and T KM ( n, k, l ) is the running time of the k -means approximation algorithm on n points in l dimensions.

Remark 3.1. The assumptions on λ k +1 and ρ ( k ) in Theorem 3.1 imply that the graph G can be partitioned into exactly k clusters of low conductance. This is related to previous results which make an assumption on the ratio λ k +1 /ρ ( k ) [21, 28, 30]. Macgregor and Sun [21] prove a guarantee like Theorem 3.1 under the assumption that λ k +1 /ρ ( k ) = Ω(1) . We achieve a faster algorithm under a slightly stronger assumption.

Remark 3.2. The running time of Theorem 3.1 improves on previous spectral clustering algorithms. Boutsidis et al. [4] describe an algorithm with running time ˜ O ( m · k · ϵ -2 ) + O ( k 2 · n ) + T KM ( n, k, k ) . Moreover, their analysis does not provide any guarantee on the number of misclassified vertices.

Remark 3.3. The constants in the definition of l and t in Algorithm 2 are based on those in the analysis of Makarychev et al. [23] and this paper. In practice, we find that setting l = log( k ) and t = 10log( n/k ) works well.

Throughout the remainder of this section, we will sketch the proof of Theorem 3.1. We assume that G = ( V, E ) is a graph with k clusters { S i } k i =1 of almost balanced size, λ k +1 = Ω(1) , and ρ ( k ) = O ( ϵ · log( n/ϵ ) -1 ) .

In order to understand the behaviour of the k -means algorithm on the computed vectors, we will analyse the k -means cost of a given partition under three different embeddings of the vertices. Let f 1 , . . . , f k be the eigenvectors of M corresponding to the eigenvalues γ 1 , . . . , γ k and let y 1 , . . . , y l be the vectors computed in Algorithm 2. We will also consider the vectors z 1 , . . . , z l given by z i = P x i , where { x i } k i =1 are the random vectors sampled in Algorithm 2, and P = ∑ k i =1 f i f ⊺ i is the projection onto the space spanned by f 1 , . . . , f k . Notice that each z i is a random vector distributed according to the k -dimensional Gaussian distribution. Furthermore, let

We will consider the vertex embeddings given by D -1 / 2 F , D -1 / 2 Z and D -1 / 2 Y and show that the k -means objective for every k -way partition is approximately equal in each of them. We will use the following result shown by Makarychev et al. [23].

Lemma 3.2 ([23], Theorem 1.3) . Given data X ∈ R n × k , let Π ∈ R k × l be a random matrix with each column sampled from the k -dimensional Gaussian distribution N( 0 , I k ) and

Then, with probability at least 1 -ϵ , it holds for all partitions { A i } k i =1 of [ n ] that

Applying this lemma with X = D -1 2 F and Π = F ⊺ Z shows that the k -means cost is approximately equal in the embeddings given by D -1 2 F and D -1 2 Z , since FF ⊺ Z = Z and each of the entries of F ⊺ Z is distributed according to the Gaussian distribution N(0 , 1) . 2 By Lemma 3.1, we can also show that the k -means objective in D -1 2 Y is within an additive error of D -1 2 Z . This allows us to prove the following lemma.

Lemma 3.3. With probability at least 0 . 9 -ϵ , for any partitioning { A i } k i =1 of the vertex set V , we have and

To complete the proof of Theorem 3.1, we will make use of the following results proved by Macgregor and Sun [21].

Lemma 3.4 ([21], Lemma 4.1) . There exists a partition { A i } k i =1 of the vertex set V such that

Lemma 3.5 ([21], Theorem 2) . Given some partition of the vertices, { A i } k i =1 , such that

then there exists a permutation σ : [ k ] - → [ k ] such that

Proof of Theorem 3.1. By Lemma 3.4 and Lemma 3.3, with probability at least 0 . 9 -ϵ , there exists some partition { ̂ A i } k i =1 of the vertex set V G such that

Since we use a constant-factor approximation algorithm for k -means, the partition { A i } k i =1 returned by Algorithm 2 satisfies COST D -1 / 2 Y ( A 1 , . . . , A k ) = O ( ϵk ) . Then, by Lemma 3.5 and Lemma 3.3, for some permutation σ : [ k ] - → [ k ] , we have

To bound the running time, notice that the number of non-zero entries in M is 2 m , and the time complexity of matrix multiplication is proportional to the number of non-zero entries. Therefore, the running time of POWERMETHOD ( M , x 0 , t ) is ˜ O ( m ) . Since the loop in Algorithm 2 is executed Θ ( log( k ) · ϵ -2 ) times, the total running time of Algorithm 2 is ˜ O ( m · ϵ -2 ) + T KM ( n, k, l ) . ■

4 Experiments

In this section, we empirically study several variants of the spectral clustering algorithm. We compare the following algorithms:

k EIGENVECTORS: the classical spectral clustering algorithm which uses k eigenvectors of the graph Laplacian matrix to embed the vertices. This is the algorithm analysed in [21, 28].

log( k ) EIGENVECTORS: spectral clustering with log( k ) eigenvectors of the graph Laplacian to embed the vertices.

KASP: the fast spectral clustering algorithm proposed by Yan et al. [35]. The algorithm proceeds by first coarsening the data with k -means before applying spectral clustering.

PM k VECTORS (Power Method with k vectors): spectral clustering with k orthogonal vectors computed with the power method. This is the algorithm analysed in [4].

PM log( k ) VECTORS: spectral clustering with log( k ) random vectors computed with the power method. This is Algorithm 2.

We implement all algorithms in Python, using the numpy [12], scipy [32], stag [22], and scikit-learn [27], libraries for matrix manipulation, eigenvector computation, graph processing, and k -means approximation respectively. We first compare the performance of the algorithms on synthetic graphs with a range of sizes drawn from the stochastic block model (SBM). We then study the algorithms' performance on several real-world datasets. We find that our algorithm is significantly faster than all other spectral clustering algorithm, while maintaining almost the same clustering accuracy. The most significant improvement is seen on graphs with a large number of clusters. All experiments are performed on an HP laptop with an 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz processor and 32 GB RAM. The code to reproduce the experiments is available at https://github.com/pmacg/fast-spectral-clustering .

0

100

200

300

400

Number of clusters

0

20

40

60

80

100

120

Running time (s)

(a)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Number of vertices

×

10 6

0

20

40

60

Running time (s)

k

EIGENVECTORS

log

(

k

)

EIGENVECTORS

PM

k

PM log

(

k

)

(b)

4.1 Synthetic Data

In this section, we evaluate the spectral clustering algorithms on synthetic data drawn from the stochastic block model. Given parameters n ∈ Z ≥ 0 , k ∈ Z ≥ 0 , p ∈ [0 , 1] , and q ∈ [0 , 1] , we generate a graph G = ( V, E ) with n vertices and k ground-truth clusters S 1 , . . . , S k of size n/k . For any pair of vertices u ∈ S i and v ∈ S j , we add the edge { u, v } with probability p if i = j and with probability q otherwise. We study the running time of the algorithms in two settings.

In the first experiment, we set n = 1000 · k , p = 0 . 04 , and q = 1 / (1000 k ) . Then, we study the running time of spectral clustering for different values of k . The results are shown in Figure 2(a). We observe that our newly proposed algorithm is much faster than existing methods for large values of k , and our algorithm is easily able to scale to large graphs with several hundred thousand vertices.

In the second experiment, we set k = 20 , p = 40 /n , and q = 1 / (20 n ) . Then, we study the running time of the spectral clustering algorithms for different values of n . The results are shown in Figure 2(b). Empirically, we find that when k is a fixed constant, our newly proposed algorithm is faster than existing methods by a constant factor. In every case, all algorithms successfully recover the ground truth clusters. 3

4.2 Real-world Data

In this section, we evaluate spectral clustering on real-world data with labeled ground-truth clusters. We compare the algorithms on the following datasets from a variety of domains.

MNIST [18]: each data point is an image with 28 × 28 greyscale pixels, representing a hand-written digit from 0 to 9 .

Pen Digits [1]: data is collected by writing digits on a digital pen tablet. Each data point corresponds to some digit from 0 to 9 and consists of 8 pairs of ( x, y ) coordinates encoding the sequence of pen positions while the digit was written.

Fashion [34]: each data point is an image with 28 × 28 greyscale pixels, representing one of 10 classes of fashion item, such as 'shoe' or 'shirt'.

HAR (Human Activity Recognition) [2]: the dataset consists of pre-processed sensor data from a body-worn smartphone. Participants were asked to perform a variety of activities,

±

±

±

±

±

such as 'walking', 'walking upstairs', and 'standing'. The task is to identify the activity from the sensor data.

Letter [9]: each data point corresponds to an upper-case letter from 'A' to 'Z'. The data was generated from distorted images of letters with a variety of fonts, and the features correspond to various statistics computed on the resulting images.

The datasets are all made available by the OpenML [31] project, and can be downloaded with the scikit-learn library [27]. We first pre-process each dataset by computing the k nearest neighbour graph from the data, for k = 10 . Table 2 shows the number of nodes and the number of ground truth clusters in each dataset.

For each dataset, we report the performance of each spectral clustering algorithm with respect to the running time in seconds, and the clustering accuracy measured with the Adjusted Rand Index (ARI) [10, 29] and the Normalised Mutual Information (NMI) [17]. Table 1 summarises the results.

Table 2: The number of vertices ( n ) and clusters ( k ) in each of the real-world datasets.

We find that Algorithm 2 is consistently very fast when compared to the other spectral clustering algorithms. Moreover, the clustering accuracy is similar for every algorithm.

5 Conclusion

In this paper, we introduced a new fast spectral clustering algorithm based on projecting the vertices of the graph into O (log( k )) dimensions with the power method. We find that the new algorithm is faster than previous spectral clustering algorithms and achieves similar clustering accuracy.

This algorithm offers a new option for the application of spectral clustering. If a large running time is acceptable and the goal is to achieve the best accuracy possible, then our experimental results suggest that the classical spectral clustering algorithm with k eigenvectors is the optimal choice. On the other hand, when the number of clusters or the number of data points is very large, our newly proposed method provides a significantly faster algorithm for a small trade-off in terms of clustering accuracy. This could allow spectral clustering to be applied in regimes that were previously intractable, such as when k = Θ( n ) .

References

Fevzi Alimoglu and Ethem Alpaydin. Methods of combining multiple classifiers based on different representations for pen-based handwriting recognition. In Proceedings of the fifth Turkish artificial intelligence and artificial neural networks symposium (TAINN 96) , 1996.

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In 21st European Symposium on Artificial Neural Networks (ESANN'13) , 2013.

David Arthur and Sergei Vassilvitskii. K-means++: the advantages of careful seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'07) , pages 1027-1035, 2007.

Christos Boutsidis, Prabhanjan Kambadur, and Alex Gittens. Spectral clustering via the power method-provably. In 32nd International Conference on Machine Learning (ICML'15) , pages 40-48. PMLR, 2015.

Christos Boutsidis and Malik Magdon-Ismail. Deterministic feature selection for k-means clustering. IEEE Transactions on Information Theory , 59(9):6099-6110, 2013.

Anna Choromanska, Tony Jebara, Hyungtae Kim, Mahesh Mohan, and Claire Monteleoni. Fast spectral clustering via the nyström method. In Algorithmic Learning Theory: 24th International Conference (ALT'13) , pages 367-381, 2013.

Fan R K Chung. Spectral graph theory . American Mathematical Society, 1997.

Dan Feldman, Morteza Monemizadeh, and Christian Sohler. A PTAS for k-means clustering based on weak coresets. In Proceedings of the 23rd Annual Symposium on Computational Geometry (SoCG'07) , pages 11-18, 2007.

Peter W Frey and David J Slate. Letter recognition using holland-style adaptive classifiers. Machine learning , 6:161-182, 1991.

Alexander J. Gates and Yong-Yeol Ahn. The impact of random models on clustering similarity. The Journal of Machine Learning Research , 18(1):3049-3076, 2017.

Gene H Golub and Henk A Van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics , 123(1-2):35-65, 2000.

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, 2020.

William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemporary Mathematics , 26:189-206, 1984.

Tapas Kanungo, David M Mount, Nathan S Netanyahu, Christine D Piatko, Ruth Silverman, and Angela Y Wu. A local search approximation algorithm for k-means clustering. In Proceedings of the 18th Annual Symposium on Computational Geometry (SoCG'02) , pages 10-18, 2002.

Pavel Kolev and Kurt Mehlhorn. A note on spectral clustering. In 24th Annual European Symposium on Algorithms (ESA'16) , pages 1-14, 2016.

Amit Kumar, Yogish Sabharwal, and Sandeep Sen. A simple linear time (1+ ϵ )-approximation algorithm for k-means clustering in any dimensions. In 45th Annual IEEE Symposium on Foundations of Computer Science (FOCS'04) , pages 454-462. IEEE, 2004.

Andrea Lancichinetti, Santo Fortunato, and János Kertész. Detecting the overlapping and hierarchical community structure in complex networks. New Journal of Physics , 11(3), 2009.

Yann LeCun. The mnist database of handwritten digits. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 1998.

James R Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher-order Cheeger inequalities. Journal of the ACM , 61(6):1-30, 2014.

Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory , 28(2):129-137, 1982.

Peter Macgregor and He Sun. A tighter analysis of spectral clustering, and beyond. In 39th International Conference on Machine Learning (ICML'22) , pages 14717-14742, 2022.

Peter Macgregor and He Sun. Spectral toolkit of algorithms for graphs: Technical report (1). CoRR , abs/2304.03170, 2023.

Konstantin Makarychev, Yury Makarychev, and Ilya Razenshteyn. Performance of JohnsonLindenstrauss transform for k-means and k-medians clustering. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing (STOC'19) , page 1027-1038, 2019.

Tomohiko Mizutani. Improved analysis of spectral algorithm for clustering. Optimization Letters , 15(4):1303-1325, 2021.

Herman Müntz et al. Solution directe de l'équation séculaire et de quelques problemes analogues transcendants. Comptes rendus de l'Académie des Sciences , 156:43-46, 1913.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. 15th Advances in Neural Information Processing Systems (NeurIPS'01) , 14, 2001.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Richard Peng, He Sun, and Luca Zanetti. Partitioning well-clustered graphs: Spectral clustering works! SIAM Journal on Computing , 46(2):710-743, 2017.

William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association , 66(336):846-850, 1971.

He Sun and Luca Zanetti. Distributed graph clustering and sparsification. ACM Transactions on Parallel Computing (TOPC) , 6(3):1-23, 2019.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. SIGKDD Explorations , 15(2):49-60, 2013.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17:395-416, 2007.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Donghui Yan, Ling Huang, and Michael I Jordan. Fast approximate spectral clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD'09) , pages 907-916, 2009.

A Omitted detail from Section 3

In this section, we prove the main theoretical result of the paper. In order that this section is selfcontained, we repeat some of the steps included in the main paper. We first show that the lengths of the random vectors x i generated in Algorithm 2 are close to their expected value. Notice that E[ ∥ x i ∥ 2 ] = √ n and E[ ∥ P x i ∥ 2 ] = √ k . We use Chebyshev's inequality to show the following.

∥ P x ∥ 2 ≤ √ 6 k , and

Lemma A.1. Let x ∈ R n be drawn from the n -dimensional Gaussian distribution N( 0 , I ) . Let f 1 , . . . , f k ∈ R n be orthogonal vectors and let P = ∑ k i =1 f i f ⊺ i be the projection onto the space spanned by f 1 , . . . , f k . With probability at least 1 -(1 / 10 k ) ,

∥ x ∥ 2 ≤ √ 6 n .

Proof of Lemma A.1. Since x is drawn from a symmetric n -dimensional Gaussian distribution, ∥ x ∥ 2 2 is distributed according to a χ 2 distribution with n degrees of freedom. Similarly, since P is a projection matrix, ∥ P x ∥ 2 2 is distributed according to a χ 2 distribution with k degrees of freedom. By the Chebyshev inequality, we have that

The lemma follows by the union bound and since k ≤ n .

■

We now show that the output of the POWERMETHOD algorithm is close to a random vector in the space spanned by f 1 , . . . , f k .

Lemma 3.1. Let M ∈ R n × n be a matrix with eigenvalues 1 ≥ γ 1 ≥ . . . ≥ γ n ≥ 0 and corresponding eigenvectors f 1 , . . . f n . Let x 0 ∈ R n be drawn from the n -dimensional Gaussian distribution N ( 0 , I ) . Let x t = POWERMETHOD ( M , x 0 , t ) for t = Θ ( log( n/ϵ 2 k ) ) . If γ k ≥ 1 -O ( ϵ · log( n/ϵ 2 k ) -1 ) and γ k +1 ≤ 1 -Ω(1) , then with probability at least 1 -1 / (10 k ) ,

where P = ∑ k i =1 f i f ⊺ i is the projection onto the space spanned by the first k eigenvectors of M .

Proof of Lemma 3.1. By the assumptions of the Lemma, we can assume that

γ k +1 ≤ c 1 < 1 ,

t = c 3 log(24 n/ϵ 2 k ) ,

γ k ≥ 1 -c 2 ϵ log(24 n/ϵ 2 k ) -1 , and

for constants c 1 , c 2 , and c 3 . Fixing c 1 < 1 , we will set

and and

Furthermore, by Lemma A.1, with probability at least 1 -(1 / 10 k ) it holds that

and and we assume that this holds in the remainder of the proof.

Now, we write x 0 in terms of its expansion in the basis given by the eigenvectors f 1 , . . . , f n :

where a j = ⟨ x 0 , f j ⟩ . Similarly, we have and

Then, where we used the fact that 1 ≥ γ 1 ≥ . . . ≥ γ n . Now, we have

Furthermore,

Combining everything together, we have

which completes the proof.

■

It remains to prove that the k -means cost is preserved in the embedding produced by the power method. Recall that f 1 , . . . , f k are the eigenvectors of M corresponding to the eigenvalues γ 1 , . . . , γ k and y 1 , . . . , y l are the vectors computed in Algorithm 2. We will also consider the vectors z 1 , . . . , z l given by z i = P x i , where { x i } k i =1 are the random vectors sampled in Algorithm 2, and P = ∑ k i =1 f i f ⊺ i is the projection onto the space spanned by f 1 , . . . , f k . We also define

We will use the following result shown by Makarychev et al. [23].

Lemma 3.2 ([23], Theorem 1.3) . Given data X ∈ R n × k , let Π ∈ R k × l be a random matrix with each column sampled from the k -dimensional Gaussian distribution N( 0 , I k ) and

Then, with probability at least 1 -ϵ , it holds for all partitions { A i } k i =1 of [ n ] that

Applying this lemma with X = D -1 2 F and Π = F ⊺ Z shows that the k -means cost is approximately equal in the embeddings given by D -1 2 F and D -1 2 Z , since FF ⊺ Z = Z and each of the entries of F ⊺ Z is distributed according to the Gaussian distribution N(0 , 1) . By Lemma 3.1, we can also show that the k -means objective in D -1 2 Y is within an additive error of D -1 2 Z . This allows us to prove the following lemma.

Lemma 3.3. With probability at least 0 . 9 -ϵ , for any partitioning { A i } k i =1 of the vertex set V , we have and

In order to prove this, we will use the fact shown by Boutsidis and Magdon-Ismail [5] that we can write the k -means cost as

where X ∈ R n × k is the indicator matrix of the partition, defined by

and ∥ B ∥ F ≜ ( ∑ i,j B 2 i,j ) 1 / 2 is the Frobenius norm.

Proof of Lemma 3.3. Notice that F ⊺ Z ∈ R k × l is a random matrix with columns drawn from the standard k -dimensional Gaussian distribution. Then, by Lemma 3.2, with probability at least 1 -ϵ , we have for any partition { A i } k i =1 that

since D -1 / 2 FF ⊺ Z = D -1 / 2 Z , where we use the fact that the columns of Z are in the span of f 1 , . . . , f k .

Furthermore, by the union bound, we can assume with probability at least 0 . 9 that the conclusion of Lemma 3.1 holds for every vector y i computed by Algorithm 2.

Now, we will establish that COST D -1 / 2 Y ( · ) is close to COST D -1 / 2 Z ( · ) which will complete the proof. For some arbitrary partition { A i } k i =1 , let X be the indicator matrix of the partition. Then, we

have

Where we use Lemma 3.1, and the fact that l ≤ k . Combining this with (2) completes the proof. ■

Now we come to the proof of the main theorem.

Theorem 3.1. Let G be a graph with λ k +1 = Ω(1) and ρ ( k ) = O ( ϵ · log( n/ϵ ) -1 ) . Additionally, let { S i } k i =1 be the k -way partition corresponding to ρ ( k ) and suppose that { S i } k i =1 are almost balanced. Let { A i } k i =1 be the output of Algorithm 2. With probability at least 0 . 9 -ϵ , there exists a permutation σ : [ k ] - → [ k ] such that

Moreover, the running time of Algorithm 2 is

where m is the number of edges in G and T KM ( n, k, l ) is the running time of the k -means approximation algorithm on n points in l dimensions.

To complete the proof, we will make use of the following results proved by Macgregor and Sun [21], which hold under the same assumptions as Theorem 3.1.

Lemma 3.4 ([21], Lemma 4.1) . There exists a partition { A i } k i =1 of the vertex set V such that

Lemma 3.5 ([21], Theorem 2) . Given some partition of the vertices, { A i } k i =1 , such that

then there exists a permutation σ : [ k ] - → [ k ] such that

Proof of Theorem 3.1. By Lemma 3.4 and Lemma 3.3, with probability at least 0 . 9 -ϵ , there exists some partition { ̂ A i } k i =1 of the vertex set V G such that

Since we use a constant-factor approximation algorithm for k -means, the partition { A i } k i =1 returned by Algorithm 2 satisfies COST D -1 / 2 Y ( A 1 , . . . , A k ) = O ( ϵk ) . Then, by Lemma 3.5 and Lemma 3.3, for some permutation σ : [ k ] - → [ k ] , we have

To bound the running time, notice that the number of non-zero entries in M is 2 m , and the time complexity of matrix multiplication is proportional to the number of non-zero entries. Therefore, the running time of POWERMETHOD ( M , x 0 , t ) is ˜ O ( m ) . Since the loop in Algorithm 2 is executed Θ ( log( k ) · ϵ -2 ) times, the total running time of Algorithm 2 is ˜ O ( m · ϵ -2 ) + T KM ( n, k, l ) . ■

Peer reviews:
comments: SUMMARY:
The paper is concerned with the spectral method for clustering vertices of graph. Specifically, the authors find that the classical spectral clustering methods need eigendecomposition of the graph Laplacian matrix which is computationally expensive, and that the authors propose to use a power method to bypass this computation-demanding part. The essence is that the power method is a random projection of the eigendecomposition into subembeddings. 

In general, I do find that the idea of the paper is novel, and that the theoretical and experimental results are convincing.

STRENGTHS:
The proposed algorithm is very simple to understand and implement. Moreover, the authors have provided both theoretical and experimental results to prove that the algorithm is valid and novel.

The structure of the paper is engaging and is written easy to follow.

WEAKNESSES:
The experiments only compared to the classical eigendecomposition-based methods. It is true that the demanding computation of the embedding step motivates the authors to use a power method for the random project, however, I am not convinced that the eigendecomposition is the only method to do the graph clustering. The authors should add both empirical discussion and experiments to other such related methods.

QUESTIONS:
The authors claim that the codes are released. however, I don't see them anywhere mentioned in the paper.

pp. 3. l91. The definition of vol is not clear.

pp. 5. l132. Definition of the inner product is not clear. Is it a dot product, or E[<.,.>], since you have random variable?

pp. 5. l142. Is it 1/100k or 1/(100k)?

pp. 5. l143. Should it be bold face P?

LIMITATIONS:
No. The authors did not discuss the limitations of their method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new spectral-based graph clustering algorithm. The main novelty of this algorithm is a fast implementation of vertex embedding, built on the power method. \n\nPrevious algorithms achieved a low dimensional embedding via two steps: \n1) compute k eigenvectors of the graph Laplacian matrix; \n2) embed k dimension datapoint into (log k)-dimension via a random projection. \n\nUnder this approach, the first step may cost a lot of time, i.e., $O(kn^2)$ in a worst-case analysis. This paper simplifies this step with a power method. Concretely, their algorithm starts with (log k) random vectors $x_1,\\dots,x_r$, and calculates $M^{t} x_{i}$. The authors showed that the vectors obtained in this way can approximate random vectors from the eigenspace of $M$. For a sparse matrix $M$, this approach can be implemented much faster. Specifically, if $M$ is a graph matrix, this algorithm can be implemented in time linear in the size of $G$.\n\nPower methods are a very well known approach to approximate eigenvectors. We can usually approximate the first eigenvector very fast for sparse matrices. However, for the second eigenvector, the running time might be slower since the matrix is no longer sparse after removing the largest eigenvector. The beauty of this paper is that we can approximate all $k$ eigenvectors in one shot in some settings. However, there is no free lunch. The main theorem of this paper requires the size of clusters should be balanced. I think this assumption plays a crucial role in this algorithm.\n\nSTRENGTHS:\n1. This paper provides a nice idea to approximate the eigenvectors of the graph Laplacian matrix. The running time of this algorithm is much faster than the previous algorithms.\n\n2. The theoretical analysis is solid and interesting!\n\n3. The power method has good potential. Beyond approximating top eigenvectors, it might have applications in different steps of clustering.\n\nWEAKNESSES:\nI first list some minor cons in this paper.\n\n1. It requires the knowledge of k\n\n2. The parameter q is too small in the SBM simulation.\n\n3. The authors used some subjective words to describe their algorithm, such as they call their algorithm a simple algorithm, and their assumptions in the main theorem are natural assumptions. I disagree with the second statement and find these assumptions to be not natural at all. However, I agree that this is a subjective viewpoint and thus put this as a minor con.\n\n4. In the real dataset experiments, it seems that none of the four algorithms are very exciting. For example, the highest NMI achieved for MNIST is 0.74 by spectral clustering through k eigenvectors. However, a quick online search shows that there are many  algorithms with NMI higher than 0.9. See the link: https://paperswithcode.com/sota/image-clustering-on-mnist-full \n\n5. My main concern in this paper is the balancedness assumption. This assumption is not true in many applications. In addition, I think this assumption is crucial for the performance of the algorithm (and not just the analysis), meaning this is an inherent barrier. If the authors want to make this algorithm useful in practice, I think this is a must-be-addressed issue. I am happy to increase my evaluation if the authors can address this barrier. Let me ask a concrete question. Let’s consider the following SBM setting with p = 0.01 and q=0.00001. We assume there is a large cluster of size n/2, and the rest of the clusters have size n^{0.99}/2. Could this power method-based algorithm recover these clusters? Regarding the experiments part, I know that MNIST and Pen Digits are balanced datasets. Are there any unbalanced datasets in the experiments? The authors did not mention the size of each cluster in the paper.\n\nQUESTIONS:\nCould the authors address my concerns about the unbalanced cases?\n\nLIMITATIONS:\nI have included my suggestions in the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper deals with the problem of spectral clustering in networks. The paper proposes a fast spectral clustering method by utilizing the power method and random projection. The paper also provides theoretical guarantee of cluster recovery.

STRENGTHS:
The main strengths of the paper are - 

(1) The paper provides a fast spectral clustering algorithm using the power method and random projections.

(2) The paper provides theoretical results on cluster recovery based on the proposed method.

(3) The paper provides a simulation study of the method and applies the proposed method to a few real-world networks.

(4) The paper is well-written.

WEAKNESSES:
The main weaknesses of the paper are - 

(1) The paper extends the recent breakthrough on o(log(k))-embedding of vertices for cluster recovery. The use of the power method as a proxy for eigenvectors is not particularly novel.

(2) The stopping criteria in Algorithms 1 and 2 are in terms of order of log(n/k). The dependence of the results on the constants is not clear.

QUESTIONS:
(1) How does the proposed algorithm depend on the sparsity of the network?

(2) What should be the choice of the constants in Algorithms 1 and 2 in terms of O(log(n/k)) and O(log(k))?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new spectral clustering algorithm that  instead of computing $k$-leading eigenvectors, the authors proposes to use power method to embed the nodes into a $\\log(k)$ dimension space, which saves computational time when $k$ is large. Theoretical guarantees on misclassification rate with constant probability is provided under balanced cluster setting. Experiments on simulated network and $k$-NN graphs on real datasets are provided to show the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\n- The idea on projecting the nodes to $\\log(k)$ random space for spectral clustering is novel and interesting. \n- The paper is well written and easy to follow\n\nWEAKNESSES:\n- The theoretical guarantee is limited on balanced cluster case and it is not a high probability bound\n- The real data experiments are not convincible as \n  - The authors manually construct $k$-NN graphs from some real data, instead of directly using existing real graph data. \n  - The authors did not study how sparsity of the network may affect the results. For example, real networks are usually very sparse, will the accuracy be hurt more for the proposed methods under this case? For sparse networks, computing $k$-leading eigenvectors is not that expensive, and the advantage on the proposed method on computing speed may not be that significant, from Table 1, the computing speed up on most of the datasets are not that remarkable.\n  - It is not clear for real dataset how balanced are the clusters.\n\nQUESTIONS:\n- Could the authors have a study on why for Letter dataset $k$ and $\\log(k)$ EIGS takes so much more time than PM $k$/$\\log(k)$? Did you also use power method to compute the leading $k$-eigenvectors? Since Letter dataset has fewer nodes than other datasets, but slightly larger $k$, it is very strange that the competing time difference is so much than other datasets.\n- Why do you not use misclassification accuracy for the experiments since it is the main theoretical result you have?\n\nLIMITATIONS:\nI do not see a section stating or broader impact of the work. The authors discussed about the limitations in conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new fast spectral clustering algorithm. The main idea of the algorithm is to, rather than embed the vertices according to the k eigenvectors of the graph Laplacian, use the power method to efficiently approximate a random projection of the spectral embedding. Then, as in the classical spectral clustering algorithm, a k-means clustering algorithm is used to partition the vertices into k clusters.\n\nThe paper starts by discussing relevant preliminaries such as conductance, a higher order Cheeger inequality, the k-means objective and the power method. Next, the result is proven that if the k-th eigenvalue of the matrix M is close to 1 then the power method can be used to compute a random vector in the span of the first k eigenvectors of M. The authors then present their fast spectral clustering algorithm. They proceed to give a bound on the number of misclassified vertices by their proposed algorithm, and analyze the time complexity of the method. Finally, the authors sketch the proof of their main result, building on a recent result by Makarychev et al [22] which states that a random projection of data onto O(log(k)) dimensions preserves the k-means objective function for all partitions of the data with high probability.\n\nIn the experimental section, the proposed algorithm is evaluated on synthetic data under the stochastic block model as well as some medium-sized real world datasets. The algorithm is compared to the classical spectral clustering algorithm using k as well as log(k) eigenvectors as well as the power method using k eigenvectors. The authors observe that the proposed method is much faster than competing methods when increasing the number of clusters and number of vertices in the stochastic block model. The same observation is done on the real world datasets, while achieving comparable clustering accuracy.\n\nSTRENGTHS:\nThe paper proposes a new algorithm for fast spectral clustering which is simple and easy to implement. The key technical contribution is a proof that they can efficiently compute an approximation of a random projection on the spectral embedding without computing the spectral embedding itself. The proposed method improves on the algorithm by Boutsidis et al [4] both in theory (as it has better time complexity, and an upper bound on the total misclassified vertices is given), as well as in practice on a number of datasets. The improved efficiency of the algorithm allows it to be applied for problems with a large amount of vertices and clusters.\n\nWEAKNESSES:\nThe main weakness of the paper is the limited experimental evaluation. The paper only compares to the method by Boutsidis et al [4] as well as the classical spectral clustering algorithm with k or log(k) eigenvectors. A plethora of algorithms have been proposed to do fast approximate spectral clustering. The paper even mentions the algorithms by Choromanska et al [4] and Yan et al [34] but an experimental comparison to these methods is missing. While the proposed method always outperforms the other considered methods in terms of runtimes, it is in some cases significantly worse in terms of ARI and NMI score (in particular for the MNIST, Fashion and HAR datasets). It would be interesting to see if some of the other competing methods achieve a better trade-off here. \n\nMoreover, one of the main selling points of the algorithm is its better time complexity and scalability compared to standard spectral clustering. However, the evaluation is done on relatively small datasets of up to 70000 vertices. The paper could be made stronger by also showing the results on some larger datasets where standard spectral clustering is not applicable any more. \n\nFinally, the paper might also benefit from an evaluation in terms of the conductance score since it is the underlying objective used in spectral clustering.\n\n\n====\nEdit after Rebuttal:\n\nIn their rebuttal, the authors have provided some additional experiments and pointed out some additional results in the paper. I have therefore increased my score.\n\nQUESTIONS:\nHow does the method perform against other fast spectral clustering techniques such as Choromanska et al [4] or Yan et al [34]?\n\nLIMITATIONS:\nThe paper discusses the limitations of the proposed approach which is that it leads to a trade-off in terms of clustering accuracy compared to classical spectral clustering. There are no potential negative societal impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}