SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {C O S T} _ {\mathbf {D} ^ {- 1 / 2} \mathbf {Y}} (\widehat {A} _ {1}, \dots , \widehat {A} _ {k}) = O \left((1 + \epsilon) \frac {\epsilon k}{\log (n / \epsilon)} + \epsilon k\right).
$$
Since we use a constant-factor approximation algorithm for  $k$ -means, the partition  $\{A_i\}_{i=1}^k$  returned by Algorithm 2 satisfies  $\mathrm{COST}_{\mathbf{D}^{-1/2}\mathbf{Y}}(A_1, \ldots, A_k) = O(\epsilon k)$ . Then, by Lemma 3.5 and Lemma 3.3, for some permutation  $\sigma : [k] \to [k]$ , we have
To bound the running time, notice that the number of non-zero entries in  $\mathbf{M}$  is  $2m$ , and the time complexity of matrix multiplication is proportional to the number of non-zero entries. Therefore, the running time of POWERMETHOD  $(\mathbf{M}, \boldsymbol{x}_0, t)$  is  $\widetilde{O}(m)$ . Since the loop in Algorithm 2 is executed  $\Theta\left(\log(k) \cdot \epsilon^{-2}\right)$  times, the total running time of Algorithm 2 is  $\widetilde{O}\left(m \cdot \epsilon^{-2}\right) + T_{\mathrm{KM}}(n, k, l)$ .
In this section, we empirically study several variants of the spectral clustering algorithm. We compare the following algorithms:
-  $k$  EIGENVECTORS: the classical spectral clustering algorithm which uses  $k$  eigenvectors of the graph Laplacian matrix to embed the vertices. This is the algorithm analysed in [21, 28].
-  $\log (k)$  EIGENVECTORS: spectral clustering with  $\log (k)$  eigenvectors of the graph Laplacian to embed the vertices.
- KASP: the fast spectral clustering algorithm proposed by Yan et al. [35]. The algorithm proceeds by first coarsening the data with  $k$ -means before applying spectral clustering.
- PM  $k$  VECTORS (Power Method with  $k$  vectors): spectral clustering with  $k$  orthogonal vectors computed with the power method. This is the algorithm analysed in [4].
- PM  $\log (k)$  VECTORS: spectral clustering with  $\log (k)$  random vectors computed with the power method. This is Algorithm 2.
We implement all algorithms in Python, using the numpy [12], scipy [32], stag [22], and scikit-learn [27], libraries for matrix manipulation, eigenvector computation, graph processing, and  $k$ -means approximation respectively. We first compare the performance of the algorithms on synthetic graphs with a range of sizes drawn from the stochastic block model (SBM). We then study the algorithms' performance on several real-world datasets. We find that our algorithm is significantly faster than all other spectral clustering algorithm, while maintaining almost the same clustering accuracy. The most significant improvement is seen on graphs with a large number of clusters. All experiments are performed on an HP laptop with an 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz processor and 32 GB RAM. The code to reproduce the experiments is available at https://github.com/pmacg/fast-spectral-clustering.
(b)
Figure 2: The running time of spectral clustering variants on graphs drawn from the stochastic block model. (a) Setting  $n = 1000 \cdot k$  and increasing the number of clusters,  $k$ , shows that Algorithm 2 is much faster than alternative methods for large values of  $k$ . (b) Setting  $k = 20$  and increasing the number of vertices,  $n$ , shows that for fixed  $k$ , Algorithm 2 is faster than the alternatives by a constant factor.
In this section, we evaluate the spectral clustering algorithms on synthetic data drawn from the stochastic block model. Given parameters  $n \in \mathbb{Z}_{\geq 0}$ ,  $k \in \mathbb{Z}_{\geq 0}$ ,  $p \in [0,1]$ , and  $q \in [0,1]$ , we generate a graph  $G = (V,E)$  with  $n$  vertices and  $k$  ground-truth clusters  $S_{1},\ldots ,S_{k}$  of size  $n / k$ . For any pair of vertices  $u \in S_{i}$  and  $v \in S_{j}$ , we add the edge  $\{u,v\}$  with probability  $p$  if  $i = j$  and with probability  $q$  otherwise. We study the running time of the algorithms in two settings.
In the first experiment, we set  $n = 1000 \cdot k$ ,  $p = 0.04$ , and  $q = 1 / (1000k)$ . Then, we study the running time of spectral clustering for different values of  $k$ . The results are shown in Figure 2(a). We observe that our newly proposed algorithm is much faster than existing methods for large values of  $k$ , and our algorithm is easily able to scale to large graphs with several hundred thousand vertices.
In the second experiment, we set  $k = 20$ ,  $p = 40 / n$ , and  $q = 1 / (20n)$ . Then, we study the running time of the spectral clustering algorithms for different values of  $n$ . The results are shown in Figure 2(b). Empirically, we find that when  $k$  is a fixed constant, our newly proposed algorithm is faster than existing methods by a constant factor. In every case, all algorithms successfully recover the ground truth clusters. $^3$
4.2 Real-world Data
In this section, we evaluate spectral clustering on real-world data with labeled ground-truth clusters. We compare the algorithms on the following datasets from a variety of domains.
- MNIST [18]: each data point is an image with  $28 \times 28$  greyscale pixels, representing a hand-written digit from 0 to 9.
- Pen Digits [1]: data is collected by writing digits on a digital pen tablet. Each data point corresponds to some digit from 0 to 9 and consists of 8 pairs of  $(x, y)$  coordinates encoding the sequence of pen positions while the digit was written.
- Fashion [34]: each data point is an image with  $28 \times 28$  greyscale pixels, representing one of 10 classes of fashion item, such as 'shoe' or 'shirt'.
- HAR (Human Activity Recognition) [2]: the dataset consists of pre-processed sensor data from a body-worn smartphone. Participants were asked to perform a variety of activities,
Note that we cannot compare with the KASP algorithm on the stochastic block model since KASP is designed to operate on vector data rather than on graphs.
Table 1: The performance of spectral clustering algorithms on real-world datasets. The PM  $\log (k)$  algorithm corresponds to Algorithm 2. We perform 10 trials and report the average performance with one standard deviation of uncertainty. We observe that Algorithm 2 is consistently very fast when compared to the other algorithms while achieving comparable clustering accuracy.
Algorithm	Dataset
MNIST	PenDigits	Fashion	HAR	Letter
Time	k EIGS	2.70 ± 0.24	0.64 ± 0.07	3.55 ± 0.17	0.58 ± 0.07	29.29 ± 11.85
log(k) EIGS	2.73 ± 0.20	1.01 ± 0.05	3.79 ± 0.11	0.83 ± 0.05	24.99 ± 11.58
KASP	15.47 ± 3.40	0.22 ± 0.03	14.33 ± 3.54	0.91 ± 0.19	0.33 ± 0.14
PM k	3.23 ± 0.15	0.49 ± 0.02	2.40 ± 0.07	0.38 ± 0.01	1.14 ± 0.02
PM log(k)	1.99 ± 0.05	0.36 ± 0.01	1.19 ± 0.06	0.30 ± 0.02	0.39 ± 0.02
ARI	k EIGS	0.61 ± 0.01	0.58 ± 0.02	0.42 ± 0.00	0.51 ± 0.00	0.17 ± 0.00
log(k) EIGS	0.49 ± 0.03	0.63 ± 0.06	0.32 ± 0.02	0.30 ± 0.01	0.17 ± 0.00
KASP	0.33 ± 0.03	0.42 ± 0.04	0.30 ± 0.03	0.48 ± 0.03	0.13 ± 0.01
PM k	0.55 ± 0.03	0.60 ± 0.07	0.40 ± 0.02	0.50 ± 0.02	0.17 ± 0.00
PM log(k)	0.51 ± 0.04	0.61 ± 0.05	0.35 ± 0.03	0.49 ± 0.02	0.17 ± 0.00
NMI	k EIGS	0.74 ± 0.01	0.78 ± 0.00	0.60 ± 0.00	0.72 ± 0.00	0.27 ± 0.02
log(k) EIGS	0.68 ± 0.01	0.77 ± 0.03	0.55 ± 0.01	0.48 ± 0.02	0.13 ± 0.02
KASP	0.48 ± 0.02	0.60 ± 0.03	0.46 ± 0.03	0.61 ± 0.03	0.35 ± 0.01
PM k	0.73 ± 0.03	0.76 ± 0.03	0.61 ± 0.02	0.69 ± 0.02	0.29 ± 0.03
PM log(k)	0.69 ± 0.02	0.77 ± 0.02	0.55 ± 0.04	0.66 ± 0.04	0.30 ± 0.01
such as 'walking', 'walking upstairs', and 'standing'. The task is to identify the activity from the sensor data.
- Letter [9]: each data point corresponds to an upper-case letter from 'A' to 'Z'. The data was generated from distorted images of letters with a variety of fonts, and the features correspond to various statistics computed on the resulting images.
The datasets are all made available by the OpenML [31] project, and can be downloaded with the scikit-learn library [27]. We first pre-process each dataset by computing the  $k$  nearest neighbour graph from the data, for  $k = 10$ . Table 2 shows the number of nodes and the number of ground truth clusters in each dataset.
For each dataset, we report the performance of each spectral clustering algorithm with respect to the running time in seconds, and the clustering accuracy measured with the Adjusted Rand Index (ARI) [10, 29] and the Normalised Mutual Information (NMI) [17]. Table 1 summarises the results.
Table 2: The number of vertices  $(n)$  and clusters  $(k)$  in each of the real-world datasets.
Dataset	n	k
MNIST	70000	10
PenDigits	7494	10
Fashion	70000	10
HAR	10299	6
Letter	20000	26
We find that Algorithm 2 is consistently very fast when compared to the other spectral clustering algorithms. Moreover, the clustering accuracy is similar for every algorithm.
In this paper, we introduced a new fast spectral clustering algorithm based on projecting the vertices of the graph into  $O(\log (k))$  dimensions with the power method. We find that the new algorithm is faster than previous spectral clustering algorithms and achieves similar clustering accuracy.
This algorithm offers a new option for the application of spectral clustering. If a large running time is acceptable and the goal is to achieve the best accuracy possible, then our experimental results suggest that the classical spectral clustering algorithm with  $k$  eigenvectors is the optimal choice. On the other hand, when the number of clusters or the number of data points is very large, our newly proposed method provides a significantly faster algorithm for a small trade-off in terms of clustering accuracy. This could allow spectral clustering to be applied in regimes that were previously intractable, such as when  $k = \Theta(n)$ .
This work is supported by EPSRC Early Career Fellowship (EP/T00729X/1).
A Omitted detail from Section 3
In this section, we prove the main theoretical result of the paper. In order that this section is self-contained, we repeat some of the steps included in the main paper. We first show that the lengths of the random vectors  $\boldsymbol{x}_i$  generated in Algorithm 2 are close to their expected value. Notice that  $\operatorname{E}\left[\| \boldsymbol{x}_i\| _2\right] = \sqrt{n}$  and  $\operatorname{E}\left[\| \mathbf{P}\boldsymbol{x}_i\| _2\right] = \sqrt{k}$ . We use Chebyshev's inequality to show the following.
Lemma A.1. Let  $\pmb{x} \in \mathbb{R}^n$  be drawn from the  $n$ -dimensional Gaussian distribution  $\mathrm{N}(0, \mathbf{I})$ . Let  $\pmb{f}_1, \dots, \pmb{f}_k \in \mathbb{R}^n$  be orthogonal vectors and let  $\mathbf{P} = \sum_{i=1}^{k} \pmb{f}_i \pmb{f}_i^\top$  be the projection onto the space spanned by  $\pmb{f}_1, \dots, \pmb{f}_k$ . With probability at least  $1 - (1/10k)$ ,
$\| \mathbf{P}\mathbf{x}\| _2\leq \sqrt{6k},$  and
$\| \pmb {x}\| _2\leq \sqrt{6n}.$
Proof of Lemma A.1. Since  $\pmb{x}$  is drawn from a symmetric  $n$ -dimensional Gaussian distribution,  $\| \pmb{x} \|_2^2$  is distributed according to a  $\chi^2$  distribution with  $n$  degrees of freedom. Similarly, since  $\mathbf{P}$  is a projection matrix,  $\| \mathbf{P} \pmb{x} \|_2^2$  is distributed according to a  $\chi^2$  distribution with  $k$  degrees of freedom. By the Chebyshev inequality, we have that
$$
\operatorname * {P r} \left[ \| \mathbf {P} \boldsymbol {x} _ {i} \| _ {2} ^ {2} \geq 6 k \right] \leq \frac {k}{(5 k) ^ {2}} = \frac {1}{2 5 k},
$$
$$
\operatorname * {P r} \left[ \| \boldsymbol {x} _ {i} \| _ {2} ^ {2} \geq 6 n \right] \leq \frac {n}{(5 n) ^ {2}} = \frac {1}{2 5 n}.
$$
The lemma follows by the union bound and since  $k \leq n$ .
We now show that the output of the POWERMETHOD algorithm is close to a random vector in the space spanned by  $f_1, \ldots, f_k$ .
Lemma 3.1. Let  $\mathbf{M} \in \mathbb{R}^{n \times n}$  be a matrix with eigenvalues  $1 \geq \gamma_1 \geq \ldots \geq \gamma_n \geq 0$  and corresponding eigenvectors  $\pmb{f}_1, \ldots, \pmb{f}_n$ . Let  $\pmb{x}_0 \in \mathbb{R}^n$  be drawn from the  $n$ -dimensional Gaussian distribution  $N(\mathbf{0}, \mathbf{I})$ . Let  $\pmb{x}_t = \text{POWERMETHOD}(\mathbf{M}, \pmb{x}_0, t)$  for  $t = \Theta\big(\log(n/\epsilon^2k)\big)$ . If  $\gamma_k \geq 1 - O\big(\epsilon \cdot \log(n/\epsilon^2k)^{-1}\big)$  and  $\gamma_{k+1} \leq 1 - \Omega(1)$ , then with probability at least  $1 - 1/(10k)$ ,
$$
\left\| \pmb {x} _ {t} - \mathbf {P} \pmb {x} _ {0} \right\| _ {2} \leq \epsilon \sqrt {k},
$$
Proof of Lemma 3.1. By the assumptions of the Lemma, we can assume that
-  $\gamma_{k + 1} \leq c_1 < 1$ ,
-  $\gamma_{k} \geq 1 - c_{2}\epsilon \log (24n / \epsilon^{2}k)^{-1}$ , and
$t = c_{3}\log (24n / \epsilon^{2}k)$
for constants  $c_{1}, c_{2}$ , and  $c_{3}$ . Fixing  $c_{1} < 1$ , we will set
$$
c _ {3} = \frac {1}{2 \log \left(\frac {1}{c _ {1}}\right)}
$$
$$
c _ {2} = \frac {1}{c _ {3} \cdot 2 \sqrt {6}}.
$$
Furthermore, by Lemma A.1, with probability at least  $1 - (1/10k)$  it holds that
$$
\left\| \mathbf {P} \boldsymbol {x} _ {0} \right\| _ {2} \leq \sqrt {6 k}
$$
$$
\left\| \boldsymbol {x} _ {0} \right\| _ {2} \leq \sqrt {6 n},
$$
and we assume that this holds in the remainder of the proof.
Now, we write  $\pmb{x}_0$  in terms of its expansion in the basis given by the eigenvectors  $\pmb{f}_1, \dots, \pmb{f}_n$ :
$$
\boldsymbol {x} _ {0} = \sum_ {j = 1} ^ {n} a _ {j} \boldsymbol {f} _ {j},
$$
where  $a_{j} = \langle \pmb{x}_{0},\pmb{f}_{j}\rangle$  . Similarly, we have
$$
\mathbf {P} \boldsymbol {x} _ {0} = \sum_ {j = 1} ^ {k} a _ {j} \boldsymbol {f} _ {j}
$$
$$
\boldsymbol {x} _ {t} = \sum_ {j = 1} ^ {n} a _ {j} \gamma_ {j} ^ {t} \boldsymbol {f} _ {j}.
$$
$$
\begin{array}{l} \left\| \boldsymbol {x} _ {t} - \mathbf {P} \boldsymbol {x} _ {0} \right\| _ {2} = \left\| \sum_ {j = 1} ^ {k} \left(a _ {j} \gamma_ {j} ^ {t} - a _ {j}\right) \boldsymbol {f} _ {j} + \sum_ {j = k + 1} ^ {n} a _ {j} \gamma_ {j} ^ {t} \boldsymbol {f} _ {j} \right\| _ {2} \\ \leq \left\| \sum_ {j = 1} ^ {k} \left(a _ {j} \gamma_ {j} ^ {t} - a _ {j}\right) \boldsymbol {f} _ {j} \right\| _ {2} + \left\| \sum_ {j = k + 1} ^ {n} a _ {j} \gamma_ {j} ^ {t} \boldsymbol {f} _ {j} \right\| _ {2} \\ \leq \left\| \left(1 - \gamma_ {k} ^ {t}\right) \sum_ {j = 1} ^ {k} a _ {j} \boldsymbol {f} _ {j} \right\| _ {2} + \left\| \gamma_ {k + 1} ^ {t} \sum_ {j = k + 1} ^ {n} a _ {j} \boldsymbol {f} _ {j} \right\| _ {2} \\ = \left(1 - \gamma_ {k} ^ {t}\right) \| \mathbf {P} \boldsymbol {x} _ {0} \| _ {2} + \gamma_ {k + 1} ^ {t} \| (\mathbf {I} - \mathbf {P}) \boldsymbol {x} _ {0} \| _ {2} \\ \leq \left(1 - \gamma_ {k} ^ {t}\right) \| \mathbf {P} \boldsymbol {x} _ {0} \| _ {2} + \gamma_ {k + 1} ^ {t} \| \boldsymbol {x} _ {0} \| _ {2} \\ \end{array}
$$
where we used the fact that  $1 \geq \gamma_{1} \geq \ldots \geq \gamma_{n}$ . Now, we have
$$
\begin{array}{l} \gamma_ {k} ^ {t} \geq \left(1 - c _ {2} \epsilon \log (2 4 n / \epsilon^ {2} k) ^ {- 1}\right) ^ {c _ {3} \log (2 4 n / \epsilon^ {2} k)} \\ \geq 1 - c _ {2} c _ {3} \epsilon \\ = 1 - \frac {\epsilon}{2 \sqrt {6}}. \\ \end{array}
$$
$$
\begin{array}{l} \gamma_ {k + 1} ^ {t} \leq c _ {1} ^ {c _ {3} \log (2 4 n / \epsilon^ {2} k)} \\ = \left(\frac {1}{c _ {1}}\right) ^ {c _ {3} \log (\epsilon^ {2} k / 2 4 n)} \\ = \left(\frac {\epsilon^ {2} k}{2 4 n}\right) ^ {c _ {3} \log (1 / c _ {1})} \\ = \epsilon \sqrt {\frac {k}{2 4 n}}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| \boldsymbol {x} _ {t} - \mathbf {P} \boldsymbol {x} _ {0} \right\| _ {2} \leq \frac {\epsilon}{2 \sqrt {6}} \left\| \mathbf {P} \boldsymbol {x} _ {0} \right\| _ {2} + \epsilon \sqrt {\frac {k}{2 4 n}} \left\| \boldsymbol {x} _ {0} \right\| _ {2} \\ \leq \frac {\epsilon}{2 \sqrt {6}} \sqrt {6 k} + \epsilon \sqrt {\frac {6 k n}{2 4 n}} \\ \leq \epsilon \sqrt {k}, \\ \end{array}
$$
It remains to prove that the  $k$ -means cost is preserved in the embedding produced by the power method. Recall that  $\pmb{f}_1, \dots, \pmb{f}_k$  are the eigenvectors of  $\mathbf{M}$  corresponding to the eigenvalues  $\gamma_1, \dots, \gamma_k$  and  $\pmb{y}_1, \dots, \pmb{y}_l$  are the vectors computed in Algorithm 2. We will also consider the vectors  $\pmb{z}_1, \dots, \pmb{z}_l$  given by  $\pmb{z}_i = \mathbf{P} \pmb{x}_i$ , where  $\{\pmb{x}_i\}_{i=1}^k$  are the random vectors sampled in Algorithm 2, and  $\mathbf{P} = \sum_{i=1}^k f_i f_i^\top$  is the projection onto the space spanned by  $\pmb{f}_1, \dots, \pmb{f}_k$ . We also define
$$
\mathbf {F} = \left[ \begin{array}{c c c} | & & | \\ \boldsymbol {f} _ {1} & \ldots & \boldsymbol {f} _ {k} \\ | & & | \end{array} \right], \quad \mathbf {Y} = \left[ \begin{array}{c c c} | & & | \\ \boldsymbol {y} _ {1} & \ldots & \boldsymbol {y} _ {l} \\ | & & | \end{array} \right] \quad \text {a n d} \quad \mathbf {Z} = \left[ \begin{array}{c c c} | & & | \\ z _ {1} & \ldots & z _ {l} \\ | & & | \end{array} \right].
$$
We will use the following result shown by Makarychev et al. [23].
Then, with probability at least  $1 - \epsilon$ , it holds for all partitions  $\{A_{ij}\}_{i = 1}^{k}$  of  $[n]$  that
$$
\operatorname {C O S T} _ {\mathbf {X}} \left(A _ {1}, \dots , A _ {k}\right) \in (1 \pm \epsilon) \operatorname {C O S T} _ {\mathbf {X N}} \left(A _ {1}, \dots , A _ {k}\right).
$$
In order to prove this, we will use the fact shown by Boutsidis and Magdon-Ismail [5] that we can write the  $k$ -means cost as
$$
\operatorname {C O S T} _ {\mathbf {B}} \left(A _ {i}, \dots , A _ {k}\right) = \left\| \mathbf {B} - \mathbf {X X} ^ {\intercal} \mathbf {B} \right\| _ {F} ^ {2} \tag {1}
$$
where  $\mathbf{X} \in \mathbb{R}^{n \times k}$  is the indicator matrix of the partition, defined by
$$
\mathbf {X} (u, i) = \left\{ \begin{array}{l l} \frac {1}{\sqrt {| A _ {i} |}} & \text {i f} u \in A _ {i} \\ 0 & \text {o t h e r w i s e} \end{array} \right.,
$$
and  $\| \mathbf{B}\| _F\triangleq (\sum_{i,j}\mathbf{B}_{i,j}^2)^{1 / 2}$  is the Frobenius norm.
Proof of Lemma 3.3. Notice that  $\mathbf{F}^{\intercal}\mathbf{Z} \in \mathbb{R}^{k \times l}$  is a random matrix with columns drawn from the standard  $k$ -dimensional Gaussian distribution. Then, by Lemma 3.2, with probability at least  $1 - \epsilon$ , we have for any partition  $\{A_i\}_{i=1}^k$  that
$$
\operatorname {C O S T} _ {\mathbf {D} ^ {- 1 / 2} \mathbf {F}} \left(A _ {1}, \dots , A _ {k}\right) \in (1 \pm \epsilon) \operatorname {C O S T} _ {\mathbf {D} ^ {- 1 / 2} \mathbf {Z}} \left(A _ {1}, \dots , A _ {k}\right) \tag {2}
$$
since  $\mathbf{D}^{-1/2}\mathbf{F}\mathbf{F}^{\intercal}\mathbf{Z} = \mathbf{D}^{-1/2}\mathbf{Z}$ , where we use the fact that the columns of  $\mathbf{Z}$  are in the span of  $f_1, \ldots, f_k$ .
Furthermore, by the union bound, we can assume with probability at least 0.9 that the conclusion of Lemma 3.1 holds for every vector  $\mathbf{y}_i$  computed by Algorithm 2.
Now, we will establish that  $\mathrm{COST}_{\mathbf{D}^{-1/2}\mathbf{Y}}(\cdot)$  is close to  $\mathrm{COST}_{\mathbf{D}^{-1/2}\mathbf{Z}}(\cdot)$  which will complete the proof. For some arbitrary partition  $\{A_i\}_{i=1}^k$ , let  $\mathbf{X}$  be the indicator matrix of the partition. Then, we
$$
\begin{array}{l} \left\| \mathbf {D} ^ {- 1 / 2} \mathbf {Y} - \mathbf {X X} ^ {\intercal} \mathbf {D} ^ {- 1 / 2} \mathbf {Y} \right\| _ {F} - \left\| \mathbf {D} ^ {- 1 / 2} \mathbf {Z} - \mathbf {X X} ^ {\intercal} \mathbf {D} ^ {- 1 / 2} \mathbf {Z} \right\| _ {F} \\ = \left\| (\mathbf {I} - \mathbf {X X} ^ {\intercal}) \mathbf {D} ^ {- 1 / 2} \mathbf {Y} \right\| _ {F} - \left\| (\mathbf {I} - \mathbf {X X} ^ {\intercal}) \mathbf {D} ^ {- 1 / 2} \mathbf {Z} \right\| _ {F} \\ \leq \left\| (\mathbf {I} - \mathbf {X X ^ {\intercal}}) \mathbf {D} ^ {- 1 / 2} (\mathbf {Y} - \mathbf {Z}) \right\| _ {F} \\ \leq \left\| (\mathbf {I} - \mathbf {X X ^ {\intercal}}) (\mathbf {Y} - \mathbf {Z}) \right\| _ {F} \\ \leq \left\| \mathbf {Y} - \mathbf {Z} \right\| _ {F} \\ = \sqrt {\sum_ {i = 1} ^ {l} \left\| \boldsymbol {y} _ {i} - \boldsymbol {z} _ {i} \right\| _ {2} ^ {2}} \\ \leq \sqrt {l \epsilon^ {2} k} \\ \leq \epsilon k, \\ \end{array}
$$
Where we use Lemma 3.1, and the fact that  $l \leq k$ . Combining this with (2) completes the proof.
Now we come to the proof of the main theorem.
$$
\sum_ {i = 1} ^ {k} \operatorname {v o l} \left(A _ {i} \triangle S _ {\sigma (i)}\right) = O (\epsilon \cdot \operatorname {v o l} \left(V _ {G}\right)).
$$
To complete the proof, we will make use of the following results proved by Macgregor and Sun [21], which hold under the same assumptions as Theorem 3.1.
$$
\operatorname {C O S T} _ {\mathbf {D} ^ {- 1 / 2} \mathbf {F}} \left(A _ {1}, \dots A _ {k}\right) \leq c \cdot k,
$$
then there exists a permutation  $\sigma :[k]\to [k]$  such that
To bound the running time, notice that the number of non-zero entries in  $\mathbf{M}$  is  $2m$ , and the time complexity of matrix multiplication is proportional to the number of non-zero entries. Therefore, the running time of POWERMETHOD  $(\mathbf{M},\boldsymbol{x}_0,t)$  is  $\widetilde{O}(m)$ . Since the loop in Algorithm 2 is executed  $\Theta\big(\log(k)\cdot\epsilon^{-2}\big)$  times, the total running time of Algorithm 2 is  $\widetilde{O}\big(m\cdot\epsilon^{-2}\big)+T_{\mathrm{KM}}(n,k,l)$ .