Title: On the Power of SVD in the Stochastic Block Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Clustering Algorithms

Stochastic Block Model

Spectral Algorithms
Abstract: A popular heuristic method for improving clustering results is to apply dimensionality reduction before running clustering algorithms.
It has been observed that spectral-based dimensionality reduction tools, such as PCA or SVD, improve the performance of clustering algorithms in many applications. This phenomenon indicates that spectral method not only serves as a dimensionality reduction tool, but also contributes to the clustering procedure in some sense. It is an interesting question to understand the behavior of spectral steps in clustering problems.

As an initial step in this direction, this paper studies the power of vanilla-SVD algorithm in the stochastic block model (SBM). We show that, in the symmetric setting, vanilla-SVD algorithm recovers all clusters correctly. This result answers an open question posed by Van Vu (Combinatorics Probability and Computing, 2018) in the symmetric setting.

On the Power of SVD in the Stochastic Block Model

Abstract

A popular heuristic method for improving clustering results is to apply dimensionality reduction before running clustering algorithms. It has been observed that spectral-based dimensionality reduction tools, such as PCA or SVD, improve the performance of clustering algorithms in many applications. This phenomenon indicates that spectral method not only serves as a dimensionality reduction tool, but also contributes to the clustering procedure in some sense. It is an interesting question to understand the behavior of spectral steps in clustering problems.

As an initial step in this direction, this paper studies the power of vanilla-SVD algorithm in the stochastic block model (SBM). We show that, in the symmetric setting, vanilla-SVD algorithm recovers all clusters correctly. This result answers an open question posed by Van Vu (Combinatorics Probability and Computing, 2018) in the symmetric setting.

1 Introduction

Clustering is a fundamental task in machine learning, with applications in many fields, such as biology, data mining, and statistical physics. Given a set of objects, the goal is to partition them into clusters according to their similarities. Objects and known relations can be represented in various ways. In most cases, objects are represented as vectors in R d , forming a data set D ⊂ R d ; each coordinate is called a feature, whose value is directly derived from raw data.

In many applications, the number of features is very large. It has been observed that the performance of classical clustering algorithms such as K-means may be worse on high-dimensional datasets. Some people call this phenomenon curse of dimensionality in machine learning [SEK04]. A popular heuristic method to address this issue is to apply dimensionality reduction before clustering. Among tools for dimensionality reduction, it is noted in practice that spectral methods such as principal component analysis (PCA) and singular value decomposition (SVD) significantly improve clustering results, e.g., [SEK04, KAH19].

A natural question arises: why do spectral methods help to cluster high-dimensional datasets? Some practitioners believe one reason is that the spectral method filters some noise from the highdimensional data [ARS + 04, SEK04, ZLWZ09, KAH19]. Simultaneously, many theory works also (partially) support this explanation [AFWZ20, EBW18, LZK22, MZ22a]. With this explanation in mind, people analyzed the behavior of spectral-based algorithms with noise perturbation. Based on these analyses, many algorithms were proposed to recover clusters in probabilistic generative models. Among them, a well-studied model is the signal-plus-noise model .

Signal-Plus-Noise model In this model, we assume that each observed sample ̂ v i has the form ̂ v i = v i + e i , where v i is a ground-truth vector and e i is a random noise vector. For any two sample vectors ̂ v i , ̂ v j , if they are from the same cluster, their corresponding ground-truth vectors are identical, i.e., v i = v j . Signal-plus-noise model is very general; it has plentiful variants with different types of ground-truth vectors and noise distribution. In this paper, we focus on an important instance known as

the stochastic block model (SBM). Though SBM is not as broad as general signal-plus-noise model, it usually serves as a benchmark for clustering and provides preliminary intuition about random graphs.

Stochastic block model The SBM is first introduced by [HLL83] and is widely used as a theoretical benchmark for graph clustering algorithms. In the paper, we focus on the symmetric version of stochastic block model (SSBM), described as follows. Given a set of n vertices V , we uniformly partition them into k disjoint sets (clusters), denoted by V 1 , . . . , V k . Based on this partition, a random (undirected) graph ̂ G = ( V, E ) is sampled in the following way: for all pairs of vertices u, v ∈ V , an edge { u, v } is added independently with probability p , if u, v ∈ V ℓ for some ℓ ; otherwise, an edge { u, v } is added independently with probability q .

We usually assume that p > q . The task is to recover the hidden partition V 1 , . . . , V k from the random graph ̂ G . We denote this model as SSBM ( V, n, k, p, q ) .

SBM as a signal-plus-noise model Though SBM was originally designed for graph clustering, we view it as a special form of vector clustering. Namely, given the adjacency matrix of a graph ̂ G ∈ { 0 , 1 } V × V , the columns of ̂ G form a set of n = | V | vectors. To see that SBM fits into the signal-plus-noise model, note that in the SBM, the adjacency matrix ̂ G ∈ { 0 , 1 } V × V can be viewed as a fixed matrix G plus a random noise, i.e., ̂ G = G + E , where G def = E [ ̂ G ] is the mean and E is a zero-mean random matrix. More precisely, in the case of SSBM,

where the random variables { E uv : u ≤ v } are independent and E vu = E uv for all u, v ∈ V . 1

1.1 Motivations: Analyzing Vanilla Spectral Algorithms

Since the seminal work by McSherry [McS01], many spectral-based algorithms have been proposed and studied in the SBM [GM05, Vu18, LR15, EBW18, Col19, AFWZ20, MZ22b] and even more general signal-plus-noise models [AFWZ20, EBW18, CTP19, LZK22, MZ22a]. These algorithms are largely based on the spectral analysis of random matrices. The purpose of designing and analyzing such algorithms is twofold.

Understanding the limitation of spectral-based algorithms SBM is specified by parameters, such as n, k, p, q in the symmetric case. Clustering is usually getting harder for larger k and smaller gap ( p -q ) . Many existing works aim to understand in which regimes of these parameters it is possible to recover the hidden partition. In this regard, the state-of-the-art bound is given by Vu [Vu18]. Concretely, [Vu18] proved that, in the symmetric setting, there is an algorithm that recovers

Understanding spectral-based algorithms in practice Besides analyzing spectral algorithms in theory, the other purpose, which is the primary purpose of this paper , is to explain the usefulness of such algorithms in practice. Indeed, as we mentioned before, many spectral-based algorithms, as observed in practice, can filter the noise and address the curse of dimensionality [ARS + 04, SEK04, ZLWZ09, KAH19]. Some representative algorithms are PCA and SVD. Furthermore, it has been observed that spectral algorithms used in practice, such as PCA or SVD, are usually very simple : they just project data points into some lower-dimension subspace, and no extra steps are conducted.

In stark contrast, most of the aforementioned theoretical algorithms have pre-processing or postprocessing steps. For example, the idea in [LR15] is that one first applies SVD, and then runs a variant of K-means to clean up the clustering; the main algorithm in [Vu18] partitions the graph into several parts and uses these parts in different ways. As noted in [Vu18], these extra steps are only for the purpose of theoretical analysis : From the perspective of algorithm design, these extra steps appear redundant. Later on, [AFWZ20] coined the phrase vanilla spectral algorithms to describe spectral algorithms that do not include any additional steps. Both [Vu18] and [AFWZ20] conjectured

that vanilla spectral algorithms are themselves good clustering algorithms. In practice, this is a widely-used heuristic; however, in theory, the analysis of vanilla spectral algorithms is not satisfactory due to the lack of techniques for analysis. We refer to [MZ22b] for a detailed discussion on barriers of the current analysis.

Whydo we study vanilla algorithms? Our main focus is particularly on vanilla spectral algorithms for two reasons:

Vanilla spectral algorithms are the most popular in practice-no extra steps are widely used. Plus, their performance seems good enough. The lack of theoretical analysis is mostly due to technical obstacles.

A vanilla spectral algorithm is often simple and is not specifically designed for theoretical models such as SBM. In contrast, some complicated algorithms use extra steps which are designed for SBM. These steps made the analysis of SBM go through (as commented by [Vu18]). Meanwhile, these extra steps exploit specific structures and may cause 'overfittings' on SBM, which makes these algorithms less powerful in practice.

The main purpose of this paper is to theoretically understand the power of practically successful vanilla spectral algorithms. To this end, we study SBM as a preliminary demonstration. We do not aim to design algorithms for SBM that outperforms existing algorithms.

1.2 Our Results

The contribution of this paper is twofold. On the one hand, we show that vanilla algorithms (alg. 1) is indeed a clustering algorithm in the SSBM for a wide range of parameters, breaking previous barrier on analyzing on only constant number of clusters. On the other hand, we provide a novel analysis on matrix perturbation with random noise. We discuss more details on this part in Section 1.4.

Recall that parameters of SBM is specified by SSBM ( V, n, k, p, q ) , where n = | V | . Let σ 2 = max { p (1 -p ) , q (1 -q ) } . Our main result is stated below.

Theorem 1.1. There exists a constant C > 0 with the following property. In the model SSBM ( V, n, k, p, q ) , if σ 2 ≥ C log n/n and n ≥ C · k ( √ kp · log 6 n + √ log n p -q ) 2 , then alg. 1 recovers all clusters with probability 1 -O ( n -1 ) .

Here we describe the vanilla-SVD algorithm in more detail. Algorithms in [McS01, Vu18, Col19] share a common idea: they both use SVD-based methods to find a clear-cut vector representation of vertices. That is, every node v ∈ V is associated with a vector ρ ( v ) , and we say a vector representation ρ is clear-cut if the following holds for some threshold ∆ : if u, v ∈ V ℓ for some ℓ , then ∥ ρ ( u ) -ρ ( v ) ∥ ≤ ∆ / 4 ; otherwise, ∥ ρ ( u ) -ρ ( v ) ∥ ≥ ∆ .

Once a clear-cut representation is found, the clustering task is easy: If the parameters n, k, p, q are all known, we can calculate ∆ and simply decide whether two vertices are in the same cluster based on their distance; in the case where ∆ is unknown, we need one more step. 2 Following [Vu18], we denote by ClusterByDistance an algorithm that recovers the partition from a clear-cut representation. One natural representation is obtained by SVD as follows. Let ̂ G ∈ { 0 , 1 } V × V be the adjacent matrix of the input graph, and let P ̂ G k be the orthogonal projection matrix onto the space spanned by the first k eigenvectors of ̂ G . Then set ρ ( u ) def = P ̂ G k ̂ G u , where ̂ G u is the column index by u ∈ V . This yields alg. 1, the vanilla-SVD algorithm.

1.3 Comparison with Existing Analysis for Vanilla Spectral Algorithms in the SBM

To the best of our knowledge, there are very few works on the analysis of vanilla spectral algorithms [AFWZ20, EBW18, PPV + 19]. All of them only apply to the case of k = O (1) . In this work, we obtain the first analysis for general parameters n, k, p, q , in the symmetric SBM setting.

Algorithm 1: Vanilla-SVD algorithm for graph clustering

1 Input: adjacent matrix ̂ G ∈ { 0 , 1 } V × V

2 Output: a partition of V

Compute ρ ( u ) def = P ̂ G k ̂ G u for each u ∈ V .

Run ClusterByDistance with representation ρ .

Davis-Kahan approaches To study spectral algorithms in signal-plus-noise models, a key step is to understand how random noise perturbs the eigenvectors of a matrix. A commonly-used technical ingredient is the Davis-Kahan sin Θ theorem (or its variant). However, this type of approach faces two challenges in the SBM.

Davis-Kahan leads to worst-case perturbation bounds. For perturbations caused by random noises, such as signal-plus-noise models, Davis-Kahan sin Θ theorem is sometimes suboptimal.

These sin Θ theorems only lead to bound on 2 -norm. However, in the SBM analysis, we may need (2 →∞ ) -norm bounds. See [CTP19] for more discussions.

Previous works such as [AFWZ20, EBW18, PPV + 19] mainly followed this approach. They proposed some novel ideas to (partially) address these two challenges, but only apply to the case of k = O (1) . In contrast, our approach, following the power-iteration-based analysis proposed by [MZ22b], completely avoids Davis-Kahan sin Θ theorem and can handle the case of k = ω (1) .

Comparison with [MZ22b] Inspired by power iteration methods, Mukherjee and Zhang [MZ22b] proposed a new approach to analyze the perturbation of random matrices. The idea is to approximate the eigenvectors of a matrix by its power. In fact, this method has been widely used in practice as a fast algorithm to approximate eigenvectors. However, there are two limitations of [MZ22b].

Their analysis requires a nice structure of the mean matrix, i.e., all large eigenvalues are more or less the same.

Their algorithm is not vanilla as it has a 'centering step'. Moreover, their algorithm requires the knowledge of parameters p, q, k , and particularly, the centering step alone requires the knowledge of q . In comparison, we only need to know k ; further, we can also guess k (by checking the number of large eigenvalues) and then make alg. 1 fully parameter-free.

To overcome these limitations, we introduce a novel 'polynomial approximation + entrywise analysis' method, which makes this analysis more robust and requires less structure. More details will be discussed in Section 1.4.

Regarding parameters in Theorem 1.1 The difference between the parameters in our results and those in Vu's paper is that we replaced σ by √ p log 6 n . If p < 0 . 9 , σ and √ p are equal up to constant, so our bound is essentially the same as [Vu18] except for the log 6 n factor. We believe the extra log 6 n factor can be improved by future works. This term stems from the new concentration inequality we used as a black box. A refined analysis of this concentration inequality may remove this factor. Here are two example settings of parameters that satisfy the conditions in Theorem 1.1:

q = Θ( log n n ) , p = Ω( k log 6 n √ n ) ;

p -q = Θ(1) and k = O ( √ n log 6 n ) .

1.4 Proof Outline and Technical Contributions

Let s u denote the size of the cluster to which u belongs. Assume for now that all V i 's are of size roughly n/k . Indeed, this happens with high probability inasmuch as the partition is uniformly sampled.

Our goal is to show that there exists some threshold ∆ > 0 such that for every u, v ∈ V : if u, v ∈ V ℓ for some ℓ , then ∥ ∥ ∥ P ̂ G k ̂ G u -P ̂ G k ̂ G v ∥ ∥ ∥ ≤ ∆ / 4 ; otherwise, ∥ ∥ ∥ P ̂ G k ̂ G u -P ̂ G k ̂ G v ∥ ∥ ∥ ≥ ∆ . Write ε ( u ) def = ∥ ∥ ∥ P ̂ G k ̂ G u -G u ∥ ∥ ∥ . Then ∣ ∣ ∣ ∥ ∥ ∥ P ̂ G k ̂ G u -P ̂ G k ̂ G v ∥ ∥ ∥ -∥ G v -G u ∥ ∣ ∣ ∣ ≤ ε ( u ) + ε ( v ) . Note that ∥ G v -G u ∥ = 0 if u, v ∈ V ℓ for some ℓ , otherwise, ∥ G v -G u ∥ = ( p -q ) · √ s u + s v > ( p -q ) √ n/k . Therefore, setting ∆ = 0 . 8( p -q ) √ n/k , it suffices to show that ε ( u ) ≤ 0 . 1( p -q ) √ n/k for every u ∈ V .

We decompose ε ( u ) into two terms:

We shall bound the two terms from above separately. Intuitively, the noise term is small means P ̂ G k reduces the noise, while the deviation term is small means P ̂ G k preserves the data.

Upper bound of the noise term It is known that P ̂ G k (resp., P G k ) can be write as a polynomial of ̂ G (resp., G ). By Weyl's inequality, the eigenvalues of ̂ G are not too far from those of G . Therefore, in our case, one can find a simple polynomial φ which only depends on G , such that φ ( ̂ G ) (resp., φ ( G ) ) is a good approximation of P ̂ G k (resp., P G k ); this is formalized in Lemma 3.2. Then we have the following decomposition: ∥ ∥ ∥ P ̂ G k E u ∥ ∥ ∥ ≤ 2 ∥ ∥ ∥ φ ( ̂ G ) E u ∥ ∥ ∥ ≤ 2 ∥ φ ( G ) E u ∥ +2 ∥ ∥ ∥ ( φ ( ̂ G ) -φ ( G ) ) E u ∥ ∥ ∥ , where the first inequality follows from Lemma 3.2, which roughly says φ ( ̂ G ) is a good approximation of P ̂ G k .

The first term, ∥ φ ( G ) E u ∥ , is small with high probability. To see this, we use Lemma 3.2 again: ∥ φ ( G ) E u ∥ ≤ 3 2 ∥ P G k E u ∥ . According to a known result (c.f. Proposition 2.4), ∥ P G k E u ∥ is small with high probability, largely because the projection P G k and the vector E u are independent.

The second term is the tricky part, and we draw on an entrywise analysis. Namely, we study every entry of ( φ ( ̂ G ) -φ ( G )) E u , using the new inequality from [MZ22b]. See Lemma 3.3 for more details.

The upper bound for the noise term is encapsulated in Lemma 3.4.

Upper bound of the deviation term The following argument is reminiscent of [Vu18]. Say u ∈ V ℓ . Note that Gχ ℓ = √ s u · G u where χ ℓ = 1 √ s u · 1 V ℓ is the normalized characteristic vector of V ℓ (i.e., 1 V ℓ ( v ) = 1 ⇐⇒ v ∈ V ℓ ). It follows that

where the second inequality holds because P ̂ G k ̂ G is the best k -rank approximation of ̂ G and rank( G ) = k , and in the third inequality, we use ∥ ∥ ∥ ( P ̂ G k -I ) ∥ ∥ ∥ 2 ≤ 1 , as P ̂ G k is a projection matrix. Therefore,

A typical result in random matrix theory (c.f. Proposition 2.3) states that with high probability, ∥ E ∥ 2 = O ( √ n ) . Combining Equation (2) and s u ≈ n/k , we get ∥ ∥ ∥ ( P ̂ G k -I ) G u ∥ ∥ ∥ = O ( √ k ) . And by our assumption on n , we have √ k = o (( p -q ) n/k ) = o (∆) .

Technical contribution The major novelty of our analysis is using the polynomial φ . [MZ22b] used a centering step to make the mean matrix nicely structured, while in our analysis, we used polynomial approximation to address this issue. Another difference is that in [MZ22b], the centering

step appears explicitly in the algorithm. By contrast, our polynomial approximation only appears in the analysis - the algorithm is vanilla.

As a byproduct, we developed new techniques for studying eigenspace perturbation , a typical topic in random matrix theory. Our high-level idea is 'polynomial approximation + entrywise analysis'. That is, we reduce the analysis of eigenspace perturbation to the analysis of a simple polynomial (of matrix) under perturbation. We have more tools to deal with the latter.

1.5 Discussion and Future Directions

In this paper, we studied the behavior of vanilla-SVD in the SSBM, a benchmark signal-plus-noise model widely studied in random matrix theory. We showed that vanilla-SVD indeed filters noise in the SSBM. In fact, our analysis technique, 'polynomial approximation + entrywise analysis', is not very limited to SSBM. A direct and interesting question yet to be answered is: Can our method be extended to prove that vanilla-SVD works in the general SBM where partitions are not uniformly sampled and edges appear with different probabilities? Moreover, our method may be useful for analyzing some other realistic, probabilistic models such as the factor model - a model which has been widely used in economics and model portfolio theory.

In the long term, it would be very interesting to understand the behavior of vanilla spectral algorithms on real data: 1) Why does it succeed in some applications? 2) How could we fix it if it has failed in other cases? A deeper understanding of vanilla spectral algorithms will provide guidelines for using them in many machine learning tasks.

2 Preliminaries

Notations Let 1 n denote the n -dimensional vector whose entries are all 1's, and let J n be the n × n matrix whose entries are all 1's. Let s u denote the size of the cluster to which u belongs. For a matrix A , A [ i ] denotes the row of A indexed by i , and A i denotes the column indexed by i ; λ i ( A ) is the i -th largest eigenvalue of A ; let P A k denote the orthogonal projection matrix onto the space spanned by the first k eigenvectors of A . For a vector x ∈ R n , ∥ x ∥ def = √ x 2 1 + · · · + x 2 n denotes the Euclidean norm.

Definition 2.1 (Matrix operator norms) . Let A ∈ R n × n . Define ∥ A ∥ 2 def = max ∥ x ∥ =1 ∥ Ax ∥ and ∥ A ∥ 2 →∞ def = max x : ∥ x ∥ =1 ∥ Ax ∥ ∞ .

Proposition 2.1 (e.g., [CTP19]) . For all matrices A,B ∈ R n × n , it holds that (1) ∥ A ∥ 2 →∞ = max i ∈ [ n ] ∥ A [ i ] ∥ ; (2) ∥ AB ∥ 2 →∞ ≤ ∥ A ∥ 2 →∞ ∥ B ∥ 2 .

Proposition 2.2 (Weyl's inequality) . For all A,E ∈ R n × n , we have | λ i ( A ) -λ i ( A + E ) | ≤ ∥ E ∥ 2 .

Proposition 2.3 (Norm of a random matrix [Vu18]) . There is a constant C 0 > 0 . Let E be a symmetric matrix whose upper diagonal entries e ij are independent random variables where e ij = 1 -p ij or -p ij with probabilities p ij and 1 -p ij respectively, where p ij ∈ [0 , 1] . Let σ 2 := max ij { p ij (1 -p ij ) } . If σ 2 ≥ C 0 log n/n , then Pr [ ∥ E ∥ 2 ≥ C 0 σn 1 / 2 ] ≤ n -3 .

Proposition 2.4 (Projection of a random vector, lemma 2.1 in [Vu18]) . There exists a constant C 1 such that the following holds. Let X = ( ξ 1 , . . . , ξ n ) be a random vector in R n whose coordinates ξ i are independent random variables with mean 0 and variance at most σ 2 ⩽ 1 . Assume furthermore that the ξ i are bounded by 1 in absolute value. Let H be a subspace of dimension d and let Π H ξ be the length of the orthogonal projection of ξ onto H . Then Pr [ Π H X ≥ σ √ d + C 1 √ log n ] ≤ n -3 .

Proposition 2.5. For a ∈ [0 , 2] and r ∈ N , if | a -1 | ≤ δ < 1 2 r , then | a r -1 | ≤ 2 rδ .

3 Analysis of Vanilla SVD Algorithm

Write s i def = | V i | . We say the partition V 1 , . . . , V k is balanced if ( 1 -1 16 log n ) n k ≤ s i ≤ ( 1 + 1 16 log n ) n k , ∀ i ∈ [ k ] . By Chernoff bound, the partition V 1 , . . . , V k is balanced with probability at least 1 -n -1 ; hence, we assume that the partition is balanced in the following argument. Since σ 2 ≥ C log n/n , the event ∥ E ∥ = O ( √ n ) holds with high probability (see Proposition 2.3).

Recall the decomposition into deviation term and noise term in Equation (1). We first state our upper bound of the deviation term, which readily follows from the argument in Section 1.4, and the complete proof is in Appendix B.

Lemma 3.1 (Upper bound of deviation term) . Let C 0 be the constant in Proposition 2.3. If the partition is balanced and n ≥ 10 4 · C 2 0 k 2 σ 2 ( p -q ) 2 , then with probability at least 1 -n -3 we have

Section 3.1 and Section 3.2 lead to an upper bound of the noise term, and Section 3.3 is the proof of main theorem.

3.1 An Approximation of P G k and P ̂ G k

In order to give some intuition on the choice of φ , we first analyze the spectrum of G , and the result is summed up in Theorem 3.1.

Without loss of generality, assume that s 1 ≥ s 2 ≥ · · · ≥ s k . It is easy to see that the eigenvalues of H are ( p -q ) s 1 , . . . , ( p -q ) s k , 0 . Viewing G as a rank-one perturbation of H , we have the following theorem that characterizes eigenvalues of G . Its proof, in Appendix C, readily follows from a theorem in [BNS79], which studies eigenvalues under rank-one perturbation.

Theorem 3.1. Write s i def = | V i | and assume that s 1 ≥ s 2 ≥ · · · ≥ s k . Define δ i def = λ i ( G ) -( p -q ) s i , then (1) δ i ≥ 0 and ∑ k i =1 δ i = nq ; (2) λ 1 ( G ) ≥ nq +( p -q ) n k , and hence ∑ k i =2 δ i ≤ ( p -q )( s 1 -n k ) .

The choice of the polynomial φ Let µ def = ( p -q ) n k , and let ψ ( t ) be the quadratic polynomial such that ψ ( λ 1 ( G )) = ψ ( µ ) = 1 , ψ (0) = 0 , i.e., ψ ( t ) def = -1 λ 1 ( G ) µ ( t -λ 1 ( G ))( t -µ ) + 1 def = At 2 + Bt, where A = -1 λ 1 ( G ) µ , B = 1 λ 1 ( G ) + 1 µ . Finally, let φ ( t ) def = ( ψ ( t )) r where r def = log n .

Here we give some intuition for the choice of φ . Let ̂ G = ∑ n i =1 ̂ λ i v i v ⊤ i be the spectral decomposition of ̂ G . Then φ ( ̂ G ) = ∑ n i =1 φ ( ̂ λ i ) v i v ⊤ i , P ̂ G k = ∑ k i =1 v i v ⊤ . The spectral decomposition of φ ( ̂ G ) -P ̂ G k is φ ( ̂ G ) -P ̂ G k = ∑ k i =1 ( φ ( ̂ λ i ) -1) v i v ⊤ + ∑ n i = k +1 φ ( λ i ) v i v ⊤ . Hence,

Recall that ̂ λ i -λ i ( G ) is bounded by Weyl's inequality. Plus, when the partition is balanced, Theorem 3.1 shows that the eigenvalues of G is nicely distributed: Except for λ 1 ( G ) , other eigenvalues are all close to µ . Hence, our choice of φ makes ∥ ∥ ∥ φ ( ̂ G ) -P ̂ G k ∥ ∥ ∥ 2 small, and thus φ ( ̂ G ) is a good approximation of P ̂ G k . Formally, we have the following lemma.

Lemma 3.2 (Polynomial approximation) . Assume that the partition is balanced and n ≥ 10 4 · C 2 0 · k 2 · p · log n ( p -q ) 2 , where C 0 is the constant in Proposition 2.3. Then with probability at least 1 -n -3 , it holds that for all x ∈ R n , 1 2 ∥ ∥ ∥ P ̂ G k x ∥ ∥ ∥ ≤ ∥ ∥ ∥ φ ( ̂ G ) x ∥ ∥ ∥ ≤ 3 2 ∥ ∥ ∥ P ̂ G k x ∥ ∥ ∥ + ∥ x ∥ /n log log n , and 1 2 ∥ P G k x ∥ ≤ ∥ φ ( G ) x ∥ ≤ 3 2 ∥ P G k x ∥ .

Proof. Let G = ∑ k i =1 λ i u i u ⊤ i (resp., ̂ G = ∑ n i =1 ̂ λ i v i v ⊤ i ) be the spectral decomposition of G (resp., ̂ G ). We shall use the following claim.

Claim 3.1. The following holds with probability 1 -n -3 (over the choice of E ): for every i ∈ [ k ] , ∣ ∣ ∣ φ ( ̂ λ i ) -1 ∣ ∣ ∣ < 1 2 , | φ ( λ i ) -1 | < 1 2 ; and for every i = k +1 , . . . , n , ∣ ∣ ∣ φ ( ̂ λ i ) ∣ ∣ ∣ < n -log log n .

Fix x ∈ R n . On the one hand, 1 2 ≤ φ ( ̂ λ i ) ≤ 3 2 , ∀ i ∈ [ k ] , and hence

which means ∥ ∥ ∥ φ ( ̂ G ) x ∥ ∥ ∥ ≥ 1 2 ∥ ∥ ∥ P ̂ G k x ∥ ∥ ∥ . On the other hand,

Since √ a + b ≤ √ a + √ b , we have ∥ ∥ ∥ φ ( ̂ G ) x ∥ ∥ ∥ ≤ 3 2 ∥ ∥ ∥ P ̂ G k x ∥ ∥ ∥ + ∥ x ∥ n log log n . This establishes the first part.

Note that ∥ φ ( G ) x ∥ = √ ∑ k i =1 φ ( λ i ) 2 ⟨ x, u i ⟩ 2 and we also have 1 2 ≤ φ ( λ i ) ≤ 3 2 , ∀ i ∈ [ k ] , and thus similar argument goes for G . This finishes the proof of Lemma 3.2.

It remains to prove Claim 3.1. The claim readily follows from the choice of φ and the fact that λ i , ̂ λ i are close. A complete proof can be found in Appendix C.

3.2 The Upper Bound of the Noise Term

According to Equation (1), in order to derive an upper bound of ∥ P G k E u ∥ , it remains to bound ∥ ∥ ∥ ( φ ( ̂ G ) -φ ( G ) ) E u ∥ ∥ ∥ from above. This is done by the following lemma.

Lemma 3.3. Let C 0 be the constant in Proposition 2.3. Assume that the partition is balanced and n ≥ (100 + C 0 ) 2 · k 2 · p · log 12 n ( p -q ) 2 . For every u ∈ V , it holds that

where C 2 def = 7 · 10 6 is a constant.

Combining Lemma 3.2 and Proposition 2.4, we get an upper bound of the noise term:

Lemma 3.4 (Upper bound of noise term) . Let C 0 be the constant in Proposition 2.3. Assume that n ≥ (100 + C 0 ) 2 · k 2 · p · log 12 n ( p -q ) 2 . Then with probability at least 1 -O ( n -1 ) , we have ∥ ∥ ∥ P ̂ G k E u ∥ ∥ ∥ ≤ C 3 ( √ kp log 2 n + √ log n ) for all u ∈ V , where C 3 is a constant.

The proof of Lemma 3.3 is deferred to Section 4. We use it to prove Lemma 3.4 here.

Proof of Lemma 3.4. It follows from Lemma 3.2 that

By Proposition 2.4, with high probability at least 1 -n -1 , ∥ P G k E u ∥ is bounded by σ √ k + C 1 √ log n , where C 1 is a universal constant. Meanwhile, by Lemma 3.3 and union bound over all u , with probability at least 1 -O ( n -1 ) , ∥ ∥ ∥ ( φ ( ̂ G ) -φ ( G ) ) E u ∥ ∥ ∥ ≤ 7 · 10 6 ( √ kp log 2 n +1 / log n ) for every u ∈ V . Therefore, with probability 1 -O ( n -1 ) , it holds that ∥ ∥ ∥ P ̂ G k E u ∥ ∥ ∥ ≤ 1 . 4 × 10 7 √ kp log 2 n + 3 σ √ k + 3 C 1 √ log n for all u ∈ V . Setting C 3 def = (1 . 4 × 10 7 + 3 + 3 C 1 ) , we have the desired result.

3.3 Putting It Together

Now we are well-equipped to prove Theorem 1.1.

Proof of Theorem 1.1. Let C def = (100+100 C 0 +100 C 3 ) 2 , where C 0 , C 3 are the constants in Proposition 2.3 and Lemma 3.4. By our assumption on n , we have ( p -q ) √ n/k > 100 C 3 ( √ kp log 6 n + √ log n ) . It is easy to verify n satisfies the conditions in Lemma 3.4 and Lemma 3.1.

Write ∆ def = 0 . 8( p -q ) √ n/k . We aim to show that for every u, v ∈ V : if u, v ∈ V ℓ for some ℓ , then ∥ ∥ ∥ P ̂ G k ̂ G u -P ̂ G k ̂ G v ∥ ∥ ∥ ≤ ∆ / 4 ; otherwise, ∥ ∥ ∥ P ̂ G k ̂ G u -P ̂ G k ̂ G v ∥ ∥ ∥ ≥ ∆ . Then by calling ClusterByDistance , alg. 1 recovers all large clusters correctly.

Let ε ( u ) def = ∥ ∥ ∥ P ̂ G k ̂ G u -G u ∥ ∥ ∥ . According to the argument in Section 1.4, it suffices to show that ε ( u ) ≤ 0 . 1( p -q ) √ n/k for all u ∈ V . We further decompose ε ( u ) into noise term and deviation term, i.e., ε ( u ) ≤ noise ( u ) + dev ( u ) , where noise ( u ) def = ∥ ∥ P ̂ G E u ∥ ∥ and dev ( u ) def = ∥ ∥ ( P ̂ G -I ) G u ∥ ∥ . By Lemma 3.4 and Lemma 3.1, with probability at least 1 -O ( n -1 ) , the following hold for all u ∈ V : (1) noise ( u ) ≤ C 3 ( √ kp log 2 n + √ log n ) ≤ 0 . 01( p -q ) √ n/k ; (2) dev ( u ) ≤ 0 . 04( p -q ) √ n/k . Therefore, with probability at least 1 -O ( n -1 ) , we indeed have ε ( u ) ≤ 0 . 1( p -q ) √ n/k, ∀ u ∈ V . This completes the proof.

4 Proof of Lemma 3.3: Entrywise Analysis

This section is dedicated to proving Lemma 3.3.

Since both ( φ ( ̂ G ) -φ ( G )) and E are symmetric, we have ∥ ∥ ∥ ( φ ( ̂ G ) -φ ( G )) E u ∥ ∥ ∥ ≤ ∥ ∥ ∥ E ( φ ( ̂ G ) -φ ( G )) ∥ ∥ ∥ 2 →∞ . The high-level idea is to write E ( φ ( ̂ G ) -φ ( G )) as a sum of matrices, where each matrix is of the form E t SQ such that ∥ Q ∥ 2 = O (1) . This way, we have ∥ E t SQ ∥ 2 →∞ ≤ ∥ E t S ∥ 2 →∞ · O (1) , and ∥ E t S ∥ 2 →∞ is bounded by a lemma from [MZ22b].

where the last step is a decomposition based on the first location of D in the product terms. And

. We bound the three terms respectively.

That is, E ( φ ( ̂ G ) -φ ( G )) = EM + ED r + EM ′ Here we first list some definitions and estimations of the quantities involved.

According to Proposition 2.3, with probability at least 1 -n -3 , we have ∥ E ∥ 2 ≤ C 0 σ √ n, where C 0 is a constant. In the following argument, we always assume this holds.

µ def = ( p -q ) n/k . By our assumption on n , we have µ ≥ (100 + C 0 ) √ np log 6 n .

A = -1 λ 1 ( G ) µ , B = ( 1 λ 1 ( G ) + 1 µ ) , r = log n ; λ 1 ( G ) > µ , and thus B ≤ 2 µ , | A | ≤ 1 µ 2 .

By Claim 3.1, ∥ F ∥ 2 ≤ 1 + 1 4 log n , ∥ ∥ ∥ ̂ F ∥ ∥ ∥ 2 ≤ 1 + 1 4 log n . By Proposition 2.5, ∥ F ∥ t 2 , ∥ ∥ ∥ ̂ F ∥ ∥ ∥ t 2 ≤ 2 , ∀ t ≤ log n .

Upper bound of ∥ EM ∥ 2 →∞ Note that ∥ EF t D ∥ 2 →∞ ≤ ∥ EF ∥ 2 →∞ ∥ F ∥ t -1 2 ∥ D ∥ 2 , and ∥ F ∥ t -1 2 ≤ 2 for all t ≤ r . Moreover, ∥ D ∥ 2 ≤ | A | (2 ∥ E ∥ 2 ∥ G ∥ 2 + ∥ E ∥ 2 2 ) + B ∥ E ∥ 2 ≤ 3 ∥ E ∥ 2 µ + ∥ E ∥ 2 2 µ 2 + ≤ 4 ∥ E ∥ 2 µ ≤ 4(log 6 n ) -1 < 1 log 3 n . And the following lemma gives an upper bound of ∥ EF ∥ 2 →∞ .

Therefore, by union bound, we have the following holds with probability at least 1 -n -1 :

Upper bound of ∥ ED r ∥ 2 →∞ Since ∥ D ∥ 2 < 1 log 3 n , we have

Lemma 4.2 (Upper bound of ∥ EM ′ ∥ 2 →∞ ) . With probability 1 -O ( n -2 ) (over the choice of E ), we have ∥ EM ′ ∥ 2 →∞ ≤ 6 C 2 √ kp log 2 n , where C 2 = 10 6 is a constant.

Finally, combining Equation (4), Equation (5), and the above lemma, we conclude that with probability at least 1 -O ( n -2 ) ,

Proofs of Lemma 4.1 and Lemma 4.2 are deferred to Appendix D.

Acknowledgments and Disclosure of Funding

We are grateful to anonymous NeurIPS reviewers for their helpful comments. This research is supported by NSF CAREER award 2141536.

References

[AFWZ20] Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Ann. Statist. , 48(3):1452-1474, 2020.

[ARS + 04] Paolo Antonelli, HE Revercomb, LA Sromovsky, WL Smith, RO Knuteson, DC Tobin, RK Garcia, HB Howell, H-L Huang, and FA Best. A principal component noise filter for high spectral resolution infrared measurements. Journal of Geophysical Research: Atmospheres , 109(D23), 2004.

[BNS79] James R. Bunch, Christopher P. Nielsen, and Danny C. Sorensen. Rank-one modification of the symmetric eigenproblem. Numer. Math. , 31(1):31-48, 1978/79.

[Col19] Sam Cole. Recovering nonuniform planted partitions via iterated projection. Linear Algebra and its Applications , 576:79-107, 2019.

[CTP19] Joshua Cape, Minh Tang, and Carey E Priebe. The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics. The Annals of Statistics , 47(5):2405-2439, 2019.

[EBW18] Justin Eldridge, Mikhail Belkin, and Yusu Wang. Unperturbed: spectral analysis beyond Davis-Kahan. In Algorithmic learning theory 2018 , volume 83 of Proc. Mach. Learn. Res. (PMLR) , page 38. Proceedings of Machine Learning Research PMLR, [place of publication not identified], 2018.

[GM05] Joachim Giesen and Dieter Mitsche. Reconstructing many partitions using spectral techniques. In International Symposium on Fundamentals of Computation Theory , pages 433-444. Springer, 2005.

[HLL83] Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks , 5(2):109-137, 1983.

[KAH19] Vladimir Yu Kiselev, Tallulah S Andrews, and Martin Hemberg. Challenges in unsupervised clustering of single-cell rna-seq data. Nature Reviews Genetics , 20(5):273-282, 2019.

[LR15] Jing Lei and Alessandro Rinaldo. Consistency of spectral clustering in stochastic block models. The Annals of Statistics , 43(1):215-237, 2015.

[LZK22] Boris Landa, Thomas TCK Zhang, and Yuval Kluger. Biwhitening reveals the rank of a count matrix. SIAM Journal on Mathematics of Data Science , 4(4):1420-1446, 2022.

[McS01] Frank McSherry. Spectral partitioning of random graphs. In 42nd IEEE Symposium on Foundations of Computer Science (Las Vegas, NV, 2001) , pages 529-537. IEEE Computer Soc., Los Alamitos, CA, 2001.

[MZ22a] Chandra Sekhar Mukherjee and Jiapeng Zhang. Compressibility: Power of pca in clustering problems beyond dimensionality reduction. arXiv preprint arXiv:2204.10888 , 2022.

[MZ22b] Chandra Sekhar Mukherjee and Jiapeng Zhang. Detecting hidden communities by power iterations with connections to vanilla spectral algorithms, 2022.

[PPV + 19] Carey E Priebe, Youngser Park, Joshua T Vogelstein, John M Conroy, Vince Lyzinski, Minh Tang, Avanti Athreya, Joshua Cape, and Eric Bridgeford. On a two-truths phenomenon in spectral graph clustering. Proceedings of the National Academy of Sciences , 116(13):5995-6000, 2019.

[SEK04] Michael Steinbach, Levent Ertöz, and Vipin Kumar. The challenges of clustering high dimensional data. In New directions in statistical physics , pages 273-309. Springer, 2004.

[Vu18] Van Vu. A simple SVD algorithm for finding hidden partitions. Combin. Probab. Comput. , 27(1):124-140, 2018.

[ZLWZ09] Lei Zhang, Rastislav Lukac, Xiaolin Wu, and David Zhang. Pca-based spatially adaptive denoising of cfa images for single-sensor digital cameras. IEEE transactions on image processing , 18(4):797-812, 2009.

A Useful inequalities

Proposition A.1 (Chernoff bound) . Let X 1 , . . . , X m be i.i.d random variables that can take values in { 0 , 1 } , with E [ X i ] ≤ p for 1 ≤ i ≤ m . Then it holds that

Proposition A.2 (Hoeffding bound) . Let X 1 , . . . , X m be independent random variables such that a i ≤ X 1 ≤ b i , and write S def = ∑ m i =1 X i . Then it holds that

Definition A.1. Let X be a Bernoulli random variable with parameter p , i.e., Pr [ X = 1] = p, Pr [ X = 0] = 1 -p . The random variable Y def = X -p = X -E [ X ] is called centered Bernoulli random variable with parameter p .

Proposition A.3 (Adapted from [MZ22b]) . Let S ∈ R n × n , and let E = ( ξ ij ) be an n × n symmetric random matrix, where

are independent, centered Bernoulli random variables with parameter at most α for all i, j . Suppose that every entry of S takes value in [ -β, β ] , and each column of S has at most γ non-zero entries. Then for every t ∈ [log n ] , it holds that

Remark A.1 . The parameter α is determined by E , which equals to p in our case. The above bound is particularly useful when β, γ are small, that is, we want the matrix S to have either small entries or sparse columns.

Proposition A.4 (Proposition 2.5 restated) . For a ∈ [0 , 2] and r ∈ N , if | a -1 | ≤ δ < 1 2 r , then | a r -1 | ≤ 2 rδ .

Proof. Let x = a -1 ∈ [ -δ, δ ] . If 0 ≤ a ≤ 1 , we have 1 ≥ a r = (1 + x ) r ≥ 1 + rx ≥ 1 -rδ . If 1 < a < 1 + 1 /r , then 0 < x < 1 /r and hence

B Bounding the Deviation Term

Proof of Lemma 3.1. Our assumption on n implies that ( p -q ) n/k > 100 C 0 σ √ n . By Proposition 2.3, with probability at least 1 -n -3 , we have

According to Equation (2) and s u ≥ n 2 k , we have

where

By union bound,

C Polynomial Approximation

The proof of Theorem 3.1 rely on the following result on rank-one pertuebation.

Proposition C.1 (Eigenvalues under rank-one perturbation, Theorem 1 in [BNS79]) . Let C = D + ρzz T , where D is diagonal, ∥ z ∥ 2 = 1 . Let d 1 ≥ d 2 ≥ · · · ≥ d n be the eigenvalues of D , and let ˜ d 1 ≥ ˜ d 2 ≥ · · · ≥ ˜ d n be the eigenvalues of C . Then

where ∑ n i =1 µ i = 1 and 0 ≤ µ i ≤ 1 .

Proof of Theorem 3.1. Let χ i ∈ { 0 , 1 } V be the indicator vector for V i , i.e., χ i ( u ) = 1 iff u ∈ V i . It is easy to see that the eigenvectors of H are 1 √ s 1 χ 1 , . . . , 1 √ s k χ k . Write U = ( 1 √ s 1 χ 1 , . . . , 1 √ s k χ k ) ∈ R V × V , D = diag(( p -q ) s 1 , . . . , ( p -q ) s k , 0 , . . . , 0) , then we have H = UDU ⊤ . Note that 1 n = U ( √ s 1 , . . . , √ s n ) ⊤ , and hence

where ρ = nq, z = 1 √ n ( √ s 1 , . . . , √ s n ) ⊤ . This means the eigenvalues of G are the same as those of D + ρzz ⊤ . Since ∥ z ∥ = 1 , Item 1 follow directly from Proposition C.1. To see Item 2, we use the Rayleigh quotient characterization of the largest eigenvalue:

where the last inequality follows from n k = 1 k ∑ k i =1 s i ≤ √ ∑ k i =1 s 2 i /k

,

Proof of Claim 3.1. The assumption on n in Lemma 3.2 implies that µ = ( p -q ) n/k ≥ 100 C 0 σ √ n · log n . By Weyl's inequality, we have

Meanwhile, by Theorem 3.1,

for i = 2 , 3 , . . . , k . Hence, write ε def = µ 6 log n , we have

First, ψ ( λ 1 ) = 1 according to the definition of ψ , and hence φ ( λ 1 ) = 1 . As for ̂ λ 1 ,

Consequently, | φ ( ̂ λ 1 ) -1 | < 2 r 4 log n ≤ 1 / 2 by Proposition 2.5.

Next, for a ∈ { λ 2 , . . . , λ k , ̂ λ 2 , . . . , ̂ λ k } , the argument is similar:

where the second inequality follows from 2 Aµ + B = 1 µ -1 λ 1 ≤ 1 µ . This yields | φ ( a ) -1 | ≤ 1 / 2 by Proposition 2.5.

Finally, for i ≥ k +1 , it holds that

D Bounding the Noise Term

Proof of Lemma 4.1. The lemma readily follows from the following entrywise bound and Chernoff bound.

Claim D.1 (Entries of ψ ( G ) ) . For every u, v ∈ X , if u, v ∈ V ℓ for some ℓ , then 0 ≤ F uv ≤ 5 k n ; otherwise, | F uv | ≤ 10 n .

We decompose F = F ′ + F ′′ , where F ′ is the intra-cluster part, i.e., F ′ uv = F uv if u, v ∈ V ℓ for some ℓ , and F ′ uv = 0 otherwise. Since for every column of F ′ v , its non-zero entries are identical and at most 5 k/n by the above claim. Hence, every entry of EF ′ equals to the sum of at most 2 n/k independent, centered Bernoulli variables with parameter p , scaled by some factor at most 5 k n . By Chernoff bound, Pr E [ | ( EF ′ ) uv | > 10 √ kp log n/n ] ≤ n -4 , ∀ u, v ∈ V , and we have Pr E [ ∥ EF ′ ∥ 2 →∞ ≤ 10 √ kp log n ] ≥ 1 -n -2 by union bound. Analogously, by Hoeffding bound, Pr E [ ∥ EF ′′ ∥ 2 →∞ ≤ 10 √ log n ] ≥ 1 -n -2 . Since ∥ EF ∥ 2 →∞ ≤ ∥ EF ′ ∥ 2 →∞ + ∥ EF ′′ ∥ 2 →∞ , the lemma follows from the above two inequalities and union bound.

Proof of Claim D.1. Write λ = λ 1 ( G ) and recall that (i) ( p -q ) s u ≤ 2 µ for all u (ii) nq + µ ≤ λ ≤ nq +( p -q ) s 1 < nq +2 µ , (iii) λ > p · µ . Assume that u, v ∈ V ℓ for some ℓ . Then

Since λ -nq ≥ µ , the numerator is at least

Because s u ≤ 2 n/k, λ ≥ nq +( p -q ) n/k , we have

2 qn/k + λ + µ -( p + q ) s u > 2 qn/k + nq +( p -q )2 n/k -( p + q )2 n/k = ( n -2 n/k ) q ≥ 0 , which means F uv ≥ 0 . Meawhile,

where the first term is at most 3 q λ ≤ 3 n by (ii); second term is at most 2( p -q ) µ ≤ 2 k n . Therefore, | F uv | ≤ 5 k n .

For the second part, assume that u, v are not in the same cluster. Then

Hence,

By (ii), the first term is at most 3 q λ < 3 n ; by (i), the second term is at most 4 q λ ≤ 4 n ; hence, | F uv | ≤ 10 n .

Upper bound of ∥ EM ′ ∥ 2 →∞ (Proof of Lemma 4.2) Write L def = A ( EG + GE ) , R def = AE 2 + BE . Then D t F = ( L + R ) t F = R t F + R t -1 LF + R t -2 LDF + · · · + RLD t -2 + LD t -1 F . It suffices to derive a good upper bound of ∥ E η L ∥ 2 →∞ and ∥ E η F ∥ 2 →∞ , as R w can be further expressed as sum of powers of E . This is done by the following lemma:

LemmaD.1. The following holds with probability 1 -O ( n -2 ) over the choice of E : for all η ≤ log n , it holds that

where C 2 def = 10 6 is an absolute constant.

Specifically,

Note that | A j B w -j | ≤ 2 w · µ -( w + j ) ≤ 2 w · (100 √ np log 6 n ) -( w + j ) . It follows that for every t ∈ [ r -1] ,

where the second inequality is by Lemma D.1, and the last step follows from Proposition 2.5. Similarly, for every t ∈ [ r -1] ,

where the second inequality follows from ∥ ∥ D t -1 -i F ∥ ∥ 2 ≤ 2 , and the third inequality is by Lemma D.1. In sum,

where in the last inequality we also use ∥ ∥ ∥ ̂ F ∥ ∥ ∥ t 2 ≤ 2 .

It remains to prove Lemma D.1; the proof draws on the entrywise bound in Proposition A.3.

Proof of Lemma D.1. Fix an η ≤ log n . Write s ∗ def = n/k for the ease of notation. Observe that E η L = A ( E η +1 G + E η GE ) = A ( E η +1 H + E η HE ) + Aq ( E η +1 J n + E η J n E ) . We can apply Proposition A.3 to E η +1 H and E η H , with α = p, β = p -q, γ = 2 s ∗ . That is, with probability at least 1 -O ( n -2 ) ,

Our assumption on n yields µ ≥ C √ np log 6 n ; moreover, | A | ( p -q ) ≤ p -q µ 2 ≤ 1 /s ∗ · 1 µ ≤ 1 /s ∗ · ( C √ np log 6 n ) -1 . Therefore,

Similarly, by applying Proposition A.3 to E j J n with α = p, β = 1 , γ = n , we have, with probability at least 1 -O ( n -2 ) ,

Since | Aq | = q λ · 1 µ ≤ 1 n · ( C √ n log 6 n ) -1 , we have

Combining Equation (7) and Equation (8), we have, with probability at least 1 -O ( n -2 ) , ∥ E η L ∥ 2 →∞ ≤ C 2 √ np/s ∗ (log n ) 5 η (100 √ np ) η -1 where C 2 def = 10 6 .

For the second part, we decompose F = F ′ + F ′′ , where F ′ is the intra-cluster part, i.e., F ′ uv = F uv if u, v ∈ V ℓ for some ℓ ; and F ′ uv = 0 otherwise. Equipped with Claim D.1, we can apply Proposition A.3 on E η F ′ (with α = p, β = 10 /s ∗ , γ = 2 s ∗ ), and E η F ′′ (with α = p, β = 5 n , γ = n ):

where the second inequality holds with probability at least 1 -O ( n -2 ) . The lemma follows from a union bound over all η ≤ log n .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the power of vanilla-SVD algorithm, algorithm without any pre-processing or post-trimming steps, in the symmetric stochastic block model and proves it recovers all clusters in the balanced case, which answers an open question in [Vu18].\n\nSTRENGTHS:\nThe main contribution of this work lies in its demonstration of the effectiveness of a truly vanilla SVD algorithm in recovering all clusters of the symmetric stochastic block model in the balanced case with high probability theoretically. Some noticeable distinctions from existing works include a truly vanilla SVD algorithm without any additional pre-processing steps and the ability of handling of cluster numbers $k = \\omega(1)$. Also, from a technical standpoint, the authors introduce a novel \"polynomial approximation + entrywise analysis\" approach, which simplifies the analysis of eigenspace perturbation by reducing it to the analysis of a simple polynomial under perturbation thus makes the analysis more robust and requires less structure.\n\nWEAKNESSES:\nThe presentation of this work could be improved. First, it might be better to explicitly mention in the Abstract and Section 1that the analysis is within the balanced case and not directly extended to other cases. Second, [MZ22b] also claims answering the open question in [Vu18] in the balanced case. The authors point out that the centered-SVD algorithm proposed in [MZ22b] is not truly vanilla, containing a pre-processing centering step which depends on the knowledge of $q$. But it would still be beneficial to compare the condition and bounds in [MZ22b] to have a more complete understanding. Third, adding more background information and necessary theoretical knowledge would make this work more accessible to a broader audience. Lastly, the analysis in Section 4 may be better positioned prior to Section 3.3, which already wraps up the entire analysis, or included within the supplementary material.\n\nOthers:\n- In Lemma 3.1, it should be $1 - n^{-3}$ rather than $1- n^3$.\n\nQUESTIONS:\nThis work argues that its analysis can handle the case of $k = \\omega(1)$ while [MZ22b] considers a stronger case of $k = \\omega(\\log n)$. Can the analysis in this work also handle the case of $k = \\omega(\\log n)$?\n\nLIMITATIONS:\nThere seems to be no discussion of limitations or potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to understand the behavior of spectral steps in clustering problems, this paper studies the power of vanilla-SVD algorithm in the SBM. This work shows that vanilla-SVD algorithm recovers all clusters correctly in the symmetric setting.\n\nSTRENGTHS:\n1. To theoretically understand the power of practically successful vanilla spectral algorithms, this paper studies SBM as a preliminary demonstration. Then, some proofs are presented for the proposed Theorem 1.1.\n2. This work shows that vanilla algorithms is indeed a clustering algorithm in SSBM for a wide range of parameters.\n3. The authors give another analysis on matrix perturbation with random noise.\n4. Detailed comparisons with existing analysis for vanilla spectral algorithms in SBM are presented in this paper.\n\nWEAKNESSES:\n1. In real applications, it is common in practice that spectral embeddings obtained by spectral clustering algorithm need a post-processing, e.g., k-means. It is reasonable that spectral embeddings themselves are clustering-friendly. Thus, the contributions of analyzing vanilla spectral methods could be highlighted.\n\n2. From Section 1.3, it can be observed that there is limited work on the vanilla spectral algorithm. Could the authors provide an explanation for the reasons behind analyzing the parameters in the SSBM setting in this paper?\n\n3. From the comparisons, it is apparent that there are some issues with the related theoretical analyses, such as [AFWZ20], [EBW18], [PPV+19], etc. Since these weaknesses are evident, could the authors consider providing quantitative experimental analysis in addition to the theoretical analysis?\n\n4. Between lines 48 and 49, '...can be view as a fixed matrix...' should be corrected to '...viewed...'.\n\n5. The coherence in the review content in the Introduction section is insufficient. The authors seem to focus more on the mathematical advancements rather than considering the theoretical and practical applications of spectral methods in the field of machine learning, especially the related work in both theoretical understandings and practical applications.\n\n6. The argument is verified using a specific setting in a typical model, SSBM,  to demonstrate that vanilla spectral algorithm is powerful and prectically successful. It may not be comprehensive enough.\n\nQUESTIONS:\n1. Experiments are suggested to conduct from the perspectives of practical applications. \n2. Considering this paper as purely theoretical work, then based on Weaknesses #3, it is recommended to provide quantitative experimental analysis to supplement and demonstrate the limitations of the theoretical analysis of the comparison methods.\n3. The arguments of this paper need verifying by more settings, not only in symmetric version of SBM.\n4. It is recommended to use reference citation format that complies with the guidelines specified by this conference. Additionally, it would be beneficial to gain a better understanding of the contributions of spectral methods in the field of machine learning or data mining.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper investigates the effectiveness of the vanilla-SVD algorithm in the stochastic block model (SBM) and demonstrates that it can accurately recover all clusters in the symmetric setting. The authors address an open question raised by Van Vu in the symmetric setting.

STRENGTHS:
1. The vanilla algorithms employed in this study are applicable to a wide range of parameters, surpassing previous limitations that only allowed analysis on a constant number of clusters.

2. They also provide a novel analysis on  matrix perturbation with random noise.

WEAKNESSES:
1. The paper lacks experimental evaluation to demonstrate the effectiveness of the proposed scheme.

2. The time and space complexity analysis are not compared with previous schemes, which could provide insights into the efficiency of the proposed approach.

3. Lots of proofs are omitted in the main paper (I understand that this is due to the space limit). Maybe the theory conferences like COLT is a better venue for this paper?

QUESTIONS:
1. What advantages does the SVD approach in this paper offer compared to the terminal version of the Johnson-Lindenstrauss Lemma mentioned in ``Optimal terminal dimensionality reduction in Euclidean space''?

2. Is the proposed  scheme  efficient for large $n$ and $k$? If $n$ and $k$ are large, can the algorithm handle them efficiently?

LIMITATIONS:
Overall, this paper provides some valuable contributions to the study of the vanilla-SVD algorithm in the SBM. However, addressing the weaknesses mentioned above, such as conducting experiments, comparing complexities, and improving readability, would enhance the quality and impact of the research.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript mention that the paper contributes by providing a theoretical understanding of the power of vanilla spectral algorithms in clustering problems, specifically in the stochastic block model (SBM).  It also presents a novel analysis of matrix perturbation with random noise.  These contributions suggest that the document offers new insights into the application of SVD in clustering problems, particularly in the context of SBM. It focuses on the stochastic block model (SBM), a benchmark for clustering, and treats it as a form of vector clustering. The manuscript proposes and analyzes a vanilla-SVD algorithm for graph clustering and demonstrates its effectiveness in SBM.\n\nSTRENGTHS:\nThe main topic of the manuscript is the analysis of the power of Singular Value Decomposition (SVD) in clustering problems. The manuscript discusses the use of dimensionality reduction techniques, specifically PCA and SVD, to improve clustering results in high-dimensional datasets. It explains that classical clustering algorithms like K-means may perform poorly in such datasets due to the curse of dimensionality. Spectral methods like PCA and SVD have been observed to significantly enhance clustering results. The manuscript explores the reasons behind this improvement, including filtering noise from high-dimensional data. It focuses on the stochastic block model (SBM), a benchmark for clustering, and treats it as a form of vector clustering. The manuscript proposes and analyzes a vanilla-SVD algorithm for graph clustering and demonstrates its effectiveness in SBM. The authors present their results, including a clustering algorithm and an analysis of matrix perturbation with random noise. They compare their approach with existing analysis methods and highlight the advantages of their approach. The document concludes with a proof outline and technical contributions, outlining the steps taken to analyze the power of vanilla spectral algorithms for clustering.\n\nWEAKNESSES:\nThere are many formulas that make the readability not very strong. Some experiments should be designed to evaluate the significance of the model.\n\nQUESTIONS:\n1] The paper focuses on the power of the vanilla-SVD algorithm in the stochastic block model (SBM).  It would be interesting to know if the algorithm's effectiveness is limited to the SBM or if it can be applied to other clustering problems as well.  Is there any generalization of its performance to different datasets? \n2] The authors mention that the lack of theoretical analysis for vanilla spectral algorithms is due to technical obstacles.  Could you elaborate on these obstacles and explain why the theoretical analysis is challenging?  What are the main difficulties in analyzing vanilla spectral algorithms?\n3] The authors state that the lack of theoretical analysis is partly due to the simplicity of vanilla spectral algorithms, which are not specifically designed for theoretical models like SBM.  How do these algorithms compare to more complicated algorithms designed for SBM in terms of performance and practicality?  Are there any trade-offs between simplicity and performance?\n4] The authors mention the potential application of vanilla spectral algorithms in real-world data, but there is no analysis or discussion on this topic.  Can you provide any insights into the behavior of these algorithms on real data?  How do they perform in different applications?  Are there any challenges or limitations when applying vanilla spectral algorithms in practice?\n\nLIMITATIONS:\nThis work should design some experiments to fully illustrate the application value of the model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides rigorous, theory-based evidence that vanilla spectral algorithms (i.e., methods that run SVD on the adjacency matrix without any further processing) succeed in finding many communities in symmetric stochastic block models. In contrast to Davis-Kahan approaches, the authors adopt an analysis similar to [MZ22b], which is inspired by power iteration. The key technical novelty of the authors' method is a new way to study eigenspace perturbation by using a polynomial approximation of the operator that projects a vector onto the space of the first k eigenvectors of the adjacency matrix.\n\nSTRENGTHS:\nWhile there are many community recovery algorithms for SBMs at this point, this paper provides an important analysis of vanilla algorithms for community recovery, showing that pre or post processing steps are unnecessary, thereby validating practical approaches. This is the key strength of the paper, in my view. \n\nA secondary strength, which is of significance to researchers working on random matrix theory, is the use of the polynomial approximation method for the projection operator. This interestingly allows one to circumvent usage of the Davis-Kahan theorem (which is the standard way to handle eigenspace perturbation), and I expect this technique will be useful in various other settings. \n\nThe paper is also very well written, with clear comparisons to related work, and key contributions highlighted.\n\nWEAKNESSES:\nThe main weakness I found was that there is little to no discussion of the condition under which Algorithm 1 recovers communities (Theorem 1.1). At face value, it seems like a similar condition as [Vu18], with a few changes, like $\\sigma \\sqrt{k}$ replaced by $\\sqrt{kp} \\log^6 n$. A few questions are:\n- Is this change an artifact of the proof, or is it something more fundamental? In most situations, I'd expect the $\\sqrt{\\log n}$ term to dominate. \n- What are commonly considered regimes for p, q, k in which this threshold is satisfied? For instance, it seems like the regime $p, q = \\Theta ( \\log (n) / n)$ is not covered by the theorem. What's the minimum scaling of $p, q$ that would work here? And what is the maximum number of communities that can be tolerated? Understanding such qualitative cases would make your results much more interpretable. In a similar vein, it was unclear to me whether the condition you have here is optimal in some sense, or if it's just to prove that vanilla methods succeed in just *some* regime.\n\nFinally, I have a question related to how your work compares to [MZ22b]. \n- It seems that your work and [MZ22b] tackle a similar parameter regime for community recovery. How do the achievability regions compare between their work and yours? In particular, does the vanilla algorithm work in almost the same parameter regime, or is the parameter regime more restricted?\n\nQUESTIONS:\n- I would mention in the abstract that your main contribution is on vanilla algorithms with a large number of communities. This would emphasize the difference between your work and that of [AFWZ20]. \n- What should the value of $\\Delta$ be in Algorithm 1?\n- If there is space, could you elaborate on the connections between your polynomial approach and the power iteration method of [MZ22b]? From a quick read of [MZ22b] it seems there may be some similar ideas and it would be valuable for readers in the field to understand those connections further.\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}