SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recovering Unbalanced Communities in the Stochastic Block Model with Application to Clustering with a Faulty Oracle*
Chandra Sekhar Mukherjee†  
chandrasekhar.mukherjee@usc.edu
Pan Peng ‡  
ppeng@ustc.edu.cn
Jiapeng Zhang†  
jiapengz@usc.edu
The stochastic block model (SBM) is a fundamental model for studying graph clustering or community detection in networks. It has received great attention in the last decade and the balanced case, i.e., assuming all clusters have large size, has been well studied. However, our understanding of SBM with unbalanced communities (arguably, more relevant in practice) is still limited. In this paper, we provide a simple SVD-based algorithm for recovering the communities in the SBM with communities of varying sizes. We improve upon a result of Ailon, Chen and Xu [ICML 2013; JMLR 2015] by removing the assumption that there is a large interval such that the sizes of clusters do not fall in, and also remove the dependency of the size of the recoverable clusters on the number of underlying clusters. We further complement our theoretical improvements with experimental comparisons. Under the planted clique conjecture, the size of the clusters that can be recovered by our algorithm is nearly optimal (up to poly-logarithmic factors) when the probability parameters are constant.
As a byproduct, we obtain an efficient clustering algorithm with sublinear query complexity in a faulty oracle model, which is capable of detecting all clusters larger than  $\tilde{\Omega}(\sqrt{n})$ , even in the presence of  $\Omega(n)$  small clusters in the graph. In contrast, previous efficient algorithms that use a sublinear number of queries are incapable of recovering any large clusters if there are more than  $\tilde{\Omega}(n^{2/5})$  small clusters.
Graph clustering (or community detection) is a fundamental problem in computer science and has wide applications in many domains, including biology, social science, and physics. Among others, the stochastic block model (SBM) is one of the most basic models for studying graph clustering, offering both a theoretical arena for rigorously analyzing the performance of different types of clustering algorithms, and synthetic benchmarks for evaluating these algorithms in practice. Since the 1980s (e.g., [19, 8, 15, 7]), there has been much progress towards the understanding of the statistical and computational tradeoffs for community detection in SBM with various parameter regimes. We refer to the recent survey [1] for a list of such results.
*Authors are in alphabetical order.
$^{\dagger}$ Thomas Lord Department of Computer Science, University of Southern California. Research supported by NSF CAREER award 2141536.
$^{\ddagger}$ School of Computer Science and Technology, University of Science and Technology of China. Research supported in part by NSFC grant 62272431 and "the Fundamental Research Funds for the Central Universities".
In this paper, we focus on a very basic version of the stochastic block model.
Definition 1.1 (The  $\mathrm{SBM}(n,k,p,q)$  model). In this model, given an  $n$ -vertex set  $V$  with a hidden partition  $V = \cup_{i=1}^{k} V_i$  such that  $V_i \cap V_j = \emptyset$  for all  $i \neq j$ , we say a graph  $G = (V,E)$  is sampled from  $\mathrm{SBM}(n,k,p,q)$ , if for all pairs of vertices  $v_i, v_j \in V$ , (1) an edge  $(v_i,v_j)$  is added independently with probability  $p$ , if  $v_i, v_j \in V_\ell$  for some  $\ell$ ; (2) an edge  $(v_i,v_j)$  is added independently with probability  $q$ , otherwise.
We are interested in the problem of fully recovering all or some of the clusters, given a graph  $G$  that is sampled from  $\mathrm{SBM}(n,k,p,q)$ . A cluster  $V_{i}$  is said to be fully recovered if the algorithm outputs a set  $S$  that is exactly  $V_{i}$ . Most of the previous algorithms on the full recovery of SBM either just work for the nearly balanced case (i.e., each cluster has size  $\Omega\left(\frac{n}{k}\right)$ ) when  $k$  is small, say  $k = o(\log n)$  (see e.g. [2]), or only work under the following assumption:
- All of the latent clusters are sufficiently large $^4$ , i.e., for each  $j$ ,  $|V_j| = \tilde{\Omega}(\sqrt{n})$  (see e.g., [25, 6, 10, 9, 1, 28, 11]).
From a practical perspective, many real-world graphs may have many communities of different sizes, that is, large and small clusters co-exist in these graphs. This motivates us to investigate how to recover the communities in SBM if the latent communities have very different sizes. In particular, we are interested in efficiently recovering all the large clusters in the presence of small clusters. However, such a task can be quite difficult, as those small clusters may be confused with noisy edges. Indeed, most previous algorithms try to find all the  $k$ -clusters in one shot, which always computes some structures/information of the graph that are sensitive to noise (and small clusters). For example, the classical SVD-based algorithms (e.g., [25, 28]) first compute the first  $k$  singular vectors of some matrix associated with the graph and then use these  $k$  vectors to find clusters. Such singular vectors are sensitive to edge insertions or deletions (e.g. [13]). In general, this difficulty was termed by Ailon et al. [3] as "small cluster barrier" for graph clustering.
To overcome such a barrier, Ailon et al. [3, 4] proposed an algorithm that recovers all large latent clusters in the presence of small clusters under the following assumptions (see [4]),
- none of the cluster sizes falls in the interval  $(\alpha / c, \alpha)$  for a number  $\alpha \sim \Theta\left(\frac{\sqrt{p(1 - q)n}}{p - q}\right)$  and  $c > 1$  is some universal constant;
- there exists a large cluster, say of size at least  $\Upsilon := \Theta\left(\max \left\{\frac{\sqrt{p(1 - q)n}}{p - q}, \frac{k \log n}{(p - q)^2}\right\}\right)$ .
The algorithm in [4] then has to exhaustively search for such a gap, and then apply a convex program-based algorithm to find a large cluster of size at least  $\Upsilon$ . As we discuss in the Appendix D, the assumption of the recoverable cluster being larger than  $\Omega(\sqrt{p(1 - q)n} / (p - q))$  is (relatively) natural as any polynomial time algorithm can only recover clusters of size  $\Omega(\sqrt{n})$ , under the planted clique conjecture. Still, two natural questions that remain are
1. Can we break the small cluster barrier without making the first assumption on the existence of a gap between the sizes of some clusters?
2. Can we remove the dependency of the size of the recoverable cluster on the number  $k$  of clusters? In particular, when  $k \gg \sqrt{n}$ , can we still recover a cluster of size  $\tilde{\Omega}(\sqrt{n})$ ?
The above questions are inherently related to the clustering problem under the faulty oracle model which was recently proposed by Mazumdar and Saha [23], as an instance from the faulty oracle model is exactly the graph that is sampled from SBM with corresponding parameters. Thus, it is natural to ask if one can advance the state-of-the-art algorithm for recovering large clusters for the graph instance from the faulty oracle model using an improved algorithm for the SBM?
We affirmatively answer all three questions mentioned above. Specifically, we demonstrate that clusters of size  $\tilde{\Omega}(\sqrt{n})$  can be successfully recovered in both the standard SBM and the faulty oracle model, regardless of the number of clusters present in the graph. This guarantee surpasses any previous achievements in related studies. The practical implications of this finding are significant
4The assumption is sometimes implicit. E.g., in [28], in their Theorem 1, the lower bound on their parameter  $\Delta$  implies a lower bound on the smallest cluster size.
since real-world networks often exhibit a substantial number of clusters (see e.g. [29]), varying in size from large to small.
1.1.1 Recovering large clusters in the SBM
We first provide a singular value decomposition (SVD) based algorithm, without assuming there is a gap between the sizes of some clusters, for recovering large latent clusters. Furthermore, the recoverability of the largest cluster is unaffected by the number of underlying clusters.
Theorem 1.2 (Recovering one large cluster). Let  $G$  be a graph that is generated from the  $SBM(n, k, p, q)$  with  $\sigma = \max \left( \sqrt{p(1 - p)}, \sqrt{q(1 - q)} \right)$ . If both of the following conditions are satisfied: (1) the size of the largest cluster, denoted by  $s_{\mathrm{max}}$ , is at least  $s^* := \frac{2^{13} \cdot \sqrt{p(1 - q) \cdot n} \cdot \log n}{(p - q)}$ ; (2)  $\sigma^2 = \Omega(\log n / n)$ . There exists a polynomial time algorithm that exactly recovers a cluster of size at least  $\frac{s_{\mathrm{max}}}{7}$  with probability  $1 - \frac{1}{n^2}$ .
We have the following remarks about Theorem 1.2. (1) By the assumption that  $\sigma^2 = \Omega (\log n / n)$ , we obtain that  $p = \Omega \left(\frac{\log n}{n}\right)$ , which further implies that the expected degrees are at least logarithmic in  $n$ . This is necessary as exact recovery in SBM requires the node degrees to be at least logarithmic even in the balanced case (i.e. when all the clusters have the same size; see e.g. [1]). (2) In contrast to the work [4], our algorithm breaks the small cluster barrier and improves upon the result of [4] in the following sense: we do not need to assume there is a large interval such that the sizes of clusters do not fall in, nor do our bounds get affected with increasing number of small clusters. (3) As a byproduct of Theorem 1.2, we give an algorithm that improves a result of [28] on partially recovering clusters in the SBM in the balanced case. We refer to Appendix C for details.
In addition, the tradeoff of the parameters in our algorithm in Theorem 1.2 is nearly optimal up to polylogarithmic factors for constant  $p$  and  $q$  under the planted clique conjecture (see Appendix D).
Recovering more clusters. We can apply the above algorithm to recover even more clusters, using a "peeling strategy" (see [3]). That is, we first recover the largest cluster (under the preconditions of Theorem 1.2), say  $V_{1}$ . Then we can remove  $V_{1}$  and all the edges incident to them and obtain the induced subgraph of  $G$  on the vertices  $V' := V \setminus \{V_{1}\}$ , denoting it as  $G'$ . Note that  $G'$  is a graph generated from  $\mathrm{SBM}(n', k - 1, p, q)$  where  $n' = n - |V_{1}|$ . Then we can invoke the previous algorithm on  $G'$  to find the largest cluster again. We can repeat the process until the we reach a point where the recovery conditions no longer hold on the residual graph. Formally, we introduce the following definition of prominent clusters.
Definition 1.3 (Prominent clusters). Let  $V_{1},\ldots ,V_{k}$  be the  $k$  latent clusters and  $s_1,\dots ,s_k$  be the size of the clusters. WLOG we assume  $s_1\geq \dots \geq s_k$ . Let  $k^{\prime}\geq 0$  be the smallest integer such that one of the following is true. (1)  $s_{k^{\prime} + 1} < \frac{2^{13}\cdot\sqrt{p(1 - q)}\sqrt{\sum_{i = k^{\prime} + 1}^{k}s_{i}}}{(p - q)}$ , (2)  $\sigma^2 < \log (\sum_{i = k' + 1}^k s_i) / (\sum_{i = k' + 1}^k s_i)$ . We call  $V_{1},\ldots ,V_{k^{\prime}}$  prominent clusters of  $V$ .
By the above definition, Theorem 1.2, and the aforementioned algorithm, which we call RECUR- SIVECLUSTER, we can efficiently recover all these prominent clusters.
Corollary 1.4 (Recovering all the prominent communities). Let  $G$  be a graph that is generated from the  $SBM(n, k, p, q)$  model. Then there exists a polynomial time algorithm RECURSIVECLUSTER that correctly recovers all the prominent clusters of  $G$ , with probability  $1 - o_{n}(1)$ .
Experimental Comparisons. We evaluate the performance of our algorithm in the simulation settings outlined in [4] and confirm its effectiveness. Moreover, the experiments conducted in [4] established that their gap constraint is an observable phenomenon. We demonstrate that our algorithm can accurately recover clusters even without this gap constraint. Specifically, we succeed in identifying large clusters in scenarios where there were  $\Omega(n)$  single-vertex clusters, a situation where the guarantees provided by [4] are inadequate. We observed that simpler spectral algorithms, such as [28], also failed to perform well in this scenario. Furthermore, we observe that the run-time of our algorithm is significantly faster than the SDP based approach of [3, 4]. Finally, we present empirical evidence of the efficacy of our techniques beyond their theoretical underpinnings.
1.1.2 An algorithm for clustering with a faulty oracle
We apply the above algorithm to give an improved algorithm for a clustering problem in a faulty oracle model, which was proposed by [23]. The model is defined as follows:
Definition 1.5. Given a set  $V = [n] \coloneqq \{1, \dots, n\}$  of  $n$  items which contains  $k$  latent clusters  $V_{1}, \dots, V_{k}$  such that  $\cup_{i} V_{i} = V$  and for any  $1 \leq i < j \leq k$ ,  $V_{i} \cap V_{j} = \emptyset$ . The clusters  $V_{1}, \dots, V_{k}$  are unknown. We wish to recover them by making pairwise queries to an oracle  $\mathcal{O}$ , which answers if the queried two vertices belong to the same cluster or not. This oracle gives correct answer with probability  $\frac{1}{2} + \frac{\delta}{2}$ , where  $\delta \in (0,1)$  is a bias parameter. It is assumed that repeating the same question to the oracle  $\mathcal{O}$ , it always returns the same answer<sup>5</sup>.
Our goal is to recover the latent clusters efficiently (i.e., within polynomial time) with high probability by making as few queries to the oracle  $\mathcal{O}$  as possible. One crucial limitation of all the previous polynomial-time algorithms ([23, 21, 27, 20, 14]) that make sublinear $^6$  number of queries is that they cannot recover large clusters, if there are at least  $\tilde{\Omega}(n^{2/5})$  small clusters. Now we present our result for the problem of clustering with a faulty oracle.
Theorem 1.6. In the faulty oracle model with parameters  $n, k, \delta$ , there exists a polynomial time algorithm NOSIYCLUSTERING(s), such that for any  $n \geq s \geq \frac{C \cdot \sqrt{n} \log^2 n}{\delta}$ , it recovers all clusters of size larger than  $s$  by making  $\mathcal{O}\left(\frac{n^4 \log^2 n}{\delta^4 \cdot s^4} + \frac{n^2 \log^2 n}{s \cdot \delta^2}\right)$  queries in the faulty oracle model.
We remark that our algorithm works without the knowledge of  $k$ , i.e., the number of clusters. Note that Theorem 1.6 says even if there are  $\Omega(n)$  small clusters, our efficient algorithm can still find all clusters of size larger than  $\Omega\left(\frac{\sqrt{n} \log n}{\delta}\right)$  with sublinear number of queries. We note that the size of clusters that our algorithm can recover is nearly optimal under the planted clique conjecture. Due to space constraints, all the missing algorithms, analyses, and proofs are deferred to Appendix E and F.
Now we describe our main idea for recovering the largest cluster in a graph  $G = (V, E)$  that is generated from  $\mathrm{SBM}(n, k, p, q)$ .
Previous SBM algorithms The starting point of our algorithm is a Singular Value Decomposition (SVD) based algorithm by [28], which in turn is built upon the seminal work of [25]. The main idea underlying this algorithm is as follows: Given the adjacency matrix  $A$  of  $G$ , project the columns of  $A$  to the space  $A_{k}$ , which is the subspace spanned by the first  $k$  left singular vectors of  $A_{k}$ . Then it is shown that for appropriately chosen parameters, the corresponding geometric representation of the vertices satisfies a separability condition. That is, there exists a number  $r > 0$  such that 1) vertices in the same cluster have a distance at most  $r$  from each other; 2) vertices from different clusters have a distance at least  $4r$  from each other. This is proven by showing that each projected point  $P_{\mathbf{u}}$  is close to its center, which is point  $\mathbf{u}$  corresponding to a column in the expected adjacency matrix  $\operatorname{E}[A]$ . There are exactly  $k$  centers corresponding to the  $k$  clusters. Then one can easily find the clusters according to the distances between the projected points.
The above SVD-based algorithm aims to find all the  $k$  clusters at once. Since the distance between two projected points depends on the sizes of the clusters they belong to, the parameter  $r$  is inherently related to the size  $s$  of the smallest cluster. Slightly more formally, in order to achieve the above separability condition, the work [28] requires that the minimum distance (which is roughly  $\sqrt{s} (p - q)$ ) between any two centers is at least  $\Omega (\sqrt{n / s})$ , which essentially leads to the requirement that the minimum cluster size is large, say  $\Omega (\sqrt{n})$ , in order to recover all the  $k$  clusters.
High-level idea of our algorithm In comparison to the work [28], we do not attempt to find all the  $k$  clusters at once. Instead, we focus on finding large clusters, one at a time. As in [28], we first project the vertices to points using the SVD. Then instead of directly finding the "perfect" clusters from the projected points, we first aim to find a set  $S$  that is somewhat close to a latent cluster that is large enough. Formally, we introduce the following definition of  $V_{i}$ -plural set.
Definition 1.7 (Plural set). We call a set  $S \subset V$  as a  $V_{i}$ -plural set if (1)  $|S \cap V_i| \geq 2^{13}\sqrt{n} \log n$ ; (2) For any  $V_{j} \neq V_{i}$  we have  $|S \cap V_j| \leq 0.1 \cdot |S \cap V_i|$ .
5This was known as persistent noise in the literature; see e.g. [17].
<sup>6</sup>Since there are  $\Theta(n^2)$  number of possible queries, by "sublinear" number of queries, we mean the number of queries made by the algorithm is  $o(n^2)$ .
That is, a plural set contains sufficiently many vertices from one cluster and much fewer vertices from any other cluster.
Recall that  $s^* \coloneqq \frac{C\sqrt{p(1 - q)\cdot n\cdot\log n}}{(p - q)}$  for  $C = 2^{13}$ , and  $s_{\max} \geq s^*$ . We will find a  $V_i$ -plural set for any cluster  $V_i$  that is large enough, i.e.,  $|V_i| \geq \frac{s_{\max}}{7}$ . To recover large clusters, our crucial observation is that it suffices to separate vertices of one large cluster from other large clusters, rather than trying to separate it from all the other clusters. This is done by setting an appropriate distance threshold  $L$  to separate points from any two different and large clusters. Then by refining Vu's analysis, we can show that for any  $u \in V_i$  with  $|V_i| \geq \frac{s_{\max}}{7}$ , the set  $S$  that consists of all vertices whose projected points belong to the ball surrounding  $u$  with radius  $L$  is a  $V_i$ -plural set, for some appropriately chosen  $L$ . It is highly non-trivial to find such a radius  $L$ . To do so, we carefully analyze the geometric properties of the projected points. In particular, we show that the distances between a point and its projection can be bounded in terms of the  $k'$ -th largest eigenvalue of the expected adjacency matrix of the graph (see Lemma 2.2), for a carefully chosen parameter  $k'$ . To bound this eigenvalue, we make use of the fact that  $A$  is a sum of many rank 1 matrices and Weyl's inequality (see Lemma 2.3). We refer to Section 2 for more details.
Now suppose that the  $V_{i}$ -plural set  $S$  is independent of the edges in  $V \times V$  (which is not true and we will show how to remedy this later). Then given  $S$ , we can run a statistical test to identify all the vertices in  $V_{i}$ . To do so, for any vertex  $v \in V$ , observe that the subgraph induced by  $S \cup \{v\}$  is also sampled from a stochastic block model. For each vertex  $v \in V_{i}$ , the expected number of its neighbors in  $S$  is
$$
p \cdot | S \cap V _ {i} | + q \cdot | S \setminus V _ {i} | = q | S | + (p - q) \cdot | S \cap V _ {i}.
$$
On the other hand, for each vertex  $u \in V_j$  for some different cluster  $V_j \neq V_i$ , the expected number of its neighbors in  $S$  is
$$
p \cdot | S \cap V _ {j} | + q \cdot | S \setminus V _ {j} | = q | S | + (p - q) \cdot | S \cap V _ {j} | \leq q | S | + (p - q) \cdot 0. 1 \cdot | S \cap V _ {i} |,
$$
since  $|S \cap V_j| \leq 0.1 \cdot |S \cap V_i|$  for any  $V_j \neq V_i$ . Hence there exists a  $\Theta((p - q) \cdot |S \cap V_i|)$  gap between them. Thus, as long as  $|S \cap V_i|$  is sufficiently large, with high probability, we can identify if a vertex belongs to  $V_i$  or not by counting the number of its neighbors in  $S$ .
To address the issue that the set  $S$  does depend on the edge set on  $V$ , we use a two-phase approach: that is, we first randomly partition  $V$  into two parts  $U, W$  (of roughly equal size), and then find a  $V_{i}$ -plural set  $S$  from  $U$ , then use the above statistical test to find all the vertices of  $V_{i}$  in  $W$  (i.e.,  $V \setminus U$ ), as described in IDENTIFYCLUSTER(S, W,  $\overline{s}$ ) (i.e. Algorithm 4).
Note that the output, say  $T_{1}$ , of this test is also  $V_{i}$ -plural set. Then we can find all vertices of  $V_{i}$  in  $U$  by running the statistical test again using  $T_{1}$  and  $U$ , i.e., invoking IDENTIFYCLUSTER( $T_{1}, U, \overline{s}$ ). Then the union of the outputs of these two tests gives us  $V_{i}$ . We note that there is correlation between  $T_{1}$  and  $U$ , which makes our analysis a bit more involved. We solve it by taking a union bound over a set of carefully defined bad events; see the proof of Lemma 2.7.
1.3 Other related work
In [11] (which improves upon [12]), the author also gave a clustering algorithm for SBM that recovers a cluster at a time, while the algorithm only works under the assumption that all latent clusters are of size  $\Omega(\sqrt{n})$ , thus they do not break the "small cluster barrier".
The model for clustering with a faulty oracle captures some applications in entity resolution (also known as the record linkage) problem [16, 24], the signed edges prediction problem in a social network [22, 26] and the correlation clustering problem [5]. A sequence of papers has studied the problem of query-efficient (and computationally efficient) algorithms for this model [23, 21, 27, 20, 14]. We refer to references [23, 21, 27] for more discussions of the motivations for this model.
2 The algorithm in the SBM
We start by giving a high-level view of our algorithm (i.e., Algorithm 1). Let  $G = (V, E)$  be a graph generated from  $\mathrm{SBM}(n, k, p, q)$ . For a vertex  $v$  and a set  $T \subset V$ , we let  $N_T(v)$  denote the number of neighbors of  $v$  in  $T$ .
We first preprocess (in Line 1) the graph  $G$  by invoking Algorithm 2 PREPROCESSING, which randomly partitions  $V$  into four subsets  $Y_{1}, Y_{2}, Z, W$  such that each vertex is added to  $Y_{1}, Y_{2}, Z, W$
with probability  $1/8, 1/8, 1/4, 1/2$ , respectively. Let  $Y = Y_{1} \cup Y_{2}, U = Y \cup Z$ . See Figure 1 for a visual presentation of the partition. Let  $\hat{A}$  (resp.  $\hat{B}$ ) be the bi-adjacency matrix between  $Y_{1}$  (resp.  $Y_{2}$ ) and  $Z$ . This part is to reduce the correlation between some random variables in the analysis, similar to in [25] and [28]. Then we invoke (in Line 2) Algorithm 3 ESTMATINGSIZE to estimate the size of the largest cluster. It first samples  $\sqrt{n} \log n$  vertices from  $Y_{2}$  and then counts their number of neighbors in  $W$ . These counters allow us to obtain a good approximation  $\overline{s}$  of  $s_{\max}$ .