SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle f _ {x}, f _ {y} \rangle = \langle f _ {x}, f _ {y} - \mu_ {j} \rangle + \langle f _ {x} - \mu_ {i}, \mu_ {j} \rangle + \langle \mu_ {i}, \mu_ {j} \rangle \\ \leq \left\| f _ {x} \right\| _ {2} \cdot \left\| f _ {y} - \mu_ {j} \right\| _ {2} + \left\| \mu_ {j} \right\| _ {2} \cdot \left\| f _ {x} - \mu_ {i} \right\| _ {2} + \langle \mu_ {i}, \mu_ {j} \rangle \\ \leq \frac {\sqrt {k}}{n} \cdot \left(\frac {\varepsilon}{\varphi^ {2}}\right) ^ {1 / 6} + \frac {\sqrt {2} k ^ {1 / 4}}{\sqrt {n}} \cdot \left(\frac {\varepsilon}{\varphi^ {2}}\right) ^ {1 / 3} \cdot \| \mu_ {j} \| _ {2} + \langle \mu_ {i}, \mu_ {j} \rangle \\ \leq \frac {\sqrt {k}}{n} \cdot \left(\frac {\varepsilon}{\varphi^ {2}}\right) ^ {1 / 6} + \frac {\sqrt {2} k ^ {1 / 4}}{\sqrt {n}} \cdot \left(\frac {\varepsilon}{\varphi^ {2}}\right) ^ {1 / 3} \cdot \sqrt {\left(1 + \frac {4 \sqrt {\varepsilon}}{\varphi}\right) \frac {1}{| C _ {j} |}} + \frac {8 \sqrt {\varepsilon}}{\varphi} \cdot \frac {1}{\sqrt {| C _ {i} | \cdot | C _ {j} |}}. \\ \end{array}
$$
The last inequality is according to item 1 and item 2 in Lemma D.7.
E Formal statement of Theorem 2 and proof
Theorem 2 (Formal; Robust against random edge deletions). Let  $k \geq 2$  be an integer,  $\varphi \in (0,1)$ . Let  $G_0 = (V,E_0)$  be a d-regular  $n$ -vertex graph that admits a  $(k,\varphi ,\varepsilon)$ -clustering  $C_1,\ldots ,C_k$ ,  $\frac{\varepsilon}{\varphi^4} \ll \frac{\gamma^3}{k^{\frac{9}{2}}\cdot\log^3k}$  and for all  $i \in [k]$ ,  $\gamma \frac{n}{k} \leq |C_i| \leq \frac{n}{\gamma k}$ , where  $\gamma$  is a constant that is in (0.001,1].
1. Let  $G$  be a graph obtained from  $G_0$  by deleting at most  $c$  edges in each cluster, where  $c$  is a constant. If  $0 \leq c \leq \frac{d\varphi^2}{2\sqrt{10}}$ , then there exists an algorithm that has query access to the adjacency list of  $G$  and constructs a clustering oracle in  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}(\frac{k \log n}{\gamma \varphi}))$  preprocessing time and takes  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}(\frac{k \log n}{\gamma}))$  space. Furthermore, with probability at least 0.95, the following hold:
1). Using the oracle, the algorithm can answer any WHICHCLUSTER query in  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \text{poly}(\frac{k \log n}{\gamma \varphi}))$  time and a WHICHCLUSTER query takes  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \text{poly}(\frac{k \log n}{\gamma}))$  space.
2). Let  $U_{i} \coloneqq \{x \in V : \text{WHICHCLUSTER}(G, x) = i\}, i \in [k]$  be the clusters recovered by the algorithm. There exists a permutation  $\pi : [k] \to [k]$  such that for all  $i \in [k]$ ,  $|U_{\pi(i)} \triangle C_i| \leq O(\frac{k^{\frac{3}{2}}}{\gamma} \cdot (\frac{\varepsilon}{\varphi^4})^{1/3}) |C_i|$ .
2. Let  $G$  be a graph obtained from  $G_0$  by randomly deleting at most  $O\left(\frac{kd^2}{\log k + d}\right)$  edges in  $G_0$ . With probability at least  $1 - \frac{1}{k^2}$ , then there exists an algorithm that has query access to the adjacency list of  $G$  and constructs a clustering oracle in  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}\left(\frac{k \log n}{\gamma \varphi}\right))$  preprocessing time and takes  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}\left(\frac{k \log n}{\gamma}\right))$  space. Furthermore, with probability at least 0.95, the following hold:
1). Using the oracle, the algorithm can answer any WHICHCLUSTER query in  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}(\frac{k \log n}{\gamma \varphi}))$  time and a WHICHCLUSTER query takes  $O(n^{1/2 + O(\varepsilon/\varphi^2)} \cdot \mathrm{poly}(\frac{k \log n}{\gamma}))$  space.
2). Let  $U_{i} \coloneqq \{x \in V : \text{WHICHCLUSTER}(G, x) = i\}$ ,  $i \in [k]$  be the clusters recovered by the algorithm. There exists a permutation  $\pi : [k] \to [k]$  such that for all  $i \in [k]$ ,
$$
| U _ {\pi (i)} \triangle C _ {i} | \leq O (\frac {k ^ {\frac {3}{2}}}{\gamma} \cdot (\frac {\varepsilon}{\varphi^ {4}}) ^ {1 / 3}) | C _ {i} |.
$$
To proof Theorem 2, we need the following lemmas.
Lemma E.1 (Cheeger's Inequality). In holds for any graph  $G$  that
$$
\frac {\lambda_ {2}}{2} \leq \phi (G) \leq \sqrt {2 \lambda_ {2}}.
$$
Lemma E.2 (Weyl's Inequality). Let  $A, B \in \mathbb{R}^{n \times n}$  be symmetric matrices. Let  $\alpha_{1}, \ldots, \alpha_{n}$  and  $\beta_{1}, \ldots, \beta_{n}$  be the eigenvalues of  $A$  and  $B$  respectively. Then for any  $i \in [n]$ , we have
$$
\left| \alpha_ {i} - \beta_ {i} \right| \leq \| A - B \|,
$$
where  $\| A - B \|$  is the spectral norm of  $A - B$ .
Proof of Theorem 2. Proof of item 1: For any  $d$ -bounded graph  $G' = (V, E)$ , we can get a  $d$ -regular graph  $G$  from  $G'$  by adding  $d - \deg(x)$  self-loops with weight  $1/2$  to each vertex  $x \in V$ . Then according to [7], the normalized Laplacian of  $G$  (denoted as  $L$ ) satisfies
$$
L (x, y) = \left\{ \begin{array}{l l} 1 - \frac {w _ {s e l f} (x)}{d} & \text {i f} \quad x = y \\ - \frac {1}{d} & \text {i f} \quad x \neq y \\ 0 & \text {o t h e r w i s e} \end{array} \right. \text {a n d} \quad (x, y) \in E  .
$$
Let  $G_0 = (V, E_0)$  be a  $d$ -regular graph that admits a  $(k, \varphi, \varepsilon)$ -clustering  $C_1, \ldots, C_k$ . Now we consider a cluster  $C_i, i \in [k]$ . Let  $C_i^0$  be a  $d$ -regular graph obtained by adding  $d - deg_i(x)$  self-loops to each vertex  $x \in C_i$ , where  $deg_i(x)$  is the degree of vertex  $x$  in the subgraph  $C_i$ ,  $i \in [k]$ . Let  $C_i^j$  be a graph obtained from  $C_i^{j-1}$  by: (1) randomly deleting one edge  $(u, v) \in E(C_i^{j-1})$ , where  $E(C_i^{j-1})$  is a set of edges that have both endpoints in  $C_i^{j-1}$ ; (2) turning the result subgraph in (1) be a  $d$ -regular graph,  $i \in [k], j \in [c]$ . Let  $L_i^j$  be the normalized Laplacian of  $C_i^j$ ,  $i \in [k], j = 0, 1, \ldots, c$ . Let  $H_i^j = L_i^j - L_i^{j-1}, i \in [k], j \in [c]$ . Then if  $u \neq v$ , we have
$$
H _ {i} ^ {j} (x, y) = \left\{ \begin{array}{l l} \frac {1}{d} & \text {i f} \quad x = u, y = v \quad \text {o r} \quad x = v, y = u \\ - \frac {1}{2 d} & \text {i f} \quad x = y = u \quad \text {o r} \quad x = y = v \\ 0 & \text {o t h e r w i s e} \end{array} \right.,
$$
and if  $u = v$ ,  $H_i^j$  is a all-zero matrix. Consider the fact that for a symmetric matrix, the spectral norm is less than or equal to its Frobenius norm, we will have  $\| H_i^j \| \leq \| H_i^j \|_F = \sqrt{2 \cdot \frac{1}{d^2} + 2 \cdot \frac{1}{4d^2}} = \sqrt{\frac{5}{2d^2}} = \frac{\sqrt{10}}{2d}$  for all  $i \in [k], j \in [c]$ . Let  $H_i = \sum_{j=1}^{c} H_i^j = L_i^c - L_i^0$ , we have  $\| H_i \| \leq \frac{\sqrt{10}}{2d} \cdot c, i \in [k]$ . Let  $\lambda_2(L_i^0)$  and  $\lambda_2(L_i^c)$  be the second smallest eigenvalue of  $L_i^0$  and  $L_i^c$  respectively. By Lemma E.2, we have  $|\lambda_2(L_i^c) - \lambda_2(L_i^0)| \leq \| H_i \| \leq \frac{\sqrt{10}}{2d} \cdot c$ , which gives us  $\lambda_2(L_i^c) \geq \lambda_2(L_i^0) - \frac{\sqrt{10}}{2d} \cdot c, i \in [k]$ . By Lemma E.1 and the precondition that  $c \leq \frac{d\varphi^2}{2\sqrt{10}}$ , we have  $\lambda_2(L_i^0) \geq \frac{\varphi^2}{2} \geq \frac{\sqrt{10}}{d} \cdot c$ . Therefore,
$$
\begin{array}{l} \lambda_ {2} \left(L _ {i} ^ {c}\right) \geq \lambda_ {2} \left(L _ {i} ^ {0}\right) - \frac {\sqrt {1 0}}{2 d} \cdot c \\ = \frac {1}{2} \lambda_ {2} \left(L _ {i} ^ {0}\right) + \frac {1}{2} \lambda_ {2} \left(L _ {i} ^ {0}\right) - \frac {\sqrt {1 0}}{2 d} \cdot c \\ \geq \frac {1}{2} \lambda_ {2} \left(L _ {i} ^ {0}\right) + \frac {\sqrt {1 0}}{2 d} \cdot c - \frac {\sqrt {1 0}}{2 d} \cdot c \\ = \frac {1}{2} \lambda_ {2} \left(L _ {i} ^ {0}\right). \\ \end{array}
$$
Again by Lemma E.1, for graph  $C_i^c$ , we have  $\phi_{\mathrm{in}}(C_i^c) = \phi (C_i^c)\geq \frac{1}{2}\lambda_2(L_i^c)\geq \frac{1}{4}\lambda_2(L_i^0)\geq \frac{1}{8}\varphi^2,i\in [k]$ . Note that we slightly abuse the notion  $C_i^c$ , for  $\phi (C_i^c)$ , we treat  $C_i^c$  as a  $d$ -regular graph, and
for  $\phi_{\mathrm{in}}(C_i^c)$ , we treat  $C_i^c$  as a cluster obtained by deleting  $c$  edges from  $E(C_i)$ . So, for a  $(k,\varphi ,\varepsilon)$ -clusterable graph  $G_0 = (V,E_0)$ , if we delete at most  $c\leq \frac{d\varphi^2}{2\sqrt{10}}$  edges in each cluster, then the resulting graph  $G$  is  $(k,\frac{1}{8}\varphi^{2},\varepsilon)$ -clusterable. Since  $\frac{\varepsilon}{\varphi^4}\ll \frac{\gamma^3}{k^{\frac{9}{2}}\cdot\log^3k}$ , we have  $\frac{\varepsilon}{(\frac{1}{8}\varphi^2)^2}\ll \frac{\gamma^3}{k^{\frac{9}{2}}\cdot\log^3k}$ . The statement of item 1 in this theorem follows from the same augments as those in the proof of Theorem 1 with parameter  $\varphi^{\prime} = \frac{1}{8}\varphi^{2}$  in  $G$ .
Proof of item 2: Let  $c = \frac{d\varphi^2}{2\sqrt{10}}$ . Since  $|C_i| \leq \frac{n}{\gamma k}$  for all  $i \in [k]$ , we have  $|E(C_i)| \leq \frac{n}{\gamma k} \cdot \frac{d}{2}$ , where  $E(C_i)$  is a set of edges that have both endpoints in  $C_i$ . So  $\frac{|E(C_i)|}{|E_0|} \leq \frac{\frac{n}{\gamma k} \cdot \frac{d}{2}}{\frac{nd}{2}} = \frac{1}{\gamma k}, i \in [k]$ . Since  $|C_i| \geq \frac{\gamma n}{k}$  and  $\phi_{\mathrm{out}}(C_i, V) \leq \varepsilon$ , we have  $E(C_i) \geq \frac{nd}{2k} (\gamma - \frac{\varepsilon}{\gamma})$ . So  $\frac{|E(C_i)|}{|E_0|} \geq \frac{\frac{nd}{2k} (\gamma - \frac{\varepsilon}{\gamma})}{\frac{nd}{2}} = \frac{1}{k} (\gamma - \frac{\varepsilon}{\gamma}), i \in [k]$ . Combining the above two results, we have  $\frac{1}{k} (\gamma - \frac{\varepsilon}{\gamma}) \leq \frac{|E(C_i)|}{|E_0|} \leq \frac{1}{\gamma k}$ .
In the following, we use  $X_{i}$  to denote the number of edges that are deleted from  $E(C_{i})$ . If we randomly delete  $\frac{kc^2\gamma(\gamma^2 - \varepsilon)}{9\log k + 2(\gamma^2 - \varepsilon)c} = O\left(\frac{kd^2}{\log k + d}\right)$  edges from  $G_0$ , then we have  $\frac{1}{k} (\gamma -\frac{\varepsilon}{\gamma})\cdot \frac{kc^2\gamma(\gamma^2 - \varepsilon)}{9\log k + 2(\gamma^2 - \varepsilon)c}\leq \mathrm{E}(X_i)\leq \frac{1}{\gamma k}\cdot \frac{kc^2\gamma(\gamma^2 - \varepsilon)}{9\log k + 2(\gamma^2 - \varepsilon)c}$ , which gives us
$$
\frac {c ^ {2} (\gamma^ {2} - \varepsilon) ^ {2}}{9 \log k + 2 (\gamma^ {2} - \varepsilon) c} \leq \operatorname {E} (X _ {i}) \leq \frac {c ^ {2} (\gamma^ {2} - \varepsilon)}{9 \log k + 2 (\gamma^ {2} - \varepsilon) c}.
$$
Chernoff-Hoeffding implies that  $\operatorname*{Pr}[X_i > (1 + \delta)\cdot \mathrm{E}(X_i)]\leq e^{\frac{-\mathrm{E}(X_i)\cdot\delta^2}{3}}$  . We set  $\delta = \frac{9\log k + 2(\gamma^2 - \varepsilon)c}{(\gamma^2 - \varepsilon)c} -$  1, then we have
$$
\begin{array}{l} \Pr \left[ X _ {i} > c \right] = \Pr \left[ X _ {i} > (1 + \delta) \cdot \frac {c ^ {2} (\gamma^ {2} - \varepsilon)}{9 \log k + 2 (\gamma^ {2} - \varepsilon) c} \right] \\ \leq \Pr \left[ X _ {i} > (1 + \delta) \cdot \mathrm {E} \left(X _ {i}\right) \right] \\ \leq e ^ {\frac {- \mathrm {E} (X _ {i}) \cdot \delta^ {2}}{3}} \\ \leq e ^ {- \frac {1}{3} \cdot \frac {c ^ {2} (\gamma^ {2} - \varepsilon) ^ {2}}{9 \log k + 2 (\gamma^ {2} - \varepsilon) c}} \cdot \delta^ {2} \\ \leq e ^ {- \frac {1}{3} \cdot \frac {c ^ {2} (\gamma^ {2} - \varepsilon) ^ {2}}{9 \log k + 2 (\gamma^ {2} - \varepsilon) c}} \cdot (\delta + 1) (\delta - 1) \\ = \frac {1}{k ^ {3}}. \\ \end{array}
$$
Using union bound, with probability at least  $1 - \frac{1}{k^3} \cdot k = 1 - \frac{1}{k^2}$ , we have that if we randomly delete  $\frac{k c^2 \gamma (\gamma^2 - \varepsilon)}{9 \log k + 2 (\gamma^2 - \varepsilon) c} = O\left(\frac{k d^2}{\log k + d}\right)$  edges from  $G_0$ , there is no cluster that is deleted more than  $c$  edges. Therefore, according to item 1 of Theorem 2, with probability at least  $1 - \frac{1}{k^3} \cdot k = 1 - \frac{1}{k^2}$ ,  $G$  is a  $(k, \frac{1}{8} \varphi^2, \varepsilon)$ -clusterable graph. The statement of item 2 in this theorem also follows from the same augments as those in the proof of Theorem 1 with parameter  $\varphi' = \frac{1}{8} \varphi^2$  in  $G$ .
Practical changes to our oracle. In order to implement our oracle, we need to make some modifications to the theoretical algorithms. To adapt the dot product oracle parameters (see Algorithm 7 and Algorithm 8 in Appendix C), i.e.,  $t$  (random walk length),  $s$  (sampling set size), and  $R_{init}$ ,  $R_{query}$  (number of random walks), we exploit the theoretical gap between intra-cluster and inter-cluster dot products in clusterable graphs. Given a clusterable graph  $G$ , by constructing the dot product oracle with various parameter settings and calculating some intra-cluster and inter-cluster dot products, we generate density graphs. The setting with the most prominent gap in the density graph is selected (see Figure 2 for an illustrative example).
Figure 2: For a random graph  $G$  generated by SBM with  $n = 3000$ ,  $k = 3$ ,  $p = 0.03$ ,  $q = 0.002$ , we build the dot product oracle for several different parameters for  $t, s, R_{init}$ ,  $R_{query}$  and plot the density graph. The setting with the most prominent gap in the density graph, i.e., the one on the right, is selected. We can further set  $\theta = 0.0005$  for  $G$  according to the right graph.
Determining the appropriate threshold  $\theta$  (at lines 2, 8, 9 of Alg.1 and line 3 of Alg.2) is the next step. By observing the density graph linked to the chosen dot product oracle parameters, we identify the fitting  $\theta$  (see Figure 2 for an illustrative example).
Determining the appropriate sampling set size  $s$  (at line 3 of Alg.1) of our oracle is the final step. Given a graph  $G = (V, E)$  generated by SBM, for all vertices in  $V$ , we know their ground-truth clusters. We can built our clustering oracle for several parameters for  $s$ . For each parameter setting, we run WHICHCLUSTER( $G, x$ ) for some  $x \in V$  and check if  $x$  was classified correctly. We pick the parameter setting with the most correct answers.
Query complexity. We conducted an experiment on a SBM graph with  $k = 3$ ,  $n = 15000$ ,  $q = 0.002$ ,  $p = 0.2$ . We calculate the fraction of edges that have been accessed given a number of invocations of WHICHCLUSTER( $G, x$ ) (Table 3). (Note that there is a trade-off between computational cost and clustering quality. Therefore, it is necessary to point out that the parameters of this experiment are set reasonably and the misclassification error is 0.) Table 3 shows that as long as the number of WHICHCLUSTER queries is not too large, our algorithm only reads a small portion of the input graph.
The above experiment shows that for a small target misclassification error, our algorithms only require a sublinear amount of data, which is often critical when analyzing large social networks, since one typically does not have access to the entire network.
Table 3: The fraction of accessed edges of queries
# queries	0	50	100	200	400	800	1600	3200
fraction	0.1277	0.2539	0.3637	0.5377	0.7517	0.9273	0.9929	0.9999
Running time. To evaluate the running time of our oracle, we conducted this experiment on some random graphs generated by SBM with  $n = 3000$ ,  $k = 3$ ,  $q = 0.002$  and  $p \in [0.02, 0.06]$ . Note that there is a trade-off between running time and clustering quality. In this experiment, we set the experimental parameters the same as those in the misclassification error experiment, which can ensure a small error. We recorded the running time of constructing a similarity graph  $H$  as construct-time. For each  $p$ , we query all the vertices in the input graph and recorded the average time of the  $n = 3000$  queries as query-time (Table 4).
Table 4: The average query time of our oracle
p	0.02	0.025	0.03	0.035	0.04	0.05	0.06
construct-time (s)	11.6533	12.4221	11.8358	11.6097	12.2473	12.2124	12.5851
query-time (s)	0.3504	0.4468	0.4446	0.4638	0.4650	0.4751	0.4874