SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We continue our analysis conditioning on the above events, happening with high probability. First, notice that for all  $l \in [k]$
$$
\max  _ {u, u ^ {\prime} \in \mathbf {B} _ {\ell}} \| u - u ^ {\prime} \| _ {2} ^ {2} \leqslant d ^ {*} \cdot b ^ {2} \leqslant 1 0 0 k ^ {- 2 5} \leqslant \frac {\Delta - C ^ {*} k ^ {- 2 5}}{k ^ {1 0}},
$$
and thus there cannot be  $\ell, \ell' \in [k]$  such that  $Q_{\Phi}(\mathcal{G}_{\ell}) \subseteq \mathbf{B}_{\ell}$  and  $Q_{\Phi}(\mathcal{G}_{\ell}') \subseteq \mathbf{B}_{\ell}$ . Moreover, by Corollary D.17 and
$$
\min  _ {\ell \in [ k ]} | \boldsymbol {G} _ {\ell} | \geqslant \frac {n ^ {0 . 0 1}}{k} \cdot \left(1 - \frac {1 5 0}{k ^ {4 7}}\right),
$$
$$
| \boldsymbol {S} \backslash \boldsymbol {\mathcal {G}} | \leqslant n ^ {0. 0 1} \cdot \frac {1 5 0}{k ^ {4 7}} = \frac {n ^ {0 . 0 1}}{k} \cdot \frac {1 5 0}{k ^ {4 6}}
$$
it must be that step 7 returned bins  $\mathbf{B}_1, \ldots, \mathbf{B}_k$ . This also implies that the algorithm does not reject. Further, by Lemma E.5 for all  $\bar{\nu}^{(i)}, \bar{\nu}^{(j)}$  such that  $\Phi \bar{\nu}^{(i)}, \Phi \bar{\nu}^{(j)} \in \mathbf{B}_l$  it holds that
$$
\left\| \bar {\boldsymbol {\nu}} ^ {(i)} - \bar {\boldsymbol {\nu}} ^ {(j)} \right\| _ {2} \leqslant C ^ {*} \cdot \left\| \Phi \bar {\boldsymbol {\nu}} ^ {(i)} - \Phi \bar {\boldsymbol {\nu}} ^ {(j)} \right\| _ {2} \leqslant C ^ {*} \cdot \sqrt {d ^ {*}} \cdot b \leqslant O (k ^ {- 1 2}).
$$
And hence, by triangle inequality, we get
$$
\left\| \bar {\boldsymbol {\nu}} ^ {(i)} - \mu_ {l} \right\| _ {2} \leqslant O (k ^ {- 1 2}).
$$
Finally, recall that for each  $\ell \in [k]$
where  $\mathbf{w}' \sim N\left(0, N\left(0, 32 \cdot k^{-120} \cdot \frac{\log(2kn / \delta)}{\varepsilon^2} \cdot \mathrm{Id}\right)\right)$ . Since by choice of  $n, k, \varepsilon$  it holds that
$$
3 2 \cdot k ^ {- 1 2 0} \cdot \frac {\log (2 k n / \delta)}{\varepsilon^ {2}} \leqslant O \left(k ^ {- 9 0}\right),
$$
we get with probability at least  $1 - e^{-k^{\Omega(1)}}$  for each  $\ell \in [k]$ , by Fact E.1, with  $m = 1$ , and a union bound that
$$
\left\| \mathbf {w} ^ {\prime} \right\| \leqslant O \left(k ^ {- 2 0}\right).
$$
Since all  $\bar{\pmb{\nu}}^{(i)}$  such that  $\Phi \bar{\pmb{\nu}}^{(i)}\in \mathbf{B}_l$  are at most  $O(k^{-12})$ -far from  $\mu_{l}$ , also their average is. We conclude that
$$
\left\| \hat {\mu} _ {\ell} - \mu_ {l} \right\| _ {2} \leqslant O \left(k ^ {- 1 2}\right) + \left\| \mathbf {w} \right\| _ {2} \leqslant O \left(k ^ {- 1 2}\right).
$$
Now Theorem D.3 is a trivial consequence.
Proof of Theorem D.3. The error guarantees and privacy guarantees immediately follows combining Lemma D.8, Lemma D.15, Lemma D.14 and Lemma D.18. The running time follows by Fact A.16.  $\square$
E Concentration inequalities
We introduce here several useful and standard concentration inequalities.
Fact E.1 (Concentration of spectral norm of Gaussian matrices). Let  $\mathbf{W} \sim \mathcal{N}(0, 1)^{m \times n}$ . Then for any  $t$ , we have
$$
\mathbb {P} \left(\sqrt {m} - \sqrt {n} - t \leqslant \sigma_ {\min } (\mathbf {W}) \leqslant \sigma_ {\max } (\mathbf {W}) \leqslant \sqrt {m} + \sqrt {n} + t\right) \geqslant 1 - 2 \exp \left(- \frac {t ^ {2}}{2}\right),
$$
where  $\sigma_{\mathrm{min}}(\cdot)$  and  $\sigma_{\mathrm{max}}(\cdot)$  denote the minimum and the maximum singular values of a matrix, respectively.
Let  $\mathbf{W}'$  be an  $n$ -by- $n$  symmetric matrix with independent entries sampled from  $N(0, \sigma^2)$ . Then  $\|\mathbf{W}'\| \leqslant 3\sigma \sqrt{n}$  with probability at least  $1 - \exp(-\Omega(n))$ .
Fact E.2 (Maximum degree of Erdős-Rényi graphs). Let  $G$  be an Erdős-Rényi graph on  $n$  vertices with edge probability  $p$ . Then with probability at least  $1 - n \exp(-np/3)$ , any vertex in  $G$  has degree at most  $2np$ .
Fact E.3 (Gaussian concentration bounds). Let  $\mathbf{X} \sim \mathcal{N}(0, \sigma^2)$ . Then for any  $t \geqslant 0$ ,
$$
\max  \left\{\mathbb {P} \left(\mathbf {X} \geqslant t\right), \mathbb {P} \left(\mathbf {X} \leqslant - t\right) \right\} \leqslant \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right).
$$
Fact E.4 (Chernoff bound). Let  $\mathbf{X_1},\ldots ,\mathbf{X_n}$  be independent random variables taking values in  $\{0,1\}$ . Let  $\mathbf{X} := \sum_{i = 1}^{n}\mathbf{X_i}$  and let  $\mu \coloneqq \mathbb{E}\mathbf{X}$ . Then for any  $\delta >0$
$$
\mathbb {P} \left(\mathbf {X} \leqslant (1 - \delta) \mu\right) \leqslant \exp \left(- \frac {\delta^ {2} \mu}{2}\right),
$$
$$
\mathbb {P} \left(\mathbf {X} \geqslant (1 + \delta) \mu\right) \leqslant \exp \left(- \frac {\delta^ {2} \mu}{2 + \delta}\right).
$$
Lemma E.5 ([31]). Let  $\Phi$  be a  $d$ -by- $n$  Gaussian matrix, with each entry independently chosen from  $N(0,1/d)$ . Then, for every vector  $u \in \mathbb{R}^n$  and every  $\alpha \in (0,1)$
$$
\mathbb {P} \left(\| \Phi u \| = (1 \pm \alpha) \| u \|\right) \geqslant 1 - e ^ {- \Omega (\alpha^ {2} d)}.
$$
F Linear algebra
Lemma F.1 (Weyl's inequality). Let  $A$  and  $B$  be symmetric matrices. Let  $R = A - B$ . Let  $\alpha_{1} \geqslant \dots \geqslant \alpha_{n}$  be the eigenvalues of  $A$ . Let  $\beta_{1} \geqslant \dots \geqslant \beta_{n}$  be the eigenvalues of  $B$ . Then for each  $i \in [n]$ ,
$$
\left| \alpha_ {i} - \beta_ {i} \right| \leqslant \| R \|.
$$
Lemma F.2 (Davis-Kahan's theorem). Let  $A$  and  $B$  be symmetric matrices. Let  $R = A - B$ . Let  $\alpha_{1} \geqslant \dots \geqslant \alpha_{n}$  be the eigenvalues of  $A$  with corresponding eigenvectors  $v_{1},\ldots ,v_{n}$ . Let  $\beta_{1} \geqslant \dots \geqslant \beta_{n}$  be the eigenvalues of  $B$  with corresponding eigenvectors  $u_{1},\ldots ,u_{n}$ . Let  $\theta_{i}$  be the angle between  $\pm v_{i}$  and  $\pm u_{i}$ . Then for each  $i \in [n]$ ,
$$
\sin \left(2 \theta_ {i}\right) \leqslant \frac {2 \| R \|}{\min  _ {j \neq i} \left| \alpha_ {i} - \alpha_ {j} \right|}.
$$
G Convex optimization
Proposition G.1. Let  $f: \mathbb{R}^m \to \mathbb{R}$  be a convex function. Let  $\mathcal{K} \subseteq \mathbb{R}^m$  be a convex set. Then  $y^{*} \in \mathcal{K}$  is a minimizer of  $f$  over  $\mathcal{K}$  if and only if there exists a subgradient  $g \in \partial f(y^{*})$  such that
$$
\langle y - y ^ {*}, g \rangle \geqslant 0 \quad \forall y \in \mathcal {K}.
$$
Proof. Define indicator function
$$
I _ {\mathcal {K}} (y) = \left\{ \begin{array}{l l} 0, & y \in \mathcal {K}, \\ \infty , & y \notin \mathcal {K}. \end{array} \right.
$$
Then for  $y \in \mathcal{K}$ , one has
$$
\partial I _ {\mathcal {K}} (y) = \left\{g \in \mathbb {R} ^ {m}: \langle g, y - y ^ {\prime} \rangle \geqslant 0 \forall y ^ {\prime} \in \mathcal {K} \right\}.
$$
Note  $y^{*}$  is a minimizer of  $f$  over  $\mathcal{K}$ , if and only if  $y^{*}$  is a minimizer of  $f + I_{\mathcal{K}}$  over  $\mathbb{R}^m$ , if and only if  $\mathbf{0}_m \in \partial (f + I_{\mathcal{K}})(y^*) = \partial f(y^*) + \partial I_{\mathcal{K}}(y^*)$ , if and only if there exists  $g \in \partial f(y^{*})$  such that  $\langle g, y - y^{*} \rangle \geqslant 0$  for any  $y \in \mathcal{K}$ .
Proposition G.2 (Pythagorean theorem from strong convexity). Let  $f: \mathbb{R}^m \to \mathbb{R}$  be a convex function. Let  $\mathcal{K} \subseteq \mathbb{R}^m$  be a convex set. Suppose  $f$  is  $\kappa$ -strongly convex over  $\mathcal{K}$ . Let  $x^* \in \mathcal{K}$  be a minimizer of  $f$  over  $\mathcal{K}$ . Then for any  $x \in \mathcal{K}$ , one has
$$
\left\| x - x ^ {*} \right\| ^ {2} \leqslant \frac {2}{\kappa} (f (x) - f (x ^ {*})).
$$
Proof. By strong convexity, for any subgradient  $g \in \partial f(x^{*})$  one has
$$
f (x) \geqslant f \left(x ^ {*}\right) + \langle x - x ^ {*}, g \rangle + \frac {\kappa}{2} \| x - x ^ {*} \| ^ {2}.
$$
By Proposition G.1,  $\langle x - x^{*},g\rangle \geqslant 0$  for some  $g\in \partial f(x^{*})$ . Then the result follows.
H Deferred proofs SBM
We prove Lemma C.9 restated below.
Lemma H.1 (Restatement of Lemma C.9). Consider the settings of Lemma C.8. With probability  $1 - \exp(-\Omega(n))$  over  $\mathbf{G} \sim \mathrm{SBM}_n(\gamma, d, x)$ ,
$$
\left\| \hat {X} (Y (\mathbf {G})) - \frac {1}{n} x x ^ {\top} \right\| _ {F} ^ {2} \leqslant \frac {8 0 0}{\gamma \sqrt {d}}.
$$
Proof. Recall  $\mathcal{K} = \{X\in \mathbb{R}^{n\times n}:X\succeq 0,X_{ii} = 1 / n\forall i\}$ . Let  $X^{*}\coloneqq \frac{1}{n} xx^{\top}$ . Since  $\hat{X} = \hat{X} (Y(\mathbf{G}))$  is a minimizer of  $\min_{X\in \mathcal{K}}\| Y(\mathbf{G}) - X\| _F^2$  and  $X^{*}\in \mathcal{K}$ , we have
$$
\left\| \hat {X} - Y (\mathbf {G}) \right\| _ {F} ^ {2} \leqslant \| X ^ {*} - Y (\mathbf {G}) \| _ {F} ^ {2} \iff \left\| \hat {X} - X ^ {*} \right\| _ {F} ^ {2} \leqslant 2 \left\langle \hat {X} - X ^ {*}, Y (\mathbf {G}) - X ^ {*} \right\rangle .
$$
The infinity-to-one norm of a matrix  $M \in \mathbb{R}^{m \times n}$  is defined as
$$
\left\| M \right\| _ {\infty \rightarrow 1} := \max  \left\{\langle u, M v \rangle : u \in \{\pm 1 \} ^ {m}, v \in \{\pm 1 \} ^ {n} \right\}.
$$
By [28, Fact 3.2], every  $Z \in \mathcal{K}$  satisfies
$$
| \langle Z, Y (\mathbf {G}) - X ^ {*} \rangle | \leqslant \frac {K _ {G}}{n} \cdot \| Y (\mathbf {G}) - X ^ {*} \| _ {\infty \rightarrow 1},
$$
where  $K_{G} \leqslant 1.783$  is Grothendieck's constant. Similar to the proof of [28, Lemma 4.1], using Bernstein's inequality and union bound, we can show (cf. Fact H.2)
$$
\left\| Y (\mathbf {G}) - X ^ {*} \right\| _ {\infty \rightarrow 1} \leqslant \frac {1 0 0 n}{\gamma \sqrt {d}}
$$
with probability  $1 - \exp (-\Omega (n))$ . Putting things together, we have
$$
\left\| \hat {X} (Y (\mathbf {G})) - \frac {1}{n} x x ^ {\top} \right\| _ {F} ^ {2} \leqslant \frac {4 0 0 \cdot K _ {G}}{\gamma \sqrt {d}},
$$
Fact H.2. Let  $\gamma > 0, d \in \mathbb{N}, x^{*} \in \{\pm 1\}^{n}$ , and  $\mathbf{G} \sim \mathrm{SBM}(\gamma, d, x^{*})$ . Let  $Y(\mathbf{G}) = \frac{1}{\gamma^d} \left(A(\mathbf{G}) - \frac{d}{n} J\right)$ , where  $A((G))$  is the adjacency matrix of  $(G)$  with entries  $d/n$  on the diagonal. Then
$$
\max  _ {x \in \{\pm 1 \} ^ {n}} \left| x ^ {\top} \left(Y (\mathbf {G}) - \frac {1}{n} x ^ {*} (x ^ {*}) ^ {\top}\right) x \right| \leqslant \frac {1 0 0 n}{\gamma \sqrt {d}}
$$
with probability at least  $1 - e^{-10n}$ .
Proof. The result will follow using Bernstein's Inequality and a union bound. Define  $\pmb{E} \coloneqq Y(\mathbf{G}) - \frac{1}{n} x^{*}(x^{*})^{\top}$ . Fix  $x \in \{\pm 1\}^{n}$  and for  $1 \leqslant i < j \leqslant n$ , let  $Z_{i,j} \coloneqq E_{i,j}x_{i}x_{j}$ . Then  $x^{\top}\pmb{Ex} = 2\sum_{1 \leqslant i < j \leqslant n}\pmb{Z}_{i,j}$ . Note that
$$
\mathbb {E} \boldsymbol {Z} _ {i, j} = 0,
$$
$$
\left| \boldsymbol {Z} _ {i, j} \right| \leqslant \frac {1}{\gamma n} \cdot \left(\frac {n}{d} - 1\right) + \frac {1}{\gamma d n} \leqslant \frac {1}{\gamma d},
$$
$$
\begin{array}{l} \mathbb {E} \boldsymbol {Z} _ {i, j} ^ {2} = \operatorname {V a r} \left[ \boldsymbol {Y} (\mathbf {G}) _ {i, j} \right] \leqslant \mathbb {E} \boldsymbol {Y} (\mathbf {G}) _ {i, j} ^ {2} \leqslant (1 + \gamma) \frac {d}{n} \cdot \frac {1}{\gamma^ {2} n ^ {2}} \left[ \left(\frac {n}{d} - 1\right) ^ {2} - \frac {1}{\gamma^ {2} n ^ {2}} \right] + \frac {1}{\gamma^ {2} n ^ {2}} \\ \leqslant (1 + \gamma) \frac {1}{d \gamma^ {2} n} + \frac {1}{\gamma^ {2} n ^ {2}} \leqslant \frac {3}{\gamma^ {2} d n}. \\ \end{array}
$$
By Bernstein's Inequality (cf. [62, Proposition 2.14]) it follows that
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {i <   j} \boldsymbol {Z} _ {i, j} \geqslant \frac {5 0 n}{\gamma \sqrt {d}}\right) \leqslant \mathbb {P} \left(\sum_ {i <   j} \boldsymbol {Z} _ {i, j} \geqslant \frac {n ^ {2}}{2} \cdot \frac {1 0 0 n}{\gamma \sqrt {d}}\right) \leqslant 2 \exp \left(- \frac {\frac {1 0 ^ {4}}{\gamma^ {2} d}}{\frac {3}{\gamma^ {2} d n} + \frac {1 0 0}{3 \gamma^ {2} d ^ {3 / 2} n}}\right) \\ = 2 \exp \left(- \frac {1 0 ^ {4} n}{3 + \frac {1 0 0}{\sqrt {d}}}\right) \leqslant \exp (- 5 0 n). \\ \end{array}
$$
Hence, by a union bound over all  $x \in \{\pm 1\}^n$  it follows that
I Deferred proofs for clustering
In this section, we will prove Lemma D.15 restated below.
Lemma (Restatement of Lemma D.15). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Let  $W(\mathbf{Y})\in \mathcal{W}(\mathbf{Y})$  be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then
We will need the following fact about our clustering program. Similar facts where used, e.g., in [30, 26]. One difference for us is that we don't have a constraint on the lower bound on the cluster size indicated by our SOS variables. However, since we maximize a variant of the  $\ell_1$  norm of the second moment matrix of the pseudo-distribution this will make up for this.
Fact I.1. Consider the same setting as in Lemma D.15. Let  $0 < \delta \leqslant \frac{1}{1.5 \cdot 10^{10}} \cdot \frac{1}{k^{201}}$  and denote by  $\mathbf{C}_1, \ldots, \mathbf{C}_k \subseteq [n]$  the indices belonging to each true cluster. Then  $W(\mathbf{Y})$  satisfies the following three properties:
1. For all  $i,j\in [n]$  it holds that  $0\leqslant \mathbf{W}_{i,j}\leqslant 1$
2. for all  $i \in [n]$  it holds that  $\sum_{j=1}^{n} \mathbf{W}_{i,j} \leqslant \frac{n}{k}$  and for at least  $(1 - \frac{1}{1000k^{100}}) n$  indices  $i \in [n]$  it holds that  $\sum_{j=1}^{n} \mathbf{W}_{i,j} \geqslant (1 - \frac{1}{(10)^6k^{200}}) \cdot \frac{n}{k}$ ,
3. for all  $r \in [k]$  it holds that  $\sum_{i \in \mathbf{C}_{r}, j \notin \mathbf{C}_{r}} \mathbf{W}_{i,j} \leqslant \delta \cdot \frac{n^{2}}{k}$ .
We will prove Fact I.1 at the end of this section. With this in hand, we can proof Lemma D.15.
Proof of Lemma D.15. For brevity, we write  $\mathbf{W} = W(\mathbf{Y})$ . Since  $\phi(\mathbf{W}^*) = \mathbf{W}^*$  and  $\phi$  is 10-Lipschitz we can also bound
$$
\left\| \phi (\mathbf {W}) - \mathbf {W} ^ {*} \right\| _ {1} \leqslant 1 0 \cdot \left\| \mathbf {W} - \mathbf {W} ^ {*} \right\| _ {1}.
$$
Let  $\delta \leqslant \frac{1}{1.5 \cdot 10^{10}} \cdot \frac{1}{k^{201}}$  and again let  $\mathbf{C}_1, \ldots, \mathbf{C}_k \subseteq [n]$  denote the indices belonging to each true cluster. Note that by assumption that  $\mathbf{Y}$  is a good sample it holds for each  $r \in [k]$  that  $\frac{n}{k} - n^{0.6} \leqslant |\mathbf{C}_r| \leqslant \frac{n}{k} + n^{0.6}$ .
Let  $r, r' \in [k]$ . We can write
$$
\left\| \mathbf {W} - \mathbf {W} ^ {*} \right\| _ {1} = \sum_ {r = 1} ^ {k} \sum_ {i, j \in \mathbf {C} _ {r}} \left| \mathbf {W} _ {i, j} - 1 \right| + \sum_ {r = 1} ^ {k} \sum_ {i \in \mathbf {C} _ {r}, j \notin \mathbf {C} _ {r}} \left| \mathbf {W} _ {i, j} - 0 \right| \tag {I.1}
$$
Note that we can bound the second sum by  $k \cdot \delta \frac{n^2}{k}$  using Item 3. Further, in what follows consider only indices  $i$  such that  $\sum_{j=1}^{n} \mathbf{W}_{i,j} \geqslant \left(1 - \frac{1}{(10)^6 k^{200}}\right) \cdot \frac{n}{k}$ . By Item 2 we can bound the contribution of the other indices by
$$
\frac {1}{1 0 0 0 k ^ {1 0 0}} n \cdot \left(\frac {n}{k} + n ^ {0. 6}\right) \leqslant \frac {2}{1 0 0 0 k ^ {1 0 0}} \cdot \frac {n ^ {2}}{k}.
$$
Focusing only on such indices, for the first sum in Eq. (I.1), fix  $r \in [k]$ . We will aim to show that most entries of  $\mathbf{W}$  are large if and only if the corresponding entry of  $\mathbf{W}^*$  is 1. By Item 3 and Markov's Inequality, it follows that for at least a  $(1 - \frac{1}{1000k^{100}})$ -fraction of the indices  $i \in \mathbf{C}_r$  it holds that
$$
\sum_ {j \not \in \mathbf {C} _ {r}} \mathbf {W} _ {i, j} \leqslant 1 0 0 0 k ^ {1 0 0} \cdot \delta \frac {n ^ {2}}{k \cdot | \mathbf {C} _ {r} |} \leqslant 1 0 0 0 k ^ {1 0 0} \delta \cdot \frac {n}{1 - k \cdot n ^ {- 0 . 4}} \leqslant 2 0 0 0 k ^ {1 0 1} \delta \cdot \frac {n}{k},
$$
where we used that  $|\mathbf{C}_r| \geqslant \frac{n}{k} - n^{0.6}$ . Call such indices good. Notice that for good indices it follows using Item 2 that
$$
\sum_ {j \in \mathbf {C} _ {r}} \mathbf {W} _ {i, j} \geqslant \frac {n}{k} \cdot \left(1 - \frac {1}{(1 0) ^ {6} k ^ {2 0 0}} - 2 0 0 0 k ^ {1 0 1} \delta\right).
$$
Denote by  $G$  the number of  $j \in \mathbf{C}_r$  such that  $\mathbf{W}_{i,j} \geqslant 1 - \frac{1}{1000k^{100}}$ . Using the previous display and that  $\mathbf{W}_{i,j} \leqslant 1$  we obtain
$$
\begin{array}{l} \frac {n}{k} \cdot \left(1 - \frac {1}{(1 0) ^ {6} k ^ {2 0 0}} - 2 0 0 0 k ^ {1 0 1} \delta\right) \leqslant \sum_ {j \in \mathbf {C} _ {r}} \mathbf {W} _ {i, j} \leqslant G \cdot 1 + (| \mathbf {C} _ {r} | - G) \cdot \left(1 - \frac {1}{1 0 0 0 k ^ {1 0 0}}\right) \\ \leqslant G \cdot \frac {1}{1 0 0 0 k ^ {1 0 0}} + \frac {n}{k} \cdot \left(1 + \frac {1}{k n ^ {0 . 4}}\right) \cdot \left(1 - \frac {1}{1 0 0 0 k ^ {1 0 0}}\right) \\ \leqslant G \cdot \frac {1}{1 0 0 0 k ^ {1 0 0}} + \frac {n}{k} \cdot \left(1 + \frac {1}{k n ^ {0 . 4}}\right), \\ \end{array}
$$
where we also used  $|\mathbf{C}_r| \leqslant \frac{n}{k} + n^{0.6}$ . Rearranging now yields
$$
G \geqslant \frac {n}{k} \cdot \left(1 - \frac {1}{1 0 0 0 k ^ {1 0 0}} - \frac {1 0 ^ {3} k ^ {9 9}}{n ^ {0 . 4}} - 2 \cdot 1 0 ^ {6} k ^ {1 0 1} \delta\right) \geqslant \frac {n}{k} \cdot \left(1 - \frac {2}{1 0 0 0 k ^ {1 0 0}} - 2 \cdot 1 0 ^ {6} k ^ {1 0 1} \delta\right).
$$
$$
\begin{array}{l} \sum_ {i, j \in \mathbf {C} _ {r}} | \mathbf {W} _ {i, j} - 1 | = \sum_ {i, j \in \mathbf {C} _ {r}, i \text {i s g o o d}} | \mathbf {W} _ {i, j} - 1 | + \sum_ {i, j \in \mathbf {C} _ {r}, i \text {i s n o t g o o d}} | \mathbf {W} _ {i, j} - 1 | \\ \leqslant \left| \mathbf {C} _ {r} \right| \cdot \left(\left(\left| \mathbf {C} _ {r} \right| - G\right) \cdot 1 + \left| \mathbf {C} _ {r} \right| \cdot \frac {1}{1 0 0 0 k ^ {1 0 0}}\right) + \frac {1}{1 0 0 0 k ^ {1 0 0}} \cdot \left| \mathbf {C} _ {r} \right| ^ {2} \\ \leqslant \left| \mathbf {C} _ {r} \right| ^ {2} \left(1 + \frac {1}{5 0 0 k ^ {1 0 0}}\right) - G \cdot \left| \mathbf {C} _ {r} \right| \\ \leqslant \frac {n ^ {2}}{k ^ {2}} \left(1 + \frac {k}{n ^ {0 . 4}}\right) ^ {2} \left(1 + \frac {1}{5 0 0 k ^ {1 0 0}}\right) - \frac {n ^ {2}}{k ^ {2}} \left(1 - \frac {2}{1 0 0 0 k ^ {1 0 0}} - 2 \cdot 1 0 ^ {6} k ^ {1 0 1} \delta\right) \left(1 - \frac {k}{n ^ {0 . 4}}\right) \\ \leqslant \frac {n ^ {2}}{k ^ {2}} \cdot (3 0 \cdot 1 0 ^ {6} k ^ {1 0 1} \delta + \frac {1 1}{5 0 0 k ^ {1 0 0}}) \leqslant \frac {n ^ {2}}{k} \cdot (3 0 \cdot 1 0 ^ {6} k ^ {1 0 0} \delta + \frac {1 1}{5 0 0 k ^ {1 0 1}}) \\ \leqslant \frac {n ^ {2}}{k} \cdot \frac {3}{1 2 5 k ^ {1 0 1}}. \\ \end{array}
$$
Putting everything together, it follows that
$$
\| \phi (\mathbf {W}) - \mathbf {W} ^ {*} \| _ {\mathrm {F}} ^ {2} \leqslant \| \phi (\mathbf {W}) - \mathbf {W} ^ {*} \| _ {1} \leqslant 1 0 \cdot \frac {n ^ {2}}{k} \left(\delta k + \frac {2}{1 0 0 0 k ^ {1 0 0}} + \frac {3}{1 2 5 k ^ {1 0 0}}\right) \leqslant \frac {n ^ {2}}{k} \cdot \frac {4}{k ^ {1 0 0}} \leqslant \frac {n ^ {2}}{k} \cdot \frac {3}{k ^ {9 8}}.
$$
It remains to verify Fact I.1.
Proof of Fact I.1. Let  $\mathcal{P} = \mathcal{P}_{n,k,t}(\mathbf{Y})$  be the system of Eq.  $(\mathcal{P}_{n,k,t}(Y))$ . Recall that  $\mathbf{W}_{i,j} = \tilde{\mathbb{E}}\sum_{l\in [k]}z_{i,l}z_{j,l}$ . Since
$$
\mathcal {P} \left| _ {\frac {1}{4}} \left\{0 \leqslant \sum_ {l \in [ k ]} z _ {i, l} z _ {j, l} \leqslant \sum_ {l \in [ k ]} z _ {i, l} \leqslant 1 \right. \right\},
$$
it follows that  $0 \leqslant \mathbf{W}_{i,j} \leqslant 1$ . Further, for each  $i \in [n]$  it holds that
$$
\mathcal {P} \left| _ {\frac {1}{4}} \left\{\sum_ {j \in [ n ], l \in [ k ]} z _ {j, l} z _ {i, l} \leqslant \frac {n}{k} \sum_ {l \in [ k ]} z _ {i, l} \leqslant \frac {n}{k} \right\} \right.
$$
implying that  $\sum_{j\in [n]}\mathbf{W}_{i,j}\leqslant \frac{n}{k}$ . Further, by Lemma D.14
$$
\left\| \mathbf {W} \right\| _ {1} \geqslant \frac {n ^ {2}}{k} \cdot \left(1 - n ^ {- 0. 4} - \frac {1}{(1 0) ^ {1 0} k ^ {3 0 0}}\right) \geqslant \frac {n ^ {2}}{k} \cdot \left(1 - \frac {1}{(1 0) ^ {9} k ^ {3 0 0}}\right).
$$
Denote by  $\mathbf{W}_i$  the  $i$ -th row of  $\mathbf{W}$  and by  $L$  the number of rows which have  $\ell_1$  norm at least  $(1 - \frac{1}{(10)^6k^{200}})\cdot \frac{n}{k}$ . Since for all  $i$  it holds that  $\| \mathbf{W}_i\| _1\leqslant \frac{n}{k}$  it follows that
$$
\begin{array}{l} \frac {n ^ {2}}{k} \cdot \left(1 - \frac {1}{(1 0) ^ {9} k ^ {3 0 0}}\right) \leqslant \sum_ {i \in [ n ]} \| \mathbf {W} _ {i} \| _ {1} \leqslant L \cdot \frac {n}{k} + (n - L) \cdot \left(1 - \frac {1}{(1 0) ^ {6} k ^ {2 0 0}}\right) \cdot \frac {n}{k} \\ = L \cdot \frac {1}{(1 0) ^ {6} k ^ {2 0 0}} \cdot \frac {n}{k} + \frac {n ^ {2}}{k} \cdot \left(1 - \frac {1}{(1 0) ^ {6} k ^ {2 0 0}}\right) \\ \end{array}
$$
Rearranging then yields  $L \geqslant \left(1 - \frac{1}{1000k^{100}}\right) \cdot n$  which proofs Item 2.
It remains to verify Item 3. Fix  $r, l \in [k]$  and define  $z_{l}(\mathbf{C}_{r}) = \frac{k}{n}\sum_{i\in \mathbf{C}_{r}}z_{i,l}$ . Let  $t > 0$  be an integer. We aim to show that for all unit vectors  $v$  it holds that
$$
\mathcal {P} \left| \overline {{1 0 t}} \left\{z _ {l} \left(\mathbf {C} _ {r}\right) \cdot \frac {1}{\Delta^ {2 t}} \sum_ {r ^ {\prime} \neq r} z _ {l} \left(\mathbf {C} _ {r ^ {\prime}}\right) \langle \mu_ {r} - \mu_ {r ^ {\prime}}, v \rangle^ {2 t} \leqslant \frac {\delta}{k} \right\}, \right. \tag {I.2}
$$
where  $\Delta$  is the minimal separation between the true means. Before proving this, let us examine how we can use this fact to prove Item 3. Note, that for all  $r \neq r'$  it holds that
$$
\sum_ {s, u \in [ k ]} \left\langle \mu_ {r} - \mu_ {r ^ {\prime}}, \frac {\mu_ {s} - \mu_ {u}}{\| \mu_ {s} - \mu_ {u} \|} \right\rangle^ {2 t} \geqslant \Delta^ {2 t}.
$$
Hence, if the above SOS proof indeed exists, we obtain
$$
\begin{array}{l} \sum_ {i \in \mathbf {C} _ {r}, j \not \in \mathbf {C} _ {r}} \mathbf {W} _ {i, j} = \sum_ {l = 1} ^ {k} \tilde {\mathbb {E}} \sum_ {i \in \mathbf {C} _ {r}, j \not \in \mathbf {C} _ {r}} z _ {i, l} z _ {j, l} = \frac {n ^ {2}}{k ^ {2}} \tilde {\mathbb {E}} z _ {l} (\mathbf {C} _ {r}) \cdot \sum_ {r ^ {\prime} \neq r} z _ {l} (\mathbf {C} _ {r ^ {\prime}}) \\ \leqslant \frac {n ^ {2}}{\Delta^ {2 t} k ^ {2}} \sum_ {s, u \in [ k ]} \tilde {\mathbb {E}} z _ {l} \left(\mathbf {C} _ {r}\right) \cdot \sum_ {r ^ {\prime} \neq r} z _ {l} \left(\mathbf {C} _ {r}\right) \left\langle \mu_ {r} - \mu_ {r ^ {\prime}}, \frac {\mu_ {s} - \mu_ {u}}{\| \mu_ {s} - \mu_ {u} \|} \right\rangle^ {2 t} \\ \leqslant \frac {\delta}{k} k ^ {2} \cdot \frac {n ^ {2}}{k ^ {2}} = \delta \cdot \frac {n ^ {2}}{k}. \\ \end{array}
$$
In the remainder of this proof we will prove Eq. (I.2). We will use the following SOS version of the triangle Inequality (cf. Fact I.2)
$$
\left| \frac {x , y}{2 t} \right. (x + y) ^ {2 t} \leqslant 2 ^ {2 t - 1} (x ^ {2 t} + y ^ {2 t}).
$$
Recall that  $\mu_l' = \frac{k}{n} \sum_{i=1}^{n} z_{i,l} y_i$  and denote by  $\mu_{\pi(i)}$  the true mean corresponding to the  $i$ -th sample. Let  $v$  be an arbitrary unit vector, it follows that
$$
\begin{array}{l} \mathcal {P} \left| \frac {}{1 0 t} \left\{z _ {l} (\mathbf {C} _ {r}) \cdot \frac {1}{\Delta^ {2 t}} \sum_ {r ^ {\prime} \neq r} z _ {l} (\mathbf {C} _ {r ^ {\prime}}) \langle \mu_ {r} - \mu_ {r ^ {\prime}}, v \rangle^ {2 t} \right. \right. \\ \leqslant z _ {l} \left(\mathbf {C} _ {r}\right) \cdot \frac {2 ^ {2 t - 1}}{\Delta^ {2 t}} \sum_ {r ^ {\prime} \neq r} z _ {l} \left(\mathbf {C} _ {r ^ {\prime}}\right) \left(\left\langle \mu_ {r} - \mu_ {l} ^ {\prime}, v \right\rangle^ {2 t} + \left\langle \mu_ {r ^ {\prime}} - \mu_ {l} ^ {\prime}, v \right\rangle^ {2 t}\right) \\ \leqslant \frac {2 ^ {2 t - 1}}{\Delta^ {2 t}} \sum_ {r = 1} ^ {k} z _ {l} (\mathbf {C} _ {r}) \langle \mu_ {r} - \mu_ {l} ^ {\prime}, v \rangle^ {2 t} = \frac {2 ^ {2 t - 1}}{\Delta^ {2 t}} \cdot \frac {k}{n} \sum_ {i = 1} ^ {n} z _ {i, l} \langle \mu_ {\pi (i)} - \mu_ {l} ^ {\prime}, v \rangle^ {2 t} \}, \\ \end{array}
$$
where we used that  $\mathcal{P}\big|_{\overline{1}}\sum_{r = 1}^{k}z_{l}(\mathbf{C}_{r})\leqslant 1$ . Using the SOS triangle inequality again and that  $\mathcal{P}\big|_{\overline{2}}z_{i,l}\leqslant 1$  we obtain