SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $g_W$  be the number of indices  $i \in \mathcal{G}$  such that  $\| \phi(W)_i \|_1 \geqslant \frac{n}{k} \cdot \left( 1 - \frac{1}{k^{50}} \right)$ . It holds that
$$
\begin{array}{l} \frac {n ^ {2}}{k} \cdot \left(1 - \frac {2}{k ^ {1 0 0}}\right) \leqslant g _ {W} \cdot \frac {n}{k} + (n - | \mathcal {G} |) \cdot \frac {n}{k} + (| G | - g _ {w}) \frac {n}{k} \cdot \left(1 - \frac {1}{k ^ {5 0}}\right) \\ \leqslant g _ {W} \cdot \frac {n}{k} \cdot \frac {1}{k ^ {5 0}} + \frac {n ^ {1 . 8}}{k} + \frac {n ^ {2}}{k} \cdot \left(1 - \frac {1}{k ^ {5 0}}\right) \\ \leqslant g _ {W} \cdot \frac {n}{k} \cdot \frac {1}{k ^ {5 0}} + \frac {n ^ {2}}{k} \cdot \left(1 + \frac {1}{k ^ {1 0 0}} - \frac {1}{k ^ {5 0}}\right). \\ \end{array}
$$
Rearrning now yields
$$
g _ {W} \geqslant n \cdot \left(1 - \frac {3}{k ^ {5 0}}\right).
$$
Similarly, let  $g_{W'}$  be the number of indices  $i \in \mathcal{G}$  such that  $\| \phi(W')_i \|_1 \geqslant \frac{n}{k} \cdot \left( 1 - \frac{1}{k^{50}} \right)$ . By an analogous argument it follows that  $g_{W'} \geqslant n \cdot \left( 1 - \frac{3}{k^{50}} \right)$ . Thus, by the pigeonhole principle there are at least  $g_W \geqslant n \cdot \left( 1 - \frac{6}{k^{50}} \right)$  indices  $i$  such that
1.  $\| \phi (W)_i\| _1\geqslant \frac{n}{k}\left(1 - \frac{1}{k^{50}}\right)$
2.  $\| \phi (W')_i\| _1\geqslant \frac{n}{k}\left(1 - \frac{1}{k^{50}}\right)$
3.  $\| \phi (W)_i - \phi (W')_i\| _1\leqslant n^{0.8}$
Combining these with Lemma D.11 we may also add
4. the distribution induced by  $\| \phi (W_i)\| _1^{-1}\phi (W_i)$  is 2-explicitly 40-bounded,
5. the distribution induced by  $\| \phi (W_i^{\prime})\| _1^{-1}\phi (W_i^{\prime})$  is 2-explicitly 40-bounded.
Using that for non-zero vectors  $x, y$  it holds that  $\left\| \frac{x}{\|x\|} - \frac{y}{\|y\|} \right\| \leqslant \frac{2}{\|x\|} \| x - y \|$  points 1 to 3 above imply that
$$
\left\| \left\| \phi (W _ {i}) \right\| _ {1} ^ {- 1} \phi (W _ {i}) - \left\| \phi (W _ {i} ^ {\prime}) \right\| _ {1} ^ {- 1} \phi (W _ {i} ^ {\prime}) \right\| _ {1} \leqslant \frac {2 n ^ {0 . 8}}{\frac {n}{k} \cdot \left(1 - \frac {1}{k ^ {5 0}}\right)} = O \left(n ^ {- 0. 2}\right).
$$
Hence, applying Theorem A.21 with  $t = 1$  it follows that
D.1.3 From low sensitivity to privacy
In this section we argue privacy of the whole algorithm, proving Lemma D.8. Before doing that we observe that low-sensitivity is preserved with high probability under subsampling.
Fact D.12 (Stability of  $\mathcal{S}$ ). Consider the settings of Theorem D.3. Suppose Algorithm D.5 does not reject in step 3, on adjacent inputs  $Y$ ,  $Y' \in \mathcal{V}$ . With probability at least  $1 - e^{-n^{\Omega(1)}}$  over the random choices of  $\mathcal{S}$ , for all but  $\frac{10n^{0.01}}{k^{50}}$  indices  $i \in \mathcal{S}$ , it holds:
$$
\left\| \nu_ {Y} ^ {(i)} - \nu_ {Y ^ {\prime}} ^ {(i)} \right\| _ {2} \leqslant O \left(n ^ {- 0. 1}\right).
$$
Proof. There are at most  $\frac{6n}{k^{50}}$  such indices in  $[n]$  by Lemma D.10. By Chernoff's bound, cf. Fact E.4, the claim follows.
Finally, we prove our main privacy lemma.
Proof of Lemma D.8. For simplicity, we will prove that the algorithm is  $(5\varepsilon, 5\delta)$ -private. Let  $Y, Y' \in \mathcal{V}$  be adjacent inputs. By Lemma A.10 and Lemma D.9 the test in step 3 of Algorithm D.5 is  $(\varepsilon, \delta)$ -private.
Thus suppose now the algorithm did not reject in step 3 on inputs  $Y, Y'$ . By composition (cf. Lemma A.4) it is enough to show that the rest of the algorithm is  $(\varepsilon, \delta)$ -private with respect to  $Y, Y'$  under this condition. Next, let  $\nu_{Y}^{(1)}, \ldots, \nu_{Y}^{(n)}$  and  $\nu_{Y'}^{(1)}, \ldots, \nu_{Y'}^{(n)}$  be the vectors computed in step 4 of the algorithm and  $S$  be the random set of indices computed in step 5.[33] By Lemma D.10 and Fact D.12 with probability  $1 - e^{-n^{\Omega(1)}}$  over the random choices of  $S$  we get that for all but  $\frac{10n^{0.01}}{k^{50}}$  indices  $i \in S$ , it holds that
Denote this set of indices by  $\mathcal{G}$ . Note, that we may incorporate the failure probability  $e^{-n^{\Omega(1)}} \leqslant \min\{\varepsilon/2, \delta/2\}$  into the final privacy parameters using Fact I.3.
Denote by  $\mathbf{V},\mathbf{V}^{\prime}$  the  $|\mathcal{S}|$ -by- $d$  matrices respectively with rows  $\nu_{Y}^{(i_{1})},\ldots ,\nu_{Y}^{(i_{|\mathcal{S}|})}$  and  $\nu_{Y^{\prime}}^{(i_1)},\ldots ,\nu_{Y^{\prime}}^{(i_{|\mathcal{S}|})}$ , where  $i_1,\dots,i_{|\mathcal{S}|}$  are the indices in  $\mathcal{S}$ . Recall, that  $|\mathcal{G}|$  rows of  $\mathbf{V}$  and  $\mathbf{V}^{\prime}$  differ by at most  $O\left(n^{-0.1}\right)$  in  $\ell_2$ -norm. Thus, by the Gaussian mechanism used in step 6 (cf. Lemma A.12) and Lemma I.4 it is enough to show that step 7 to step 9 of the algorithm are private with respect to pairs of inputs  $V$  and  $V^{\prime}$  differing in at most 1 row.[34] In particular, suppose these steps are  $(\varepsilon_{1},\delta_{1})$ -private. Then, for  $m = n^{0.01} - |\mathcal{G}|\leqslant \frac{10n^{0.01}}{k^{50}}$ , by Lemma I.4 it follows that step 6 to step 9 are  $(\varepsilon^{\prime},\delta^{\prime})$ -differentially private with
$$
\varepsilon^ {\prime} := \varepsilon + m \varepsilon_ {1},
$$
$$
\delta^ {\prime} := e ^ {\varepsilon} m e ^ {(m - 1) \varepsilon_ {1}} \delta_ {1} + \delta .
$$
Consider steps 7 and 8. Recall, that in step 7 we invoke the histogram learner with parameters
$$
b = k ^ {- 1 5}, \mathbf {q} \stackrel {u. a. r.} {\sim} [ 0, b ], \alpha = k ^ {- 1 0}, \beta = n ^ {- 1 0}, \delta^ {*} = \frac {\delta}{n}, \varepsilon^ {*} = \varepsilon \cdot \frac {1 0 k ^ {5 0}}{n ^ {0 . 0 1}}.
$$
Hence, by Lemma A.13 this step is  $(\varepsilon^{*},\delta^{*})$  -private since
$$
\frac {8}{\varepsilon^ {*} \alpha} \cdot \log \left(\frac {2}{\delta^ {*} \beta}\right) \leqslant \frac {2 0 0 \cdot k ^ {1 0} \cdot n ^ {0 . 0 1}}{1 0 \cdot k ^ {5 0} \cdot \varepsilon} \cdot \log n = \frac {2 0 \cdot n ^ {0 . 0 1}}{k ^ {4 0} \cdot \varepsilon} \cdot \log n \leqslant n,
$$
for  $\varepsilon \geqslant k^{-10}$ . Step 8 is private by post-processing.
33Note that since this does not depend on  $Y$  or  $Y^{\prime}$ , respectively, we can assume this to be the same in both cases. Formally, this can be shown, e.g., via a direct calculation or using Lemma A.4.
34Note that for the remainder of the analysis, these do not correspond to  $\mathbf{V}$  and  $\mathbf{V}'$ , since those differ in  $m$  rows. Lemma I.4 handles this difference.
Next, we argue that step 9 is private by showing that the average over the bins has small  $\ell_2$ -sensitivity. By Lemma A.4 we can consider the bins  $\mathbf{B}_1, \ldots, \mathbf{B}_k$  computed in the previous step as fixed. Further, we can assume that the algorithm did not reject in step 8, i.e., that each bin contains at least  $\frac{n^{0.01}}{2k}$  points of  $V$  and  $V'$  respectively. As a consequence, every bin contains at least two (projections of) points of the input  $V$  or  $V'$  respectively. In particular, it contains at least one (projection of a) point which is present in both  $V$  and  $V'$ . Fix a bin  $\mathbf{B}_l$  and let  $\bar{\nu}^*$  be such that it is both in  $V$  and  $V'$  and  $\Phi \bar{\nu}^* \in \mathbf{B}_l$ . Also, define
$$
S _ {l} := \left| \left\{j \left| \Phi \bar {\nu} _ {Y} ^ {(j)} \in \mathbf {B} _ {i} \right. \right\} \right|,
$$
$$
S _ {l} ^ {\prime} := \left| \left\{j \middle | \Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {i} \right\} \right|.
$$
Assume  $V$  and  $V'$  differ on index  $j$ . We consider two cases. First, assume that  $\Phi \bar{\nu}_{Y}^{(j)}$  and  $\Phi \bar{\nu}_{Y'}^{(j)}$  both lie in  $\mathbf{B}_l$ . In this case,  $S_l = S_l'$  and using Lemma E.5 it follows that with probability  $n^{-100} \leqslant \min \left\{ \varepsilon / 2, \delta / 2 \right\}$  it holds that
$$
\begin{array}{l} \left\| \bar {\nu} _ {Y} ^ {(j)} - \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \right\| _ {2} \leqslant \left\| \bar {\nu} _ {Y} ^ {(j)} - \bar {\nu} ^ {*} \right\| _ {2} + \left\| \bar {\nu} ^ {*} - \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \right\| \leqslant 1 0 \cdot \left(\left\| \Phi \bar {\nu} _ {Y} ^ {(j)} - \Phi \bar {\nu} ^ {*} \right\| _ {2} + \left\| \Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} - \Phi \bar {\nu} ^ {*} \right\| _ {2}\right) \\ \leqslant 2 0 \cdot \sqrt {d ^ {*}} \cdot b \leqslant 2 0 0 \cdot k ^ {- 1 2}. \\ \end{array}
$$
And hence we can bound
$$
\left\| \frac {1}{S _ {l}} \cdot \left(\sum_ {\Phi \bar {\nu} _ {Y} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y} ^ {(j)}\right) - \frac {1}{S _ {l} ^ {\prime}} \cdot \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)}\right) \right\| _ {2} \leqslant \frac {\left\| \bar {\nu} _ {Y} ^ {(j)} - \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \right\| _ {2}}{S _ {l}} \leqslant \frac {4 0 0 \cdot k ^ {- 1 1}}{n ^ {0 . 0 1}}.
$$
Next, assume that  $\Phi \bar{\nu}_{Y}^{(j)} \notin \mathbf{B}_l$  and  $\Phi \bar{\nu}_{Y'}^{(j)} \in \mathbf{B}_l$  (the other case works symmetrically). It follows that  $S_l = S_l' - 1$  and we can bound
$$
\begin{array}{l} \left\| \frac {1}{S _ {l}} \cdot \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y} ^ {(j)}\right) - \frac {1}{S _ {l} ^ {\prime}} \cdot \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)}\right) \right\| _ {2} = \frac {1}{S _ {l} \cdot S _ {l} ^ {\prime}} \cdot \left\| S _ {l} ^ {\prime} \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y} ^ {(j)}\right) - (S _ {l} ^ {\prime} - 1) \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)}\right) \right\| _ {2} \\ = \frac {1}{S _ {l} \cdot S _ {l} ^ {\prime}} \cdot \left\| S _ {l} ^ {\prime} \cdot \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} + \left(\sum_ {\Phi \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)}\right) \right\| _ {2} \\ = \frac {1}{S _ {l}} \cdot \left\| \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} - \frac {1}{S _ {l} ^ {\prime}} \left(\sum_ {\boldsymbol {\Phi} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\nu} _ {Y ^ {\prime}} ^ {(j)}\right) \right\| _ {2} \\ \leqslant \frac {\sqrt {d ^ {*}} \cdot b}{S _ {t}} \leqslant \frac {2 0 \cdot k ^ {- 1 1}}{n ^ {0 . 0 1}}. \\ \end{array}
$$
Hence, the  $\ell_2$ -sensitivity is at most  $\Delta \coloneqq \frac{400 \cdot k^{-11}}{n^{0.01}}$ . Since
$$
2 \Delta^ {2} \cdot \frac {\log (2 / (\delta^ {*} / k))}{(\varepsilon^ {*} / k) ^ {2}} = 3 2 \cdot k ^ {- 1 2 0} \cdot \frac {\log (2 k n / \delta)}{\varepsilon^ {2}}
$$
and  $\mathbf{w}' \sim N\left(0, 32 \cdot k^{-120} \cdot \frac{\log(2kn / \delta)}{\varepsilon^2} \cdot \mathrm{Id}\right)$  it follows that outputting  $\hat{\mu}_l$  is  $(\varepsilon^* / k, \delta^* / k)$ -DP by the Gaussian Mechanism that. By Lemma A.4 it follows step 9 is  $(\varepsilon^*, \delta^*)$ -private.
Hence, by Lemma A.4 it follows that step 7 to step 9 are  $(2\varepsilon^{*},2\delta^{*})$  -differentially private. Using  $m\leqslant \frac{10n^{0.01}}{k^{10}}$  it now follows by Lemma I.4 that step 6 to step 9 are  $(\varepsilon^{\prime},\delta^{\prime})$  -private for
$$
\varepsilon^ {\prime} = \varepsilon + 2 m \varepsilon^ {*} \leqslant 3 \varepsilon ,
$$
$$
\delta^ {\prime} = 2 e ^ {\varepsilon} m e ^ {(m - 1) 2 \varepsilon^ {*}} \delta^ {*} + \delta \leqslant 2 m e ^ {3 \varepsilon} \cdot \frac {\delta}{n} + \delta \leqslant 3 \delta .
$$
Thus, combined with the private check and Fact I.3 in step 3 the whole algorithm is  $(5\varepsilon, 5\delta)$ -private.
D.2 Utility analysis
In this section we reason about the utility of Algorithm D.5 and prove Theorem D.3. We first introduce some notation.
Definition D.13 (True solution). Let  $\mathbf{Y}$  be an input sampled from Model 1.2. Denote by  $W^{*}(\mathbf{Y})\in$ $\mathcal{W}(\mathbf{Y})$  the matrix induced by the true solution (or ground truth). I.e., let
$$
W ^ {*} (\mathbf {Y}) _ {i j} = \left\{ \begin{array}{l l} 1 & \text {i f i , j w e r e b o t h s a m p l e d f r o m t h e s a m e c o m p o n e n t o f t h e m i x t u r e}, \\ 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
Whenever the context is clear, we simply write  $\mathbf{W}^*$  to ease the notation.
First, we show that in the utility case step 3 of Algorithm D.5 rejects only with low probability.
Lemma D.14 (Algorithm does not reject on good inputs). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Then  $\left\| W(\hat{\zeta} (\mathbf{Y}))\right\|_{1}\geqslant \frac{n^{2}}{k}\cdot \left(1 - n^{-0.4} - \frac{1}{(10)^{10}k^{300}}\right)$  and Algorithm D.5 rejects with probability at most  $\exp \left(-\Omega \left(n^{1.7}\right)\right)$ .
Proof. Since  $\mathbf{Y}$  is good, there exists  $\mathbf{W}^* \in \mathcal{W}(\mathbf{Y})$ , corresponding to the indicator matrix of the true solution, such that
$$
\begin{array}{l} g \left(\mathbf {W} ^ {*}\right) = \| \mathbf {W} ^ {*} \| _ {\mathrm {F}} ^ {2} - 1 0 ^ {1 0} k ^ {3 0 0} \langle J, \mathbf {W} ^ {*} \rangle \leqslant \frac {n ^ {2}}{k} + n ^ {1. 6} - (1 0) ^ {1 0} k ^ {3 0 0} \left(\frac {n ^ {2}}{k} - n ^ {1. 6}\right) \\ = \frac {n ^ {2}}{k} \left(1 + \frac {k}{n ^ {0 . 4}} - (1 0) ^ {1 0} k ^ {3 0 0} \left(1 - \frac {k}{n ^ {0 . 4}}\right)\right). \\ \end{array}
$$
Since  $g(W(\hat{\zeta} (\mathbf{Y})))\leqslant g(\mathbf{W}^{*})$  it follows that
$$
(1 0) ^ {1 0} k ^ {3 0 0} \langle J, W (\hat {\zeta} (\mathbf {Y})) \rangle \geqslant | g (W (\hat {\zeta} (\mathbf {Y}))) | \geqslant \frac {n ^ {2}}{k} \left((1 0) ^ {1 0} k ^ {3 0 0} \left(1 - \frac {k}{n ^ {0 . 4}}\right) - 1 - \frac {k}{n ^ {0 . 4}}\right).
$$
Since,  $\left\| W(\hat{\zeta} (\mathbf{Y}))\right\| _1\geqslant \langle J,W(\hat{\zeta} (\mathbf{Y}))\rangle$  the first claim follows rearranging the terms. This means that the algorithm rejects only if  $|\pmb {\tau}|\geqslant n^{1.7}$ . Recall that  $\pmb {\tau}\sim \mathrm{tLap}\left(-n^{1.6}\left(1 + \frac{\log(1 / \delta)}{\varepsilon}\right),\frac{n^{1.6}}{\varepsilon}\right)$ . Hence, by Lemma A.11 it follows that
$$
\mathbb {P} \left(| \boldsymbol {\tau} | \geqslant n ^ {1. 7}\right) \leqslant \frac {\exp \left(- n ^ {1 . 7} + \varepsilon + \log (1 / \delta)\right)}{2 - \exp \left(- \varepsilon - \log (1 / \delta)\right)} = \exp \left(- \Omega \left(n ^ {1. 7}\right)\right).
$$
The next step shows that on a good input  $\mathbf{Y}$  the matrix  $\phi(W(\hat{\zeta}(\mathbf{Y})))$  is close to the true solution.
Lemma D.15 (Closeness to true solution on good inputs). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Let  $W(\mathbf{Y})\in \mathcal{W}(\mathbf{Y})$  be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then
$$
\left\| \phi (W (\mathbf {Y})) - \mathbf {W} ^ {*} \right\| _ {1} \leqslant \frac {n ^ {2}}{k} \cdot \frac {3}{k ^ {9 8}}.
$$
The proof is similar to the classical utility analysis of the sum-of-squares program found, e.g., in [30, 26]. We defer it to Appendix I.
Together, the above results imply that the vectors  $\nu^{(i)}$  computed by the algorithm are close to the true centers of the mixture.
Lemma D.16 (Closeness to true centers). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Let  $\mathbf{W} \in \mathcal{W}(\mathbf{Y})$  be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject in step 3. Then for each  $\ell \in [k]$ , there exists  $\frac{n}{k} \cdot \left(1 - \frac{2}{k^{47}}\right)$  indices  $i \in [n]$ , such that
$$
\left\| \nu^ {(i)} (\mathbf {W}) - \mu_ {\ell} \right\| _ {2} \leqslant O (k ^ {- 2 5}).
$$
Proof. We aim to show that for most indices  $i \in [n]$  the vectors  $\| \phi(\mathbf{W}_i) \|_1^{-1} \phi(\mathbf{W}_i)$  and  $\| \mathbf{W}_i^* \|_1^{-1} \mathbf{W}_i^*$  induce a 2-explicitly 40-bounded distribution over  $\mathbf{Y}$ . If additionally the two vectors are close in  $\ell_1$ -norm, the result will follow by Theorem A.21.
Note that  $\| \mathbf{W}_i^*\| _1^{-1}\mathbf{W}_i^*$  induces a 2-explicitly 40-bounded distribution by Lemma D.7. By Markov's inequality and Lemma D.15 there can be at most  $n / k^{48}$  indices  $j\in [n]$  such that
$$
\left\| \phi (\mathbf {W}) _ {j} - \mathbf {W} _ {j} ^ {*} \right\| _ {1} \geqslant \frac {n}{k} \cdot \frac {3}{k ^ {5 0}}.
$$
Consider all remaining indices  $i$ . It follows that
$$
\| \phi (\mathbf {W} _ {i}) \| _ {1} \geqslant \| \mathbf {W} _ {i} ^ {*} \| _ {1} - \| \phi (\mathbf {W}) _ {i} - \mathbf {W} _ {i} ^ {*} \| _ {1} \geqslant \frac {n}{k} \cdot \left(1 - \frac {k}{n ^ {0 . 4}} - \frac {3}{k ^ {5 0}}\right) \geqslant \frac {n}{k} \cdot \left(1 - \frac {1 0}{k ^ {5 0}}\right).
$$
Hence, by Lemma D.11 the distribution induced by  $\| \phi (\mathbf{W}_i)\| _1^{-1}\phi (\mathbf{W}_i)$  is 2-explicitly 40-bounded distribution. Further, using  $\| \mathbf{W}_i^*\| _1\geqslant \frac{n}{k}\left(1 - \frac{k}{n^{0.4}}\right)$  we can bound
$$
\begin{array}{l} \left\| \left\| \phi (\mathbf {W} _ {i}) \right\| _ {1} ^ {- 1} \phi (\mathbf {W} _ {i}) - \left\| \mathbf {W} _ {i} ^ {*} \right\| _ {1} ^ {- 1} \mathbf {W} _ {i} ^ {*} \right\| _ {1} = \left\| \phi (\mathbf {W} _ {i}) \right\| _ {1} ^ {- 1} \left\| \mathbf {W} _ {i} ^ {*} \right\| _ {1} ^ {- 1} \cdot \left\| \left\| \mathbf {W} _ {i} ^ {*} \right\| _ {1} \phi (\mathbf {W} _ {i}) - \left\| \phi (\mathbf {W} _ {i}) \right\| _ {1} \mathbf {W} _ {i} ^ {*} \right\| _ {1} \\ \leqslant \| \phi (\mathbf {W} _ {i}) \| _ {1} ^ {- 1} \| \mathbf {W} _ {i} ^ {*} \| _ {1} ^ {- 1} \cdot (| | \phi (\mathbf {W} _ {i}) \| _ {1} - \| \mathbf {W} _ {i} ^ {*} \| _ {1} | \cdot \| \phi (\mathbf {W} _ {i}) \| _ {1} + \| \phi (\mathbf {W} _ {i}) \| _ {1} \cdot \| \phi (\mathbf {W} _ {i}) - \mathbf {W} _ {i} ^ {*} \| _ {1}) \\ \leqslant \left\| \mathbf {W} _ {i} ^ {*} \right\| _ {1} ^ {- 1} \cdot 2 \left\| \phi (\mathbf {W} _ {i}) - \mathbf {W} _ {i} ^ {*} \right\| _ {1} \leqslant \frac {6}{k ^ {5 0} \cdot \left(1 - \frac {k}{n ^ {0 . 4}}\right)} \leqslant \frac {7}{k ^ {5 0}}. \\ \end{array}
$$
Hence, by Theorem A.21 for each  $l \in [k]$  there are at least  $\frac{n}{k} - n^{0.6} - \frac{n}{k^{48}} \geqslant \frac{n}{k} \cdot \left(1 - \frac{2}{k^{47}}\right)$  indices  $i$  such that
$$
\left\| \nu^ {(i)} (\mathbf {W}) - \| \mathbf {W} _ {i} ^ {*} \| _ {1} ^ {- 1} \sum_ {j = 1} ^ {n} \mathbf {W} _ {i, j} ^ {*} \mathbf {y} _ {j} \right\| _ {2} \leqslant O (k ^ {- 2 5}).
$$
The result now follows by standard concentration bounds applied to the distribution induced by  $\| \mathbf{W}_i^*\| _1^{-1}\mathbf{W}_i^*$ .
An immediate consequence of Lemma D.16 is that the vectors  $\bar{\nu}^{(i)}$  inherits the good properties of the vectors  $\nu^{(i)}$  with high probability.
Corollary D.17 (Closeness to true centers after sub-sampling). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Let  $\mathbf{W} \in \mathcal{W}(\mathbf{Y})$  be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then with high probability for each  $\ell \in [k]$ , there exists  $\frac{n^{0.01}}{k} \cdot \left(1 - \frac{150}{k^{47}}\right)$  indices  $i \in S$ , such that
$$
\left\| \bar {\nu} ^ {(i)} - \mu_ {\ell} \right\| _ {2} \leqslant O \left(k ^ {- 2 5}\right).
$$
Proof. For each  $\ell \in [k]$ , denote by  $\mathcal{T}_{\ell}$  the set of indices in  $[n]$  satisfying
By Lemma D.16 we know that  $\mathcal{T}_{\ell}$  has size at least  $\frac{n}{k} \cdot \left(1 - \frac{2}{k^{47}}\right)$ . Further, let  $S$  be the set of indices selected by the algorithm. By Chernoff's bound Fact E.4 with probability  $1 - e^{-n^{\Omega(1)}}$ , we have  $|\mathcal{S} \cap \mathcal{T}_{\ell}| \geqslant \frac{n^{0.01}}{k} \cdot \left(1 - \frac{150}{k^{47}}\right)$ . Taking a union bound over all  $\ell \in [k]$  we get that with probability  $1 - e^{-n^{\Omega(1)}}$ , for each  $\ell \in [k]$ , there exists  $\frac{n^{0.01}}{k} \cdot \left(1 - \frac{150}{k^{47}}\right)$  indices  $i \in S$  such that
Now, we obtain the corollary observing (cf. Fact E.1 with  $m = 1$ ) that with probability at least  $1 - e^{-n^{\Omega(1)}}$ , for all  $i \in \mathcal{S}$
$$
\left\| \bar {\boldsymbol {\nu}} ^ {(i)} - \nu^ {(i)} (\mathbf {W}) \right\| _ {2} = \| \mathbf {w} \| _ {2} \leqslant n ^ {- 0. 0 5} \cdot \frac {\sqrt {\log (2 / \delta)}}{\varepsilon} \cdot \sqrt {d} \leqslant n ^ {- 0. 0 4} \leqslant O \left(k ^ {- 2 5}\right).
$$
For each  $\ell$ , denote by  $\mathcal{G}_{\ell} \subseteq S$  the set of indices  $i \in S$  satisfying
$$
\left\| \bar {\nu} ^ {(i)} - \mu_ {\ell} \right\| _ {2} \leqslant O (k ^ {- 2 5}).
$$
Let  $\mathcal{G} \coloneqq \bigcup_{\ell \in [k]} \mathcal{G}_{\ell}$ . We now have all the tools to prove utility of Algorithm D.5. We achieve this by showing that with high probability, each bin returned by the algorithm at step 7 satisfies  $\mathcal{G}_{\ell'} \subseteq \mathbf{B}_{\ell}$  for some  $\ell, \ell' \in [k]$ . Choosing the bins small enough will yield the desired result.
Lemma D.18 (Closeness of estimates). Consider the settings of Theorem D.3. Suppose  $\mathbf{Y}$  is a good set as per Definition D.6. Let  $\mathbf{W} \in \mathcal{W}(\mathbf{Y})$  be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then with high probability, there exists a permutation  $\pi : [k] \to [k]$  such that
$$
\max  _ {\ell \in [ k ]} \left\| \mu_ {\ell} - \hat {\boldsymbol {\mu}} _ {\pi (\ell)} \right\| _ {2} \leqslant O \left(k ^ {- 2 0}\right)
$$
Proof. Consider distinct  $\ell, \ell' \in [k]$ . By Corollary D.17 for each  $\bar{\nu}^{(i)}, \bar{\nu}^{(j)} \in \mathcal{G}_{\ell}$  it holds that
$$
\left\| \bar {\boldsymbol {\nu}} ^ {(i)} - \bar {\boldsymbol {\nu}} ^ {(j)} \right\| _ {2} \leqslant C \cdot k ^ {- 2 5},
$$
for some universal constant  $C > 0$ . Moreover, by assumption on  $\mu_{\ell}, \mu_{\ell'}$  for each  $\bar{\nu}^{(i)} \in \mathcal{G}_{\ell}$  and  $\bar{\nu}^{(j)} \in \mathcal{G}_{\ell'}$
$$
\left\| \bar {\boldsymbol {\nu}} ^ {(i)} - \bar {\boldsymbol {\nu}} ^ {(j)} \right\| _ {2} \geqslant \Delta - O \left(k ^ {- 2 5}\right).
$$
Thus, by Lemma E.5 with probability at least  $1 - e^{\Omega (d^{*})}\geqslant 1 - n^{-100}$  it holds that or each  $\bar{\pmb{\nu}}^{(i)},\bar{\pmb{\nu}}^{(j)}\in \mathcal{G}_{\ell}$  and  $\bar{\pmb{\nu}}^r\in \mathcal{G}_{\ell '}$  with  $\ell^{\prime}\neq \ell$
$$
\left\| \Phi \bar {\nu} ^ {(i)} - \Phi \bar {\nu} ^ {(j)} \right\| _ {2} \leqslant C ^ {*} \cdot k ^ {- 2 5} \quad \text {a n d} \quad \left\| \Phi \bar {\nu} ^ {(i)} - \Phi \bar {\nu} ^ {(r)} \right\| _ {2} \geqslant \Delta - C ^ {*} \cdot k ^ {- 2 5}
$$
for some other universal constant  $C^* > C$ . Let  $Q_{\Phi}(\mathcal{G}_{\ell}) \subseteq \mathbb{R}^{d^*}$  be a ball of radius  $C^* \cdot (k^{-25})$  such that  $\forall i \in \mathcal{G}_{\ell}$  it holds  $\Phi \bar{\nu}^{(i)} \in Q_{\Phi}(\mathcal{G}_{\ell})$ . That is,  $Q_{\Phi}(\mathcal{G}_{\ell})$  contains the projection of all points in  $\mathcal{G}_{\ell}$ .
Recall that  $d^{*} = 100\log (n)\leqslant 100k^{5}$  and  $b = k^{-15}$ . Let  $\mathcal{B} = \{\mathbf{B}_i\}_{i = 1}^{\infty}$  be the sequence of bins computed by the histogram learner of Lemma A.13 for  $\mathbb{R}^{d^*}$  at step 7 of the algorithm. By choice of  $b$ , and since  $\mathbf{q}$  is chosen uniformly at random in  $[0,b]$ , the probability that there exists a bin  $\mathbf{B}\in \mathcal{B}$  containing  $Q_{\Phi}(\mathcal{G}_{\ell})$  is at least
$$
1 - d ^ {*} \cdot \frac {C ^ {*}}{b} \cdot (k ^ {- 2 5}) \geqslant 1 - \frac {1 0 0 C ^ {*}}{b} \cdot k ^ {- 2 0} \geqslant 1 - O (k ^ {- 5}),
$$
where we used that  $d^{*} = 100\log n \leqslant 100k^{5}$ . A simple union bound over  $\ell \in [k]$  yields that with high probability for all  $\ell \in [k]$ , there exists  $\mathbf{B} \in \mathcal{B}$  such that  $Q_{\Phi}(\mathcal{G}_{\ell}) \subseteq \mathbf{B}$ . For simplicity, denote such bin by  $\mathbf{B}_{\ell}$ .