SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Rightarrow | B _ {\mathrm {H a m}} (x, 4 \zeta) | \leqslant \sum_ {i = 1} ^ {m} 2 \cdot \left| B _ {\mathrm {H a m}} \left(x ^ {i}, 2 \zeta\right) \right| = 2 m \cdot | B _ {\mathrm {H a m}} (x, 2 \zeta) | \\ \Rightarrow 2 m \geqslant \frac {\left| B _ {\mathrm {H a m}} (x , 4 \zeta n) \right|}{\left| B _ {\mathrm {H a m}} (x , 2 \zeta n) \right|} = \frac {\binom {n / 2} {2 \zeta n} ^ {2}}{\binom {n / 2} {\zeta n} ^ {2}} \geqslant \frac {\left(\frac {1}{4 \zeta}\right) ^ {4 \zeta n}}{\left(\frac {e}{2 \zeta}\right) ^ {2 \zeta n}} = \left(\frac {1}{8 e \zeta}\right) ^ {2 \zeta n} \tag {C.12} \\ \end{array}
$$
For each  $i \in [m]$ , define  $Y_i := \{w \in \{\pm 1\}^n : \mathrm{err}(w, x^i) \leqslant \zeta\}$ . Then  $Y_i$ 's are pairwise disjoint. For each  $i \in [m]$ , let  $P_i$  be the distribution over  $n$ -vertex graphs generated by  $\mathsf{SBM}_n(d, \gamma, x^i)$ . By our assumption on the algorithm, we have for any  $i \in [m]$  that
$$
\underset {\mathbf {G} \sim P _ {i}} {\mathbb {P}} \left(\hat {x} (\mathbf {G}) \in Y _ {i}\right) \geqslant 1 - \eta .
$$
Combining the fact that  $Y_{i}$ 's are pairwise disjoint, we have
$$
\sum_ {i = 1} ^ {m} _ {\mathbf {G} \sim P _ {1}} \mathbb {P} (\hat {x} (\mathbf {G}) \in Y _ {i}) = \mathbb {P} _ {\mathbf {G} \sim P _ {1}} (\hat {x} (\mathbf {G}) \in \cup_ {i = 1} ^ {m} Y _ {i}) \leqslant 1 \Rightarrow \sum_ {i = 2} ^ {m} _ {\mathbf {G} \sim P _ {1}} \mathbb {P} (\hat {x} (\mathbf {G}) \in Y _ {i}) \leqslant \eta . \tag {C.13}
$$
In the following, we will lower bound  $\mathbb{P}_{\mathbf{G}\sim P_1}(\hat{x} (\mathbf{G})\in Y_i)$  for each  $i\in [m]\setminus \{1\}$  using group privacy.
Note each  $P_{i}$  is a product of  $\binom{n}{2}$  independent Bernoulli distributions. Thus for any  $i,j\in [m]$ , there exists a coupling  $\omega_{ij}$  of  $P_{i}$  and  $P_{j}$  such that, if  $(\mathbf{G},\mathbf{H})\sim \omega$ , then
$$
\operatorname {H a m} (\mathbf {G}, \mathbf {H}) \sim \operatorname {B i n o m i a l} (N _ {i j}, p),
$$
where  $p = 2\gamma d / n$  and  $N_{ij} = \mathrm{Ham}(x^i,x^j)\cdot (n - \mathrm{Ham}(x^i,x^j))$ . Applying group privacy, we have for any two graphs  $G,H$  and for any  $S\subseteq \{\pm 1\} ^n$  that<sup>31</sup>
$$
\mathbb {P} (\hat {x} (G) \in S) \leqslant \exp (\varepsilon \cdot \operatorname {H a m} (G, H)) \cdot \mathbb {P} (\hat {x} (H) \in S). \tag {C.14}
$$
For each  $i \in [m]$ , taking expectations on both sides of Eq. (C.14) with respect to coupling  $\omega_{i1}$  and setting  $S = Y_i$ , we have
$$
\underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \mathbb {P} (\hat {x} (\mathbf {G}) \in Y _ {i}) \leqslant \underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \exp (\varepsilon \cdot \operatorname {H a m} (\mathbf {G}, \mathbf {H})) \cdot \mathbb {P} (\hat {x} (\mathbf {H}) \in Y _ {i}). \tag {C.15}
$$
The left side of Eq. (C.15) is equal to
$$
\underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \mathbb {P} \left(\hat {x} (\mathbf {G}) \in Y _ {i}\right) = \underset {\mathbf {G} \sim P _ {i}} {\mathbb {P}} \left(\hat {x} (\mathbf {G}) \in Y _ {i}\right) \geqslant 1 - \eta .
$$
Upper bounding the right side of Eq. (C.15) by Cauchy-Schwartz inequality, we have
$$
\begin{array}{l} \underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \exp (\varepsilon \cdot \operatorname {H a m} (\mathbf {G}, \mathbf {H})) \cdot \mathbb {P} \left(\hat {x} (\mathbf {H}) \in Y _ {i}\right) \\ \leqslant \left(\underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \exp (2 \varepsilon \cdot \operatorname {H a m} (\mathbf {G}, \mathbf {H}))\right) ^ {1 / 2} \cdot \left(\underset {(\mathbf {G}, \mathbf {H}) \sim \omega_ {i 1}} {\mathbb {E}} \mathbb {P} (\hat {x} (\mathbf {H}) \in Y _ {i}) ^ {2}\right) ^ {1 / 2} \\ = \left(\underset {\mathbf {X} \sim \operatorname {B i n o m i a l} (N _ {i 1}, p)} {\mathbb {E}} \exp (2 \varepsilon \cdot \mathbf {X})\right) ^ {1 / 2} \cdot \left(\underset {\mathbf {H} \sim P _ {1}} {\mathbb {E}} \left(\hat {x} (\mathbf {H}) \in Y _ {i}\right) ^ {2}\right) ^ {1 / 2}. \\ \end{array}
$$
Using the formula for the moment generating function of binomial distributions, we have
$$
\underset {\mathbf {X} \sim \operatorname {B i n o m i a l} (N _ {i 1}, p)} {\mathbb {E}} \exp (2 \varepsilon \cdot \mathbf {X}) = (1 - p + p \cdot e ^ {2 \varepsilon}) ^ {N _ {i 1}},
$$
and it is easy to see
$$
\underset {\mathbf {H} \sim P _ {1}} {\mathbb {E}} \mathbb {P} \left(\hat {x} (\mathbf {H}) \in Y _ {i}\right) ^ {2} = \underset {\mathbf {H} \sim P _ {1}} {\mathbb {E}} \left(\mathbb {E} \mathbb {1} _ {[ \hat {x} (\mathbf {H}) \in Y _ {i} ]}\right) ^ {2} \leqslant \underset {\mathbf {H} \sim P _ {1}} {\mathbb {P}} \left(\hat {x} (\mathbf {H}) \in Y _ {i}\right).
$$
Putting things together, Eq. (C.15) implies for each  $i \in [m]$  that
$$
\underset {\mathbf {H} \sim P _ {1}} {\mathbb {P}} (\hat {x} (\mathbf {H}) \in Y _ {i}) \geqslant \frac {(1 - \eta) ^ {2}}{(1 - p + p \cdot e ^ {2 \varepsilon}) ^ {N _ {i 1}}}. \tag {C.16}
$$
31In Eq. (C.14), the randomness only comes from the algorithm.
Since  $x^{i} \in B_{\mathrm{err}}(x, 4\zeta)$  for  $i \in [m]$ , by assuming  $\zeta \leqslant 1 / 16$ , we have
$$
N _ {i 1} = \operatorname {H a m} \left(x ^ {i}, x ^ {1}\right) \cdot \left(n - \operatorname {H a m} \left(x ^ {1}, x ^ {i}\right)\right) \leqslant 8 \zeta n (n - 8 \zeta n). \tag {C.17}
$$
Recalling  $p = 2\gamma d / n$  and combining Eq. (C.12), Eq. (C.13), Eq. (C.16) and Eq. (C.17), we have
$$
(m - 1) \cdot \frac {(1 - \eta) ^ {2}}{(1 - p + p \cdot e ^ {2 \varepsilon}) ^ {8 \zeta n (n - 8 \zeta n)}} \leqslant \eta .
$$
By taking logarithm on both sides, using  $t \geqslant \log (1 + t)$  for any  $t > -1$ , and assuming  $\zeta \leqslant 1 / (8e)$ , we have
$$
e ^ {2 \varepsilon} - 1 \gtrsim \frac {\log \frac {1}{8 e \zeta}}{\gamma d} + \frac {\log \frac {1}{\eta}}{\zeta n \gamma d}.
$$
D Private algorithms for learning mixtures of spherical Gaussians
In this section we present a private algorithm for recovering the centers of a mixtures of  $k$  Gaussians (cf. Model 1.2). Let  $\mathcal{Y} \subseteq (\mathbb{R}^d)^{\otimes n}$  be the collection of sets of  $n$  points in  $\mathbb{R}^d$ . We consider the following notion of adjacency.
Definition D.1 (Adjacent datasets). Two datasets  $Y, Y' \in \mathcal{V}$  are said to be adjacent if  $|Y \cap Y'| \geqslant n - 1$ .
Remark D.2 (Problem parameters as public information). We consider the parameters  $n, k, \Delta$  to be public information given as input to the algorithm.
Next we present the main theorem of the section.
Theorem D.3 (Privately learning spherical mixtures of Gaussians). Consider an instance of Model 1.2. Let  $t \in \mathbb{N}$  be such that  $\Delta \geqslant O\left(\sqrt{t} k^{1 / t}\right)$ . For  $n \geqslant \Omega \left(k^{O(1)} \cdot d^{O(t)}\right)$ ,  $k \geqslant (\log n)^{1 / 5}$ , there exists an algorithm, running in time  $(nd)^{O(t)}$ , that outputs vectors  $\hat{\mu}_1, \ldots, \hat{\mu}_\ell$  satisfying
with high probability, for some permutation  $\pi : [k] \to [k]$ . Moreover, for  $\varepsilon \geqslant k^{-10}$ ,  $\delta \geqslant n^{-10}$ , the algorithm is  $(\varepsilon, \delta)$ -differentially private for any input  $Y$ .
We remark that our algorithm not only works for mixtures of Gaussians but for all mixtures of  $2t$ -explicitly bounded distributions (cf. Definition A.19).
Our algorithm is based on the sum-of-squares hierarchy and at the heart lies the following sum-of-squares program. The indeterminates  $z_{11},\ldots ,z_{1k},\ldots ,z_{nk}$  and vector-valued indeterminates  $\mu_1^{\prime},\dots,\mu_k^{\prime}$ , will be central to the proof of Theorem D.3. Let  $n,k,t$  be fixed parameters.
$$
\left\{ \begin{array}{l l} z _ {i \ell} ^ {2} = z _ {i \ell} & \forall i \in [ n ], \ell \in [ k ] \quad (\text {i n d i c a t o r s}) \\ \sum_ {\ell \in [ k ]} z _ {i \ell} \leqslant 1 & \forall i \in [ n ] \quad (\text {c l u s t e r m e m .}) \\ z _ {i \ell} \cdot z _ {i \ell^ {\prime}} = 0 & \forall i \in [ n ], \ell \in [ k ] \quad (\text {u n i q . m e m .}) \\ \sum_ {i} z _ {i \ell} \leqslant n / k & \forall \ell \in [ k ] \quad (\text {s i z e o f c l u s t e r s}) \\ \mu_ {\ell} ^ {\prime} = \frac {k}{n} \sum_ {i} z _ {i \ell} \cdot y _ {i} & \forall \ell \in [ k ] \quad (\text {m e a n s o f c l u s t e r s}) \\ \forall v \in \mathbb {R} ^ {d}: \frac {k}{n} \sum_ {i = 1} ^ {n} z _ {i \ell} \langle y _ {i} - \mu_ {\ell} ^ {\prime}, v \rangle^ {2 s} + \| Q v ^ {\otimes s} \| ^ {2} = (2 s) ^ {s} \cdot \| v \| _ {2} ^ {2 s} & \forall s \leqslant t, \ell \in [ k ] \quad (t m o m e n t) \\ & (\mathcal {P} _ {n, k, t} (Y)) \end{array} \right\}
$$
32We remark that we chose constants to optimize readability and not the smallest possible ones.
We remark that the moment constraint encodes the  $2t$ -explicit 2-boundedness constraint introduced in Definition A.19. Note that in the form stated above there are infinitely many constraints, one for each vector  $v$ . This is just for notational convenience. This constraint postulates equality of two polynomials in  $v$ . Formally, this can also be encoded by requiring there coefficients to agree and hence eliminating the variable  $v$ . It is not hard to see that this can be done adding only polynomially many constraints. Further, the matrix variable  $Q$  represents the SOS proof of the  $2t$ -explicit 2-boundedness constraint and we can hence deduce that for all  $0 \leq s \leq t$
$$
\mathcal {P} \left| \frac {v}{2 s} \left\{\frac {k}{n} \sum_ {i = 1} ^ {n} z _ {i \ell} \langle y _ {i} - \mu_ {\ell} ^ {\prime}, v \rangle^ {2 s} \leqslant (2 s) ^ {s} \| s \| _ {2} ^ {2 s} \right. \right\}.
$$
Before presenting the algorithm we will introduce some additional notation which will be convenient. We assume  $t, n, k$  to be fixed throughout the section and drop the corresponding subscripts. For  $Y \in \mathcal{V}$ , let  $\mathcal{Z}(Y)$  be the set of degree-10t pseudo-distributions satisfying  $\mathcal{P}(Y)$ . For each  $\zeta \in \mathcal{Z}(Y)$  define  $W(\zeta)$  as the  $n$ -by- $n$  matrix satisfying
$$
W (\zeta) _ {i j} = \tilde {\mathbb {E}} _ {\zeta} \left[ \sum_ {\ell \in [ k ]} z _ {i \ell} \cdot z _ {j \ell} \right].
$$
We let  $\mathcal{W}(Y) \coloneqq \{W(\zeta) \mid \zeta \in \mathcal{Z}(Y)\}$ .
Recall that  $J$  denotes the all-ones matrix. We define the function  $g: \mathbb{R}^{n \times n} \to \mathbb{R}$  as
$$
g (W) = \| W \| _ {\mathrm {F}} ^ {2} - (1 0) ^ {1 0} k ^ {3 0 0} \langle J, W \rangle
$$
$$
W (\hat {\zeta} (Y)) := \operatorname {a r g m i n} _ {W \in \mathcal {W} (Y)} g (W).
$$
We also consider the following function
Definition D.4 (Soft thresholding function). We denote by  $\phi :[0,1]\to [0,1]$  the function
$$
\phi (x) = \left\{ \begin{array}{l l} 0 & \text {i f} x \leqslant 0. 8  , \\ 1 & \text {i f} x \geqslant 0. 9  , \\ \frac {x - 0 . 8}{0 . 9 - 0 . 8} & \text {o t h e r w i s e}  . \end{array} \right.
$$
Notice that  $\phi(\cdot)$  is  $\frac{1}{0.9 - 0.8} = 10$  Lipschitz. Next we introduce our algorithm. Notice the algorithm relies on certain private subroutines. We describe them later in the section to improve the presentation.
Algorithm D.5 (Private algorithm for learning mixtures of Gaussians).
Input: Set of  $n$  points  $Y \subseteq \mathbb{R}^d$ ,  $\varepsilon, \delta > 0$ ,  $k, t \in \mathbb{N}$ ,  $d^* = 100 \log n$ ,  $b = k^{-15}$ .
1. Compute  $W = W(\hat{\zeta}(Y))$ .
2. Pick  $\pmb{\tau} \sim tLap\left(-n^{1.6}\left(1 + \frac{\log(1 / \delta)}{\varepsilon}\right), \frac{n^{1.6}}{\varepsilon}\right)$ .
3. If  $|\pmb{\tau}| \geqslant n^{1.7}$  or  $\| \phi(W)\|_1 \leqslant \frac{n^2}{k} \cdot \left(1 - \frac{1}{n^{0.1}} - \frac{1}{k^{100}}\right) + \pmb{\tau}$  reject.
4. For all  $i \in [n]$ , compute the  $n$ -dimensional vector
$$
\nu^ {(i)} = \left\{ \begin{array}{l l} \mathbf {0} & i f \| \phi (W _ {i}) \| _ {1} = 0 \\ \| \phi (W _ {i}) \| _ {1} ^ {- 1} \sum_ {j} \phi (W _ {i j}) \cdot y _ {j} & o t h e r w i s e. \end{array} \right.
$$
5. Pick a set  $\mathcal{S}$  of  $n^{0.01}$  indices  $i \in [n]$  uniformly at random.
6. For each  $i \in \mathcal{S}$  let  $\bar{\nu}^{(i)} = \nu^{(i)} + \mathbf{w}$  where  $\mathbf{w} \sim N\left(0, n^{-0.18} \cdot \frac{\log(2 / \delta)}{\varepsilon^2} \cdot \mathrm{Id}\right)$ .
7. Pick  $\Phi \sim N\left(0, \frac{1}{d^*}\right)^{d^* \times d}$ ,  $\mathbf{q} \stackrel{u.a.r.}{\sim}[0, b]$  and run the histogram learner of Lemma A.13 with input  $\Phi \bar{\nu}^{(1)}, \ldots, \Phi \bar{\nu}^{(n^{0.01})}$  and parameters
$$
\mathbf {q}, b, \alpha = k ^ {- 1 0}, \beta = n ^ {- 1 0}, \delta^ {*} = \frac {\delta}{n}, \varepsilon^ {*} = \varepsilon \cdot \frac {1 0 k ^ {5 0}}{n ^ {0 . 0 1}}.
$$
Let  $\mathbf{B}_1, \ldots, \mathbf{B}_k$  be the resulting  $d^*$ -dimensional bins with highest counts. Break ties randomly.
8. Reject if  $\min_{i\in [k]}\left|\left\{j\mid \Phi \bar{\pmb{\nu}}^{(j)}\in \mathbf{B}_i\right\} \right| <   \frac{n^{0.01}}{2k}.$
9. For each  $l \in [k]$  output
$$
\hat {\boldsymbol {\mu}} _ {l} := \frac {1}{| \left\{j \mid \Phi \bar {\boldsymbol {\nu}} ^ {(j)} \in \mathbf {B} _ {i} \right\} |} \cdot \left(\sum_ {\Phi \bar {\boldsymbol {\nu}} ^ {(j)} \in \mathbf {B} _ {l}} \bar {\boldsymbol {\nu}} ^ {(j)}\right) + \mathbf {w} ^ {\prime},
$$
where  $\mathbf{w}' \sim N\left(0, N\left(0, 32 \cdot k^{-120} \cdot \frac{\log(2kn / \delta)}{\varepsilon^2} \cdot \mathrm{Id}\right)\right)$ .
For convenience, we introduce some preliminary facts.
Definition D.6 (Good  $Y$ ). Let  $\mathbf{Y}$  be sampled according to Model 1.2. We say that  $\mathbf{Y}$  is good if:
1. for each  $\ell \in [k]$ , there are at least  $\frac{n}{k} - n^{0.6}$  and most  $\frac{n}{k} + n^{0.6}$  points sampled from  $D_{\ell}$  in  $\mathbf{Y}$ . Let  $\mathbf{Y}_{\ell} \subseteq \mathbf{Y}$  be such set of points.
2. Each  $\mathbf{Y}_{\ell}$  is 2t-explicitly 2-bounded.
It turns out that typical instances  $\mathbf{Y}$  are indeed good.
Lemma D.7 ([30, 36]). Consider the settings of Theorem D.3. Then  $\mathbf{Y}$  is good with high probability. Further, in this case the sets  $\mathcal{Z}(Y)$  and  $\mathcal{W}(Y)$  are non-empty.
D.1 Privacy analysis
In this section we show that our clustering algorithm is private.
Lemma D.8 (Differential privacy of the algorithm). Consider the settings of Theorem D.3. Then Algorithm D.5 is  $(\varepsilon, \delta)$ -differentially private.
We split our analysis in multiple steps and combine them at the end. On a high level, we will argue that on adjacent inputs  $Y, Y'$  many of the vectors  $\nu^{(i)}$  by the algorithm are close to each other and a small part can be very far. We can then show that we can mask this small difference using the
Gaussian mechanism and afterwards treat this subset of the vectors as privatized (cf. Lemma I.4). Then we can combine this with known histogram learners to deal with the small set of  $\nu^{(i)}$ 's that is far from each other on adjacent inputs.
D.1.1 Sensitivity of the matrix W
Here we use Lemma B.1 to reason about the sensitivity of  $\phi(W(\hat{\zeta}(Y)))$ . For adjacent datasets  $Y, Y' \in \mathcal{Y}$  we let  $\hat{\zeta}, \hat{\zeta}'$  be the pseudo-distribution corresponding to  $W(\hat{\zeta}(Y))$  and  $W(\hat{\zeta}(Y'))$  computed in step 1 of the algorithm, respectively. We prove the following result.
Lemma D.9 ( $\ell_1$ -sensitivity of  $\phi(W)$ ). Consider the settings of Theorem D.3. Let  $W, W'$  be respectively by the matrices computed in step 1 by Algorithm D.5 on adjacent inputs  $Y, Y' \in \mathcal{V}$ . Then
$$
\left\| \phi (W) - \phi (W ^ {\prime}) \right\| _ {1} \leqslant n ^ {1. 6}.
$$
For all but  $n^{0.8}$  rows  $i$  of  $\phi (W),\phi (W^{\prime})$  , it holds
$$
\left\| \phi (W) _ {i} - \phi (W ^ {\prime}) _ {i} \right\| _ {1} \leqslant n ^ {0. 8}.
$$
Proof. The second inequality is an immediate consequence of the first via Markov's inequality. Thus it suffices to prove the first. Since  $\phi(\cdot)$  is 10-Lipschitz, we immediately obtain the result if
$$
\left. \left\| W (\hat {\zeta} (Y)) - W (\hat {\zeta} (Y ^ {\prime})) \right\| _ {1} \leqslant n ^ {1. 5 5} \right..
$$
Thus we focus on this inequality. To prove it, we verify the two conditions of Lemma B.1. First notice that  $g$  is 2-strongly convex with respect to its input  $W$ . Indeed for  $W, W' \in \mathcal{W}(Y)$ , since  $\forall i, j \in [n]$ ,  $W_{ij} \geqslant 0$  it holds that
$$
\begin{array}{l} \| W ^ {\prime} \| _ {\mathrm {F}} ^ {2} = \| W \| _ {\mathrm {F}} ^ {2} + \| W - W ^ {\prime} \| _ {\mathrm {F}} ^ {2} + 2 \langle W ^ {\prime} - W, W \rangle \\ = \| W \| _ {\mathrm {F}} ^ {2} + \| W - W ^ {\prime} \| _ {\mathrm {F}} ^ {2} + 2 \langle W ^ {\prime} - W, W \rangle + \langle W ^ {\prime} - W, (1 0) ^ {1 0} k ^ {3 0 0} (J - J) \rangle \\ = g (W) + \| W - W ^ {\prime} \| _ {F} ^ {2} + \langle W ^ {\prime} - W, \nabla g (W) \rangle + \langle W ^ {\prime}, (1 0) ^ {1 0} k ^ {3 0 0} J \rangle , \\ \end{array}
$$
where we used that  $\nabla g(W) = 2W - (10)^{10}k^{300}J$ . Thus it remains to prove  $(i)$  of Lemma B.1.
Let  $\hat{\zeta} \in \mathcal{Z}(Y)$ ,  $\hat{\zeta}' \in \mathcal{Z}(Y')$  be the pseudo-distributions such that  $W_{Y}(\hat{\zeta}) = W$  and  $W_{Y}(\hat{\zeta}') = W'$ . We claim that there always exists  $\zeta_{\mathrm{adj}} \in \mathcal{Z}(Y) \cap \mathcal{Z}(Y')$  such that
1.  $|g(W(\zeta)) - g(W(\zeta_{\mathrm{adj}})| \leqslant \frac{2n}{k} \cdot ((10)^{10}k^{300} + 1) \leqslant 3 \cdot (10)^{10}k^{300}n$ ,
2.  $|g_{Y^{\prime}}(W(\zeta_{\mathrm{adj}})) - g(W(\zeta_{\mathrm{adj}})| = 0$
Note that in this case the second point is always true since  $g$  doesn't depend on  $Y$ . Together with Lemma B.1 these two inequalities will imply that
$$
\left\| W (\hat {\zeta} (Y)) - W (\hat {\zeta} (Y ^ {\prime})) \right\| _ {F} ^ {2} \leqslant 1 8 \cdot (1 0) ^ {1 0} k ^ {3 0 0} n.
$$
By assumption on  $n$ , an application of Cauchy-Schwarz will give us the desired result.
So, let  $i$  be the index at which  $Y, Y'$  differ. We construct  $\zeta_{\mathrm{adj}}$  as follows: for all polynomials  $p$  of degree at most  $10t$  we let
$$
\tilde {\mathbb {E}} _ {\zeta_ {\mathrm {a d j}}} [ p ] = \left\{ \begin{array}{l l} \tilde {\mathbb {E}} _ {\zeta} [ p ] & \text {i f} p \text {d o e s n o t c o n t a i n v a r i a b l e s} z _ {i \ell} \text {f o r a n y} \ell \in [ k ] \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
By construction  $\zeta_{\mathrm{adj}} \in \mathcal{Z}(Y) \cap \mathcal{Z}(Y')$ . Moreover,  $W(\zeta), W(\zeta_{\mathrm{adj}})$  differ in at most  $2n / k$  entries. Since all entries of the two matrices are in  $[0,1]$ , the first inequality follows by definition of the objective function.
D.1.2 Sensitivity of the resulting vectors
In this section we argue that if the algorithm does not reject in step 3 then the vectors  $\nu^{(i)}$  are stable on adjacent inputs. Concretely our statement goes as follows:
Lemma D.10 (Stability of the  $\nu^{(i)}$ 's). Consider the settings of Theorem D.3. Suppose Algorithm D.5 does not reject in step 3, on adjacent inputs  $Y$ ,  $Y' \in \mathcal{V}$ . Then for all but  $\frac{6n}{k^{50}}$  indices  $i \in [n]$ , it holds:
$$
\left\| \nu_ {Y} ^ {(i)} - \nu_ {Y ^ {\prime}} ^ {(i)} \right\| _ {2} \leqslant O (n ^ {- 0. 1}).
$$
The proof of Lemma D.10 crucially relies on the next statement.
Lemma D.11 (Covariance bound). Consider the settings of Theorem D.3. Let  $W$  be the matrix computed by Algorithm D.5 on input  $Y \in \mathcal{V}$ . For  $i \in [n]$ , if  $\| \phi(W_i) \|_1 \geqslant \frac{n}{k} \cdot \left(1 - \frac{10}{k^{50}}\right)$  then  $\nu^{(i)}$  induces a 2-explicitly 40-bounded distribution over  $Y$ .
Proof. First, by assumption notice that there must be at least  $\frac{n}{k} \cdot \left(1 - \frac{10}{k^{50}}\right)$  entries of  $\phi(W_i)$  larger than 0.8. We denote the set of  $j \in [n]$  such that  $W_{ij} \geqslant 0.8$  by  $\mathcal{G}$ . Let  $\zeta \in \mathcal{Z}(Y)$  be the degree 10t pseudo-distribution so that  $W = W(\zeta(Y))$ . Since  $\zeta$  satisfies  $\mathcal{P}(Y)$ , for  $\ell \in [k]$  it follows from the moment bound constraint for  $s = 1$  that for all unit vectors  $u$  it holds that
$$
\mathcal {P} \left. \right| _ {4} \left\{0 \leqslant \frac {k}{n} \sum_ {j = 1} ^ {n} z _ {j \ell} \langle \mathbf {y} _ {j} - \mu_ {l} ^ {\prime}, u \rangle^ {2} \leqslant 2 \right\},
$$
Using the SOS triangle inequality (cf. Fact I.2)  $\left| \frac{a, b}{2} (a + b)^2 \leqslant 2(a^2 + b^2) \right|$  it now follows that
$$
\mathbf {0} \preceq \tilde {\mathbb {E}} _ {\zeta} \left[ \frac {k ^ {2}}{n ^ {2}} \sum_ {j, j ^ {\prime} \in [ n ]} z _ {j \ell} z _ {j ^ {\prime} \ell} \cdot \left(y _ {j} - y _ {j ^ {\prime}}\right) ^ {\otimes 2} \right] \preceq 8 \mathrm {I d}
$$
$$
\mathbf {0} \preceq \tilde {\mathbb {E}} _ {\zeta} \left[ \frac {k ^ {2}}{n ^ {2}} \sum_ {\ell \in [ k ]} \sum_ {j, j ^ {\prime} \in [ n ]} z _ {i \ell} z _ {j \ell} z _ {j ^ {\prime} \ell} \cdot \left(y _ {j} - y _ {j ^ {\prime}}\right) ^ {\otimes 2} \right] \preceq 8 \mathrm {I d}.
$$
Furthermore using  $\mathcal{P}(Y)\left|\frac{1}{2}\left\{z_{i\ell}z_{i\ell^{\prime}} = 0\right\}\right.$  for  $\ell \neq \ell^{\prime}$  we have
$$
\tilde {\mathbb {E}} _ {\zeta} \left[ \sum_ {\ell \in [ k ]} \sum_ {j, j ^ {\prime} \in [ n ]} z _ {i \ell} z _ {j \ell} z _ {j ^ {\prime} \ell} \right] = \tilde {\mathbb {E}} _ {\zeta} \left[ \left(\sum_ {\ell \in [ k ], j \in [ n ]} z _ {i \ell} z _ {j \ell}\right) \cdot \left(\sum_ {\ell^ {\prime} \in [ k ], j ^ {\prime} \in [ n ]} z _ {i \ell^ {\prime}} z _ {j ^ {\prime} \ell^ {\prime}}\right) \right].
$$
Now, for fixed  $j,j^{\prime}\in [n]$  , using
$$
\left\{a ^ {2} = a, b ^ {2} = b \right\} \mid_ {\overline {{O (1)}}} \left\{1 + a b - a - b = 1 - a b - (a - b) ^ {2} \geqslant 0 \right\}
$$
with  $a = \sum_{\ell \in [k]}z_{i\ell}z_{j\ell}$  and  $b = \sum_{\ell^{\prime}\in [k]}z_{i\ell^{\prime}}z_{j^{\prime}\ell^{\prime}}$  we get
$$
\begin{array}{l} \tilde {\mathbb {E}} _ {\zeta} \left[ \left(\sum_ {\ell \in [ k ]} z _ {i \ell} z _ {j \ell}\right) \left(\sum_ {\ell^ {\prime} \in [ k ]} z _ {i \ell^ {\prime}} z _ {j ^ {\prime} \ell^ {\prime}}\right) \right] \geqslant \tilde {\mathbb {E}} _ {\zeta} \left[ \sum_ {\ell \in [ k ]} z _ {i \ell} z _ {j \ell} + \sum_ {\ell^ {\prime} \in [ k ]} z _ {i \ell^ {\prime}} z _ {j ^ {\prime} \ell^ {\prime}} \right] - 1 \\ = W _ {i j} + W _ {i j ^ {\prime}} - 1. \\ \end{array}
$$
Now if  $j,j^{\prime}\in \mathcal{G}$  we must have
$$
\sum_ {\ell \in [ k ]} \tilde {\mathbb {E}} _ {\zeta} \left[ z _ {i \ell} z _ {j \ell} z _ {j ^ {\prime} \ell} \right] = \tilde {\mathbb {E}} _ {\zeta} \left[ \left(\sum_ {\ell \in [ k ]} z _ {i \ell} z _ {j \ell}\right) \left(\sum_ {\ell^ {\prime} \in [ k ]} z _ {i \ell^ {\prime}} z _ {j ^ {\prime} \ell^ {\prime}}\right) \right] \geqslant 0. 6.
$$
Since  $\phi(W_{ij}) \leqslant 1$  by definition and  $\| \phi(W_i) \|_1 \geqslant \frac{n}{k} \cdot \left(1 - \frac{10}{k^{50}}\right)$ , we conclude
$$
\| \phi (W _ {i}) \| _ {1} ^ {- 2} \left[ \sum_ {j, j ^ {\prime} \in [ n ]} \phi (W _ {i j}) \phi (W _ {i j ^ {\prime}}) \left(y _ {j} - y _ {j ^ {\prime}}\right) ^ {\otimes 2} \right]
$$
$$
\begin{array}{l} \preceq 5 \cdot \frac {k ^ {2}}{n ^ {2}} \sum_ {j, j ^ {\prime} \in [ n ], \ell \in [ k ]} \tilde {\mathbb {E}} _ {\zeta} \left[ z _ {i \ell} z _ {j \ell} z _ {j ^ {\prime} \ell} \right] \cdot \left(y _ {j} - y _ {j ^ {\prime}}\right) ^ {\otimes 2} \\ \succ 4 0 \mathrm {I d}. \\ \end{array}
$$
We can now prove Lemma D.10.
Proof of Lemma D.10. Let  $W, W'$  be the matrices computed by Algorithm D.5 in step 1 on input  $Y, Y'$ , respectively. Let  $\mathcal{G} \subseteq [n]$  be the set of indices  $i$  such that
$$
\left\| \phi (W) _ {i} - \phi \left(W ^ {\prime}\right) _ {i} \right\| _ {1} \leqslant n ^ {0. 8}.
$$
Notice that  $|\mathcal{G}| \geqslant n - n^{0.8}$  by Lemma D.9. Since on input  $Y$  the algorithm did not reject in step 3 we must have
$$
\| \phi (W) \| _ {1} \geqslant \frac {n ^ {2}}{k} \cdot \left(1 - \frac {1}{n ^ {0 . 1}} - \frac {1}{k ^ {1 0 0}}\right) - n ^ {1. 7} \geqslant \frac {n ^ {2}}{k} \cdot \left(1 - \frac {2}{k ^ {1 0 0}}\right).
$$