SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {P} \left| ^ {\frac {1 0 t}{2}} \left\{z _ {l} \left(\mathbf {C} _ {r}\right) \cdot \frac {1}{\Delta^ {2 t}} \sum_ {r ^ {\prime} \neq r} z _ {l} \left(\mathbf {C} _ {r ^ {\prime}}\right) \langle \mu_ {r} - \mu_ {r ^ {\prime}}, v \rangle^ {2 t} \right. \right. \\ \leqslant \frac {2 ^ {4 t - 1}}{\Delta^ {2 t}} \cdot \left(k \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \mathbf {y} _ {i} - \mu_ {\pi (i)}, v \rangle^ {2 t} + \frac {k}{n} \sum_ {i = 1} ^ {n} z _ {i, l} \langle \mathbf {y} _ {i} - \mu_ {l} ^ {\prime}, v \rangle^ {2 t}\right) \}. \\ \end{array}
$$
We start by bounding the first sum. Recall that by assumption the uniform distribution over each true cluster is  $2t$ -explicitly 2-bounded. It follows that
$$
\begin{array}{l} \left| \frac {1}{2 t} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \mathbf {y} _ {i} - \mu_ {\pi (i)}, v \rangle^ {2 t} = \frac {1}{k} \sum_ {r = 1} ^ {k} \frac {k}{n} \sum_ {i \in \mathbf {C} _ {r}} \langle \mathbf {y} _ {i} - \mu_ {r}, v \rangle^ {2 t} \leqslant \frac {1}{k} \sum_ {r = 1} ^ {k} \frac {k}{n} \cdot | \mathbf {C} _ {r} | \cdot (2 t) ^ {t} \cdot \| v \| _ {2} ^ {2 t} \right. \right. (I.3) \\ \leqslant \left(1 + \frac {k}{n ^ {0 . 4}}\right) \cdot (2 t) ^ {t} \leqslant 2 (2 t) ^ {t} \}, (I.4) \\ \end{array}
$$
where we used that  $|\mathbf{C}_r| \leqslant \frac{n}{k} + n^{0.6}$ . To bound the second sum, we will use the moment bound constraints. In particular, we know that
$$
\mathcal {P} \left| \overline {{1 0 t}} \left\{\frac {k}{n} \sum_ {i = 1} ^ {n} z _ {i, l} \left\langle \mathbf {y} _ {i} - \mu_ {l} ^ {\prime}, v \right\rangle^ {2 t} \leqslant (2 t) ^ {t} \right\}. \right. \tag {I.5}
$$
Combining Eq. (I.4) and Eq. (I.5) now yields
$$
\mathcal {P} \left| \right. \overline {{1 0 t}} \left\{z _ {l} (\mathbf {C} _ {r}) \cdot \frac {1}{\Delta^ {2 t}} \sum_ {r ^ {\prime} \neq r} z _ {l} (\mathbf {C} _ {r ^ {\prime}}) \langle \mu_ {r} - \mu_ {r ^ {\prime}}, v \rangle^ {2 t} \leqslant k \frac {2 ^ {2 t + 1} (2 t) ^ {t}}{\Delta^ {2 t}} \leqslant k \left(\frac {8 t}{\Delta^ {2}}\right) ^ {t} \right\}.
$$
Note that by assumption  $\Delta \geqslant O(\sqrt{tk^{1 / t}})$ . Overloading notation, we can choose the  $t$  parameter in the SOS proof to be 202 times the  $t$  parameter in the lower bound in the separation to obtain<sup>35</sup>
$$
\sum_ {i \in \mathbf {C} _ {r}, j \notin \mathbf {C} _ {r}} \mathbf {W} _ {i, j} \leqslant \delta \cdot \frac {n ^ {2}}{k}.
$$
I.1 Small Lemmas
Fact I.2 (Lemma A.2 in [36]). For all integers  $t > 0$  it holds that
$$
\left| \frac {x , y}{2 t} (x + y) ^ {2 t} \leqslant 2 ^ {2 t - 1} \left(x ^ {2 t} + y ^ {2 t}\right). \right.
$$
Fact I.3. Let  $\varepsilon, \delta > 0$ . Let  $\mathcal{M} \colon \mathcal{Y} \to \mathcal{O}$  be a randomized algorithm that, for every pair of adjacent inputs, with probability at least  $1 - \gamma \geqslant 1/2$  over the internal randomness of  $\mathcal{Y}^{36}$  satisfies  $(\varepsilon, \delta)$ -privacy. Then  $\mathcal{M}$  is  $(\varepsilon + 2\gamma, \delta + \gamma)$ -private.
Proof. Let  $X, X'$  be adjacent input and let  $B$  be the event under which  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -private. By assumption, we know that  $\mathbb{P}(B) \geqslant 1 - \gamma$ . Let  $S \in \mathcal{O}$ , it follows that
$$
\begin{array}{l} \mathbb {P} \left(\mathcal {M} (X) \in S\right) = \mathbb {P} (B) \cdot \mathbb {P} \left(\mathcal {M} (X) \in S \mid B\right) + \mathbb {P} \left(B ^ {c}\right) \cdot \mathbb {P} \left(\mathcal {M} (X) \in S \mid B ^ {c}\right) \\ \leqslant \mathbb {P} (\mathcal {M} (X) \in S \mid B) + \gamma \\ \leqslant e ^ {\varepsilon} \mathbb {P} (\mathcal {M} (X) \in S \mid B) + \delta + \gamma \\ \leqslant \frac {e ^ {\varepsilon}}{\mathbb {P} (B)} \cdot \mathbb {P} (\mathcal {M} (X) \in S) + \delta + \gamma \\ \leqslant e ^ {\varepsilon + \log \left(\frac {1}{1 - \gamma}\right)} \cdot \mathbb {P} (\mathcal {M} (X) \in S) + (\delta + \gamma) \\ \leqslant e ^ {\varepsilon + 2 \gamma} \cdot \mathbb {P} (\mathcal {M} (X) \in S) + (\delta + \gamma), \\ \end{array}
$$
where we used that  $\log (1 - \gamma)\geqslant -2\gamma$  for  $\gamma \in [0,1 / 2]$
I.2 Privatizing input using the Gaussian Mechanism
In this section, we will proof the following helpful lemma used in the privacy analysis of our clustering algorithm (Algorithm D.5). In summary, it says that when restricted to some set our input has small  $\ell_2$  sensitivity, we can first add Gaussian noise proportional to this sensitivity and afterwards treat this part of the input as "privatized". In particular, for the remainder of the privacy analysis we can treat this part as the same on adjacent inputs. Note that we phrase the lemma in terms of matrix inputs since this is what we use in our application. Of course, it also holds for more general inputs.
35Note that this influences the exponent in the running time and sample complexity only by a constant factor and hence doesn't violate the assumptions of Theorem D.3.
36 In particular, this randomness is independent of the input
Lemma I.4. Let  $V, V' \in \mathbb{R}^{n \times d}$ ,  $m \in [n]$  and  $\Delta > 0$  be such that there exists a set  $S$  of size at least  $n - m$  satisfying
$$
\forall i \in S. \| V _ {i} - V _ {i} ^ {\prime} \| _ {2} ^ {2} \leqslant \Delta^ {2},
$$
where  $V_{i}, V_{i}^{\prime}$  denote the rows of  $V, V^{\prime}$ , respectively. Let  $\mathcal{A}_2\colon \mathbb{R}^{n\times d}\to \mathcal{O}$  be an algorithm that is  $(\varepsilon_2,\delta_2)$ -differentially private in the standard sense, i.e., for all sets  $S\subseteq \mathcal{O}$  and datasets  $X,X^{\prime}\in \colon \mathbb{R}^{n\times d}$  differing only in a single row it holds that
$$
\mathbb {P} \left(\mathcal {A} _ {2} (X) \in S\right) \leqslant e ^ {\varepsilon_ {2}} \mathbb {P} \left(\mathcal {A} _ {2} \left(X ^ {\prime}\right) \in S\right) + \delta_ {2}.
$$
Further, let  $\mathcal{A}_1\colon \mathbb{R}^{n\times d}\to \mathbb{R}^{n\times d}$  be the Gaussian Mechanism with parameters  $\Delta ,\varepsilon_1,\delta_1$  . I.e., on input  $M$  it samples  $\mathbf{W}\sim N\left(0,2\Delta^2\cdot \frac{\log(2 / \delta_1)}{\varepsilon_1^2}\right)^{n\times d}$  and outputs  $M + \mathbf{W}$
$$
\varepsilon^ {\prime} := \varepsilon_ {1} + m \varepsilon_ {2},
$$
$$
\delta^ {\prime} := e ^ {\varepsilon_ {1}} m e ^ {(m - 1) \varepsilon_ {2}} \delta_ {2} + \delta_ {1}.
$$
$\mathcal{A}_2 \circ \mathcal{A}_1$  is  $(\varepsilon', \delta')$ -differentially private with respect to  $V$  and  $V'$ , i.e., for all sets  $\mathcal{S} \subseteq \mathcal{O}$  it holds that
$$
\mathbb {P} \left(\left(\mathcal {A} _ {2} \circ \mathcal {A} _ {1}\right) (V) \in S\right) \leqslant e ^ {\varepsilon^ {\prime}} \mathbb {P} \left(\left(\mathcal {A} _ {2} \circ \mathcal {A} _ {1}\right) \left(V ^ {\prime}\right) \in S\right) + \delta^ {\prime}.
$$
Proof. Without loss of generality, assume that  $S = \{1, \dots, m\}$ . Denote by  $V_{1}, V_{2}$  the first  $m$  and last  $n - m$  rows of  $V$  respectively. Analogously for  $V_{1}', V_{2}'$ . We will later partit in the noise  $\mathbf{W}$  of the Gaussian mechanism in the same way. Further, for a subset  $A$  of  $\mathbb{R}^{n \times n}$  and  $Y \in \mathbb{R}^{m \times n}$  define
$$
T _ {A, Y} = \left\{X \in \mathbb {R} ^ {(n - m) \times n} \middle | \left( \begin{array}{c} X \\ Y \end{array} \right) \in A \right\} \subseteq \mathbb {R} ^ {(n - m) \times n}.
$$
Note that  $\left( \begin{array}{c}X\\ Y \end{array} \right)\in A$  if and only if  $X\in T_{A,Y}$
Let  $S \subseteq \mathcal{O}$ . It now follows that
$$
\begin{array}{l} \mathbb {P} _ {\mathcal {A} _ {2}, \mathbf {W}} \left[ \left(\mathcal {A} _ {2} \circ \mathcal {A} _ {1}\right) (V) \in S \right] = \underset {\mathcal {A} _ {2}, \mathbf {W}} {\mathbb {E}} \left[ \upharpoonright \left\{V + \mathbf {W} \in \mathcal {A} _ {2} ^ {- 1} (S) \right\} \right] \\ = \underset {\mathcal {A} _ {2}, \mathbf {W} _ {2}} {\mathbb {E}} \left[ \underset {\mathbf {W} _ {1}} {\mathbb {E}} \left[ \mathbb {1} \left\{\binom {V _ {1} + \mathbf {W} _ {1}} {V _ {2} + \mathbf {W} _ {2}} \in \mathcal {A} _ {2} ^ {- 1} (S) \right\} \right] \mid \mathbf {W} _ {2} \right] \\ = \underset {\mathcal {A} _ {2}, \mathbf {W} _ {2}} {\mathbb {E}} \left[ \underset {\mathbf {W} _ {1}} {\mathbb {E}} \left[ \nVdash \left\{V _ {1} + \mathbf {W} _ {1} \in T _ {\mathcal {A} _ {2} ^ {- 1} (S), V _ {2} + \mathbf {W} _ {2}} \right\} \right] \mid \mathbf {W} _ {2} \right] \\ \leqslant e ^ {\varepsilon_ {1}} \cdot \underset {\mathcal {A} _ {2}, \mathbf {W} _ {2}} {\mathbb {E}} \left[ \underset {\mathbf {W} _ {1}} {\mathbb {E}} \left[ \nVdash \left\{V _ {1} ^ {\prime} + \mathbf {W} _ {1} \in T _ {\mathcal {A} _ {2} ^ {- 1} (S), V _ {2} + \mathbf {W} _ {2}} \right\} \right] \mid \mathbf {W} _ {2} \right] + \delta_ {1} \\ = e ^ {\varepsilon_ {1}} \cdot \underset {\mathcal {A} _ {2}, \mathbf {W}} {\mathbb {E}} \left[ \mathbb {1} \left\{\left( \begin{array}{c} V _ {1} ^ {\prime} + \mathbf {W} _ {1} \\ V _ {2} + \mathbf {W} _ {2} \end{array} \right) \in \mathcal {A} _ {2} ^ {- 1} (S) \right\} \right] + \delta_ {1}, \\ \end{array}
$$
where the inequality follows by the guarantees of the Gaussian Mechanism. Further, we can bound
$$
\begin{array}{l} \underset {\mathcal {A} _ {2}, \mathbf {W}} {\mathbb {E}} \left[ \mathbb {1} \left\{\left( \begin{array}{c} V _ {1} ^ {\prime} + \mathbf {W} _ {1} \\ V _ {2} + \mathbf {W} _ {2} \end{array} \right) \in \mathcal {A} _ {2} ^ {- 1} (S) \right\} \right] = \underset {\mathbf {W}} {\mathbb {E}} \left[ \underset {\mathcal {A} _ {2}} {\mathbb {E}} \left[ \mathbb {1} \left\{\mathcal {A} _ {2} \left( \begin{array}{c} V _ {1} ^ {\prime} + \mathbf {W} _ {1} \\ V _ {2} + \mathbf {W} _ {2} \end{array} \right) \in S \right\} \mid \mathbf {W} \right] \right] \\ \leqslant e ^ {m \varepsilon_ {2}} \cdot \underset {\mathbf {W}} {\mathbb {E}} \left[ \underset {\mathcal {A} _ {2}} {\mathbb {E}} \left[ \nVdash \left\{\mathcal {A} _ {2} \left( \begin{array}{c} V _ {1} ^ {\prime} + \mathbf {W} _ {1} \\ V _ {2} ^ {\prime} + \mathbf {W} _ {2} \end{array} \right) \in S \right\} \mid \mathbf {W} \right] \right] + m e ^ {(m - 1) \varepsilon_ {2}} \delta_ {2} \\ = e ^ {m \varepsilon_ {2}} \cdot \underset {\mathcal {A} _ {2}, \mathbf {W}} {\mathbb {E}} \left[ \mathbb {1} \left\{\left( \begin{array}{c} V _ {1} ^ {\prime} + \mathbf {W} _ {1} \\ V _ {2} ^ {\prime} + \mathbf {W} _ {2} \end{array} \right) \in \mathcal {A} _ {2} ^ {- 1} (S) \right\} \right] + m e ^ {(m - 1) \varepsilon_ {2}} \delta_ {2}, \\ \end{array}
$$
where the inequality follows by the privacy guarantees of  $\mathcal{A}_2$  combined with standard group privacy arguments.
Putting the above two displays together and plugging in the definition of  $\varepsilon^{\prime},\delta^{\prime}$  we finally obtain
$$
\mathbb {P} _ {\mathcal {A} _ {2}, \mathbf {W}} \left[ \left(\mathcal {A} _ {2} \circ \mathcal {A} _ {1}\right) (V) \in S \right] \leqslant e ^ {\varepsilon^ {\prime}} \mathbb {P} _ {\mathcal {A} _ {2}, \mathbf {W}} \left[ \left(\mathcal {A} _ {2} \circ \mathcal {A} _ {1}\right) \left(V ^ {\prime}\right) \in S \right] + \delta^ {\prime}.
$$