SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
E _ {j} ^ {(m)} := \left\{| \hat {f} _ {j} ^ {(m)} - f _ {j} ^ {(m)} | \leq C \cdot \sqrt {\frac {\log (1 / p)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {(m)}\right) ^ {2}} \right\}, m \in [ M ], j \in [ d ].
$$
Then by Proposition 2.1 we have
$$
\mathbb {P} \left\{E _ {j} ^ {(m)} \right\} \geq 1 - p.
$$
$$
\mathbb {P} \bigg \{\bigcap_ {m = 1} ^ {M} E _ {j} ^ {(m)} \bigg \} \geq 1 - M p.
$$
Conditional on the event of  $\bigcap_{m=1}^{M} E_j^{(m)}$ , we know that every random variable  $\hat{f}_j^{(m)} - f_j^{(m)}$  is bounded within
$$
\big (- F ^ {(m)}, F ^ {(m)} \big),
$$
$$
F ^ {(m)} := C \cdot \sqrt {\frac {\log (1 / p)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {(m)}\right) ^ {2}}.
$$
So by Hoeffding inequality, we have
$$
\mathbb {P} \left\{\left| \frac {1}{M} \sum_ {m = 1} ^ {M} \hat {f} _ {j} ^ {(m)} - \frac {1}{M} \sum_ {m = 1} ^ {M} f _ {j} ^ {(m)} \right| \leq \sqrt {\frac {\log (2 / p _ {1})}{2 M ^ {2}} \sum_ {m = 1} ^ {M} \left(F ^ {(m)}\right) ^ {2}} \Bigg | \bigcap_ {m = 1} ^ {M} E _ {j} ^ {(m)} \right\} \geq 1 - p _ {1}
$$
$$
\mathbb {P} \left\{\left| \frac {1}{M} \sum_ {m = 1} ^ {M} \hat {f} _ {j} ^ {(m)} - \frac {1}{M} \sum_ {m = 1} ^ {M} f _ {j} ^ {(m)} \right| \leq \sqrt {\frac {\log (2 / p _ {1})}{2 M ^ {2}} \sum_ {m = 1} ^ {M} \left(F ^ {(m)}\right) ^ {2}} \right\} \geq 1 - p _ {1} - M p.
$$
$$
\begin{array}{l} \frac {\log (2 / p _ {1})}{2 M ^ {2}} \sum_ {m = 1} ^ {M} \left(F ^ {(m)}\right) ^ {2} = \frac {\log (2 / p _ {1})}{2 M ^ {2}} \sum_ {m = 1} ^ {M} C ^ {2} \cdot \frac {\log (1 / p)}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(f _ {i} ^ {(m)}\right) ^ {2} \\ = C ^ {2} \cdot \frac {\log (2 / p _ {1}) \log (1 / p)}{2 L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(F _ {i}\right) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left\{\left| \frac {1}{M} \sum_ {m = 1} ^ {M} \hat {f} _ {j} ^ {(m)} - \frac {1}{M} \sum_ {m = 1} ^ {M} f _ {j} ^ {(m)} \right| \leq \sqrt {C ^ {2} \cdot \frac {\log (2 / p _ {1}) \log (1 / p)}{2 L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(F _ {i}\right) ^ {2}} \right\} \\ \geq 1 - p _ {1} - M p. \\ \end{array}
$$
Note replace  $p_1 = p' / 2$  and  $p = p' / (2M)$ , we have that
$$
\mathbb {P} \bigg \{\left| \frac {1}{M} \sum_ {m = 1} ^ {M} \hat {f} _ {j} ^ {(m)} - \frac {1}{M} \sum_ {m = 1} ^ {M} f _ {j} ^ {(m)} \right| \leq \sqrt {C ^ {\prime} \cdot \frac {\log (1 / p ^ {\prime}) \log (M / p ^ {\prime})}{L} \cdot \frac {1}{W} \cdot \sum_ {i > W} \left(F _ {i}\right) ^ {2}} \bigg \} \geq 1 - p ^ {\prime}.
$$
Proof of Theorem 3.2. Let us consider the hybrid sketch approach in Algorithm 2. Recall that within a round, clients use the same set of hash functions to construct their sketching matrices. Across different rounds, clients use the same set of location hashes but a fresh set of sign hashes. Denote the hash functions by:
$$
h _ {\ell}: [ d ] \to [ w ], \quad \ell = 1, \dots , L;
$$
$$
\sigma_ {\ell} ^ {(m)}: [ d ] \rightarrow \{+ 1, - 1 \}, \quad \ell = 1, \dots , L; m = 1 \dots , M.
$$
Recall the local frequency in each round is defined by
$$
\mathbf {f} ^ {(m)} := \frac {1}{n} \sum_ {t = 1} ^ {n} \mathbf {x} ^ {(m, t)}, \quad m = 1, \ldots , M.
$$
And the global frequency vector is defined by
$$
\mathbf {f} := \frac {1}{M} \sum_ {m = 1} ^ {M} \mathbf {f} ^ {(m)}.
$$
Then according to the communication protocol, the server receives  $M$  sketching matrices (each corresponds to a summation of clients' sketches within the same round). From the  $m$ -th sketch, we can extract  $L$  estimators for each index  $j \in [d]$ , i.e.,
$$
\begin{array}{l} \tilde {\mathbf {f}} _ {j} ^ {(m, \ell)} := \sum_ {i = 1} ^ {d} \mathbb {1} \left[ h _ {\ell} (i) = h _ {\ell} (j) \right] \cdot \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)}, \qquad j \in [ d ], m \in [ M ], \ell \in [ L ] \\ = \mathbf {f} _ {j} ^ {(m)} + \sum_ {i \neq j} \mathbb {1} [ h _ {\ell} (i) = h _ {\ell} (j) ] \cdot \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)}. \\ \end{array}
$$
For each index, we will first average the estimators from different rounds to reduce the variance, then take the median over different rows to amplify the success probability. In particular, denote the round-wise averaging by
$$
\begin{array}{l} \tilde {\mathbf {f}} _ {j} ^ {(\ell)} := \frac {1}{M} \sum_ {m = 1} ^ {M} \tilde {\mathbf {f}} _ {j} ^ {(m, \ell)}, \qquad j \in [ d ],   \ell \in [ L ] \\ = \frac {1}{M} \sum_ {m = 1} ^ {M} \mathbf {f} _ {j} ^ {(m)} + \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {i \neq j} \mathbb {1} [ h _ {\ell} (i) = h _ {\ell} (j) ] \cdot \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)} \\ \end{array}
$$
$$
\begin{array}{l} = \underbrace{\mathbf{f}_{j}}_{\text{signal}} + \underbrace{\frac{1}{M}\sum_{i\neq j}\mathbb{1}\left[h_{\ell}(i) = h_{\ell}(j)\right]\cdot\sum_{m = 1}^{M}\sigma_{\ell}^{(m)}(j)\cdot\sigma_{\ell}^{(m)}(i)\cdot\mathbf{f}_{i}^{(m)}}_{\text{noise}} \\ = \underbrace{\mathbf{f}_{j}}_{\text{signal}} + \underbrace{\frac{1}{M}\sum_{i\neq j,i\in\mathbb{W}}\mathbb{1}\left[h_{\ell}(i) = h_{\ell}(j)\right]\cdot\sum_{m = 1}^{M}\sigma_{\ell}^{(m)}(j)\cdot\sigma_{\ell}^{(m)}(i)\cdot\mathbf{f}_{i}^{(m)}}_{\text{headNoise}} \\ + \underbrace {\frac {1}{M} \sum_ {i \neq j , i \notin \mathbb {W}} \mathbb {1} \left[ h _ {\ell} (i) = h _ {\ell} (j) \right] \cdot \sum_ {m = 1} ^ {M} \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)}} _ {\text {t a i l N o i s e}}. \tag {5} \\ \end{array}
$$
Then we take the median over these estimators to obtain
$$
\tilde {\mathbf {f}} _ {j} := \operatorname {m e d i a n} \{\tilde {\mathbf {f}} _ {j} ^ {(\ell)}, \ell \in [ L ] \}, \quad j \in [ d ].
$$
Head Noise. The only randomness comes from the algorithm. Note that the head noise contains at most  $|\mathbb{W}| \leq 0.1W$  independent terms, and each is zero with probability  $1 - 1 / W$ . Thus the head noise is zero with probability at least  $(1 - 1 / W)^{|\mathbb{W}|} \geq (1 - 1 / W)^{0.1W} \geq 0.9$  provided that  $W > 10$ .
Tail Noise. Now consider the second noise term in (5). Fixing  $\ell$  and  $j$ . Define
$$
\xi_ {i} ^ {(m)} := \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)}
$$
$$
\xi_ {i} := \sum_ {m = 1} ^ {M} \xi_ {i} ^ {(m)} = \sum_ {m = 1} ^ {M} \sigma_ {\ell} ^ {(m)} (j) \cdot \sigma_ {\ell} ^ {(m)} (i) \cdot \mathbf {f} _ {i} ^ {(m)}
$$
$$
\eta_ {i} := \mathbb {1} [ h (i) = h (j) ]
$$
$$
\text{tailNoise} := \frac{1}{M}\sum_{i\neq j,i\notin \mathbb{W}}\eta_{i}\cdot \xi_{i}.
$$
First notice that  $\left(\xi_i^{(m)}\right)_{m = 1}^M$  are independent random variables and
$$
\mathbb {E} \left[ \xi_ {i} ^ {(m)} \right] = 0, \quad \operatorname {V a r} \left[ \xi_ {i} ^ {(m)} \right] = \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}.
$$
These imply that
$$
\mathbb {E} [ \xi_ {i} ] = 0, \quad \operatorname {V a r} [ \xi_ {i} ] = \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}.
$$
Moreover, notice that  $(\eta_i,\xi_i)_{i\neq j}$  are independent random variables, and
$$
\mathbb {E} [ \eta_ {i} ^ {2} ] = \frac {1}{W},
$$
$$
\begin{array}{l} \mathbb {E} [ \eta_ {i} \xi_ {i} ] = 0; \\ \operatorname {V a r} \left[ \eta_ {i} \xi_ {i} \right] = \mathbb {E} \left[ \eta_ {i} ^ {2} \right] \cdot \operatorname {V a r} \left[ \xi_ {i} \right] + \operatorname {V a r} \left[ \eta_ {i} \right] \cdot \left(\mathbb {E} \left[ \xi_ {i} \right]\right) ^ {2} \\ = \frac {1}{W} \cdot \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}. \\ \end{array}
$$
Therefore we conclude that
$$
\mathbb{E}[\text{tailNoise}] = \frac{1}{M}\sum_{i\neq j,i\notin \mathbb{W}}\mathbb{E}[\eta_{i}\xi_{i}] = 0;
$$
$$
\operatorname {V a r} [ \text {t a i l N o i s e} ] = \frac {1}{M ^ {2}} \sum_ {i \neq j, i \notin \mathbb {W}} \operatorname {V a r} [ \eta_ {i} \xi_ {i} ]
$$
$$
\begin{array}{l} = \frac {1}{M ^ {2} W} \cdot \sum_ {i \neq j, i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2} \\ \leq \frac {1}{M ^ {2} W} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}. \\ \end{array}
$$
Then by Chebyshev we see that: for fixed  $j \in [d]$  and  $\ell \in [L]$  it holds that
$$
\mathbb {P} \left\{\left| \text {t a i l N o i s e} \right| \geq \sqrt {\frac {1 0}{M ^ {2} W} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}} \right\} <   0. 1.
$$
By a union bound we see that: for fixed  $j \in [d]$  and  $\ell \in [L]$  it holds that
$$
\mathbb {P} \left\{\left| \tilde {\mathbf {f}} _ {j} ^ {(\ell)} - \mathbf {f} _ {j} \right| <   \sqrt {\frac {1 0}{M ^ {2} W} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}} \right\} > 0. 8 > 0. 5.
$$
Probability Amplification. Fixing  $j$ . Recall that  $(\tilde{\mathbf{f}}_j^{(\ell)})_{\ell=1}^L$  are i.i.d. random variables and that  $\tilde{\mathbf{f}}_j := \text{median}\{\tilde{\mathbf{f}}_j^{(\ell)} : \ell \in [L]\}$ . By Chernoff over  $\ell$  and union bound over  $j$  we see that:
$$
\mathbb {P} \left\{\text {f o r e a c h} j \in [ d ], \quad | \tilde {\mathbf {f}} _ {j} - \mathbf {f} _ {j} | \geq \sqrt {\frac {1 0}{M ^ {2} W} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}} \right\} <   2 d \cdot \exp (\Omega (L)).
$$
By choosing  $L = \Theta (\log (2d / \delta))$  we obtain that, with probability at least  $1 - \delta$ ,
$$
\text {f o r e a c h} j \in [ d ], \quad | \tilde {\mathbf {f}} _ {j} - \mathbf {f} _ {j} | \lesssim \sqrt {\frac {1 0}{M ^ {2} W} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}}.
$$
C Missing Proofs for Section 4
Proof of Theorem 4.1. We follow the method of Pagh and Thorup [2022], Zhao et al. [2022] to add DP noise to all  $M$  sketches. Suppose  $\mathcal{F} = \left(\mathbf{f}^{(m)}\right)_{m=1}^{M}$  and  $\mathring{\mathcal{F}} = \left(\mathring{\mathbf{f}}^{(m)}\right)_{m=1}^{M}$  are the sets of local frequencies for two neighboring datasets respectively, then
$$
\| \mathcal {F} - \mathring {\mathcal {F}} \| _ {2} \leq \frac {1}{n}.
$$
Denote the sketches to be released by  $\mathcal{S} \circ \mathcal{F} \coloneqq \left(\mathbf{S}^{(m)} \circ \mathbf{f}^{(m)}\right)_{m=1}^{M}$ . One can then calculate the  $\ell_2$ -sensitivity:
$$
\left\| \mathcal {S} \circ \mathcal {F} - \mathcal {S} \circ \mathring {\dot {\mathcal {F}}} \right\| _ {2} \leq \frac {\sqrt {L}}{n},
$$
where  $L \asymp \log(d / \delta)$  is the sketch length. Therefore the sketching will be  $(\epsilon, \delta)$ -DP by adding Gaussian noise  $\mathcal{N}(0, \sigma^2)$  to each bucket of each sketch, where
$$
\sigma \approx \frac {\sqrt {L \log (1 / \delta)}}{n \epsilon}.
$$
The final released frequency estimator is obtained by post-processing the sketch, so it is also  $(\epsilon, \delta)$ -DP. We then calculate the error for the noisy sketch matrix. For each row estimator, we have that with probability at least  $2/3$ :
$$
\tilde {\mathbf {f}} _ {j} ^ {(\ell)} - \mathbf {f} _ {j} ^ {\ell} = \text {t a i l N o i s e} + \frac {1}{M} \sum_ {m = 1} ^ {M} \operatorname {r a d} _ {m} \cdot \mathcal {N} (0, \sigma^ {2})
$$
$$
\begin{array}{l} = \text {t a i l N o i s e} + \mathcal {N} (0, \sigma^ {2} / M) \\ \lesssim \sqrt {\frac {1}{M ^ {2} w} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}} + \frac {\sqrt {L \log (1 / \delta)}}{\sqrt {M} n \epsilon}. \\ \end{array}
$$
By taking median over  $L \approx \log (d / \delta)$  repeats, we see that with probability at least  $1 - \delta$ , it holds that
$$
\text {f o r e a c h} j \in [ d ], \quad | \hat {\mathbf {f}} _ {j} - \mathbf {f} _ {j} | \lesssim \sqrt {\frac {1}{M ^ {2} w} \cdot \sum_ {i \notin \mathbb {W}} \sum_ {m = 1} ^ {M} \left(\mathbf {f} _ {i} ^ {(m)}\right) ^ {2}} + + \frac {\sqrt {\log (d / \delta) \cdot \log (1 / \delta)}}{\sqrt {M} n \epsilon}.
$$