SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first introduce the idea of a tight frame in Kashin's representation. A tight frame is a set of vectors  $\{u_j\}_{j=1}^D \subset \mathbb{R}^d$  that satisfy Parseval's identity, i.e.,  $\|x\|_2^2 = \sum_{j=1}^D \langle u_j, x \rangle^2$  for all  $x \in \mathbb{R}^d$ .
A frame can be viewed as a generalization of the notion of an orthogonal basis in  $\mathbb{R}^d$  for  $D > d$ . To increase robustness, we wish the information to be spread evenly across different coefficients, which motivates the following definition of a Kashin's representation:
Definition H.1. For a set of vectors  $\{u_j\}_{j=1}^D$ , we say the expansion
$$
x = \sum_ {j = 1} ^ {D} a _ {j} u _ {j}, w i t h \max  _ {j} | a _ {j} | \leq \frac {K}{\sqrt {D}} \| x \| _ {2}
$$
is a Kashin's representation of the vector  $x$  at level  $K$ .
By Theorem 3.5 and Theorem 4.1 in [67], we have the following lemma:
Lemma H.2. There exists a tight frame  $U = [u_{1},\dots,u_{D}]$  with (1)  $D = \Theta (d)$  and (2) level  $K = O(1)$ .
The above lemma implies that for each  $x_{i} \in \mathbb{R}^{d}$  such that  $\| x_{i}\|_{2} \leq 1$ , one can always represent each  $x_{i}$  with coefficients  $y_{i} \in [-\gamma_{0} / \sqrt{d}, \gamma_{0} / \sqrt{d}]^{\gamma_{1}d}$  for some  $\gamma_0, \gamma_1 > 0$  and  $x_{i} = Uy_{i}$ .
I Rényi-DP for Shuffled SQKR
In this section we restate some results for RDP which are useful for privacy accounting in experiments.
Following the proof of Corollary 4.3 in [44], applying Theorem 4.1 in the same paper yields the following.
Lemma I.1. Let  $\mathcal{M}_i$  be an independent  $(\varepsilon_0, 0)$ -LDP mechanism for each  $i \in [n]$  with  $\varepsilon_0 \leq 1$  and  $\pi$  be a random permutation of  $[n]$ . Then for any  $\alpha < \frac{n}{16\varepsilon_0 \exp(\varepsilon_0)}$ , the mechanism
$$
\mathcal {S}: \left(x _ {1}, \dots , x _ {n}\right) \mapsto \left(\mathcal {M} _ {1} \left(x _ {\pi (1)}\right), \dots , \mathcal {M} _ {n} \left(x _ {\pi (n)}\right)\right)
$$
is  $(\varepsilon_{1}(\alpha),\delta)$  -RDP with
$$
\varepsilon_ {1} (\alpha) = \frac {\log \left(e ^ {2 \alpha^ {2} \sigma^ {2}} + 4 \delta_ {\min } e ^ {\alpha \varepsilon_ {0}}\right)}{\alpha - 1}, \tag {36}
$$
$$
\sigma = 8 \sqrt {\frac {e ^ {\varepsilon_ {0}}}{n}} \tag {37}
$$
$$
\delta_ {\min } = e ^ {- \frac {n}{8 \left(e ^ {\bar {\varepsilon} 0} + 1\right)}}. \tag {38}
$$
For small  $\varepsilon_0$ , the result below is useful.
Lemma I.2 ([40]). Under the same assumptions as Lemma I.1,  $S$  is  $(\varepsilon(\alpha), \delta)$ -RDP
$$
\varepsilon_ {1} (\alpha) = 2 \alpha e ^ {4 \varepsilon_ {0}} \left(e ^ {\varepsilon_ {0}} - 1\right) ^ {2} / n. \tag {39}
$$
Applying Lemma G.1, by construction the mapping from each  $x_{i}$  to  $y(i,1),\ldots ,y(i,b_0)$  is  $(\varepsilon_0,0)$ -LDP. By Lemma I.1, respectively Lemma I.2, the mapping  $x^{n}\mapsto \hat{\mu}^{(k)}$  is  $(\varepsilon_1(\alpha),\alpha)$ -RDP where  $\varepsilon_1(\alpha)$  is given by (36), respectively (39). By composition, Algorithm 4 is  $(T\varepsilon (\alpha),\alpha)$ -RDP.
We can convert this bound back to  $(\varepsilon, \delta)$ -DP using Proposition 12 from [30].
Proposition I.3. For all  $\delta > 0$ , Algorithm 4 is  $(\varepsilon, \delta)$ -DP where
$$
\varepsilon = \inf  _ {\alpha \in (1, \infty)} T \varepsilon_ {1} (\alpha) + \frac {\log (1 / \delta) + (\alpha - 1) \log (1 - 1 / \alpha) - \log (\alpha)}{\alpha - 1}, \tag {40}
$$
$$
\varepsilon_ {1} (\alpha) = \min  \left(2 \alpha e ^ {4 \varepsilon_ {0}} \left(e ^ {\varepsilon_ {0}} - 1\right) ^ {2} / n, \frac {\log \left(e ^ {2 \alpha^ {2} \sigma^ {2}} + 4 \delta_ {\min } e ^ {\alpha \varepsilon_ {0}}\right)}{\alpha - 1}\right) \tag {41}
$$
and  $\sigma, \delta_{\mathrm{min}}$  are given by (37), (38) respectively.
J Additional Experiments
Here experiments are done with the same setup as in Section 6, with local vectors  $X_{i}(j) \stackrel{\mathrm{i.i.d.}}{\sim} \frac{1}{\sqrt{d}} (2 \cdot \operatorname{Ber}(0.8) - 1)$ . We set  $\delta = 10^{-6}$ .
Figure 2: Comparison of MSE vs. number of clients  $n$  for LDP scheme (SQKR) and shuffled SQKR. For shuffled SQKR, we set  $b_{0} = 1$  and choose  $\varepsilon_0$  using results in Section I. Communication cost is  $\lceil (\log_2(2000) + 1) \rceil = 12$  bits per round.
Figure 2 illustrates separation between Algorithm 4 and LDP schemes. Algorithm 4 achieves error decreasing quadratically with  $n$  as guaranteed by Theorem 5.3. With only one round of shuffling, there is separation from the LDP scheme only when  $n$  is sufficiently large, and thus order-optimal error performance only occurs for large  $n$  (or equivalently small  $\varepsilon$ ). This problem is avoided with multiple rounds of shuffling.
Figure 3: CSGM with and without coordinate pre-selection using  $d' = 833$ .
Figure 3 compares the performance of CSGM with and without coordinate pre-selection. In this regime coordinate pre-selection improves performance for all  $b$ . As predicted by Corollary 4.3 and Corollary 4.5, the MSE decreases with  $b$  but is effectively constant for sufficiently high  $b$  where the
privacy term dominates. We can determine the communication cost needed for order-optimal central DP error performance to be the  $b$  at which the MSE is within some fixed constant factor away from the limiting value. We see that the communication cost increases with dimension  $d$  with the vanilla CSGM scheme, but a dimension-free communication cost is achieved with coordinate pre-selection.