SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The main step of the proof is to show that all the probabilities should be either the maximum or the minimum in order to minimize the error, and the proof is given in Appendix E.
3.4  $k$ -closest Encoding for General Rotationally Symmetric Codebook
In this section, we demonstrate that the  $k$ -closest encoding consistently yields an unbiased scheme for any rotationally symmetric codebook. To be more specific, for any given spherically symmetric
Algorithm 1 Randomly Rotating Simplex Coding RRSC  $(k)$
Inputs:  $v \in \mathbb{S}^{d - 1}, k, r_k$ , codebook size  $M = 2^b$ .
Codebook Generation:
Generate the simplex  $s_1, \ldots, s_M \in \mathbb{R}^d$  in (23).
Sample orthogonal matrix  $A \in \mathbb{R}^{d \times d}$  uniformly using the shared random SEED.
Generate the codebook  $U^{M}\colon (U_{1},U_{2},\ldots ,U_{M})\gets (r_{k}As_{1},r_{k}As_{2},\ldots ,r_{k}As_{M})$
for  $m\in [M]$  do
if  $\langle v,U_m\rangle$  is one of the  $k$  largest then
$$
Q _ {f} (m | v, U ^ {M}) \leftarrow \frac {e ^ {\varepsilon}}{k e ^ {\varepsilon} + (M - k)}
$$
$$
Q _ {f} (m | v, U ^ {M}) \leftarrow \frac {1}{k e ^ {\varepsilon} + (M - k)}
$$
Sample codeword index  $m^{*}\gets Q_{f}(\cdot |v,U^{M})$
Output:  $m^*$ , encoded in  $b = \log M$  bits.
codebook  $U^{M}$ , there exists a scalar  $r_k$  that ensures that the  $k$ -closest encoding with  $r_k U^M = (r_k U_1, \ldots, r_k U_M)$  is unbiased. Let  $T_k(v, U^M) = \{m : U_m \text{ is one of the } k$ -closest\} and without loss of generality, let us assume  $v = e_1$ . Then,
$$
\begin{array}{l} \mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | e _ {1}, U ^ {M}) U _ {m} \right] \\ = \mathbb {E} _ {P _ {U M}} \left[ \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \sum_ {m \in T _ {k} \left(e _ {1}, U ^ {M}\right)} U _ {m} + \frac {1}{k e ^ {\varepsilon} + (M - k)} \sum_ {m = 1} ^ {M} U _ {m} \right] (27) \\ = \mathbb {E} _ {P _ {U M}} \left[ \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \sum_ {m \in T _ {k} \left(e _ {1}, U ^ {M}\right)} U _ {m} \right], (28) \\ \end{array}
$$
where  $\mathbb{E}\left[\sum U_m\right] = 0$  due to rotationally symmetric codebook and we assume an integer  $k$  for the sake of simplicity. Since the codebook is rotationally symmetric and we pick  $k$ -closest vectors toward  $v = e_1$ , each codeword  $U_{m} \in T_{k}(e_{1}, U^{M})$  is symmetric in all directions other than  $v = e_1$ . Thus, in expectation, the decoded vector is aligned with  $e_1$ , and there exists  $r_k$  such that
$$
r _ {k} \times \mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} \left(m \mid e _ {1}, U ^ {M}\right) U _ {m} \right] = e _ {1}. \tag {29}
$$
For a rotationally symmetric simplex codebook, where  $U_{m} = As_{m}$  for a uniform random orthogonal matrix  $A$ , we have an (almost) analytic formula.
Lemma 3.7. Normalization constant  $r_k$  for  $RRSC(k)$  is
$$
r _ {k} = \frac {k e ^ {\varepsilon} + M - k}{e ^ {\varepsilon} - 1} \sqrt {\frac {M - 1}{M}} \frac {1}{C _ {k}}, \tag {30}
$$
where  $C_k^2$  is an expected sum of top- $k$  coordinates of uniform random vector  $a \in \mathbb{S}^{d-1}$ .
The key idea in the proof is to show that encoding  $e_1$  with  $A s^M$  is equivalent to encoding uniform random vector  $a \in \mathbb{S}^{d-1}$  with  $s^M$ . The formal proof is provided in Appendix F.
The following lemma controls the asymptotic behavior of  $C_k$ :
Lemma 3.8. Let  $C_k$  be defined as in Lemma 3.7. Then, it holds that
$$
C _ {k} = O \left(\sqrt {\frac {k ^ {2} \log M}{d}}\right). \tag {31}
$$
Note that  $C_k$  depends on  $k, d,$  and  $M$ , but for ease of presentation, we suppress the dependency on  $d$  and  $m$  here and only present the full expression in the proof.
In addition, there exist absolute constants  $C_1, C_2 > 0$  such that as long as  $\lfloor M / k \rfloor > C_1$  and  $k > C_2$ ,
$$
C _ {k} = \Omega \left(\sqrt {\frac {k ^ {2}}{d} \log \left(\frac {M}{k}\right)}\right). \tag {32}
$$
As a corollary, Lemma 3.8 implies the order-wise optimality of RRSC:
$$
\operatorname {E r r} (\operatorname {R R S C}) \leq r _ {k} ^ {2} - 1 = O \left(\frac {\left(e ^ {\varepsilon} - 1 - \frac {M}{k}\right) ^ {2}}{\left(e ^ {\varepsilon} - 1\right) ^ {2}} \cdot \frac {d}{\log \left(\frac {M}{k}\right)}\right).
$$
By picking  $k = \max(1, Me^{-\varepsilon})$ , the above error is  $O\left(\frac{d}{\min(\varepsilon^2, \varepsilon, b)}\right)$ . We provide the proof of Lemma 3.8 in Appendix G.
3.5 Convergence to PrivUnit
As the communication constraint  $b$  increases, the exact-optimal scheme with communication constraint should coincide with the exact-optimal scheme without communication constraint, which is PrivUnit. Note that the rotationally symmetric simplex can be defined only when  $M = 2^b < d$ , due to its simplex structure. However, we have a natural extension where the codebook is a collection of  $M$  (nearly) maximally separated vectors on the sphere of radius  $r$ , where we can assume that  $M$  codewords are uniformly distributed on the sphere of radius  $r_k$  if  $M$  is large enough. Consider the case where  $q = \frac{k}{M}$  is fixed and  $M = 2^b$  is large. Since the  $k$ -closest encoding yields an unbiased scheme with error  $\mathrm{Err}(f, P_{U^M}) = r_k^2 - 1$ , where  $r_k$  is normalizing constant, for uniformly distributed  $M$  codewords on the sphere, the constant  $r_k$  should satisfy
$$
r _ {k} \times \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \mathbb {E} \left[ \sum_ {m \in \operatorname {t o p} - k} U _ {m, 1} \right] = 1 \tag {33}
$$
where  $U_{m,1}$  is the first coordinate of uniformly drawn  $U_{m}$  from the unit sphere  $\mathbb{S}^{d - 1}$ . Then, as  $M$  increases,  $U_{m,1}$  being one of the top-  $k$  becomes equivalent to  $U_{m,1} > \gamma$ , where  $\gamma$  is the threshold such that  $\operatorname*{Pr}[U_{m,1} > \gamma] = q$ . Hence, assigning higher probabilities to the top-  $k$  closest codewords becomes equivalent to assigning high probabilities to the codewords with  $\langle U_m,e_1\rangle >\gamma$  where  $v = e_1$ . This is essentially how PrivUnit operates.
3.6 Complexity of RRSC
Each user has  $d \times d$  orthonormal matrix  $A$  and needs to find  $k$  smallest  $\langle v, As_m \rangle$  for  $1 \leq m \leq M$ . Since  $\langle v, As_m \rangle = \langle A^\intercal v, s_m \rangle$ , it requires  $O(d^2)$  to compute  $A^\intercal v$  and additional  $O(Md)$  to compute all inner products for  $1 \leq m \leq M$ . However, if  $M \ll d$ , we have a simpler equivalent scheme using
$$
\langle A ^ {\intercal} v, s _ {m} \rangle = \sqrt {\frac {M}{M - 1}} a _ {m} ^ {\intercal} v - \sum_ {i = 1} ^ {M} a _ {i} ^ {\intercal} v \frac {1}{\sqrt {M (M - 1)}}, \tag {34}
$$
where  $a_{m}^{\intercal}$  is the  $m$ -th row of the matrix  $A$ . Then, it only requires storing the first  $M$  rows of the matrix and  $O(Md)$  to obtain all inner products in (34) by avoiding  $O(d^{2})$  to construct  $A^{\intercal}v$ .
On the other hand, the server computes  $As_{m}$  upon receiving a message  $m$ . The corresponding time complexity is  $O(Md)$  (per user) since  $s_{m}$  has  $M$  non-zero values. We note that both MMRC [30] and FT21 [12] require the same encoding complexity  $O(Md)$  as RRSC, where they choose  $M = O(\exp(\varepsilon))$ .
We empirically demonstrate the communication-privacy-utility tradeoffs of RRSC and compare it with order-optimal schemes under privacy and communication constraints, namely SQKR [6] and MMRC [30]. We also show that RRSC performs comparably with PrivUnit [4], which offers the exact-optimal privacy-utility tradeoffs without communication constraints [2]. In our simulations, we use
the "optimized" PrivUnit mechanism, called PrivUnitG, introduced in [2], which performs better than PrivUnit in practice since it provides an easy-to-analyze approximation of PrivUnit but with analytically better-optimized hyperparameters. Similar to [6, 30], we generate data independently but non-identically to capture the distribution-free setting with  $\mu \neq 0$ . More precisely, for the first half of the users, we set  $v_{1}, \ldots, v_{n/2} \stackrel{\mathrm{i.i.d}}{\sim} N(1,1)^{\otimes d}$ ; and for the second half of the users, we set  $v_{n/2+1}, \ldots, v_{n} \stackrel{\mathrm{i.i.d}}{\sim} N(10,1)^{\otimes d}$ . We further normalize each  $v_{i}$  to ensure that they lie on  $\mathbb{S}^{d-1}$ . We report the average  $\ell_2$  error over 10 rounds together with the confidence intervals. To find the optimal values for  $k$  and  $r_k$ , we compute the optimal  $r_k$  using the formula in (33) for  $k = 1, \ldots, M$  and pick the  $k$  that gives the smallest  $r_k$  (which corresponds to the bias). To estimate the expectation  $C_k$  in (33), we run a Monte Carlo simulation with  $1M$  trials. We report the  $k$  we use for each experiment in the captions. Additional experimental results are provided in Appendix H.
In Figure 1-(left, middle), we report  $\ell_2$  error for  $\varepsilon = 1,\ldots ,8$ , where for each method (except PrivUnitG), the number of bits is equal to  $b = \varepsilon$ . In Figure 1-(right), we report  $\ell_2$  error by fixing  $\epsilon = 6$  and sweeping the bitrate from  $b = 1$  to  $b = 8$  for RRSC and MMRC. For SQKR, we only sweep for  $b\leq \varepsilon$  as it leads to poor performance for  $b > \varepsilon$ . In each figure, RRSC performs comparably to PrivUnitG even for small  $b$  and outperforms both SKQR and MMRC by large margins.
The codebase for this work is open-sourced at https://github.com/BerivanIsik/rrsc.
We proved that using a rotationally symmetric codebook is a necessary condition for the exact optimality of mean estimation mechanisms with privacy and communication constraints. We then proposed Random Rotating Simplex Coding (RRSC) based on a  $k$ -closest encoding mechanism and proved that RRSC is exact-optimal for the random rotating simplex codebook. We now discuss some important features of RRSC and provide conjectures for future work.
Unified Framework It turns out that SQKR [6], FT21 [12] and MMRC [30] can be viewed as special cases in our framework. Specifically, SQKR [6] uses Kashin's representation of  $v = \sum_{j=1}^{N} a_j u_j$ , where  $\{a_j\}_{j=1}^N \in [-c/\sqrt{d}, c/\sqrt{d}]$  for some  $(1 + \mu)d$  with  $\mu > 0$  and  $c > 0$ . Then the SQKR encoder quantizes each  $a_j$  into a 1-bit message  $q_i$ , and draws  $k$  samples with the help of shared randomness. This can be viewed as random coding with a codebook-generating distribution. More concretely, the corresponding codebook  $U^M$  consists of  $k$  non-zero values of  $\pm c/\sqrt{d}$  where the randomness is from selecting  $k$  indices using shared randomness. On the other hand, since MMRC [30] is simulating the channel corresponding to a privacy mechanism, it can be viewed as pre-generating random codewords  $U^M$  according to the reference distribution, where the importance sampling is also a way of assigning probabilities to each codeword. As elaborated in Section 3.5, it is observed that with an increase in the communication constraint  $b$ , the suggested  $k$ -closest encoding gradually transforms into a threshold-based encoding, analogous to that of MMRC. The codebook associated with FT21 [12]
depends on the PRG it uses. Let  $\mathrm{PRG}:\{0,1\}^b\to \{0,1\}^{\Theta (d)}$  be a PRG that takes a  $b$ -bit seed and maps it into  $\Theta (d)$  bits, where  $b\ll d$ , and let  $g:\{0,1\}^{\Theta (d)}\to \mathbb{R}^{d}$ . For example, if we represent each coordinate of  $x\in \mathbb{R}^d$  as a 32-bit float, then  $g(\cdot)$  maps the float representation of  $x$  (a 32-bit string) to  $x$ . With a PRG, FT21 mimics PrivUnit by first generating a  $b$ -bit seed  $m$ , computing  $g(\mathrm{PRG}(m))$ , and then performing rejection sampling on the seed space. The above procedure can be treated as a special case in our framework, where the deterministic codebook consists of  $2^{b}$  points on  $\mathbb{R}^d$ :  $\mathcal{C}_{\mathrm{FT21}}\coloneqq \{g(\mathrm{PRG}(m)):m\in \{0,1\}^b\}$ . The probabilities assigned to each codeword according to the rejection sampling are equivalent to a threshold-based assignment.
Shared randomness When  $M \leq d + 1$ , additional randomization is required during codebook generation to achieve an unbiased scheme, as discussed in [13]. Furthermore, both the encoder and decoder must possess this randomization information. In the proposed RRSC scheme, this randomization is achieved through the random rotation of the simplex code using shared randomness. However, it is possible to circumvent the need for shared randomness by having the server generate random rotation matrices using its private coin and communicate them to the users. This approach replaces shared randomness with downlink communication, which is typically more affordable than uplink communication. It should be noted that directly transmitting the rotation matrices would require  $O(d^{2})$  bits. Nonetheless, the server can generate them using a predetermined pseudo-random generator (PRG) and transmit only the seeds of it to the users. Drawing from a similar argument as in [12], assuming the existence of exponentially strong PRGs, seeds with polylog(d) bits are sufficient.
Future Work We showed the exact-optimality of  $k$ -closest encoding for the rotating simplex codebook. In general, it also achieves unbiasedness and the following error formulation  $\mathbb{E}_{P_{UM}}\left[\sum_{m = 1}^{M}Q_{f}(m|v,U^{M})\| v - U_{m}\|^{2}\right]$  implies the exact-optimality of  $k$ -closest encoding for any rotationally symmetric codebook, which leads us to the following conjecture.
Conjecture 5.1. The proposed  $k$ -closest encoding is exact-optimal for any rotationally symmetric codebook.
It also remains unclear whether  $k$  can depend on the realization of the codebook  $U^{M}$  in general, which we leave to future work. We also proved that the exact-optimal codebook must be rotationally symmetric. We conjecture that the maximally separated codebook (simplex codebook) is exact-optimal as it provides the most effective coverage of the space  $\mathbb{S}^{d - 1}$ . This, too, is left as a topic for future work.
Conjecture 5.2. The rotationally symmetric simplex codebook is the exact-optimal codebook.
Limitations and Broader Impact While we take an important step towards exact optimality by proving several necessary conditions and by providing a mechanism that is exact-optimal for a family of codebooks, we still have the above conjectures left to be proven in future work.
The authors would like to thank Peter Kairouz for the helpful discussions in the early stage of this work; and Hilal Asi, Vitaly Feldman, and Kunal Talwar for sharing their implementation for the PrivUnitG mechanism [2]. BI was supported by a Stanford Graduate Fellowship, a Google PhD Fellowship, and a Meta research grant. WNC was supported by NSF under Grant CCF-2213223. AN was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1F1A1059567).
[1] N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan. cpsgd: Communication-efficient and differentially-private distributed sgd. Advances in Neural Information Processing Systems, 31, 2018.
[2] H. Asi, V. Feldman, and K. Talwar. Optimal algorithms for mean estimation under local differential privacy. In International Conference on Machine Learning, pages 1046-1056. PMLR, 2022.
A Proof of Lemma 3.1
Proof. For  $n$ -user mean estimation protocol  $(f, \mathcal{A}, P_{U^M})$ , following the notation and steps from [2, Proof of Lemma 3.1], we define the marginalized output
$$
\tilde {g} _ {i} \left(m _ {i}, U _ {i}; v ^ {n}\right) = \mathbb {E} _ {\left\{m _ {j}, U _ {j} \right\} _ {j \neq i}} \left[ n \mathcal {A} \left(\left\{m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) \mid f _ {i} \left(v _ {i}, U _ {i}\right) = m _ {i}, U _ {i}, v ^ {n \backslash i} \right]. \tag {35}
$$
Then, we define the user-specific decoder by averaging  $g_{i}(m_{i},U_{i};v^{n})$  with respect to i.i.d. uniform  $P_{\mathrm{unif}}$ :
$$
g _ {i} \left(m _ {i}, U _ {i}\right) = \mathbb {E} _ {v ^ {n} \backslash i \sim P _ {\text {u n i f}}} \left[ \tilde {g} _ {i} \left(m _ {i}, U _ {i}; v ^ {n}\right) \right] \tag {36}
$$
where  $v^{n \setminus i}$  indicates the  $v^n$  vector except  $v_i$ . Due to the symmetry of  $P_{\mathrm{unif}}$ , it is clear that  $g_i$  is unbiased. We also define
$$
\hat {\mathcal {R}} _ {\leq i} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {i}\right) = \mathbb {E} _ {v _ {j} \sim P _ {\text {u n i f}}, j > i} \left[ n \mathcal {A} \left(\left\{m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) - \sum_ {j = 1} ^ {i} v _ {j} \mid \left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {i} \right] \tag {37}
$$
Consider an average error where  $v_{1}, \ldots, v_{n}$  are drawn i.i.d. uniformly on the sphere  $\mathbb{S}^{d - 1}$ .
$$
\begin{array}{l} \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}} \left[ \left\| n \mathcal {A} \left(\left\{m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) - \sum_ {j = 1} ^ {n} v _ {j} \right\| ^ {2} \right] \\ = \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}} \left[ \left\| \hat {\mathcal {R}} _ {\leq n} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) \right\| ^ {2} \right] (38) \\ = \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}} \left[ \left\| \hat {\mathcal {R}} _ {\leq n} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) - \hat {\mathcal {R}} _ {\leq n - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n - 1}\right) + \hat {\mathcal {R}} _ {\leq n - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n - 1}\right) \right\| ^ {2} \right] (39) \\ = \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}} \left[ \left\| \hat {\mathcal {R}} _ {\leq n} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) - \hat {\mathcal {R}} _ {\leq n - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n - 1}\right) \right\| ^ {2} \right] \\ + \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n - 1}} \left[ \left\| \hat {\mathcal {R}} _ {\leq n - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n - 1}\right) \right\| ^ {2} \right] (40) \\ = \sum_ {i = 1} ^ {n} \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {i}} \left[ \left\| \hat {\mathcal {R}} _ {\leq i} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {i = 1} ^ {n}\right) - \hat {\mathcal {R}} _ {\leq i - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {i - 1}\right) \right\| ^ {2} \right] (41) \\ \geq \sum_ {i = 1} ^ {n} \mathbb {E} _ {m _ {i}, U _ {i}} \left[ \left\| \mathbb {E} _ {\{v _ {j}, m _ {j}, U _ {j} \} _ {j = 1} ^ {i - 1}} \left[ \hat {\mathcal {R}} _ {\leq i} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {i = 1} ^ {n}\right) - \hat {\mathcal {R}} _ {\leq i - 1} \left(\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {i - 1}\right) \right] \right\| ^ {2} \right] (42) \\ = \sum_ {i = 1} ^ {n} \mathbb {E} _ {m _ {i}, U _ {i}} \left[ \| g _ {i} \left(m _ {i}, U _ {i}\right) - v _ {i} \| ^ {2} \right]. (43) \\ \end{array}
$$
Then, we need to show the same inequality for the worst-case error.
$$
\begin{array}{l} \sup  _ {v _ {1}, \dots , v _ {n}} \mathbb {E} _ {\{m _ {j}, U _ {j} \} _ {j = 1} ^ {n}} \left[ \left\| n \mathcal {A} (\{m _ {j}, U _ {j} \} _ {j = 1} ^ {n}) - \sum_ {j = 1} ^ {n} v _ {j} \right\| ^ {2} \right] \\ \geq \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}} \left[ \left\| n \mathcal {A} \left(\left\{m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) - \sum_ {j = 1} ^ {n} v _ {j} \right\| ^ {2} \right] (44) \\ = \sum_ {i = 1} ^ {n} \mathbb {E} _ {v _ {i}, m _ {i}, U _ {i}} \left[ \left\| g _ {i} \left(m _ {i}, U _ {i}\right) - v _ {i} \right\| ^ {2} \right] (45) \\ = \sum_ {i = 1} ^ {n} \sup  _ {v _ {i}} \mathbb {E} _ {m _ {i}, U _ {i}} \left[ \| g _ {i} \left(m _ {i}, U _ {i}\right) - v _ {i} \| ^ {2} \right] (46) \\ \end{array}
$$
where the last equality (46) is from Lemma 3.2, Lemma 3.4, and Lemma 3.5. Thus, the user-specific decoder achieves lower MSE:
$$
\operatorname {E r r} _ {n} \left(f, \mathcal {A}, P _ {U ^ {n}}\right) \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {E r r} _ {1} \left(f _ {i}, g _ {i}, P _ {U _ {i}}\right). \tag {47}
$$
Since we keep random encoder  $f_{i}$  the same, the canonical protocol with  $g_{i}$  also satisfies  $\varepsilon$ -LDP constraint. This concludes the proof.
B Proof of Lemma 3.2
Proof. Let  $\tilde{U}_m = g(m,U)$  for all  $1\leq m\leq M$ . Without loss of generality  $g(\cdot ,U)$  is one-to-one, i.e.,  $\{u:\tilde{u}_m = g(m,u)$  for all  $m\}$  has at most one element (with probability 1), and  $u = g^{-1}(\tilde{u}^M)$  is well-defined. Then, we define a randomizer  $f_{0}(v,\tilde{U}^{M})$  that satisfies
$$
Q _ {f _ {0}} \left(m | v, \tilde {u} ^ {M}\right) = Q _ {f} \left(m | v, g ^ {- 1} \left(\tilde {u} ^ {M}\right)\right). \tag {48}
$$
It is clear that  $f_{0}$  satisfies  $\varepsilon$ -LDP constraint. Then,
$$
\begin{array}{l} D \left(v, f _ {0}, g ^ {+}, P _ {\tilde {U} ^ {M}}\right) = \mathbb {E} _ {f _ {0}, P _ {\tilde {U} ^ {M}}} \left[ \| g ^ {+} \left(f _ {0} \left(v, \tilde {U} ^ {M}\right), \tilde {U} ^ {M}\right) - v \| ^ {2} \right] (49) \\ = \mathbb {E} _ {P _ {\tilde {U} ^ {M}}} \left[ \sum_ {m = 1} ^ {M} Q _ {f _ {0}} (m | v, \tilde {U} ^ {M}) \| \tilde {U} _ {m} - v \| ^ {2} \right] (50) \\ = \mathbb {E} _ {f, P _ {U}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | v, U) \| g (m, U) - v \| ^ {2} \right] (51) \\ = \mathbb {E} _ {f, P _ {U}} \left[ \| g (f (v, U), U) - v \| ^ {2} \right] (52) \\ = D (v, f, g, P _ {U}). (53) \\ \end{array}
$$
We also need to show that the composition of the new randomizer  $f_{0}$  and selector  $g^{+}$  is unbiased.
$$
\begin{array}{l} \mathbb {E} _ {P _ {\tilde {U} M}} \left[ g ^ {+} \left(f _ {0} (v, \tilde {U} ^ {M}), \tilde {U} ^ {M}\right) \right] = \mathbb {E} _ {f _ {0}, P _ {\tilde {U} M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f _ {0}} (m | v, \tilde {U} ^ {M}) \tilde {U} _ {m} \right] (54) \\ = \mathbb {E} _ {f, P _ {U}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | v, U) g (m, U) \right] (55) \\ = \mathbb {E} _ {f, P _ {U}} [ g (f (v, U), U) ] (56) \\ = v. (57) \\ \end{array}
$$
Finally,  $Q_{f_0}(m|v,\tilde{u}^M)$  is a valid transition probability, since
$$
\sum_ {m = 1} ^ {M} Q _ {f _ {0}} (m | v, \tilde {u} ^ {M}) = \sum_ {m = 1} ^ {M} Q _ {f} (m | v, g ^ {- 1} (\tilde {u} ^ {M})) = 1 \tag {58}
$$
for all  $\tilde{u}^M$ . This concludes the proof.
C Proof of Lemma 3.4
Proof. Let  $A$  be a uniformly random orthogonal matrix and  $\bar{U}^M = A^\intercal U^M$ . We further let  $f_1$  be a randomized encoder that satisfies
$$
Q _ {f _ {1}} (m | v, \bar {U} ^ {M}) = \mathbb {E} _ {A} \left[ Q _ {f} (m | A v, A \bar {U} ^ {M}) | \bar {U} ^ {M} \right]. \tag {59}
$$
Then,  $Q_{f_1}$  is a valid probability since
$$
\sum_ {m = 1} ^ {M} Q _ {f _ {1}} (m | v, \bar {U} ^ {M}) = \mathbb {E} _ {A} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, A \bar {U} ^ {M}) | \bar {U} ^ {M} \right] = 1. \tag {60}
$$
$$
\frac {Q _ {f _ {1}} (m | v , \bar {U} ^ {M})}{Q _ {f _ {1}} (m | v ^ {\prime} , \bar {U} ^ {M})} = \frac {\mathbb {E} _ {A} \left[ Q _ {f} (m | A v , A \bar {U} ^ {M}) | \bar {U} ^ {M} \right]}{\mathbb {E} _ {A} \left[ Q _ {f} (m | A v ^ {\prime} , A \bar {U} ^ {M}) | \bar {U} ^ {M} \right]} \tag {61}
$$
$$
\leq \frac {\mathbb {E} _ {A} \left[ e ^ {\varepsilon} Q _ {f} \left(m \mid A v ^ {\prime} , A \bar {U} ^ {M}\right) \mid \bar {U} ^ {M} \right]}{\mathbb {E} _ {A} \left[ Q _ {f} \left(m \mid A v ^ {\prime} , A \bar {U} ^ {M}\right) \mid \bar {U} ^ {M} \right]} \tag {62}
$$
$$
= e ^ {\varepsilon}. \tag {63}
$$
Finally, we need to check unbiasedness.