SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Berivan Isik
berivan.isik@stanford.edu
Wei-Ning Chen
wnchen@stanford.edu
aozgur@stanford.edu
Tsachy Weissman
tsachy@stanford.edu
Albert No
Hongik University
albertno@hongik.ac.kr
We study the mean estimation problem under communication and local differential privacy constraints. While previous work has proposed order-optimal algorithms for the same problem (i.e., asymptotically optimal as we spend more bits), exact optimality (in the non-asymptotic setting) still has not been achieved. In this work, we take a step towards characterizing the exact-optimal approach in the presence of shared randomness (a random variable shared between the server and the user) and identify several conditions for exact optimality. We prove that one of the conditions is to utilize a rotationally symmetric shared random codebook. Based on this, we propose a randomization mechanism where the codebook is a randomly rotated simplex - satisfying the properties of the exact-optimal codebook. The proposed mechanism is based on a  $k$ -closest encoding which we prove to be exact-optimal for the randomly rotated simplex codebook.
The distributed mean estimation problem has attracted attention from the machine learning community as it is a canonical statistical formulation for many stochastic optimization problems such as distributed SGD [1, 3, 31, 32] and federated learning [33, 34]. As these tasks require data collection from the users, the mean estimation problem has often been studied under privacy constraints to protect users' sensitive information. More specifically, several works [2, 4, 7, 8, 9, 29, 35] have analyzed and improved the tradeoff between the utility and  $\varepsilon$ -local differential privacy ( $\varepsilon$ -LDP) - the predominant paradigm in privacy mechanisms, which guarantees that an adversary cannot distinguish the user data from the outcome of the privacy mechanism [10, 24]. Among them, [4, 8, 9] developed algorithms that are asymptotically optimal, achieving an optimal mean squared error (MSE) proportional to  $\Theta \left( \frac{d}{n \min(\varepsilon, \varepsilon^2)} \right)$ , where  $n$  is the number of users, and  $d$  is the input dimension. Later, [7] proved the corresponding lower bounds that hold for all privacy regimes. However, only PrivUnit [4] enjoys exact optimality among a large family of mechanisms, as proved by [2], while others provide only order optimality and their performance in practice depends heavily on the constant factors.
Another important consideration in the applications of mean estimation is the communication cost during user data collection. For instance, in federated learning, clients need to send overparameterized machine learning models at every round, which becomes a significant bottleneck due to limited resources and bandwidth available to the clients [22, 25, 28]. This has motivated extensive research on mean estimation [31, 34, 38] and distributed SGD [1, 3, 13, 26, 37] under communication constraints; and communication-efficient federated learning [17, 18, 21, 33].
In addition to the lines of work that studied these constraints (either privacy or communication) separately, recently, there has also been advancement in the joint problem of mean estimation under both privacy and communication constraints. [6] introduced an order-optimal mechanism SQKR requiring only  $O(\varepsilon)$  bits by using shared randomness – a random variable shared between the server and the user (see Section 2 for the formal definition). Later, [30] demonstrated better MSE with another order-optimal algorithm, MMRC, by simulating PrivUnit using an importance sampling technique [5, 15] – again with shared randomness. In the absence of shared randomness, the order-optimal mechanisms proposed by [6] do not achieve the best-known accuracy under this setting and are outperformed by the lossless compression approach in [12] that compresses PrivUnit using a pseudorandom generator (PRG). Due to not using shared randomness, these mechanisms require more bits than others [6, 30] that use shared randomness in the scenarios where it is actually available.
To our knowledge, no existing mechanism achieves exact optimality under both privacy and communication constraints with shared randomness<sup>1</sup>. In this work, we address this gap by treating the joint problem as a lossy compression problem under  $\varepsilon$ -LDP constraints.
Our first contribution is to demonstrate that the exact optimal scheme with shared randomness can be represented as random coding with a codebook-generating distribution. Specifically, under  $b$  bits of communication constraint, the server and the user generate a codebook consisting of  $M = 2^b$  vectors (codewords) using shared randomness. The user then selects an index of a vector under a distribution that satisfies  $\varepsilon$ -LDP constraints, and the server claims the corresponding vector upon receiving the index. We term this approach as "random coding with a codebook" and demonstrate that this (random codebook generation) is the optimal way to use shared randomness.
Next, we prove that the exact-optimal codebook-generating distribution must be rotationally symmetric. In other words, for any codebook-generating distribution, the distribution remains the same after random rotation. Based on this insight, we propose Random Rotating Simplex Coding (RRSC), where the codebook-generating distribution is a uniformly randomly rotating simplex. This choice of codebook distribution is reasonable as it maximizes the separation between codewords, which efficiently covers the sphere. The corresponding encoding scheme is the  $k$ -closest encoding, where the top- $k$ -closest codewords to the input obtain high sampling probability, and the remaining ones are assigned low probabilities. We show that this scheme is exact-optimal for the random rotating simplex codebook.
The proposed codebook generation is valid only when  $M < d$  (or  $b \leq \log d$  where  $b$  is the communication budget) due to the simplex structure of the codebook. Note that as shown in [6],  $b \leq \log d$  bits of communication budget is sufficient to achieve orderwise optimal MSEs under an  $\varepsilon$ -LDP constraint for any  $\varepsilon \leq O(\log d)$ , which is usually a common scenario in practical applications such as federated learning where  $d$  can range from millions to billions. In addition, we can also extend the scheme for cases when  $M \geq d$ , by using a codebook consisting of (nearly) maximally separated  $M$  vectors on the sphere. As the number of bits  $b$  used for communication increases, we demonstrate that the proposed scheme approaches PrivUnit, which is the exact-optimal scheme without communication constraints.
Finally, through empirical comparisons, we demonstrate that RRSC outperforms the existing order-optimal methods such as SQKR [6] and MMRC [30]. We also observe that the performance of RRSC is remarkably close to that of PrivUnit when the number of bits is set to  $b = \epsilon$ .
The  $\ell_2$  mean estimation problem is a canonical statistical formulation for many distributed stochastic optimization methods, such as communication (memory)-efficient SGD [31, 34] or private SGD [24]. For instance, as shown in [14], as long as the final estimator of the mean is unbiased, the  $\ell_2$  estimation error (i.e., the variance) determines the convergence rate of the distributed SGD. As a result, there is a long thread of works that study the mean estimation problem under communication constraints [3, 8, 13, 31, 34, 39], privacy constraints [2, 16], or a joint of both [1, 6, 12, 30].
<sup>1</sup> Note that we can also eliminate shared randomness with a private coin setting. See Section 5 for a discussion.
Among them, [6] shows that  $\Theta(\varepsilon)$  bits are sufficient to achieve the order-optimal MSE  $\Theta\left(\frac{d}{n\min(\varepsilon,\varepsilon^2)}\right)$  and proposes SQKR, an order-optimal mean estimation scheme under both privacy and communication constraints. Notice that the MSE of SQKR is orderwise optimal up to a constant factor. Later on, in [12], it is shown that the pre-constant factor in SQKR is indeed suboptimal, resulting in an ignorable gap in the MSE compared to PrivUnit - an optimal  $\ell_2$  mean estimation scheme under  $\varepsilon$ -LDP. In the original PrivUnit, the output space is a  $d$ -dimensional sphere  $\mathbb{S}^{d-1}$  and hence requires  $O(d)$  bits of communication, which is far from the optimal  $O(\varepsilon)$  communication bound. However, [12] shows that one can (almost) losslessly compress PrivUnit via a pseudo-random generator (PRG). Under the assumption of an existing exponentially strong PRG, [12] proves that one can compress the output of PrivUnit using polylog  $(d)$  bits with negligible performance loss. Similarly, [30] shows that with the help of shared randomness, PrivUnit can be (nearly) losslessly compressed to  $\Theta(\varepsilon)$  bits via a channel simulation technique, called MMRC. We remark that although the privacy-utility trade-offs in [12] and [30] are (nearly) exactly optimal, the communication efficiency is only order-optimal. That is, under an exact  $b$ -bit communication constraint, the MSEs of [12] (denoted as FT21) and MMRC [30] may be suboptimal. In this work, we aim to achieve the exact-optimal MSE under both communication and privacy constraints.
Furthermore, we show that SQKR, FT21, and MMRC can be viewed as special cases in our framework – i.e., (random) coding with their own codebook design. We elaborate on this in Section 5 and provide more details on prior work in Appendix I.
In this section, we formally define LDP (with shared randomness) and describe our problem setting.
Local Differential Privacy (LDP) A randomized algorithm  $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$  is  $\varepsilon$ -LDP if
$$
\forall x, x ^ {\prime} \in \mathcal {X}, y \in \mathcal {Y}, \frac {\mathcal {Q} (y | x)}{\mathcal {Q} (y | x ^ {\prime})} \leq e ^ {\varepsilon}. \tag {1}
$$
LDP with Shared Randomness In this work, we assume that the encoder and the decoder have access to a shared source of randomness  $U \in \mathcal{U}$ , where the random encoder (randomizer)  $\mathcal{Q}$  privatizes  $x$  with additional randomness  $U$ . Then, the corresponding  $\varepsilon$ -LDP constraint is
$$
\forall x, x ^ {\prime} \in \mathcal {X}, y \in \mathcal {Y}, \frac {\mathcal {Q} (y | x , u)}{\mathcal {Q} (y | x ^ {\prime} , u)} \leq e ^ {\varepsilon} \tag {2}
$$
for  $P_U$  -almost all  $u$
Notation We let  $\mathbb{S}^{d - 1} = \{u\in \mathbb{R}^d:||u||_2 = 1\}$  denote the unit sphere,  $e_i\in \mathbb{R}^d$  the standard basis vectors for  $i = 1,\ldots ,d$ $\lfloor k\rfloor$  the greatest integer less than or equal to  $k$ , and  $[M] = \{1,\dots ,M\}$ .
Problem Setting We consider the  $\ell_2$  mean estimation problem with  $n$  users where each user  $i$  has a private unit vector  $v_{i}\in \mathbb{S}^{d - 1}$  for  $1\leq i\leq n$ . The server wants to recover the mean  $\frac{1}{n}\sum_{i = 1}^{n}v_{i}$  after each user sends a message using up to  $b$ -bits under an  $\varepsilon$ -LDP constraint. We allow shared randomness between each user and the server. More concretely, the  $i$ -th user and the server both have access to a random variable  $U_{i}\in \mathbb{R}^{t}$  (which is independent of the private local vector  $v_{i}$ ) for some  $t\geq 1$  and the  $i$ -th user has a random encoder (randomizer)  $f_{i}:\mathbb{S}^{d - 1}\times \mathbb{R}^{t}\to [M]$ , where  $M = 2^b$ . We denote by  $Q_{f_i}(m_i|v_i,u_i)$  the transition probability induced by the random encoder  $f_{i}$ , i.e., the probability that  $f_{i}$  outputs  $m_{i}$  given the source  $v_{i}$  and the shared randomness  $u_{i}$  is
$$
\Pr \left[ f _ {i} \left(v _ {i}, u _ {i}\right) = m _ {i} \right] = Q _ {f _ {i}} \left(m _ {i} \mid v _ {i}, u _ {i}\right). \tag {3}
$$
We require that the random encoder  $f_{i}$  satisfies  $\varepsilon$ -LDP, i.e.,
$$
\frac {Q _ {f _ {i}} \left(m _ {i} \mid v _ {i} , u _ {i}\right)}{Q _ {f _ {i}} \left(m _ {i} \mid v _ {i} ^ {\prime} , u _ {i}\right)} \leq e ^ {\varepsilon} \tag {4}
$$
for all  $v_{i}, v_{i}^{\prime} \in \mathbb{S}^{d - 1}$ ,  $m_{i} \in [M]$  and  $P_{U_i}$ -almost all  $u_{i} \in \mathbb{R}^{t}$ .
The server receives  $m_{i} = f_{i}(v_{i}, U_{i})$  from all users and generates unbiased estimate of the mean  $\mathcal{A}(m_1, \ldots, m_n, U_1, \ldots, U_n)$  that satisfies
$$
\mathbb {E} \left[ \mathcal {A} \left(m _ {1}, \dots , m _ {n}, U _ {1}, \dots , U _ {n}\right) \right] = \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i}. \tag {5}
$$
Then, the goal is to minimize the worst-case error
$$
\operatorname {E r r} _ {n} \left(f, \mathcal {A}, P _ {U ^ {n}}\right) = \sup  _ {v _ {1}, \dots , v _ {n} \in \mathbb {S} ^ {d - 1}} \mathbb {E} \left[ \left\| \mathcal {A} \left(m _ {1}, \dots , m _ {n}, U _ {1}, \dots , U _ {n}\right) - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| _ {2} ^ {2} \right], \tag {6}
$$
where  $f$  denotes the collection of all encoders  $(f_{1},\ldots ,f_{n})$ . We note that the error is also a function of the distribution of shared randomness, which was not the case for PrivUnit [2, 4].
3.1 Canonical Protocols
Similar to Asi et al. [2], we first define the canonical protocol when both communication and privacy constraints exist. The canonical protocols are where the server recovers each user's vector and estimates the mean by averaging them. In other words, the server has a decoder  $g_{i}:[M]\times \mathbb{R}^{t}\to \mathbb{S}^{d - 1}$  for  $1\leq i\leq M$  which is dedicated to the  $i$ -th user's encoder  $f_{i}$ , where the mean estimation is a simple additive aggregation, i.e.,
$$
\mathcal {A} ^ {+} \left(m _ {1}, \dots , m _ {n}, U _ {1}, \dots , U _ {n}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} \left(m _ {i}, U _ {i}\right). \tag {7}
$$
Our first result is that the exact-optimal mean estimation scheme should follow the above canonical protocol.
Lemma 3.1. For any  $n$ -user mean estimation protocol  $(f, \mathcal{A}, P_{U^n})$  that satisfies unbiasedness and  $\varepsilon$ -LDP, there exists an unbiased canonical protocol with decoders  $g = (g_1, \dots, g_n)$  that satisfies  $\varepsilon$ -LDP and achieves lower MSE, i.e.,
$$
\begin{array}{l} \operatorname {E r r} _ {n} \left(f, \mathcal {A}, P _ {U ^ {n}}\right) \geq \sup  _ {v _ {1}, \dots , v _ {n} \in \mathbb {S} ^ {d - 1}} \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} \left(m _ {i}, U _ {i}\right) - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| ^ {2} \right] (8) \\ \geq \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \operatorname {E r r} _ {1} \left(f _ {i}, g _ {i}, P _ {U _ {i}}\right), (9) \\ \end{array}
$$
where  $\mathrm{Err}_1(f,g,P_U)$  is the worst-case error for a single user with a decoder  $g$ .
The main proof techniques are similar to [2], where we define the marginalizing decoder:
$$
g _ {i} \left(m _ {i}, U _ {i}\right) = \mathbb {E} _ {\left\{v _ {j}, m _ {j}, U _ {j} \right\} _ {j \neq i}} \left[ n \mathcal {A} \left(\left\{m _ {j}, U _ {j} \right\} _ {j = 1} ^ {n}\right) \mid f _ {i} \left(v _ {i}, U _ {i}\right) = m _ {i}, U _ {i} \right]. \tag {10}
$$
The expectation in (10) is with respect to the uniform distribution of  $v_{j}$ 's. We defer the full proof to Appendix A.
Since the exact-optimal  $n$ -user mean estimation scheme is simply additively aggregating user-wise exact-optimal scheme, throughout the paper, we will focus on the single-user case and drop the index  $i$  when it is clear from the context. In this simpler formulation, we want the server to have an unbiased estimate  $\hat{v} = g(m,U)$ , i.e.,
$$
\begin{array}{l} v = \mathbb {E} _ {P _ {U}, f} [ g (f (v, U), U) ] (11) \\ = \mathbb {E} _ {P _ {U}} \left[ \sum_ {m = 1} ^ {M} g (m, U) Q _ {f} (m | v, U) \right] (12) \\ \end{array}
$$
for all  $v \in \mathbb{S}^{d - 1}$ . We assume that the decoder  $g: [M] \times \mathbb{R}^t \to \mathbb{R}^d$  is deterministic, since the randomized decoder does not improve the performance. Then, the corresponding error becomes
$$
\begin{array}{l} D (v, f, g, P _ {U}) = \mathbb {E} _ {P _ {U}, f} \left[ \| g (f (U, v), U) - v \| ^ {2} \right] (13) \\ = \mathbb {E} _ {P _ {U}} \left[ \sum_ {m = 1} ^ {M} \| g (m, U) - v \| ^ {2} Q _ {f} (m | v, U) \right]. (14) \\ \end{array}
$$
Finally, we want to minimize the following worst-case error over all  $(f,g)$  pairs that satisfy the unbiasedness condition in (12)
$$
\operatorname {E r r} _ {1} (f, g, P _ {U}) = \sup  _ {v \in \mathbb {S} ^ {d - 1}} D (v, f, g, P _ {U}). \tag {15}
$$
3.2 Exact Optimality of the Codebook
We propose a special way of leveraging shared randomness, which we term as random codebook. First, we define a codebook  $U^{M} = (U_{1},\ldots ,U_{M})\in (\mathbb{R}^{d})^{M}$ , consisting of  $M$  number of  $d$ -dimensional random vectors generated via shared randomness (i.e., both the server and the user know these random vectors). We then define the corresponding simple selecting decoder  $g^{+}:[M]\times (\mathbb{R}^{d})^{M}\to \mathbb{R}^{d}$ , which simply picks the  $m$ -th vector of the codebook upon receiving the message  $m$  from the user:
$$
g ^ {+} (m, U ^ {M}) = U _ {m}. \tag {16}
$$
Our first theorem shows that there exists a scheme with a random codebook and a simple selecting decoder that achieves the exact-optimal error. More precisely, instead of considering the general class of shared randomness (with general dimension  $t$ ) and the decoder, it is enough to consider the random codebook  $U^{M} \in (\mathbb{R}^{d})^{M}$  as the shared randomness and the simple selector  $g^{+}$  as the decoder.
Lemma 3.2. For any  $f, g$ ,  $P_U$  with  $U \in \mathbb{R}^t$  that are unbiased and that satisfy  $\varepsilon$ -LDP, there exists a shared randomness  $\tilde{U}^M \in (\mathbb{R}^d)^M$  and random encoder  $f_0: \mathbb{S}^{d-1} \times (\mathbb{R}^d)^M \to [M]$  such that
$$
D (v, f, g, P _ {U}) = D \left(v, f _ {0}, g ^ {+}, P _ {\tilde {U} ^ {M}}\right) \tag {17}
$$
for all  $v\in \mathbb{S}^{d - 1}$ , where  $f_0,g^+,P_{\tilde{u}^M}$  also satisfy unbiasedness and  $\varepsilon$  LDP.
The main step of the proof is to set an implicit random codebook with codewords  $\tilde{U}_m = g(m,U)$  for  $m = 1,\ldots ,2^b$  and show that we can obtain an essentially equivalent scheme with a different form of shared randomness  $\tilde{U}^{M}$ , which is an explicit random codebook. The detailed proof is given in Appendix B. Thus, without loss of generality, we can assume  $t = M\times d$  and the random codebook  $U^{M}$  is the new shared randomness, where the decoder is a simple selector. Since we fix the decoder, we drop  $g$  to simplify our notation. We say the random encoder  $f$  satisfies unbiasedness condition if
$$
\mathbb {E} _ {P _ {U}} \left[ \sum_ {m = 1} ^ {M} U _ {m} Q _ {f} (m | v, U ^ {M}) \right] = v, \tag {18}
$$
and the worst-case error is
$$
\begin{array}{l} \operatorname {E r r} (f, P _ {U ^ {M}}) = \sup  _ {v \in \mathbb {S} ^ {d - 1}} D (v, f, P _ {U ^ {M}}) (19) \\ = \sup  _ {v \in \mathbb {S} ^ {d - 1}} \mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} \| U _ {m} - v \| ^ {2} Q _ {f} (m | v, U) \right]. (20) \\ \end{array}
$$
Thus, the goal is now to find the exact-optimum codebook generating distribution  $P_{U^M}$ , and the random encoder  $f$  (or the probability assignment  $Q_f(\cdot | v, U)$ ). We then argue that the exact-optimal codebook should be rotationally symmetric.
Definition 3.3. A random codebook  $U^M \in (\mathbb{R}^d)^M$  is rotationally symmetric if  $(U_1, \ldots, U_M) \stackrel{(d)}{=} (A_0U_1, \ldots, A_0U_M)$  for any  $d \times d$  orthonormal matrix  $A_0$ .
The next lemma shows that the exact-optimal  $P_{U^M}$  is rotationally symmetric.
Lemma 3.4. Let  $P_{U^M}$  be a codebook generating distribution, and suppose random encoder  $f$  satisfies unbiasedness and  $\varepsilon$ -LDP. Then, there exists a random encoder  $f_1$  and rotationally symmetric random codebook  $\tilde{U}^M$  such that
$$
\operatorname {E r r} \left(f, P _ {U M}\right) \geq \operatorname {E r r} \left(f _ {1}, P _ {\bar {U} M}\right), \tag {21}
$$
which also satisfies unbiasedness and  $\varepsilon$ -LDP.
This is mainly because the goal is to minimize the worst-case error, and the codebook-generating distribution should be symmetric in all directions. The proof is provided in Appendix C. The next lemma shows that the exact-optimal scheme has constant error for all  $v \in \mathbb{S}^{d - 1}$ .
Lemma 3.5. For any rotationally symmetric codebook generating distribution  $P_{U^M}$  and an unbiased randomized encoder  $f$  that satisfies  $\varepsilon$ -LDP, there exists a random encoder  $f_2$  such that
$$
\operatorname {E r r} \left(f, P _ {U ^ {M}}\right) \geq \operatorname {E r r} \left(f _ {2}, P _ {U ^ {M}}\right), \text {w h e r e} D \left(v, f _ {2}, P _ {U ^ {M}}\right) = D \left(v ^ {\prime}, f _ {2}, P _ {U ^ {M}}\right) \tag {22}
$$
for all  $v, v' \in \mathbb{S}^{d-1}$ .
The formal proof is given in Appendix D. Since the codebook is symmetric (Lemma 3.4), the exact-optimal encoding strategy remains the same for any input  $v$ . Thus, without loss of generality, we can assume that the input is a standard unit vector  $v = e_1 = (1,0,\dots,0)$ .
3.3 Rotationally Symmetric Simplex Codebook
Now, we focus on a particular rotationally symmetric codebook. Notice that the codebook  $U^{M}$  has a similar role to the codebook in lossy compression, in the sense that we prefer the codeword  $U_{m}$  close to the input vector  $v$ . Thus, it is natural to consider the maximally separated codebook so that the  $M$  vectors  $U_{1},\ldots ,U_{M}$  cover the source space effectively. For  $M < d$ , the maximally separated  $M$  vectors on the unit sphere  $\mathbb{S}^{d - 1}$  is a simplex. More precisely, let  $s_1,\dots,s_M\in \mathbb{R}^d$  form a simplex:
$$
\left(s _ {i}\right) _ {j} = \left\{ \begin{array}{l l} \frac {M - 1}{\sqrt {M (M - 1)}} & \text {i f} i = j \\ - \frac {1}{\sqrt {M (M - 1)}} & \text {i f} i \neq j \text {a n d} j \leq M. \\ 0 & \text {i f} j > M \end{array} \right. \tag {23}
$$
Then, we can define the rotationally symmetric simplex codebook  $U^{M}$
$$
\left(U _ {1}, U _ {2}, \dots , U _ {M}\right) \stackrel {(d)} {=} \left(r A s _ {1}, r A s _ {2}, \dots , r A s _ {M}\right), \tag {24}
$$
where  $A$  is uniformly drawn orthogonal matrix and  $r > 0$  is a normalizing constant. We then need to find the corresponding encoder  $f$  that minimizes the error. Recall that the error is
$$
\mathbb {E} _ {P _ {U M}} \left[ \sum_ {m = 1} ^ {M} \| U _ {m} - v \| ^ {2} Q _ {f} (m | v, U) \right], \tag {25}
$$
and it is natural to assign high probabilities to the message  $m$  with low distortion  $\| U_m - v\|^2$  as long as  $\varepsilon$ -LDP constraint allows. More precisely, we call the following probability assignment "k-closest" encoding:
$$
Q _ {f} (m | v, U ^ {M}) = \left\{ \begin{array}{l l} \frac {e ^ {\varepsilon}}{k e ^ {\varepsilon} + (M - k)} & \text {i f} \| v - U _ {m} \| ^ {2} \text {i s o n e o f t h e} \lfloor k \rfloor \text {s m a l l e s t} \\ \frac {(k - \lfloor k \rfloor) (e ^ {\varepsilon} - 1) + 1}{k e ^ {\varepsilon} + (M - k)} & \text {i f} \| v - U _ {m} \| ^ {2} \text {i s t h e} \lfloor k \rfloor + 1 \text {- t h s m a l l e s t}, \\ \frac {1}{k e ^ {\varepsilon} + (M - k)} & \text {o t h e r w i s e} \end{array} \right. \tag {26}
$$
where we allow non-integer  $k$ . The choice of  $r = r_k$  is described in Section 3.4. We call this approach Randomly Rotating Simplex Coding (RRSC) and provide the pseudocode in Algorithm 1. We note that the codewords  $U_m$ 's with smallest  $\| v - U_m \|^2$  and codewords  $U_m$ 's with largest  $\langle v, U_m \rangle$  coincide for a codebook with fixed-norm codewords  $U_m$ 's, which is the case for the rotationally symmetric simplex codebook. Our main theorem is that the  $k$ -closest encoding is exact-optimum if the codebook generating distribution is rotationally symmetric simplex.
Theorem 3.6. For a rotationally symmetric simplex codebook, there exists a  $k$  such that the "  $k$ -closest" encoding is the exact-optimum unbiased scheme that satisfies  $\varepsilon$ -LDP constraint.