SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {P _ {\bar {U} M}} \left[ Q _ {f 1} (m | v, \bar {U} ^ {M}) \bar {U} _ {m} \right] = \mathbb {E} _ {A, P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, A \bar {U} ^ {M}) \bar {U} _ {m} \right] (64) \\ = \mathbb {E} _ {A, P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) A ^ {\intercal} U _ {m} \right] (65) \\ = \mathbb {E} _ {A} \left[ A ^ {\intercal} \mathbb {E} _ {P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) U _ {m} \right] \right] (66) \\ = \mathbb {E} _ {A} [ A ^ {\intercal} A v ] (67) \\ = v. (68) \\ \end{array}
$$
The key step is that the original encoder  $f$  is unbiased, which implies
$$
\mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) U _ {m} \right] = A v \tag {69}
$$
for all  $A$ .
Now, we are ready to prove the main inequality.
$$
\begin{array}{l} \operatorname {E r r} (f, P _ {U ^ {M}}) = \sup  _ {v} D (v, f, P _ {U ^ {M}}) (70) \\ \geq \mathbb {E} _ {A} \left[ D \left(A v, f, P _ {U ^ {M}}\right) \right] (71) \\ = \mathbb {E} _ {A} \left[ \mathbb {E} _ {P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) \| U _ {m} - A v \| ^ {2} \right] \right] (72) \\ = \mathbb {E} _ {P _ {U M}, A} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, A \bar {U} ^ {M}) \| \bar {U} _ {m} - v \| ^ {2} \right] (73) \\ = \mathbb {E} _ {P _ {\bar {U} M}} \left[ \sum_ {m = 1} ^ {M} \mathbb {E} _ {A} \left[ Q _ {f} (m | A v, A \bar {U} ^ {M}) | \bar {U} ^ {M} \right] \| \bar {U} _ {m} - v \| ^ {2} \right] (74) \\ = \mathbb {E} _ {P _ {\bar {U} M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f _ {1}} (m | v, \bar {U} ^ {M}) \| \bar {U} _ {m} - v \| ^ {2} \right] (75) \\ = D \left(v, f _ {1}, P _ {\bar {U} ^ {M}}\right). (76) \\ \end{array}
$$
for all  $v$ . This concludes the proof.
D Proof of Lemma 3.5
Proof. For  $v, v' \in \mathbb{S}^{d-1}$ , let  $A_0$  be an orthonormal matrix such that  $v' = A_0v$ . Let  $f_2$  be a randomized encoder such that
$$
f _ {2} (v, U ^ {M}) = f (A v, A U ^ {M}) \tag {77}
$$
for uniform random orthonormal matrix. Then,
$$
Q _ {f _ {2}} (m | v, U ^ {M}) = \mathbb {E} _ {A} \left[ Q _ {f} (m | A v, A U ^ {M}) \right]. \tag {78}
$$
Similar to the previous proofs,  $Q_{f_2}$  is a well-defined probability distribution, and  $f_2$  is unbiased as well as  $\varepsilon$ -LDP. Since  $P_{U^M}$  is rotationally symmetric and  $f_2$  is also randomized via the uniform random orthogonal matrix, we have
$$
D \left(v ^ {\prime}, f _ {2}, P _ {U ^ {M}}\right) = D \left(A _ {0} v, f _ {2}, P _ {U ^ {M}}\right) = D \left(v, f _ {2}, P _ {U ^ {M}}\right). \tag {79}
$$
Compared to a given randomizer  $f$ , we have
$$
\begin{array}{l} \operatorname {E r r} \left(f, P _ {U ^ {M}}\right) \geq \mathbb {E} _ {A} \left[ D \left(A v, f, P _ {U ^ {M}}\right) \right] (80) \\ = \mathbb {E} _ {A, P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) \| A v - U ^ {M} \| ^ {2} \right] (81) \\ = \mathbb {E} _ {A, P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, U ^ {M}) \| v - A ^ {\intercal} U ^ {M} \| ^ {2} \right] (82) \\ = \mathbb {E} _ {A, P _ {U M}} \left[ \sum_ {m = 1} ^ {M} Q _ {f} (m | A v, A U ^ {M}) \| v - U ^ {M} \| ^ {2} \right] (83) \\ = \mathbb {E} _ {P _ {U M}} \left[ \sum_ {m = 1} ^ {M} \mathbb {E} _ {A} \left[ Q _ {f} (m | A v, A U ^ {M}) \right] \| v - U ^ {M} \| ^ {2} \right] (84) \\ = D \left(v, f _ {2}, P _ {U ^ {M}}\right) (85) \\ \end{array}
$$
for all  $v\in \mathbb{S}^{d - 1}$  . This concludes the proof.
E Proof of Theorem 3.6
Proof. The rotationally symmetric simplex codebook with normalization constant  $r$  is  $(rAs_1, \ldots, rAs_M)$ . Let  $f$  be the unbiased encoder satisfying  $\varepsilon$ -LDP. Let  $Q_{\mathrm{max}} = \max Q_f(m|v, rAs^M)$  and  $Q_{\mathrm{min}} = \min Q_f(m|v, rAs^M)$ , our objective is to demonstrate that  $Q_{\mathrm{max}}$  is less than or equal to  $e^{\varepsilon}Q_{\mathrm{min}}$ . We will employ a proof by contradiction to establish this. Suppose  $Q_f(m_1|v_1, rA_1s^M) > e^{\varepsilon}Q_f(m_2|v_2, rA_2s^M)$  for some  $m_1, v_1, A_1, m_2, v_2$ , and  $A_2$ . Let  $\tilde{A}$  be the row switching matrix where  $r\tilde{A}A_1s_{m_1} = rA_1s_{m_2}$  and  $r\tilde{A}A_1s_{m_2} = rA_1s_{m_1}$ , then we have
$$
Q _ {f} \left(m _ {1} \mid v _ {1}, r A _ {1} s ^ {M}\right) = Q _ {f} \left(m _ {2} \mid \tilde {A} v _ {1}, r \tilde {A} A _ {1} s ^ {M}\right). \tag {86}
$$
We further let  $A'$  be an orthogonal matrix such that  $A' \tilde{A} A_1 = A_2$ , then
$$
Q _ {f} \left(m _ {2} | \tilde {A} v _ {1}, r \tilde {A} A _ {1} s ^ {M}\right) = Q _ {f} \left(m _ {2} | A ^ {\prime} \tilde {A} v _ {1}, r A ^ {\prime} \tilde {A} A _ {1} s ^ {M}\right) \tag {87}
$$
$$
= Q _ {f} \left(m _ {2} \mid A ^ {\prime} \tilde {A} v _ {1}, r A _ {2} s ^ {M}\right) \tag {88}
$$
If we let  $v_{1}^{\prime} = A^{\prime}\tilde{A} v_{1}$ , then
$$
Q _ {f} \left(m _ {2} \mid v _ {1} ^ {\prime}, r A _ {2} s ^ {M}\right) = Q _ {f} \left(m _ {1} \mid v _ {1}, r A _ {1} s ^ {M}\right) \tag {89}
$$
$$
> e ^ {\varepsilon} Q _ {f} \left(m _ {2} \mid v _ {2}, r A _ {2} s ^ {M}\right), \tag {90}
$$
which contradicts the  $\varepsilon$ -LDP constraint.
For an unbiased encoder, the error is
$$
\begin{array}{l} \mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} \| U _ {m} - v \| ^ {2} Q _ {f} (m | v, U ^ {M}) \right] = \mathbb {E} _ {P _ {U ^ {M}}} \left[ \sum_ {m = 1} ^ {M} \| U _ {m} \| ^ {2} Q _ {f} (m | v, U ^ {M}) \right] - 1 (91) \\ = r ^ {2} - 1. (92) \\ \end{array}
$$
Thus, we need to find  $r$  that minimizes the error.
On the other hand, the encoder needs to satisfy unbiasedness. Without loss of generality, we assume  $v = e_1$ , then we need
$$
\mathbb {E} _ {A} \left[ \sum_ {m = 1} ^ {M} r A s _ {m} Q _ {f} \left(m \mid e _ {1}, r A s ^ {M}\right) \right] = e _ {1}, \tag {93}
$$
where the expectation is with respect to the random orthonormal matrix  $A$ . If we focus on the first index of the vector, then
$$
r \times \mathbb {E} _ {a} \left[ \sum_ {m = 1} ^ {M} a ^ {\mathsf {T}} s _ {m} Q _ {f} \left(m \mid e _ {1}, r A s ^ {M}\right) \right] = 1, \tag {94}
$$
where  $a^{\mathsf{T}} = (a_{1},\ldots ,a_{d})$  is the first row of  $A$  and has uniform distribution on the sphere  $\mathbb{S}^{d - 1}$ . Thus, it is clear that assigning higher probability (close to  $Q_{\mathrm{max}}$ ) to the larger  $a^{\mathsf{T}}s_m$ .
If  $Q_{\mathrm{max}}$  is strictly smaller than  $e^{\varepsilon}Q_{\mathrm{min}}$ , then we can always scale up the larger probabilities and scale down the lower probabilities to keep the probability sum to one (while decreasing the error). Hence, we can assume that  $Q_{\mathrm{min}} = q_0$  and  $Q_{\mathrm{max}} = e^{\varepsilon}q_{0}$  for some  $1 > q_{0} > 0$ .
Now, let  $k$  be such that
$$
(M - \lfloor k \rfloor - 1) q _ {0} + q _ {i} + \lfloor k \rfloor e ^ {\varepsilon} q _ {0} = 1, \tag {95}
$$
where  $q_{i}$  is an intermediate value such that  $q_{i} \in [q_{0}, e^{\varepsilon} q_{0}]$ . Then, the optimal strategy is clear: (i) assign  $e^{\varepsilon} q_{0}$  to  $\lfloor k \rfloor$ -th closest codewords  $s_{m}$ 's, (ii) assign  $q_{i}$  to the  $(\lfloor k \rfloor + 1)$ -th closest codeword, and (iii) assign  $q_{0}$  to the remaining codewords. This implies that the  $k$ -closest coding is optimal.
F Proof of Lemma 3.7
Proof. Following (28) with  $U_{m} = As_{m}$  and  $v = e_{1}$ , we have
$$
r _ {k} \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \mathbb {E} \left[ \sum_ {m \in T _ {k} (e _ {1}, A \cdot S)} A \cdot s _ {m} \right]
$$
$$
\begin{array}{l} = r _ {k} \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \mathbb {E} \left[ \sum_ {m \in \arg \max  _ {k} (\{\langle e _ {1}, A s _ {1} \rangle , \dots , \langle e _ {1}, A s _ {M} \rangle \})} A \cdot s _ {m} \right] \\ = e _ {1}. \\ \end{array}
$$
By focusing on the first coordinate of the above equation and observing that  $\langle e_1, As_M \rangle = \langle a, s_m \rangle$  where  $a$  is the first row of the rotation matrix  $A$ , we must have
$$
r _ {k} \cdot \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \mathbf {E} _ {a \sim \operatorname {u n i f} \left(\mathbf {S} ^ {d - 1}\right)} \left[ \sum_ {m \in \operatorname {T o p} _ {k} (\{\langle a, s _ {1} \rangle , \dots , \langle a, s _ {M} \rangle \})} \langle a, s _ {m} \rangle \right] = 1. \tag {96}
$$
Note that since  $A$  is a random orthogonal matrix drawn from the Haar measure on  $SO(d)$ ,  $a$  is distributed uniformly over the unit sphere  $\mathbb{S}^{d-1}$ .
Next, observe that by definition,
$$
s _ {m} = \frac {M}{\sqrt {M (M - 1)}} e _ {m} - \frac {1}{\sqrt {M (M - 1)}} \mathbf {1} _ {M},
$$
where  $\mathbf{1}_M = (\underbrace{1,1,\ldots,1}_{M\text{entries}},0,\ldots,0) \in \{0,1\}^d$  (that is,  $(\mathbf{1}_M)_m = \mathbb{1}_{\{m \leq M\}}$ ). Therefore,
$$
\langle a, s _ {m} \rangle = \frac {M}{\sqrt {M (M - 1)}} a _ {m} - \frac {1}{\sqrt {M (M - 1)}} \langle a, \mathbf {1} _ {M} \rangle ,
$$
and hence plugging in (96) yields
$$
\begin{array}{l} r _ {k} \cdot \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \mathbf {E} _ {a \sim \operatorname {u n i f} \left(\mathbf {S} ^ {d - 1}\right)} \left[ \sum_ {m \in \operatorname {T o p} _ {k} (\{\langle a, s _ {1} \rangle , \dots , \langle a, s _ {M} \rangle \})} \langle a, s _ {m} \rangle \right] \\ = r _ {k} \cdot \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \cdot \frac {M}{\sqrt {M (M - 1)}} \mathbf {E} _ {a \sim \operatorname {u n i f} (\mathbf {S} ^ {d - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i | M)} - \frac {k}{M} \langle a, \mathbf {1} _ {M} \rangle \right] \\ = r _ {k} \cdot \frac {e ^ {\varepsilon} - 1}{k e ^ {\varepsilon} + (M - k)} \cdot \sqrt {\frac {M}{M - 1}} \cdot \underbrace {\mathbf {E} _ {a \sim \operatorname {u n i f} (\mathbf {S} ^ {d - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i | M)} \right]} _ {:= C _ {k}}, \\ \end{array}
$$
where (1)  $a_{(i|M)}$  denotes the  $i$ -th largest entry of the first  $M$  coordinates of  $a$  and (2) the last equality holds since  $a$  is uniformly distributed over  $\mathbb{S}^{d-1}$ .
G Proof of Lemma 3.8
Proof. First of all, observe that
$$
\begin{array}{l} \mathbb {E} _ {a \sim \operatorname {u n i f} (\mathbb {S} ^ {d - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i | M)} \right] \\ = \mathbb {E} _ {a \sim \operatorname {u n i f} (\mathbb {S} ^ {d - 1})} \left[ \mathbb {E} \left[ \sum_ {i = 1} ^ {k} a _ {(i | M)} \left| \sum_ {i = 1} ^ {M} a _ {i} ^ {2} \right. \right] \right] \\ \stackrel {\mathrm {(a)}} {=} \mathbb {E} _ {a \sim \operatorname {u n i f} (\mathbb {S} ^ {d - 1})} \left[ \sqrt {\sum_ {i = 1} ^ {M} a _ {i} ^ {2}} \cdot \mathbb {E} _ {\left(a _ {1} ^ {\prime}, \dots , a _ {M} ^ {\prime}\right) \sim \operatorname {u n i f} (\mathbb {S} ^ {M - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i)} ^ {\prime} \right] \right] \\ = \underbrace {\mathbb {E} _ {a \sim \operatorname {u n i f} (\mathbb {S} ^ {d - 1})} \left[ \sqrt {\sum_ {i = 1} ^ {M} a _ {i} ^ {2}} \right]} _ {\text {(i)}} \cdot \underbrace {\mathbb {E} _ {\left(a _ {1} ^ {\prime} , \dots , a _ {M} ^ {\prime}\right) \sim \operatorname {u n i f} (\mathbb {S} ^ {M - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i)} ^ {\prime} \right]} _ {\text {(i i)}}, \\ \end{array}
$$
where (a) holds due to the spherical symmetry of  $a$ . Next, we bound (i) and (ii) separately.
Claim G.1 (Bounding (i)). For any  $d \geq M > 2$ , it holds that
$$
\sqrt {\frac {M - 2}{d - 2}} \leq \mathbb {E} _ {a \sim \operatorname {u n i f} \left(\mathbb {S} ^ {d - 1}\right)} \left[ \sqrt {\sum_ {i = 1} ^ {M} a _ {i} ^ {2}} \right] \leq \sqrt {\frac {M}{d - 2}}. \tag {97}
$$
Proof of Claim G.1. Observe that when  $a$  is distributed uniformly over  $\mathbb{S}^{d - 1}$ , it holds that
$$
\left(a _ {1}, a _ {2}, \dots , a _ {d}\right) \stackrel {{d}} {{=}} \left(\frac {Z _ {1}}{\sqrt {\sum_ {i = 1} ^ {d} Z _ {i} ^ {2}}}, \frac {Z _ {2}}{\sqrt {\sum_ {i = 1} ^ {d} Z _ {i} ^ {2}}}, \dots , \frac {Z _ {d}}{\sqrt {\sum_ {i = 1} ^ {d} Z _ {i} ^ {2}}}\right),
$$
where  $A \stackrel{d}{=} B$  denotes  $A$  and  $B$  have the same distribution, and  $Z_{1}, \ldots, Z_{d} \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ . As a result, we must have
$$
\mathbb {E} _ {a \sim \mathrm {u n i f} (\mathbb {S} ^ {d - 1})} \left[ \sqrt {\sum_ {i = 1} ^ {M} a _ {i} ^ {2}} \right] = \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} \stackrel {\mathrm {i . i . d .}} {\sim} \mathcal {N} (0, 1)} \left[ \sqrt {\frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2} + \sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}} \right].
$$
By Jensen's inequality, it holds that
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} ^ {\mathrm {i . i . d .}} \sim \mathcal {N} (0, 1)} \left[ \sqrt {\frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2} + \sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}} \right] \\ = \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} \stackrel {\mathrm {i . i . d}} {\sim}} \mathcal {N} (0, 1) \left[ \sqrt {\frac {1}{1 + \frac {\sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}}} \right] \\ \stackrel {\mathrm {(a)}} {\geq} \sqrt {1} \frac {1}{1 + \mathbb {E} _ {Z _ {1} , \dots , Z _ {M} \stackrel {\text {i . i . d}} {\sim}} \mathcal {N} (0 , 1) \left[ \frac {\sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}} \right]} \\ \stackrel {\mathrm {(b)}} {=} \sqrt {\frac {1}{1 + \frac {d - M}{M - 2}}} \\ = \sqrt {\frac {M - 2}{d - 2}}, \\ \end{array}
$$
where (a) holds since  $x \mapsto \sqrt{1 / (1 + x)}$  is a convex mapping for  $x > 0$ , and (b) holds due to the fact that  $\sum_{i} Z_{i}^{2}$  follows from a  $\chi^2$  distribution and that the ratio of two independent  $\chi^2$  random variables follows an  $F$ -distribution.
On the other hand, it also holds that
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} \stackrel {\mathrm {i . i . d .}} {\sim}} \mathcal {N} (0, 1) \left[ \sqrt {\frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2} + \sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}} \right] \\ \stackrel {\mathrm {(a)}} {\leq} \sqrt {\mathbb {E} _ {Z _ {1} , \ldots , Z _ {M} \stackrel {\mathrm {i . i . d .}} {\sim}} \mathcal {N} (0 , 1)} \left[ \frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2} + \sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}} \right] \\ = \sqrt {\mathbb {E} _ {Z _ {1} , \dots , Z _ {M} \stackrel {\mathrm {i . i . d .}} {\sim}} \mathcal {N} (0 , 1)} \left[ 1 - \frac {\sum_ {i = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}{\sum_ {i = 1} ^ {M} Z _ {i} ^ {2} + \sum_ {i ^ {\prime} = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}} \right] \\ = \sqrt {1 - \mathbb {E} _ {Z _ {1} , \ldots , Z _ {M} ^ {\mathrm {i . d .}}} \mathcal {N} (0 , 1)} \left[ \frac {1}{1 + \frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}}} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} \sqrt {1 - \frac {1}{1 + \mathbb {E} _ {Z _ {1} , \dots , Z _ {M} ^ {\text {i . i . d}}} \sim \mathcal {N} (0 , 1) \left[ \frac {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}{\sum_ {i = M + 1} ^ {d} Z _ {i ^ {\prime}} ^ {2}} \right]}} \\ \stackrel {\mathrm {(c)}} {=} \sqrt {1 - \frac {1}{1 + \frac {M}{d - M - 2}}} \\ = \sqrt {\frac {M}{d - 2}}, \\ \end{array}
$$
where (a) holds since  $\sqrt{\cdot}$  is concave, (b) holds since  $x\mapsto \frac{1}{1 + x}$  is convex, and (c) again is due to the fact that the ratio of two independent  $\chi^2$  random variables follows an  $F$ -distribution.
Claim G.2 (Bounding (ii)). As long as
-  $k \geq 400 \cdot \log 10$ ,
$\log (M / k)\geq \left(\frac{10^3\pi\log 2}{9}\right)^2,$
$$
\sqrt {\frac {k \log \left(\frac {M}{k}\right)}{2 4 \pi \log 2 M}} \leq \mathbb {E} _ {\left(a _ {1} ^ {\prime}, \dots , a _ {M} ^ {\prime}\right) \sim \operatorname {u n i f} \left(\mathbb {S} ^ {M - 1}\right)} \left[ \sum_ {i = 1} ^ {k} a _ {(i)} ^ {\prime} \right] \leq \sqrt {\frac {4 k \log M}{M}}. \tag {98}
$$
Proof of Claim G.2. We start by re-writing  $a^\prime$  ..
$$
(a _ {1} ^ {\prime}, a _ {2} ^ {\prime}, \dots , a _ {M} ^ {\prime}) \stackrel {d} {=} \left(\frac {Z _ {1}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i}}}, \frac {Z _ {2}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i}}}, \dots , \frac {Z _ {M}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i}}}\right).
$$
$$
\left(a _ {(1)} ^ {\prime}, a _ {(2)} ^ {\prime}, \dots , a _ {(k)} ^ {\prime}\right) \stackrel {d} {=} \left(\frac {Z _ {(1)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}}, \frac {Z _ {(2)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}}, \dots , \frac {Z _ {(k)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}}\right),
$$
$$
\mathbb {E} _ {(a _ {1} ^ {\prime}, \ldots , a _ {M} ^ {\prime}) \sim \mathsf {u n i f} (\mathbb {S} ^ {M - 1})} \left[ \sum_ {i = 1} ^ {k} a _ {(i)} ^ {\prime} \right] = \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} ^ {\mathrm {i . i . d .}}} \mathcal {N} (0, 1) \left[ \frac {1}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \sum_ {i = 1} ^ {k} Z _ {(i)} \right].
$$
Upper bound. To upper bound the above, observe that
$$
\mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} ^ {\mathrm {i . i . d}} \sim \mathcal {N} (0, 1)} \left[ \frac {1}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \sum_ {i = 1} ^ {k} Z _ {(i)} \right] \leq k \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} ^ {\mathrm {i . i . d}} \sim \mathcal {N} (0, 1)} \left[ \frac {1}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} Z _ {(1)} \right].
$$
Let  $\mathcal{E}_1\coloneqq \left\{(Z_1,\dots,Z_M)\big|\sum_{i = 1}^{M}Z_i^2\leq M(1 - \gamma)\right\}$  where  $\gamma >0$  will be optimized later. Then it holds that
$$
\Pr \left\{\mathcal {E} _ {1} \right\} \leq e ^ {- \frac {M \gamma^ {2}}{4}}. \tag {99}
$$
On the other hand, the Borell-TIS inequality ensures
$$
\Pr \left\{\left| Z _ {(1)} - \mathbb {E} \left[ Z _ {(1)} \right] \right| > \xi \right\} \leq 2 e ^ {- \frac {\xi^ {2}}{2 \sigma^ {2}}}, \tag {100}
$$
where  $Z_{i}\sim \mathcal{N}(0,\sigma^{2})$  (in our case,  $\sigma = 1$ ). Since  $\mathbb{E}\left[Z_{(1)}\right]\leq \sqrt{2\log M}$ , it holds that
$$
\Pr \left\{Z _ {(1)} \geq \sqrt {2 \log M} + \xi \right\} \leq 2 e ^ {- \xi^ {2}}.
$$
Therefore, define  $\mathcal{E}_2\coloneqq \left\{Z_{(1)}\geq \sqrt{2\log M} +\xi \right\}$  and we obtain
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1}, \ldots , Z _ {M} \stackrel {\mathrm {i . i . d .}} {\sim}} \mathcal {N} (0, 1) \left[ \frac {1}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \sum_ {i = 1} ^ {k} Z _ {(i)} \right] \\ \leq k \mathbb {E} _ {Z _ {1}, \dots , Z _ {M}} \overset {\text {i . d .}} {\sim} \mathcal {N} (0, 1) \left[ \frac {1}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} Z _ {(1)} \right] \\ \leq k \cdot \left(\mathbb {E} \left[ \frac {Z _ {(1)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {1} \cap \mathcal {E} _ {2} \right] + \sup  _ {z _ {1}, \dots , z _ {m}} \left(\frac {z _ {(1)}}{\sqrt {\sum_ {i = 1} ^ {M} z _ {i} ^ {2}}}\right) \cdot \Pr \left(\mathcal {E} _ {1} ^ {c} \cup \mathcal {E} _ {2} ^ {c}\right)\right) \\ \leq k \cdot \left(\frac {\sqrt {2 \log M + \xi}}{M (1 - \gamma)} + 1 \cdot \left(e ^ {- M \gamma^ {2} / 4} + 2 e ^ {- \xi^ {2}}\right)\right) \\ \leq k \cdot \left(\frac {\sqrt {2 \log M + \sqrt {\log (M)}}}{0 . 9 \cdot M} + 1 \cdot \left(e ^ {- M / 4 0 0} + 2 / M\right)\right) \\ = \Theta \left(\frac {k \sqrt {\log M}}{M}\right), \\ \end{array}
$$
where the last inequality holds by picking  $\gamma = 0.1$  and  $\xi = \sqrt{\log M}$ .
Lower bound. The analysis of the lower bound is more sophisticated. To begin with, let
$$
\mathcal {E} _ {M} := \left\{\left(Z _ {1}, \dots , Z _ {M}\right) \left| \sum_ {i = 1} ^ {M} Z _ {i} ^ {2} \in [ M (1 - \gamma), M (1 + \gamma) ] \right. \right\}
$$
denote the good event such that the denominator of our target is well-controlled, where  $\gamma > 0$  again will be optimized later. By the concentration of  $\chi^2$  random variables, it holds that
$$
\Pr \left\{\mathcal {E} _ {M} ^ {c} \right\} \leq e ^ {- \frac {M}{2} (\gamma - \log (1 + \gamma))} + e ^ {- \frac {M \gamma^ {2}}{4}} \leq e ^ {- \frac {M}{2} \left(1 - \frac {1}{\sqrt {1 + \gamma}}\right) \gamma} + e ^ {- \frac {M \gamma^ {2}}{4}} \leq 2 e ^ {- \frac {M \gamma^ {2}}{4}}. \tag {101}
$$
Next, to lower bound  $\sum_{i=1}^{k} Z_{(i)}$ , we partition  $(Z_1, Z_2, \ldots, Z_M)$  into  $k$  blocks  $B_1, B_2, \ldots, B_k$  where each block contains at least  $N = \lfloor M / k \rfloor$  samples:  $B_j := [(j - 1) \cdot N + 1 : j \cdot N]$  for  $j \in [k - 1]$  and  $B_k = [M] \setminus \left( \bigcup_{j = 1}^{k - 1} B_j \right)$ . Define  $\tilde{Z}_{(1)}^{(j)}$  be the maximum samples in the  $j$ -th block:  $\tilde{Z}_{(1)}^{(j)} := \max_{i \in B_j} Z_i$ . Then, it is obvious that
$$
\sum_ {i = 1} ^ {k} Z _ {(i)} \geq \sum_ {j = 1} ^ {k} \tilde {Z} _ {(1)} ^ {(j)}.
$$
To this end, we define  $\mathcal{E}_1$  to be the good event that  $90\%$  of  $\tilde{Z}_{(1)}^{(j)}$ 's are large enough (i.e., concentrated to the expectation):
$$
\mathcal {E} _ {1} := \left\{\left| \left\{j \in [ k ] \middle | \tilde {Z} _ {(1)} ^ {(j)} \geq \frac {\sqrt {\log N}}{\sqrt {\pi \log 2}} - \log 1 0 0 \right\} \right| > 0. 9 k \right\}.
$$
Note that by the Borell-TIS inequality, for any  $j \in [k]$ ,
$$
\operatorname * {P r} \left\{\tilde {Z} _ {(1)} ^ {(j)} \geq \frac {\sqrt {\log N}}{\sqrt {\pi \log 2}} - \xi \right\} \geq 1 - 2 e ^ {- \xi^ {2}},
$$
so setting  $\xi = \log 100$  implies  $\operatorname*{Pr}\left\{\tilde{Z}_{(1)}^{(j)}\geq \frac{\sqrt{\log N}}{\sqrt{\pi\log 2}} -\xi \right\} \geq 0.98$ . Since blocks are independent with each other, applying Hoeffding's bound yields
$$
\Pr \left\{\mathcal {E} _ {1} \right\} \geq 1 - \Pr \left\{\operatorname {B i n o m} (k, 0. 9 8) \leq 0. 9 \right\} \geq 1 - e ^ {- k (0. 0 8) ^ {2}} \geq 0. 9,
$$
when  $k \geq 400 \cdot \log 10 \geq \log 10 / 0.08^2$ .
Next, we define a "not-too-bad" event where  $\sum_{j=1}^{k} \tilde{Z}_{(1)}^{(j)}$  is not catastrophically small:
$$
\mathcal {E} _ {2} := \left\{\sum_ {j = 1} ^ {k} \tilde {Z} _ {(1)} ^ {(j)} \geq - \frac {k}{\sqrt {M}} \xi \right\},
$$
for some  $\xi > 0$  to be optimized later. Observe that  $\mathcal{E}_2$  holds with high probability:
$$
\Pr \left\{\mathcal {E} _ {2} \right\} \stackrel {\text {(a)}} {\geq} \Pr \left\{\frac {k}{M} \sum_ {i = 1} ^ {M} Z _ {i} \geq - \frac {k}{\sqrt {M}} \xi \right\}
$$
$$
\stackrel {(b)} {\geq} 1 - e ^ {- \xi^ {2} / 2},
$$
where (a) holds since the each of the top-  $k$  values must be greater than  $k$  times the average, and (b) holds due to the Hoeffding's bound on the sum of i.i.d. Gaussian variables.
Lastly, a trivial bound implies that
$$
\inf  _ {a \in \mathbb {S} ^ {M - 1}} \sum_ {i = 1} ^ {k} a _ {(i)} \geq - \frac {k}{\sqrt {M}}.
$$
Now, we are ready to bound  $\mathbb{E}_{Z_1,\ldots,Z_M} \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0,1) \left[ \frac{1}{\sqrt{\sum_{i=1}^M Z_i^2}} \sum_{i=1}^k Z_i \right]$ . We begin by decomposing it into three parts:
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1}, \dots , Z _ {M} \stackrel {\mathrm {i . d .}} {\sim}} \mathcal {N} (0, 1) \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \right] = \operatorname * {P r} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {1} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right] \\ + \Pr \left\{\mathcal {E} _ {1} ^ {c} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {1} ^ {c} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right] \\ + \operatorname * {P r} \left\{\mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right]. \\ \end{array}
$$