SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We verify that  $X_{i}$  satisfies the small-ball condition by utilizing Eq (20). Recall that  $\langle \theta ,X_i\rangle^2 /\sigma_X^2\| \theta \| ^2$  follows the non-central  $\chi^2$  distribution with degree of freedom 1 and non-centrality parameter  $\lambda^2 = \langle \theta ,\mu \rangle^2 /\sigma_X^2\| \theta \| ^2$  for any  $\theta \in \mathbb{R}^d\setminus \{0\}$ . By Eq (20), we have
$$
\mathbb {P} \left\{\frac {\left| \langle \theta , X _ {i} \rangle \right|}{\sigma_ {X} ^ {2} \| \theta \| ^ {2}} ^ {2} <   t ^ {2} \right\} \leq \sqrt {\frac {2}{\pi}} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t.
$$
Noting that  $\left\| \Sigma^{1 / 2}\theta \right\| ^2 = \sigma_X^2\| \theta \| ^2 +\langle \theta ,\mu \rangle^2$  , we have
$$
\begin{array}{l} \mathbb {P} \left\{\langle \theta , X _ {i} \rangle^ {2} \leq t ^ {2} \left\| \Sigma^ {1 / 2} \theta \right\| ^ {2} \right\} \leq \sqrt {\frac {2}{\pi} \left(1 + \frac {\langle \theta , \mu \rangle^ {2}}{\sigma_ {X} ^ {2} \| \theta \| ^ {2}}\right)} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t \\ = \sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t. \tag {21} \\ \end{array}
$$
We divide into two cases,  $\lambda > t$  and  $\lambda \leq t$ , to derive an upper bound on Eq (21).
(Case  $\lambda > t$ ) Since  $(\lambda - t)^2 = \lambda^2 - 2\lambda t + t^2 \geq \lambda^2 - 2t^2 + t^2 = \lambda^2 - t^2$ , an upper bound on Eq (21) is obtained as
$$
\begin{array}{l} \sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t \\ \leq \sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {\lambda^ {2} - t ^ {2}}{2}} t. \\ \end{array}
$$
For positive numbers  $a$  and  $b$ ,  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ . Using this fact, we have
$$
\begin{array}{l} \sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t \\ \leq \sqrt {\frac {2}{\pi}} \left(\sqrt {1 + t ^ {2}} + \sqrt {\lambda^ {2} - t ^ {2}} e ^ {- \frac {\lambda^ {2} - t ^ {2}}{2}}\right) t. \\ \end{array}
$$
Since a function  $x \to xe^{-x^2 / 2}$  admits a maximum on  $x \in (0, \infty)$  of  $e^{-1/2}$ , we have  $\sqrt{\lambda^2 - t^2} e^{-\frac{\lambda^2 - t^2}{2}} \leq e^{-1/2}$ . Consequently, we have
$$
\sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t \leq \sqrt {\frac {1 6}{\pi e} (1 + t ^ {2})} t, \tag {22}
$$
where we use the fact  $(1 + e^{-1 / 2})^2\leq 8e^{-1}$
(Case  $\lambda \leq t$ ) We can easily verify that
$$
\sqrt {\frac {2}{\pi} (1 + \lambda^ {2})} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t \leq \sqrt {\frac {2}{\pi} (1 + t ^ {2})} t \tag {23}
$$
Combining Eqs (22) and (23), we have for every  $t > 0$
$$
\mathbb {P} \left\{\langle \theta , X _ {i} \rangle^ {2} \leq t ^ {2} \left\| \Sigma^ {1 / 2} \theta \right\| ^ {2} \right\} \leq \sqrt {\frac {1 6}{\pi e} (1 + t ^ {2})} t.
$$
For  $t^2 \in (0, -\frac{1}{2} + \frac{1}{2}\sqrt{1 + \frac{\pi e}{4}}]$ , we have
$$
\sqrt {\frac {1 6}{\pi e} (1 + t ^ {2})} t \leq \sqrt {\frac {8}{\pi e} \left(1 + \sqrt {1 + \frac {\pi e}{4}}\right)} t.
$$
For  $t\geq -\frac{1}{2} +\frac{1}{2}\sqrt{1 + \frac{\pi e}{4}}$
$$
\sqrt {\frac {8}{\pi e} \left(1 + \sqrt {1 + \frac {\pi e}{4}}\right)} t \geq \sqrt {\frac {4}{\pi e} \left(\left(1 + \frac {\pi e}{4}\right) - 1\right)} = 1.
$$
Hence, for every  $t > 0$ , we have
$$
\mathbb {P} \left\{\langle \theta , X _ {i} \rangle^ {2} \leq t ^ {2} \left\| \Sigma^ {1 / 2} \theta \right\| ^ {2} \right\} \leq \sqrt {\frac {8}{\pi e} \left(1 + \sqrt {1 + \frac {\pi e}{4}}\right)} t \leq 7 ^ {1 / 4} t.
$$
It confirms  $X_{i}$  satisfies the small-ball condition with  $C = 7^{1/4}$  and  $\alpha = 1$ . Application of Theorem 13 yields the desired claim.
Proof of Lemma 5. For a positive random variable  $X$ , we can express the expected value of  $X$  as  $\mathbf{E}[X] = \int_0^\infty \mathbb{P}\{X > t\} dt$ . Applying this to our problem, we obtain
$$
\begin{array}{l} \mathbf {E} \left[ \lambda_ {\max } \left(\left(\frac {1}{n} X ^ {\top} X\right) ^ {- 1}\right) \right] = \int_ {0} ^ {\infty} \mathbb {P} \left\{\lambda_ {\max } \left(\left(\frac {1}{n} X ^ {\top} X\right) ^ {- 1}\right) > t \right\} d t \\ = \int_ {0} ^ {\infty} \mathbb {P} \Bigg \{\lambda_ {\min } \left(\left(\frac {1}{n} X ^ {\top} X\right)\right) <   t ^ {- 1} \Bigg \} d t. \\ \end{array}
$$
Now, let us set  $C = \frac{21e^{10}}{\sigma_X^2}$ . Using the previous result, we can rewrite the expectation of interest as
$$
\begin{array}{l} \mathbf {E} \left[ \lambda_ {\max } \left(\left(\frac {1}{n} X ^ {\top} X\right) ^ {- 1}\right) \right] = C + \int_ {C} ^ {\infty} \mathbb {P} \left\{\lambda_ {\min } \left(\left(\frac {1}{n} X ^ {\top} X\right)\right) <   t ^ {- 1} \right\} d t \\ \leq C + \int_ {C} ^ {\infty} (C t ^ {- 1}) ^ {n / 6} d t \\ = C + C ^ {n / 6} \left(1 - \frac {n}{6}\right) ^ {- 1} \left(- C ^ {1 - n / 6}\right) \\ = C \left(1 + \frac {6}{n - 6}\right), \\ \end{array}
$$
which yields the claim.
Proofs of theorems. By utilizing the results of Lemmas 2 to 5, we provide the complete proofs for both Theorem 4 and Theorem 5.
Proof of Theorem 4. We begin by demonstrating that obtaining an upper bound on the expected error of the direction estimator can be reduced to finding a lower bound on the inner product  $A_{n} = \langle \frac{\hat{\beta}}{\|\hat{\beta}\|},\frac{\beta^{*}}{\|\beta^{*}\|}\rangle$ . Specifically, a straightforward calculation yields
$$
\mathbf {E} \left[ \left\| \frac {\hat {\beta}}{\| \hat {\beta} \|} - \frac {\beta^ {*}}{\| \beta^ {*} \|} \right\| ^ {2} \right] = 2 \left(1 - \mathbf {E} \left[ \left\langle \frac {\hat {\beta}}{\| \hat {\beta} \|}, \frac {\beta^ {*}}{\| \beta^ {*} \|} \right\rangle \right]\right). \tag {24}
$$
Therefore, it suffices to establish a lower bound on  $\mathbf{E}[A_n]$ .
Taking advantage of Lemma 3, we derive a lower bound on  $A_{n}$ . Let  $r = (\hat{\beta} -\beta^{*})^{\top}(\frac{n}{\sigma_{\xi}^{2}}\Sigma_{n})(\hat{\beta} -\beta^{*})$ . From Lemma 3, it follows that
$$
A _ {n} \geq \sqrt {1 - \frac {\sigma_ {\xi} ^ {2} r}{\| \beta^ {*} \| ^ {2} n} \lambda_ {\max} \left(\Sigma_ {n} ^ {- 1}\right)},
$$
provided that  $r \leq n\|\beta^{*}\|^{2} / \sigma_{\xi}^{2}\lambda_{\min}(\Sigma_{n}^{-1})$ . Since  $1 - \sqrt{1 - x} \leq x$  for  $x \in [0,1]$ , it follows that
$$
1 - A _ {n} \leq \frac {\sigma_ {\xi} ^ {2} r}{\| \beta^ {*} \| ^ {2} n} \lambda_ {\max} \big (\Sigma_ {n} ^ {- 1} \big),
$$
as long as  $r \leq n\|\beta^*\|^2/\sigma_\xi^2\lambda_{\max}(\Sigma_n^{-1})$ .
Next, we derive an upper bound on the expectation of  $1 - A_{n}$ . Noting that conditioned on  $X$ ,  $r$  follows the  $\chi^2$  distribution with degree of freedom  $d$ , we have
$$
\begin{array}{l} \mathbf {E} \left[ 1 - A _ {n} | X \right] \\ = \mathbf {E} \left[ \left(\mathbb {1} \left\{r \leq \frac {n \| \beta^ {*} \| ^ {2}}{\sigma_ {\xi} ^ {2} \lambda_ {\max } (\Sigma_ {n} ^ {- 1})} \right\} + \mathbb {1} \left\{r > \frac {n \| \beta^ {*} \| ^ {2}}{\sigma_ {\xi} ^ {2} \lambda_ {\max } (\Sigma_ {n} ^ {- 1})} \right\}\right) (1 - A _ {n}) \Bigg | X \right] \\ \leq \mathbf {E} \left[ \frac {\sigma_ {\xi} ^ {2} r}{\| \beta^ {*} \| ^ {2} n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right) \Bigg | X \right] + \mathbb {P} \left\{r > \frac {n \| \beta^ {*} \| ^ {2}}{\sigma_ {\xi} ^ {2} \lambda_ {\max } (\Sigma_ {n} ^ {- 1})} \Bigg | X \right\} \\ \leq 2 \mathbf {E} \left[ \frac {\sigma_ {\xi} ^ {2} r}{\| \beta^ {*} \| ^ {2} n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right) \mid X \right] (25) \\ = \frac {2 \sigma_ {\xi} ^ {2} d}{\left\| \beta^ {*} \right\| ^ {2} n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right), (26) \\ \end{array}
$$
where we use the Markov inequality to obtain Eq (25).
By utilizing Eq (24), an upper bound on the expected error can be obtained by deriving an upper bound on the expectation of Eq (26). The random variable in Eq (26) is  $\lambda_{\max}(\Sigma_n^{-1})$ , which allows us to derive the upper bound on the expected error by obtaining an upper bound on the expectation of  $\lambda_{\max}(\Sigma_n^{-1})$ . To accomplish this, we apply Lemma 5. The upper bound from Lemma 5 can be substituted into Eq (26), resulting in the claimed upper bound.
Proof of Theorem 5. We first utilize Lemma 2 to get an upper bound on the squared error of the norm estimator. Let  $r = (\hat{\beta} - \beta^{*})^{\top}(\frac{n}{\sigma_{\xi}^{2}} \Sigma_{n})(\hat{\beta} - \beta^{*})$ . From Lemma 2, we have
$$
\left\| \hat {\beta} - \beta^ {*} \right\| ^ {2} \leq \frac {\sigma_ {\xi} ^ {2} r}{n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right).
$$
Application of the triangle and reverse triangle inequality yields
$$
\| \beta^ {*} \| - \sqrt {\frac {\sigma_ {\xi} ^ {2} r}{n} \lambda_ {\max} \left(\Sigma_ {n} ^ {- 1}\right)} \leq \left\| \hat {\beta} \right\| \leq \| \beta^ {*} \| + \sqrt {\frac {\sigma_ {\xi} ^ {2} r}{n} \lambda_ {\max} \left(\Sigma_ {n} ^ {- 1}\right)},
$$
equivalently
$$
\left(\left\| \hat {\beta} \right\| - \| \beta^ {*} \|\right) ^ {2} \leq \frac {\sigma_ {\xi} ^ {2} r}{n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right).
$$
Taking expectation conditioned on  $X$  yields
$$
\begin{array}{l} \mathbf {E} \left[ \left(\| \hat {\beta} \| - \| \beta^ {*} \|\right) ^ {2} \mid X \right] \\ = \mathbf {E} \left[ \frac {\sigma_ {\xi} ^ {2} r}{n} \lambda_ {\operatorname * {m a x}} \left(\Sigma_ {n} ^ {- 1}\right) \Bigg | X \right] \\ \leq \frac {\sigma_ {\xi} ^ {2} d}{n} \lambda_ {\max } \left(\Sigma_ {n} ^ {- 1}\right), \tag {27} \\ \end{array}
$$
where we use the fact that  $r$  follows the  $\chi^2$  distribution with degree of freedom  $d$  to obtain the last line. Again, application of Lemma 5 into expectation of Eq (27) yields the claim.
F Details of Upper Bound Analyses
This section presents a detailed proof of the upper bound stated in Theorem 1, which is achieved through an analysis of the estimator constructed in Section 5. Specifically, we establish the following theorem:
Theorem 14. Let  $\hat{\beta}_n$  be the estimator constructed in Section 5. Then, there exists a universal constant  $C > 0$  such that for any  $\delta \in (0,1)$  and  $n\geq 12(3d\vee 4\ln (M / \delta)) / \min_{s\in [M]}p_s$
$$
\mathcal {E} _ {n} \bigg (\frac {1}{2}, \delta \bigg) \leq C \frac {\sigma_ {\xi} ^ {2} B ^ {2} d M \vee \sigma_ {X} ^ {2} B ^ {2} M \vee B ^ {2} U ^ {2}}{n} + o \bigg (\frac {1}{n} \bigg).
$$
To establish the validity of Theorem 14, we begin by proving Theorem 3, which demonstrates that the estimation error can be decomposed into the sum of errors associated with individual components. Subsequently, we derive upper bounds for the estimation errors of each component. Finally, we synthesize these results to provide a proof of Theorem 14.
F.1 Proof of Theorem 3
We commence the error analysis of our estimator by decomposing the estimation error, as presented in Theorem 3. Recall the statement of Theorem 3
Theorem 15. For the estimator defined in Eq (6), the mean square deviation from  $f_{\mathrm{DP}}^{*}$  is bounded above by
$$
\begin{array}{l} \sum_ {s \in [ M ]} p _ {s} \mathbf {E} \left[ \right.\left( \right.\mathbf {E} \left[ \right. \widehat {\| \beta . \|} ^ {2} \left. \right\rvert n. \left. \right] ^ {1 / 2} \mathbf {E} \left[ \right.\left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \left. \right\rvert n. \left. \right] ^ {1 / 2} + \sigma_ {X} \mathbf {E} \left[ \right.\left(\widehat {\| \beta . \|} - \overline {{\| \beta_ {:} ^ {*} \|}}\right) ^ {2} \left. \right\rvert n. \left. \right] ^ {1 / 2} + \\ \sigma_ {X} \overline {{\| \beta_ {s} ^ {*} \|}} \mathbf {E} \left[ \left\| \tilde {\beta} _ {s} - \beta_ {s} ^ {*} / \| \beta_ {s} ^ {*} \| \right\| ^ {2} \mid n. \right] ^ {1 / 2} + \mathbf {E} \left[ \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \left\langle \hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} \right\rangle\right) ^ {2} \mid n. \right] ^ {1 / 2} + \\ \mathbf {E} \left[ \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \langle \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} - \mu_ {s ^ {\prime}} \rangle\right) ^ {2} \mid n. \right] ^ {1 / 2} + \left| \sum_ {s ^ {\prime} \in [ M ]} (\hat {p} _ {s ^ {\prime}} - p _ {s ^ {\prime}}) \langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \rangle \right|) ^ {2} ]. \\ \end{array}
$$
We provide a proof of Theorem 15 as follows:
Proof of Theorem 15. We begin by decomposing  $\hat{f}_n(X,S) - f_{\mathrm{DP}}^* (X,S)$  into six terms. Recall the definitions of  $\hat{f}_n(x,s)$  and  $f_{\mathrm{DP}}^{*}(x,s)$ :
$$
\hat {f} _ {n} (x, s) = \widehat {\| \beta . \|} \Bigl \langle \tilde {\beta} _ {s}, x - \hat {\mu} _ {s} \Bigr \rangle + \sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \Bigl \langle \hat {\beta} _ {s} ^ {\prime}, \hat {\mu} _ {s} ^ {\prime} \Bigr \rangle
$$
$$
f _ {\mathrm {D P}} ^ {*} (x, s) = \overline {{\| \beta_ {\cdot} ^ {*} \|}} \Bigg \langle \frac {\beta_ {s} ^ {*}}{\| \beta_ {s} ^ {*} \|}, x - \mu_ {s} \Bigg \rangle + \sum p _ {s ^ {\prime}} \langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \rangle .
$$
Through elementary calculations, we obtain:
$$
\begin{array}{l} \hat {f} _ {n} (X, S) - f _ {\mathrm {D P}} ^ {*} (X, S) = \widehat {\| \beta \cdot \|} \left\langle \tilde {\beta} _ {S}, \mu_ {S} - \hat {\mu} _ {S} \right\rangle + \left(\widehat {\| \beta . \|} - \| \beta_ {.} ^ {*} \|\right) \left\langle \tilde {\beta} _ {S}, X - \mu_ {S} \right\rangle \\ + \| \beta_ {\cdot} ^ {*} \| \Big \langle \tilde {\beta} _ {S} - \frac {\beta_ {S} ^ {*}}{\| \beta_ {S} ^ {*} \|}, X - \mu_ {S} \Big \rangle + \sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \Big \langle \hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} \Big \rangle \\ + \sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \left\langle \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} - \mu_ {s ^ {\prime}} \right\rangle + \sum_ {s ^ {\prime} \in [ M ]} \left(\hat {p} _ {s ^ {\prime}} - p _ {s ^ {\prime}}\right) \left\langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \right\rangle . \tag {28} \\ \end{array}
$$
By the Cauchy-Schwarz inequality, for two random variables  $Z_{1}$  and  $Z_{2}$ , we have  $\mathbf{E}[(Z_1 + Z_2)^2 ]^{1 / 2}\leq \mathbf{E}[Z_1^2 ]^{1 / 2} + \mathbf{E}[Z_2^2 ]^{1 / 2}$ . By applying this fact into the expectation of Eq (28) conditioned on  $S$  and  $n$ . multiple times, we have
$$
\begin{array}{l} \mathbf {E} \Big [ \Big (\hat {f} _ {n} (X, S) - f _ {\mathrm {D P}} ^ {*} (X, S) \Big) ^ {2} \Big ] \\ = \sum_ {s \in [ M ]} p _ {s} \mathbf {E} \left[ \mathbf {E} \left[ \left(\hat {f} _ {n} (X, S) - f _ {\mathrm {D P}} ^ {*} (X, S)\right) ^ {2} \mid S = s, n. \right] \mid S = s \right] \\ \leq \sum_ {s \in [ M ]} p _ {s} \mathbf {E} \left[ \left(\mathbf {E} \left[ \left(\widehat {\| \beta \cdot \|} \left\langle \tilde {\beta} _ {S}, \mu_ {S} - \hat {\mu} _ {S} \right\rangle\right) ^ {2} \right\rvert   S = s, n. \right] \right. ^ {1 / 2} \tag {29} \\ + \mathbf {E} \left[ \left(\left(\widehat {\| \beta_ {.} \|} - \| \beta_ {.} ^ {*} \|\right) \left\langle \tilde {\beta} _ {S}, X - \mu_ {S} \right\rangle\right) ^ {2} \mid S = s, n. \right] ^ {1 / 2} \\ + \mathbf {E} \left[ \left(\| \beta_ {\cdot} ^ {*} \| \left\langle \tilde {\beta} _ {S} - \frac {\beta_ {S} ^ {*}}{\| \beta_ {S} ^ {*} \|}, X - \mu_ {S} \right\rangle\right) ^ {2} \mid S = s, n. \right] ^ {1 / 2} \\ + \mathbf {E} \left[ \left. \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \left\langle \hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} \right\rangle\right) ^ {2} \right| S = s, n. \right] ^ {1 / 2} \\ + \mathbf {E} \left[ \left. \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \langle \beta_ {s ^ {\prime}} ^ {*}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} - \mu_ {s ^ {\prime}} \rangle\right) ^ {2} \right| S = s, n. \right] ^ {1 / 2} \\ + \mathbf {E} \left[ \left. \left(\sum_ {s ^ {\prime} \in [ M ]} \left(\hat {p} _ {s ^ {\prime}} - p _ {s ^ {\prime}}\right) \langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \rangle\right) ^ {2} \right| S = s, n. \right] ^ {1 / 2}) ^ {2} \Bigg | S = s \Bigg ]. \\ \end{array}
$$
In the subsequent analyses, we derive upper bounds for each term in Eq (29).
(First term in Eq (29)) Due to the splitting of the sample,  $\| \beta_{\cdot}\|$ ,  $\tilde{\beta}_s$ , and  $\hat{\mu}_s$  are independent conditioned on  $n$ . Thus, we have:
$$
\begin{array}{l} \mathbf {E} \left[ \left(\widehat {\| \beta \cdot \|} \left\langle \tilde {\beta} _ {S}, \mu_ {S} - \hat {\mu} _ {S} \right\rangle\right) ^ {2} \mid S = s, n. \right] ^ {1 / 2} \\ = \mathbf {E} \left[ \left(\widehat {\| \beta . \|} \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle\right) ^ {2} \mid n. \right] ^ {1 / 2} \\ = \mathbf {E} \left[ \left(\widehat {\| \beta . \|} ^ {2}\right) | n. \right] ^ {1 / 2} \mathbf {E} \left[ \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} | n. \right] ^ {1 / 2}. \\ \end{array}
$$
This term matches the first term of the desired bound.
(Second term in Eq (29)) Since  $\widehat{\|\beta\|}, \tilde{\beta}_s$ , and  $X$  are independent conditioned on  $n$ , we have
$$
\begin{array}{l} \mathbf {E} \left[ \left(\left(\widehat {\| \beta . \|} - \overline {{\| \beta_ {*} ^ {*} \|}}\right) \left\langle \tilde {\beta} _ {s}, X - \mu_ {s} \right\rangle\right) ^ {2} \mid S = s, n. \right] \\ = \sigma_ {X} ^ {2} \mathbf {E} \left[ \left(\widehat {\| \beta . \|} - \overline {{\| \beta . ^ {*} \|}}\right) ^ {2} \Bigg | n. \right], \\ \end{array}
$$
where we use the fact that  $X - \mu_s \sim N(0, \sigma_X^2 I)$  conditioned on  $S = s$ , and  $\tilde{\beta}_s \in S_{d-1}$  almost surely. This result corresponds to the second term of the desired bound.
(Third term in Eq (29)) Since  $X - \mu_{s}\sim N(0,\sigma_{X}^{2}I)$  , we have
$$
\mathbf {E} \left[ \left(\overline {{\| \beta_ {s} ^ {*} \|}} \left\langle \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\| \beta_ {s} ^ {*} \|}, X - \mu_ {s} \right\rangle\right) ^ {2} \mid S = s, n. \right]
$$
$$
= \sigma_ {X} ^ {2} \overline {{\left\| \beta_ {z} ^ {*} \right\|}} ^ {2} \mathbf {E} \left[ \left\| \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\left\| \beta_ {s} ^ {*} \right\|} \right\| ^ {2} \middle | n. \right],
$$
which corresponds to the third term of the desired bound.
(Forth and fifth terms in Eq (29)) These terms are independent of  $S$ , so we can omit  $S = s$  from the condition, resulting in the fourth and fifth terms of the desired bound.
(Sixth term in Eq (29)) This term does not contain any random variable when  $n$  is fixed. Thus, we can remove the expectation, yielding the sixth term of the desired bound.
F.2 Estimation Error Analyses for Each Component
This subsection presents an analysis of the estimation errors associated with each component estimator. In particular, we investigate the estimation errors of  $\hat{\mu}_s$ ,  $\widehat{\|\beta\|}$ ,  $\tilde{\beta}_s$ ,  $\hat{\beta}_s'$ , and  $\hat{\mu}_s'$ .
F.2.1 Estimation Error Analysis for  $\hat{\mu}_s$
Here, we presents the proof of the following theorem.
Theorem 16. Given  $s \in [M]$ , if  $n_s > 18d$ , we have
$$
\sup  _ {v \in \mathbb {S} _ {d - 1}} \mathbf {E} \left[ \langle v, \mu_ {s} - \hat {\mu} _ {s} \rangle^ {2} \mid n. \right] \leq \frac {6 \sigma_ {X} ^ {2}}{n _ {s}}.
$$
Proof of Theorem 16. Given  $v \in \mathbb{S}_{d-1}$ , we have
$$
\mathbf {E} \left[ \langle v, \mu_ {s} - \hat {\mu} _ {s} \rangle^ {2} \mid n. \right] = \left\langle v, \mathbf {E} \left[ (\mu_ {s} - \hat {\mu} _ {s}) (\mu_ {s} - \hat {\mu} _ {s}) ^ {\top} \mid n. \right] v \right\rangle .
$$
According to the definition,  $\hat{\mu}_s$  is an average of  $n_{3,s}$  i.i.d. random variables following  $N(\mu_s,\sigma_X^2 I)$ . Hence, we have  $\mu_s - \hat{\mu}_s\sim N(0,\frac{\sigma_X^2}{n_{3,s}} I)$ , which implies  $\mathbf{E}[(\mu_s - \hat{\mu}_s)(\mu_s - \hat{\mu}_s)^\top |n.] = \frac{\sigma_X^2}{n_{3,s}} I$ . Consequently, we obtain:
$$
\begin{array}{l} \mathbf {E} \left[ \langle v, \mu_ {s} - \hat {\mu} _ {s} \rangle^ {2} \mid n. \right] = \left\langle v, \frac {\sigma_ {X} ^ {2}}{n _ {3 , s}} I v \right\rangle \\ = \frac {\sigma_ {X} ^ {2}}{n _ {3 , s}} \langle v, v \rangle = \frac {\sigma_ {X} ^ {2}}{n _ {3 , s}}. \\ \end{array}
$$
Since  $n_{3,s} \geq \lfloor n_s / 3 \rfloor \geq n_s / 6$  for  $n_s \geq 6$ , the claim follows.
F.2.2 Estimation Error Analysis for  $\|\beta\|$
Here, we present the proof of the following theorem.
Theorem 17. For any  $s \in [M]$ , we have
$$
\mathbf {E} \left[ \left(\widehat {\| \beta_ {\cdot} \|} - \overline {{\| \beta_ {\cdot} ^ {*} \|}}\right) ^ {2} \Big | n. \right] ^ {1 / 2} \leq \sqrt {\frac {1 8 9 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \mathbb {1} \{n _ {s} \leq 1 8 d \} \frac {n _ {s}}{n}} + \left| \sum_ {s \in [ M ]} (\hat {p} _ {s} - p _ {s}) \| \beta_ {s} ^ {*} \| \right|.
$$
Proof of Theorem 17. By combining the definitions of  $\widehat{\|\beta\|}$  and  $\overline{\|\beta_*^*}$  and utilizing the Cauchy-Schwarz inequality, we obtain
$$
\begin{array}{l} \mathbf {E} \left[ \left(\widehat {\| \beta_ {.} \|} - \overline {{\| \beta_ {.} ^ {*} \|}}\right) ^ {2} \mid n. \right] ^ {1 / 2} \\ = \mathbf {E} \left[ \left(\sum_ {s \in [ M ]} \widehat {p _ {s}} \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right) + \sum_ {s \in [ M ]} (\widehat {p _ {s}} - p _ {s}) \overline {{\| \beta_ {s} ^ {*} \|}}\right) ^ {2} \Bigg | n. \right] ^ {1 / 2} \\ \end{array}
$$
$$
\leq \mathbf {E} \left[ \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right)\right) ^ {2} \mid n. \right] ^ {1 / 2} + \left| \sum_ {s \in [ M ]} \left(\hat {p} _ {s} - p _ {s}\right) \overline {{\| \beta_ {s} ^ {*} \|}} \right|. \tag {30}
$$
Next, we derive an upper bound for the first term in Eq (30). Applying Jensen's inequality, we have:
$$
\begin{array}{l} \mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right)\right) ^ {2} \right\rvert   n. \right] \\ \leq \sum_ {s \in [ M ]} \frac {n _ {s}}{n} \mathbf {E} \left[ \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right) ^ {2} \Bigg | n. \right] \\ = \sum_ {s \in [ M ]} \frac {n _ {s}}{n} \bigg (\mathbb {1} \{n _ {s} > 1 8 d \} \mathbf {E} \bigg [ \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right) ^ {2} \Big | n. \bigg ] + \mathbb {1} \{n _ {s} \leq 1 8 d \} \| \beta_ {s} ^ {*} \| ^ {2} \bigg) \\ \end{array}
$$
Using the fact that  $n_{1,s} > 6d$  for  $n_s > 18d$  and employing Theorem 5, we obtain:
$$
\begin{array}{l} \mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left(\widehat {\| \beta_ {s} \|} - \overline {{\| \beta_ {s} ^ {*} \|}}\right)\right) ^ {2} \right\rvert   n. \right] \\ \leq \sum_ {s \in [ M ]} \frac {n _ {s}}{n} \frac {2 1 e ^ {1 0} \sigma_ {\xi} ^ {2} d}{n _ {1 , s}} \left(1 + \frac {6}{n _ {1 , s} - 6}\right) + \sum_ {s \in [ M ]} \mathbb {1} \left\{n _ {s} \leq 1 8 d \right\} \frac {\| \beta_ {s} ^ {*} \| ^ {2} n _ {s}}{n} \\ \leq \frac {1 8 9 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \mathbb {1} \{n _ {s} \leq 1 8 d \} \frac {B ^ {2} n _ {s}}{n}, \\ \end{array}
$$
where the last line follows from the fact that  $n_{1,s} \geq \lfloor n_s / 3 \rfloor \geq n_s / 6$  for  $n_s \geq 6$ ,  $\frac{6}{n_{1,s} - 6} \leq 1/2$  for  $n_s \geq 18$ , and  $\| \beta_s^* \| \leq B$ .
F.2.3 Estimation Error Analysis for  $\tilde{\beta}_s$
Here, we will prove the following theorem.
Theorem 18. For any  $s \in [M]$ , we have
$$
\mathbf{E}\left[\left|\left|\tilde{\beta}_{s}-\frac{\beta_{s}^{*}}{\|\beta_{s}^{*}\|}\right|\right|^{2}\Bigg|n.\right]\leq\left\{\begin{array}{ll}\hline 756e^{10}\sigma_{\xi}^{2}d&\mbox{if }n_{s}>18d,\\ \hline 1&\mbox{otherwise.}\end{array}\right.
$$