SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem 18. If  $n_s \leq 18d$ ,  $\tilde{\beta}_s = 0$ , and we thus have  $\mathbf{E}[\| \tilde{\beta}_s - \frac{\beta_s^*}{\|\beta_s^*\|} \|^2 | n.] = \| \frac{\beta_s^*}{\|\beta_s^*\|} \|^2 = 1$ . For  $n_s > 18d$ , we have  $n_{2,s} > 6d$ . Application of Theorem 4 yields
$$
\mathbf {E} \left[ \left\| \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\| \beta_ {s} ^ {*} \|} \right\| ^ {2} \right\rvert   n. \Bigg ] \leq \frac {8 4 e ^ {1 0} \sigma_ {\xi} ^ {2} d}{\sigma_ {X} ^ {2} \| \beta_ {s} ^ {*} \| ^ {2} n _ {2 , s}} \left(1 + \frac {6}{n _ {2 , s} - 6}\right)
$$
We get the claim in the same manner as the proof of Theorem 17.
F.2.4 Estimation Error Analysis for  $\hat{\beta}_{s'}'$
Theorem 19. Given  $s \in [M]$ , let  $\Sigma_s = \mathbf{E}[X_s X_s^\top]$  for  $X_s \sim N(\mu_s, \sigma_X^2 I)$ . Then, if  $n_s > 12d$ , we have
$$
\mathbf {E} \left[ \left\| \Sigma_ {s} ^ {1 / 2} \left(\hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}\right) \right\| ^ {2} \mid n. \right] \leq \frac {4 \sigma_ {\xi} ^ {2} d}{n _ {s}} + 5 0 4 e ^ {1 0} \sigma_ {\xi} ^ {2} \left(\frac {4 d}{n _ {s}}\right) ^ {2}.
$$
To prove Theorem 19, we utilize the following theorem presented by Mourtada [15].
Theorem 20 (Theorem 3 in [15]). Let  $X$  be a random vector in  $\mathbb{R}^d$  such that it satisfies the small-ball condition of Eq (19) and  $\mathbf{E}[\| \Sigma^{-1/2}X\|^4] \leq \kappa d$  for some  $\kappa > 0$ , where  $\Sigma = \mathbf{E}[XX^\top]$ . Let  $\hat{\Sigma}_n = \frac{1}{n}\sum_{i=1}^{n}X_iX_i^\top$ , where  $X_i$  are i.i.d. copies of  $X$ . If  $n \geq 6\alpha^{-1}d \wedge 12\alpha^{-1}\ln(12\alpha^{-1})$ ,
$$
\frac {1}{n} \mathbf {E} \left[ \mathrm {T r} \left(\hat {\boldsymbol {\Sigma}} _ {n} ^ {- 1} \boldsymbol {\Sigma}\right) \right] \leq \frac {d}{n} + 8 C ^ {\prime} \kappa \left(\frac {d}{n}\right) ^ {2},
$$
where  $\alpha$  and  $C^\prime$  are as in Theorem 13.
Proof of Theorem 19. We can easily confirm that  $\hat{\beta}_{s}^{\prime}\sim N(\beta_{s}^{*},\frac{\sigma_{\xi}^{2}}{n_{1,s}^{\prime}} (\Sigma_{1,s}^{\prime})^{-1})$  conditioned on  $n$ . and  $X_{1,s}^{\prime}$ , where  $\Sigma_{1,s}^{\prime} = \frac{1}{n_{1,s}^{\prime}} X_{1,s}^{\prime}(X_{1,s}^{\prime})^{\top}$ . Noting that  $\mathbf{E}[(\hat{\beta}_s^\prime -\beta_s^*)(\hat{\beta}_s^\prime -\beta_s^*)^\top |X,n.] = \frac{\sigma_\xi^2}{n_{1,s}^\prime} (\Sigma_{1,s}^\prime)^{-1}$  we have
$$
\begin{array}{l} \mathbf {E} \left[ \left. \left\| \Sigma_ {m, s} ^ {1 / 2} \left(\hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s} ^ {*}\right) \right\| ^ {2} \right\rvert   n. \right] \\ = \operatorname {T r} \left(\Sigma_ {s} \mathbf {E} \left[ \left(\hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s} ^ {*}\right) \left(\hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s} ^ {*}\right) ^ {\top} \mid n. \right]\right) \\ = \frac {\sigma_ {\xi} ^ {2}}{n _ {1 , s} ^ {\prime}} \mathbf {E} \left[ \operatorname {T r} \left(\Sigma_ {s} \left(\Sigma_ {1, s} ^ {\prime}\right) ^ {- 1}\right) \mid n. \right]. \tag {31} \\ \end{array}
$$
We apply Theorem 20 to the expected trace term in Eq (31). To do so, we need to check  $X_{1,s}^{\prime}$  satisfies the small-ball condition of Eq (19) and the kurtosis condition  $\mathbf{E}[\| \Sigma_s^{-1 / 2}X_{1,s}^{\prime}\| ^4 ]\leq \kappa d$
The small-ball condition is confirmed by the same manner in the proof of Lemma 5, with  $C = 7^{1/4}$  and  $\alpha = 1$ . Here, we prove the satisfaction of the kurtosis condition. For a multivariate Gaussian random variable  $X \sim N(\mu, \Lambda)$  such that  $\lambda_{\min}(\Lambda) > 0$ ,  $\Sigma \coloneqq \mathbf{E}[XX^{\top}] = \mu \mu^{\top} + \Lambda$ , and  $\| \Sigma^{-1/2}X\|^4 = \langle X, \Sigma^{-1}X \rangle$
$$
\begin{array}{l} \mathbf {E} \left[ \left\| \Sigma^ {- 1 / 2} X \right\| ^ {4} \right] = \mathbf {E} \left[ \langle X, \Sigma^ {- 1} X \rangle^ {2} \right] \\ = \operatorname {V a r} \left[ \langle X, \Sigma^ {- 1} X \rangle \right] + \left(\mathbf {E} \left[ \langle X, \Sigma^ {- 1} X \rangle \right]\right) ^ {2}. \\ \end{array}
$$
$$
\mathbf {E} \left[ \langle X, \Sigma^ {- 1} X \rangle \right] = \operatorname {T r} \left(\Sigma^ {- 1} \left(\mu \mu^ {\top} + \Lambda\right)\right) = d
$$
$$
\begin{array}{l} \mathbf {V a r} \left[ \langle X, \Sigma^ {- 1} X \rangle \right] = 2 \operatorname {T r} \left(\Sigma^ {- 1} \Lambda \Sigma^ {- 1} \left(\mu \mu^ {\top} + \Lambda\right)\right) + 2 \operatorname {T r} \left(\Sigma^ {- 1} \Lambda \Sigma^ {- 1} \mu \mu^ {\top}\right) \\ = 2 \operatorname {T r} \left(\Sigma^ {- 1} \Lambda\right) + 2 \operatorname {T r} \left(\Sigma^ {- 1} \Lambda \Sigma^ {- 1} \mu \mu^ {\top}\right) \\ = 2 \operatorname {T r} \left(\Sigma^ {- 1} \left(\mu \mu^ {\top} + \Lambda\right)\right) - 2 \operatorname {T r} \left(\mu \mu^ {\top} \Sigma^ {- 1} \mu \mu^ {\top}\right) \\ = 2 d - 2 \| \mu \| ^ {2} \left\langle \mu , \Sigma^ {- 1} \mu \right\rangle \leq 2 d. \\ \end{array}
$$
Hence, the kurtosis condition satisfies with  $\kappa = 3$
Application of Theorem 20 into Eq (31) yields
$$
\mathbf {E} \left[ \left| \left| \Sigma_ {m, s} ^ {1 / 2} \left(\hat {\beta} _ {s ^ {\prime}} ^ {\prime} - \beta_ {s} ^ {*}\right) \right| \right| ^ {2} \right\rvert   n. \Bigg ] \leq \frac {\sigma_ {\xi} ^ {2} d}{n _ {1 , s} ^ {\prime}} + 5 0 4 e ^ {1 0} \sigma_ {\xi} ^ {2} \left(\frac {d}{n _ {1 , s} ^ {\prime}}\right) ^ {2},
$$
provided that  $n_s \geq 12d$ . We get the claim from the fact that  $n_{1,s}' \geq \lfloor n_s / 2 \rfloor \geq n_s / 4$  for  $n_s \geq 4$ .
F.2.5 Estimation Error Analysis for  $\hat{\mu}_s^{\prime}$
Theorem 21. Given  $s \in [M]$  and  $v \in \mathbb{R}^d$ , if  $n_s > 12d$ , we have
$$
\mathbf {E} \Big [ \langle v, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \rangle^ {2} \Big | n. \Big ] \leq \frac {4 \sigma_ {X} ^ {2} \| v \| ^ {2}}{n _ {s}}.
$$
Proof of Theorem 21. By definition, we have  $\hat{\mu}_s^\prime \sim N(\mu_s,\frac{\sigma_X^2}{n_{2,s}^\prime} I)$  conditioned on  $n$ . for  $n_s > 12d$ . Hence, we have
$$
\mathbf {E} \left[ \langle v, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \rangle^ {2} \mid n. \right] \leq \frac {\sigma_ {X} ^ {2} \| v \| ^ {2}}{n _ {2 , s} ^ {\prime}}.
$$
We get the claim following the same manner of the proof of Theorem 19.
F.3 Some Auxiliary Lemmas
This subsections introduce some auxiliary lemmas for use to prove Theorem 14. Specifically, we demonstrate the following lemmas:
Lemma 6. Let  $a_1, \ldots, a_M \in \mathbb{R}$  be arbitrary numbers. Then, we have
$$
\mathbf {E} \left[ \left(\sum_ {s \in [ M ]} a _ {s} \left(\hat {p} _ {s} - p _ {s}\right) ^ {2}\right) \right] = \frac {1}{n} \mathbf {V a r} \left[ a _ {S} \right].
$$
Lemma 7. For a constant  $c > 0$ , we have for any  $s \in [M]$
$$
\mathbf {E} \left[ n _ {s} ^ {- 1} \mathbb {1} \left\{n _ {s} > c \right\} \right] \leq \frac {1 + c ^ {- 1}}{p _ {s} (n + 1)}.
$$
Lemma 8. Let  $c > 0$  be a constant. If  $n > 2c / \min_{s\in [M]}p_s$ , we have for any  $s\in [M]$
$$
\mathbb {P} \{n _ {s} \leq c \} \leq e ^ {- n p _ {s} / 8}.
$$
Proof of Lemma 6. Since  $n\hat{p}$  follows the multinomial distribution with the parameters  $n$  and  $p$ , using the variance and covariance of the multinomial distribution, we have
$$
\begin{array}{l} \mathbf {E} \left[ \left(\sum_ {s \in [ M ]} a _ {s} \left(\hat {p} _ {s} - p _ {s}\right)\right) ^ {2} \right] \\ = \sum_ {s \in [ M ]} \frac {a _ {s} ^ {2} p _ {s} (1 - p _ {s})}{n} - \sum_ {s, s ^ {\prime} \in [ M ]: s \neq s ^ {\prime}} \frac {a _ {s} a _ {s ^ {\prime}} p _ {s} p _ {s ^ {\prime}}}{n} \\ = \sum_ {s \in [ M ]} \frac {a _ {s} p _ {s}}{n} \left(a _ {s} - \sum_ {s ^ {\prime} \in [ M ]} a _ {s ^ {\prime}} p _ {s ^ {\prime}}\right). \\ \end{array}
$$
Let  $\bar{a} = \sum_{s\in [M]}p_s a_s$  . Then, we have
$$
\begin{array}{l} \sum_ {s \in [ M ]} a _ {s} p _ {s} (a _ {s} - \bar {a}) \\ = \sum_ {s \in [ M ]} p _ {s} \left(a _ {s} - \bar {a}\right) ^ {2} + \sum_ {s \in [ M ]} \bar {a} p _ {s} \left(a _ {s} - \bar {a}\right) \\ = \sum_ {s \in [ M ]} p _ {s} \left(a _ {s} - \bar {a}\right) ^ {2} = \mathbf {V a r} \left[ a _ {S} \right]. \\ \end{array}
$$
$$
\mathbf {E} \left[ \left(\sum_ {s \in [ M ]} a _ {s} \left(\hat {p} _ {s} - p _ {s}\right)\right) ^ {2} \right] = \frac {1}{n} \mathbf {V a r} \left[ a _ {S} \right].
$$
Proof of Lemma 7. For a random variable  $X$  following the binomial distribution with the parameters  $n$  and  $p$ ,  $\mathbf{E}[\frac{1}{X + 1}] = \frac{1}{p(n + 1)} (1 - (1 - p)^{n + 1})$  [5]. Since  $n_s$  follows the binomial distribution with the parameters  $n$  and  $p_s$ , we have We have
$$
\begin{array}{l} \mathbf {E} \left[ n _ {s} ^ {- 1} \mathbb {I} \left\{n _ {s} > c \right\} \right] \\ = \mathbf {E} \left[ \frac {1}{n _ {s} + 1} \left(1 + \frac {1}{n _ {s}}\right) \mathbb {1} \{n _ {s} > c \} \right] \\ \leq \left(1 + c ^ {- 1}\right) \mathbf {E} \left[ \frac {1}{n _ {s} + 1} \right] \\ = \frac {1 + c ^ {- 1}}{p _ {s} (n + 1)} \left(1 - \left(1 - p _ {s}\right) ^ {n + 1}\right) \leq \frac {1 + c ^ {- 1}}{p _ {s} (n + 1)}. \\ \end{array}
$$
Proof of Lemma 8. From the Chernoff bound, we have
$$
\mathbb {P} \{n _ {s} \leq c \} \leq \exp \left(- \frac {n}{2 p _ {s}} \left(\frac {c}{n} - p _ {s}\right) ^ {2}\right).
$$
Under the assumption, we have  $c \leq np_s / 2$ . Then, we have  $\frac{n}{2p_s} \left( \frac{c}{n} - p_s \right)^2 \geq n / 8$ , which gives the claim.
F.4 Proof of Theorem 14
Proof of Theorem 14. We begin by characterizing the estimation error by each component's estimation error shown in Theorems 16 to 19 and 21. Specifically, we characterize the estimation error using the following error terms:
$$
e _ {\text {m e a n}, s} ^ {2} = \sup  _ {v \in \mathbb {S} _ {d - 1}} \mathbf {E} \left[ \langle v, \mu_ {s} - \hat {\mu} _ {s} \rangle^ {2} \Big | n. \right]
$$
$$
e _ {\mathrm {n o r m}} ^ {2} = \mathbf {E} [ (\overline {{\| \boldsymbol {\beta} . ^ {*} \|}} - \widehat {\| \boldsymbol {\beta} . \|}) ^ {2} | n. ]
$$
$$
e _ {\mathrm {c o e f}, s} ^ {2} = \mathbf {E} [ \| \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\| \beta_ {s} ^ {*} \|} \| ^ {2} | n. ]
$$
$$
e _ {\operatorname {c o e f} ^ {\prime}, s} ^ {2} = \mathbf {E} \left[ \left\| \Sigma_ {s} ^ {1 / 2} \left(\hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}\right) \right\| ^ {2} \mid n. \right]
$$
$$
e _ {\text {m e a n} ^ {\prime}, s} ^ {2} = \sup  _ {v \in \mathbb {S} _ {d - 1}} \mathbf {E} \left[ \langle \hat {\mu} _ {s} ^ {\prime} - \mu_ {s}, v \rangle^ {2} | n. \right]
$$
$$
e _ {\text {p r o b}} ^ {2} = \left| \sum_ {s ^ {\prime} \in [ M ]} \left(\hat {p} _ {s ^ {\prime}} - p _ {s ^ {\prime}}\right) \left\langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \right\rangle \right| ^ {2},
$$
where  $\Sigma_{x} = \mathbf{E}[X_{s}X_{s}^{\top}]$  for  $X_{s}\sim N(\mu_{s},\sigma_{X}^{2}I)$ .
We analyze each term in Eq (7) one by one.
(First term in Theorem 3) Recall the first term in Theorem 3
$$
\mathbf {E} \left[ \widehat {\left\| \beta_ {\cdot} \right\|} ^ {2} \Bigg | n. \right] ^ {1 / 2} \mathbf {E} \left[ \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \Bigg | n. \right] ^ {1 / 2}.
$$
From the Cauchy-Schwarz inequality, we have
$$
\begin{array}{l} \mathbf {E} \left[ \widehat {\left\| \beta_ {.} \right\| ^ {2}} \mid n. \right] ^ {1 / 2} \leq \mathbf {E} \left[ \overline {{\left\| \beta_ {.} ^ {*} \right\| ^ {2}}} \mid n. \right] ^ {1 / 2} + \mathbf {E} \left[ \left(\overline {{\left\| \beta_ {.} ^ {*} \right\|}} - \widehat {\left\| \beta_ {.} \right\|}\right) ^ {2} \mid n. \right] ^ {1 / 2} \\ \leq B + e _ {\mathrm {n o r m}}. \\ \end{array}
$$
By definition,  $\tilde{\beta}_s = 0$  for  $n_s\leq 18d$ , which indicates that
$$
\mathbf {E} \left[ \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \mid n. \right] ^ {1 / 2} = 0.
$$
In the case of  $n_s > 18d$ , by utilizing the fact  $\tilde{\beta}_s$  and  $\hat{\mu}_s$  are independent conditioned on  $n$ . Due to the sample splitting and  $\tilde{\beta}_s \in \mathbb{S}_{d-1}$ , we obtain
$$
\mathbf {E} \left[ \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \middle | n. \right] \leq \sup  _ {v \in \mathbb {S} _ {d - 1}} \mathbf {E} \left[ \left\langle v, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \middle | n. \right] = e _ {\mathrm {m e a n}, s} ^ {2}.
$$
$$
\mathbf {E} \left[ \widehat {\left\| \beta . \right\|} ^ {2} \mid n. \right] ^ {1 / 2} \mathbf {E} \left[ \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle^ {2} \mid n. \right] ^ {1 / 2} \leq (B + e _ {\text {n o r m}}) e _ {\text {m e a n}, s} \mathbb {1} \{n _ {s} > 1 8 d \}. \tag {32}
$$
(Second term in Theorem 3) Recall the second term in Theorem 3
$$
\sigma_ {X} \mathbf {E} \left[ \left(\widehat {\| \beta_ {\cdot} \|} - \overline {{\| \beta_ {\cdot} ^ {*} \|}}\right) ^ {2} \Bigg | n. \right] ^ {1 / 2}.
$$
Using the notation of  $e_{\mathrm{norm}}$ , we have
$$
\sigma_ {X} \mathbf {E} \left[ \left(\widehat {\left\| \beta . \right\|} - \overline {{\left\| \beta . ^ {*} \right\|}}\right) ^ {2} \mid n. \right] ^ {1 / 2} = \sigma_ {X} e _ {\text {n o r m}}. \tag {33}
$$
(Third term in Theorem 3) Recall the third term in Theorem 3
$$
\sigma_ {X} \overline {{\| \beta_ {*} ^ {*} \|}} \mathbf {E} \left[ \right.\left\| \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\| \beta_ {s} ^ {*} \|} \right\| ^ {2} \left. \right\rvert   n. \left. \right] ^ {1 / 2}.
$$
Substituting  $e_{\mathrm{coef},s}$  yields
$$
\sigma_ {X} \overline {{\left\| \beta_ {s} ^ {*} \right\|}} \mathbf {E} \left[ \left\| \tilde {\beta} _ {s} - \frac {\beta_ {s} ^ {*}}{\left\| \beta_ {s} ^ {*} \right\|} \right\| ^ {2} \right\rvert   n. \Bigg ] ^ {1 / 2} = \sigma_ {X} \overline {{\left\| \beta_ {s} ^ {*} \right\|}} e _ {\operatorname {c o e f}, s}. \tag {34}
$$
(Fourth term in Theorem 3) Recall the fourth term in Theorem 3
$$
\mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left\langle \hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} \right\rangle\right) ^ {2} \right| n. \right] ^ {1 / 2}.
$$
Due to the sample splitting,  $\hat{\beta}_s^\prime$  and  $\hat{\mu}_s^\prime$  are mutually independent. Also, we have  $\mathbf{E}[\langle \hat{\beta}_s^\prime -\beta_s^*,\hat{\mu}_s^\prime \rangle ] = 0$  Hence,
$$
\begin{array}{l} \mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left\langle \hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} \right\rangle\right) ^ {2} \right| n. \right] \\ = \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \mathbf {E} \left[ \left\langle \hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} \right\rangle^ {2} \mid n. \right] \\ = \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \mathbf {E} \left[ \left\langle \hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} \right\rangle^ {2} \mid n. \right] \mathbb {1} \{n _ {s} > 1 2 d \}. \\ \end{array}
$$
Since  $\mathbf{E}[(\hat{\mu}_s')(\hat{\mu}_s')^\top] = \mu \mu^\top + \frac{\sigma_X^2}{n_{2,s}'} I = \Sigma_s - (1 - \frac{1}{n_{2,s}'})\sigma_X^2 I$ , we have
$$
\mathbf {E} \left[ \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left\langle \hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} \right\rangle\right) ^ {2} \middle | n. \right]
$$
$$
\begin{array}{l} = \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \mathbf {E} \left[ \right.\left\|\left(\Sigma_ {s} - \left(1 - \frac {1}{n _ {2 , s} ^ {\prime}}\right) \sigma_ {X} ^ {2} I\right) ^ {1 / 2} \left(\hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}\right)\right\| ^ {2} \left. \right\rvert   n. \left. \right] \mathbb {1} \{n _ {s} > 1 2 d \} \\ \leq \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \mathbf {E} \left[ \left\| \Sigma_ {s} ^ {1 / 2} \left(\hat {\beta} _ {s} ^ {\prime} - \beta_ {s} ^ {*}\right) \right\| ^ {2} \mid n. \right] \mathbb {1} \{n _ {s} > 1 2 d \} = \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} e _ {\operatorname {c o e f} ^ {\prime}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 2 d \}, \tag {35} \\ \end{array}
$$
where we use the fact  $\Sigma_s\succeq \Sigma_s - (1 - \frac{1}{n_{2,s}'})\sigma_X^2 I$ , and for symmetric matrices  $A$  and  $B$  such that  $A\succeq B$ ,  $\langle v,Av\rangle \geq \langle v,Bv\rangle$  for any  $v\in \mathbb{R}$ . For  $n_s\leq 18d$ , the error is zero because  $\hat{\mu}_s^\prime = 0$
(Fifth term in Theorem 3) Recall the fifth term in Theorem 3
$$
\mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \left\langle \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \right\rangle\right) ^ {2} \right| n. \right] ^ {1 / 2}.
$$
Since  $\hat{\mu}_s^\prime$  are independent, we have
$$
\mathbf {E} \left[ \left. \left(\sum_ {s \in [ M ]} \hat {p} _ {s} \langle \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \rangle\right) ^ {2} \right| n. \right]
$$
$$
\begin{array}{l} = \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \mathbf {E} \left[ \left\langle \beta_ {s} ^ {*}, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \right\rangle^ {2} \mid n. \right] \mathbb {1} \left\{n _ {s} > 1 2 d \right\} \\ + \sum_ {s, s ^ {\prime} \in [ M ]} \hat {p} _ {s} \hat {p} _ {s ^ {\prime}} \mathbb {1} \left\{n _ {s} <   1 2 d, n _ {s ^ {\prime}} <   1 2 d \right\} \langle \beta_ {s} ^ {*}, \mu_ {s} \rangle \langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \rangle \\ \end{array}
$$
$$
\begin{array}{l} \leq \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} \| \beta_ {s} ^ {*} \| ^ {2} \sup  _ {v \in \mathbb {S} _ {d - 1}} \mathbf {E} \left[ \langle v, \hat {\mu} _ {s} ^ {\prime} - \mu_ {s} \rangle^ {2} \mid n. \right] \mathbb {1} \{n _ {s} > 1 2 d \} + \frac {1 4 4 d ^ {2} M ^ {2}}{n ^ {2}} \\ \leq \sum_ {s \in [ M ]} \hat {p} _ {s} ^ {2} B ^ {2} e _ {\text {m e a n} ^ {\prime}, s} ^ {2} \mathbb {1} \left\{n _ {s} > 1 2 d \right\} + o \left(\frac {1}{n}\right). \tag {36} \\ \end{array}
$$
(Sixth term in Theorem 3) Recall the sixth term in Theorem 3
$$
\left| \sum_ {s ^ {\prime} \in [ M ]} \left(\hat {p} _ {s ^ {\prime}} - p _ {s ^ {\prime}}\right) \left\langle \beta_ {s ^ {\prime}} ^ {*}, \mu_ {s ^ {\prime}} \right\rangle \right|,
$$
which is equivalent to  $e_{\mathrm{prob}}$
By combining Theorem 3 and Eqs (32) to (36), we get
$$
\begin{array}{l} \mathbf {E} \left[ \left(\hat {f} _ {n} (X, S) - f _ {\mathrm {D P}} ^ {*} (X, S)\right) ^ {2} \right] \\ \leq \sum_ {s \in [ M ]} p _ {s} \mathbf {E} \left[ \left((B + e _ {\text {n o r m}}) e _ {\text {m e a n}, s} \mathbb {1} \{n _ {s} > 1 8 d \} + \sigma_ {X} e _ {\text {n o r m}} + \sigma_ {X} \overline {{\| \beta_ {:} ^ {*} \|}} e _ {\text {c o e f}, s} \right. \right. \\ + \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} ^ {2} e _ {\operatorname {c o e f} ^ {\prime}, s ^ {\prime}} ^ {2} \mathbb {1} \{n _ {s ^ {\prime}} > 1 2 d \}\right) ^ {1 / 2} \\ \left. + \left(\sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} ^ {2} B ^ {2} e _ {\text {m e a n} ^ {\prime}, s ^ {\prime}} ^ {2} \mathbb {1} \{n _ {s ^ {\prime}} > 1 2 d \} + o \left(\frac {1}{n}\right)\right) ^ {1 / 2} + e _ {\text {p r o b}}\right) ^ {2} \rbrack . \\ \end{array}
$$
The triangle inequality gives that
$$
\begin{array}{l} \mathbf {E} \left[ \left(\hat {f} _ {n} (X, S) - f _ {\mathrm {D P}} ^ {*} (X, S)\right) ^ {2} \right] \\ \leq \sum_ {s \in [ M ]} 7 p _ {s} \left(B ^ {2} \mathbf {E} \left[ e _ {\text {m e a n}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 8 d \} \right] + \mathbf {E} \left[ e _ {\text {n o r m}} ^ {2} \right] \mathbf {E} \left[ e _ {\text {m e a n}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 8 d \} \right] \right. \\ \left. + \sigma_ {X} ^ {2} \mathbf {E} \left[ e _ {\text {n o r m}} ^ {2} \right] + \sigma_ {X} ^ {2} \overline {{\left\| \beta_ {s} ^ {*} \right\|}} ^ {2} \mathbf {E} \left[ e _ {\text {c o e f}, s} ^ {2} \right]\right) + \sum_ {s \in [ M ]} 7 \left(\mathbf {E} \left[ \hat {p} _ {s} ^ {2} e _ {\text {c o e f} ^ {\prime}, s} ^ {2} \right. \mathbb {1} \left\{n _ {s} > 1 2 d \right\} \right] \\ \left. + \mathbf {E} \left[ \hat {p} _ {s} ^ {2} B ^ {2} e _ {\text {m e a n} ^ {\prime}, s} ^ {2} \mathbb {1} \left\{n _ {s} > 1 2 d \right\} \right]\right) + 7 \mathbf {E} \left[ e _ {\text {p r o b}} ^ {2} \right] + o \left(\frac {1}{n}\right). \\ \end{array}
$$
By applying Theorem 16 and Lemma 7, we have
$$
\mathbf {E} \big [ e _ {\mathrm {m e a n}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 8 d \} \big ] \leq \mathbf {E} \bigg [ \frac {6 \sigma_ {X} ^ {2}}{n _ {s}} \mathbb {1} \{n _ {s} > 1 8 d \} \bigg ] \leq \frac {6 \sigma_ {X} ^ {2}}{p _ {s} (n + 1)} \bigg (1 + \frac {1}{1 8 d} \bigg).
$$
Also, from Theorem 17 and Lemmas 6 and 8, we have
$$
\begin{array}{l} \mathbf {E} \left[ e _ {\mathrm {n o r m}} ^ {2} \right] \leq \mathbf {E} \left[ \left(\sqrt {\frac {1 8 9 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \mathbb {1} \{n _ {s} \leq 1 8 d \} \frac {n _ {s}}{n}} + \left| \sum_ {s \in [ M ]} (\hat {p} _ {s} - p _ {s}) \| \beta_ {s} ^ {*} \| \right|\right) ^ {2} \right] \\ \leq \frac {3 7 8 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \mathbf {E} \left[ \mathbb {1} \left\{n _ {s} \leq 1 8 d \right\} \frac {2 n _ {s}}{n} \right] + \frac {2}{n} \mathbf {V a r} (\| \beta_ {S} ^ {*} \|) \\ \leq \frac {3 7 8 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \mathbb {P} \left\{n _ {s} \leq 1 8 d \right\} \frac {3 6 d}{n} + \frac {2 \max _ {s} \| \beta_ {s} ^ {*} \| ^ {2}}{n} \\ \leq \frac {3 7 8 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \sum_ {s \in [ M ]} \frac {3 6 d}{n} e ^ {- n p _ {s} / 8} + \frac {2 B ^ {2}}{n} \\ = \frac {3 7 8 e ^ {1 0} \sigma_ {\xi} ^ {2} M d}{\sigma_ {X} ^ {2} n} + \frac {2 B ^ {2}}{n} + o \left(\frac {1}{n}\right), \\ \end{array}
$$
provided that  $n > 36d / \min_{s\in [M]}p_s$ . By utilizing Theorem 18 and Lemmas 7 and 8, we have
$$
\begin{array}{l} \mathbf {E} \left[ e _ {\operatorname {c o e f}, s} ^ {2} \right] \leq \mathbf {E} \left[ \frac {7 5 6 e ^ {1 0} \sigma_ {\xi} ^ {2} d}{\sigma_ {X} ^ {2} \| \beta_ {s} ^ {*} \| ^ {2} n _ {s}} \mathbb {1} \{n _ {s} > 1 8 d \} \right] + \mathbb {P} \{n _ {s} \leq 1 8 d \} \\ \leq \frac {7 5 6 e ^ {1 0} \sigma_ {\xi} ^ {2} d}{p _ {s} \sigma_ {X} ^ {2} \| \beta_ {s} ^ {*} \| ^ {2} n} \left(1 + \frac {1}{1 8 d}\right) + e ^ {- n p _ {s} / 8} \\ = \frac {7 5 6 e ^ {1 0} \sigma_ {\xi} ^ {2} d}{p _ {s} \sigma_ {X} ^ {2} \| \beta_ {s} ^ {*} \| ^ {2} n} \left(1 + \frac {1}{1 8 d}\right) + o \left(\frac {1}{n}\right), \\ \end{array}
$$
provided that  $n > 36d / \min_{s\in [M]}p_s$ . Application of Theorem 19 gives
$$
\begin{array}{l} \mathbf {E} \left[ \hat {p} _ {s} ^ {2} e _ {\mathrm {c o e f} ^ {\prime}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 2 d \} \right] \leq \mathbf {E} \left[ \left(\frac {n _ {s}}{n}\right) ^ {2} \left(\frac {4 \sigma_ {\xi} ^ {2}}{n _ {s}} + 5 4 0 e ^ {1 0} \sigma_ {\xi} ^ {2} \left(\frac {4 d}{n _ {s}}\right) ^ {2}\right) \mathbb {1} \{n _ {s} > 1 2 d \} \right] \\ \leq \mathbf {E} \left[ \frac {n _ {s}}{n} \mathbb {1} \left\{n _ {s} > 1 2 d \right\} \right] \frac {4 \sigma_ {\xi} ^ {2}}{n _ {s}} + o \left(\frac {1}{n}\right) \leq p _ {s} \frac {4 \sigma_ {\xi} ^ {2}}{n _ {s}} + o \left(\frac {1}{n}\right). \\ \end{array}
$$
By Theorem 21, we have
$$
\mathbf {E} \left[ \hat {p} _ {s} ^ {2} e _ {\mathrm {m e a n} ^ {\prime}, s} ^ {2} \mathbb {1} \{n _ {s} > 1 2 d \} \right] \leq \mathbf {E} \left[ \frac {n _ {s}}{n} \frac {4 \sigma_ {X} ^ {2}}{n} \mathbb {1} \{n _ {s} > 1 2 d \} \right] \leq p _ {s} \frac {4 \sigma_ {X} ^ {2}}{n}.
$$
Application of Lemma 6 with  $a_{s} = \langle \beta_{s}^{*},\mu_{s}\rangle$  into  $e_{\mathrm{prob}}^2$  yields
$$
\mathbf {E} \left[ e _ {\text {p r o b}} ^ {2} \right] \leq \frac {1}{n} \mathbf {V a r} \left(\left\langle \beta_ {S} ^ {*}, \mu_ {S} \right\rangle\right) \leq \frac {B ^ {2} U ^ {2}}{n}.
$$
Synthesizing the results so far, there exists an universal constant  $C > 0$  such that