SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
g \left(\boldsymbol {x} ^ {*}, \boldsymbol {x} _ {0}\right) = g \left(\widehat {\boldsymbol {x}}, \boldsymbol {x} _ {0}\right) + 2 \left(\boldsymbol {x} ^ {*} - \widehat {\boldsymbol {x}}\right) ^ {\intercal} A \left(\boldsymbol {x} ^ {*} - \boldsymbol {x} _ {0}\right) - g \left(\boldsymbol {x} ^ {*}, \widehat {\boldsymbol {x}}\right) \leq g \left(\widehat {\boldsymbol {x}}, \boldsymbol {x} _ {0}\right),
$$
where the last inequality uses the PSD property of  $A$ .
Now we consider the more general case and let  $\pmb{x}$  be any vector in  $\mathcal{B}$ . Following the same steps in the earlier case, we have
$$
\left(\boldsymbol {x} ^ {*} - \widehat {\boldsymbol {x}}\right) ^ {\intercal} A \left(\boldsymbol {x} ^ {*} - \boldsymbol {x}\right) \leq 0.
$$
$$
\begin{array}{l} g \left(\boldsymbol {x} ^ {*}, \boldsymbol {z} _ {0}\right) - g \left(\widehat {\boldsymbol {x}}, \boldsymbol {z} _ {0}\right) = 2 \left(\boldsymbol {x} ^ {*} - \widehat {\boldsymbol {x}}\right) ^ {\intercal} A \left(\boldsymbol {x} ^ {*} - \boldsymbol {z} _ {0}\right) - g \left(\boldsymbol {x} ^ {*}, \widehat {\boldsymbol {x}}\right) \\ \leq 2 \left(\boldsymbol {x} ^ {*} - \widehat {\boldsymbol {x}}\right) ^ {\top} A \left(\boldsymbol {x} - \boldsymbol {z} _ {0}\right) - g \left(\boldsymbol {x} ^ {*}, \widehat {\boldsymbol {x}}\right) \\ = g \left(\boldsymbol {z} _ {0}, \boldsymbol {x}\right) - \left(\boldsymbol {x} - \boldsymbol {z} _ {0} - \boldsymbol {x} ^ {*} + \widehat {\boldsymbol {x}}\right) ^ {\top} A \left(\boldsymbol {x} - \boldsymbol {z} _ {0} - \boldsymbol {x} ^ {*} + \widehat {\boldsymbol {x}}\right) \\ \leq g \left(\boldsymbol {z} _ {0}, \boldsymbol {x}\right). \\ \end{array}
$$
Note that the above inequality holds for any  $\pmb{x} \in \mathcal{B}$ . The proposition is proved by taking the minimum over  $\pmb{x}$ .
B Proof of Proposition 3.3
Proof. We first prove for the case where  $Z$  is deterministic. Let  $\mu_Z$  denote the conditional expectation of  $\theta$ . By Cauchy's inequality,
$$
\mathbb {E} \left[ (\theta - \mu_ {Z}) ^ {2} | \boldsymbol {Z} \right] \cdot \mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {Z} (\theta)\right) ^ {2} | \boldsymbol {Z} \right] \geq \mathbb {E} \left[ \left| (\theta - \mu_ {Z}) \cdot \frac {\partial}{\partial \theta} \ln f _ {Z} (\theta) \right| | \boldsymbol {Z} \right] ^ {2}. \tag {22}
$$
The quantity on the RHS above can be bounded as follows.
$$
\begin{array}{l} \mathbb {E} \left[ \left| (\theta - \mu_ {\boldsymbol {Z}}) \cdot \frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta) \right| \middle | \boldsymbol {Z} \right] = \int \left| (\theta - \mu_ {\boldsymbol {Z}}) \cdot \frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta) \right| f _ {\boldsymbol {Z}} (\theta) d \theta \\ = \int \left| (\theta - \mu_ {Z}) \cdot \frac {\partial}{\partial \theta} f _ {Z} (\theta) \right| d \theta \\ \geq \limsup_ {T \rightarrow + \infty} \left| \int_ {- T} ^ {T} (\theta - \mu_ {\mathbf {Z}}) \cdot \frac {\partial}{\partial \theta} f _ {\mathbf {Z}} (\theta) d \theta \right| \\ = \limsup_{T\to +\infty}\left|\left((\theta -\mu_ {\boldsymbol {Z}})f_{\boldsymbol {Z}}(\theta)\big|_{\theta = -T}^{\theta = T}\right) - \mathbb{P}[ \theta \in [-T,T] | \boldsymbol {Z}]\right| \\ \geq 1, \\ \end{array}
$$
where the last inequality uses the integrability of  $f_{Z}$ , which implies
$$
\liminf_{T\to +\infty}\left(\theta -\mu_{Z}\right)f_{Z}(\theta)\big|_{\theta = -T}^{\theta = T}\leq 0.
$$
Then we evaluate the second factor on the LHS of inequality (22). Recall that  $\frac{\partial^2}{\partial\theta^2}\ln f_Z(\theta)$  is integrable, the following limit exists.
$$
\mathbb {E} \left[ \frac {\partial^ {2}}{\partial \theta^ {2}} \ln f _ {\boldsymbol {Z}} (\theta) \Bigg | \boldsymbol {Z} \right] = \lim _ {T \to + \infty} \int_ {- T} ^ {T} f _ {\boldsymbol {Z}} (\theta) \frac {\partial^ {2}}{\partial \theta^ {2}} \ln f _ {\boldsymbol {Z}} (\theta) d \theta .
$$
Then by positivity, we also have
$$
\mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2} \Bigg | \boldsymbol {Z} \right] = \lim _ {T \to + \infty} \int_ {- T} ^ {T} f _ {\boldsymbol {Z}} (\theta) \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2} d \theta .
$$
If we focus the non-trivial case where the first limit is not  $-\infty$ , the above two equations imply the existence of the following limit.
$$
\begin{array}{l} \mathbb {E} \left[ \frac {\partial^ {2}}{\partial \theta^ {2}} \ln f _ {\boldsymbol {Z}} (\theta) \middle | \boldsymbol {Z} \right] + \mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2} \middle | \boldsymbol {Z} \right] \\ = \lim  _ {T \rightarrow + \infty} \int_ {- T} ^ {T} f _ {\boldsymbol {Z}} (\theta) \left(\frac {\partial^ {2}}{\partial \theta^ {2}} \ln f _ {\boldsymbol {Z}} (\theta) + \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2}\right) d \theta \\ = \lim  _ {T \rightarrow + \infty} f _ {\mathcal {Z}} (\theta) \frac {\partial}{\partial \theta} \ln f _ {\mathcal {Z}} (\theta) \Big | _ {\theta = - T} ^ {\theta = T} \\ = \lim  _ {T \rightarrow + \infty} \frac {\partial}{\partial \theta} f _ {Z} (\theta) \Big | _ {\theta = - T} ^ {\theta = T}. \\ \end{array}
$$
The result of the above equation has to be zero, because the limit points of  $\frac{\partial}{\partial\theta} f_Z(\theta)$  must contain zero on both ends of the real line, which is implied by the integrability of  $f_{Z}$ . Consequently, we have
$$
\mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {Z} (\theta)\right) ^ {2} \mid Z \right] = \mathbb {E} \left[ - \frac {\partial^ {2}}{\partial \theta^ {2}} \ln f _ {Z} (\theta) \mid Z \right]. \tag {23}
$$
Then, the special case of Proposition 3.3 with fixed  $Z$  is implied by inequality (22).
When  $Z$  is variable, we simply have
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {V a r} [ \theta | \boldsymbol {Z} ] \right] \geq \mathbb {E} \left[ 1 / \mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2} \mid \boldsymbol {Z} \right] \right] \\ \geq \frac {1}{\mathbb {E} \left[ \mathbb {E} \left[ \left(\frac {\partial}{\partial \theta} \ln f _ {\boldsymbol {Z}} (\theta)\right) ^ {2} | \boldsymbol {Z} \right] \right]}. \\ \end{array}
$$
Then the proposition is implied by equation (23).
C Proof of Theorem 2.2
We first investigate the lower bounds. Observe that the proof provided in Section 3.1 only fails when the constructed hard instances have  $||\pmb{x}_0||_2 > 1$ . Hence, we have already covered the  $T \geq \left(\sum_{k=1}^{d} \lambda_k^{-\frac{1}{2}}\right) \left(\sum_{k=1}^{d} \lambda_k^{-\frac{3}{2}}\right)$  case, i.e., when  $k^* = \dim A = d$ . It remains to consider the other scenarios, where  $k^* < d$  is satisfied.
By the assumption that  $T \geq \left( \sum_{k=1}^{k^*} \lambda_k^{-\frac{1}{2}} \right) \left( \sum_{k=1}^{k^*} \lambda_k^{-\frac{3}{2}} \right)$ , one can instead set the entries of  $x_0$  in the earlier proof with indices greater than  $k^*$  to be zero, so that  $||x_0||_2 \leq 1$  is satisfied. Formally, let the hard-instance functions be constructed by the following set.
$$
\boldsymbol {x} _ {0} \in \mathcal {X} _ {\mathrm {H}} \triangleq \left\{(x _ {1}, x _ {2}, \dots , x _ {k ^ {*}}, 0, \dots , 0) \Bigg | x _ {k} = \pm \sqrt {\frac {\lambda_ {k} ^ {- \frac {3}{2}} \left(\sum_ {j} \lambda_ {j} ^ {- \frac {1}{2}}\right)}{2 T}}, \forall k \in [ k ^ {*} ] \right\}.
$$
Then by the identical proof steps, we have  $\Re (T;A) = \Omega \left(\left(\sum_{k = 1}^{k^*}\lambda_k^{-\frac{1}{2}}\right)^2\Big / T\right)$ .
Next, we show that  $\Re (T;A) = \Omega (\lambda_{k^{*} + 1})$ . We assume the non-trivial case where  $\lambda_{k^{*} + 1}\neq 0$ . Note that  $\Re (T;A)$  is non-increasing w.r.t.  $T$ . We can lower bound  $\Re (T;A)$  through the above steps but by replacing  $T$  with any larger quantity. Specifically, recall that  $k^*$  is largest integer satisfying  $T\geq \left(\sum_{k = 1}^{k^{*}}\lambda_{k}^{-\frac{1}{2}}\right)\left(\sum_{k = 1}^{k^{*}}\lambda_{k}^{-\frac{3}{2}}\right)$ , which implies  $T\leq \left(\sum_{k = 1}^{k^{*} + 1}\lambda_{k}^{-\frac{1}{2}}\right)\left(\sum_{k = 1}^{k^{*} + 1}\lambda_{k}^{-\frac{3}{2}}\right)$ . We have,
$$
\Re (T; A) \geq \Re \left(\left(\sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {1}{2}}\right) \left(\sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {3}{2}}\right); A\right).
$$
Notice that this change of sampling time allows us to apply the earlier lower bound with  $k^*$  incremented by 1.
$$
\begin{array}{l} \Re (T; A) \geq \Omega \left(\frac {\left(\sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {1}{2}}\right) ^ {2}}{\left(\sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {1}{2}}\right) \left(\sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {3}{2}}\right)}\right) \\ = \Omega \left( \begin{array}{c} \sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {1}{2}} \\ \hline \sum_ {k = 1} ^ {k ^ {*} + 1} \lambda_ {k} ^ {- \frac {3}{2}} \end{array} \right) = \Omega (\lambda_ {k ^ {*} + 1}). \\ \end{array}
$$
To conclude,
$$
\Re (T; A) = \Omega \left(\max  \left\{\frac {\left(\sum_ {k = 1} ^ {k ^ {*}} \lambda_ {k} ^ {- \frac {1}{2}}\right) ^ {2}}{T}, \lambda_ {k ^ {*} + 1} \right\}\right) = \Omega \left(\frac {\left(\sum_ {k = 1} ^ {k ^ {*}} \lambda^ {- \frac {1}{2}}\right) ^ {2}}{T} + \lambda_ {k ^ {*} + 1}\right),
$$
which completes the proof of the lower bounds.
The needed upper bounds can be obtained by only estimating the first  $k^*$  entries of  $\pmb{x}_0$ .
Remark C.1. The requirement of  $T > 3 \dim A$  in the Theorem statement is simply due to the integer constraints for the achievability bounds. Indeed, when  $\lambda_{\dim A}$  is large, it requires at least  $\Omega(\dim A)$  samples to achieve  $O(1)$  expected simple regret.
D Proof Details for Theorem 2.4
D.1 Truncation Method and Its Applications
The truncation method is based on the following facts.
Proposition D.1. For any sequence of independent random variables  $X_{1}, X_{2}, \ldots, X_{n}$  and any fixed parameter  $m$  satisfying  $m > \max_k |\mathbb{E}[X_k]|$ . Let  $Z_{k} = \max \{\min \{X_{k}, m\}, -m\}$  for any  $k \in [n]$ , we have
$$
\left| \mathbb {E} \left[ Z _ {k} \right] - \mathbb {E} \left[ X _ {k} \right] \right| \leq \frac {1}{4} \cdot \frac {\operatorname {V a r} \left[ X _ {k} \right]}{m - \left| \mathbb {E} \left[ X _ {k} \right] \right|}, \tag {24}
$$
$$
\operatorname {V a r} \left[ Z _ {k} \right] \leq \mathbb {E} \left[ \left(Z _ {k} - \mathbb {E} \left[ X _ {k} \right]\right) ^ {2} \right] \leq \operatorname {V a r} \left[ X _ {k} \right]. \tag {25}
$$
Moreover, for any  $z > 0$ , we have
$$
\mathbb {P} \left[ \left| \sum_ {k} Z _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ] \right| \geq z \right] \leq 2 \exp \left(\sum_ {k} \frac {\operatorname {V a r} [ X _ {k} ]}{m (m - | \mathbb {E} [ X _ {k} ] |)} - \frac {z}{m}\right). \tag {26}
$$
Proof. The first inequality is proved by expressing the LHS with piecewise linear functions. Note that by the definition of  $Z_{k}$ , we have
$$
\begin{array}{l} \left| \mathbb {E} \left[ Z _ {k} \right] - \mathbb {E} \left[ X _ {k} \right] \right| = \left| \mathbb {E} \left[ \max  \{- m - X _ {k}, 0 \} \right] - \mathbb {E} \left[ \max  \{X _ {k} - m, 0 \} \right] \right| \\ \leq | \mathbb {E} [ \max  \{- m - X _ {k}, 0 \} ] | + | \mathbb {E} [ \max  \{X _ {k} - m, 0 \} ] | \\ = \mathbb {E} [ \max  \{| X _ {k} | - m, 0 \} ]. \\ \end{array}
$$
We apply the following inequalities, which holds for any  $m \geq |\mathbb{E}[X_k]|$ .
$$
\left| X _ {k} \right| - m \leq \left| X _ {k} - \mathbb {E} \left[ X _ {k} \right] \right| - m + \mathbb {E} \left[ X _ {k} \right] \leq \frac {1}{4} \cdot \frac {\left| X _ {k} - \mathbb {E} \left[ X _ {k} \right] \right| ^ {2}}{m - \mathbb {E} \left[ X _ {k} \right]}.
$$
$$
\begin{array}{l} \left| \mathbb {E} \left[ Z _ {k} \right] - \mathbb {E} \left[ X _ {k} \right] \right| \leq \mathbb {E} \left[ \frac {1}{4} \cdot \frac {\left| X _ {k} - \mathbb {E} \left[ X _ {k} \right] \right| ^ {2}}{m - \mathbb {E} \left[ X _ {k} \right]} \right] \\ = \frac {1}{4} \cdot \frac {\operatorname {V a r} \left[ X _ {k} \right]}{m - \left| \mathbb {E} \left[ X _ {k} \right] \right|}. \\ \end{array}
$$
The second inequality is due to the following elementary facts,
$$
\mathbb {E} \left[ \left(Z _ {k} - \mathbb {E} [ X _ {k} ]\right) ^ {2} \right] \leq \mathbb {E} \left[ \left(X _ {k} - \mathbb {E} [ X _ {k} ]\right) ^ {2} \right] = \operatorname {V a r} [ X _ {k} ],
$$
where the inequality step is implied by the definition of  $Z_{k}$  and the condition  $m > \max_k|\mathbb{E}[X_k]|$ .
To prove the third inequality, we first investigate the following upper bound, which is due to Markov's inequality.
$$
\begin{array}{l} \mathbb {P} \left[ \sum_ {k} Z _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ] \geq z \right] \leq \frac {\mathbb {E} \left[ e ^ {\frac {1}{m} \left(\sum_ {k} Z _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ]\right)} \right]}{e ^ {\frac {z}{m}}} \\ = \frac {\prod_ {k} \mathbb {E} \left[ e ^ {\frac {1}{m} \left(Z _ {k} - \mathbb {E} \left[ X _ {k} \right]\right)} \right]}{e ^ {\frac {z}{m}}} \tag {27} \\ \end{array}
$$
The equality step above is by the fact that  $Z_{k}$ 's are jointly independent. For each  $k$ , using the fact that  $Z_{k}$  is bounded, particularly,  $Z_{k} - \mathbb{E}[X_{k}] \leq m + |\mathbb{E}[X_{k}]|$ , we have the following inequality
$$
e ^ {\frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ])} - 1 - \frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ]) \leq (Z _ {k} - \mathbb {E} [ X _ {k} ]) ^ {2} \cdot \frac {e ^ {\frac {1}{m} (m + | \mathbb {E} [ X _ {k} ] |)} - 1 - \frac {1}{m} (m + | \mathbb {E} [ X _ {k} ] |)}{(m + | \mathbb {E} [ X _ {k} ] |) ^ {2}}.
$$
For brevity, let  $\theta \triangleq \frac{|\mathbb{E}[X_k]|}{m}$ . We combine the above bound with inequality (24) and (25) to obtain that
$$
\begin{array}{l} \mathbb {E} [ e ^ {\frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ])} ] = 1 + \mathbb {E} \left[ \frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ]) \right] + \mathbb {E} \left[ e ^ {\frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ])} - 1 - \frac {1}{m} (Z _ {k} - \mathbb {E} [ X _ {k} ]) \right] \\ \leq 1 + \frac {\operatorname {V a r} \left[ X _ {k} \right]}{m \left(m - \left| \mathbb {E} \left[ X _ {k} \right] \right|\right)} \cdot \left(\frac {1}{4} + (1 - \theta) \cdot \frac {e ^ {1 + \theta} - 2 - \theta}{(1 + \theta) ^ {2}}\right). \tag {28} \\ \end{array}
$$
Recall that  $\theta < 1$  as assumed in the proposition. From elementary calculus, we have
$$
\begin{array}{l} \mathbb {E} \left[ e ^ {\frac {1}{m} \left(Z _ {k} - \mathbb {E} \left[ X _ {k} \right]\right)} \right] \leq 1 + \frac {\operatorname {V a r} \left[ X _ {k} \right]}{m (m - | \mathbb {E} \left[ X _ {k} \right] |)} \\ \leq \exp \left(\frac {\operatorname {V a r} [ X _ {k} ]}{m (m - | \mathbb {E} [ X _ {k} ] |)}\right). \\ \end{array}
$$
Therefore, recall inequality (27), we have
$$
\mathbb {P} \left[ \sum_ {k} Z _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ] \geq z \right] \leq \exp \left(\sum_ {k} \frac {\operatorname {V a r} [ X _ {k} ]}{m (m - | \mathbb {E} [ X _ {k} ] |)} - \frac {z}{m}\right).
$$
By symmetry, one can also prove the following bound through the same steps.
$$
\mathbb {P} \left[ \sum_ {k} Z _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ] \leq - z \right] \leq \exp \left(\sum_ {k} \frac {\operatorname {V a r} [ X _ {k} ]}{m (m - | \mathbb {E} [ X _ {k} ] |)} - \frac {z}{m}\right).
$$
Hence, the needed inequality is obtained by adding the two inequalities above.
Now equation (14) for the 1D case is immediately implied by Proposition D.1. Recall the construction of  $\widehat{A}$  in the proof, for any sufficiently large  $T_0$ , we have
$$
\mathbb {P} \left[ \left| \widehat {A} - A \right| \geq T _ {0} ^ {- \alpha} \right] \leq 2 \exp \left(3 - \frac {T _ {0} ^ {0 . 5 - \alpha}}{3}\right) = o \left(\frac {1}{T _ {0} ^ {\beta}}\right).
$$
Remark D.2. Instead of projecting to a bounded interval, the same achievability result can be obtained if the we average over any functions that map the samples to  $[-T_0^{0.5}, T_0^{0.5}]$  while imposing an additional error of  $o(T^{-\alpha})$  everywhere. This includes  $\Theta(\ln T)$ -bit uniform quantizers, which naturally appear in digital systems, over which exact computation can be performed to eliminate numerical errors. We present this simple generalization in the following corollary.
Corollary D.3. Consider the setting in Proposition D.1. Let  $Y_{1}, \ldots, Y_{n}$  be variables that satisfy  $|Y_{k} - Z_{k}| \leq b$  for all  $k$  with probability 1. We have
$$
\mathbb {P} \left[ \left| \sum_ {k} Y _ {k} - \sum_ {k} \mathbb {E} [ X _ {k} ] \right| \geq z \right] \leq 2 \exp \left(\sum_ {k} \frac {\operatorname {V a r} [ X _ {k} ]}{m (m - | \mathbb {E} [ X _ {k} ] |)} - \frac {z - b n}{m}\right).
$$
D.2 Proof of Proposition 4.2
$$
\boldsymbol {y} ^ {\intercal} Z \boldsymbol {y} - \boldsymbol {y} ^ {\intercal} \widehat {A} _ {0} \boldsymbol {y} = \boldsymbol {y} ^ {\intercal} (Z - \widehat {A} _ {0}) \boldsymbol {y} \leq | | Z - \widehat {A} _ {0} | | _ {\mathrm {F}} | | \boldsymbol {y} | | _ {2} ^ {2} \leq | | Z - \widehat {A} _ {0} | | _ {\mathrm {F}} | | \widehat {A} _ {0} ^ {- 1} | | _ {\mathrm {F}} (\boldsymbol {y} ^ {\intercal} \widehat {A} _ {0} \boldsymbol {y}).
$$
D.3 Proof of inequality (19)
We apply Proposition 4.2 to inequality (17) and let  $Z = A\widehat{A}_0^{-1}A$ . Note that
$$
\left| \left| Z - \widehat {A} _ {0} \right| \right| _ {\mathrm {F}} \leq 2 \left| \left| A - \widehat {A} _ {0} \right| \right| _ {\mathrm {F}} + \left| \left| (A - \widehat {A} _ {0}) \widehat {A} _ {0} ^ {- 1} (A - \widehat {A} _ {0}) \right| \right| _ {\mathrm {F}} = o (1),
$$
which satisfies the condition of Proposition 4.2. Using the fact that  $\widehat{A}_0^{-1}\widehat{A}_0\widehat{A}_0^{-1} = \widehat{A}_0^{-1}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(\widehat {A} _ {0} ^ {- 1} A (\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0})\right) ^ {\top} \widehat {A} _ {0} \left(\widehat {A} _ {0} ^ {- 1} A (\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0})\right) \right] \\ = \mathbb {E} \left[ \left(\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) ^ {\top} Z \left(\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) \right] = O \left((\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)) ^ {2}\right) / T. \tag {29} \\ \end{array}
$$
Then by the triangle inequality for the PSD matrix  $\hat{A}_0$ , the combination of the above inequality and inequality (18) gives
$$
\mathbb {E} \left[ (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z}) ^ {\intercal} \widehat {A} _ {0} (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z}) \right] = O \left(\left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}\right) / T.
$$
D.4 Proof of Proposition 4.3
Proof. When  $A_{1}$  and  $A$  has the same rank, the map  $P_{1}$  is invertible over the column space of  $A$ . Under such condition, there exists a matrix  $X$  such that  $A = XP_{1}A$ . Note that  $A_{1}A_{1}^{-1} = P_{1}$ . We have  $XP_{1} = XP_{1}A_{1}A_{1}^{-1} = AA_{1}^{-1}$ . Therefore, the needed  $A = AA_{1}^{-1}A$  is obtained by multiplying  $A$  on the right-hand sides in the above identity.