SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ | | \boldsymbol {x} _ {t} - \boldsymbol {x} _ {0} | | _ {2} ^ {2} \right] \leq \mathbb {E} \left[ (1 + | | \boldsymbol {x} _ {0} | | _ {2}) ^ {2} \right] \leq \left(1 + \mathbb {E} \left[ | | \boldsymbol {x} _ {0} | | _ {2} ^ {2} \right] ^ {\frac {1}{2}}\right) ^ {2} = \left(1 + d ^ {\frac {1}{2}} T ^ {- \frac {1}{3}}\right) ^ {2},
$$
where  $d$  is the dimension of the action space. We have obtained a lower bound of  $\mathbb{E}_{\boldsymbol{x}_0}[\mathbb{E}[f(\boldsymbol{x}_T)]]$  that is independent of the algorithm. Formally,
$$
\inf  _ {\mathcal {A}} \mathbb {E} _ {\boldsymbol {x} _ {0}} \left[ \mathbb {E} \left[ f (\boldsymbol {x} _ {T}) \right] \right] \geq \frac {1}{2} \left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2} / \left(T \left(1 + d ^ {\frac {1}{2}} T ^ {- \frac {1}{3}}\right) ^ {2} + T ^ {\frac {2}{3}} \operatorname {T r} \left(A ^ {- 2}\right)\right).
$$
We apply the above estimate to inequality (13). By taking the infimum over all algorithms,
$$
\liminf_{T\to \infty}\Re (T;A)\cdot T\geq \liminf_{T\to \infty}\frac{\inf_{\mathcal{A}}\mathbb{E}_{\boldsymbol{x}_{0}}\left[\mathbb{E}[f(\boldsymbol{x}_{T})]\right] - \mathbb{E}\left[ \mathbb{1}(\boldsymbol{x}_{0}\notin \mathcal{B})\cdot \sup_{\boldsymbol {x}\in \mathcal{B}}f(\boldsymbol {x})\right]}{\mathbb{P}[\boldsymbol{x}_{0}\in\mathcal{B}]}\cdot T
$$
For the purpose of our proof, the covariance of  $\pmb{x}_0$  can be arbitrary as long as their inverse is asymptotically large but sublinear w.r.t.  $T$ .
Observe that for our Gaussian prior,
$$
\lim_{T\to \infty}\mathbb{P}[\boldsymbol{x}_{0}\in \mathcal{B}] = 1,
$$
$$
\lim  _ {T \to \infty} \mathbb {E} \left[ \mathbb {1} (\boldsymbol {x} _ {0} \notin \mathcal {B}) \cdot \sup  _ {\boldsymbol {x} \in \mathcal {B}} f (\boldsymbol {x}) \right] \cdot T = 0.
$$
Hence, all terms above can be replaced by closed-form functions. and we have
$$
\begin{array}{l} \liminf_{T\to \infty}\mathfrak{R}(T;A)\cdot T\geq \liminf_{T\to \infty}\inf_{\mathcal{A}}\mathbb{E}_{\boldsymbol{x}_{0}}\left[\mathbb{E}[f(\boldsymbol{x}_{T})]\right]\cdot T \\ \geq \liminf_{T\to \infty}\frac{1}{2}\left(\operatorname {Tr}(A^{-\frac{1}{2}})\right)^{2} / \Bigg{/}\left(\left(1 + d^{\frac{1}{2}}T^{-\frac{1}{3}}\right)^{2} + T^{-\frac{1}{3}}\operatorname {Tr}(A^{-2})\right) \\ = \frac {1}{2} \left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}. \\ \end{array}
$$
4 Proof of Theorem 2.4
To prove the universal achievability result, we need to develop a new algorithm to learn and incorporate the needed Hessian information. Particularly, the procedure required for achieving the optimal rates is beyond simply adding an initial stage with an arbitrary Hessian estimator, for there are two challenges. First, any Hessian estimation algorithm would result in a mean squared error of  $\Omega(1/T)$  in the worst case, which translates to a cost of  $\Omega(1/T)$  in the minimax simple regret. This induced cost often introduces a multiplicative factor that is order-wise larger than the desired  $O\left((\mathrm{Tr}(A^{-\frac{1}{2}}))^2\right)$ . Second, to utilize any estimated Hessian, the analysis for the subsequent stages often requires the estimation error to have a light-tail distribution, which is not guaranteed for general linear estimators when the additive noise in the observation model has a heavy-tail distribution.
To overcome these challenges, we present two main algorithmic building blocks in the following subsections. The first uses  $o(T)$  samples to obtain a rough estimate of the Hessian, and achieves low-error guarantees with high probability through the introduction of a truncation method. The second estimates the global minimum with carefully designed sample points to minimize the error contributed from the Hessian estimation. We show that this sample phase can be written in the form of a two-step descent. Finally, we show the combination of two stages provides an algorithm that proves Theorem 2.4.
4.1 Initial Hessian Estimate and Truncation Method
Consider any sufficiently large  $T$ , we first use  $T_0 = \lceil T^{0.8} \rceil$  samples to find a rough estimate of  $A$ . Particularly, we rely on the following result.
Proposition 4.1. For any fixed dimension  $d$ , there is an algorithm that samples on  $T_{0}$  predetermined points and returns an estimation  $\widehat{A}$ , such that for any fixed  $\alpha \in (0, 0.5)$ ,  $\beta \in (0, +\infty)$ , and PSD matrix  $A$ ,
$$
\lim  _ {T _ {0} \rightarrow \infty} \sup  _ {f \in \mathcal {F} _ {A}} \mathbb {P} \left[ | | \widehat {A} - A | | _ {\mathrm {F}} \geq T _ {0} ^ {- \alpha} \right] \cdot T _ {0} ^ {\beta} = 0, \tag {14}
$$
where  $||\cdot ||_{\mathrm{F}}$  denotes the Frobenius norm.
Proof. We first consider the 1D case. Let  $y_{+}$ ,  $y_{-}$ , and  $y$  each be samples of  $f$  at 1, -1, and 0, respectively. When the additive noises are subgaussian, we have that  $(y_{+} + y_{-} - 2y)$  is an unbiased estimator of  $A$  with an error that is subgaussian. By repeating and averaging over this process  $\lfloor T_0 / 3 \rfloor$  times, the squared estimation error is reduced to  $O(1 / T_0)$ , and the normalized error is subgaussian, which satisfies the statement in the proposition.
However, recall that in Section 2 we did not assume the subgaussianity of  $w_{t}$ . A modification of the above estimator is required to achieve the same guarantee in equation (14). Here we propose a truncation method, which projects each measurement  $(y_{+} + y_{-} - 2y)$  to a bounded interval  $[-T_0^{0.5}, T_0^{0.5}]$ . Specifically, the returned  $\widehat{A}$  is the average of  $\lfloor T_0 / 3 \rfloor$  samples of  $\max \{\min \{y_{+} + y_{-} - 2y, T_0^{0.5}\}, -T_0^{0.5}\}$ , which provides a guaranteed superpolynomial tail bound. A more detailed discussion and analysis for the truncation method is provided in Appendix D.1.
For general  $d$ , one can return an estimator that satisfies the same light-tail requirement, for example, by applying the above 1D estimator repetitively  $\mathrm{poly}(d)$  times to obtain estimates of all entries of  $A$ . Then the overall error probability is controlled by the union bound.
4.2 Two-Step Descent Stage
Let  $\widehat{A}$  be any estimator that satisfies the condition in Proposition 4.1 for  $\alpha = 0.4$  and  $\beta = 1.6$ . Asymptotically, this implies the eigenvalues and eigenvectors of  $\widehat{A}$  are close to that of  $A$ . We choose an eigenbasis of  $\widehat{A}$  and remove vectors with vanishingly small eigenvalues to approximate the row space of  $A$ . Then, we estimate the entries of  $x_0$  in these remaining directions following an energy allocation defined based on  $\widehat{A}$ .
Specifically, consider any fixed realization of  $\widehat{A}$ , let  $\widehat{\lambda}_1, \widehat{\lambda}_2, \ldots$  be its eigenvalues in the non-increasing order,  $\widehat{e}_1, \widehat{e}_2, \ldots$  be the corresponding eigenvectors, and  $k^*$  be the largest integer such that  $\widehat{\lambda}_{k^*} \geq T^{-0.2}$ . We present the detailed steps in Algorithm 2, where  $T_1$  denotes the remaining available samples, i.e.,  $T - T_0$ .
Algorithm 2
procedure QUADRATIC SEARCH  $(\widehat{\lambda}_1,\widehat{\lambda}_2,\dots ,\widehat{\lambda}_{k^*},\widehat{e}_1,\dots ,\widehat{e}_{k^*},T_1)$    
for  $k\gets 1$  to  $k^{*}$  do Let  $p_k = \frac{\widehat{\lambda}_k^{-\frac{1}{2}}}{4\sum_{j = 1}^{k^*}\widehat{\lambda}_j^{-\frac{1}{2}}},t_k = [p_k\cdot (T_1 - 4d - 1)]$  Let  $\alpha_{k} = -\frac{1}{2\widehat{\lambda}_{k}}$  Truncated Diff  $(\widehat{e}_k, - \widehat{e}_k,t_k)$    
end for   
Let  $\widetilde{x} = \sum_{k = 1}^{k^{*}}\alpha_{k}\widehat{e}_{k},\widehat{x} = \widetilde{x}\cdot \min \left\{1,1.5 / ||\widetilde{x} ||_{2}\right\}$  Projecting to a O-centered hyperball   
for  $k\gets 1$  to  $k^{*}$  do Let  $\beta_{k} = -\frac{4}{\widetilde{\lambda}_{k}}$  Truncated Diff  $(\frac{\widehat{e}_k + 2\widehat{x}}{4},\frac{-\widehat{e}_k + 2\widehat{x}}{4},t_k)$  .Obtain an unbounded estimator   
end for   
return  $\boldsymbol {x}_T = \arg \min_{\boldsymbol {x} = \sum_{k = 1}^{k^*}\theta_k\widehat{e}_k\in \mathcal{B}}\sum_{k = 1}^{k^*}\widehat{\lambda}_k(\beta_k - \theta_k)^2$  Projection to B   
end procedure   
procedure TRUNCATED DIFF(xo, x1, t)   
for  $k\gets 1$  to  $t$  do Let  $y_{+},y_{-}$  be a sample of  $f$  at  $x_0,x_1$  , respectively Compute the projection of the difference  $y_{+} - y_{-}$  to the interval  $[-t^{0.5},t^{0.5}]$  , i.e., let   
 $z_{k} = \max \{\min \{y_{+} - y_{-},t^{0.5}\} , - t^{0.5}\}$    
end for   
return  $\frac{1}{t}\sum_{k = 1}^{t}z_{k}$    
end procedure
We use the eigenbasis of  $\widehat{A}$  and let  $\widehat{A}_0$  be the diagonal matrix given by  $\mathrm{diag}(\widehat{\lambda}_1, \widehat{\lambda}_2, \dots, \widehat{\lambda}_{k^*}, 0, \dots, 0)$ . In the first for-loop, we essentially estimated  $Ax_0$ , and then computed the first  $k^*$  entries of its product with the pseudo-inverse of  $\widehat{A}_0$  (note that the rest of the entries are all zero). Since  $\widehat{A}_0$  is a good estimate of  $A$  with high probability, we use it to compute the optimal energy spectrum similar to the instance-dependent case, and allocate the measurements accordingly.
Recall the proof in Section 3. By the analysis of the truncation method in Appendix D.1, the variable  $\widehat{\pmb{x}}$  could have served as an estimator of  $\pmb{x}_0$  if  $\widehat{A}_0 = A$ , where the estimation process reduces to the Hessian-dependent case. However,  $\widehat{A}_0$  relies only on  $O(T^{0.8})$  samples, which generally leads to an expected penalty of  $\Theta (T^{-0.8}) = \omega (1 / T)$  in simple regret if used in place of  $A$ . We reserve a fraction of samples for a second for-loop to fine-tune the estimation in order to achieve the optimal Hessian-dependent simple regrets.
4.3 Regret Analysis
Recall our assumption on the Hessian Estimator and  $T_0 = O(T^{0.8})$ . The probability for  $||\widehat{A} - A||_{\mathrm{F}} \geq T^{-0.32}$  is  $o(1 / T)$ . As our algorithm always returns  $x_{T}$  with bounded norms, the simple regret contributed by this exceptional case is negligible. Hence, we can focus on instances where  $||\widehat{A} - A||_{\mathrm{F}} \leq T^{-0.32}$ .
Under such scenario, any non-zero  $\widehat{\lambda}_k$  is close to a non-zero eigenvalue of  $A$ . Specifically, we have  $|\widehat{\lambda}_k - \lambda_k| \leq T^{-0.32}$  for all  $k$ , where each  $\lambda_k$  is the  $k$ th largest eigenvalue of  $A$ . Recall the definition of  $\widehat{A}_0$ . It implies that for sufficiently large  $T$ ,  $k^*$  equals the rank of  $A$ , and all non-zero eigenvalues of  $\widehat{A}_0$  are bounded away from zero. Consequently,  $||\widehat{A}_0^{-1}||_{\mathrm{F}} = ||A^{-1}||_{\mathrm{F}}(1 + o(1))$ , which does not scale w.r.t.  $T$ , where  $M^{-1}$  denotes the pseudo-inverse for any symmetric  $M$ . We also have  $(\mathrm{Tr}(\widehat{A}_0^{-\frac{1}{2}}))^2 = O\left((\mathrm{Tr}(A^{-\frac{1}{2}}))^2\right)$ .
The closeness between  $\widehat{A}$  and  $A$  also implies information on the eigenvectors of  $\widehat{A}_0$ . Recall that  $\widehat{A}_0$  is obtained by removing the diagonal entries of  $\widehat{A}$  (in their eigenbasis) that are below a threshold  $T^{-0.2}$ . For sufficiently large  $T$ , the removed entries are associated with the  $d - k^*$  smallest eigenvalues, which are no greater than  $T^{-0.32}$ . Hence, as a rough estimate, we have  $||\widehat{A}_0 - A||_{\mathrm{F}} = o(T^{-0.3})$ .
These conditions can be used to characterize the distribution of  $\widehat{\pmb{x}}$ . As mentioned earlier, from the analysis of the truncation method, each  $\alpha_{k}$  concentrates around  $\widehat{\pmb{e}}_k\widehat{A}_0^{-1}A\pmb{x}_0$ . Hence,  $\widetilde{\pmb{x}}$  concentrates near  $\widehat{A}_0^{-1}A\pmb{x}_0$ , and the truncation method ensures that
$$
\begin{array}{l} \mathbb {E} \left[ \left(\widetilde {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) ^ {\top} \widehat {A} _ {0} \left(\widetilde {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) \right] = O \left((\operatorname {T r} \left(\widehat {A} _ {0} ^ {- \frac {1}{2}}\right)) ^ {2}\right) / T = O \left((\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)) ^ {2}\right) / T, (15) \\ \mathbb {P} \left[ \left| \left| \widetilde {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0} \right| \right| _ {2} \geq T ^ {- 0. 4} \right] = o (1 / T). (16) \\ \end{array}
$$
Note that the  $L_{2}$ -norm of  $\widehat{A}_0^{-1}Ax_0$  is no greater than  $1 + o(1)$  under the condition of  $||\widehat{A}_0 - A||_{\mathrm{F}} = o(T^{-0.3})$ . Formally, by triangle inequality
$$
| | \widehat {A} _ {0} ^ {- 1} A \pmb {x} _ {0} | | _ {2} \leq | | \widehat {A} _ {0} ^ {- 1} \widehat {A} _ {0} \pmb {x} _ {0} | | _ {2} + | | \widehat {A} _ {0} ^ {- 1} (\widehat {A} _ {0} - A) \pmb {x} _ {0} | | _ {2} \leq 1 + | | \widehat {A} _ {0} ^ {- 1} | | _ {\mathrm {F}} \cdot | | \widehat {A} _ {0} - A | | _ {\mathrm {F}} = 1 + o (1).
$$
We can apply inequality (16) to show that the  $L_{2}$ -norm of  $\widetilde{\pmb{x}}$  is no greater than  $1 + o(1)$  with  $1 - o(1 / T)$  probability. Under such high probability cases, the projection of  $\widetilde{\pmb{x}}$  to the hyperball of radius 1.5 remains identical. Hence, by the PSD property of  $\widehat{A}_0$ , which is due to the convergence of its eigenvalues, we can replace all  $\widetilde{\pmb{x}}$  in inequality (15) with  $\widehat{\pmb{x}}$  and obtain that
$$
\mathbb {E} \left[ \left(\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) ^ {\top} \widehat {A} _ {0} \left(\widehat {\boldsymbol {x}} - \widehat {A} _ {0} ^ {- 1} A \boldsymbol {x} _ {0}\right) \right] = O \left(\left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}\right) / T. \tag {17}
$$
The same analysis can also be performed for each  $\beta_{k}$ , which concentrates near  $\widehat{\pmb{e}}_k\widehat{A}_0^{-1}A(2\pmb{x}_0 - \widehat{x})$ . Formally, let  $\widetilde{\pmb{x}}_T\triangleq \sum_k\beta_k\widehat{\pmb{e}}_k$ , we have
$$
\mathbb {E} \left[ \left(\widetilde {\boldsymbol {x}} _ {T} - \widehat {A} _ {0} ^ {- 1} A \left(2 \boldsymbol {x} _ {0} - \widehat {x}\right)\right) ^ {\top} \widehat {A} _ {0} \left(\widetilde {\boldsymbol {x}} _ {T} - \widehat {A} _ {0} ^ {- 1} A \left(2 \boldsymbol {x} _ {0} - \widehat {x}\right)\right) \right] = O \left((\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)) ^ {2}\right) / T. \tag {18}
$$
The above results for the two descent steps can be combined, using triangle inequality and Proposition 4.2 below, to obtain the following inequality (See Appendix D for their proofs).
$$
\mathbb {E} \left[ (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z}) ^ {\top} \widetilde {A} _ {0} (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z}) \right] = O \left(\left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}\right) / T. \tag {19}
$$
where we denote  $z \triangleq \left(2\widehat{A}_0^{-1}A - \left(\widehat{A}_0^{-1}A\right)^2\right)x_0$  for brevity.
Proposition 4.2. Let  $\widehat{A}_0, Z, \mathbf{y}$  be variables dependent on a parameter  $T \in \mathbb{N}$ , where  $\widehat{A}_0, Z$  are PSD matrices and  $\mathbf{y}$  belongs to the column space of  $\widetilde{A}_0$ . If  $\lim \sup_{T \to \infty} ||\widehat{A}_0^{-1}||_{\mathrm{F}} < \infty$  and  $\lim_{T \to \infty} ||Z - \widetilde{A}_0||_{\mathrm{F}} = 0$ , then
$$
\boldsymbol {y} ^ {\intercal} Z \boldsymbol {y} \leq (1 + | | Z - \widehat {A} _ {0} | | _ {\mathrm {F}} | | \widehat {A} _ {0} ^ {- 1} | | _ {\mathrm {F}}) (\boldsymbol {y} ^ {\intercal} \widehat {A} _ {0} \boldsymbol {y}) = (1 + o (1)) (\boldsymbol {y} ^ {\intercal} \widehat {A} _ {0} \boldsymbol {y}).
$$
Assume the correctness of inequality (19), the remainder of the proof consists of two parts. First, we show that the vector  $\mathbf{z}$  can be viewed as a Taylor approximation for a projection of  $x_0$  onto the column space of  $\widehat{A}_0$ , hence,  $\widetilde{\boldsymbol{x}}_T$  could achieve the needed simple regret. Then, we show that the projection of  $\widetilde{\boldsymbol{x}}_T$  to the unit hypersphere to obtain  $\boldsymbol{x}_T$  induces negligible cost, so that the validity constraint can be satisfied the same time.
For the first part, let  $P_{1}$  denote the projective map onto the column space of  $\widehat{A}_0$ , i.e.,  $P_{1} = \widehat{A}_{0}\widehat{A}_{0}^{-1}$  (here and in the following,  $\widehat{A}_0^{-1}$  denotes the pseudo inverse of  $\widehat{A}_0$ ). We consider the Hessian of  $f$  on this restricted subspace, which is given by  $A_{1} \triangleq P_{1}AP_{1}$ . Note that the definition of  $\widehat{A}_0$  implies  $\widehat{A}_0 = P_1\widehat{A}P_1$ . We have  $||\widehat{A}_0 - A_1||_{\mathrm{F}} = ||P_1(\widehat{A} - A)P_1||_{\mathrm{F}} \leq ||\widehat{A} - A||_{\mathrm{F}} \leq T^{-0.32}$ . Then recall that the non-zero eigenvalues of  $\widehat{A}_0$  are bounded away from 0, i.e.,  $||\widehat{A}_0^{-1}||_{\mathrm{F}} = o(T^{0.32})$ . We have that  $A_{1}$  is invertible within the column space of  $\widehat{A}_0$  (i.e.,  $A_{1}A_{1}^{-1} = P_{1}$ ) for sufficiently large  $T$ .
The pseudo inversion of  $A_{1}$  provides a point of equivalence to  $\pmb{x}_0$  within the column space of  $\widehat{A}_0$ . Particularly, let  $\pmb{z}_0 \triangleq A_1^{-1} A \pmb{x}_0$ , we have  $P_1 \pmb{z} = \pmb{z}$  and  $f(\pmb{x}) = (\pmb{x} - \pmb{z}_0)^{\top} A (\pmb{x} - \pmb{z}_0)$  for any  $\pmb{x} \in \mathbb{R}^d$  and sufficiently large  $T$ . The second equality is due to  $A(\pmb{x}_0 - \pmb{z}_0) = (A - A A_1^{-1} A) \pmb{x}_0 = 0$  for sufficiently large  $T$ , which is implied by the following proposition (see Appendix D.4 for a proof).
Proposition 4.3. For any symmetric matrix  $A$  and any symmetric projection map  $P_{1}$  (i.e.,  $P_{1}^{2} = P_{1} = P_{1}^{\intercal}$ ), if  $A_{1} \triangleq P_{1}AP_{1}$  and  $A$  has the same rank, then the pseudo inverse  $A_{1}^{-1}$  satisfies  $A = AA_{1}^{-1}A$ .
We show that the error  $z - z_0$  is bounded by  $o(T^{-0.5})$ . Observe that  $P_1\widehat{A}_0^{-1} = \widehat{A}_0^{-1}P_1 = \widehat{A}_0^{-1}$  and  $A_1^{-1}P_1 = A_1^{-1}$ , we have
$$
\boldsymbol {z} - \boldsymbol {z} _ {0} = \left(A _ {1} ^ {- 1} A - \left(2 \widehat {A} _ {0} ^ {- 1} A - \left(\widehat {A} _ {0} ^ {- 1} A\right) ^ {2}\right)\right) \boldsymbol {x} _ {0} = A _ {1} ^ {- 1} \left(\left(\widehat {A} _ {0} - A _ {1}\right) \widehat {A} _ {0} ^ {- 1}\right) ^ {2} A \boldsymbol {x} _ {0}.
$$
From  $||\widehat{A}_0 - A_1||_{\mathrm{F}} \leq T^{-0.32}$ ,  $||\widehat{A}_0^{-1}||_{\mathrm{F}} = ||\widehat{A}^{-1}||_{\mathrm{F}} \cdot O(1)$  and the fact that  $A_1$  is restricted to the column space of  $\widehat{A}_0$ , we can derive that  $||A_1^{-1}||_{\mathrm{F}} = ||\widehat{A}^{-1}||_{\mathrm{F}} \cdot O(1)$ , which also does not scale w.r.t.  $T$ . Therefore, the above equality implies that  $||z - z_0||_2 = o(T^{-0.5})$ . As a consequence, we have following inequalities due to inequality (19) and triangle inequality.
$$
\mathbb {E} \left[ (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z} _ {0}) ^ {\top} \widetilde {A} _ {0} (\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z} _ {0}) \right] = O \left(\left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}\right) / T, \tag {20}
$$
For the second part, we first note that the L2 norm of  $z_0$  is bounded by the spectrum norm of  $A_1^{-1}A$ . Specifically,
$$
\left| \left| \boldsymbol {z} _ {0} \right| \right| _ {2} = \left| \left| A _ {1} ^ {- 1} A \boldsymbol {x} _ {0} \right| \right| _ {2} \leq \left| \left| \boldsymbol {x} _ {0} \right| \right| _ {2} \cdot \left| \left| A _ {1} ^ {- 1} A \right| \right| \leq \left| \left| A _ {1} ^ {- 1} A \right| \right|,
$$
where  $||\cdot ||$  denotes the spectrum norm. Recall the definition of  $A_{1}$ , we have  $A_{1}^{-1}AA_{1} = A_{1}^{-1}P_{1}AP_{1}A_{1} = A_{1}^{-1}A_{1}A_{1}$ . Hence,
$$
\begin{array}{l} \left| \left| A _ {1} ^ {- 1} A \right| \right| ^ {2} = \left| \left| A _ {1} ^ {- 1} A ^ {2} A _ {1} ^ {- 1} \right| \right| = \left| \left| A _ {1} ^ {- 1} \left(A _ {1} + (A - A _ {1})\right) ^ {2} A _ {1} ^ {- 1} \right| \right| = \left| \left| P _ {1} + A _ {1} ^ {- 1} (A - A _ {1}) ^ {2} A _ {1} ^ {- 1} \right| \right| \\ \leq \left| \left| P _ {1} \right| \right| + \left| \left| A _ {1} ^ {- 1} (A - A _ {1}) ^ {2} A _ {1} ^ {- 1} \right| \right|. \\ \end{array}
$$
Note that  $||A - A_1||_{\mathrm{F}} \leq ||\widehat{A}_0 - A_1||_{\mathrm{F}} + ||\widehat{A}_0 - A_1||_{\mathrm{F}} \leq o(T^{-0.3})$ . By the fact that  $||A_1^{-1}||_{\mathrm{F}} = ||\widehat{A}^{-1}||_{\mathrm{F}} \cdot O(1)$ , we have  $||A_1^{-1}A||^2 \leq 1 + o(T^{-0.6})$ , and  $||z_0||_2 \leq 1 + o(T^{-0.6})$ .
Therefore, we have  $\min_{\pmb{x} \in \mathcal{B}} (\pmb{z}_0 - \pmb{x})^{\top} \widehat{A}_0(\pmb{z}_0 - \pmb{x}) = o(T^{-1.2})$ . The projection of  $\widetilde{\pmb{x}}_T$  to the unit hypersphere guarantees the following bound (see Appendix A for a proof).
$$
\mathbb {E} \left[ \left(\boldsymbol {x} _ {T} - \boldsymbol {z} _ {0}\right) ^ {\intercal} \widetilde {\boldsymbol {A}} _ {0} \left(\boldsymbol {x} _ {T} - \boldsymbol {z} _ {0}\right) \right] \leq \mathbb {E} \left[ \left(\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z} _ {0}\right) ^ {\intercal} \widetilde {\boldsymbol {A}} _ {0} \left(\widetilde {\boldsymbol {x}} _ {T} - \boldsymbol {z} _ {0}\right) \right] + o (T ^ {- 1. 2}) = O \left((\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)) ^ {2}\right) / T.
$$
Note that both  $\boldsymbol{x}_T$  and  $z_0$  belong to the column space of  $\widehat{A}_0$ . From Proposition 4.2, we can substitute  $\widehat{A}_0$  by  $A$  for the above inequality and obtain
$$
\mathbb {E} \left[ \left(\boldsymbol {x} _ {T} - \boldsymbol {z} _ {0}\right) ^ {\top} A \left(\boldsymbol {x} _ {T} - \boldsymbol {z} _ {0}\right) \right] = O \left(\left(\operatorname {T r} \left(A ^ {- \frac {1}{2}}\right)\right) ^ {2}\right) / T. \tag {21}
$$
Then the theorem is proved from the fact that  $f(\pmb{x}_T) = (\pmb{x}_T - \pmb{z}_0)^\top A(\pmb{x}_T - \pmb{z}_0)$ .
[26] P. Zhao and L. Lai. Optimal stochastic nonconvex optimization with bandit feedback. arXiv preprint arXiv:2103.16082, 2021.
A Projection Lemma
Proposition A.1. For any PSD matrix  $A$  with dimension  $d$ , any closed convex set  $\mathcal{B}$  in the Euclidean space  $\mathbb{R}^d$ , and  $\widehat{\pmb{x}} \in \mathbb{R}^d$ , let
$$
\boldsymbol{x}^{*} = \operatorname *{argmin}_{\boldsymbol {x}\in \mathcal{B}}g(\widehat{\boldsymbol{x}},\boldsymbol {x})
$$
$$
g (\boldsymbol {u}, \boldsymbol {v}) \triangleq (\boldsymbol {u} - \boldsymbol {v}) ^ {\intercal} A (\boldsymbol {u} - \boldsymbol {v}),
$$
$$
g \left(\boldsymbol {x} ^ {*}, \boldsymbol {x} _ {0}\right) \leq g \left(\widehat {\boldsymbol {x}}, \boldsymbol {x} _ {0}\right) \quad \forall \boldsymbol {x} _ {0} \in \mathcal {B}.
$$
More generally,
$$
g (\boldsymbol {x} ^ {*}, \boldsymbol {z} _ {0}) \leq g (\widehat {\boldsymbol {x}}, \boldsymbol {z} _ {0}) + \min  _ {\boldsymbol {x} \in \mathcal {B}} g (\boldsymbol {z} _ {0}, \boldsymbol {x}) \quad \forall \boldsymbol {z} _ {0} \in \mathbb {R} ^ {d}.
$$
Proof. We first provide a proof for  $\pmb{x}_0 \in \mathcal{B}$ . For any  $\alpha \in [0,1]$ , let
$$
\boldsymbol {x} _ {\alpha} \triangleq \alpha \boldsymbol {x} ^ {*} + (1 - \alpha) \boldsymbol {x} _ {0}.
$$
By convexity, we have  $\pmb{x}_{\alpha} \in \mathcal{B}$  for any  $\alpha$ . Note that  $g(\widehat{\pmb{x}}, \pmb{x}_{\alpha})$  is differentiable. By the definition of  $x^{*}$ , we have
$$
\left. \left(\boldsymbol {x} ^ {*} - \widehat {\boldsymbol {x}}\right) ^ {\intercal} A \left(\boldsymbol {x} ^ {*} - \boldsymbol {x} _ {0}\right) = \frac {1}{2} \frac {\partial}{\partial \alpha} g \left(\widehat {\boldsymbol {x}}, \boldsymbol {x} _ {\alpha}\right) \right| _ {\alpha = 1} \leq 0.
$$