SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24:2312-2320, 2011.
Jacob Abernethy and Alexander Rakhlin. Beating the adaptive bandit with high probability. In 2009 Information Theory and Applications Workshop, pages 280-289. IEEE, 2009.
Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Conference on Learning Theory, 2009.
Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics, pages 19-26. PMLR, 2012.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, pages 1638-1646. PMLR, 2014.
Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory, pages 12-38. PMLR, 2017.
Alexander Rakhlin and Karthik Sridharan. Bistro: An efficient relaxation-based method for contextual bandits. In International Conference on Machine Learning, pages 1977-1985. PMLR, 2016.
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Aadirupa Saha, Pierre Gaillard, and Michal Valko. Improved sleeping bandits with stochastic action sets and adversarial rewards. In International Conference on Machine Learning, pages 8357-8366. PMLR, 2020.
Uri Sherman, Tomer Koren, and Yishay Mansour. Improved regret for efficient online reinforcement learning with linear function approximation. In International Conference on Machine Learning, 2023.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 47(3):1904-1931, 2022.
Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert Schapire. Efficient algorithms for adversarial contextual learning. In International Conference on Machine Learning, pages 2159-2168. PMLR, 2016a.
Vasilis Syrgkanis, Haipeng Luo, Akshay Krishnamurthy, and Robert E Schapire. Improved regret bounds for oracle-based adversarial contextual bandits. Advances in Neural Information Processing Systems, 29, 2016b.
Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications, pages 495-517, 2017.
Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference on Learning Theory, pages 1263-1291. PMLR, 2018.
Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory, pages 1043-1096. PMLR, 2022.
Yunbei Xu and Assaf Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876, 2020.
Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In Conference on Learning Theory, 2023.
Yinglun Zhu, Dylan J Foster, John Langford, and Paul Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning, pages 27428-27453. PMLR, 2022.
Julian Zimmert and Tor Lattimore. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory, pages 3285-3312. PMLR, 2022.
Julian Zimmert, Naman Agarwal, and Satyen Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory, pages 182-226. PMLR, 2022.
A Summary of Notation 15
B Auxiliary Lemmas 15
C Concentration Inequalities 16
C.1 General Concentration Inequalities 16
C.2 Concentration Inequalities under a Fixed Policy  $p$  17
C.3 Union Bound over Policies 21
D Regret Analysis 26
D.1 Bounding the Bias term 26
D.2 Bounding the Bonus term 27
D.3 Bounding the Penalty term 30
D.4 Bounding the Stability-1 term 30
D.5 Bounding the Stability-2 term 32
D.6 Bounding the Error term 33
D.7 Finishing up 33
E Handling Misspecification 34
E.1 Known misspecification 34
E.2 Unknown misspecification 36
F Analysis for Linear EXP4 42
G Comparison with Dai et al. [2023] and Sherman et al. [2023] 43
G.1 Regret Analysis Sketch 44
A Summary of Notation
We summarize the notations that have been defined in Algorithm 1 and Definition 1.
$$
\beta_ {t} = \Theta \left(\frac {(d + 1) ^ {3} \log (T / \delta)}{t - 1}\right)
$$
$$
\hat {x} _ {t} = \frac {1}{t - 1} \sum_ {\tau = 1} ^ {t - 1} \mathbb {E} _ {a \sim p _ {t} ^ {A _ {\tau}}} [ a ]
$$
$$
\hat {H} _ {t} = \frac {1}{t - 1} \sum_ {\tau = 1} ^ {t - 1} \mathbb {E} _ {a \sim p _ {t} ^ {A _ {\tau}}} \left[ (a - \hat {x} _ {t}) (a - \hat {x} _ {t}) ^ {\top} \right]
$$
$$
\hat {\boldsymbol {H}} _ {t} = \frac {1}{t - 1} \sum_ {\tau = 1} ^ {t - 1} \mathbb {E} _ {a \sim p _ {t} ^ {A _ {\tau}}} \left[ \begin{array}{c c} a a ^ {\top} & a \\ a ^ {\top} & 1 \end{array} \right] = \left[ \begin{array}{c c} \hat {H} _ {t} + \hat {x} _ {t} \hat {x} _ {t} ^ {\top} & \hat {x} _ {t} \\ \hat {x} _ {t} ^ {\top} & 1 \end{array} \right]
$$
$$
\hat {\Sigma} _ {t} = \hat {H} _ {t} + \beta_ {t} I
$$
$$
\hat {\boldsymbol {\Sigma}} _ {t} = \hat {\boldsymbol {H}} _ {t} + \beta_ {t} \boldsymbol {I} = \left[ \begin{array}{c c} \hat {\boldsymbol {\Sigma}} _ {t} + \hat {x} _ {t} \hat {x} _ {t} ^ {\top} & \hat {x} _ {t} \\ \hat {x} _ {t} ^ {\top} & 1 + \beta_ {t} \end{array} \right]
$$
$$
x _ {t} = \mathbb {E} _ {\mathcal {A} \sim \mathcal {D}} \mathbb {E} _ {a \sim p _ {t} ^ {A}} [ a ]
$$
$$
H _ {t} = \mathbb {E} _ {\mathcal {A} \sim \mathcal {D}} \mathbb {E} _ {a \sim p _ {t} ^ {\mathcal {A}}} \left[ (a - \hat {x} _ {t}) (a - \hat {x} _ {t}) ^ {\top} \right]
$$
$$
\boldsymbol {H} _ {t} = \mathbb {E} _ {\mathcal {A} \sim \mathcal {D}} \mathbb {E} _ {a \sim p _ {t} ^ {\mathcal {A}}} \left[ \begin{array}{c c} a a ^ {\top} & a \\ a ^ {\top} & 1 \end{array} \right]
$$
B Auxiliary Lemmas
Lemma 5 (FTRL regret bound, Lemma 18 of Dann et al. [2023a]). Let  $\Omega \subset \mathbb{R}^d$  be a convex set,  $g_1,\ldots ,g_T\in \mathbb{R}^d$ , and  $\eta_{1},\dots ,\eta_{T} > 0$ . Then the FTRL update
$$
w _ {t} = \underset {w \in \Omega} {\operatorname {a r g m i n}} \left\{\left\langle w, \sum_ {\tau = 1} ^ {t - 1} g _ {\tau} \right\rangle + \frac {1}{\eta_ {t}} \psi (w) \right\}
$$
ensures for any  $u\in \Omega$  and  $\eta_0 > 0$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \langle w _ {t} - u, g _ {t} \rangle \\ \leq \underbrace {\frac {\psi (u) - \min _ {w \in \Omega} \psi (w)}{\eta_ {0}} + \sum_ {t = 1} ^ {T} (\psi (u) - \psi (w _ {t})) \left(\frac {1}{\eta_ {t}} - \frac {1}{\eta_ {t - 1}}\right)} _ {\text {P e n a l t y}} + \underbrace {\sum_ {t = 1} ^ {T} \left(\max _ {w \in \Omega} \langle w _ {t} - w , g _ {t} \rangle - \frac {D _ {\psi} (w , w _ {t})}{\eta_ {t}}\right)} _ {\text {S t a b i l i t y}}. \\ \end{array}
$$
When  $\eta_0, \eta_1, \ldots, \eta_T$  is non-increasing, the penalty term can further be upper bounded by
$$
\text {P e n a l t y} \leq \frac {\psi (u) - \min  _ {w \in \Omega} \psi (w)}{\eta_ {T}}.
$$
Lemma 6 (Bernstein's inequality). Let  $X_{1}, \dots, X_{n}$  be iid random variables; let  $\mathbb{E}[X]$  be the expectation and  $\operatorname{Var}(X)$  be the variance of these random variables. If for any  $i$ ,  $|X_{i} - \mathbb{E}[X_{i}]| \leq R$ , then with probability of at least  $1 - \delta$ ,
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} - \mathbb {E} [ X ] \right| \leq \sqrt {\frac {4 \operatorname {V a r} (X) \log \frac {2}{\delta}}{n}} + \frac {4 R \log \frac {2}{\delta}}{3 n}.
$$
Lemma 7 (Hoeffding's inequality). Let  $X_{1}, \dots, X_{n}$  be iid random variables; let  $a \leq X_{i} \leq b$  and let  $\mathbb{E}[X]$  be the expectation. Then with probability of at least  $1 - \delta$ ,
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} - \mathbb {E} [ X ] \right| \leq (b - a) \sqrt {\frac {1}{2 n} \log (\frac {2}{\delta})}
$$
Given  $F(X) = -\log \det(X)$ ,  $D^2 F(X) = X^{-1} \otimes X^{-1}$  where  $\otimes$  is the Kronecker product. For any matrix  $A = \left[ \begin{array}{cccc} a_1 & a_2 & \dots & a_n \end{array} \right]$ , let  $\operatorname{vec}(A) = \begin{bmatrix} a_1 \\ \vdots \\ a_n \end{bmatrix}$  which vectorizes matrix
$A$  to a column vector by stacking the columns  $A$ . The second order directional derivative for  $F$  is  $D^{2}F(X)[A,A] = \operatorname{vec}(A)^{T}\left(X^{-1}\otimes X^{-1}\right)\operatorname{vec}(A) = \operatorname{Tr}(A^{\top}X^{-1}AX^{-1})$ . We define  $\| A\|_{\nabla^{2}F(X)} = \sqrt{\operatorname{Tr}(A^{\top}X^{-1}AX^{-1})}$  and  $\| A\|_{\nabla^{-2}F(X)} = \sqrt{\operatorname{Tr}(A^{\top}XAX)}$ . It is a pseudo-norm and more discussion can be found in Appendix D of Zimmert et al. [2022]. In the following analysis, we will only use one property of this pseudo-norm shown below which is similar to the Holder inequality.
Lemma 8. For any two symmetric matrices  $A, B$  and positive definite matrix  $X$ ,
$$
\langle A, B \rangle \leq \| A \| _ {\nabla^ {2} F (X)} \| B \| _ {\nabla^ {- 2} F (X)}
$$
Proof. Since  $(X\otimes X)^{-1} = X^{-1}\otimes X^{-1}$ , from Holder inequality, we have
$$
\langle A, B \rangle = \left\langle \operatorname {v e c} (A), \operatorname {v e c} (B) \right\rangle \leq \| \operatorname {v e c} (A) \| _ {X ^ {- 1} \otimes X ^ {- 1}} \| \operatorname {v e c} (B) \| _ {(X ^ {- 1} \otimes X ^ {- 1}) ^ {- 1}} = \| A \| _ {\nabla^ {2} F (X)} \| B \| _ {\nabla^ {- 2} F (X)}
$$
C Concentration Inequalities
The goal of this section is to show Lemma 18 and Lemma 19, which are key to bound the bias term. We first introduce a useful lemma from Dai et al. [2023], which will be used later to prove our concentration bounds.
C.1 General Concentration Inequalities
Lemma 9 (Lemma A.4 in Dai et al. [2023]). Let  $H_{1}, H_{2}, \ldots, H_{n}$  be i.i.d. PSD matrices such that  $\mathbb{E}[H_i] = H$ ,  $H_{i} \preceq I$  almost surely and  $H \succeq \frac{1}{dn} \log \frac{d}{\delta} I$ . Then with probability  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} - H \succeq - \sqrt {\frac {d}{n} \log \frac {d}{\delta}} H ^ {1 / 2}
$$
Corollary 10. Let  $H_1, H_2, \ldots, H_n$  be i.i.d. PSD matrices such that  $\mathbb{E}[H_i] = H$  and  $H_i \preceq cI$  almost surely for some positive constant  $c$ . Let  $\hat{H} = \frac{1}{n}\sum_{i=1}^{n}H_i$ , then with probability  $1 - \delta$ ,
$$
\hat {H} + \frac {3 c}{2} \cdot \frac {d}{n} \log \left(\frac {d}{\delta}\right) I \succeq \frac {1}{2} H \tag {9}
$$
Proof. A simple corollary of Lemma 9 under the condition of Lemma 9 is that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} - H \succeq - \sqrt {\frac {d}{n} \log \frac {d}{\delta}} H ^ {1 / 2} \succeq - \frac {1}{2} H - \frac {d}{2 n} \log \left(\frac {d}{\delta}\right) I \\ \Rightarrow \frac {1}{n} \sum_ {i = 1} ^ {n} H _ {i} + \frac {d}{2 n} \log \left(\frac {d}{\delta}\right) I \succeq \frac {1}{2} H, \tag {10} \\ \end{array}
$$
where we use that  $H^{\frac{1}{2}} \preceq \frac{k}{2} H + \frac{1}{2k}$  for any  $k > 0$ .
Now consider the condition of this corollary. We first consider the case where  $\frac{d}{n}\log \left(\frac{d}{\delta}\right)\leq 1$ . In this case, we apply Eq. (10) with  $H_{i}^{\prime} = \frac{1}{2c} H_{i} + \frac{d}{2n}\log \left(\frac{d}{\delta}\right)I$ , which satisfies the condition for Eq. (10) to hold. This gives
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {1}{2 c} H _ {i} + \frac {d}{2 n} \log \left(\frac {d}{\delta}\right) I\right) + \frac {d}{2 n} \log \left(\frac {d}{\delta}\right) I \succeq \frac {1}{2} \left(\frac {1}{2 c} H + \frac {d}{2 n} \log \left(\frac {d}{\delta}\right) I\right) \\ \Rightarrow \hat {H} + \frac {3 c}{2} \cdot \frac {d}{n} \log \left(\frac {d}{\bar {\delta}}\right) I \succeq \frac {1}{2} H \\ \end{array}
$$
with probability at least  $1 - \delta$ . When  $\frac{d}{n}\log \left(\frac{d}{\delta}\right) > 1$ . Eq. (9) is trivial because  $\frac{1}{2} H \preceq \frac{c}{2} I \preceq \frac{c}{2} \cdot \frac{d}{n}\log \left(\frac{d}{\delta}\right)I$ .
C.2 Concentration Inequalities under a Fixed Policy  $p$
In this subsection, we establish concentration bounds for a fixed policy  $p$  (with  $p^{\mathcal{A}} \in \Delta(\mathcal{A})$  denoting the action distribution it uses over  $\mathcal{A}$ ) over i.i.d. contexts. The results in this subsection are preparation for Appendix C.3 where we take union bounds over policies.
The setting and notation to be used in this subsection are defined in Definition 11.
Definition 11. Let  $\{\mathcal{A}_1,\dots ,\mathcal{A}_n\}$  be i.i.d. context samples drawn from  $D$ . Let  $\hat{D}$  be the uniform distribution over  $\{\mathcal{A}_1,\dots ,\mathcal{A}_n\}$ .
Over this set of context samples, define for any policy  $p$
$$
x (p) = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ].
$$
$$
\hat {x} (p) = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ].
$$
$$
H (p) = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x} (p)) (a - \hat {x} (p)) ^ {\top} \right],
$$
$$
\hat {H} (p) = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x} (p)) (a - \hat {x} (p)) ^ {\top} \right],
$$
$$
\boldsymbol {H} (p) = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ \boldsymbol {a a} ^ {\top} \right],
$$
$$
\hat {\boldsymbol {H}} (p) = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ \boldsymbol {a a} ^ {\top} \right],
$$
$$
\hat {\Sigma} (p) = \hat {H} (p) + \beta I,
$$
$$
\hat {\boldsymbol {\Sigma}} (p) = \hat {\boldsymbol {H}} (p) + \beta \boldsymbol {I},
$$
where  $\beta = \frac{5d\log(6d / \delta)}{n}$
Lemma 12. Under the setting of Definition 11, for any fixed  $p$ , with probability at least  $1 - \delta$
$$
\hat {H} (p) + \frac {4 d \log (6 d / \delta)}{n} I \succeq \frac {1}{2} H (p),
$$
$$
\hat {\boldsymbol {H}} (p) + \frac {3 d \log (d / \delta)}{n} \boldsymbol {I} \succeq \frac {1}{2} \boldsymbol {H} (p).
$$
Proof. In this proof, we use  $\hat{x}, x, \hat{H}, H, \hat{H}, H$  to denote  $\hat{x}(p), x(p), \hat{H}(p), H(p), \hat{H}(p), H(p)$  since  $p$  is fixed throughout the proof.
Since  $\| a \| \leq 1$ ,  $H \preceq 2I$  and  $\hat{H} \preceq 2I$ . Thus, we can directly apply Corollary 10 with  $c = 2$  to get with probability  $1 - \frac{\delta}{3}$
$$
\hat {\boldsymbol {H}} + \frac {3 d \log (3 d / \delta)}{n} \boldsymbol {I} \succeq \frac {1}{2} \boldsymbol {H}.
$$
To prove the first inequality, we first decompose  $H$  and  $\tilde{H}$
$$
\begin{array}{l} H = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x}) (a - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x + x - \hat {x}) (a - x + x - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right] + (x - \hat {x}) (x - \hat {x}) ^ {\top} \quad (\text {b e c a u s e} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} (a - x) = 0) \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} \hat {H} = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x}) (a - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x + x - \hat {x}) (a - x + x - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right] - (x - \hat {x}) (x - \hat {x}) ^ {\top} \\ \end{array}
$$
$$
\left(\text {b e c a u s e} \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} (a - x) = \hat {x} - x\right) \tag {12}
$$
From Hoeffding inequality (Lemma 7) and union bound, with probability  $1 - \frac{\delta}{3}$ , for all  $k \in [d]$ , we have
$$
\left| \mathrm {e} _ {k} ^ {\top} x - \mathrm {e} _ {k} ^ {\top} \hat {x} \right| \leq \sqrt {\frac {1}{2 n} \log \left(\frac {6 d}{\delta}\right)},
$$
which implies that  $\mathrm{e}_k^\top (x - \hat{x})(x - \hat{x})^\top \mathrm{e}_k\leq \frac{1}{2n}\log (\frac{6d}{\delta})$  for all  $k$ , and thus
$$
(x - \hat {x}) (x - \hat {x}) ^ {\top} \preceq \frac {1}{2 n} \log \left(\frac {6 d}{\delta}\right) I. \tag {13}
$$
By directly applying Corollary 10 with  $c = 2$ , we get with probability at least  $1 - \frac{\delta}{3}$
$$
\mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right] + \frac {3 d \log (3 d / \delta)}{n} I \succeq \frac {1}{2} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right]
$$
Further using Eq. (11), Eq. (12) and Eq. (13), we get with probability at least  $1 - \frac{2\delta}{3}$ ,
$$
\hat {H} + \frac {4 d \log (6 d / \delta)}{n} I \succeq \frac {1}{2} H
$$
Taking union bound for both inequality finishes the proof.
Lemma 13. Under the setting of Definition 11, for any fixed policy  $p$ , with probability at least  $1 - \mathcal{O}(\delta)$ ,
$$
\| x (p) - \hat {x} (p) \| _ {\hat {\Sigma} (p) ^ {- 1}} ^ {2} \leq \mathcal {O} \left(\frac {d \log (d / \delta)}{n}\right)
$$
Proof. In this proof, we use  $\hat{x}, x, \hat{H}, H, \hat{H}, H, \hat{\Sigma}, \hat{\Sigma}$  to denote  $\hat{x}(p), x(p), \hat{H}(p), H(p), \hat{H}(p), H(p), \hat{\Sigma}(p), \hat{\Sigma}(p)$  since  $p$  is fixed throughout the proof.
We first rewrite  $H$ .
$$
\begin{array}{l} H = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x}) (a - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x + x - \hat {x}) (a - x + x - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right] + (x - \hat {x}) (x - \hat {x}) ^ {\top} \quad (\text {b e c a u s e} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} (a - x) = 0) \tag {14} \\ \end{array}
$$
To simplify analysis, we perform diagonalization. Suppose that  $\mathbb{E}_{\mathcal{A}\sim D}\mathbb{E}_{a\sim p^{\mathcal{A}}}[(a - x)(a - x)^{\top}]$  admits the following eigen-decomposition:
$$
\mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ (a - x) (a - x) ^ {\top} ] = V \Lambda V ^ {\top}
$$
where  $V$  is an orthogonal matrix and  $\Lambda$  is a diagonal matrix. By Lemma 12 and the definition of  $\beta$  in Definition 11, we have with probability  $1 - \delta$ ,
$$
\hat {\Sigma} \succeq \frac {1}{2} H + \rho I \succeq \frac {1}{2} V \Lambda V ^ {\top} + \rho I
$$
with some  $\rho = \Theta\left(\frac{d\log(d / \delta)}{n}\right)$ , where the second inequality is by Eq. (14). Thus,
$$
\begin{array}{l} \left\| x - \hat {x} \right\| _ {\hat {\Sigma} ^ {- 1}} ^ {2} = (x - \hat {x}) ^ {\top} \hat {\Sigma} ^ {- 1} (x - \hat {x}) \\ \leq \left(x - \hat {x}\right) ^ {\top} \left(\frac {1}{2} V \Lambda V ^ {\top} + \rho I\right) ^ {- 1} (x - \hat {x}) \\ = (\hat {x} - x) ^ {\top} V \left(\frac {1}{2} \Lambda + \rho I\right) ^ {- 1} V ^ {\top} (\hat {x} - x). \\ \end{array}
$$
$$
\Delta_ {k} = \mathrm {e} _ {k} ^ {\top} V ^ {\top} (\hat {x} - x) = \frac {1}{n} \sum_ {i = 1} ^ {n} \underbrace {\mathrm {e} _ {k} ^ {\top} V ^ {\top} \mathbb {E} _ {a \sim p ^ {\mathcal {A}} i} [ a ]} _ {\text {D e f i n e a s} Z _ {k} ^ {(i)}} - \underbrace {\mathrm {e} _ {k} ^ {\top} V ^ {\top} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ]} _ {\text {D e f i n e a s} Z _ {k}}
$$
Since  $\mathbb{E}_{\mathcal{A}_i\sim D}\left[Z_k^{(i)}\right] = Z_k$ , by Bernstein's inequality, with probability at least  $1 - \delta$ , we have
$$
\left| \Delta_ {k} \right| \leq \mathcal {O} \left(\sqrt {\frac {\operatorname {V a r} \left(Z _ {k} ^ {(i)}\right) \log (d / \delta)}{n}} + \frac {\log (d / \delta)}{n}\right) \tag {15}
$$
for all  $k$ , where
$$
\mathrm {V a r} (Z _ {k} ^ {(i)}) = \mathbb {E} _ {\mathcal {A} \sim D} \left[ \left(\mathrm {e} _ {k} ^ {\top} V ^ {\top} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ] - \mathrm {e} _ {k} ^ {\top} V ^ {\top} x\right) ^ {2} \right].
$$
$$
\begin{array}{l} \Lambda_ {k k} = \mathrm {e} _ {k} ^ {\top} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ V ^ {\top} (a - x) (a - x) ^ {\top} V ] \mathrm {e} _ {k} \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ \left(\mathrm {e} _ {k} ^ {\top} V ^ {\top} a - \mathrm {e} _ {k} ^ {\top} V ^ {\top} x\right) ^ {2} \right]. \\ \end{array}
$$
From Jensen's inequality,
$$
\Lambda_ {k k} = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ \left(\mathrm {e} _ {k} ^ {\top} V ^ {\top} a - \mathrm {e} _ {k} ^ {\top} V ^ {\top} x\right) ^ {2} \right] \geq \mathbb {E} _ {\mathcal {A} \sim D} \left[ \left(\mathrm {e} _ {k} ^ {\top} V ^ {\top} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ] - \mathrm {e} _ {k} ^ {\top} V ^ {\top} x\right) ^ {2} \right] = \operatorname {V a r} \left(Z _ {k} ^ {(i)}\right)
$$
$$
\begin{array}{l} \left\| x - \hat {x} \right\| _ {\hat {\Sigma} ^ {- 1}} ^ {2} \leq (\hat {x} - x) ^ {\top} V \left(\frac {1}{2} \Lambda + \rho I\right) ^ {- 1} V ^ {\top} (\hat {x} - x) \\ = \sum_ {k = 1} ^ {d} \frac {\left(\Delta_ {k}\right) ^ {2}}{\frac {1}{2} \Lambda_ {k k} + \rho} \\ \leq \mathcal {O} \left(\frac {\log (d / \delta)}{n} \sum_ {k = 1} ^ {d} \frac {\operatorname {V a r} \left(Z _ {k} ^ {(i)}\right) + \frac {\log (d / \delta)}{n}}{\Lambda_ {k k} + \rho}\right) \tag {byEq.(15)} \\ \leq \mathcal {O} \left(\frac {d \log (d / \delta)}{n}\right). \quad (\Lambda_ {k k} \geq \operatorname {V a r} (Z _ {k} ^ {(i)}) \text {a n d} \rho = \Theta (\frac {d \log (d / \delta)}{n})) \\ \end{array}
$$
Lemma 14. Under the setting of Definition 11, for any fixed policy  $p$ , with probability at least  $1 - \mathcal{O}(\delta)$ ,
$$
\left\| \left(\hat {\Sigma} (p) - H (p)\right) y \right\| _ {\hat {\Sigma} (p) ^ {- 1}} ^ {2} \leq \mathcal {O} \left(\frac {d \log (d / \delta)}{n}\right)
$$
for any  $y\in \mathbb{B}_2^d$
First, we re-write  $H$  and  $\hat{H}$ :
$$
\begin{array}{l} H = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - \hat {x}) (a - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x + x - \hat {x}) (a - x + x - \hat {x}) ^ {\top} \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right] + (x - \hat {x}) (x - \hat {x}) ^ {\top} \quad (\text {b e c a u s e} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} (a - x) = 0) \tag {16} \\ \end{array}
$$
$$
(b e c a u s e \mathbb {E} _ {\mathcal {A} \sim \hat {D}} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} (a - x) = \hat {x} - x)
$$
Then, by definition (in Definition 11) and the calculation above,
$$
\hat {\Sigma} - H
$$
$$
= \hat {H} - H + \beta I
$$
$$
= \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {a \sim p ^ {\mathcal {A}} i} \left[ (a - x) (a - x) ^ {\top} \right] - \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ (a - x) (a - x) ^ {\top} \right]} _ {\mathrm {d e f i n e t h i s a s} \Gamma} - 2 (x - \hat {x}) (x - \hat {x}) ^ {\top} + \beta I.
$$
Using  $\| a + b + c\| ^2\leq 3\| a\| ^2 +3\| b\| ^2 +3\| c\| ^2$  , we have