SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| (\hat {\Sigma} - H) y \| _ {\hat {\Sigma} ^ {- 1}} ^ {2} \leq 3 \| \Gamma y \| _ {\hat {\Sigma} ^ {- 1}} ^ {2} + 1 2 \| (x - \hat {x}) (x - \hat {x}) ^ {\top} y \| _ {\hat {\Sigma} ^ {- 1}} ^ {2} + \beta^ {2} \| y \| _ {\hat {\Sigma} ^ {- 1}} ^ {2} \\ \leq 3 \| \Gamma y \| _ {\hat {\Sigma} - 1} ^ {2} + 1 2 \| x - \hat {x} \| _ {\hat {\Sigma} - 1} ^ {2} + \mathcal {O} (\beta). \tag {17} \\ \end{array}
$$
The second and third term are bounded by  $\mathcal{O}\left(\frac{d\log(d / \delta)}{n}\right)$  using Lemma 13 and the definition of  $\beta$  with probability at least  $1 - \mathcal{O}(\delta)$ . Below, we further deal with the first term. To simplify analysis, we perform diagonalization. Suppose that  $\mathbb{E}_{\mathcal{A}\sim D}\mathbb{E}_{a\sim p^{\mathcal{A}}}[(a - x)(a - x)^{\top}]$  admits the following eigen-decomposition:
where  $V$  is an orthogonal matrix and  $\Lambda$  is a diagonal matrix. Then
$$
\left\| \Gamma y \right\| _ {\hat {\Sigma} ^ {- 1}} ^ {2} = y ^ {\top} \Gamma \hat {\Sigma} ^ {- 1} \Gamma y = \left(V ^ {\top} y\right) ^ {\top} \left(V ^ {\top} \Gamma V\right) \left(V ^ {\top} \hat {\Sigma} V\right) ^ {- 1} \left(V ^ {\top} \Gamma V\right) \left(V ^ {\top} y\right). \tag {18}
$$
Below, we further deal with the  $V^{\top}\Gamma V$  and  $V^{\top}\Lambda V$  terms in Eq. (18). By Lemma 12, with probability at least  $1 - \delta$ ,
$$
\hat {\Sigma} \succeq \frac {1}{2} H + \rho I \succeq \frac {1}{2} V \Lambda V ^ {\top} + \rho I,
$$
for some  $\rho = \Theta\left(\frac{d\log(d / \delta)}{n}\right)$ , where we use Eq. (16) in the second inequality. Therefore,
$$
V ^ {\top} \hat {\Sigma} V \succeq \frac {1}{2} \Lambda + \rho I. \tag {19}
$$
Next, denote  $\Delta = V^{\top}\Gamma V$ . By definition, it can be written as the following:
$$
\Delta = \frac{1}{n}\sum_{i = 1}^{n}\underbrace{\mathbb{E}_{a\sim p^{\mathcal{A}}i}\left[V^{\top}(a - x)(a - x)^{\top}V\right]}_{\text{defining this as $\Lambda^{(i)}}} - \underbrace{\mathbb{E}_{\mathcal{A}\sim D}\mathbb{E}_{a\sim p^{\mathcal{A}}}\left[V^{\top}(a - x)(a - x)^{\top}V\right]}_{= \Lambda}
$$
with  $\Lambda^{(i)}$  being i.i.d. samples with mean  $\mathbb{E}[\Lambda^{(i)}] = \Lambda$ . While these are  $d\times d$  matrices, we will apply concentration inequalities to individual entries.
Let  $\lambda_{ikh} = \mathrm{e}_k^\top \Lambda^{(i)}\mathrm{e}_h$  be the  $(k,h)$ -th entry of  $\Lambda^{(i)}$ . Notice that  $\mathbb{E}[\lambda_{ikh}] = \mathrm{e}_k^\top \Lambda \mathrm{e}_h = \Lambda_{kh}$ , the  $(k,h)$ -th entry of  $\Lambda$ .
By Bernstein's inequality, with probability at least  $1 - \delta$ , we have
$$
\left| \Delta_ {k h} \right| = \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\lambda_ {i k h} - \Lambda_ {k h}\right) \right| \leq \mathcal {O} \left(\sqrt {\frac {\operatorname {V a r} \left(\lambda_ {i k h}\right) \log (d / \delta)}{n}} + \frac {\log (d / \delta)}{n}\right). \tag {20}
$$
With the manipulations and notations above, we continue to bound Eq. (18) by
$$
\begin{array}{l} \left\| \Gamma y \right\| _ {\hat {\Sigma} ^ {- 1}} ^ {2} = y ^ {\prime \top} \Delta \left(V ^ {\top} \hat {\Sigma} V\right) ^ {- 1} \Delta y ^ {\prime} \quad (\text {l e t} y ^ {\prime} = V ^ {\top} y) \\ \leq 2 y ^ {\prime \top} \Delta (\Lambda + \rho I) ^ {- 1} \Delta y ^ {\prime} \quad \text {(b y E q . (1 9))} \\ \leq 2 \operatorname {T r} \left(\Delta (\Lambda + \rho I) ^ {- 1} \Delta\right) \\ \end{array}
$$
By direct expansion and the fact that  $\Lambda$  is diagonal,
$$
\begin{array}{l} \operatorname {T r} \left(\Delta (\Lambda + \rho I) ^ {- 1} \Delta\right) = \sum_ {k = 1} ^ {d} \left(\Delta (\Lambda + \rho I) ^ {- 1} \Delta\right) _ {k k} \\ = \sum_ {k = 1} ^ {d} \sum_ {h = 1} ^ {d} \frac {\Delta_ {k h} \Delta_ {h k}}{\Lambda_ {h h} + \rho} \\ \leq \mathcal {O} \left(\sum_ {k = 1} ^ {d} \sum_ {h = 1} ^ {d} \frac {1}{\Lambda_ {h h} + \rho} \left(\frac {\operatorname {V a r} \left(\lambda_ {i k h}\right) \log (d / \delta)}{n} + \frac {\log^ {2} (d / \delta)}{n ^ {2}}\right)\right) (byEq.(20)) \\ \leq \mathcal {O} \left(\sum_ {k = 1} ^ {d} \sum_ {h = 1} ^ {d} \frac {1}{\Lambda_ {h h} + \rho} \frac {\mathbb {E} \left(\lambda_ {i k h} ^ {2}\right) \log (d / \delta)}{n} + \frac {d ^ {2} \log^ {2} (d / \delta)}{\rho n ^ {2}}\right) (21) \\ \end{array}
$$
By definition,
$$
\lambda_ {i k h} = \mathbb {E} _ {a \sim p ^ {A _ {i}}} \left[ \mathrm {e} _ {k} V ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {h} \right]
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {d} \lambda_ {i k h} ^ {2} \leq \mathbb {E} _ {a \sim p ^ {\mathcal {A}} _ {i}} \left[ \sum_ {k = 1} ^ {d} \left(\mathrm {e} _ {k} V ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {h}\right) ^ {2} \right] \\ = \mathbb {E} _ {a \sim p ^ {A _ {i}}} \left[ \sum_ {k = 1} ^ {d} \mathrm {e} _ {h} ^ {\top} V ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {k} \mathrm {e} _ {k} ^ {\top} V ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {h} \right] \\ = \mathbb {E} _ {a \sim p ^ {A _ {i}}} \left[ \mathrm {e} _ {h} ^ {\top} V ^ {\top} (a - x) (a - x) ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {h} \right] \\ \leq \mathbb {E} _ {a \sim p ^ {\mathcal {A} _ {i}}} \left[ \mathrm {e} _ {h} ^ {\top} V ^ {\top} (a - x) (a - x) ^ {\top} V \mathrm {e} _ {h} \right] \\ = \lambda_ {i h h} \\ \end{array}
$$
and  $\sum_{k=1}^{d} \mathbb{E}[\lambda_{ikh}^2] \leq \mathbb{E}[\lambda_{ihh}] = \Lambda_{hh}$ . Continuing from Eq. (21) and using that  $\rho = \Theta\left(\frac{d\log(d / \delta)}{n}\right)$ ,
$$
\operatorname {T r} \left(\Delta (\Lambda + \rho I) ^ {- 1} \Delta\right) \leq \mathcal {O} \left(\sum_ {h = 1} ^ {d} \frac {\Lambda_ {h h} \log (d / \delta)}{(\Lambda_ {h h} + \rho) n} + \frac {d ^ {2} \log^ {2} (d / \delta)}{n ^ {2}}\right) \leq \mathcal {O} \left(\frac {d \log (d / \delta)}{n}\right).
$$
This gives a bound on  $\| \Gamma y \|_{\hat{\Sigma}^{-1}}^2$  and finishes the proof after combining Eq. (17).
C.3 Union Bound over Policies
In Lemma 12, Lemma 13, and Lemma 14, we have obtained the desired concentration inequalities under a fixed policy  $p$ . In this subsection, we proceed to take union bound over all policies that are possibly used by Algorithm 1.
The set of policies that could be generated by Algorithm 1 is the following:
$$
\mathbf {P} = \left\{p: \widehat {\operatorname {C o v}} (p ^ {\mathcal {A}}) = \operatorname * {a r g m i n} _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A}}} \left\{\langle \boldsymbol {H}, \boldsymbol {Z} \rangle + F (\boldsymbol {H}) \right\}, \text {f o r} \boldsymbol {Z} \in \mathcal {Z} \right\}
$$
where  $\mathcal{Z} = [-T^2, T^2]^{(d+1) \times (d+1)} \cap \mathbb{S}$  with  $\mathbb{S}$  denoting the set of symmetric matrices. To see this, notice that Algorithm 1 at round  $t$  corresponds to the policy defined above with  $\boldsymbol{Z} = \eta_t \sum_{s=1}^{t-1} (\hat{\gamma}_s - \alpha_s \hat{\Sigma}_s^{-1})$ .
Our goal is to construct a  $\epsilon$ -cover  $\mathbf{P}'$  so that every policy  $p \in \mathbf{P}$  can find a policy  $p' \in \mathbf{P}'$  making  $-\epsilon I \preceq \widehat{\operatorname{Cov}}(p^{\mathcal{A}}) - \widehat{\operatorname{Cov}}(p'^{\mathcal{A}}) \preceq \epsilon I$  on every action set  $\mathcal{A}$ . The size of such a cover is bounded in the Proposition below.
Lemma 15. There exists an  $\epsilon$ -cover  $\mathbf{P}'$  of  $\mathbf{P}$  with size  $\log |\mathbf{P}'| = \mathcal{O}\left(d^{2}\log \frac{d}{\epsilon}\right)$  such that for any  $p\in \mathbf{P}$ , there exists an  $p'\in \mathbf{P}'$  satisfying
$$
\left\| \widehat {\operatorname {C o v}} (p ^ {\mathcal {A}}) - \widehat {\operatorname {C o v}} (p ^ {\prime \mathcal {A}}) \right\| _ {F} \leq \epsilon
$$
for all  $\mathcal{A}$ .
Proof. It is straightforward to construct an  $\frac{\epsilon}{4}$ -cover  $\mathcal{C}$  for  $\mathcal{Z} = [-T^2, T^2]^{(d+1) \times (d+1)} \cap \mathbb{S}$  in Frobenius norm with size  $|\mathcal{C}| = \left(\frac{24(d+1)^2}{\epsilon}\right)^{(d+1)^2}$  (Exercise 27.6 of Lattimore and Szepesvári [2020]). Now define  $\mathbf{P}'$  as
$$
\mathbf {P} ^ {\prime} = \left\{p: \widehat {\operatorname {C o v}} (p ^ {\mathcal {A}}) = \underset {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A}}} {\operatorname {a r g m i n}} \left\{\langle \boldsymbol {H}, \boldsymbol {Z} \rangle + F (\boldsymbol {H}) \right\}, \text {f o r} \boldsymbol {Z} \in \mathcal {C} \right\} \tag {22}
$$
Below, we show that this is a  $\epsilon$ -cover for  $\mathbf{P}$ .
Consider two policies  $p_1$  and  $p_2$  defined as the following:
$$
\widehat {\operatorname {C o v}} (p _ {1} ^ {\mathcal {A}}) = \underset {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A}}} {\operatorname {a r g m i n}} \left\{\langle \boldsymbol {H}, \boldsymbol {Z} _ {1} \rangle + F (\boldsymbol {H}) \right\}
$$
$$
\widehat {\operatorname {C o v}} \left(p _ {2} ^ {\mathcal {A}}\right) = \underset {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A}}} {\operatorname {a r g m i n}} \left\{\left\langle \boldsymbol {H}, \boldsymbol {Z} _ {2} \right\rangle + F (\boldsymbol {H}) \right\}
$$
with  $\| Z_1 - Z_2\| _F\leq \frac{\epsilon}{4}$ . Consider an arbitrary  $\mathcal{A}$  and define  $\pmb {H}_1 = \widehat{\mathrm{Cov}} (p_1^{\mathcal{A}}),\pmb {H}_2 = \widehat{\mathrm{Cov}} (p_2^{\mathcal{A}})$ . Below we show  $\| H_{1} - H_{2}\|_{F}\leq \epsilon$
Since  $F(\pmb{H})$  is convex for  $\pmb{H}$ , from the first-order optimality condition for convex function, we have
$$
\begin{array}{l} \langle \boldsymbol {H} _ {1}, \boldsymbol {Z} _ {1} \rangle + F (\boldsymbol {H} _ {1}) \leq \langle \boldsymbol {H} _ {2}, \boldsymbol {Z} _ {1} \rangle + F (\boldsymbol {H} _ {2}) - D _ {F} (\boldsymbol {H} _ {2}, \boldsymbol {H} _ {1}) \\ = \left\langle \mathbf {H} _ {2}, \mathbf {Z} _ {2} \right\rangle + \left\langle \mathbf {H} _ {2}, \mathbf {Z} _ {1} - \mathbf {Z} _ {2} \right\rangle + F (\mathbf {H} _ {2}) - D _ {F} (\mathbf {H} _ {2}, \mathbf {H} _ {1}) \\ \end{array}
$$
$$
\begin{array}{l} \langle \boldsymbol {H} _ {2}, \boldsymbol {Z} _ {2} \rangle + F (\boldsymbol {H} _ {2}) \leq \langle \boldsymbol {H} _ {1}, \boldsymbol {Z} _ {2} \rangle + F (\boldsymbol {H} _ {1}) - D _ {F} (\boldsymbol {H} _ {1}, \boldsymbol {H} _ {2}) \\ = \left\langle \boldsymbol {H} _ {1}, \boldsymbol {Z} _ {1} \right\rangle + \left\langle \boldsymbol {H} _ {1}, \boldsymbol {Z} _ {2} - \boldsymbol {Z} _ {1} \right\rangle + F (\boldsymbol {H} _ {1}) - D _ {F} (\boldsymbol {H} _ {1}, \boldsymbol {H} _ {2}) \\ \end{array}
$$
Adding up these the two inequalities, we get
$$
2 \min  \left\{D _ {F} \left(\boldsymbol {H} _ {1}, \boldsymbol {H} _ {2}\right), D _ {F} \left(\boldsymbol {H} _ {2}, \boldsymbol {H} _ {1}\right) \right\} \leq D _ {F} \left(\boldsymbol {H} _ {1}, \boldsymbol {H} _ {2}\right) + D _ {F} \left(\boldsymbol {H} _ {2}, \boldsymbol {H} _ {1}\right) \leq \langle \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2}, \boldsymbol {H} _ {2} - \boldsymbol {H} _ {1} \rangle
$$
Since the second order directional derivative for  $F$  is  $D^{2}F(\pmb{H})[\pmb{X},\pmb{X}] = \mathrm{Tr}(\pmb{X}\pmb{H}^{-1}\pmb{X}\pmb{H}^{-1})$  for any symmetric matrix  $\pmb{X}$ , from the Taylor series, there exists  $\pmb{H}'$  that is a line segment between  $\pmb{H}_{1}$  and  $\pmb{H}_{2}$  such that
$$
\begin{array}{l} \left\| \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \right\| _ {\nabla^ {2} F (\boldsymbol {H} ^ {\prime})} ^ {2} = 2 \min  \left\{D _ {F} \left(\boldsymbol {H} _ {1}, \boldsymbol {H} _ {2}\right), D _ {F} \left(\boldsymbol {H} _ {2}, \boldsymbol {H} _ {1}\right) \right\} \leq \langle \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2}, \boldsymbol {H} _ {2} - \boldsymbol {H} _ {1} \rangle \\ \leq \left\| Z _ {1} - Z _ {2} \right\| _ {\nabla^ {- 2} F \left(\boldsymbol {H} ^ {\prime}\right)} \left\| H _ {1} - H _ {2} \right\| _ {\nabla^ {2} F \left(\boldsymbol {H} ^ {\prime}\right)} \tag {Lemma8} \\ \end{array}
$$
Thus we have  $\| H_1 - H_2\|_{\nabla^2 F(H')}\leq \| Z_1 - Z_2\|_{\nabla^{-2}F(H')}$ . Since  $\| a\| _2\leq 1$ ,  $H^{\prime}\preceq 2I$ . The left-hand side and right-hand side can be bounded as follows,
$$
\| \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \| _ {\nabla^ {2} F (\boldsymbol {H} ^ {\prime})} = \sqrt {\operatorname {T r} \left((\boldsymbol {H} _ {1} - \boldsymbol {H} _ {2}) (\boldsymbol {H} ^ {\prime}) ^ {- 1} (\boldsymbol {H} _ {1} - \boldsymbol {H} _ {2}) (\boldsymbol {H} ^ {\prime}) ^ {- 1}\right)} \geq \frac {1}{2} \| \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \| _ {F}
$$
$$
\| \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \| _ {\nabla^ {- 2} F (\boldsymbol {H} ^ {\prime})} = \sqrt {\operatorname {T r} \left((\boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2}) \boldsymbol {H} ^ {\prime} (\boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2}) \boldsymbol {H} ^ {\prime}\right)} \leq 2 \| \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \| _ {F} \leq \frac {\epsilon}{2}
$$
Combining the three inequalities above, we conclude that
$$
\begin{array}{l} \| \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \| _ {F} \leq 2 \| \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \| _ {\nabla^ {2} F (\boldsymbol {H} ^ {\prime})} \leq 2 \| \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \| _ {\nabla^ {- 2} F (\boldsymbol {H} ^ {\prime})} \leq 4 \| \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \| _ {F} \leq \epsilon . \\ - \epsilon \boldsymbol {I} \preceq \boldsymbol {H} _ {1} - \boldsymbol {H} _ {2} \preceq \epsilon \boldsymbol {I}. \\ \end{array}
$$
Lemma 16. Suppose that  $p, p'$  are two policies such that for all action set  $A$ ,
$$
\left\| \widehat {\operatorname {C o v}} \left(p ^ {\mathcal {A}}\right) - \widehat {\operatorname {C o v}} \left(p ^ {\prime \mathcal {A}}\right) \right\| _ {F} \leq \epsilon \tag {23}
$$
Then all quantities defined in Definition 11 under  $p$  and  $p'$  are close. That is,
$$
\left\| x (p) - x \left(p ^ {\prime}\right) \right\| \leq \epsilon \tag {24}
$$
$$
\left\| \hat {x} (p) - \hat {x} \left(p ^ {\prime}\right) \right\| \leq \epsilon \tag {25}
$$
$$
\left\| H (p) - H \left(p ^ {\prime}\right) \right\| _ {F} \leq 7 \epsilon \tag {26}
$$
$$
\left\| \hat {H} (p) - \hat {H} \left(p ^ {\prime}\right) \right\| _ {F} \leq 7 \epsilon \tag {27}
$$
$$
\left\| \boldsymbol {H} (p) - \boldsymbol {H} \left(p ^ {\prime}\right) \right\| _ {F} \leq \epsilon \tag {28}
$$
$$
\left\| \hat {\boldsymbol {H}} (p) - \hat {\boldsymbol {H}} \left(p ^ {\prime}\right) \right\| _ {F} \leq \epsilon \tag {29}
$$
$$
\| \hat {\Sigma} (p) - \hat {\Sigma} \left(p ^ {\prime}\right) \| _ {F} \leq 7 \epsilon \tag {30}
$$
$$
\left\| \hat {\boldsymbol {\Sigma}} (p) - \hat {\boldsymbol {\Sigma}} \left(p ^ {\prime}\right) \right\| _ {F} \leq \epsilon \tag {31}
$$
Proof. Eq. (28) and Eq. (29) are direct consequences of Eq. (23) since  $H(p)$  and  $\hat{H}(p)$  are expectations of  $\widehat{\operatorname{Cov}}(p^{\mathcal{A}})$  over distributions over  $\mathcal{A}$ . Eq. (31) is directly implied by Eq. (29) because  $\hat{\Sigma}(p) = \hat{H}(p) + \beta I$ .
To show Eq. (24) and Eq. (25), observe that by the definition of  $x(p)$  and  $H(p)$ ,
$$
\begin{array}{l} \boldsymbol {H} (p) = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} \left[ \begin{array}{c c} a a ^ {\top} & a \\ a ^ {\top} & 1 \end{array} \right] = \left[ \begin{array}{c c} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a a ^ {\top} ] & \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ] \\ \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ^ {\top} ] & 1 \end{array} \right] \\ = \left[ \begin{array}{c c} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a a ^ {\top} ] & x (p) \\ x (p) ^ {\top} & 1 \end{array} \right] \\ \end{array}
$$
Therefore,  $\| x(p) - x(p')\| \leq \| H(p) - H(p')\| _F\leq \epsilon$  . Similarly,  $\| \hat{x} (p) - \hat{x} (p')\| \leq \| \hat{H} (p) -$ $\hat{\pmb{H}} (p^{\prime})\|_{F}\leq \epsilon$
If remains to show Eq. (26), Eq. (27) and Eq. (30). Next, we show Eq. (26):
$$
\begin{array}{l} H (p) - H \left(p ^ {\prime}\right) \\ = \mathbb {E} _ {\mathcal {A} \sim D} \left[ \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ (a - \hat {x} (p)) (a - \hat {x} (p)) ^ {\top} ] - \mathbb {E} _ {a \sim p ^ {\prime} \mathcal {A}} [ (a - \hat {x} (p ^ {\prime})) (a - \hat {x} (p ^ {\prime})) ^ {\top} ] \right] \\ = \mathbb {E} _ {\mathcal {A} \sim D} \left[ \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a a ^ {\top} ] - \mathbb {E} _ {a \sim p ^ {\prime \mathcal {A}}} [ a a ^ {\top} ] \right] \\ - x (p) \hat {x} (p) ^ {\top} - \hat {x} (p) x (p) ^ {\top} + x \left(p ^ {\prime}\right) \hat {x} \left(p ^ {\prime}\right) ^ {\top} + \hat {x} \left(p ^ {\prime}\right) x \left(p ^ {\prime}\right) ^ {\top} (\text {u s i n g} \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p ^ {\mathcal {A}}} [ a ] = x (p)) \\ + \hat {x} (p) \hat {x} (p) ^ {\top} - \hat {x} \left(p ^ {\prime}\right) \hat {x} \left(p ^ {\prime}\right) ^ {\top} \tag {32} \\ \end{array}
$$
Using the property
$$
\left\| a b ^ {\top} - c d ^ {\top} \right\| _ {F} \leq \left\| a b ^ {\top} - c b ^ {\top} \right\| _ {F} + \left\| c b ^ {\top} - c d ^ {\top} \right\| _ {F} \leq \left\| a - c \right\| \left\| b \right\| + \left\| c \right\| \left\| b - d \right\|
$$
we continue from Eq. (32) and bound
$$
\begin{array}{l} \| H (p) - H \left(p ^ {\prime}\right) \| _ {F} \\ \leq \| \boldsymbol {H} (p) - \boldsymbol {H} \left(p ^ {\prime}\right) \| _ {F} + 2 \left(\| \hat {x} (p) - \hat {x} \left(p ^ {\prime}\right) \| + \| x (p) - x \left(p ^ {\prime}\right) \|\right) + \| \hat {x} (p) - \hat {x} \left(p ^ {\prime}\right) \| + \| \hat {x} (p) - \hat {x} \left(p ^ {\prime}\right) \| \\ \leq 7 \epsilon . \\ \end{array}
$$
Eq. (27) can be shown in the same manner, which further implies Eq. (30) by the definition of  $\hat{\Sigma}(p)$ .
Lemma 17. With probability  $1 - \delta$ , for all  $t = 1, \dots, T$
$$
\hat {H} _ {t} + \frac {5 0 (d + 1) ^ {3} \log (3 T / \delta)}{t - 1} I \succeq \frac {1}{2} H _ {t},
$$
$$
\hat {\boldsymbol {H}} _ {t} + \frac {5 0 (d + 1) ^ {3} \log (3 T / \delta)}{t - 1} \boldsymbol {I} \succeq \frac {1}{2} \boldsymbol {H} _ {t}.
$$
Proof. Notice that  $\hat{H}_t, \hat{H}_t, H_t, H_t$  corresponds to  $\hat{H}(p_t), \hat{H}(p_t), H(p_t), H(p_t)$  defined in Definition 11 with  $n = t - 1$ . To show the lemma, our strategy is to argue the following two facts: 1) the two desired inequalities hold for all policies in the cover  $\mathbf{P}'$  (defined in Eq. (22)) with high probability. This is simply by applying Lemma 12 with an union bound over policies in  $\mathbf{P}'$ . 2)  $p_t$  is sufficiently close to the nearest element in  $\mathbf{P}'$  so the desired inequalities still approximately hold.
By Lemma 15, we can find  $p' \in \mathbf{P}'$  such that for all  $\mathcal{A}$ ,
$$
\left\| \widehat {\operatorname {C o v}} (p _ {t} ^ {\mathcal {A}}) - \widehat {\operatorname {C o v}} (p ^ {\prime \mathcal {A}}) \right\| _ {F} \leq \epsilon .
$$
By Lemma 16, it holds that
$$
\left\| H \left(p _ {t}\right) - H \left(p ^ {\prime}\right) \right\| _ {F} \leq 7 \epsilon , \quad \left\| \hat {H} \left(p _ {t}\right) - \hat {H} \left(p ^ {\prime}\right) \right\| _ {F} \leq 7 \epsilon \tag {33}
$$
$$
\left\| \boldsymbol {H} \left(p _ {t}\right) - \boldsymbol {H} \left(p ^ {\prime}\right) \right\| _ {F} \leq \epsilon , \quad \left\| \hat {\boldsymbol {H}} \left(p _ {t}\right) - \hat {\boldsymbol {H}} \left(p ^ {\prime}\right) \right\| _ {F} \leq \epsilon \tag {34}
$$
On the other hand, using Lemma 12 and union bound, with probability  $1 - \delta$ , we have
$$
\hat {H} \left(p ^ {\prime}\right) + \frac {4 d \log (6 d | \mathbf {P} ^ {\prime} | / \delta)}{n} I \succeq \frac {1}{2} H \left(p ^ {\prime}\right), \tag {35}
$$
$$
\hat {\boldsymbol {H}} \left(p ^ {\prime}\right) + \frac {3 d \log (d | \mathbf {P} ^ {\prime} | / \delta)}{n} \boldsymbol {I} \succeq \frac {1}{2} \boldsymbol {H} \left(p ^ {\prime}\right). \tag {36}
$$
Combining Eq. (35) and Eq. (33), we get
$$
\hat {H} (p _ {t}) + 7 \epsilon I + \frac {4 d \log (6 d | \mathbf {P} ^ {\prime} | / \delta)}{n} I \succeq \hat {H} (p ^ {\prime}) + \frac {4 d \log (6 d | \mathbf {P} ^ {\prime} | / \delta)}{n} I \succeq \frac {1}{2} H (p ^ {\prime}) \succeq \frac {1}{2} H (p _ {t}) - \frac {7}{2} \epsilon I
$$
which implies the first inequality in the lemma by plugging in the choice of  $\epsilon = \frac{1}{T^3}$  and the upper bound of  $\log |\mathbf{P}'|$  in Lemma 16. The second inequality in the lemma can be obtained similarly by combining Eq. (34) and Eq. (36).
Lemma 18. With probability of at least  $1 - \delta$ , for all  $t = 1, \dots, T$ ,
$$
\left\| x _ {t} - \hat {x} _ {t} \right\| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq \mathcal {O} \left(\frac {d ^ {3} \log (d T / \delta)}{t}\right)
$$
Proof. Notice that  $x_{t}, \hat{x}_{t}, \hat{\Sigma}_{t}$  corresponds to  $x(p_{t}), \hat{x}(p_{t}), \hat{\Sigma}(p_{t})$  defined in Definition 11 with  $n = t - 1$ . To show the lemma, our strategy is to argue the following two facts: 1) the two desired inequalities hold for all policies in the cover  $\mathbf{P}'$  with high probability. This is simply by applying Lemma 13 with an union bound over policies in  $\mathbf{P}'$ . 2)  $p_{t}$  is sufficiently close to the nearest element in  $\mathbf{P}'$  so the desired inequalities still approximately hold.
By Lemma 16, we have
$$
\left\| x \left(p ^ {\prime}\right) - x \left(p _ {t}\right) \right\| \leq \epsilon , \quad \left\| \hat {x} \left(p ^ {\prime}\right) - \hat {x} \left(p _ {t}\right) \right\| \leq \epsilon , \quad \left\| \hat {\Sigma} \left(p ^ {\prime}\right) - \hat {\Sigma} \left(p _ {t}\right) \right\| _ {F} \leq 7 \epsilon \tag {37}
$$
$$
\begin{array}{l} \left\| x \left(p _ {t}\right) - \hat {x} \left(p _ {t}\right) \right\| _ {\hat {\Sigma} \left(p _ {t}\right) ^ {- 1}} ^ {2} \\ = \left(\| x (p _ {t}) - \hat {x} (p _ {t}) \| _ {\hat {\Sigma} (p _ {t}) ^ {- 1}} ^ {2} - \| x (p ^ {\prime}) - \hat {x} (p ^ {\prime}) \| _ {\hat {\Sigma} (p ^ {\prime}) ^ {- 1}} ^ {2}\right) + \| x (p ^ {\prime}) - \hat {x} (p ^ {\prime}) \| _ {\hat {\Sigma} (p ^ {\prime}) ^ {- 1}} ^ {2} \\ \leq \left(\| x (p _ {t}) - \hat {x} (p _ {t}) \| _ {\hat {\Sigma} (p _ {t}) ^ {- 1}} ^ {2} - \| x (p ^ {\prime}) - \hat {x} (p ^ {\prime}) \| _ {\hat {\Sigma} (p ^ {\prime}) ^ {- 1}} ^ {2}\right) + \mathcal {O} \left(\frac {d \log (d | \mathbf {P} ^ {\prime} | / \delta)}{t - 1}\right) \\ \end{array}
$$
(by Lemma 13 with an union bound over  $\mathbf{P}'$ )
$$
\begin{array}{l} = \theta_ {t} ^ {\top} \hat {\Sigma} \left(p _ {t}\right) ^ {- 1} \theta_ {t} - \theta^ {\prime \top} \hat {\Sigma} \left(p ^ {\prime}\right) ^ {- 1} \theta^ {\prime} + \mathcal {O} \left(\frac {d \log (d | \mathbf {P} ^ {\prime} | / \delta)}{t - 1}\right) \\ (\text {d e f i n e} \theta_ {t} = x (p _ {t}) - \hat {x} (p _ {t}) \text {a n d} \theta^ {\prime} = x \left(p ^ {\prime}\right) - \hat {x} \left(p ^ {\prime}\right)) \\ \end{array}
$$
$$
\begin{array}{l} = \left(\theta_ {t} - \theta^ {\prime}\right) ^ {\top} \hat {\Sigma} \left(p _ {t}\right) ^ {- 1} \theta_ {t} + \theta^ {\prime \top} \left(\hat {\Sigma} \left(p _ {t}\right) ^ {- 1} - \hat {\Sigma} \left(p ^ {\prime}\right) ^ {- 1}\right) \theta_ {t} + \theta^ {\prime \top} \hat {\Sigma} \left(p ^ {\prime}\right) ^ {- 1} \left(\theta_ {t} - \theta^ {\prime}\right) + \mathcal {O} \left(\frac {d \log (d | \mathbf {P} ^ {\prime} | / \delta)}{t - 1}\right) \\ \leq \left(\theta_ {t} - \theta^ {\prime}\right) ^ {\top} \left(\hat {\Sigma} \left(p _ {t}\right) ^ {- 1} \theta_ {t} + \hat {\Sigma} \left(p ^ {\prime}\right) ^ {- 1} \theta^ {\prime}\right) + \theta^ {\prime \top} \hat {\Sigma} \left(p ^ {\prime}\right) ^ {- 1} \left(\hat {\Sigma} \left(p ^ {\prime}\right) - \hat {\Sigma} \left(p _ {t}\right)\right) \hat {\Sigma} \left(p _ {t}\right) ^ {- 1} \theta_ {t} + \mathcal {O} \left(\frac {d \log (d | \mathbf {P} ^ {\prime} | / \delta)}{t - 1}\right) \\ \end{array}
$$
The first two terms above can be bounded by the order of  $\mathcal{O}(\epsilon t^2)$  by Eq. (37). Using the choice  $\epsilon = \frac{1}{T^3}$  and recalling that  $\log |\mathbf{P}'| = \mathcal{O}(d^2\log (d / \epsilon))$  finishes the proof.
Lemma 19. With probability of at least  $1 - \delta$ , for all  $t = 1,2,\ldots,T$