SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {c o s t} _ {h} ^ {n} (\boldsymbol {\tau}) \leftarrow \frac {1}{M} \langle \Xi_ {n}, \psi_ {h} (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \rangle \quad \text {f o r} \quad \Xi_ {n} \leftarrow \nabla_ {\boldsymbol {\Gamma}} \Phi (\boldsymbol {\Gamma}) | _ {\boldsymbol {\Gamma} = \boldsymbol {\Gamma} _ {n - 1}}
$$
for  $K$  episodes, collect trajectories  $\mathfrak{D}_n = \{\pmb{\tau}_k^n\}_{k=1}^K$ , denote policies run as  $\Pi_n$
6:  $\Gamma_{n}\gets (1 - \gamma_{n})\Gamma_{n - 1} + \gamma_{n}K^{-1}\sum_{k = 1}^{K}\psi (\pmb{\tau}_{k}^{n})$
7: return  $(N + 1)K\Gamma_{N},\cup_{n = 0}^{N}\Pi_{n},\cup_{n = 0}^{N}\mathfrak{D}_{n}$
We define DYNAMICOED as in Algorithm 4. We then have the following generalization of Theorem 3.
Theorem 5 (Full Version of Theorem 3). Let Assumption 7 hold, and assume that we have access to a learner  $\mathbb{A}_{\mathcal{R}}$  satisfying Assumption 8. Fix  $N,K > 0$ . Then, with probability at least  $1 - \delta$ , DYNAMICOED runs for at most  $(N + 1)K$  episodes, and collects a dataset satisfying  $\mathfrak{D} = \{\{\pmb{\tau}_k^n\}_{k=1}^K\}_{n=0}^N$  satisfying
$$
\begin{array}{l} \Phi \left(\frac {1}{K (N + 1)} \sum_ {n = 0} ^ {N} \sum_ {k = 1} ^ {K} \psi \left(\boldsymbol {\tau} _ {k} ^ {n}\right)\right) - \min  _ {\boldsymbol {\Gamma} \in \boldsymbol {\Omega} _ {\psi}} \Phi (\boldsymbol {\Gamma}) \leq \frac {\beta R ^ {2} (\log N + 1)}{2 (N + 1)} + M \cdot \left(C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta} \cdot K ^ {\alpha - 1} \right. \\ \left. + \sqrt {\frac {8 \log (4 N / \delta)}{K}}\right) \\ \end{array}
$$
where  $R = \sup_{\mathbf{\Gamma},\mathbf{\Gamma}'\in \widehat{\Omega}_{\psi}}\| \mathbf{\Gamma} - \mathbf{\Gamma}'\|$
In this work we are particularly interested in the case where  $\psi(\tau) \in S_{+}^{d_{\psi}}$ . We encapsulate this in the following assumption.
Assumption 11 (Matrix Experiment Design). We assume that  $\psi(\tau) \in S_{+}^{d_{\psi}}$  and that, for all  $\tau \in \mathcal{T}$ ,  $\operatorname{tr}(\psi(\tau)) \leq D$  for some  $D > 0$ .
The following corollary instantiates Theorem 3 under Assumption 11 with objective  $\Phi (\Gamma) =$  tr  $(\mathcal{H}(\Gamma +\Gamma_0)^{-1})$  , the objective considered in Algorithm 1.
Corollary 1. Consider the objective
$$
\Phi (\boldsymbol {\Gamma}) = \operatorname {t r} \left(\mathcal {H} \cdot \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right)
$$
and assume that  $\mathcal{H} \succeq 0$  and Assumption 10 holds with  $\alpha = 1/2$  and Assumption 11 holds. Fix  $N, K$ , let  $T := (N + 1)K$ , and consider running Algorithm 4 on this objective and with these choices of  $N$  and  $K$ . Then Algorithm 4 will run for at most  $T$  episodes, and, with probability at least  $1 - \delta$ , will return data satisfying
$$
\begin{array}{l} \operatorname {t r} \left(\mathcal {H} \left(\sum_ {t = 1} ^ {T} \psi (\boldsymbol {\tau} _ {t}) + T \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \leq \frac {1}{T} \cdot \min _ {\boldsymbol {\Gamma} \in \Omega_ {\psi}} \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}) ^ {- 1}\right) + \frac {8 D ^ {4} \| \mathcal {H} \| _ {\mathrm {o p}} \| \boldsymbol {\Gamma} _ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {3}}{T (N + 1)} \\ + \frac {8 D \| \mathcal {H} \| _ {\mathrm {o p}} \| \boldsymbol {\Gamma} _ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {2} (\log^ {1 / 2} \frac {4 T}{\delta} + C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 T}{\delta})}{T \sqrt {K}} \\ \end{array}
$$
C.1 Proof of Theorem 3 and Theorem 5
Lemma C.1 (Lemma C.1 of [56]). Consider running Algorithm 5 with some convex function  $f$  that is  $\beta$ -smooth with respect to some norm  $\| \cdot \|$ , assume that  $y_{n} \in \mathcal{V}$  for some  $\mathcal{V}$  and all  $n$ , and let
Algorithm 5 Approximate Frank-Wolfe
1: input: function to optimize  $f$ , number of iterations to run  $N$ , starting iterate  $x_{1}$
2: for  $t = 1, 2, \dots, N$  do
3: Set  $\gamma_{n}\gets \frac{1}{n + 1}$
4: Choose  $\pmb{y}_n$  to be any point such that
$$
\nabla f (\boldsymbol {z} _ {n}) ^ {\top} \boldsymbol {y} _ {n} \leq \min  _ {\boldsymbol {y} \in \mathcal {Z}} \nabla f (\boldsymbol {z} _ {n}) ^ {\top} \boldsymbol {y} + \epsilon_ {n}
$$
5:  $z_{n + 1}\gets (1 - \gamma_n)z_n + \gamma_ny_n$
6: return  $\mathbf{x}_{N + 1}$
$R\coloneqq \sup_{\pmb {z},\pmb {y}\in \mathcal{Z}\cup \mathcal{Y}}\| \pmb {z} - \pmb {y}\|$  . Then for  $N\geq 2$  , we have
$$
f \left(\boldsymbol {z} _ {N + 1}\right) - \min  _ {\boldsymbol {z} \in \mathcal {Z}} f (\boldsymbol {z}) \leq \frac {\beta R ^ {2} (\log N + 1)}{2 (N + 1)} + \frac {1}{N + 1} \sum_ {n = 1} ^ {N} \epsilon_ {n}.
$$
Lemma C.2 (Lemma C.2 of [56]). When running Algorithm 5, we have
$$
\boldsymbol {z} _ {N + 1} = \frac {1}{N + 1} \left(\sum_ {n = 1} ^ {N} \boldsymbol {y} _ {n} + \boldsymbol {z} _ {1}\right).
$$
Proof of Theorem 3. By our assumption on  $\mathbb{A}_{\mathcal{R}}$ , Assumption 8, we have that, at round  $n$ , with probability at least  $1 - \delta /2N$
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k}} [ \operatorname {c o s t} ^ {n} (\boldsymbol {\tau} _ {k}) ] - K \cdot \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {\pi} [ \operatorname {c o s t} ^ {n} (\boldsymbol {\tau}) ] \leq C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta} \cdot K ^ {\alpha}
$$
where we have used that, under Assumption 9 and by the definition of  $\mathrm{cost}_h^n (\pmb {\tau})$ ,  $|\mathrm{cost}^n (\pmb {\tau})|\leq 1$  for all  $\pmb {\tau}\in \mathcal{T}$ . This implies that
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k}} [ \langle \Xi_ {n}, \psi (\boldsymbol {\tau} _ {k}) \rangle ] \leq M \cdot \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {\pi} [ \cos^ {n} (\boldsymbol {\tau}) ] + M C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta} \cdot K ^ {\alpha - 1}.
$$
Furthermore, by Azuma-Hoeffding and under Assumption 9, we have that, with probability at least  $1 - \delta /2N$ ,
$$
\left| \frac {1}{K} \sum_ {k = 1} ^ {K} \langle \Xi_ {n}, \psi (\boldsymbol {\tau} _ {k}) \rangle - \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi_ {k}} [ \langle \Xi_ {n}, \psi (\boldsymbol {\tau} _ {k}) \rangle ] \right| \leq \sqrt {\frac {8 M ^ {2} \log (4 N / \delta)}{K}}.
$$
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \langle \Xi_ {n}, \psi (\boldsymbol {\tau} _ {k}) \rangle \leq M \cdot \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {\pi} [ \cos^ {n} (\boldsymbol {\tau}) ] + M \cdot \left(C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {K}{\delta} \cdot K ^ {\alpha - 1} + \sqrt {\frac {8 \log (4 N / \delta)}{K}}\right). \tag {C.1}
$$
$$
\langle \Xi_ {n}, \psi (\tau_ {k}) \rangle = \langle \nabla_ {\Gamma} \Phi (\Gamma) | _ {\Gamma = \Gamma_ {n}}, \psi (\tau) \rangle ,
$$
and that for any  $\Gamma \in \Omega_{\psi}$ , we have
$$
\left\langle \nabla_ {\Gamma} \Phi (\Gamma) \mid_ {\Gamma = \Gamma_ {n}}, \Gamma \right\rangle = \mathbb {E} _ {\pi \sim \omega} \left[ \mathbb {E} _ {\pi} [ \left\langle \nabla_ {\Gamma} \Phi (\Gamma) \mid_ {\Gamma = \Gamma_ {n}}, \psi (\boldsymbol {\tau}) \right\rangle ] ] \right.
$$
for some  $\omega$ . This implies that
$$
\begin{array}{l} \inf  _ {\boldsymbol {\Gamma} \in \Omega_ {\psi}} \langle \nabla_ {\boldsymbol {\Gamma}} \Phi (\boldsymbol {\Gamma}) | _ {\boldsymbol {\Gamma} = \boldsymbol {\Gamma} _ {n}}, \boldsymbol {\Gamma} \rangle = \inf  _ {\omega \in \triangle_ {\Pi}} \mathbb {E} _ {\pi \sim \omega} [ \mathbb {E} _ {\pi} [ \langle \nabla_ {\boldsymbol {\Gamma}} \Phi (\boldsymbol {\Gamma}) | _ {\boldsymbol {\Gamma} = \boldsymbol {\Gamma} _ {n}}, \boldsymbol {\psi} (\boldsymbol {\tau}) \rangle ] ] \\ = \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {\pi} [ \langle \Xi_ {n}, \psi (\boldsymbol {\tau}) \rangle ] \\ \end{array}
$$
$$
= M \cdot \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {\pi} [ \operatorname {c o s t} ^ {n} (\boldsymbol {\tau}) ].
$$
By (C.1) above, we have that
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \psi (\boldsymbol {\tau} _ {k})
$$
is an approximate minimizer of  $M \cdot \sup_{\pi \in \Pi_{\mathrm{exp}}} \mathbb{E}_{\pi}[\cos^n(\tau)]$ , with approximation tolerance  $M(C_{\mathcal{R}} \log^{p\mathcal{R}} \frac{2NK}{\delta} \cdot K^{\alpha - 1} + \sqrt{\frac{8 \log(4N / \delta)}{K}})$ . We can therefore apply Lemma C.1 with
$$
\epsilon_ {n} = M \cdot \left(C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {K}{\delta} \cdot K ^ {\alpha - 1} + \sqrt {\frac {8 \log (4 N / \delta)}{K}}\right)
$$
to get that
$$
\Phi (\mathbf {\Gamma} _ {N + 1}) - \min _ {\mathbf {\Gamma} \in \boldsymbol {\Omega} _ {\psi}} \Phi (\mathbf {\Gamma}) \leq \frac {\beta R ^ {2} (\log N + 1)}{2 (N + 1)} + M \cdot \left(C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta} \cdot K ^ {\alpha - 1} + \sqrt {\frac {8 \log (4 N / \delta)}{K}}\right).
$$
The result then follows since  $\Gamma_{N + 1} = \frac{1}{K(N + 1)}\sum_{n = 0}^{N}\sum_{k = 1}^{K}\psi (\pmb{\tau}_k^n)$  by Lemma C.2.
Proof of Corollary 1. By Theorem 5, for any setting of  $N$  and  $K$ , we have that with probability at least  $1 - \delta$ :
$$
\begin{array}{l} \operatorname {t r} \left(\mathcal {H} \left(\frac {1}{K (N + 1)} \sum_ {n = 0} ^ {N} \sum_ {k = 1} ^ {K} \psi \left(\boldsymbol {\tau} _ {k} ^ {n}\right) + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) - \min  _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \\ \leq \frac {\beta R ^ {2} \log N}{N + 1} + \frac {M C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta}}{K ^ {1 - \alpha}} + M \sqrt {\frac {8 \log \frac {4 N}{\delta}}{K}} \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {t r} \left(\mathcal {H} \left(\sum_ {n = 0} ^ {N} \sum_ {k = 1} ^ {K} \psi (\boldsymbol {\tau} _ {k} ^ {n}) + T \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) - \frac {\min _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}) ^ {- 1}\right)}{T} \\ \leq \frac {\beta R ^ {2} \log N}{T (N + 1)} + \frac {M C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 N K}{\delta}}{T \sqrt {K}} + M \frac {1}{T} \sqrt {\frac {8 \log \frac {4 N}{\delta}}{K}} \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {t r} \left(\mathcal {H} \left(\sum_ {n = 0} ^ {N} \sum_ {k = 1} ^ {K} \psi \left(\boldsymbol {\tau} _ {k} ^ {n}\right) + T \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \leq \frac {\min  _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right)}{T} \\ + \frac {\beta R ^ {2} \log T}{T (N + 1)} + \frac {M (3 \log^ {1 / 2} \frac {4 T}{\delta} + C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 T}{\delta})}{T \sqrt {K}}. \\ \end{array}
$$
It then remains to bound  $R, \beta$ , and  $M$ . By Lemma D.6 of [56], we have that
$$
\nabla_ {\mathbf {r}} \Phi (\mathbf {r}) [ \widetilde {\mathbf {r}} ] = - \operatorname {t r} \left(\mathcal {H} (\mathbf {r} + \mathbf {r} _ {0}) ^ {- 1} \widetilde {\mathbf {r}} (\mathbf {r} + \mathbf {r} _ {0}) ^ {- 1}\right).
$$
We can then compute the second derivative as, using Lemma D.6 of [56]:
$$
\begin{array}{l} \nabla_ {\mathbf {F}} ^ {2} \Phi (\boldsymbol {\Gamma}) [ \widetilde {\boldsymbol {\Gamma}}, \bar {\boldsymbol {\Gamma}} ] = \frac {\mathrm {d}}{\mathrm {d} t} \left[ - \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0} + t \bar {\boldsymbol {\Gamma}}) ^ {- 1} \widetilde {\boldsymbol {\Gamma}} (\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0} + t \bar {\boldsymbol {\Gamma}}) ^ {- 1}\right) \right] \\ = \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \bar {\boldsymbol {\Gamma}} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \widetilde {\boldsymbol {\Gamma}} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \\ \end{array}
$$
$$
+ \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \widetilde {\boldsymbol {\Gamma}} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \bar {\boldsymbol {\Gamma}} \left(\boldsymbol {\Gamma} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right).
$$
Recall that  $M$  is any bound on
$$
\sup_{\boldsymbol {\Gamma}\in \widehat{\Omega}_{\psi}}\sup_{\boldsymbol {\tau}\in \mathcal{T}}|\langle \nabla_{\boldsymbol{\Gamma}}\Phi (\boldsymbol {\Gamma}),\boldsymbol {\psi}(\boldsymbol {\tau})\rangle |.
$$
By the above computation of the gradient, we can bound this as
$$
\begin{array}{l} \sup  _ {\Gamma \in \widehat {\Omega} _ {\psi}} \sup  _ {\tau \in \mathcal {T}} | \langle \nabla_ {\Gamma} \Phi (\Gamma), \psi (\tau) \rangle | \leq \sup  _ {\Gamma \in \widehat {\Omega} _ {\psi}} \sup  _ {\tau \in \mathcal {T}} \left| \operatorname {t r} \left(\mathcal {H} (\Gamma + \Gamma_ {0}) ^ {- 1} \psi (\tau) (\Gamma + \Gamma_ {0}) ^ {- 1}\right) \right| \\ \leq \| \mathcal {H} \| _ {\mathrm {o p}} \| \boldsymbol {\Gamma} _ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {2} \cdot \sup  _ {\boldsymbol {\tau} \in \mathcal {T}} \operatorname {t r} (\boldsymbol {\psi} (\boldsymbol {\tau})) \tag {C.2} \\ \leq D \| \mathcal {H} \| _ {\mathrm {o p}} \| \boldsymbol {\Gamma} _ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {2}. \\ \end{array}
$$
To bound  $\beta$ , by the Mean Value Theorem it suffices to bound the operator norm of  $\nabla_{\Gamma}^{2}\Phi(\Gamma)$ . Using the expression above, we can bound this as
$$
\begin{array}{l} \sup  _ {\widetilde {\Gamma}, \bar {\Gamma} \in \widehat {\Omega} _ {\psi}} | \nabla_ {\Gamma} ^ {2} \Phi (\Gamma) [ \psi (\tau_ {1}), \psi (\tau_ {2}) ] | \leq 2 \| \mathcal {H} \| _ {\mathrm {o p}} \| \Gamma_ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {3} \cdot \sup  _ {\widetilde {\Gamma}, \bar {\Gamma} \in \widehat {\Omega} _ {\psi}} \operatorname {t r} (\widetilde {\Gamma} \bar {\Gamma}) \\ \leq 2 D ^ {2} \| \mathcal {H} \| _ {\mathrm {o p}} \| \boldsymbol {\Gamma} _ {0} ^ {- 1} \| _ {\mathrm {o p}} ^ {3}. \\ \end{array}
$$
Finally, it's straightforward to bound  $R \leq 2D$ . Putting all of this together gives the result.
C.2 Collecting Full-Rank Data
Algorithm 6 Minimum Eigenvalue Maximization (MINEIG)
1: input: scale  $N$ , confidence  $\delta$ , regret minimization algorithm  $\mathbb{A}_{\mathcal{R}}$ , exploration policies  $\Pi_{\mathrm{exp}}$
2: for  $j = 1, 2, 3, \ldots$  do
3:  $N_{j}\gets \lceil 2^{j / 3}\rceil -1,K_{j}\gets \lceil 2^{2j / 3}\rceil ,T_{j}\gets (N_{j} + 1)K_{j},\lambda_{j}\gets T_{j}^{-1 / 18},\delta_{j}\gets \frac{\delta}{4j^{2}}$
4:  $\Sigma_{j},\Pi_{j}\gets \mathrm{DYNAMICOED}(\Phi ,N_{j},K_{j},\delta_{j},\mathbb{A}_{\mathcal{R}},\Pi_{\mathrm{exp}})$  for  $\Phi (\Gamma) = \mathrm{tr}((\Gamma +\lambda_j\cdot I)^{-1})$
5: if  $\lambda_{\min}(\Sigma_j) \geq 12544Dd_\psi \log \frac{2N(2 + 32T_j)}{\delta}$  then
6: break
7: return  $\Pi_j$
In this section, we consider the setting where  $\psi(\tau) \in S_{+}^{d_{\psi}}$ , and our goal is to collect  $\{\pmb{\tau}_t\}_{t=1}^T$  such that  $\psi\left(\frac{1}{T}\sum_{t=1}^{T}\psi(\pmb{\tau}_t)\right) > 0$ . For this to be achievable, we need the following assumption, a generalization of Assumption 3.
Assumption 12 (Full-Rank Data). Consider  $\psi(\tau)$  such that  $\psi(\tau) \in \mathbb{S}_+^{d_\psi}$ . Then we have  $\sup_{\Gamma \in \Omega_\psi} \lambda_{\min}(\Gamma) \geq \lambda_{\min}^\star$  for some  $\lambda_{\min}^\star > 0$ .
Throughout this section we also assume that Assumption 10 is satisfied with  $\alpha = 1 / 2$  (though all results generalize in a straightforward way for  $\alpha \neq 1 / 2$ ). We have the following result.
Lemma C.3. Under Assumptions 10 to 12, running Algorithm 6 we have that with probability at least  $1 - \delta$ , it will terminate after collecting at most
$$
\text {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\operatorname* {m i n}} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {N}{\delta}\right)
$$
episodes, and return policy set  $\Pi$  such that
$$
\lambda_ {\min } \left(\sum_ {\pi \in \Pi} \mathbf {T} _ {\pi}\right) \geq 6 2 7 2 D d _ {\psi} \log \frac {6 8 N}{\delta}.
$$
Furthermore, if we rerun each policy in  $\Pi$  once, the resulting features  $\Sigma$  will satisfy, with probability at least  $1 - \delta / N$ :
$$
\lambda_ {\min } (\boldsymbol {\Sigma}) \geq 6 2 7 2 D d _ {\psi} \log \frac {6 8 N}{\delta}.
$$
Proof. By Lemma C.4 and our choice of  $N_{j}$  and  $K_{j}$  in Algorithm 6, we have that if  $\lambda_{j} \leq \frac{\lambda_{\min}^{\star}}{4d_{\psi}}$  and
$$
T _ {j} ^ {1 / 3} \geq \widetilde {\Omega} \left(\left(D \lambda_ {j} ^ {- 2} \left(D ^ {3} \lambda_ {j} ^ {- 1} + C _ {\mathcal {R}} \cdot \log^ {p _ {\mathcal {R}}} \frac {1}{\delta_ {j}}\right)\right) \cdot \frac {\lambda_ {\min } ^ {\star}}{d _ {\psi}}\right), \tag {C.3}
$$
then  $\lambda_{\min}(\pmb{\Sigma}_j) \geq \frac{\lambda_{\min}^*}{4d_{\psi}} \cdot T_j$  with probability at least  $1 - \delta_j$ . It follows that, with probability at least  $1 - \delta_j$ , the if statement on Line 5 will be true once  $\lambda_j \leq \frac{\lambda_{\min}^*}{4d_{\psi}}$ , (C.3) holds, and
$$
\frac {\lambda_ {\min } ^ {\star}}{4 d _ {\psi}} \cdot T _ {j} \geq 1 2 5 4 4 D d _ {\psi} \log \frac {2 N (2 + 3 2 T _ {j})}{\delta}. \tag {C.4}
$$
By our choice of  $\lambda_{j} = T_{j}^{-1 / 18}$ , a sufficient condition to ensure  $\lambda_{j} \leq \frac{\lambda_{\min}^{\star}}{4d_{\psi}}$ , (C.3), and (C.4) is
$$
T _ {j} \geq \widetilde {\Omega} \left(\max  \left\{\left(\frac {d _ {\psi}}{\lambda_ {\min } ^ {\star}}\right) ^ {1 8}, \left(\frac {D ^ {4} \lambda_ {\min } ^ {\star}}{d _ {\psi}}\right) ^ {6}, \left(D C _ {\mathcal {R}} \cdot \log^ {p _ {\mathcal {R}}} \frac {1}{\delta_ {j}} \cdot \frac {\lambda_ {\min} ^ {\star}}{d _ {\psi}}\right) ^ {9 / 2}, \frac {D d _ {\psi} ^ {2}}{\lambda_ {\min} ^ {\star}} \cdot \log \frac {N T _ {j}}{\delta} \right\}\right).
$$
Since  $T_{j} = \lceil 2^{j / 3}\rceil \lceil 2^{2j / 3}\rceil \in [2^{j},4\cdot 2^{j}]$ , it follows that the if statement on Line 5 will be met after running for at most
$$
\left. \widetilde {\mathcal {O}} \left(\max  \left\{\left(\frac {d _ {\psi}}{\lambda_ {\min } ^ {\star}}\right) ^ {1 8}, \left(\frac {D ^ {4} \lambda_ {\min } ^ {\star}}{d _ {\psi}}\right) ^ {6}, \left(D C _ {\mathcal {R}} \cdot \log^ {p _ {\mathcal {R}}} \frac {1}{\delta_ {j}} \cdot \frac {\lambda_ {\min } ^ {\star}}{d _ {\psi}}\right) ^ {9 / 2}, \frac {D d _ {\psi} ^ {2}}{\lambda_ {\min } ^ {\star}} \cdot \log \frac {N}{\delta} \right\}\right) \right. \tag {C.5}
$$
By Lemma C.5, if  $\lambda_{\mathrm{min}}(\pmb{\Sigma}_j) \geq 12544Dd_\psi \log \frac{2N(2 + 32T_j)}{\delta}$  and we rerun all policies in  $\Pi_j$ , then we will collect data  $\widetilde{\pmb{\Sigma}}$  such that  $\lambda_{\mathrm{min}}(\widetilde{\pmb{\Sigma}}) \geq \frac{1}{2}\lambda_{\mathrm{min}}(\pmb{\Sigma}_j)$ , with probability at least  $1 - \delta / 2N$ . As the if statement on Line 5 will only be true once this is met, it follows that, with probability at least  $1 - \delta / 2N$ , rerunning all policies in  $\Pi_j$  once, we will collect data  $\pmb{\Sigma}$  which satisfies
$$
\lambda_ {\mathrm {m i n}} (\pmb {\Sigma}) \geq \frac {1}{2} \lambda_ {\mathrm {m i n}} (\pmb {\Sigma} _ {j}) \geq 6 2 7 2 D d _ {\psi} \log \frac {2 N (2 + 3 2 T _ {j})}{\delta} \geq 6 2 7 2 D d _ {\psi} \log \frac {6 8 N}{\delta}.
$$
The lower bound on  $\lambda_{\mathrm{min}}(\sum_{\pi \in \Pi}\Gamma_{\pi})$  follows analogously from Lemma C.5.
The result then follows noting that the failure probability of running DYNAMICOED is at most
$$
\sum_ {j = 1} ^ {\infty} \frac {\delta}{4 j ^ {2}} \leq \delta / 2.
$$
C.2.1 Supporting Lemmas
Lemma C.4. Under Assumptions 10 to 12, consider running DYNAMICOED on the objective
$$
\Phi (\Gamma) = \operatorname {t r} ((\Gamma + \lambda \cdot I) ^ {- 1})
$$
with  $N = \lceil 2^{i / 3}\rceil -1$  and  $K = \lceil 2^{2i / 3}\rceil$ , for some  $\lambda >0$  and  $i$ . Let  $T\coloneqq (N + 1)K$ . Then if  $\lambda \leq \frac{\lambda_{\min}^{\star}}{4d_{\psi}}$  and
$$
T ^ {1 / 3} \geq \widetilde {\Omega} \left(\left(D \lambda^ {- 2} \left(D ^ {3} \lambda^ {- 1} + C _ {\mathcal {R}} \cdot \log^ {p _ {\mathcal {R}}} \frac {1}{\delta}\right)\right) \cdot \frac {\lambda_ {\operatorname* {m i n}} ^ {\star}}{d _ {\psi}}\right), \tag {C.6}
$$
$$
\lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \psi (\boldsymbol {\tau} _ {t})\right) \geq \frac {\lambda_ {\min } ^ {\star}}{4 d _ {\boldsymbol {\psi}}} \cdot T.
$$
Proof. Applying Corollary 1 with  $\mathcal{H} = I$  and  $\Gamma_0 = \lambda \cdot I$ , we have that, with probability at least  $1 - \delta$ :
$$
\begin{array}{l} \mathrm {t r} \left(\left(\sum_ {t = 1} ^ {T} \psi (\pmb {\tau} _ {t}) + T \lambda \cdot I\right) ^ {- 1}\right) \leq \frac {1}{T} \cdot \min _ {\pmb {\Gamma} \in \Omega_ {\psi}} \mathrm {t r} ((\pmb {\Gamma} + \lambda \cdot I) ^ {- 1}) + \frac {8 D ^ {4} \lambda^ {- 3}}{T (N + 1)} + \frac {8 D \lambda^ {- 2} (\log^ {1 / 2} \frac {T}{\delta} + C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {T}{\delta})}{T \sqrt {K}} \\ \leq \frac {1}{T} \cdot \min _ {\mathbf {\Gamma} \in \Omega_ {\psi}} \mathrm {t r} ((\mathbf {\Gamma} + \boldsymbol {\lambda} \cdot I) ^ {- 1}) + \frac {2 4 D ^ {4} \lambda^ {- 3}}{T ^ {4 / 3}} + \frac {2 4 D \lambda^ {- 2} (\log^ {1 / 2} \frac {T}{\delta} + C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {T}{\delta})}{T ^ {4 / 3}} \\ \end{array}
$$
where the second inequality follows since
$$
T = \lceil 2 ^ {i / 3} \rceil \lceil 2 ^ {2 i / 3} \rceil \leq 4 \cdot 2 ^ {i}
$$
which implies  $N + 1 = \lceil 2^{i / 3} \rceil \geq T^{1 / 3} / 4^{1 / 3}$  and  $K = \lceil 2^{2i / 3} \rceil \geq T^{2 / 3} / 4^{2 / 3}$ . If  $T$  satisfies (C.6), then we can bound
$$
\frac {2 4 D ^ {4} \lambda^ {- 3}}{T ^ {4 / 3}} + \frac {2 4 D \lambda^ {- 2} (\log^ {1 / 2} \frac {T}{\delta} + C _ {\mathcal {R}} \log^ {p \mathcal {R}} \frac {T}{\delta})}{T ^ {4 / 3}} \leq \frac {1}{T} \cdot \frac {d _ {\psi}}{\lambda_ {\min} ^ {\star}}.
$$
Furthermore, under Assumption 3 there exists some  $\Gamma \in \Omega_{\psi}$  such that  $\Gamma \succeq \lambda_{\min}^{\star} \cdot I$ , so we can upper bound
$$
\min  _ {\mathbf {\Gamma} \in \Omega_ {\psi}} \operatorname {t r} \left(\left(\mathbf {\Gamma} + \lambda \cdot I\right) ^ {- 1}\right) \leq \frac {d _ {\psi}}{\lambda_ {\operatorname* {m i n}} ^ {\star}}
$$
and we can lower bound
$$
\operatorname {t r} \left(\left(\sum_ {t = 1} ^ {T} \boldsymbol {\psi} (\boldsymbol {\tau} _ {t}) + T \lambda \cdot I\right) ^ {- 1}\right) \geq \frac {1}{\lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {\psi} (\boldsymbol {\tau} _ {t})\right) + T \lambda}.
$$
$$
\frac {1}{\lambda_ {\min} \left(\sum_ {t = 1} ^ {T} \boldsymbol {\psi} (\boldsymbol {\tau} _ {t})\right) + T \lambda} \leq \frac {1}{T} \cdot \frac {2 d _ {\boldsymbol {\psi}}}{\lambda_ {\min} ^ {\star}} \Rightarrow \lambda_ {\min} \left(\sum_ {t = 1} ^ {T} \boldsymbol {\psi} (\boldsymbol {\tau} _ {t})\right) \geq \frac {T \lambda_ {\min} ^ {\star}}{2 d _ {\boldsymbol {\psi}}} - T \lambda .
$$
It follows that if  $\lambda \leq \frac{\lambda_{\min}^{\star}}{4d_{\psi}}$ , then we have
$$
\lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \psi (\boldsymbol {\tau} _ {t})\right) \geq T \cdot \frac {\lambda_ {\min } ^ {\star}}{4 d _ {\psi}}
$$
Lemma C.5. Consider running some policies  $(\pi_{\tau})_{\tau = 1}^{T}$ , for  $\pi_{\tau}$ $\mathcal{F}_{\tau -1}$ -measurable, and collecting covariance  $\boldsymbol{\Sigma}_T = \sum_{t = 1}^T\psi (\pmb {\tau}_t)$ . Then under Assumption 11, as long as
$$
\lambda_ {\min } (\boldsymbol {\Sigma} _ {T}) \geq 1 2 5 4 4 D d _ {\psi} \log \frac {2 + 3 2 T}{\delta}
$$
with probability at least  $1 - \delta$ , if we rerun each  $(\pi_{\tau})_{\tau = 1}^{T}$ , we will collect features  $\tilde{\Sigma}_T$  such that
$$
\lambda_ {\min } (\widetilde {\boldsymbol {\Sigma}} _ {T}) \geq \frac {1}{2} \lambda_ {\min } (\boldsymbol {\Sigma} _ {T}).
$$
$$
\lambda_ {\min } \left(\sum_ {\tau = 1} ^ {T} \boldsymbol {\Gamma} _ {\pi_ {\tau}}\right) \geq \frac {1}{2} \lambda_ {\min } (\boldsymbol {\Sigma} _ {T}).
$$