SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. This follows from applying Lemma D.7 of [56] to the matrix  $\frac{1}{D}\pmb{\Sigma}_T$ . Note that while [56] considers the setting of linear MDPs, the proof of Lemma D.7 of [56] does not make use of the linear MDP assumption, and the proof therefore extends immediately to our setting. Furthermore, though it is not explicitly stated, the lower bound on  $\lambda_{\mathrm{min}}(\sum_{\tau = 1}^{T}\Gamma_{\pi_{\tau}})$  is also proved in Lemma D.7 of [56].
Algorithm 7 Learn Minimizing Exploration Policies (LEARNEXPII)
1: input:  $\mathcal{H}$ , iterates bound  $\tilde{N}$ , confidence  $\delta$ , regret minimization algorithm  $\mathbb{A}_{\mathcal{R}}$ , exploration policies  $\Pi_{\mathrm{exp}}$
2: for  $i = 1, 2, 3, \ldots$  do
3:  $N_{i}\gets \lceil 2^{i / 3}\rceil -1,K_{i}\gets \lceil 2^{2i / 3}\rceil ,T_{i}\gets (N_{i} + 1)K_{i},\delta_{i}\gets \delta /4i^{2}$
4:  $\Pi_{\mathrm{MINEIG}}^{\mathrm{I}} \gets \mathrm{MINEIG}(\widetilde{NT}_i, \delta_i, \mathbb{A}_{\mathcal{R}}, \Pi_{\mathrm{exp}})$
5: Run policies in  $\Pi_{\mathrm{MINEIG}}^i\left[T_i / \left|\Pi_{\mathrm{MINEIG}}^i\right|\right]$  times, set  $\Gamma_0^i$  to collected features
6:  $\Phi (\Gamma)\gets \mathrm{tr}(\mathcal{H}(\Gamma +T_i^{-1}\Gamma_0^i)^{-1})$
7:  $\Gamma_{\mathrm{fw}}^i,\Pi_{\mathrm{fw}}^i\gets \mathrm{DYNAMICOED}(\Phi ,N_i,K_i,\delta ,\mathbb{A}_{\mathcal{R}},\Pi_{\mathrm{exp}})$
8: if
$$
\max  _ {j = 1, \dots , i} \left| \Pi_ {\mathrm {M I N E I G}} ^ {j} \right| \leq T _ {i} \tag {C.7}
$$
$$
\frac {1 6 D ^ {4} \| \mathcal {H} \| _ {\mathrm {o p}} \| \left(T _ {i} ^ {- 1} \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \| _ {\mathrm {o p}} ^ {3}}{T _ {i} \left(N _ {i} + 1\right)} + \frac {1 6 D \| \mathcal {H} \| _ {\mathrm {o p}} \| \left(T _ {i} ^ {- 1} \boldsymbol {\Gamma} _ {0}\right) ^ {- 1} \| _ {\mathrm {o p}} ^ {2} \left(\log^ {1 / 2} \frac {4 T _ {i}}{\delta} + C _ {\mathcal {R}} \log^ {p \mathcal {R}} \frac {2 T _ {i}}{\delta}\right)}{T _ {i} \sqrt {K _ {i}}} \leq \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) ^ {- 1}\right) \tag {C.8}
$$
$$
\operatorname {t r} (\mathcal {H}) \cdot D \sqrt {2 T _ {i}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {2 T _ {i}}) + 8 \log 1 / \delta} \cdot \frac {2}{\lambda_ {\min } \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) ^ {2}} \leq \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) ^ {- 1}\right) \tag {C.9}
$$
$$
D \sqrt {2 T _ {i}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {2 T _ {i}}) + 8 \log 1 / \delta} \leq \frac {1}{2} \lambda_ {\min } \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) \tag {C.10}
$$
9: then
10:  $\Gamma_{\mathrm{out}}\gets \Gamma_{\mathrm{fw}}^{i} + \Gamma_{0}^{i},\Pi_{\mathrm{out}}\gets \Pi_{\mathrm{MINEIG}}^{i}\cup (\cup_{j = 1}^{\lceil T_i / \rceil \Pi_{\mathrm{MINEIG}}\rceil}\Pi_{\mathrm{fw}}^{i})$
11: return  $\Gamma_{\mathrm{out}}, \Pi_{\mathrm{out}}$
C.3 Rerunning Policies
In this section, we build on the analysis of the DYNAMICOED algorithm to show that, not only do the features collected by DYNAMICOED approximately minimize  $\Phi$ , but that, under certain conditions, if we rerun the policies that DYNAMICOED ran to collect this data, we will collect a new set of features which also approximately minimizes  $\Phi$ .
In particular, we specialize this argument to objectives of the form  $\Phi(\Gamma) = \mathrm{tr}(\mathcal{H}\Gamma^{-1})$ . LEARNEXPII (Algorithm 7) proceeds by first calling MINEIG to collect full-rank data, using this data as a regularizer of  $\Phi(\Gamma)$ , and the running DYNAMICOED on this objective. After meeting a certain termination criteria, it terminates, and returns the policies it has run over its operation.
Lemma C.6. Let  $\mathcal{E}_{\mathrm{exp}}$  denote the event that, for all  $i = 1,2,3,\ldots$ , the success event of MINEIG and DYNAMICOED occurs, and
$$
\lambda_ {\min } \left(\boldsymbol {\Gamma} _ {0} ^ {i}\right) \geq \left| T _ {i} / \left| \Pi_ {\text {M I N E I G}} ^ {i} \right| \right| \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Then if Assumptions 10 to 12 hold,  $\mathbb{P}[\mathcal{E}_{\mathrm{exp}}] \geq 1 - \delta$ .
Lemma C.7. Consider rerunning each policy in  $\Pi_{\mathrm{out}}N\leq \tilde{N}$  times, and let  $\widetilde{\Gamma}$  denote the obtained features. Then, if Assumptions 10 to 12 hold, with probability at least  $1 - 3\delta$  on the event  $\mathcal{E}_{\mathrm{exp}}$ :
$$
\left\| \widetilde {\boldsymbol {\Gamma}} - N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \boldsymbol {\Gamma} _ {\pi} \right\| _ {\mathrm {o p}} \leq \frac {\sqrt {T _ {\mathrm {o u t}}} \cdot \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {N T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta}}{\sqrt {N} \cdot 6 2 7 2 d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}} \cdot \lambda_ {\min } \left(N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \boldsymbol {\Gamma} _ {\pi}\right),
$$
$$
N \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta} \leq \min  \left\{\lambda_ {\min } \left(N \cdot \sum_ {\pi \in \Pi_ {\text {o u t}}} \mathbf {T} _ {\pi}\right), \lambda_ {\min } (\widetilde {\Gamma}) \right\}, \tag {C.12}
$$
$$
\operatorname {t r} \left(\mathcal {H} \left(\sum_ {\pi \in \Pi_ {\text {o u t}}} \boldsymbol {\Gamma} _ {\pi}\right) ^ {- 1}\right) \leq \frac {1 2}{T _ {\text {o u t}}} \cdot \min  _ {\boldsymbol {\Gamma} \in \boldsymbol {\Omega}} \operatorname {t r} \left(\mathcal {H} \boldsymbol {\Gamma} ^ {- 1}\right) \tag {C.13}
$$
for  $T_{\mathrm{out}} \coloneqq |\Pi_{\mathrm{out}}|$ .
Lemma C.8. On the event  $\mathcal{E}_{\mathrm{exp}}$ , under Assumptions 10 to 12, we can bound
$$
T _ {\text {o u t}} \leq \operatorname {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\min} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {\widetilde {N}}{\delta}\right).
$$
Furthermore, the total number of episodes collected by Algorithm 7 is bounded by  $(16 + 2\log (T_{\mathrm{out}}))\cdot$ $T_{\mathrm{out}}$
C.3.1 Supporting Lemmas and Proofs
Lemma C.9. Under Assumption 11, for any  $\Gamma = \mathbb{E}_{\tau \sim \omega}[\psi (\tau)]$  and  $\mathcal{H}\succeq 0$  we can bound
$$
\operatorname {t r} \left(\mathcal {H} \boldsymbol {\Gamma} ^ {- 1}\right) \geq D ^ {- 1} \cdot \operatorname {t r} (\mathcal {H}).
$$
Proof. By Von Neumann's Trace Inequality we can lower bound
$$
\operatorname {t r} \left(\mathcal {H} \boldsymbol {\Gamma} ^ {- 1}\right) \geq \lambda_ {\min } \left(\boldsymbol {\Gamma} ^ {- 1}\right) \cdot \operatorname {t r} (\mathcal {H}) = \left\| \boldsymbol {\Gamma} \right\| _ {\mathrm {o p}} ^ {- 1} \cdot \operatorname {t r} (\mathcal {H}).
$$
By our assumption that  $\mathrm{tr}(\psi (\pmb {\tau}))\leq D$  , we can bound  $\| \mathbf{T}\|_{\mathrm{op}}\leq D$  , which proves the result.
Lemma C.10. Assume  $\mathrm{tr}(\psi(\tau)) \leq D$  for all  $\tau$ . Let  $\Gamma_K$  denote the time-normalized features obtained by playing policies  $\{\pi_k\}_{k=1}^K$ , where  $\pi_k$  is  $\mathcal{F}_{k-1}$ -measurable. Then, with probability at least  $1 - \delta$ ,
$$
\left\| \frac {1}{K} \sum_ {k = 1} ^ {K} \boldsymbol {\Gamma} _ {\pi_ {k}} - \boldsymbol {\Gamma} _ {K} \right\| _ {\mathrm {o p}} \leq D \sqrt {\frac {8 d _ {\psi} \log (1 + 8 \sqrt {K}) + 8 \log 1 / \delta}{K}}.
$$
Proof. This follows from an argument identical to the proof of Lemma C.4 of [56]. While [56] considers the setting of linear MDPs, we note that the proof of Lemma C.4 of [56] nowhere relies on the linear MDP assumption. The result stated here then follows identically as Lemma C.4 of [56], after normalizing  $\psi(\tau)$  by  $D$ .
Proof of Lemma C.6. By Lemma C.3, the failure probability of running MINEIG at round  $i$  is  $\delta_i = \delta / 8i^2$ , and by Corollary 1 the failure probability of DYNAMICOED at round  $i$  is also bounded by  $\delta_i = \delta / 8i^2$ . It follows that the total failure probability of running MINEIG and DYNAMICOED is bounded by
$$
\sum_ {i = 1} ^ {\infty} \frac {2 \cdot \delta}{8 i ^ {2}} \leq \frac {\delta}{2}.
$$
Furthermore, by Lemma C.3, we have that rerunning all policies in  $\Pi_{\mathrm{MINEIG}}^i$ , we will obtain features  $\Gamma$  satisfying, with probability at least  $1 - \delta_i / \widetilde{NT}_i$ :
$$
\lambda_ {\min } (\Gamma) \geq 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta_ {i}}.
$$
Repeating this  $\left\lceil T_{i} / \left|\Pi_{\mathrm{MINEIG}}^{i}\right| \right\rceil$  times and union bounding, we have that
$$
\lambda_ {\mathrm {m i n}} (\Gamma_ {0} ^ {i}) \geq \lceil T _ {i} / | \Pi_ {\mathrm {M I N E I G}} ^ {i} | \rceil \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta_ {i}}
$$
with probability at least  $1 - \delta /\widetilde{NT}_i\cdot \lceil T_i / |\Pi_{\mathrm{MINEIG}}^i | \rceil \geq 1 - \delta /\widetilde{N}$
Proof of Lemma C.7. Let  $\Pi_{\mathrm{MINEIG}}, \Gamma_0, \Pi_{\mathrm{fw}}, \Gamma_{\mathrm{fw}}$  denote the policies and features obtained on the round at which Algorithm 7 terminates. Let  $T_{\mathrm{fw}} = |\Pi_{\mathrm{fw}}|$  denote the number of episodes of DYNAMICOED on the terminating round, and  $N_{\mathrm{fw}}, K_{\mathrm{fw}}$  the corresponding values of  $N_i$  and  $K_i$ . Throughout the proof we make use of the fact that at termination of Algorithm 7, all of (C.7)-(C.10) are met.
Proof of (C.11) and (C.12). By Lemma C.10 we have that, with probability at least  $1 - \delta$ :
$$
\left\| \widetilde {\Gamma} - N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \Gamma_ {\pi} \right\| _ {\mathrm {o p}} \leq D \sqrt {N T _ {\mathrm {o u t}}} \cdot \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {N T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta}.
$$
On  $\mathcal{E}_{\mathrm{exp}}$  , by Lemma C.3, we can bound
$$
\left| \Pi_ {\mathrm {M I N E I G}} \right| \leq \operatorname {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\min} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {\widetilde {N} T _ {\text {o u t}}}{\delta}\right)
$$
and, furthermore, we can lower bound
$$
\lambda_ {\min } \left(\sum_ {\pi \in \Pi_ {\mathrm {M i N E I G}}} \boldsymbol {\Gamma} _ {\pi}\right) \geq 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Since  $\Pi_{\mathrm{MINEIG}} \subseteq \Pi_{\mathrm{out}}$ , it follows that
$$
\lambda_ {\mathrm {m i n}} \left(N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \Gamma_ {\pi}\right) \geq N \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Combining these, we therefore have that, with probability at least  $1 - \delta$ :
$$
\left\| \widetilde {\Gamma} - N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \Gamma_ {\pi} \right\| _ {\mathrm {o p}} \leq \frac {\sqrt {N T _ {\mathrm {o u t}}} \cdot \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {N T _ {\mathrm {o u t}}} + 8 \log 1 / \delta}}{N \cdot 6 2 7 2 d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}} \cdot \lambda_ {\min} \left(N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \Gamma_ {\pi}\right).
$$
In addition, also by Lemma C.3, we have that with probability at least  $1 - \delta /\widetilde{N} T_{\mathrm{out}}\cdot N\geq 1 - \delta$  ,that
$$
\lambda_ {\min } (\widetilde {\mathbf {T}}) \geq N \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Proof of (C.13). By Corollary 1, on  $\mathcal{E}_{\mathrm{exp}}$  we have that:
$$
\begin{array}{l} \Phi (\boldsymbol {\Gamma} _ {\mathrm {f w}}) = \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} _ {\mathrm {f w}} + \boldsymbol {\Gamma} _ {0}) ^ {- 1}\right) \leq \frac {\min _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} + T _ {\mathrm {f w}} ^ {- 1} \boldsymbol {\Gamma} _ {0}) ^ {- 1}\right)}{T _ {\mathrm {f w}}} \\ + \frac {8 D ^ {4} \| \mathcal {H} \| _ {\mathrm {o p}} \| (T _ {\mathrm {f w}} ^ {- 1} \Gamma_ {0}) ^ {- 1} \| _ {\mathrm {o p}} ^ {3}}{T _ {\mathrm {f w}} (N _ {\mathrm {f w}} + 1)} + \frac {8 D \| \mathcal {H} \| _ {\mathrm {o p}} \| (T _ {\mathrm {f w}} ^ {- 1} \Gamma_ {0}) ^ {- 1} \| _ {\mathrm {o p}} ^ {2} (\log^ {1 / 2} \frac {4 T _ {\mathrm {f w}}}{\delta} + C _ {\mathcal {R}} \log^ {p \mathcal {R}} \frac {2 T _ {\mathrm {f w}}}{\delta})}{T _ {\mathrm {f w}} \sqrt {K _ {\mathrm {f w}}}}. \\ \end{array}
$$
Since  $T_{\mathrm{fw}}$  satisfies (C.8), we can bound
$$
\frac {8 D ^ {4} \| \mathcal {H} \| _ {\mathrm {o p}} \| (T _ {\mathrm {f w}} ^ {- 1} \Gamma_ {0}) ^ {- 1} \| _ {\mathrm {o p}} ^ {3}}{T _ {\mathrm {f w}} (N _ {\mathrm {f w}} + 1)} + \frac {8 D \| \mathcal {H} \| _ {\mathrm {o p}} \| (T _ {\mathrm {f w}} ^ {- 1} \Gamma_ {0}) ^ {- 1} \| _ {\mathrm {o p}} ^ {2} (\log^ {1 / 2} \frac {4 T _ {\mathrm {f w}}}{\delta} + C _ {\mathcal {R}} \log^ {p _ {\mathcal {R}}} \frac {2 T _ {\mathrm {f w}}}{\delta})}{T _ {\mathrm {f w}} \sqrt {K _ {\mathrm {f w}}}} \leq \frac {1}{2} \Phi (\Gamma_ {\mathrm {f w}}).
$$
$$
\begin{array}{l} \Phi (\Gamma_ {\mathrm {f w}}) \leq \frac {\operatorname* {m i n} _ {\mathbf {R} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\mathbf {R} + T _ {\mathrm {f w}} ^ {- 1} \mathbf {r e s p} _ {0}) ^ {- 1}\right)}{T _ {\mathrm {f w}}} + \frac {1}{2} \Phi (\Gamma_ {\mathrm {f w}}) \\ \Longrightarrow \Phi (\mathbf {\Gamma} _ {\mathrm {f w}}) \leq 2 \cdot \frac {\operatorname* {m i n} _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} + T _ {\mathrm {f w}} ^ {- 1} \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right)}{T _ {\mathrm {f w}}}. \tag {C.14} \\ \end{array}
$$
By Lemma C.10, we have that, with probability at least  $1 - \delta$ :
$$
\left\| \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \Gamma_ {\pi} - (\Gamma_ {\mathrm {f w}} + \Gamma_ {0}) \right\| _ {\mathrm {o p}} \leq D \sqrt {T _ {\mathrm {o u t}}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta}.
$$
Since (C.10) is satisfied and  $|\Pi_{\mathrm{MINEIG}}^i |\leq T_i$  , we have
$$
D \sqrt {T _ {\mathrm {o u t}}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta} \leq \frac {1}{2} \lambda_ {\min} (\mathbf {\Gamma} _ {\mathrm {f w}} + \mathbf {\Gamma} _ {0}).
$$
By Lemma A.2 it follows that
$$
\left\| \left(\sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \mathbf {\Gamma} _ {\pi}\right) ^ {- 1} - \left(\mathbf {\Gamma} _ {\mathrm {f w}} + \mathbf {\Gamma} _ {0}\right) ^ {- 1} \right\| _ {\mathrm {o p}} \leq D \sqrt {T _ {\mathrm {o u t}}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta} \cdot \frac {2}{\lambda_ {\min} \left(\mathbf {\Gamma} _ {\mathrm {f w}} + \mathbf {\Gamma} _ {0}\right) ^ {2}}.
$$
$$
\begin{array}{l} \operatorname {t r} \left(\mathcal {H} \left(\sum_ {\pi \in \Pi_ {\text {o u t}}} \boldsymbol {\Gamma} _ {\pi}\right) ^ {- 1}\right) \leq \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \\ + \operatorname {t r} (\mathcal {H}) \cdot D \sqrt {T _ {\text {o u t}}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {T _ {\text {o u t}}}) + 8 \log 1 / \delta} \cdot \frac {2}{\lambda_ {\min} \left(\Gamma_ {\mathrm {f w}} + \Gamma_ {0}\right) ^ {2}}. \\ \end{array}
$$
Now if
$$
\operatorname {t r} (\mathcal {H}) \cdot D \sqrt {T _ {\text {o u t}}} \sqrt {8 d _ {\psi} \log (1 + 8 \sqrt {T _ {\text {o u t}}}) + 8 \log 1 / \delta} \cdot \frac {2}{\lambda_ {\min } \left(\mathbf {\Gamma} _ {\mathrm {f w}} + \mathbf {\Gamma} _ {0}\right) ^ {2}} \leq \operatorname {t r} \left(\mathcal {H} \left(\mathbf {\Gamma} _ {\mathrm {f w}} + \mathbf {\Gamma} _ {0}\right) ^ {- 1}\right), \tag {C.15}
$$
we can bound this all by
$$
\leq 2 \operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} + \boldsymbol {\Gamma} _ {0}\right) ^ {- 1}\right) \leq \frac {4}{T _ {\mathrm {f w}}} \cdot \min  _ {\boldsymbol {\Gamma} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\boldsymbol {\Gamma} + T _ {\mathrm {f w}} ^ {- 1} \boldsymbol {\Gamma} _ {0}) ^ {- 1}\right)
$$
where the last inequality follows from (C.14). However, note that (C.15) since (C.9) holds. Finally, note that
$$
T _ {\text {o u t}} = T _ {\mathrm {f w}} + \left| T _ {\mathrm {f w}} / \left| \Pi_ {\mathrm {M I N E I G}} \right| \right| \left| \Pi_ {\mathrm {M I N E I G}} \right| \leq 2 T _ {\mathrm {f w}} + \left| \Pi_ {\mathrm {M I N E I G}} \right| \leq 3 T _ {\mathrm {f w}}
$$
where the last inequality follows since (C.7) holds. We can therefore upper bound  $\frac{4}{T_{\mathrm{fw}}} \leq \frac{12}{T_{\mathrm{out}}}$ . Putting this together proves the result.
Proof of Lemma C.8. To bound  $T_{\mathrm{out}}$ , it suffices to show that (C.7)-(C.10) are satisfied for sufficiently large  $T_{i}$ .
$$
\left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right| \leq \operatorname {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\min} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {\widetilde {N} T _ {i}}{\delta}\right),
$$
so to ensure (C.7) is met it suffices that
$$
T _ {i} \geq \operatorname {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\operatorname* {m i n}} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {\widetilde {N}}{\delta}\right).
$$
On  $\mathcal{E}_{\mathrm{exp}}$  , we have
$$
\lambda_ {\min } \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) \geq \lambda_ {\min } \left(\boldsymbol {\Gamma} _ {0} ^ {i}\right) \geq \lceil T _ {i} / \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right| \rceil \cdot 6 2 7 2 D d _ {\psi} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Which also implies
$$
\| (T _ {i} ^ {- 1} \mathbf {\Gamma} _ {0} ^ {i}) ^ {- 1} \| _ {\mathrm {o p}} = \frac {T _ {i}}{\lambda_ {\mathrm {m i n}} (\mathbf {\Gamma} _ {0} ^ {i})} \leq \frac {T _ {i}}{\lceil T _ {i} / | \Pi_ {\mathrm {M I N E I G}} ^ {i} | \rceil} \cdot \frac {1}{6 2 7 2 D d _ {\varphi} \log \frac {6 8 \tilde {N}}{\delta}} \leq \frac {| \Pi_ {\mathrm {M I N E I G}} ^ {i} |}{6 2 7 2 D d _ {\varphi} \log \frac {6 8 \tilde {N}}{\delta}}
$$
Furthermore, by Lemma C.9 we can lower bound
$$
\operatorname {t r} \left(\mathcal {H} \left(\boldsymbol {\Gamma} _ {\mathrm {f w}} ^ {i} + \boldsymbol {\Gamma} _ {0} ^ {i}\right) ^ {- 1}\right) \geq \frac {\operatorname {t r} (\mathcal {H})}{D \left(T _ {i} + \lceil T _ {i} / | \Pi_ {\mathrm {M I N E I G}} ^ {i} | \rceil | \Pi_ {\mathrm {M I N E I G}} ^ {i} |\right)} \geq \frac {\operatorname {t r} (\mathcal {H})}{3 D T _ {i}}.
$$
Combining these and using that  $N_{i} = \mathcal{O}(T_{i}^{1 / 3})$  and  $K_{i} = \mathcal{O}(T_{i}^{2 / 3})$ , it is easy to see that (C.8)-(C.10) will be met once
$$
T _ {i} \geq \operatorname {p o l y} \left(d _ {\psi}, \frac {1}{\lambda_ {\min} ^ {\star}}, D, C _ {\mathcal {R}}, \log^ {p _ {\mathcal {R}}} \frac {\widetilde {N}}{\delta}\right).
$$
The bound on  $T_{\mathrm{out}}$  then follows since  $T_{i} = \lceil 2^{i / 3}\rceil \lceil 2^{2i / 3}\rceil \in [2^{i},4\cdot 2^{i}]$ , so it can be at most a constant larger than the sufficient condition before terminating.
Let  $i^{\star}$  denote the round that Algorithm 7 terminates on. Note that at round  $i$ , MINEIG runs for at most  $|\Pi_{\mathrm{MINEIG}}^i|$ , DYNAMICOED runs for at most  $T_i$  episodes, and we run for an additional  $|T_i / |\Pi_{\mathrm{MINEIG}}^i| \cdot |\Pi_{\mathrm{MINEIG}}^i|$  episodes on Line 5. In total, then, the number of episodes Algorithm 7 runs for is bounded by
$$
\begin{array}{l} \sum_ {i = 1} ^ {i ^ {\star}} \left(T _ {i} + \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right| + \left| T _ {i} / \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right| \right| \cdot \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right|\right) \leq 2 \sum_ {i = 1} ^ {i ^ {\star}} \left(T _ {i} + \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right|\right) \\ \leq 1 6 T _ {i ^ {\star}} + 2 \sum_ {i = 1} ^ {i ^ {\star}} \left| \Pi_ {\mathrm {M I N E I G}} ^ {i} \right| \\ \end{array}
$$
where the last inequality follows since  $T_{i} \in [2^{i}, 4 \cdot 2^{i}]$ . Now note that, since Algorithm 7 only terminates once (C.7) is met, we will have  $\max_{j=1,\dots,i^{\star}} |\Pi_{\mathrm{MINEIG}}^{j}| \leq T_{i^{\star}}$ . This implies that  $2 \sum_{i=1}^{i^{\star}} |\Pi_{\mathrm{MINEIG}}^{i}| \leq 2 i^{\star} T_{i^{\star}} \leq 2 \log(T_{i^{\star}}) \cdot T_{i^{\star}}$ . Bounding  $T_{i^{\star}} \leq T_{\mathrm{out}}$  gives the result.
D Smooth Nonlinear Systems
In this section we restrict to the nonlinear regulator system of (1.1). Our goal will be to show that, under our assumptions, the nonlinear regulator system exhibits certain smooth behavior. As we have assumed
$$
\Pi^ {\star} = \left\{\pi^ {\boldsymbol {\theta}}: \boldsymbol {\theta} \in \mathbb {R} ^ {d _ {\boldsymbol {\theta}}} \right\},
$$
it will be convenient to define  $\mathcal{J}(\theta; A) \coloneqq \mathcal{J}(\pi^{\theta}; A)$  and  $\theta_{\star}(A) = \theta_{\star}$ . For the remainder of this section, we will typically use  $\theta$  in place of  $\pi^{\theta}$ . In addition, when considering radius terms such as  $r_{\theta}(A_{\star})$  and  $r_{\mathrm{cost}}(A_{\star})$ , to simplify results we assume that  $r_{\theta}(A_{\star}) \leq 1$  and  $r_{\mathrm{cost}}(A_{\star}) \leq 1$ . Note that this does not change the validity of the result since, for example, if a result holds with  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r)$  for some  $r > r_{\theta}(A_{\star})$ , it also holds for  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\theta}(A_{\star}))$ . Throughout this section, we let  $\nabla_x f(x)[\Delta]$  refer to the directional gradient of  $f(x)$  in direction  $\Delta$ .
We first have the following result, which shows that under our assumptions, the controller loss is differentiable.
Lemma D.1. Under Assumptions 1, 2, 4 and 5, for any  $A$  satisfying  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\theta}(A_{\star}))$ , the controller loss  $\mathcal{J}(\pmb{\theta}; A)$  is four-times differentiable in  $\pmb{\theta}$  and  $A$ . Furthermore, we can bound
$$
\| \nabla_ {A} ^ {(i)} \nabla_ {\boldsymbol {\theta}} ^ {(j)} \mathcal {J} (\boldsymbol {\theta}; A) \| _ {\mathrm {o p}} \leq \operatorname {p o l y} \left(\| A \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\text {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right)
$$
for  $i,j\in \{0,1,2,3,4\}$  satisfying  $1\leq i + j\leq 3$
In this section, we generalize Assumption 6 to the following.
Assumption 13. We assume there exists some  $r_{\theta}(A_{\star}) > 0$  such that, for all  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\theta}(A_{\star}))$ ,  $\theta_{\star}(A)$  satisfies:
-  $\nabla_{\pmb{\theta}}\mathcal{J}(\pmb {\theta};A)|_{\pmb {\theta} = \pmb{\theta}_{*}(A)} = 0,$
-  $\pmb{\theta}_{\star}(A)$  is three-times differentiable in  $A$ , and we can bound  $\|\nabla_{A}^{(i)}\pmb{\theta}_{\star}(A)\|_{\mathrm{op}} \leq L_{\pi_{\star}}$  for some  $L_{\pi_{\star}} > 0$  and  $i \in \{1,2,3\}$ .
The first condition requires that  $\theta_{\star}(A)$  corresponds to a stationary point of the loss. This will be met, for example, by choosing  $\theta_{\star}(A)$  to be a minima (local or global) of  $\mathcal{J}(\theta; A)$ . It is not obvious, however, that the first and second condition can be simultaneously satisfied. In the following we show that, assuming  $\nabla_{\theta}^{2}\mathcal{J}(\theta; A_{\star})|_{\theta = \theta_{\star}(A_{\star})}$  is full-rank (which will be the case, for example, when  $\theta_{\star}(A_{\star})$  is a strict local minimum of  $\mathcal{J}(\pi^{\theta}; A_{\star})$ ), there always exists some  $\theta_{\star}(A)$  satisfying both conditions of Assumption 13, with  $L_{\pi_{\star}}$  scaling polynomially in problem parameters, and  $r_{\theta}(A_{\star})$  scaling inverse polynomially in problem parameters. Note that this definition of  $\pi_{\star}(A)$  is general enough to capture settings where the global minimum of  $\mathcal{J}(\pi; A)$  cannot be efficiently computed—it suffices to take  $\pi_{\star}(A)$  a local minimum of the loss.