SlowGuess/NeurIPS_2023_OCR huggingface dataset:
always plays  $a = 0$ , so the maximum number of  $(s, a)$  pairs reachable at layer  $h \in [H]$  is  $h + 1$ . Observe that in light of Proposition 2, the spanning capacity for  $\Pi_{\mathrm{sing}}$  is "on the right order" for characterizing the minimax sample complexity for agnostic PAC RL.
Before proceeding, we note that for any policy class  $\Pi$ , the spanning capacity is always bounded.
Proposition 3. For any policy class  $\Pi$ , we have  $\mathfrak{C}(\Pi) \leq \min \{A^H, |\Pi|, SA\}$ .
Proposition 3 recovers the worst-case upper and lower bound from Section 2.2. However, for many policy classes, spanning capacity is substantially smaller than upper bound of Proposition 3. In addition to the examples we provided above, we list several additional policy classes with small spanning capacity. For these policy classes we set the state/action spaces to be  $\mathcal{S}_h = \{s_{(i,h)} : i \in [K]\}$  for all  $h \in [H]$  and  $\mathcal{A} = \{0,1\}$ , respectively. All proofs are deferred to Appendix B.
-  $\ell$ -tons: A natural generalization of singletons. We define  $\Pi_{\ell - \mathrm{ton}} := \{\pi_I : I \subset S, |I| \leq \ell\}$ , where the policy  $\pi_I$  is defined s.t.  $\pi_I(s) = \mathbb{1}\{s \in I\}$  for any  $s \in S$ . Here,  $\mathfrak{C}(\Pi_{\ell - \mathrm{ton}}) = \Theta(H^\ell)$ .
- 1-Active Policies: We define  $\Pi_{1\text{-act}}$  to be the class of policies which can take both possible actions on a single state  $s_{(1,h)}$  in each layer  $h$ , but on other states  $s_{(i,h)}$  for  $i \neq 1$  must take action 0. Formally,  $\Pi_{1\text{-act}} := \{\pi_b \mid b \in \{0,1\}^H\}$ , where for any  $b \in \{0,1\}^H$  the policy  $\pi_b$  is defined such that  $\pi_b(s) = b[h]$  if  $s = s_{(1,h)}$ , and  $\pi_b(s) = 0$  otherwise.
- All-Active Policies: We define  $\Pi_{j\text{-act}} := \{\pi_b \mid b \in \{0,1\}^H\}$ , where for any  $b \in \{0,1\}^H$  the policy  $\pi_b$  is defined such that  $\pi_b(s) = b[h]$  if  $s = s_{(j,h)}$ , and  $\pi_b(s) = 0$  otherwise. We let  $\Pi_{\mathrm{act}} := \bigcup_{j=1}^{K} \Pi_{j\text{-act}}$ . Here,  $\mathfrak{C}(\Pi_{\mathrm{act}}) = \Theta(H^2)$ .
A natural interpretation of the spanning capacity is that it represents the largest "needle in a haystack" that can be embedded in a deterministic MDP using the policy class  $\Pi$ . To see this, let  $(M^{\star}, h^{\star})$  be the MDP and layer which witnesses  $\mathfrak{C}(\Pi)$ , and let  $\{(s_i, a_i)\}_{i=1}^{\mathfrak{C}(\Pi)}$  be the set of state-action pairs reachable by  $\Pi$  in  $M^{\star}$  at layer  $h^{\star}$ . Then one can hide a reward of 1 on one of these state-action pairs; since every trajectory visits a single  $(s_i, a_i)$  at layer  $h^{\star}$ , we need at least  $\mathfrak{C}(\Pi)$  samples in order to discover which state-action pair has the hidden reward. Note that in this agnostic learning setup, we only need to care about the states that are reachable using  $\Pi$ , even though the  $h^{\star}$  layer may have other non-reachable states and actions.
3.1 Connection to Coverability
The spanning capacity has another interpretation as the worst-case coverability, a structural parameter defined in a recent work by Xie et al. (2022).
Definition 2 (Coverability, Xie et al. (2022)). For any MDP  $M$  and policy class  $\Pi$ , the coverability coefficient  $C^{\mathrm{cov}}$  is denoted
$$
C ^ {\operatorname {c o v}} (\Pi ; M) := \inf  _ {\mu_ {1}, \dots \mu_ {H} \in \Delta (\mathcal {S} \times \mathcal {A})} \sup  _ {\pi \in \Pi , h \in [ H ]} \left\| \frac {d _ {h} ^ {\pi}}{\mu_ {h}} \right\| _ {\infty} = \max  _ {h \in [ H ]} \sum_ {s, a} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s, a). \tag {2}
$$
The last equality is shown in Lemma 3 of Xie et al. (2022), and it says that the coverability coefficient is equivalent to a notion of cumulative reachability (one can check that their definition coincides with ours for deterministic MDPs).
Coverage conditions date back to the analysis of the classic Fitted Q-Iteration (FQI) algorithm (Munos, 2007; Munos and Szeptsvári, 2008), and have extensively been studied in offline RL. Various models like tabular MDPs, linear MDPs, low-rank MDPs, and exogenous MDPs satisfy the above coverage condition (Antos et al., 2008; Chen and Jiang, 2019; Jin et al., 2021b; Rashidinejad et al., 2021; Zhan et al., 2022; Xie et al., 2022), and recently, Xie et al. showed that coverability can be used to prove regret guarantees for online RL, albeit under the additional assumption of value function realizability.
It is straightforward from Definition 2 that our notion of spanning capacity is worst-case coverability when we maximize over deterministic MDPs, since for any deterministic MDP,  $\sup_{\pi \in \Pi} d_h^\pi(s, a) = \mathbb{1}\{(s, a)$  is reachable by  $\Pi$  at layer  $h\}$ . The next lemma shows that our notion of spanning capacity is exactly worst-case coverability even when we maximize over the larger class of stochastic MDPs. As a consequence, there always exists a deterministic MDP that witnesses worst-case coverability.
Lemma 1. For any policy class  $\Pi$ , we have  $\sup_{M \in \mathcal{M}^{\mathrm{sto}}} C^{\mathrm{cov}}(\Pi; M) = \mathfrak{C}(\Pi)$ .
While spanning capacity is equal to the worst-case coverability, we remark that the two definitions have different origins. The notion of coverability bridges offline and online RL, and was introduced in Xie et al. (2022) to characterize when sample efficient learning is possible in value-based RL, where the learner has access to a realizable value function class. On the other hand, spanning capacity is developed for the much weaker agnostic RL setting, where the learner only has access to a policy class (and does not have access to a realizable value function class). Note that a realizable value function class can be used to construct a policy class that contains the optimal policy, but the converse is not true. Furthermore, note that the above equivalence only holds in a worst-case sense (over MDPs). In fact, as we show in Appendix C, coverability alone is not sufficient for sample efficient agnostic PAC RL in the online interaction model.
4 Generative Model: Spanning Capacity is Necessary and Sufficient
In this section, we show that for any policy class, the spanning capacity characterizes the minimax sample complexity for agnostic PAC RL under generative model.
Theorem 1 (Upper Bound for Generative Model). For any  $\Pi$ , the minimax sample complexity  $(\varepsilon, \delta)$ -PAC learning  $\Pi$  is at most  $n_{\mathrm{gen}}(\Pi; \varepsilon, \delta) \leq \mathcal{O}\left(\frac{H \cdot \mathfrak{C}(\Pi)}{\varepsilon^2} \cdot \log \frac{|\Pi|}{\delta}\right)$ .
The proof can be found in Appendix D.1, and is a straightforward modification of the classic trajectory tree method from Kearns et al. (1999): using generative access, sample  $\mathcal{O}(\log |\Pi | / \varepsilon^2)$  deterministic trajectory trees from the MDP to get unbiased evaluations for every  $\pi \in \Pi$ ; the number of generative queries made is bounded since the size of the maximum deterministic tree is at most  $H\cdot \mathfrak{C}(\Pi)$ .
Theorem 2 (Lower Bound for Generative Model). For any  $\Pi$ , the minimax sample complexity  $(\varepsilon, \delta)$ -PAC learning  $\Pi$  is at least  $n_{\mathrm{gen}}(\Pi; \varepsilon, \delta) \geq \Omega \left( \frac{\mathfrak{C}(\Pi)}{\varepsilon^2} \cdot \log \frac{1}{\delta} \right)$ .
The proof can be found in Appendix D.2. Intuitively, given an MDP  $M^{\star}$  which witnesses  $\mathfrak{C}(\Pi)$ , one can embed a bandit instance on the relevant  $(s,a)$  pairs spanned by  $\Pi$  in  $M^{\star}$ . The lower bound follows by a reduction to the lower bound for  $(\varepsilon,\delta)$ -PAC learning in multi-armed bandits.
Together, Theorem 1 and Theorem 2 paint a relatively complete picture for the minimax sample complexity of learning any policy class  $\Pi$ , in the generative model, up to an  $H \cdot \log |\Pi|$  factor.
Deterministic MDPs. A similar guarantee holds for online RL over deterministic MDPs.
Corollary 1. Over the class of MDPs with deterministic transitions, the minimax sample complexity of  $(\varepsilon, \delta)$ -PAC learning any  $\Pi$  is
$$
\Omega \Big (\frac {\mathfrak {C} (\Pi)}{\varepsilon^ {2}} \cdot \log \frac {1}{\delta} \Big) \leq n _ {\mathrm {o n}} (\Pi , \mathcal {M} ^ {\det  \mathrm {P}}; \varepsilon , \delta) \leq \mathcal {O} \Big (\frac {H \cdot \mathfrak {C} (\Pi)}{\varepsilon^ {2}} \cdot \log \frac {| \Pi |}{\delta} \Big).
$$
The upper bound follows because the trajectory tree algorithm for deterministic transitions samples the same tree over and over again (with different stochastic rewards). The lower bound trivially extends because the lower bound of Theorem 2 actually uses an MDP  $M \in \mathcal{M}^{\mathrm{detP}}$  (in fact, the transitions of  $M$  are also known to the learner).
5 Online RL: Spanning Capacity is Not Sufficient
Given that fact that spanning capacity characterizes the minimax sample complexity of agnostic PAC RL in the generative model, one might be tempted to conjecture that spanning capacity is also the right characterization in online RL. The lower bound is clear since online RL is at least as hard as learning with a generative model, so Theorem 2 already shows that spanning capacity is necessary. But is it also sufficient?
In this section, we prove a surprising negative result showing that bounded spanning capacity by itself is not sufficient to characterize the minimax sample complexity in online RL. In particular, we provide an example for which we have a superpolynomial (in  $H$ ) lower bound on the number of trajectories needed for learning in online RL, that is not captured by any polynomial function of spanning capacity. This implies that, contrary to RL with a generative model, one can not hope for  $n_{\mathrm{on}}(\Pi; \varepsilon, \delta) = \widetilde{\Theta}(\mathrm{poly}(\mathfrak{C}(\Pi), H, \varepsilon^{-1}, \log \delta^{-1}))$  in online RL.
Theorem 3 (Lower Bound for Online RL). Fix any sufficiently large  $H$ . Let  $\varepsilon \in (1/2^{\mathcal{O}(H)}, \mathcal{O}(1/H))$  and  $\ell \in \{2, \ldots, H\}$  such that  $1/\varepsilon^\ell \leq 2^H$ . There exists a policy class  $\Pi^{(\ell)}$  of size  $\mathcal{O}(1/\varepsilon^\ell)$  with  $\mathfrak{C}(\Pi^{(\ell)}) \leq \mathcal{O}(H^{4\ell+2})$  and a family of MDPs  $\mathcal{M}$  with state space  $S$  of size  $2^{\mathcal{O}(H)}$ , binary action space, and horizon  $H$  such that: for any  $(\varepsilon, 1/8)$ -PAC algorithm, there exists an MDP  $M \in \mathcal{M}$  for which the algorithm must collect at least  $\Omega(\min\{\frac{1}{\varepsilon^\ell}, 2^{H/3}\})$  online trajectories in expectation.
Informally speaking, the lower bound shows that there exists a policy class  $\Pi$  for which  $n_{\mathrm{on}}(\Pi ;\varepsilon ,\delta) = 1 / \varepsilon^{\Omega (\log_H\mathfrak{C}(\Pi))}$ . In order to interpret this theorem, we can instantiate choices of  $\varepsilon = 1 / 2^{\sqrt{H}}$  and  $\ell = \sqrt{H}$  to show an explicit separation.
Corollary 2. For any sufficiently large  $H$ , there exists a policy class  $\Pi$  with  $\mathfrak{C}(\Pi) = 2^{\mathcal{O}(\sqrt{H}\log H)}$  such that for any  $(1 / 2^{\sqrt{H}},1 / 8)$ -PAC algorithm, there exists an MDP for which the algorithm must collect at least  $2^{\Omega (H)}$  online trajectories in expectation.
In conjunction with the results of Section 4, Theorem 3 shows that (1) online RL is strictly harder than RL with generative access, and (2) online RL for stochastic MDPs is strictly harder than online RL for MDPs with deterministic transitions. We defer the proof of Theorem 3 to Appendix E. Our lower bound introduces several technical novelties: the family  $\mathcal{M}$  utilizes a contextual variant of the combination lock, and the policy class  $\Pi$  is constructed via a careful probabilistic argument such that it is hard to explore despite having small spanning capacity.
6 Statistically Efficient Agnostic Learning in Online RL
The lower bound in Theorem 3 suggests that further structural assumptions on  $\Pi$  are needed for statistically efficient agnostic RL under the online interaction model. Essentially, the lower bound example provided in Theorem 3 is hard to agnostically learn because any two distinct policies  $\pi, \pi' \in \Pi$  can differ substantially on a large subset of states (of size at least  $\varepsilon \cdot 2^{2H}$ ). Thus, we cannot hope to learn "in parallel" via a low variance IS strategy that utilizes extrapolation to evaluate all policies  $\pi \in \Pi$ , as we did for singletons.
In the sequel, we consider the following sunflower property to rule out such problematic scenarios, and show how bounded spanning capacity along with the sunflower property enable sample-efficient agnostic RL in the online interaction model. The sunflower property only depends on the state space, action space, and policy class, and is independent of the transition dynamics and rewards of the underlying MDP. We first define a petal, a key ingredient of a sunflower.
Definition 3 (Petal). For a policy set  $\bar{\Pi}$ , and states  $\bar{S} \subseteq S$ , a policy  $\pi$  is said to be a  $\bar{S}$ -petal on  $\bar{\Pi}$  if for all  $h \leq h' \leq H$ , and partial trajectories  $\tau = (s_h, a_h, \dots, s_{h'}, a_{h'})$  that are consistent with  $\pi$ : either  $\tau$  is also consistent with some  $\pi' \in \bar{\Pi}$ , or there exists  $i \in (h, h']$  s.t.  $s_i \in \bar{S}$ .
Informally,  $\pi$  is a  $\bar{S}$ -petal on  $\bar{\Pi}$  if any trajectory that can be obtained using  $\pi$  can either also be obtained using a policy in  $\bar{\Pi}$  or must pass through  $\bar{S}$ . Thus, any policy that is a  $\bar{S}$ -petal on  $\bar{\Pi}$  can only differentiate from  $\bar{\Pi}$  in a structured way. A policy class is said to be a sunflower if it is a union of petals as defined below:
Definition 4 (Sunflower). A policy class  $\Pi$  is said to be a  $(K, D)$ -sunflower if there exists a set  $\Pi_{\mathrm{core}}$  of Markovian policies with  $|\Pi_{\mathrm{core}}| \leq K$  such that for every policy  $\pi \in \Pi$  there exists a set  $\mathcal{S}_{\pi} \subseteq \mathcal{S}$  of size at most  $D$ , so that  $\pi$  is an  $S_{\pi}$ -petal on  $\Pi_{\mathrm{core}}$ .
Our next theorem provides a sample complexity bound for Agnostic PAC RL for policy classes that have  $(K, D)$ -sunflower structure. This bound is obtained via a new exploration algorithm called POPLER that takes as input the set  $\Pi_{\mathrm{core}}$  and corresponding petals  $\{S_{\pi}\}_{\pi \in \Pi}$  and leverages importance sampling as well as reachable state identification techniques to simultaneously estimate the value of every policy in  $\Pi$ .
Theorem 4. Let  $\varepsilon, \delta > 0$ . Suppose the policy class  $\Pi$  satisfies Definition 1 with spanning capacity  $\mathfrak{C}(\Pi)$ , and is a  $(K, D)$ -sunflower. Then, for any MDP  $M$ , with probability at least  $1 - \delta$ , POPLER (Algorithm 1) succeeds in returning a policy  $\widehat{\pi}$  that satisfies  $V^{\widehat{\pi}} \geq \max_{\pi \in \Pi} V^{\pi} - \varepsilon$ , after collecting
$$
\widetilde {\mathcal {O}} \left(\left(\frac {1}{\varepsilon^ {2}} + \frac {H D ^ {6} \mathfrak {C} (\Pi)}{\varepsilon^ {4}}\right) \cdot K ^ {2} \log \frac {| \Pi |}{\delta}\right) \quad \text {o n l i n e t r a j e c t o r i e s i n} M.
$$
The proof of Theorem 4, and the corresponding hyperparameters in POPLER needed to obtain the above bound, can be found in Appendix F. Before diving into the algorithm and proof details, let us highlight several key aspects of the above sample complexity bound:
- Note that a class  $\Pi$  may be a  $(K, D)$ -sunflower for many different choices of  $K$  and  $D$ . Barring computational issues, one can enumerate over all choices of  $\Pi_{\mathrm{core}}$  and  $\{S_{\pi}\}_{\pi \in \Pi}$ , and check if  $\Pi$  is a  $(K, D)$ -sunflower for that choice of  $K = |\Pi_{\mathrm{core}}|$  and  $D = \max_{\pi \in \Pi} |S_{\pi}|$ . Since our bound in Theorem 4 scales with  $K$  and  $D$ , we are free to choose  $K$  and  $D$  to minimize the corresponding sample complexity bound.
- In order to get a polynomial sample complexity in Theorem 4, both  $\mathfrak{C}(\Pi)$  and  $(K,D)$  are required to be poly  $(H,\log |\Pi |)$ . All of the policy classes considered in Section 3 have the sunflower property, with both  $K,D = \mathrm{poly}(H)$ , and thus our sample complexity bound extends for all these classes. See Appendix B for details.
- Notice that for Theorem 4 to hold, we need both bounded spanning capacity as well as the sunflower structure on the policy class with bounded  $(K, D)$ . Thus, one may wonder if we can obtain a similar polynomial sample complexity guarantee in online RL under weaker assumptions. In Theorem 3, we already showed that bounded  $\mathfrak{C}(\Pi)$  alone is not sufficient to obtain polynomial sample complexity in online RL. Likewise, as we show in Appendix F, sunflower property with bounded  $(K, D)$  alone is also not sufficient for polynomial sample complexity, and hence both assumptions cannot be individually removed. However, it is an interesting question if there is some other structural assumption that combines both spanning capacity and the sunflower property, and is both sufficient and necessary for agnostic PAC learning in online RL. See Section 7 for further discussions on this.
Why Does the Sunflower Property Enable Sample-Efficient Learning? Intuitively, the sunflower property captures the intuition of simultaneous estimation of all policies  $\pi \in \Pi$  via importance sampling (IS), and allows control of both bias and variance. Let  $\pi$  be a  $S_{\pi}$ -petal on  $\Pi_{\mathrm{core}}$ . Any trajectory  $\tau \rightsquigarrow \pi$  that avoids  $S_{\pi}$  must be consistent with some policy in  $\Pi_{\mathrm{core}}$ , and will thus be covered by the data collected using  $\pi' \sim \mathrm{Uniform}(\Pi_{\mathrm{core}})$ . Thus, using IS with variance scaling with  $K$ , one can create a biased estimator for  $V^{\pi}$ , where the bias is only due to trajectories that pass through  $S_{\pi}$ . There are two cases. If every state in  $S_{\pi}$  has small reachability under  $\pi$ , i.e.  $d^{\pi}(s) \ll \varepsilon$  for every  $s \in S_{\pi}$ , then the IS estimate will have a low bias (linear in  $|S_{\pi}|$ ), so we can compute  $V^{\pi}$  up to error at most  $\varepsilon |S_{\pi}|$ . On the other hand, if  $d^{\pi}(s)$  is large for some  $s \in S_{\pi}$ , it is possible to explicitly control the bias that arises from trajectories passing through them since there are at most  $D$  of them.
6.1 Algorithm and Proof Ideas
POPLER (Algorithm 1) takes as input a policy class  $\Pi$ , as well as sets  $\Pi_{\mathrm{core}}$  and  $\{S_{\pi}\}_{\pi \in \Pi}$ , which can be computed beforehand by enumeration. POPLER has two phases: a state identification phase, where it finds "petal" states  $s \in \bigcup_{\pi \in \Pi} S_{\pi}$  that are reachable with sufficiently large probability; and an evaluation phase where it computes estimates  $\hat{V}^{\pi}$  for every  $\pi \in \Pi$ . It uses three subroutines DataCollector, EstReachability, and Evaluate, whose pseudocodes are stated in Appendix F.2.
The structure of the algorithm is reminiscent of reward-free exploration algorithms in tabular RL, e.g. Jin et al. (2020), where we first identify (petal) states that are reachable with probability at least  $\Omega (\varepsilon /D)$  and build a policy cover for these states, and then use dynamic programming to estimate the values. However, contrary to the classical tabular RL setting, because the state space can be large, our setting is much more challenging and necessitates technical innovations. In particular, we can no longer enumerate over all petal states and check if they are sufficiently reachable by some policy  $\pi \in \Pi$  (since the total number of petal states  $\sum_{\pi \in \Pi}|S_{\pi}|$  could scale linearly in  $|\Pi|$ , a factor that we do not want to appear in our sample complexity). Instead, the key observation that we rely on is that if spanning capacity is bounded, then by the equivalence of spanning capacity and worst-case coverability (Lemma 1) and due to (2), the number of highly reachable (and thus relevant) petal states is also bounded. Thus, we only need to build a policy cover to reach these relevant petal states. Our algorithm does this in a sample-efficient sequential manner. For both state identification as well as evaluation, we interleave importance sampling estimates with the construction of a policy-specific Markov Reward Processes (MRPs), which are defined for every  $\pi \in \Pi$ . The challenge is doing all of this "in parallel" for every  $\pi \in \Pi$  through extensive sample reuse to avoid a blowup of  $|\Pi|$  or  $S$  in the sample complexity. For the exact construction of these policy-specific MRPs and how they are used in the algorithm, please see Appendix F.2.
Algorithm 1 Policy OPtimization by Learning  $\varepsilon$ -Reachable States (POPLER)
Require: Policy class  $\Pi$ , Sets  $\Pi_{\mathrm{core}}$  and  $\{\mathcal{S}_{\pi}\}_{\pi \in \Pi}$ , Parameters  $K, D, n_1, n_2, \varepsilon, \delta$ .  
1: Define an additional start state  $s_\top$  (at  $h = 0$ ) and end state  $s_\perp$  (at  $h = H + 1$ ).  
2: Initialize  $\mathcal{S}^{\mathrm{rch}} = \{s_\top\}$ ,  $\mathcal{T} \gets \{(s_\top, \mathrm{Null})\}$ , and for every  $\pi \in \Pi$ , define  $\mathcal{S}_{\pi}^{+} := \mathcal{S}_{\pi} \cup \{s_\top, s_\perp\}$ .  
3:  $\mathcal{D}_T \gets \text{DataCollector}(s_\top, \mathrm{Null}, \Pi_{\mathrm{core}}, n_1)$   
4: /* Identification of Petal States that are Reachable with  $\Omega(\varepsilon/D)$  Probability */  
5: while Terminate = False do  
6: Set Terminate = True.  
7: for  $\pi \in \Pi$  do  
8: Compute the set of already explored reachable states  $\mathcal{S}_{\pi}^{\mathrm{rch}} = \mathcal{S}_{\pi}^{+} \cap \mathcal{S}^{\mathrm{rch}}$ , and the remaining states  $\mathcal{S}_{\pi}^{\mathrm{rem}} = \mathcal{S}_{\pi} \setminus (\mathcal{S}_{\pi}^{\mathrm{rch}} \cup \{s_\perp\})$ .  
9: Estimate the policy-specific MRP  $\widehat{\mathfrak{M}}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$  according to (14) and (15).  
10: for  $\bar{s} \in S_{\pi}^{\mathrm{rem}}$  do  
11: Estimate probability of reaching  $\bar{s}$  under  $\pi$  as  $\widehat{d}^{\pi}(\bar{s}) \gets \text{EstReachability}(\mathcal{S}_{\pi}^{+}, \widehat{\mathfrak{M}}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}, \bar{s})$ .  
12: if  $\widehat{d}^{\pi}(\bar{s}) \geq \varepsilon/6D$  then  
13: Update  $\mathcal{S}^{\mathrm{rch}} \gets \mathcal{S}^{\mathrm{rch}} \cup \{\bar{s}\}$ ,  $\mathcal{T} \gets \mathcal{T} \cup \{(\bar{s}, \pi)\}$  and set Terminate = False.  
14: Collect dataset  $\mathcal{D}_{\bar{s}} \gets \text{DataCollector}(\bar{s}, \pi, \Pi_{\mathrm{core}}, n_2)$ .  
15: end if  
16: end for  
17: end for  
18: end while  
19: /* Policy Evaluation and Optimization */  
20: for  $\pi \in \Pi$  do  
21:  $\widehat{V}^{\pi} \gets \text{Evaluate}(\Pi_{\mathrm{core}}, \mathcal{S}^{\mathrm{rch}}, \{\mathcal{D}_s\}_{s \in S^{\mathrm{rch}}}, \pi)$ .  
22: end for  
23: Return  $\widehat{\pi} \in \arg \max_{\pi} \widehat{V}^{\pi}$ .