SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\frac {1}{| \Pi_ {\text {c o r e}} |} \sum_ {\pi^ {\prime} \in \Pi_ {\text {c o r e}}} \mathbb {1} \left\{\pi^ {\prime} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\begin{array}{c}\tau_ {h: h ^ {\prime}} \text {g o e s f r o m s t o} s ^ {\prime}\\\text {w i t h o u t p a s s i n g t h r o u g h a n y o t h e r} s ^ {\prime \prime} \in S _ {\pi}\end{array}\right\}. \tag {14}
$$
Transition  $\widehat{R}_{s\rightarrow s^{\prime}}^{\pi}$  as:
$$
\widehat {R} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {R \left(\tau_ {h : h ^ {\prime}}\right) \cdot \mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\frac {1}{| \Pi_ {\text {c o r e}} |} \sum_ {\pi^ {\prime} \in \Pi_ {\text {c o r e}}} \mathbb {1} \left\{\pi^ {\prime} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\begin{array}{c}\tau_ {h: h ^ {\prime}} \text {g o e s f r o m s t o s ^ {\prime}}\\\text {w i t h o u t p a s s i n g t h r o u g h a n y o t h e r s ^ {\prime \prime}} \in S _ {\pi}\end{array}\right\}. \tag {15}
$$
Clearly, the above definition implies that  $\widehat{P}_{s\to s^{\prime}}^{\pi} = 0$  and  $\widehat{R}_{s\rightarrow s^{\prime}}^{\pi} = 0$  for any  $s\notin S_{\pi}^{\mathrm{rch}}$  since  $\mathcal{D}_s$  would be empty corresponding to these unexplored states. Furthermore,  $P_{s_\perp \to s_\perp}^\pi = 1$  and  $R_{s_\perp \to s_\perp}^\pi = 0$ .
Note that since the set  $S^{\mathrm{rch}}$  is changing in each iteration of the loop as the algorithm collects more data, the policy-specific MRP  $\widehat{\mathfrak{M}}_{S^{\mathrm{rch}}}^{\pi}$  also changes in every iteration of the loop — in particular, more and more transitions/rewards are assigned nonzero values due to new states being added to  $S^{\mathrm{rch}}$ . More details on policy-specific MRPs is given in Appendix F.2.
The key advantage of constructing the empirical versions of policy-specific MRPs is that they allow us to explore and find new leaf states in  $S_{\pi}$  which are reachable with probability at least  $\Omega(\varepsilon/6D)$ . In particular, using standard dynamic programming (subroutine EstReachability), we can check whether a candidate petal  $\bar{s}$  is reachable with decent probability by  $\pi$  (lines 11-12); If it is, then we add  $(\bar{s}, \pi)$  to the set  $\mathcal{T}$  and collect a fresh dataset using DataCollector (lines 13-14). Crucially, the importance sampling technique enables us to be sample efficient, since the same dataset  $\mathcal{D}_s$  can be used to evaluate transitions/rewards in Eqs. (14) and (15) for multiple  $\pi \in \Pi$  for which  $s$  is a petal state. Furthermore, the number of such datasets we collect must be bounded—each  $(s, \pi_s) \in \mathcal{T}$  contributes  $\Omega(\varepsilon/D)$  to cumulative reachability, but since cumulative reachability is bounded from above by  $\mathfrak{C}(\Pi)$  (Lemma 1), we know that  $|\mathcal{T}| \leq \mathcal{O}(D \cdot \mathfrak{C}(\Pi)/\varepsilon)$ .
Evaluation Phase
Next, POPLER moves to the evaluation phase. Using the collected data, it executes the Evaluate subroutine for every  $\pi \in \Pi$  to get estimates  $\widehat{V}^{\pi}$  (line 21) corresponding to  $V^{\pi}$ . For a given  $\pi \in \Pi$ , the Evaluate subroutine also constructs an empirical policy-specific MRP  $\widehat{\mathfrak{M}}_{S^{\mathrm{rch}}}^{\pi}$  for every  $\pi \in \Pi$  and computes the value of  $\pi$  via dynamic programming on  $\widehat{\mathfrak{M}}_{S^{\mathrm{rch}}}^{\pi}$ . While the returned estimate  $\widehat{V}^{\pi}$  is biased, in the complete proof, we will show that the bias is negligible since it is now only due to the states in the petal  $S_{\pi}$  which are not  $\Omega(\varepsilon / D)$ -reachable. Thus, we can guarantee that  $\widehat{V}^{\pi}$  closely estimates  $V^{\pi}$  for every  $\pi \in \Pi$ , and therefore POPLER returns a near-optimal policy.
F.2 Algorithmic Details and Preliminaries
In this subsection, we provide the details of the subroutines that do not appear in the main body, in Algorithms 3, 4, and 5. The transitions and reward functions in line 5 in Algorithm 5 are computed using Eqs. (19) and (20), which are specified below, after introducing additional notation.
Algorithm 3 DataCollector
Require: State:  $s$ , Reacher policy:  $\pi_s$ , Exploration policy set:  $\Pi_{\mathrm{core}}$ , Number of samples:  $n$ .  
1: if  $s = s_{\top}$  then // Uniform sampling for start state  $s_{\top}$   
2: for  $t = 1, \ldots, n$  do  
3: Sample  $\pi^{\prime} \sim \mathrm{Uniform}(\Pi_{\mathrm{core}})$ , and run  $\pi^{\prime}$  to collect  $\tau = (s_1, a_1, \dots, s_H, a_H)$ .  
4:  $\mathcal{D}_s \gets \mathcal{D}_s \cup \{\tau\}$ .  
5: end for  
6: else //  $\pi_s$ -based sampling for all other states  $s \neq s_{\top}$   
7: Identify the layer  $h$  such that  $s \in \mathcal{S}_h$ .  
8: for  $t = 1, \ldots, n$  do  
9: Run  $\pi_s$  for the first  $h - 1$  time steps, and collect trajectory  $(s_1, a_1, \dots, s_{h-1}, a_{h-1}, s_h)$ .  
10: if  $s_h = s$  then  
11: Sample  $\pi^{\prime} \sim \mathrm{Uniform}(\Pi_{\mathrm{core}})$ , and run  $\pi^{\prime}$  to collect remaining  $(s_h, a_h, \dots, s_H, a_H)$ .  
12:  $\mathcal{D}_s \gets \mathcal{D}_s \cup \{\tau = (s_1, a_1, \dots, s_H, a_H)\}$ .  
13: end if  
14: end for  
15: end if  
16: Return dataset  $\mathcal{D}_s$ .
Algorithm 4 EstReachability
Require: State space  $S^{\mathrm{tab}}$ , MRP  $\mathfrak{M}$ , State  $\bar{s} \in S^{\mathrm{tab}}$ .  
1: Let  $P$  be the transition of  $\mathfrak{M}$ .  
2: Initialize  $V(s) = \mathbb{1}\{s = \bar{s}\}$  for all  $s \in S^{\mathrm{tab}}$ .  
3: Repeat  $H + 1$  times:  
4: For all  $s \in S^{\mathrm{tab}}$ , calculate  $V(s) \gets \sum_{s' \in S^{\mathrm{tab}}} P_{s \rightarrow s'} \cdot V(s')$ . // Dynamic Programming  
5: Return  $V(s_{\top})$ .
We recall the definition of petals and sunflowers given in the main body (in Definitions 3 and 4). In the rest of this section, we assume that  $\Pi$  is a  $(K, D)$ -sunflower with  $\Pi_{\mathrm{core}}$  and  $S_{\pi}$  for any  $\pi \in \Pi$ .
Definition 6 (Petals and Sunflowers (Definitions 3 and 4 in the main body)). For a policy set  $\bar{\Pi}$ , and states  $\bar{S} \subseteq S$ , a policy  $\pi$  is said to be a  $\bar{S}$ -petal on  $\bar{\Pi}$  if for all  $h \leq h' \leq H$ , and partial trajectories  $\tau = (s_h, a_h, \dots, s_{h'}, a_{h'})$  that are consistent with  $\pi$ : either  $\tau$  is also consistent with some  $\pi' \in \bar{\Pi}$ , or there exists  $i \in (h, h']$  s.t.  $s_i \in \bar{S}$ .
A policy class  $\Pi$  is said to be a  $(K, D)$ -sunflower if there exists a set  $\Pi_{\mathrm{core}}$  of Markovian policies with  $|\Pi_{\mathrm{core}}| \leq K$  such that for every policy  $\pi \in \Pi$  there exists a set  $S_{\pi} \subseteq S$ , of size at most  $D$ , so that  $\pi$  is an  $S_{\pi}$ -petal on  $\Pi_{\mathrm{core}}$ .
Additional notation. Recall that we assumed that the state space  $\mathcal{S} = \mathcal{S}_1 \times \ldots \mathcal{S}_H$  is layered. Thus, given a state  $s$ , we can infer the layer  $h$  such that  $s \in \mathcal{S}_h$ . By definition  $s_\top$  belongs to the layer  $h = 0$  and  $s_\perp$  belongs to the layer  $h = H$ . In the following, we define additional notation:
Algorithm 5 Evaluate
Require: Policy set  $\Pi_{\mathrm{core}}$ , Reachable states  $\mathcal{S}^{\mathrm{rch}}$ , Datasets  $\{\mathcal{D}_s\}_{s\in \mathcal{S}^{\mathrm{rch}}}$ , Policy  $\pi$  to be evaluated.
1: Compute  $\mathcal{S}_{\pi}^{\mathrm{rch}} \gets \mathcal{S}_{\pi}^{+} \cap \mathcal{S}^{\mathrm{rch}}$  and  $S^{\mathrm{tab}} = \mathcal{S}_{\pi}^{\mathrm{rch}} \cup \{s_{\perp}\}$ .
2: for  $s, s'$  in  $S^{\mathrm{tab}}$  do // Compute transitions and rewards on  $S^{\mathrm{tab}}$
3: Let  $h, h'$  be such that  $s \in S_h$  and  $s' \in S_{h'}$ .
4: if  $h < h'$  then
5: Calculate  $\widetilde{P}_{s\rightarrow s^{\prime}}^{\pi}$ $\widehat{r}_{s\to s^{\prime}}^{\pi}$  according to (19) and (20);
6: else
7: Set  $\widetilde{P}_{s\rightarrow s^{\prime}}^{\pi}\gets 0,\widehat{r}_{s\rightarrow s^{\prime}}^{\pi}\gets 0.$
8: end if
9: end for
0: Set  $\tilde{V}(s) = 0$  for all  $s \in S^{\mathrm{tab}}$ .
1: Repeat for  $H + 1$  times:
2: For all  $s \in S^{\mathrm{tab}}$ , calculate  $\widehat{V}(s) \gets \sum_{S^{\mathrm{tab}}} \widehat{P}_{s \to s'}^{\pi} \cdot \left( \widehat{r}_{s \to s'}^{\pi} + \widehat{V}(s') \right)$ .
13: Return  $\widehat{V}(s_{\top})$ .
(a) Sets  $\mathfrak{T}(s\to s^{\prime};\neg \bar{S})$  : For any set  $\bar{S}$ , and states  $s,s'\in S$ , we define  $\mathfrak{T}(s\to s';\neg \bar{S})$  as the set of all the trajectories that go from  $s$  to  $s^\prime$  without passing through any state in  $\bar{S}$  in between. More formally, let state  $s$  be at layer  $h$ , and  $s^\prime$  be at layer  $h^\prime$ . Then,  $\mathfrak{T}(s\to s';\neg \bar{S})$  denotes the set of all the trajectories  $\tau = (s_1,a_1,\dots ,s_H,a_H)$  that satisfy all of the following:
-  $s_h = s$ , where  $s_h$  is the state at timestep  $h$  in  $\tau$ .
-  $s_{h'} = s'$ , where  $s_{h'}$  is the state at timestep  $h'$  in  $\tau$ .
- For all  $h < \widetilde{h} < h'$ , the state  $s_{\widetilde{h}}$ , at time step  $\widetilde{h}$  in  $\tau$ , does not lie in the set  $\bar{S}$ .
Note that when  $h' \leq h$ , we define  $\mathfrak{T}(s \to s'; \neg \bar{S}) = \emptyset$ . Additionally, we define  $\mathfrak{T}(s_{\top} \to s; \neg \bar{S})$  as the set of all trajectories that go to  $s'$  (from a start state) without going through any state in  $\bar{S}$  in between. Finally, we define  $\mathfrak{T}(s \to s_{\perp}; \neg \bar{S})$  as the set of all the trajectories that go from  $s$  at time step  $h$  to the end of the episode without passing through any state in  $\bar{S}$  in between.
Furthermore, we use the shorthand  $\mathfrak{T}_{\pi}(s \to s') := \mathfrak{T}(s \to s'; \neg S_{\pi})$  to denote the set of all the trajectories that go from  $s$  to  $s'$  without passing though any leaf state  $S_{\pi}$ .
(b) Using the above notation, for any  $s \in S$  and set  $\bar{S} \subseteq S$ , we define  $\bar{d}^{\pi}(s; \neg \bar{S})$  as the probability of reaching  $s$  (from a start state) without passing through any state in  $\bar{S}$  in between, i.e.
$\bar{d}^{\pi}(s; \neg \bar{S}) = \mathbb{P}^{\pi}\left[\tau$  reaches  $s$  without passing through any state in  $\bar{S}$  before reaching  $s\right]$
$$
= \mathbb {P} ^ {\pi} [ \tau \in \mathfrak {T} (s _ {\top} \rightarrow s; \neg \bar {\mathcal {S}}) ]. \tag {16}
$$
We next recall the notation of Markov Reward Process and formally define both the population versions of policy-specific MRPs.
Markov Reward Process (MRP). A Markov reward process  $\mathfrak{M} = \mathrm{MRP}(S, P, R, H, s_{\top}, s_{\bot})$  is defined over the state space  $S$  with start state  $s_{\top}$  and end state  $s_{\bot}$ , for trajectory length  $H + 2$ . Without loss of generality, we assume that  $\{s_{\top}, s_{\bot}\} \in S$ . The transition kernel is denoted by  $P: S \times S \to [0,1]$ , such that for any  $s \in S$ ,  $\sum_{s'} P_{s \to s'} = 1$ ; the reward kernel is denoted  $R: S \times S \to \Delta([0,1])$ . Throughout, we use the notation  $\rightarrow$  to signify that the transitions and rewards are defined along the edges of the MRP.
A trajectory in  $\mathfrak{M}$  is of the form  $\tau = (s_{\top}, s_1, \dots, s_H, s_{\bot})$ , where  $s_h \in S$  for all  $h \in [H]$ . Furthermore, from any state  $s \in S$ , the MRP transitions to another state  $s' \in S$  with probability  $P_{s \to s'}$ , and
<sup>7</sup>Our definition of Markov Reward Processes (MRP) deviates from MDPs that we considered in the paper, in the sense that we do not assume that the state space  $S$  is layered in an MRP. This variation is only adapted to simplify the proofs and the notation in the rest of the paper.
obtains the rewards  $r_{s\to s'}\sim R_{s\to s'}$  . Thus,
$$
\mathbb {P} ^ {\mathfrak {M}} [ \tau ] = P _ {s _ {\top} \to s _ {1}} \cdot \left(\prod_ {h = 1} ^ {H - 1} P _ {s _ {h} \to s _ {h + 1}}\right) \cdot P _ {s _ {H} \to s _ {\perp}},
$$
and the rewards
$$
R ^ {\mathfrak {M}} (\tau) = r _ {s _ {\top} \rightarrow s _ {1}} + \sum_ {h = 1} ^ {H} r _ {s _ {h} \rightarrow s _ {h + 1}} + r _ {s _ {H} \rightarrow s _ {\perp}}.
$$
Furthermore, in all the MRPs that we consider in the paper, we have  $P_{s_\perp \to s_\perp} = 1$  and  $r_{s_\perp \to s_\perp} = 0$ .
Policy-Specific Markov Reward Processes. A key technical tool in our analysis will be policy-specific MRPs that depend on the set  $S^{\mathrm{rch}}$  of the states that we have explored so far. Recall that for any policy  $\pi$ ,  $S_{\pi}^{+} = S_{\pi} \cup \{s_{\top}, s_{\bot}\}$ ,  $S_{\pi}^{\mathrm{rch}} = S_{\pi}^{+} \cap S^{\mathrm{rch}}$  and  $S_{\pi}^{\mathrm{rem}} = S_{\pi}^{+} \setminus (S_{\pi}^{\mathrm{rch}} \cup \{s_{\bot}\})$ . We define the expected and the empirical versions of policy-specific MRPs below; see Figure 2 for an illustration.
(a) Expected Version of Policy-Specific MRP. We define  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$  =  $\mathrm{MRP}(S_{\pi}^{+}, P^{\pi}, r^{\pi}, H, s_{\top}, s_{\bot})$  where
- Transition Kernel  $P^{\pi}$ : For any  $s \in S_{\pi}^{\mathrm{rch}}$  and  $s' \in S_{\pi}^{+}$ , we have
$$
P _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \mathbb {E} ^ {\pi} \left[ \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\} \mid s _ {h} = s \right], \tag {17}
$$
where the expectation above is w.r.t. the trajectories drawn using  $\pi$  in the underlying MDP, and  $h$  denotes the time step such that  $s\in S_h$  (again, in the underlying MDP). Thus, the transition  $P_{s\to s^{\prime}}^{\pi}$  denotes the probability of taking policy  $\pi$  from  $s$  and directly transiting to  $s^\prime$  without visiting any other states in  $S_{\pi}$ . Furthermore,  $P_{s\rightarrow s^{\prime}}^{\pi} = \mathbb{1}\{s^{\prime} = s_{\perp}\}$  for all  $s\in S_{\pi}^{\mathrm{rem}}\cup \{s_{\perp}\}$ .
- Reward Kernel  $r^\pi$ : For any  $s \in S_\pi^{\mathrm{rch}}$  and  $s' \in S_\pi^+$ , we have
$$
r _ {s \rightarrow s ^ {\prime}} ^ {\pi} := \mathbb {E} ^ {\pi} \left[ R \left(\tau_ {h: h ^ {\prime}}\right) \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} \left(s \rightarrow s ^ {\prime}\right)\right\} \mid s _ {h} = s \right], \tag {18}
$$
where  $R(\tau_{h:h'})$  denotes the reward for the partial trajectory  $\tau_{h:h'}$  in the underlying MDP. The reward  $r_{s\rightarrow s'}^{\pi}$  denotes the expectation of rewards collected by taking policy  $\pi$  from  $s$  and directly transiting to  $s'$  without visiting any other states in  $S_{\pi}$ . Furthermore,  $r_{s\rightarrow s'}^{\pi} = 0$  for all  $s\in S_{\pi}^{\mathrm{rem}}\cup \{s_{\perp}\}$ .
Throughout the analysis, we use  $\mathbb{P}^{\mathfrak{M}}[\cdot] := \mathbb{P}^{\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}}[\cdot]$  and  $\mathbb{E}^{\mathfrak{M}}[\cdot] := \mathbb{E}^{\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}}[\cdot]$  as a shorthand, whenever clear from the context.
(b) Empirical Version of Policy-Specific MRPs. Since the learner only has sampling access to the underlying MDP, it can not directly construct the MRP  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$ . Instead, in Algorithm 1, the learner constructs an empirical estimate for  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$ , defined as  $\widehat{\mathfrak{M}}_{\mathcal{S}^{\mathrm{rch}}}^{\pi} = \mathrm{MRP}(S_{\pi}^{+}, \widehat{P}^{\pi}, \widehat{r}^{\pi}, H, s_{\top}, s_{\bot})$  where
Transition Kernel  $\widehat{P}^{\pi}$ : For any  $s\in S_{\pi}^{\mathrm{rch}}$  and  $s^\prime \in S_\pi^+$ , we have
$$
\widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {| \Pi_ {\text {c o r e}} |}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\sum_ {\pi^ {\prime} \in \Pi_ {\text {c o r e}}} \mathbb {1} \left\{\pi^ {\prime} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\}, \tag {19}
$$
where  $\Pi_{\mathrm{core}}$  denotes the core of the sunflower corresponding to  $\Pi$  and  $\mathcal{D}_s$  denotes a dataset of trajectories collected via DataCollector(s,  $\pi_s,\Pi_{\mathrm{core}},n_2$ ). Furthermore,  $\widehat{P}_{s\rightarrow s^{\prime}}^{\pi} = \mathbb{1}\{s^{\prime} = s_{\perp}\}$  for all  $s\in S_{\pi}^{\mathrm{rem}}\cup \{s_{\perp}\}$ .
- Reward Kernel  $\widehat{r}^{\pi}$ : For any  $s\in S_{\pi}^{\mathrm{rch}}$  and  $s^\prime \in S_\pi^+$ , we have
$$
\hat {r} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {| \Pi_ {\mathrm {c o r e}} |}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\sum_ {\pi^ {\prime} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \left\{\pi^ {\prime} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\} R \left(\tau_ {h: h ^ {\prime}}\right), \tag {20}
$$
where  $\Pi_{\mathrm{core}}$  denotes the core of the sunflower corresponding to  $\Pi$ ,  $\mathcal{D}_s$  denotes a dataset of trajectories collected via DataCollector(s,  $\pi_s$ ,  $\Pi_{\mathrm{core}}$ ,  $n_2$ ), and  $R(\tau_{h:h'}) = \sum_{i=h}^{h' - 1} r_i$ . Furthermore,  $\hat{r}_{s \to s'}^{\pi} = 0$  for all  $s \in S_{\pi}^{\mathrm{rem}}$ .
The above approximates the MRP given by (14) and (15) in the main body.
Figure 2: Illustration of an MRP  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$  with  $S_{\pi} = \{s_1,s_2,s_3\}$  and  $S^{\mathrm{rch}} = \{s_1,s_2\}$ . In the original MDP  $M$ ,  $s_1\in S_1$ ,  $s_2\in S_3$ , and  $s_3\in S_2$ . The edges are labeled with the values of  $P_{s\to s'}^{\pi}$  and  $R_{s\to s'}^{\pi}$ . Notice that (1) there are no edges from  $s_2\rightarrow s_3$  or  $s_3\rightarrow s_2$  because trajectories cannot go from later layers to earlier ones; (2) since  $s_3\notin S^{\mathrm{rch}}$ , there is no edge from  $s_3\rightarrow s_2$ , and instead we have  $P_{s_3\to s_\perp}^\pi = 1$  and  $R_{s_3\to s_\perp}^\pi = 0$ ; (3)  $s_\perp$  is an absorbing state with no rewards.
Properties of Trajectories in the Policy-Specific MRPs. We state several properties of trajectories in the policy-specific MRPs, which will be used in the proofs. Let  $\tau = (s_{\top}, s_1, \dots, s_H, s_{\bot})$  denote a trajectory from either  $\mathfrak{M}_{\mathcal{S}^{\mathrm{reh}}}^{\pi}$  or  $\widehat{\mathfrak{M}}_{\mathcal{S}^{\mathrm{reh}}}^{\pi}$ .
(1) For some  $k \leq H$  we have  $s_1, \dots, s_k \in S_\pi$  and  $s_{k+1} = \dots = s_H = s_\perp$  (if  $k = H$  we say the second condition is trivially met).
(2) Each state in  $s_1, \dots, s_k$  is unique.
(3) Letting  $\mathsf{h}(s)$  denote the layer that a state  $s \in S_{\pi}$  is in, we have  $\mathsf{h}(s_1) < \dots < \mathsf{h}(s_k)$ .
(4) Either (a)  $s_1, \dots, s_k \in S_{\pi}^{\mathrm{rch}}$ , or (b)  $s_1, \dots, s_{k-1} \in S_{\pi}^{\mathrm{rch}}$  and  $s_k \in S_{\pi}^{\mathrm{rem}}$ .
Parameters Used in Algorithm 1. Here, we list all the parameters that are used in Algorithm 1 and its subroutines:
$$
n _ {1} = C _ {1} \frac {(D + 1) ^ {4} K ^ {2} \log (| \Pi | (D + 1) / \delta)}{\varepsilon^ {2}},
$$
$$
n _ {2} = C _ {2} \frac {D ^ {3} (D + 1) ^ {2} K ^ {2} \log \left(\left| \Pi \right| (D + 1) ^ {2} / \delta\right)}{\varepsilon^ {3}}, \tag {21}
$$
where  $C_1, C_2 > 0$  are absolute numerical constants, which will be specified later in the proofs.
F.3 Supporting Technical Results
We start by stating the following variant of the classical simulation lemma (Kearns and Singh, 2002; Agarwal et al., 2019; Foster et al., 2021a).
Lemma 9 (Simulation lemma (Foster et al., 2021a, Lemma F.3)). Let  $\mathfrak{M} = (S, P, r, H, s_{\top}, s_{\bot})$  be a markov reward process. Then, the empirical version  $\widehat{\mathfrak{M}} = (\mathcal{S}, \widehat{P}, \widehat{r}, H, s_{\top}, s_{\bot})$  corresponding to  $\mathfrak{M}$
satisfies:
$$
\left| V _ {\mathrm {M R P}} - \widehat {V} _ {\mathrm {M R P}} \right| \leq \sum_ {s \in \mathcal {S}} d _ {\mathfrak {M}} (s) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} \left| P _ {s \rightarrow s ^ {\prime}} - \widehat {P} _ {s \rightarrow s ^ {\prime}} \right| + \left| r _ {s \rightarrow s ^ {\prime}} - \widehat {r} _ {s \rightarrow s ^ {\prime}} \right|\right),
$$
where  $d_{\mathfrak{M}}(s)$  is the probability of reaching  $s$  under  $\mathfrak{M}$ , and  $V_{\mathrm{MRP}}$  and  $\widehat{V}_{\mathrm{MRP}}$  denotes the value of  $s_{\top}$  under  $\mathfrak{M}$  and  $\widehat{\mathfrak{M}}$  respectively.
The following technical lemma shows that for any policy  $\pi$ , the empirical version of policy-specific MRP closely approximates its expected version.
Lemma 10. Let Algorithm 1 be run with the parameters given in Eq. (21), and consider any iteration of the while loop in line 5 with the instantaneous set  $S^{\mathrm{rch}}$ . Further, suppose that  $|\mathcal{D}_s| \geq \frac{\varepsilon n_2}{24D}$  for all  $s \in S^{\mathrm{rch}}$ . Then, with probability at least  $1 - \delta$ , the following hold:
(a) For all  $\pi \in \Pi$ ,  $s \in S_{\pi}^{\mathrm{rch}}$  and  $s' \in S_{\pi} \cup \{s_{\perp}\}$ ,
$$
\max  \left\{\left| P _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right|, \left| r _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {r} _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right|\right\} \leq \frac {\varepsilon}{1 2 D (D + 1)}.
$$
(b) For all  $\pi \in \Pi$  and  $s' \in S_{\pi} \cup \{s_{\perp}\}$ ,
$$
\max \{| P _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} - \widehat {P} _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} |, | r _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} - \widehat {r} _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} | \} \leq \frac {\varepsilon}{1 2 (D + 1) ^ {2}}.
$$
In the sequel, we define the event that the conclusion of Lemma 10 holds as  $\mathcal{E}_{\mathrm{est}}$ .
Proof. Fix any  $\pi \in \Pi$ . We first prove the bound for  $s \in S_{\pi}^{\mathrm{rch}}$ . Let  $s$  be at layer  $h$ . Fix any policy  $\pi \in \Pi$ , and consider any state  $s' \in S_{\pi} \cup \{s_{\perp}\}$ , where  $s'$  is at layer  $h'$ . Note that since  $\Pi$  is a  $(K, D)$ -sunflower, with its core  $\Pi_{\mathrm{core}}$  and petals  $\{S_{\pi}\}_{\pi \in \Pi}$ , we must have that any trajectory  $\tau \in \mathfrak{T}_{\pi}(s \to s')$  is also consistent with at least one  $\pi_e \in \Pi_{\mathrm{core}}$ . Furthermore, for any such  $\pi_e$ , we have
$$
\begin{array}{l} \mathbb {P} ^ {\pi_ {e}} \left[ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s \right] = \prod_ {i = h} ^ {h ^ {\prime} - 1} P \left[ s _ {i + 1} \mid s _ {i}, \pi_ {e} \left(s _ {i}\right), s _ {h} = s \right] \\ = \prod_ {i = h} ^ {h ^ {\prime} - 1} P \left[ s _ {i + 1} \mid s _ {i}, \pi \left(s _ {i}\right), s _ {h} = s \right] = \mathbb {P} ^ {\pi} \left[ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s \right], \tag {22} \\ \end{array}
$$
where the second line holds because both  $\pi \rightsquigarrow \tau_{h:h'}$  and  $\pi_e\rightsquigarrow \tau_{h:h'}$ . Next, recall from Eq. (17), that
$$
P _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \mathbb {E} ^ {\pi} \left[ \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} \left(s \rightarrow s ^ {\prime}\right)\right\} \mid s _ {h} = s \right]. \tag {23}
$$
Furthermore, from Eq. (19), recall that the empirical estimate  $\widehat{P}_{s\rightarrow s^{\prime}}^{\pi}$  of  $P_{s\to s^{\prime}}^{\pi}$  is given by: