SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {\mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\}}{\frac {1}{| \Pi_ {\text {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\text {c o r e}}} \mathbb {1} \left\{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}, \tag {24}
$$
where the dataset  $\mathcal{D}_s$  consists of i.i.d. samples, and is collected in lines 10-12 in Algorithm 3 (DataCollector), by first running the policy  $\pi_s$  for  $h$  timesteps and if the trajectory reaches  $s$ , then executing  $\pi_e \sim \mathrm{Uniform}(\Pi_{\mathrm{core}})$  for the remaining time steps (otherwise this trajectory is rejected). Let the law of this process be  $q$ . We thus note that,
$$
\begin{array}{l} \mathbb {E} _ {\tau \sim q} \left[ \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right] \\ = \mathbb {E} _ {\tau \sim q} \left[ \frac {\mathbb {1} \{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \}}{\frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \}} \mid s _ {h} = s \right] \\ = \sum_ {\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime})} \mathbb {P} _ {q} [ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s ] \cdot \frac {1}{\frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \left\{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {=} \sum_ {\tau \in \mathfrak {T} _ {\pi} (s \to s ^ {\prime})} \frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} ^ {\prime} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \big \{\tau \in \mathfrak {T} _ {\pi_ {e} ^ {\prime}} (s \to s ^ {\prime}) \big \} \mathbb {P} ^ {\pi_ {e} ^ {\prime}} [ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s ] \cdot \frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \big \{\pi_ {e} \rightsquigarrow \tau_ {h: h ^ {\prime}} \big \} \\ \stackrel {(i i)} {=} \sum_ {\tau \in \mathfrak {T} _ {\pi} (s \to s ^ {\prime})} \frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} ^ {\prime} \in \Pi_ {\mathrm {c o r e}}} \mathbb {P} ^ {\pi} [ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s ] \cdot \frac {\mathbb {1} \{\pi_ {e} ^ {\prime} \rightsquigarrow \tau_ {h : h ^ {\prime}} \}}{\frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \}} \\ = \sum_ {\tau \in \mathfrak {T} _ {\pi} (s \to s ^ {\prime})} \mathbb {P} ^ {\pi} [ \tau_ {h: h ^ {\prime}} \mid s _ {h} = s ] \\ \stackrel {(i i i)} {=} \mathbb {E} ^ {\pi} \big [ \mathbb {1} \big \{\tau \in \mathfrak {T} _ {\pi} (s \to s ^ {\prime}) \big \} \mid s _ {h} = s \big ] = P _ {s \to s ^ {\prime}} ^ {\pi}, \\ \end{array}
$$
where  $(i)$  follows from the sampling strategy in Algorithm 3 after observing  $s_h = s$ , and  $(ii)$  simply uses the relation (22) since both  $\pi_e' \rightsquigarrow \tau_{h:h'}$  and  $\pi \rightsquigarrow \tau_{h:h'}$  hold. Finally, in  $(iii)$ , we use the relation (23).
We have shown that  $\widehat{P}_{s\rightarrow s^{\prime}}^{\pi}$  is an unbiased estimate of  $P_{s\to s^{\prime}}^{\pi}$  for any  $\pi$  and  $s,s'\in S_{\pi}^{+}$ . Thus, using Hoeffding's inequality (Lemma 17), followed by a union bound, we get that with probability at least  $1 - \delta /4$ , for all  $\pi \in \Pi$ ,  $s\in S_{\pi}^{\mathrm{rch}}$ , and  $s^\prime \in S_\pi \cup \{s_\perp \}$ ,
$$
\left| \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} - P _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right| \leq K \sqrt {\frac {2 \log (4 | \Pi | D (D + 1) / \delta)}{| \mathcal {D} _ {s} |}},
$$
where the additional factor of  $K$  appears because for any  $\tau \in \mathfrak{T}_{\pi}(s \to s')$ , there must exist some  $\pi_e \in \Pi_{\mathrm{core}}$  that is also consistent with  $\tau$  (as we showed above), which implies that each of the terms in Eq. (24) satisfies the bound a.s.:
$$
\left| \frac {\mathbb {1} \{\tau \in \mathfrak {T} _ {\pi} (s \to s ^ {\prime}) \}}{\frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi_ {e} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \}} \right| \leq | \Pi_ {\mathrm {c o r e}} | = K.
$$
Since  $|\mathcal{D}_s| \geq \frac{\varepsilon n_2}{24D}$  by assumption, we have
$$
| \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} - P _ {s \rightarrow s ^ {\prime}} ^ {\pi} | \leq K \sqrt {\frac {4 8 D \log (4 | \Pi | D (D + 1) / \delta)}{\varepsilon n _ {2}}}.
$$
Repeating a similar argument for the empirical reward estimation in Eq. (19), we get that with probability at least  $1 - \delta / 4$ , for all  $\pi \in \Pi$ , and  $s \in S_{\pi}^{\mathrm{rch}}$  and  $s' \in S_{\pi} \cup \{s_{\perp}\}$ , we have that
$$
| \widehat {r} _ {s \rightarrow s ^ {\prime}} ^ {\pi} - r _ {s \rightarrow s ^ {\prime}} ^ {\pi} | \leq K \sqrt {\frac {4 8 D \log (4 | \Pi | D (D + 1) / \delta)}{\varepsilon n _ {2}}}.
$$
Similarly, we can also get for any  $\pi \in \Pi$  and  $s' \in S_{\pi} \cup \{s_{\perp}\}$ , with probability at least  $1 - \delta / 2$ ,
$$
\begin{array}{l} \max  \left\{\left| \widehat {r} _ {s _ {\top} \rightarrow s ^ {\prime}} ^ {\pi} - r _ {s _ {\top} \rightarrow s ^ {\prime}} ^ {\pi} \right|, \left| \widehat {P} _ {s _ {\top} \rightarrow s ^ {\prime}} ^ {\pi} - P _ {s _ {\top} \rightarrow s ^ {\prime}} ^ {\pi} \right|\right\} \leq K \sqrt {\frac {2 \log (4 | \Pi | (D + 1) / \delta)}{| \mathcal {D} _ {s _ {\top}} |}} \\ = K \sqrt {\frac {2 \log (4 | \Pi | (D + 1) / \delta)}{n _ {1}}}, \\ \end{array}
$$
where the last line simply uses the fact that  $|\mathcal{D}_{s_{\top}}| = n_1$ . The final statement is due to a union bound on the above results. This concludes the proof of Lemma 10.
Lemma 11. Fix a policy  $\pi \in \Pi$  and a set of reachable states  $S^{\mathrm{rch}}$ , and consider the policy-specific MRP  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$  (as defined by Eqs. (17) and (18)). Then for any  $s \in S_{\pi}^{\mathrm{rem}}$ , the quantity  $\bar{d}^{\pi}(s; \neg S_{\pi}^{\mathrm{rem}}) = d^{\mathfrak{M}}(s)$ , where  $d^{\mathfrak{M}}(s)$  is the occupancy of state  $s$  in  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$ .
Proof. We use  $\bar{\tau}$  to denote a trajectory in  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$  and  $\tau$  to denote a "corresponding" (in a sense which will be described shortly) trajectory in the original MDP  $M$ . For any  $s \in S_{\pi}^{\mathrm{rem}}$ , we have
$$
d ^ {\mathfrak {M}} (s) = \sum_ {\bar {\tau} \text {s . t .} s \in \bar {\tau}} \mathbb {P} ^ {\mathfrak {M}} [ \bar {\tau} ] = \sum_ {k = 0} ^ {H - 1} \sum_ {\bar {s} _ {1}, \bar {s} _ {2}, \dots , \bar {s} _ {k} \in S _ {\pi} ^ {\mathrm {r c h}}} \mathbb {P} ^ {\mathfrak {M}} [ \bar {\tau} = (s _ {\top}, \bar {s} _ {1}, \dots , \bar {s} _ {k}, s, s _ {\bot}, \dots) ].
$$
The first equality is simply due to the definition of  $d^{\mathfrak{M}}$ . For the second equality, we sum up over all possible sequences which start at  $s_{\top}$ , pass through some (variable length) sequence of states  $\bar{s}_1,\dots ,\bar{s}_k\in S_\pi^{\mathrm{rch}}$ , then reach  $s$  and the terminal state  $s_{\perp}$ . By definition of the policy-specific MRP, we know that once the MRP transits to a state  $s\in S_{\pi}^{\mathrm{rem}}$ , it must then transit to  $s_{\perp}$  and repeat  $s_{\perp}$  until the end of the episode.
Now fix a sequence  $\bar{s}_1, \dots, \bar{s}_k \in S_{\pi}^{\mathrm{rch}}$ . We relate the term in the summand to the probability of corresponding trajectories in the original MDP  $M$ . To avoid confusion, we let  $s_{h_1}, \ldots, s_{h_k} \in S_{\pi}^{\mathrm{rch}}$  denote the corresponding sequence of states in the original MDP, which are unique and satisfy  $h_1 < h_2 < \dots < h_k$ . We also denote  $s_{h_{k+1}} = s$ .
Using the definition of  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$ , we write
$$
\begin{array}{l} \mathbb {P} ^ {\mathfrak {M}} [ \bar {\tau} = (s _ {\top}, \bar {s} _ {1}, \dots , \bar {s} _ {k}, s, s _ {\perp}, \dots) ] \\ = \prod_ {i = 1} ^ {k} \mathbb {P} ^ {M, \pi} \left[ \tau_ {h _ {i}: h _ {i + 1}} \in \mathfrak {T} _ {\pi} \left(s _ {h _ {i}} \rightarrow s _ {h _ {i + 1}}\right) \mid \tau [ h _ {i} ] = s _ {h _ {i}} \right] \\ = \mathbb {P} ^ {M, \pi} [ \forall i \in [ k + 1 ], \tau [ h _ {i} ] = s _ {h _ {i}}, \text {a n d} \forall h \in [ h _ {k + 1} ] \backslash \{h _ {1}, \dots , h _ {k + 1} \}, \tau [ h ] \notin S _ {\pi} ]. \\ \end{array}
$$
Now we sum over all sequences  $\bar{s}_1,\dots ,\bar{s}_k\in S_{\pi}^{\mathrm{rch}}$  to get
$$
\begin{array}{l} d ^ {\mathfrak {M}} (s) \\ = \sum_ {k = 0} ^ {H - 1} \sum_ {s _ {h _ {1}}, \dots , s _ {h _ {k}} \in \mathcal {S} _ {\pi} ^ {\text {r c h}}} \mathbb {P} ^ {M, \pi} [ \forall i \in [ k + 1 ], \tau [ h _ {i} ] = s _ {h _ {i}}, \text {a n d} \forall h \in [ h _ {k + 1} ] \backslash \{h _ {1}, \dots , h _ {k + 1} \}, \tau [ h ] \notin \mathcal {S} _ {\pi} ] \\ = \mathbb {P} ^ {M, \pi} [ s \in \tau \text {a n d} \forall h \in [ h _ {k + 1} - 1 ], \tau [ h ] \notin S _ {\pi} ^ {\text {r e m}} ] \\ = \mathbb {P} ^ {\pi} [ \tau \in \mathfrak {T} (s _ {\top} \to s; \neg \mathcal {S} _ {\pi} ^ {\mathrm {r e m}}) ] = \bar {d} ^ {\pi} (s; \neg \mathcal {S} _ {\pi} ^ {\mathrm {r e m}}). \\ \end{array}
$$
The second equality follows from the definition of  $S_{\pi}^{\mathrm{rem}}$ , and the last line is the definition of the  $\bar{d}$  notation. This concludes the proof of Lemma 11.
Lemma 12. With probability at least  $1 - 2\delta$ , any  $(\bar{s}, \pi)$  that is added into  $\mathcal{T}$  (in line 13 of Algorithm 1) satisfies  $d^{\pi}(\bar{s}) \geq \varepsilon / 12D$ .
Proof. For any  $(\bar{s},\pi)\in \mathcal{T}$ , when we collect  $\mathcal{D}_{\bar{s}}$  in Algorithm 3, the probability that a trajectory will be accepted (i.e. the trajectory satisfies the "if" statement in line 10) is exactly  $d^{\pi}(\bar{s})$ . Thus, using Hoeffding's inequality (Lemma 17), with probability at least  $1 - \delta /D|\Pi |$
$$
\left| \frac {\left| \mathcal {D} _ {\bar {s}} \right|}{n _ {2}} - d ^ {\pi} (\bar {s}) \right| \leq \sqrt {\frac {2 \log (D | \Pi | / \delta)}{n _ {2}}}.
$$
Since  $|\mathcal{T}| \leq D|\Pi|$ , by union bound, the above holds for every  $(\bar{s}, \pi) \in \mathcal{T}$  with probability at least  $1 - \delta$ . Let us denote this event as  $\mathcal{E}_{\mathrm{data}}$ . Under  $\mathcal{E}_{\mathrm{data}}$ , for any  $(\bar{s}, \pi)$  that satisfies  $d^{\pi}(\bar{s}) \geq \frac{\varepsilon}{12D}$ ,
$$
\left| \mathcal {D} _ {\bar {s}} \right| \geq n _ {2} d ^ {\pi} (\bar {s}) - \sqrt {2 n _ {2} \log (D | \Pi | / \delta)} \geq \frac {\varepsilon n _ {2}}{1 2 D} - \frac {\varepsilon n _ {2}}{2 4 D} = \frac {\varepsilon n _ {2}}{2 4 D}, \tag {25}
$$
where the second inequality follows by the bound on  $d^{\pi}(\bar{s})$  and our choice of parameter  $n_2$  in Eq. (21).
In the following, we prove by induction that every  $(\bar{s},\pi)$  that is added into  $\mathcal{T}$  in the while loop from lines 7-17 in Algorithm 1 satisfies  $d^{\pi}(\bar{s})\geq \frac{\varepsilon}{12D}$ . This is trivially true at initialization when  $\mathcal{T} = \{(s_{\top},\mathrm{Null})\}$ , since every trajectory starts at the dummy state  $s_{\top}$ , for which we have  $d^{\mathrm{Null}}(s_{\top}) = 1$ .
We now proceed to the induction hypothesis. Suppose that in some iteration of the while loop, every tuple  $(\bar{s},\pi)\in \mathcal{T}$  satisfies  $d^{\pi}(\bar{s})\geq \varepsilon /12D$  , and that  $(\bar{s} ',\pi ')$  is a new tuple that will be added to  $\mathcal{T}$  . We will show that  $(\bar{s}^{\prime},\pi^{\prime})$  will also satisfy  $d^{\pi '}(\bar{s} ')\geq \varepsilon /12D$
Recall that  $\mathcal{S}_{\pi'}^+ = \mathcal{S}_{\pi'} \cup \{s_{\top}, s_{\bot}\}$ ,  $\mathcal{S}_{\pi'}^{\mathrm{rch}} = \mathcal{S}_{\pi'}^+ \cap \mathcal{S}^{\mathrm{rch}}$ , and  $\mathcal{S}_{\pi'}^{\mathrm{rem}} = \mathcal{S}_{\pi'}^+ \setminus \mathcal{S}_{\pi'}^{\mathrm{rch}}$ . Let  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi'} = \mathrm{MRP}(\mathcal{S}_{\pi'}, P^{\pi'}, r^{\pi'}, H, s_{\top}, s_{\bot})$  be the policy-specific MRP, where  $P^{\pi'}$  and  $r^{\pi'}$  are defined in Eqs. (17) and (18) respectively for the policy  $\pi'$ . Similarly let  $\widehat{\mathfrak{M}}_{\mathcal{S}^{\mathrm{rch}}}^{\pi'} =$
MRP  $(S_{\pi^{\prime}}^{+},\widehat{P}^{\pi^{\prime}},\widehat{r}^{\pi^{\prime}},H,s_{\top},s_{\bot})$  denote the estimated policy-specific MRP, where  $\widehat{P}^{\pi^{\prime}}$  and  $r^{\pi^{\prime}}$  are defined using (19) and (20) respectively. Note that for any state  $s\in S_{\pi '}^{\mathrm{rch}}$  , the bound in (25) holds.
For the rest of the proof, we assume that the event  $\mathcal{E}_{\mathrm{est}}$ , defined in Lemma 10, holds (this happens with probability at least  $1 - \delta$ ). By definition of  $\mathcal{E}_{\mathrm{est}}$ , we have
$$
\left| P _ {s \rightarrow s ^ {\prime}} ^ {\pi^ {\prime}} - \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi^ {\prime}} \right| \leq \frac {\varepsilon}{1 2 D (D + 1)}, \quad \text {f o r a l l} \quad s ^ {\prime} \in \mathcal {S} _ {\pi^ {\prime}} \cup \{s _ {\perp} \}. \tag {26}
$$
Furthermore, note that  $\widehat{d}^{\pi'}(\bar{s}') \gets \text{EstReachability}(S_{\pi'}^+, \widehat{\mathfrak{M}}_{\mathcal{S}^{\text{reh}}}^{\pi'}, \bar{s}')$  since in Algorithm 4 we start with  $V(s) = \mathbb{1}\{s = \bar{s}'\}$ . Furthermore, using Lemma 9, we have
$$
\begin{array}{l} \left| \widehat {d} ^ {\pi^ {\prime}} (\bar {s} ^ {\prime}) - d ^ {\mathfrak {M}} (\bar {s} ^ {\prime}) \right| \leq (D + 1) \quad \sup  \quad \left| \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi^ {\prime}} - P _ {s \rightarrow s ^ {\prime}} ^ {\pi^ {\prime}} \right| \\ s \in S _ {\pi^ {\prime}} ^ {\text {r c h}}, s ^ {\prime} \in S _ {\pi^ {\prime}} \cup \{s _ {\perp} \} \tag {27} \\ \leq \frac {\varepsilon}{1 2 D (D + 1)} \cdot (D + 1) = \frac {\varepsilon}{1 2 D}. \\ \end{array}
$$
where the second inequality follows from (26). Additionally, Lemma 11 states that  $d^{\mathfrak{M}}(\bar{s}^{\prime}) = \bar{d}^{\pi^{\prime}}(\bar{s}^{\prime};\neg S_{\pi^{\prime}}^{\mathrm{rem}})$ . Therefore we obtain
$$
\left| \bar {d} ^ {\pi^ {\prime}} \left(\bar {s} ^ {\prime}; \neg \mathcal {S} _ {\pi^ {\prime}} ^ {\mathrm {r e m}}\right) - \widehat {d} ^ {\pi^ {\prime}} \left(\bar {s} ^ {\prime}\right) \right| \leq \frac {\varepsilon}{1 2 D}.
$$
Thus, if the new state-policy pair  $(\bar{s}^{\prime},\pi^{\prime})$  is added into  $\mathcal{T}$ , we will have
$$
\bar {d} ^ {\pi^ {\prime}} \left(\bar {s} ^ {\prime}; \neg \mathcal {S} _ {\pi^ {\prime}} ^ {\text {r e m}}\right) \geq \frac {\varepsilon}{6 D} - \frac {\varepsilon}{1 2 D} = \frac {\varepsilon}{1 2 D}.
$$
Furthermore, by definition of  $\bar{d}$  we have
$$
\bar {d} ^ {\pi^ {\prime}} (\bar {s} ^ {\prime}; \neg \mathcal {S} _ {\pi^ {\prime}} ^ {\mathrm {r e m}}) = \mathbb {P} ^ {\pi^ {\prime}} [ \tau \in \mathfrak {T} (s _ {\top} \to \bar {s} ^ {\prime}; \neg \mathcal {S} _ {\pi^ {\prime}} ^ {\mathrm {r e m}}) ] \leq \mathbb {P} ^ {\pi^ {\prime}} [ \bar {s} ^ {\prime} \in \tau ] = d ^ {\pi^ {\prime}} (\bar {s} ^ {\prime}),
$$
so we have proved the induction hypothesis  $d^{\pi '}(\vec{s}')\geq \varepsilon /12D$  for the next round. This concludes the proof of Lemma 12.
The next lemma establishes that Algorithm 1 will terminate after finitely many rounds, and that after termination will have explored all sufficiently reachable states.
Lemma 13. With probability at least  $1 - 2\delta$
(a) The while loop in line 5 in Algorithm 1 will terminate after at most  $\frac{12HD\mathfrak{C}(\Pi)}{\varepsilon}$  rounds.
(b) After the termination of the while loop, for any  $\pi \in \Pi$ , the remaining states  $s \in S_{\pi}^{\mathrm{rem}}$  that are not added to  $\mathcal{S}^{\mathrm{rch}}$  satisfy  $\vec{d}^{\pi}(s; \neg S_{\pi}^{\mathrm{rem}}) \leq \varepsilon / 4D$ .
Notice that according to our algorithm, the same state cannot be added multiple times into  $S^{\mathrm{rch}}$ . Therefore,  $|S^{\mathrm{rch}}| \leq D|\Pi|$ , and the maximum number of rounds of the while loop is  $D|\Pi|$  (i.e., the while loop eventually terminates).
(a) First, note that from the definition of coverability and Lemma 1, we have
$$
\sum_ {s \in \mathcal {S}} \sup  _ {\pi \in \Pi} d ^ {\pi} (s) \leq H C ^ {\operatorname {c o v}} (\Pi ; M) \leq H \mathfrak {C} (\Pi).
$$
Furthermore, Lemma 12 states that every  $(s,\pi_s)\in \mathcal{T}$  satisfies  $d^{\pi_s}(s)\geq \varepsilon /12D$ . Thus, at any point in Algorithm 1, we have
$$
\sum_ {s \in \mathcal {S} ^ {\text {r c h}}} \sup  _ {\pi \in \Pi} d ^ {\pi} (s) \geq \sum_ {s \in \mathcal {S} ^ {\text {r c h}}} d ^ {\pi_ {s}} (s) \geq | \mathcal {T} | \cdot \frac {\varepsilon}{1 2 D}.
$$
Since,  $S^{\mathrm{rch}}\subseteq S$  , the two bounds indicate that
$$
| \mathcal {T} | \leq \frac {1 2 H D \mathfrak {C} (\Pi)}{\varepsilon}.
$$
Since every iteration of the while loop adds one new  $(s, \pi_s)$  to  $\mathcal{T}$ , the while loop terminates after at most  $12HD\mathfrak{C}(\Pi)/\varepsilon$  many rounds.
(b) We know that once the while loop has terminated, for every  $\pi \in \Pi$  and  $\bar{s} \in S_{\pi}^{\mathrm{rem}}$ , we must have  $\widehat{d}^{\pi}(\bar{s}) \leq \varepsilon / 6D$ , or else the condition in line 12 in Algorithm 1 is violated.
Fix any such  $(\bar{s},\pi)$  pair. Inspecting the proof of Lemma 12, we see that
$$
\left| \bar {d} ^ {\pi} (\bar {s}; \neg \mathcal {S} _ {\pi} ^ {\text {r e m}}) - \widehat {d} ^ {\pi} (\bar {s}) \right| \leq \frac {\varepsilon}{1 2 D}.
$$
To conclude, we get
$$
\bar {d} ^ {\pi} (s; \neg \mathcal {S} _ {\pi} ^ {\text {r e m}}) \leq \frac {\varepsilon}{6 D} + \frac {\varepsilon}{1 2 D} = \frac {\varepsilon}{4 D}.
$$
Lemma 14. Suppose that the conclusions of Lemmas 10 and 13 hold. Then for every  $\pi \in \Pi$ , the estimated value  $\widehat{V}^{\pi}$  computed in Algorithm 1 satisfies
$$
\left| \hat {V} ^ {\pi} - V ^ {\pi} \right| \leq \varepsilon .
$$
Proof. We will break up the proof into two steps. First, we show that for any  $\pi$ , the value estimate  $\widehat{V}^{\pi}$  obtained using the empirical policy-specific MRP  $\widehat{\mathfrak{M}}_{S^{\mathrm{rch}}}^{\pi}$  is close to its value in the policy-specific MRP  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$ , as defined via (17) and (18). We denote this quantity as  $V_{\mathrm{MRP}}^{\pi}$ . Then, we will show that  $V_{\mathrm{MRP}}^{\pi}$  is close to  $V^{\pi}$ , the value of the policy  $\pi$  in the original MDP  $M$ .
Part 1:  $\widehat{V}^{\pi}$  is close to  $V_{\mathrm{MRP}}^{\pi}$ . Note that the output  $\widehat{V}^{\pi}$  of Algorithm 5 is exact the value function of MRP  $\widehat{\mathfrak{M}}_{\mathcal{S}_{\mathrm{rch}}}^{\pi}$  defined by Eqs. (19) and (20). When  $D = 0$ , by part (b) of Lemma 10, we obtain
$$
| \widehat {V} ^ {\pi} - V _ {\mathrm {M R P}} ^ {\pi} | = | \widehat {r} _ {s _ {\top} \to s _ {\bot}} ^ {\pi} - r _ {s _ {\top} \to s _ {\bot}} ^ {\pi} | \leq \frac {\varepsilon}{1 2 (D + 1) ^ {2}} \leq \frac {\varepsilon}{2}.
$$
When  $D \geq 1$ , using Lemma 10, we have
$$
\left| r _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} - \widehat {r} _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} \right| \leq \frac {\varepsilon}{1 2 (D + 1) ^ {2}}, \quad \left| P _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} - \widehat {P} _ {s _ {\top} \to s ^ {\prime}} ^ {\pi} \right| \leq \frac {\varepsilon}{1 2 (D + 1) ^ {2}}, \quad \forall s ^ {\prime} \in \mathcal {S} _ {\pi} \cup \{s _ {\perp} \}
$$
$$
| r _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {r} _ {s \rightarrow s ^ {\prime}} ^ {\pi} | \leq \frac {\varepsilon}{1 2 D (D + 1)}, \quad | P _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} | \leq \frac {\varepsilon}{1 2 D (D + 1)}, \qquad \forall s \in \mathcal {S} _ {\pi} ^ {\mathrm {r c h}}, s ^ {\prime} \in \mathcal {S} _ {\pi} ^ {+} \cup \{s _ {\perp} \}.
$$
By the simulation lemma (Lemma 9), we get
$$
\begin{array}{l} \left| \widehat {V} ^ {\pi} - V _ {\mathrm {M R P}} ^ {\pi} \right| \leq 2 (D + 2) \max  _ {s, s ^ {\prime} \in S _ {\pi} ^ {+}} \left(\left| P _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right| + \left| r _ {s \rightarrow s ^ {\prime}} ^ {\pi} - \widehat {r} _ {s \rightarrow s ^ {\prime}} ^ {\pi} \right|\right) \\ \leq 2 (D + 2) \left(\frac {\varepsilon}{1 2 D (D + 1)} + \frac {\varepsilon}{1 2 D (D + 1)}\right) \leq \frac {\varepsilon}{2}. \\ \end{array}
$$
Part 2:  $V_{\mathrm{MRP}}^{\pi}$  is close to  $V^{\pi}$ . As in the proof of Lemma 11, let us consider different trajectories  $\bar{\tau}$  that are possible in  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$ . We can represent  $\bar{\tau} = (s_{\top}, \bar{s}_1, \dots, \bar{s}_k, s_{\bot}, \dots)$  where the states  $\bar{s}_1, \dots, \bar{s}_k$  are distinct and all except possibly  $\bar{s}_k$  belong to  $S_{\pi}^{\mathrm{rch}}$ , and the states after  $s_{\bot}$  are just repeats of  $s_{\bot}$  until the end of the episode. Let  $s_{h_1}, s_{h_2}, \ldots, s_{h_k}$  be the same sequence (in the original MDP  $M$ ) Again, we have
$$
\begin{array}{l} \mathbb {P} ^ {\mathfrak {M}} \left[ \bar {\tau} = \left(s _ {\top}, \bar {s} _ {1}, \dots , \bar {s} _ {k}, s _ {\perp}, \dots\right) \right] \\ = \mathbb {P} ^ {\pi} [ \forall i \in [ k ], \tau [ h _ {i} ] = s _ {h _ {i}}, \text {a n d} \forall h \in [ H ] \backslash \{h _ {1}, \dots , h _ {k} \}, \tau [ h ] \notin \mathcal {S} _ {\pi} ], \\ \end{array}
$$
where recall that  $\mathbb{P}^{\mathfrak{M}}$  denotes probability under the  $\mathfrak{M}_{S^{\mathrm{rch}}}^{\pi}$ , and  $\mathbb{P}^{\pi}$  denotes the probability under trajectories drawn according to  $\pi$  in the underlying MDP;  $\mathbb{E}^{\mathfrak{M}}$  and  $\mathbb{E}^{\pi}$  are defined similarly.
Furthermore, the expectation of rewards we collected in  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$  with trajectories  $\bar{\tau}$  is
$$
\begin{array}{l} \mathbb {E} ^ {\mathfrak {M}} \left[ R [ \bar {\tau} ] \mathbb {1} \left\{\left(s _ {\top}, \bar {s} _ {1}, \dots , \bar {s} _ {k}, s _ {\perp}, \dots\right) \right\} \right] \\ = \mathbb {E} ^ {\pi} [ R [ \tau ] \mathbb {1} \left\{\forall i \in [ k ], \tau \left[ h _ {i} \right] = s _ {h _ {i}}, \text {a n d} \forall h \in [ H ] \backslash \left\{h _ {1}, \dots , h _ {k} \right\}, \tau [ h ] \notin \mathcal {S} _ {\pi} \right\} ]. \\ \end{array}
$$
Next, we sum over all possible trajectories. However, note that the only trajectories that are possible in  $M$  whose corresponding trajectories are not accounted for in  $\mathfrak{M}_{\mathcal{S}^{\mathrm{rch}}}^{\pi}$  are precisely those that visit states in  $S_{\pi}$ , after visiting some  $s_{h_k}$  in the remaining states  $S_{\pi}^{\mathrm{rem}}$  (since, by construction, the MRP transitions directly to  $s_{\perp}$  after encountering a state in  $S_{\pi}^{\mathrm{rem}}$ ). Thus,
$$
V _ {\mathrm {M R P}} ^ {\pi} = \mathbb {E} ^ {\pi} [ R [ \tau ] (\mathbb {1} \{\tau \cap S _ {\pi} ^ {\mathrm {r e m}} = \emptyset \} + \mathbb {1} \{\exists k \in [ H ]: s _ {h _ {k}} \in S _ {\pi} ^ {\mathrm {r e m}} \text {a n d} \forall h > h _ {k}: s _ {h} \notin S _ {\pi} \}) ],
$$
where the first term corresponds to trajectories that do not pass through  $S_{\pi}^{\mathrm{rem}}$ , and the second term corresponds to trajectories that passes through some state in  $S_{\pi}^{\mathrm{rem}}$  but then does not go through any other state in  $S_{\pi}$ . On the other hand,
$$
V ^ {\pi} = \mathbb {E} ^ {\pi} [ R [ \tau ] ].
$$
Clearly,  $V_{\mathrm{MRP}}^{\pi} \leq V^{\pi}$ . Furthermore, we also have