SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{Q}_t(s_t, a_{t:t+k-1})$  is the empirical mean return-to-go from  $m$  episodes starting in state  $s_t$  and taking actions  $a_t, \ldots, a_{t+k-1}$  followed by actions sampled from the exploration policy. To simplify notation, let  $\vec{a}_t$  denote  $a_{t:t+k-1}$ . We use various inequalities, described in detail below, to bound the probability that a particular  $k$ -action sequence is chosen:
$$
\begin{array}{l} \underline {{p}} (\vec {a} _ {t}) \leq \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) > \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right) \\ \leq \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \geq \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right) \leq \overline {{p}} (\vec {a} _ {t}). \\ \end{array}
$$
Letting  $p(\vec{a}_t)$  denote the actual probability an action sequence is chosen, we can rewrite (16) to
$$
\mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t}, s _ {t}\right) = \sum_ {a _ {t} \in \mathcal {A}} \mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t}, s _ {t}, a _ {t}\right) \sum_ {a _ {t + 1: t + k - 1} \in \mathcal {A} ^ {k - 1}} p \left(\vec {a} _ {t}\right). \tag {18}
$$
Given the bounds on  $p(\vec{a}_t)$  (i.e.,  $\underline{p}$  and  $\overline{p}$ ), we formulate a linear program with the bounds as constraints, plus the constraint that  $\sum_{\vec{a}_t \in \mathcal{A}^k} p(\vec{a}_t) = 1$ , with the objective of maximizing (18). Solving this gives an upper bound on  $\mathbb{P}(\mathcal{E} \mid \mathcal{O}_t, s_t)$ . We can then propagate this bound recursively using (17) to bound  $\mathbb{P}(\mathcal{E})$ .
Calculating  $p$  and  $\bar{p}$
We use up to four methods to bound  $p(\vec{a}_t)$ , and pick the one which gives the lowest conditional failure probability after solving the linear program described above. As previously, let  $\mathcal{D}_t(s_t,\vec{a}_t)$  denote the distribution of the reward-to-go starting in  $s_t$  and taking actions  $a_{t},\ldots ,a_{t + k - 1}$  followed by actions sampled from the exploration policy. Thus we can write
$$
\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) = \frac {1}{m} \sum_ {i = 1} ^ {m} X _ {i} \qquad \text {w h e r e} \quad X _ {1}, \ldots , X _ {m} \stackrel {{\text {i . i . d .}}} {{\sim}} \mathcal {D} _ {t} (s _ {t}, \vec {a} _ {t}).
$$
Most of the following methods use the following decomposition:
$$
\begin{array}{l} \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) > \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right) \\ \stackrel {\mathrm {(i)}} {=} \int \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime}) <   \hat {Q} _ {t} (s _ {t}, \vec {a} _ {t})\right) d \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t})\right) \\ \geq \sum_ {i = 1} ^ {N} \mathbb {P} _ {m} \left(q _ {i - 1} <   \hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \leq q _ {i}\right) \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \leq q _ {i - 1}\right) \tag {19} \\ \end{array}
$$
for some sequence  $-\infty = q_0 \leq q_1 \leq \ldots \leq q_N = \infty$ . Here, (i) uses the fact that the random variables  $\hat{Q}_t(s_t, \vec{a}_t)$  across all action sequences  $\vec{a}_t \in \mathcal{A}^k$  are independent. (ii) is a lower bound on the integral via a Riemann sum.
Alternatively, suppose we know that the CDF of  $\hat{Q}_t(s_t,\vec{a}_t)$  is bounded by
$$
\mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \leq x\right) \geq F _ {\underline {{Z}}} (x)
$$
where  $F_{Z}(x)$  is the continuous CDF of some random variable  $\underline{Z}$ . Then
$$
\begin{array}{l} \mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) > \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)\right) \\ \geq \mathbb {P} _ {m} \left(\underline {{Z}} > \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right) \\ = \int \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime}) <   \hat {Q} _ {t} (s _ {t}, \vec {a} _ {t})\right) d \mathbb {P} _ {m} (\underline {{Z}}) \\ \geq \frac {1}{N} \sum_ {i = 1} ^ {N} \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \leq F _ {\underline {{Z}}} ^ {- 1} \left(\frac {i - 1}{N}\right)\right) \tag {20} \\ \end{array}
$$
by a similar argument. We also have equivalent bounds in the other direction:
$$
\begin{array}{l} \mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \geq \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)\right) \leq \sum_ {i = 1} ^ {N} \mathbb {P} _ {m} \left(q _ {i - 1} <   \hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \leq q _ {i}\right) \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \leq q _ {i}\right) (21) \\ \mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \geq \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)\right) \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \prod_ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \mathbb {P} _ {m} \left(\hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \leq F _ {\bar {Z}} ^ {- 1} \left(\frac {i}{N}\right)\right) (22) \\ \end{array}
$$
where the CDF of  $\overline{Z}$  is greater than or equal to that of  $\hat{Q}_t(s_t, \vec{a}_t)$ . We use  $N = 100$  when using these bounds.
Binomial bounds In the case where for all  $\vec{a}_t \in \mathcal{A}^k$ , the distribution  $\mathcal{D}_t(s_t, \vec{a}_t)$  has mass on only 0 and some other value  $C$ , we have
$$
\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \sim \frac {C}{m} \operatorname {B i n o m} \left(m, \frac {1}{C} Q _ {t} \left(s _ {t}, \vec {a} _ {t}\right)\right).
$$
This case occurs in many environments that are goal-based, i.e. where the agent gets reward only for reaching some goal and then the episode ends. We find that it significantly improves the sample complexity bounds in those environments. Without loss of generality, we may assume  $C = 1$ . We then apply (19) and (21) to obtain  $\underline{p}(\vec{a}_t)$  and  $\overline{p}(\vec{a}_t)$ . We let  $q_0, \ldots, q_N$  be set such that
$$
\mathbb {P} _ {m} \left(q _ {i - 1} <   \hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \leq q _ {i}\right) \approx \frac {1}{N}
$$
using either the exact inverse CDF of the binomial distribution for small  $m$  or a normal approximation for large  $m$ . Then, we can calculate all terms in the bounds (19) and (21) using the binomial CDF. Since the CDF is more expensive to calculate for larger  $m$ , we only use the binomial-based bounds when  $m \leq 10^6$  and  $k = 1$ .
Berry-Esseen bounds For this type of bound, we calculate the variance  $\sigma^2 = \mathrm{Var}(\mathcal{D}_t(s_t,\vec{a}_t))$  and third absolute moment
$$
\rho = \mathbb {E} _ {X \sim \mathcal {D} _ {t} (s _ {t}, \vec {a} _ {t})} [ | X | ^ {3} ]
$$
for each  $\vec{\alpha}_t\in \mathcal{A}^k$  . Then by the Berry-Esseen theorem [51], we have that
$$
\left| \mathbb {P} _ {m} (\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \leq u) - \mathbb {P} _ {X \sim \mathcal {N} (Q _ {t} (s _ {t}, \vec {a} _ {t}), \sigma^ {2} / m)} (X \leq u) \right| \leq \frac {\min  \{0 . 3 3 2 8 (\rho / \sigma^ {3} + 0 . 4 2 9) , 0 . 3 3 5 5 4 (\rho / \sigma^ {3} + 0 . 4 1 5) \}}{\sqrt {m}}.
$$
The resulting upper and lower bounds on the CDFs of  $\hat{Q}_t(s_t, \vec{a}_t)$  for all  $\vec{a}_t \in \mathcal{A}^k$  can be used in (20) and (22) to calculate  $\underline{p}(\vec{a}_t)$  and  $\overline{p}(\vec{a}_t)$ . Since this bound requires order  $N$  evaluations of the normal CDF and inverse CDF, which is somewhat expensive, we only use it when  $A^k \leq 100$ .
Bernstein bounds Similarly to the Berry-Esseen bounds, Bernstein's inequality can be used to bound the CDF of  $\hat{Q}_t(s_t,\vec{a}_t)$ , and is sometimes superior to Berry-Esseen for large  $m$  due to giving tail bounds that decay exponentially rather than quadratically. In particular, suppose  $\mathcal{D}_t(s_t,\vec{a}_t)$  is supported on the interval  $[\alpha ,\beta ]$ ; we can compute these bounds via value iteration. Then
$$
\operatorname {V a r} \left(\mathcal {D} _ {t} \left(s _ {t}, \vec {a} _ {t}\right)\right) \leq V = \left(\beta - Q _ {t} \left(s _ {t}, \vec {a} _ {t}\right)\right) \left(Q _ {t} \left(s _ {t}, \vec {a} _ {t}\right) - \alpha\right).
$$
Bernstein's inequality gives the following bounds on the CDF of  $\hat{Q}_t(s_t,\vec{a}_t)$ :
$$
\mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \leq Q _ {t} (s _ {t}, \vec {a} _ {t}) + u\right) \geq \left\{ \begin{array}{l l} 1 & u \leq 0 \\ 1 - \exp \left\{- \frac {m u ^ {2} / 2}{V + (\beta - \alpha) u / 3} \right\} & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \leq Q _ {t} (s _ {t}, \vec {a} _ {t}) + u\right) \leq \left\{ \begin{array}{l l} 1 & \quad u \geq 0 \\ \exp \left\{- \frac {m u ^ {2} / 2}{V + (\beta - \alpha) u / 3} \right\} & \quad \text {o t h e r w i s e .} \end{array} \right.
$$
Similarly to the Berry-Esseen bounds, we use these in (20) and (22) to calculate  $\underline{p} (\vec{a}_t)$  and  $\bar{p} (\vec{a}_t)$  when  $A^k\leq 100$
Bennett bounds The final method we use to calculate  $\underline{p}(\vec{a}_t)$  and  $\overline{p}(\vec{a}_t)$  is computationally cheaper than the others, so we can use it no matter the size of  $A^k$ . As in the Bernstein bounds, we calculate
the interval support and bound the variance of each  $\mathcal{D}_t(s_t,\vec{\alpha}_t)$ . We then let  $u$  be the arithmetic mean of the highest action sequence Q value and the second-highest, i.e.
$$
u = \frac {1}{2} \left(\max  _ {\vec {a} _ {t} \in \mathcal {A} ^ {k}} Q _ {t} (s _ {t}, \vec {a} _ {t}) + \max  _ {\vec {a} _ {t} ^ {\prime} \notin \arg \max  _ {\vec {a} _ {t} \in \mathcal {A} ^ {k}} Q _ {t} (s _ {t}, \vec {a} _ {t})} Q _ {t} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right).
$$
Then, we for each action sequence with less-than-highest Q-values, i.e. for each  $\vec{a}_t \notin \arg \max_{\vec{a}_t \in \mathcal{A}^k} Q_t(s_t, \vec{a}_t)$ , we calculate the upper bound
$$
\begin{array}{l} \mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \geq \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k} \backslash \{\vec {a} _ {t} \}} \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)\right) \\ \leq \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) \geq \max  _ {\vec {a} _ {t} ^ {\prime} \in \arg \max  _ {\vec {a} _ {t} \in \mathcal {A} ^ {k}} Q _ {t} (s _ {t}, \vec {a} _ {t})} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})\right) \\ = 1 - \mathbb {P} _ {m} \left(\exists \vec {a} _ {t} ^ {\prime} \in \arg \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k}} Q _ {t} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \quad \hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) <   \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)\right) \\ \leq 1 - \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) <   u \quad \wedge \quad \forall \vec {a} _ {t} ^ {\prime} \in \arg \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k}} Q _ {t} (s _ {t}, \vec {a} _ {t} ^ {\prime}) \quad \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime}) > u\right) \\ = 1 - \mathbb {P} _ {m} \left(\hat {Q} _ {t} (s _ {t}, \vec {a} _ {t}) <   u\right) \prod_ {\vec {a} _ {t} ^ {\prime} \in \arg \max _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k}} \hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime})} \mathbb {P} _ {m} \left(\hat {Q} (s _ {t}, \vec {a} _ {t} ^ {\prime}) > u\right) \\ = 1 - \left(1 - \mathbb {P} _ {m} \left(\hat {Q} _ {t} \left(s _ {t}, \vec {a} _ {t}\right) \geq u\right)\right) \prod_ {\vec {a} _ {t} ^ {\prime} \in \arg \max  _ {\vec {a} _ {t} ^ {\prime} \in \mathcal {A} ^ {k}} \hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right)} \left(1 - \mathbb {P} _ {m} \left(\hat {Q} \left(s _ {t}, \vec {a} _ {t} ^ {\prime}\right) \leq u\right)\right). \tag {23} \\ \end{array}
$$
Each of the tail bounds in (23) can be upper bounded using Bennett's inequality to obtain  $\overline{p}(\vec{a}_t)$ . We let  $\overline{p}(\vec{a}_t) = 1$  if  $\vec{a}_t \in \arg \max_{\vec{a}_t \in \mathcal{A}^k} Q_t(s_t, \vec{a}_t)$  and we set  $\underline{p}(\vec{a}_t) = 0$  for all  $\vec{a}_t \in A^k$ .
D Previously proposed sample complexity bounds
In this appendix, we give proofs of sample complexity bounds based on properties previously proposed in the RL theory literature. We also compare these bounds to our effective horizon-based bounds in examples that showcase their failure modes.
D.1 Upper confidence bounds (UCB) and strategic exploration
A central problem in RL is exploration: how to efficiently reach enough states in an MDP in order to identify the optimal policy. One common way of approaching exploration is with upper-confidence bounds (UCB), which originated in the bandit literature. Algorithms using UCBs generally choose actions based on the current best estimate of that action's value plus an exploration "bonus" that incentivizes exploration of little-seen states. Examples in the RL literature include Kakade [18], Azar et al. [4], Jiang et al. [19], Jin et al. [5, 20], Du et al. [21], Jin et al. [22]. Generally, these UCB algorithms achieve minimax sample complexity in terms of some measure of the "size" of the state space—either the number of states  $S$  [4], or quantities like the Bellman-Eluder dimension [22].
A very simple UCB-type algorithm, R-MAX [23, 18], achieves a sample complexity bounded by  $SAT$  in deterministic, tabular MDPs. It depends on knowing the maximum reward at any state-action pair in the MDP,  $R_{\max} = \max_{(s,a)\in S\times A}R(s,a)$ . It also requires access to a computational oracle that can calculate an optimal policy for any transition function  $f$  and reward function  $R$ , for instance via value iteration.
1: procedure R-MAX
2: initialize  $\hat{f}(s, a) \gets s$  for all  $(s, a) \in S \times \mathcal{A}$
3: initialize  $\hat{R}(s, a) \gets R_{\max}$  for all  $(s, a) \in S \times \mathcal{A}$
4: for  $j = 1, \ldots, SA$  do
5: for  $t = 1,\dots ,T$  do
6: take an action  $a$  in the current state  $s$  according to the optimal policy for  $\hat{f}$  and  $\hat{R}$
7:  $R(s,a)\gets$  the observed reward
8:  $\hat{f} (s,a)\gets$  the observed next state  $s^\prime$
9: end for
10: end for
11: return an optimal policy for  $\hat{f}$  and  $\hat{R}$
12: end procedure
This is the version of R-MAX for deterministic MDPs; there is a more complex version for stochastic MDPs. The exploration bonuses in R-MAX are simply the initialization of  $\hat{R}$  to the maximum possible reward. This ensures that an optimal value function computed from  $\hat{f}$  and  $\hat{R}$  is always an upper bound on the true optimal value function. The following result shows that R-MAX finds the optimal policy, and thus proves that its sample complexity is at most  $SAT$ .
Theorem D.1. R-MAX returns an optimal policy.
Proof. Let  $\hat{V}^*$  and  $\hat{Q}^*$  be the optimal value function and Q-function under  $\hat{f}$  and  $\hat{R}$ . We will begin by showing that at any point in the algorithm,  $V_{t}^{*}(s)\leq \hat{V}_{t}^{*}(s)$  and  $Q_{t}^{*}(s,a)\leq \hat{Q}_{t}^{*}(s,a)$  for all  $(t,s,a)\in [T]\times S\times \mathcal{A}$ .
The proof is via induction on  $t$  from  $T$  to 1. To begin, clearly  $\hat{Q}_T^* (s,a)\in \{Q_T^* (s,a),R_{\max}\}$ , so the bound holds for  $\hat{Q}_T^*$ . Now, suppose that at some  $t\in [T]$  and all  $(s,a)\in \mathcal{S}\times \mathcal{A},Q_t^* (s,a)\leq \hat{Q}_t^* (s,a)$ . Then
$$
\hat {V} _ {t} ^ {*} (s) = \max _ {a \in \mathcal {A}} \hat {Q} _ {t} ^ {*} (s, a) \geq \max _ {a \in \mathcal {A}} Q _ {t} ^ {*} (s, a) = V _ {t} ^ {*} (s),
$$
so the bound holds for  $\hat{V}_t^*$  as well. Finally, say that at some  $t\in [T - 1]$  and for all  $s\in S$ $V_{t + 1}^{*}(s)\leq \hat{V}_{t + 1}^{*}(s)$ . To show this implies  $Q_{t}^{*}(s,a)\leq \hat{Q}_{t}^{*}(s,a)$  for any  $(s,a)\in S\times \mathcal{A}$ , consider two cases. First, if  $(s,a)$  has been seen by the algorithm, then
$$
\hat {Q} _ {t} ^ {*} (s, a) = \hat {R} (s, a) + \hat {V} _ {t + 1} ^ {*} (\hat {f} (s, a) = R (s, a) + \hat {V} _ {t + 1} ^ {*} (f (s, a) \geq R (s, a) + V _ {t + 1} ^ {*} (f (s, a) = Q _ {t} ^ {*} (s, a).
$$
Otherwise, if  $(s,a)$  has not been seen, then  $\hat{R}(s,a) = R_{\max}$  and  $\hat{f}(s,a) = s$ . In this case,
$$
\hat {Q} _ {t} ^ {*} (s, a) = (T - t + 1) R _ {\max} \geq Q _ {t} ^ {*} (s, a).
$$
By induction we see that  $V_{t}^{*}(s)\leq \hat{V}_{t}^{*}(s)$  and  $Q_{t}^{*}(s,a)\leq \hat{Q}_{t}^{*}(s,a)$  for all  $(t,s,a)\in [T]\times S\times \mathcal{A}$
Next, we will prove that any optimal policy  $\pi$  for  $\hat{f}$  and  $\hat{R}$  must either (a) be optimal for  $f$  and  $R$  or (b) reach a previously unseen state-action pair. In particular, we will show that if  $V_{1}^{\pi}(s_{1}) < \hat{V}_{1}^{*}(s_{1})$ , then  $\pi$  must reach a previously unseen state-action pair. Otherwise,  $V_{1}^{\pi}(s_{1}) \geq \hat{V}_{1}^{*}(s_{1}) \geq V_{1}^{*}(s_{1})$ , showing that  $\pi$  is optimal for  $f$  and  $R$ .
We will again work inductively starting from the last timestep. First, suppose that  $Q_T^\pi(s,a) < \hat{Q}_T^*(s,a)$  for some  $(s,a) \in S \times \mathcal{A}$ . This is equivalent to  $R(s,a) < \hat{R}(s,a)$ , which clearly means that  $(s,a)$  cannot have been explored.
Now, suppose that at some timestep  $t \in [T]$ , we know that for any  $(s,a) \in S \times \mathcal{A}$ ,  $Q_{t}^{\pi}(s,a) < \hat{Q}_{t}^{*}(s,a)$  implies that  $\pi$  must explore some new state-action pair at or after timestep  $t$  starting in  $(s,a)$ . If  $V_{t}^{\pi}(s) < \hat{V}_{t}^{*}(s)$  for some  $s \in S$ , then
$$
Q _ {t} ^ {\pi} (s, \pi_ {t} (s)) <   \max  _ {a \in \mathcal {A}} \hat {Q} _ {t} ^ {*} (s, a) = \hat {Q} _ {t} ^ {*} (s, \pi_ {t} (s)).
$$
By assumption this means  $\pi$  must explore some new state-action pair at or after timestep  $t$ , since it takes an action  $a$  satisfying  $Q_{t}^{\pi}(s,a) < \hat{Q}_{t}^{*}(s,a)$ .
Finally, suppose that for some  $t \in [T]$ , we know that for any  $s \in S$ ,  $V_{t+1}^{\pi}(s) < \hat{V}_{t+1}^{*}(s)$  implies that  $\pi$  must explore some new state-action pair at or after timestep  $t+1$  starting in  $s$ . Suppose for some  $(s, a) \in S \times \mathcal{A}$  that  $Q_{t}^{\pi}(s, a) < \hat{Q}_{t}^{*}(s, a)$ . Then
$$
R (s, a) + V _ {t + 1} ^ {\pi} (f (s, a)) <   \hat {R} (s, a) + V _ {t + 1} ^ {\pi} (\hat {f} (s, a))
$$
which implies that either  $R(s, a) < \hat{R}(s, a)$  or  $V_{t+1}^{\pi}(f(s, a)) < V_{t+1}^{\pi}(\hat{f}(s, a))$ . In the first case,  $(s, a)$  must be unexplored. In the second case, either  $(s, a)$  is unexplored, or
$$
V _ {t + 1} ^ {\pi} (f (s, a)) <   V _ {t + 1} ^ {\pi} (f (s, a)).
$$
In any of these cases,  $\pi$  must explore a new state-action pair either in this timestep or in the future starting from  $(s,a)$ .
Inductively, this shows that  $V_{1}^{\pi}(s_{1}) < \hat{V}_{1}^{*}(s_{1})$  implies that  $\pi$  must reach a previously unseen state-action pair. We will show that this property implies that R-MAX must return an optimal policy.
In particular, note that after the  $j$ th loop iteration in R-MAX, it must either have an optimal policy or have explored at least  $j$  of the state-action pairs in the MDP. This is a simple consequence of the above property: at each iteration, either the policy used by R-MAX must be optimal or it must explore at least one additional state-action pair. This means that after all the SA loop iterations, R-MAX will either have an optimal policy or have explored all the state-action pairs, in which case it will also have an optimal policy.
D.2 Covering length
The covering length of an MDP was originally proposed by Even-Dar and Mansour [52] and later used by Liu and Brunskill [7] to prove sample complexity bounds on RL algorithms which use random exploration. Liu and Brunskill [7] show various bounds on the covering length using graph-theoretic notions. While they focus on discounted infinite-horizon MDPs, we use a version of covering length adapted to finite-horizon MDPs, similar to that used by Dann et al. [6].
Definition D.2 (Covering length). The covering length  $L$  of an MDP under an exploration policy  $\pi^{expl}$  is the number of episodes needed until all state-action pairs have been visited with probability at least  $1/2$ .
One can easily show sample complexity bounds based on the covering length.
Theorem D.3 (Covering length sample complexity bound). There is an RL algorithm which can solve any MDP with sample complexity  $TL$  given an exploration policy  $\pi^{expl}$ , where  $L$  is the covering length of  $\pi^{expl}$ .
1: procedure COVERINGLENGTHRL  $(\pi^{\mathrm{exp}},L)$
2: collect a dataset of  $L$  episodes, sampling actions according to  $\pi^{\mathrm{expl}}$
3: record  $\hat{R}(s, a)$  and  $\hat{f}(s, a)$  for all state-action pairs seen in the dataset
4: define  $\hat{R}(s, a)$  and  $\hat{f}(s, a)$  arbitrarily for state-action pairs not seen in the dataset
5: run value iteration using  $\hat{R}$  and  $\hat{f}$  to obtain a policy  $\pi$
6: return  $\pi$
7: end procedure
By the definition of covering length, with probability at least  $1/2$  the algorithm should produce  $\hat{R}(s,a) = R(s,a)$  and  $\hat{f}(s,a) = f(s,a)$  for all  $(s,a) \in S \times \mathcal{A}$ . In this case,  $\pi$  will be an optimal policy. Thus, COVERINGLENGTHRL returns an optimal policy with probability at least  $1/2$  while interacting with the environment for  $TL$  timesteps. This means the sample complexity of COVERINGLENGTHRL is at most  $TL$ .
To bound the covering length for MDPs in the BRIDGE dataset, we make use of the following result.
Lemma D.4 (Bounds on the covering length). Define the occupancy measure  $\mu$  of  $\pi^{expl}$  as
$$
\mu_ {t} (s, a) = \mathbb {P} _ {\pi^ {e x p l}} \left(s _ {t} = s \wedge a _ {t} = a\right).
$$
Suppose that for every state-action pair  $(s, a)$ , there is some timestep  $t$  when  $\mu_t(s, a) > 0$ . Then
$$
\frac {\log (2)}{2 \min  _ {(s , a) \in \mathcal {S} \times \mathcal {A}} \sum_ {t \in [ T ]} \mu_ {t} (s , a)} \leq L \leq \left\lceil \frac {\log (2 S A)}{\min  _ {(s , a) \in \mathcal {S} \times \mathcal {A}} \max  _ {t \in [ T ]} \mu_ {t} (s , a)} \right\rceil .
$$
We calculate  $\mu_{\mathrm{min}} = \min_{(s,a)\in S\times A}\max_{t\in [T]}\mu_t(s,a)$  for each MDP in BRIDGE and use the upper bound from Lemma D.4 to obtain a sample complexity bound of  $T\log (2SAT) / \mu_{\mathrm{min}}$ . Since  $\sum_{t\in [T]}\mu_t(s,a)\leq T\max_{t\in [T]}\mu_t(s,a)$  the upper and lower bounds in Lemma D.4 agree up to a factor of  $T\log (2SA) / \log (2)$ , so this is reasonably tight. In fact, in 122 of the 155 MDPs in BRIDGE,  $\min_{(s,a)\in S\times A}\sum_{t\in [T]}\mu_t(s,a) = \min_{(s,a)\in S\times A}\max_{t\in [T]}\mu_t(s,a)$ , making the upper and lower bounds tight up to only logarithmic factors.
Proof. Let  $\mathbb{P}_m$  be a probability measure corresponding to sampling  $m$  episodes following  $\pi^{\mathrm{expl}}$ . Let