SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We will show that Algorithm 2 converges with probability at least  $1/2$  with this choice of parameter, giving the sample complexity bound in the theorem since the algorithm clearly samples  $T^2 m$  total timesteps from the environment.
Similarly to the proof of Theorem 5.4, we will prove this by showing that with probability at least  $1 - 1/(2T)$ , at each iteration  $\pi_i(s_i) \in \arg \max_{a \in \mathcal{A}} Q_i^{\pi^{\mathrm{exp}}} (s_i, a)$ . This gives  $\pi \in \Pi(Q^1)$ , which by 1-QVI-solvability means  $\pi$  must be optimal.
Consider the  $i$ th iteration of Algorithm 2. Define for each  $a \in \mathcal{A}$  and  $j \in [m]$  the random variable
$$
X _ {j} (a) = \frac {\mathbf {1} _ {a _ {i} ^ {j} = a}}{\pi_ {i} (a \mid s _ {i})} \sum_ {t = i} ^ {T} \gamma^ {t - i} R (s _ {t} ^ {j}, a _ {t} ^ {j}).
$$
First, can see that
$$
0 \leq X _ {j} (a) \leq \frac {V _ {i} ^ {*} (s _ {i})}{\pi_ {i} (a \mid s _ {i})} = \frac {V _ {i} ^ {*} (s _ {i})}{\pi_ {i} ^ {\mathrm {e x p l}} (a \mid s _ {i})} \leq A V _ {i} ^ {*} (s _ {i}) \left\| \frac {\pi^ {\mathrm {r a n d}}}{\pi^ {\mathrm {e x p l}}} \right\| _ {\infty},
$$
since  $\pi_i = \pi_i^{\mathrm{expl}}$  until line 8.
Second, we have
$$
\mathbb {E} \left[ X _ {j} (a) \right] = \frac {1}{\pi_ {i} (a \mid s _ {i})} \mathbb {P} \left(a _ {i} ^ {j} = a\right) \mathbb {E} \left[ \sum_ {t = i} ^ {T} \gamma^ {t - i} R \left(s _ {t} ^ {j}, a _ {t} ^ {j}\right) \mid a _ {i} ^ {j} = a \right] = Q _ {i} ^ {\pi^ {\mathrm {e x p l}}} (s _ {i}, a).
$$
Finally, using same the reasoning as in the proof of Theorem 2, we can bound the variance of  $X_{j}(a)$ :
$$
\operatorname {V a r} \left(X _ {j}\right) \leq A V _ {i} ^ {*} (s _ {i}) \left\| \frac {\pi^ {\text {r a n d}}}{\pi^ {\text {e x p l}}} \right\| _ {\infty} \mathbb {E} \left[ X _ {j} \right] = A Q _ {i} ^ {\pi^ {\text {e x p l}}} (s _ {i}, a) V _ {i} ^ {*} (s _ {i}) \left\| \frac {\pi^ {\text {r a n d}}}{\pi^ {\text {e x p l}}} \right\| _ {\infty}.
$$
We now apply Bernstein's inequality to
$$
\hat {\nabla} _ {i} (a \mid s _ {i}) = \frac {1}{m} \sum_ {j = 1} ^ {m} X _ {j} (a)
$$
for each  $a \in \mathcal{A}$ . If  $a \in \arg \max_{a \in \mathcal{A}} Q_i^{\mathrm{exp}}(s_i, a)$ , we apply a lower tail bound:
$$
\begin{array}{l} \mathbb {P} \left(\hat {\nabla} _ {i} (a \mid s _ {i}) \leq Q _ {i} ^ {\pi^ {\exp}} (s _ {i}, a) - \frac {1}{2} \Delta_ {i} ^ {1} (s _ {i})\right) \\ \leq \exp \left\{- \frac {m \left(\Delta_ {i} ^ {1} (s _ {i})\right) ^ {2} / 8}{\left(Q _ {i} ^ {\pi^ {\mathrm {e x p l}}} (s _ {i} , a) + \frac {1}{3} \Delta_ {i} ^ {1} (s _ {i})\right) V _ {i} ^ {*} (s _ {i}) \left\| \frac {\pi^ {\mathrm {r a n d}}}{\pi^ {\mathrm {e x p l}}} \right\| _ {\infty}} \right\} \\ \leq \exp \left\{- \frac {m \left(\Delta_ {i} ^ {1} (s _ {i})\right) ^ {2}}{1 2 Q _ {i} ^ {\pi^ {\mathrm {e x p l}}} (s _ {i} , a) V _ {i} ^ {*} (s _ {i}) \left\| \left. \frac {\pi^ {\mathrm {r a n d}}}{\pi^ {\mathrm {e x p l}}} \right\| _ {\infty} \right.} \right\} \\ \leq \frac {1}{2 T A}. \\ \end{array}
$$
If  $a \notin \arg \max_{a \in \mathcal{A}} Q_i^{\pi^{\mathrm{exp}}}(s_i, a)$ , we apply an identical upper tail bound:
$$
\mathbb {P} \left(\hat {\nabla} _ {i} (a \mid s _ {i}) \geq Q _ {i} ^ {\pi^ {\mathrm {e x p l}}} \left(s _ {i}, a\right) + \frac {1}{2} \Delta_ {i} ^ {1} \left(s _ {i}\right)\right) \leq \frac {1}{2 T A}.
$$
These tail bounds hold simultaneously for all actions with probability at least  $1 - \frac{1}{2T}$ . Furthermore, assuming they hold and using the definition of the gap  $\Delta_i^1 (s_i)$ , it must be that
$$
\arg \max _ {a \in \mathcal {A}} \hat {\nabla} _ {i} (a \mid s _ {i}) \subseteq \arg \max _ {a \in \mathcal {A}} Q _ {i} ^ {\pi^ {\mathrm {e x p l}}} (s _ {i}, a),
$$
which is enough to show that  $\pi_i(s_i) \in \arg \max_{a \in \mathcal{A}} Q_i^{\pi^{\mathrm{expl}}}(s_i, a)$  with probability at least  $1 - \frac{1}{2T}$  and thus prove the theorem.
Now that we have seen that PG-GORP enjoys similar sample complexity bounds to GORP in the common case that an MDP is 1-QVI-solvable, we introduce FQI-GORP. FQI-GORP derives its name from fitted Q-iteration (FQI), which was originally proposed by Ernst et al. [49]. DQN was inspired by neural FQI [50], so FQI-GORP provides a natural connection to DQN.
Algorithm 3 The FQI-GORP algorithm.
1: procedure FQI-GORP(k,m)  
2: for  $i = 1,\ldots,T$  do  
3: Sample  $A^k m$  episodes, following  $\pi$  for timesteps 1 to  $i - 1$  and then  $\pi^{\mathrm{expl}}$ .  
4:  $\hat{Q}_{i+k-1}^1 \leftarrow \arg \min_{\hat{Q}_{i+k-1}^1} \frac{1}{A^k m} \sum_{j=1}^{A^k m} \left( \hat{Q}_{i+k-1}^1 (s_{i+k-1}^j, a_{i+k-1}^j) - \sum_{t=i+k-1}^{T} R(s_t^j, a_t^j) \right)^2$ .  
5: for  $t = i + k - 2,\ldots,i$  do  
6:  $\hat{Q}_t^{i+k-t} \leftarrow \arg \min_{\hat{Q}_t^{i+k-t}} \frac{1}{A^k m} \sum_{j=1}^{A^k m} \left( \hat{Q}_t^{i+k-t}(s_t^j, a_t^j) - R(s_t^j, a_t^j) - \max_{a' \in \mathcal{A}} \hat{Q}_{t+1}^{i+k-t-1}(s_{t+1}^j, a') \right)^2$ .  
7: end for  
8:  $\pi_i(s_i) \leftarrow \arg \max_{a \in \mathcal{A}} \hat{Q}_i^k(s_i, a)$ .  
9: end for  
10: return  $\pi$   
11: end procedure
FQI-GORP iteratively constructs a series of Q-functions at each iteration by minimizing a mean-squared temporal difference error loss, similar to FQI and DQN.
Theorem B.2 (Sample complexity of FQI-GORP). Suppose that an MDP is  $k$ -QVI-solvable and that all rewards are nonnegative. Then the sample complexity of FQI-GORP is at most
$$
T ^ {2} \left\| \frac {\pi^ {r a n d}}{\pi^ {e x p l}} \right\| _ {\infty} ^ {k} \max \left\{4 A ^ {\bar {H} _ {k}}, 1 0 \log (4 T A ^ {k}) \right\}.
$$
$$
m = \left\| \frac {\pi^ {\mathrm {r a n d}}}{\pi^ {\mathrm {e x p l}}} \right\| _ {\infty} ^ {k} \max  \left\{2 4 \log (2 T A) \max  _ {t \in [ T ], s \in \mathcal {S} _ {i} ^ {\mathrm {o p t}}, a \in \mathcal {A}} \left(\frac {Q _ {t} ^ {k} (s , a) V _ {t} ^ {*} (s)}{\Delta_ {t} ^ {1} (s) ^ {2}}\right), \frac {1 0 \log (4 T A ^ {k})}{A ^ {k}} \right\}.
$$
We will show that FQI-GORP with parameters  $k$  and  $m$  will return an optimal policy with probability at least  $1/2$ . Since FQI-GORP samples a number of timesteps from the environment equal to  $T^2 A^k m$ , this will prove the bound in the theorem.
Consider the  $i$ th iteration of Algorithm 2. We will show that with probability at least  $1 - 1 / (2T)$ , for every  $s_{i + k - 1} \in S$  reachable at timestep  $i + k - 1$  starting from  $s_i$ , and for every action  $a_{i + k - 1}$
$$
\left| \hat {Q} _ {i + k - 1} ^ {1} \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) - Q _ {i + k - 1} ^ {1} \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) \right| <   \frac {\Delta_ {i} ^ {1} \left(s _ {i}\right)}{2}. \tag {11}
$$
To prove (11), it is first helpful to write an explicit formula fitted Q-value, assuming that the loss in line 4 of Algorithm 3 is minimized:
$$
\hat {Q} _ {i + k - 1} ^ {1} (s, a) = \frac {\sum_ {j = 1} ^ {A ^ {k} m} \mathbf {1} _ {s _ {i + k - 1} ^ {j} = s \wedge a _ {i + k - 1} ^ {j} = a} \sum_ {t = i + k - 1} ^ {T} R (s _ {t} ^ {j} , a _ {t} ^ {j})}{\sum_ {j = 1} ^ {A ^ {k} m} \mathbf {1} _ {s _ {i + k - 1} ^ {j} = s \wedge a _ {i + k - 1} ^ {j} = a}}.
$$
That is, the Q-value is a simple average of several reward-to-go values, each of which has expectation  $Q_{i + k - 1}^{1}(s,a)$ . The probability of reaching some reachable state-action pair  $(s,a)$  at timestep  $i + k - 1$  must be at least  $A^{-k}\left\| \frac{\pi^{\mathrm{rand}}}{\pi^{\mathrm{exp}}}\right\|_{\infty}^{-k}$ . Thus, we can bound the sample size below via a concentration inequality for binomial random variables:
$$
\mathbb {P} \left(\sum_ {j = 1} ^ {A ^ {k} m} \mathbf {1} _ {s _ {i + k - 1} ^ {j} = s \wedge a _ {i + k - 1} ^ {j} = a} <   1 2 \log (2 T A) \max _ {a _ {i} \in \mathcal {A}} \frac {Q _ {i} ^ {k} (s _ {i} , a _ {i}) V _ {i} ^ {*} (s _ {i})}{\Delta_ {i} ^ {1} (s _ {i}) ^ {2}}\right) \leq \exp \{\frac {- 3 (1 0 \log (4 T A ^ {k}))}{2 8} \} \leq \frac {1}{4 T A ^ {k}}.
$$
If the sample size is at least  $12\log (2TA)\max_{a_i\in \mathcal{A}}\frac{Q_i^k(s_i,a_i)V_i^*(s_i)}{\Delta_i^1(s_i)^2}$ , then Bernstein's inequality (as applied in the proofs of Theorems 5.4 and B.1) gives
$$
\mathbb {P} \left(\left| \hat {Q} _ {i + k - 1} ^ {1} \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) - Q _ {i + k - 1} ^ {1} \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) \right| \geq \frac {\Delta_ {i} ^ {1} \left(s _ {i}\right)}{2}\right) \leq \frac {1}{4 T A ^ {k}}.
$$
Thus, taking a union bound, we have that the probability (11) does not hold for some reachable state-action pair at timestep  $i + k - 1$  must be at most  $1 / (2T)$ , since there can be at most  $A^k$  such pairs.
We will now show by induction that given that (11) holds for all reachable state-action pairs,
$$
\left| \hat {Q} _ {t} ^ {i + k - t} \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) - \hat {Q} _ {t} ^ {i + k - t} i + k - 1 \left(s _ {i + k - 1} ^ {j}, a _ {i + k - 1} ^ {j}\right) \right| <   \frac {\Delta_ {i} ^ {1} \left(s _ {i}\right)}{2}. \tag {12}
$$
holds for  $t = i + k - 1, \ldots, i$  for all reachable state-action pairs at  $t$ . The base case of  $t = i + k - 1$  is already taken care of, so we only need to show the inductive step.
Assume (12) holds for all reachable state-action pairs at  $t + 1$ . We can write an explicit formula for the fitted Q-values at timestep  $t$ , given that the loss function on line 6 of Algorithm 3 is minimized:
$$
\begin{array}{l} \hat {Q} _ {t} ^ {i + k - t} (s, a) = \frac {\sum_ {j = 1} ^ {A ^ {k} m} \mathbf {1} _ {s _ {t} ^ {j} = s \wedge a _ {t} ^ {j} = a} \left(R (s _ {t} ^ {j} , a _ {t} ^ {j}) + \max _ {a ^ {\prime} \in \mathcal {A}} \hat {Q} _ {t + 1} ^ {i + k - t - 1} (s _ {t + 1} ^ {j} , a ^ {\prime})\right)}{\sum_ {j = 1} ^ {A ^ {k} m} \mathbf {1} _ {s _ {t} ^ {j} = s \wedge a _ {t} ^ {j} = a}} \\ = R(s,a) + \max_{a^{\prime}\in \mathcal{A}}\hat{Q}_{t + 1}^{i + k - t - 1}(f(s,a),a^{\prime}). \\ \end{array}
$$
Given that (12) holds for each pair  $(f(s,a),a^{\prime})$ , it is now easy to see that (12) must hold for  $t$  as well.
By induction (12) must hold for  $t = i$  with probability at least  $1 - 1 / (2T)$ . Given that it holds, and by definition of the gap, this implies that  $\pi_i(s_i) \in \arg \max_{a \in \mathcal{A}} Q_i^k(s_i, a)$ . Thus with probability at least  $1/2$ ,  $\pi \in \Pi(Q^k)$ . By the assumption that the MDP is  $k$ -QVI solvable,  $\pi$  must be optimal with probability at least  $1/2$ .
B.2 Goal MDPs
Now, we will prove bounds on the effective horizon in one particular class of MDPs: goal MDPs.
Definition B.3 (Goal MDP). An MDP is considered a goal MDP if there is some set of goal states  $S_{\text{goal}}$  which are absorbing, i.e.,  $f(s, a) = s$  for every  $s \in S_{\text{goal}}$ , and furthermore the reward function is of the form
$$
R (s, a) = \left\{ \begin{array}{l l} 1 & \quad s \notin \mathcal {S} _ {\text {g o a l}} \wedge f (s, a) \in \mathcal {S} _ {\text {g o a l}} \\ 0 & \quad \text {o t h e r w i s e .} \end{array} \right.
$$
That is, in a goal MDP reward is only received for reaching some set of goal states; the total episode reward is 1 if a goal state is reached and 0 otherwise. As an example, all of the Minigrid environments in BRIDGE are goal MDPs. We can show the following bound on the effective horizon in goal MDPs.
Theorem B.4 (The effective horizon in goal MDPs). Suppose that  $\pi^{expl}(a\mid s) > 0$  for all  $s\in S$  and  $a\in \mathcal{A}$ . Then any goal MDP is 1-QVI-solvable. Furthermore, suppose that there is some  $p > 0$  such that, for all timesteps  $t\in [T]$  and all state-action pairs  $s_t,a_t$  at that timestep from which a goal state can be reached,
$$
\mathbb {P} _ {\pi^ {e x p l}} \left(s _ {T} \in \mathcal {S} _ {\text {g o a l}} \mid s _ {t}, a _ {t}\right) \geq p.
$$
Then the effective horizon can be bounded as
$$
H \leq 1 + \log_ {A} \frac {\log (2 T)}{p}. \tag {13}
$$
Before we see the proof, note that Theorem B.4 agrees with our intuition that it should be harder to find an optimal policy for a goal MDP when it is less likely that the exploration policy reaches the goal, i.e., when  $p$  is smaller.
For instance, consider the MDP in Figure 2a. In this MDP, the minimum probability of reaching the goal with the random exploration policy after taking some action is exponentially small:  $p = 1 / A^{T - 1}$ . Applying Theorem B.4 gives a bound of  $H \leq T + \log_A \log (2T)$ .
Figure 4: Empty-5x5, one of the Minigrid MDPs from BRIDGE and an example of a goal MDP (Definition B.3). The agent (red triangle) can turn left, turn right, or go forward, and its goal is to reach the green square, which gives a reward of 1.
In contrast, consider the Minigrid gridworld in Figure 4 from BRIDGE. Here we can bound  $p \approx 0.00137$ , which gives  $H \leq 1 + \log_A(1/p) + \log_A \log(2T) = 1 + \log_3 729 + \log_3 \log(200) \leq 7 + 1.52 = 8.52 \ll T = 100$ . The techniques in Appendix C give a much tighter bound of  $H \leq 1.64$ .
Proof. We can assume that  $V_{1}^{*}(s_{1}) = 1$ , since otherwise every trajectory in the MDP gives reward 0 and thus the effective horizon is trivially bounded. First, we will show that
$$
Q _ {t} ^ {\pi^ {\mathrm {e x p l}}} (s, a) > 0 \quad \Leftrightarrow \quad Q _ {t} ^ {*} (s, a) = 1. \tag {14}
$$
This is enough to imply the MDP is 1-QVI-solvable, since any policy in  $\Pi(Q^{\pi^{\mathrm{expl}}})$  must take only actions with  $Q_{t}^{\pi^{\mathrm{expl}}}(s,a) > 0$ , which must be optimal according to (14).
To show that (14) holds, first consider the  $\Rightarrow$  implication. Assume  $Q_{t}^{\pi^{\mathrm{exp}}}(s,a) > 0$  and by way of contradiction suppose that  $Q_{t}^{*}(s,a)\neq 1$ , which means that  $Q_{t}^{*}(s,a) = 0$ . Clearly this cannot happen since this would imply  $Q_{t}^{\pi^{\mathrm{exp}}}(s,a)\leq 0$ . Now, consider the  $\Leftarrow$  direction. If  $Q_{t}^{*}(s,a) = 1$ , then there must be some sequence of actions starting with  $a$  which leads from  $s$  to a goal state. By assumption,  $\pi^{\mathrm{exp}}$  assigns positive probability to each action in this sequence. Thus  $Q_{t}^{\pi^{\mathrm{exp}}}(s,a) = \mathbb{P}_{\pi^{\mathrm{exp}}}(s_{T}\in S_{\mathrm{goal}}\mid s,a) > 0$ .
Next, we will prove the bound on  $H$  from (13). From (14), we can see that Algorithm 1 (GORP) will return an optimal policy for the MDP as long as at each iteration  $i$  it picks some action  $a_i$  with  $Q_i^{\pi^{\mathrm{expl}}}(s_i, a_i) > 0$ . In turn, this will happen as long as  $\hat{Q}_i^1(s_i, a_i) > 0$  for some such  $a_i$ , since  $\hat{Q}_i^1(s_i, a)$  must be 0 for any suboptimal  $a$ .
Thus, we need only show that
$$
\mathbb {P} \left(\exists a _ {i} \in \mathcal {A} \quad \hat {Q} _ {i} ^ {1} \left(s _ {i}, a _ {i}\right) > 0\right) \geq 1 - \frac {1}{2 T} \tag {15}
$$
holds for each  $i$  when  $m \geq \log(2T)/p$ . This will allow us to conclude via a union bound that Algorithm 1 will find an optimal policy with probability at least  $1/2$  in this case, which gives the desired bound on the effective horizon.
To show (15), consider iteration  $i$  of Algorithm 1 and let  $a_{i} \in \mathcal{A}$  such that  $Q_{i}^{\pi^{\mathrm{rand}}}(s_{i}, a_{i}) > 0$ . We can assume that such an action exists as long as Algorithm 1 has succeeded in iterations previous to  $i$ . Let  $X_{j} = \sum_{t=i}^{T} \gamma^{t-i} R(s_{t}^{j}, a_{t}^{j})$  be the reward-to-go from the  $j$ th episode sampled to evaluate  $\hat{Q}_{i}^{1}(s_{i}, a_{i})$ . By the definition of a goal MDP, each  $X_{j}(a_{i}) \in \{0,1\}$ . Furthermore, since  $\hat{Q}_{i}^{1}(s_{i}, a_{i}) =$
$\frac{1}{m}\sum_{j = 1}^{m}X_{j}$ , then  $\hat{Q}_i^1 (s_i,a) > 0$  as long as some  $X_{j} = 1$ . This implies
$$
\begin{array}{l} \mathbb {P} \left(\exists a _ {i} \in \mathcal {A} \quad \hat {Q} _ {i} ^ {1} (s _ {i}, a _ {i}) > 0\right) \\ \geq \mathbb {P} \left(\hat {Q} _ {i} ^ {1} \left(s _ {i}, a _ {i}\right) > 0\right) \\ = \mathbb {P} \left(\exists j \in [ m ] \quad X _ {j} = 1\right) \\ = 1 - \mathbb {P} \left(\forall j \in [ m ] X _ {j} = 0\right) \\ = 1 - \mathbb {P} \left(X _ {j} = 0\right) ^ {m} \\ \geq 1 - (1 - p) ^ {m} \\ \geq 1 - \exp (- m p) \\ \geq 1 - \frac {1}{2 T}, \\ \end{array}
$$
the bound previously proposed in (15) which we argued gives the desired bound. (i) uses the assumption that  $Q_{i}^{\pi^{\mathrm{expl}}}(s_{i},a_{i}) = \mathbb{P}(X_{j} = 1)\geq p$ .
B.3 Lower bounds
Next, we will show that there are MDPs in which exponential dependence on the effective horizon is unavoidable. That is, in some cases there is an information-theoretic lower bound on the sample complexity of RL proportional to  $A^H$ .
Theorem B.5. Fix  $T \geq 1$ ,  $A \geq 2$ , and  $H \in [T]$ . Then for any RL algorithm, there is an MDP with  $A$  actions, horizon  $T$ , and effective horizon at most  $H$  for which the algorithm's sample complexity is at least
$$
T \lfloor T / H \rfloor (\lceil A ^ {H} / 2 \rceil - 1) = \Omega (T ^ {2} A ^ {H} / H).
$$
Note that this matches the upper bound on the sample complexity of GORP given in Lemma 5.3 up to a factor of roughly  $2H$ . When  $H = T$ , it exactly agrees with the lower bound given in Theorem 2.1.
Proof. The proof uses MDPs with the same state space and transition function as in Theorem 2.1. Define  $A^T$  such MDPs which differ only in their reward functions:
$$
\mathbb {M} = \{\mathcal {M} _ {a _ {1: T}} \mid a _ {1: T} \in \mathcal {A} ^ {T} \} \quad \text {w h e r e t h e M D P} \mathcal {M} _ {a _ {1: T}} \text {h a s} R (s _ {a _ {1: \ell} ^ {\prime}}, a ^ {\prime}) = \left\{ \begin{array}{l l} 1 & a _ {1: \ell} ^ {\prime}, a ^ {\prime} = a _ {1: \ell + 1} \text {a n d} \ell \equiv 0 \pmod {H} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
That is, each MDP has a single optimal sequence of actions that gives reward 1 every  $H$  timesteps.
By the same argument as in the proof of Theorem 2.1, for any RL algorithm there must be some MDP in  $\mathbb{M}$  such that after interacting with the environment for less than  $\lceil A^H /2\rceil -1$  episodes, the algorithm cannot with probability at least  $1 / 2$  identify the optimal actions for timesteps  $t = 1$  to  $t = H$ . We can repeat this line of reasoning for timesteps  $t = H + 1$  to  $t = 2H$ , and so on for a total of  $\lfloor T / H\rfloor$  steps, to show that with less than  $\lfloor T / H\rfloor$  ( $\lceil A^H /2\rceil -1$ ) episodes, there must be some MDP in  $\mathbb{M}$  whose optimal action sequence cannot be identified with probability greater than  $(1 / 2)^{\lfloor T / H\rfloor}\leq 1 / 2$ . Thus, the sample complexity of the RL algorithm on this MDP must be at least
$$
T \lfloor T / H \rfloor \left(\lceil A ^ {H} / 2 \rceil - 1\right),
$$
It only remains to be shown that the effective horizon of the MDPs in  $\mathbb{M}$  is actually  $H$ . To see why, consider running Algorithm 1 with  $k = H$  and  $m = 1$ . That is, at each iteration  $i$ , GORP will try all  $H$ -length action sequences followed by actions from  $\pi^{\mathrm{expl}}$ . Then, it will pick the action sequence with the highest empirical reward-to-go. From the definition of the MDPs in  $\mathbb{M}$ , all action sequences starting with a suboptimal action must have empirical reward-to-go of 0. Furthermore, at least one  $H$ -length action sequence starting with an optimal action must get reward-to-go of at least 1. Thus, GORP will with probability 1 choose an optimal action at each timestep. This means that the effective horizon must be at most  $H + \log_A 1 = H$ .
C Tighter bounds on the effective horizon
In Theorem 5.4, we obtained bounds on the effective horizon and thus on the sample complexity of GORP. However, we find that the bounds given by Theorem 5.4 are often very loose compared to the empirical performance of GORP due to two factors. First, Theorem 5.4 requires considering the worst case of  $Q_{t}^{k}(s,a)V_{t}^{*}(s) / \Delta_{t}^{k}(s)^{2}$  over all optimal states. However, in many MDPs in our dataset, there are optimal states with extremely small gaps that in practice are almost never reached by GORP. When Theorem 5.4 is applied, these states make the sample complexity bounds very large despite GORP working well empirically. Second, Theorem 5.4 uses asymptotically tight techniques for bounding the sample complexity that can be quite loose for small sample sizes. Below, we describe the algorithm we use to provably bound the sample complexity of GORP (and thus the effective horizon) that gives much tighter results.
Consider the GORP algorithm as given in Algorithm 1. Let  $a_{t}$  denote the random variable corresponding to the action ultimately chosen by the algorithm for timestep  $t$ . Let  $s_{t}$  denote the state reached by actions  $a_{1},\ldots ,a_{t - 1}$ . Denote by  $\mathbb{P}_m$  the probability measure given by running the algorithm with parameter  $m$ . We would like to bound the probability that the algorithm does not achieve the optimal return in the MDP. Let this event be denoted as
$$
\mathcal {E} := \sum_ {t = 1} ^ {T} R _ {t} \left(s _ {t}, a _ {t}\right) <   V _ {1} ^ {*} \left(s _ {1}\right)
$$
$$
\Leftrightarrow \exists t a _ {t} \notin \mathcal {A} _ {t} ^ {*} (s _ {t}),
$$
where  $\mathcal{A}_t^* (s)$  denotes the set of optimal actions in state  $s$  at timestep  $t$ , i.e.
$$
\mathcal {A} _ {t} ^ {*} (s) = \arg \max  _ {a \in \mathcal {A}} Q _ {t} ^ {*} (s).
$$
It is straightforward to see from Algorithm 1 that it requires  $T^2 A^k m$  timesteps of interaction the environment. Thus the sample complexity of the algorithm is
$$
T ^ {2} A ^ {k} m \quad \text {w h e r e} \quad m = \min  \{m \in \mathbb {N} \mid \mathbb {P} _ {m} (\mathcal {E}) <   1 / 2 \}.
$$
Clearly, we can upper bound the sample complexity using any  $m$  such that the probability of failure is bounded as  $\mathbb{P}_m(\mathcal{E}) < 1/2$ . To do so, for each value of  $k$ , we perform a binary search over values of  $m$  from 1 to  $10^{100}$ . For each possible  $m$ , we calculate an upper bound on  $\mathbb{P}_m(\mathcal{E})$ . If the upper bound is below  $1/2$ , we then search below  $m$ ; if it is greater, we search above  $m$ . When the search has converged to a relative precision of  $1/100$ , we output  $T^2 A^k m$  as the sample complexity and  $H_k = k + \log_A m$  as the effective horizon for that particular value of  $k$ .
C.1 Upper bounding  $\mathbb{P}_m(\mathcal{E})$
We upper bound the failure probability  $\mathbb{P}_m(\mathcal{E})$  recursively. Let  $\mathcal{O}_t$  denote the event that all actions taken before  $t$  have been optimal, i.e.,
$$
\mathcal {O} _ {t} := \forall t ^ {\prime} <   t \quad a _ {t} \in \mathcal {A} _ {t} ^ {*} (s _ {t}).
$$
To begin the recursion, note that at the final timestep,
$$
\mathbb {P} (\mathcal {E} \mid \mathcal {O} _ {T}, s _ {T}, a _ {T}) = \mathbf {1} \{a _ {T} \notin \mathcal {A} _ {T} ^ {*} (s _ {T}) \}.
$$
We will use two recursion rules: one from next states to state-action pairs and one from state-action pairs to states. The first rule is
$$
\mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t}, s _ {t}\right) = \sum_ {a _ {t} \in \mathcal {A}} \mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t}, s _ {t}, a _ {t}\right) \mathbb {P} \left(a _ {t} \mid s _ {t}\right) \tag {16}
$$
and the second (for  $t < T$ ) is
$$
\mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t}, s _ {t}, a _ {t}\right) = \left\{ \begin{array}{l l} 1 & a _ {t} \notin \mathcal {A} _ {t} ^ {*} \left(s _ {t}\right) \\ \mathbb {P} \left(\mathcal {E} \mid \mathcal {O} _ {t + 1}, s _ {t + 1}\right) & a _ {t} \in \mathcal {A} _ {t} ^ {*} \left(s _ {t}\right). \end{array} \right. \tag {17}
$$
We apply these results recursively from  $t = T,\ldots ,1$  to finally obtain  $\mathbb{P}(\mathcal{E}\mid \mathcal{O}_1,s_1) = \mathbb{P}(\mathcal{E})$
The remaining difficulty is calculating  $\mathbb{P}(a_t\mid s_t)$ . Recall from Algorithm 1 that  $a_{t}$  is chosen as the first action of a  $k$ -action sequence
$$
a _ {t: t + k - 1} \in \arg \max  _ {a _ {t: t + k - 1} \in \mathcal {A} ^ {k}} \hat {Q} _ {t} (s _ {t}, a _ {t: t + k - 1}),
$$