SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since the MDP is deterministic, an RL algorithm only needs to find an optimal sequence of actions. Clearly, there is at least a  $1/2$  chance that some optimal sequence of actions is in the first  $\lceil A^T / 2 \rceil$  elements of  $\mathcal{T}$ . If this is the case, then EXHAUSTIVESEARCH will return an optimal policy corresponding to that optimal sequence. Since the number of environment timesteps taken by EXHAUSTIVESEARCH is equal to  $T \lceil A^T / 2 \rceil$ , we have that  $N \leq T \lceil A^T / 2 \rceil$ .
For the converse, fix  $A$  and  $T$  along with any RL algorithm. Consider a set of states indexed by sequences of actions of length 0 to  $T$ :
$$
\mathcal {S} = \left\{s _ {a _ {1: \ell}} \mid \ell \in 0, \dots , T, a _ {1: \ell} \in \mathcal {A} ^ {\ell} \right\}.
$$
Then, define a transition function
$$
f (s _ {a _ {1: \ell}}, a) = s _ {a _ {1: \ell}, a}.
$$
Now consider  $A^T$  different MDPs which share the state space  $S$  and transition function  $f$ , differing only in their reward functions:
$$
\mathbb {M} = \{\mathcal {M} _ {a _ {1: T}} \mid a _ {1: T} \in \mathcal {A} ^ {T} \} \quad \text {w h e r e t h e M D P} \mathcal {M} _ {a _ {1: T}} \text {h a s} R (s _ {a _ {1: \ell}}, a) = \left\{ \begin{array}{l l} 1 & a _ {1: \ell}, a = a _ {1: T} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
That is, each MDP has a single optimal sequence of actions that gives reward 1 on the final timestep; all other rewards are 0.
Let the RL algorithm in question take in some source of randomness  $z$  and output a policy  $\pi_t^n (a \mid s;z,\mathcal{M})$  after  $n$  timesteps in MDP  $\mathcal{M}$ . Now suppose by way of contradiction the sample complexity of the algorithm is less than  $T(\lceil A^T /2\rceil -1)$  in all MDPs in  $\mathbb{M}$ . By our definition of sample complexity, this means that
$$
\forall \mathcal {M} \in \mathbb {M} \quad \mathbb {P} _ {z} \left(J \left(\pi_ {t} ^ {T (\lceil A ^ {T} / 2 \rceil - 2)} (\cdot ; z, \mathcal {M})\right) = 1\right) \geq 1 / 2. \tag {1}
$$
Clearly a policy can only be optimal in these MDPs if it is deterministic, so let  $\tau(z,\mathcal{M})$  be the sequence of actions that  $\pi_t^{T(\lceil A^T/2 \rceil - 2)}(a \mid s; z, \mathcal{M})$  takes if it is optimal, and let it be any suboptimal sequence of actions if the policy is suboptimal. We can rewrite (1) as
$$
\forall \mathcal {M} _ {a _ {1: T}} \in \mathbb {M} \quad \mathbb {P} _ {z} (\tau (z, \mathcal {M} _ {a _ {1: T}}) = a _ {1: T}) \geq 1 / 2.
$$
Letting  $\mathrm{Unif}(\mathcal{A}^T)$  define a uniform distribution over action sequences of length  $T$ , this implies
$$
\mathbb {P} _ {a _ {1: T} \sim \operatorname {U n i f} \left(\mathcal {A} ^ {T}\right), z} \left(\tau (z, \mathcal {M} _ {a _ {1: T}}) = a _ {1: T}\right) \geq 1 / 2
$$
which means that there must be some particular  $z$  such that
$$
\mathbb {P} _ {a _ {1: T} \sim \operatorname {U n i f} (\mathcal {A} ^ {T})} \left(\tau (z, \mathcal {M} _ {a _ {1: T}}) = a _ {1: T}\right) \geq 1 / 2
$$
$$
\left| \left\{a _ {1: T} \in \mathcal {A} ^ {T} \mid \tau (z, \mathcal {M} _ {a _ {1: T}}) = a _ {1: T} \right\} \right| \geq A ^ {T} / 2 \tag {2}
$$
Given that the RL algorithm is now deterministic due to the fixed  $z$ , we will prove that the LHS of (2) must be less than or equal to  $\lceil A^T / 2 \rceil - 1$ . This can be shown via induction. For the first episode, the algorithm must take the same actions in every MDP since all MDPs give zero reward until the final action (so there is no way to distinguish them). After the first episode, only one MDP can be distinguished from the others: the one corresponding to the action sequence taken in the first episode, which has reward 1 instead of 0. Thus in the remaining  $A^T - 1$  MDPs the algorithm must take the same actions in the second episode. Continuing this argument shows that after episode  $\lceil A^T / 2 \rceil - 2$ , the algorithm must still be unable to distinguish between  $A^T - (\lceil A^T / 2 \rceil - 2) = \lfloor A^T / 2 \rfloor + 2$  of the MDPs, and so  $\tau(z, \mathcal{M})$  must be the same for all MDPs in this set. Since all of these MDPs have different optimal action sequences,  $\tau(z, \mathcal{M})$  can only be optimal in one of them. Thus  $\tau(z, \mathcal{M})$  must be suboptimal in at least  $\lfloor A^T / 2 \rfloor + 1$  MDPs, which means the LHS of (2) must be at most  $A^T - (\lfloor A^T / 2 \rfloor + 1) = \lceil A^T / 2 \rceil - 1$ .
Combining this with (2) gives  $\lceil A^T / 2 \rceil - 1 \geq A^T / 2$ , which is a contradiction. Thus, the sample complexity of the RL algorithm must be at least  $T(\lceil A^T / 2 \rceil - 1)$ .
A.2 Proof that  $k = T$  in the worst case
Lemma A.1. Let  $Q^1 = Q^{\pi^{rand}}$ ,  $Q^{i+1} = QVI(Q^i)$  for  $i = 2, \ldots, T$ , and  $\Pi(Q^i)$  be defined as in Section 4. Then for any horizon  $T$  and number of actions  $A \geq 2$  there is an MDP such that no policy in  $\Pi(Q^i)$  is optimal for  $i < T$ .
Proof. As in the proof of Theorem 2.1, define an MDP where every action sequence leads to a different state. Pick an arbitrary action sequence  $a_{1:T}$ , and let the reward of taking the final action in that sequence be 1:
$$
R \left(s _ {a _ {1: T - 1}}, a _ {T}\right) = 1.
$$
Now take some action  $a_1' \neq a_1$ , and let the reward for taking that action at the beginning of an episode be  $3/4$ :
$$
R (s _ {1}, a _ {1} ^ {\prime}) = 3 / 4.
$$
Let the rewards for all other state-action pairs be 0. We will show by induction that
$$
Q _ {t} ^ {i} \left(s _ {a _ {1: t - 1}}, a _ {t}\right) = \frac {1}{A ^ {\max  \{T - i - t + 1 , 0 \}}} \quad \text {a n d} \quad Q _ {1} ^ {i} \left(s _ {1}, a _ {1} ^ {\prime}\right) = 3 / 4. \tag {3}
$$
The left half of (3) is clearly true for  $i = 1$ , since the random policy will take all action sequences following the  $t - 1$ -th optimal action with probability  $1 / A^{T - t}$ , and exactly one of those gives reward 1. The right half is also clear since following  $a_1'$  gives immediate reward of  $3/4$  and then no reward afterwards.
For the inductive step, we begin with the left half of (3); assume it holds for some  $i$ . By the definition of Q-value iteration, we have for  $t < T$
$$
\begin{array}{l} Q _ {t} ^ {i + 1} \left(s _ {a _ {1: t - 1}}, a _ {t}\right) \\ \stackrel {\text {(i)}} {=} R \left(s _ {a _ {1: t - 1}}, a _ {t}\right) + \max  _ {a} Q _ {t + 1} ^ {i} \left(s _ {a _ {1: t}}, a\right) \\ = 0 + \frac {1}{A ^ {\max  \{T - i - (t + 1) + 1 , 0 \}}} \\ = \frac {1}{A ^ {\max  \{T - (i + 1) - t + 1 , 0 \}}} \\ \end{array}
$$
and for  $t = T$
$$
Q _ {T} ^ {i + 1} \left(s _ {a _ {1: T - 1}}, a _ {T}\right) = R \left(s _ {a _ {1: T - 1}}, a _ {T}\right) = 1 = \frac {1}{A ^ {0}} = \frac {1}{A ^ {\operatorname* {m a x} \{T - (i + 1) - t + 1 , 0 \}}}.
$$
(i) is because only one action at timestep  $t + 1$  can have a Q-value greater than 0, since only one action sequence leads to reward after taking  $a_1$ . For the right half of (3), we have
$$
Q _ {1} ^ {i + 1} (s _ {1}, a _ {1} ^ {\prime}) = R (s _ {1}, a _ {1} ^ {\prime}) + \max _ {a} Q _ {2} ^ {i} (s _ {a _ {1} ^ {\prime}}, a) = 3 / 4 + 0 = 3 / 4.
$$
which is due to no reward being possible after taking  $a_1'$  at the first timestep.
Given that (3) holds for  $i = 1, \ldots, T$ , it is easy to see that for  $i < T$ , any  $\pi \in \Pi(Q^i)$  will take  $a_1'$  on the first timestep, since
$$
Q _ {1} ^ {i} (s _ {1}, a _ {1}) = \frac {1}{A ^ {\max \{T - i , 0 \}}} = \frac {1}{A ^ {T - i}} \leq \frac {1}{A} \leq \frac {1}{2} \leq \frac {3}{4} = Q _ {1} ^ {i} (s _ {1}, a _ {1} ^ {\prime}).
$$
This means that  $\pi$  will be suboptimal, since  $a_1$  is the only optimal action at  $t = 1$ .
A.3 Proof of Lemma 5.3
Proof. Recall the definition of effective horizon:
Let  $k \in \arg \min_k H_k$ . Then by Definition 5.2, GORP (Algorithm 1) with parameters  $k, m_k$  will converge to an optimal policy with probability at least  $1/2$ . Clearly, Algorithm 1 interacts with the environment for  $T$  iterations, each of which require evaluating  $A^k$  action sequences with  $m_k$  episodes of  $T$  timesteps each, for a total of
$$
T ^ {2} A ^ {k} m _ {k} = T ^ {2} A ^ {H _ {k}} = T ^ {2} A ^ {H}
$$
timesteps. Thus the sample complexity of GORP satisfies  $N_{\mathrm{GORP}} \leq T^2 A^H$ .
Now, suppose by way of contradiction that  $N_{\mathrm{GORP}} < T^2 A^H$ . Then this must mean that there are parameters  $k', m'$  such running GORP with these parameters converges to an optimal policy with probability at least  $1/2$ , and
$$
k ^ {\prime} + \log_ {A} m ^ {\prime} <   k + \log_ {A} m _ {k}.
$$
By Definition 5.2, this means that  $m_{k'} \leq m'$ , and thus
$$
H = \min  _ {k} H \leq k ^ {\prime} + \log_ {A} m _ {k ^ {\prime}} \leq k ^ {\prime} + \log_ {A} m _ {\prime} <   k + \log_ {A} m _ {k} = H
$$
which is clearly a contradiction. Thus, it must be that  $N_{\mathrm{GORP}} = T^2 A^H$
A.4 Proof of Theorem 5.4
Theorem 5.4. Suppose that an MDP is  $k$ -QVI-solvable and that all rewards are nonnegative, i.e.  $R(s,a) \geq 0$  for all  $s,a$ . Let  $\Delta^k$  denote the gap of the  $Q$ -function  $Q^k$  as defined in Definition 5.1. Then
$$
m = \log \left(2 T A ^ {k}\right) \max  _ {t \in [ T ], s \in \mathcal {S} _ {t} ^ {\mathrm {o p t}}, a \in \mathcal {A}} \frac {6 Q _ {t} ^ {k} (s , a) V _ {t} ^ {*} (s)}{\Delta_ {t} ^ {k} (s) ^ {2}}.
$$
We will show that GORP (Algorithm 1) converges to the optimal policy with probability at least  $1/2$  given parameters  $k$  and  $m$ . By Definition 5.2, this means the effective horizon must be at most  $k + \log_A m$ , which gives the bound in the theorem. More precisely, we will show that GORP converges to a policy in  $\Pi(Q^k)$  with probability at least  $1/2$ , which must be optimal because of the assumption that the MDP is  $k$ -QVI-solvable.
First, we will show the following relationship between the  $k$ -action  $Q^1$  values and  $Q^k$ :
$$
Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right) = \max  _ {a _ {i + 1: i + k - 1} \in \mathcal {A} ^ {k - 1}} Q _ {i} ^ {1} \left(s _ {i}, a _ {i}, a _ {i + 1: i + k - 1}\right). \tag {4}
$$
We prove that (4) holds inductively. For  $k = 1$ , (4) is obviously true. Supposing it holds for  $k$ , then
$$
\begin{array}{l} Q _ {i} ^ {k + 1} \left(s _ {i}, a _ {i}\right) = \operatorname {Q V I} \left(Q _ {i} ^ {k}\right) \left(s _ {i}, a _ {i}\right) \\ = R \left(s _ {i}, a _ {i}\right) + \max  _ {a _ {i + 1} \in \mathcal {A}} Q _ {i + 1} ^ {k} \left(f \left(s _ {i}, a _ {i}\right), a _ {i + 1}\right) \\ \stackrel {\mathrm {(i)}} {=} R \left(s _ {i}, a _ {i}\right) + \max  _ {a _ {i + 1: i + k} \in \mathcal {A} ^ {k}} Q _ {i + 1} ^ {1} \left(f \left(s _ {i}, a _ {i}\right), a _ {i + 1: i + k}\right) \\ = \max  _ {a _ {i + 1: i + k} \in \mathcal {A} ^ {k}} Q _ {i} ^ {1} (s _ {i}, a _ {i}, a _ {i + 1: i + k}), \\ \end{array}
$$
which shows (4) holds for  $k + 1$ . (i) holds due to the inductive hypothesis.
Recall that in Algorithm 1, we use  $\hat{Q}_i(s_i,a_{i:i + k - 1})$  to denote the estimated Q-value of the  $k$ -action sequence  $a_{i:i + k - 1}$ . Analogously to (4), define
$$
\hat {Q} _ {i} ^ {k} \left(s _ {i}, a _ {i}\right) = \max  _ {a _ {i + 1: i + k - 1} \in \mathcal {A} ^ {k - 1}} \hat {Q} _ {i} \left(s _ {i}, a _ {i}, a _ {i + 1: i + k - 1}\right)
$$
to be the maximum estimated Q-value of any action sequence that starts with  $a_i$ . We can rewrite line 7 of Algorithm 1 as
$$
\pi_ {i} (s _ {i}) \leftarrow \arg \max  _ {a _ {i} \in \mathcal {A}} \hat {Q} _ {i} ^ {k} (s _ {i}, a _ {i}).
$$
That is, the action that GORP selects for timestep  $i$  is chosen from those with the highest values of  $\hat{Q}_i^k (s_i,a_i)$ . Suppose we can show that
$$
\mathbb {P} \left(\arg \max  _ {a _ {i} \in \mathcal {A}} \hat {Q} _ {i} ^ {k} \left(s _ {i}, a _ {i}\right) \subseteq \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right)\right) \geq 1 - \frac {1}{2 T} \tag {5}
$$
holds for each  $i\in [T]$  . Then by a union bound,
$$
\mathbb {P} \left(\forall i \in [ T ] \quad \pi_ {i} \left(s _ {i}\right) \in \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right)\right) \geq \frac {1}{2}. \tag {6}
$$
This implies that  $\pi \in \Pi(Q^k)$ , which is the desired result.
It remains to show that (5) holds. We will actually prove the bound assuming that  $s_i \in S_i^{\mathrm{opt}}$ . This is still sufficient to imply (6) since one can inductively assume that previous actions are optimal. We can write (5) equivalently as
$$
\mathbb {P} \left(\exists a _ {i} \in \arg \max  _ {a _ {i} \in \mathcal {A}} \hat {Q} _ {i} ^ {k} \left(s _ {i}, a _ {i}\right): a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right)\right) \leq \frac {1}{2 T}. \tag {7}
$$
Let  $a_{i:i+k-1}^* \in \arg \max_{a_{i:i+k-1} \in \mathcal{A}^k} Q_i^1(s_i, a_{i:i+k-1})$  be chosen arbitrarily. By (4), this implies that  $a_i^* \in \arg \max_{a_i \in \mathcal{A}} Q_i^k(s_i, a_i)$ . Then
$$
\begin{array}{l} \mathbb {P} \left(\exists a _ {i} \in \arg \max  _ {a _ {i} \in \mathcal {A}} \hat {Q} _ {i} ^ {k} (s _ {i}, a _ {i}): a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} (s _ {i}, a _ {i})\right) \\ \leq \mathbb {P} \left(\exists a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} (s _ {i}, a _ {i}): \hat {Q} _ {i} ^ {k} (s _ {i}, a _ {i}) \geq \hat {Q} _ {i} ^ {k} (s _ {i}, a _ {i} ^ {*})\right) \\ \leq \mathbb {P} \left(\exists a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} (s _ {i}, a _ {i}), a _ {i + 1: i + k - 1} \in \mathcal {A} ^ {k - 1}: \hat {Q} _ {i} (s _ {i}, a _ {i}, a _ {i + 1: k - 1}) \geq \hat {Q} _ {i} (s _ {i}, a _ {i: i + k - 1} ^ {*})\right) \\ \leq \sum_ {a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right), a _ {i + 1: i + k - 1} \in \mathcal {A} ^ {k - 1}} \mathbb {P} \left(\hat {Q} _ {i} \left(s _ {i}, a _ {i}, a _ {i + 1: k - 1}\right) \geq \hat {Q} _ {i} \left(s _ {i}, a _ {i: i + k - 1} ^ {*}\right)\right). \tag {8} \\ \end{array}
$$
Consider a single term of the sum in (8). By the definition of the gap and the fact that  $a_i \notin \arg \max_{a_i \in \mathcal{A}} Q_i^k(s_i, a_i)$ , we know that
$$
Q _ {i} ^ {k} \left(s _ {i}, a _ {i}\right) \leq Q _ {i} ^ {k} \left(s _ {i}, a _ {i} ^ {*}\right) - \Delta_ {i} ^ {k} \left(s _ {i}\right).
$$
Combining this with (4) implies that
$$
Q _ {i} ^ {1} \left(s _ {i}, a _ {i: i + k - 1}\right) \leq Q _ {i} ^ {1} \left(s _ {i}, a _ {i: i + k - 1} ^ {*}\right) - \Delta_ {i} ^ {k} \left(s _ {i}\right)
$$
$$
Q _ {i} ^ {1} \left(s _ {i}, a _ {i: i + k - 1} ^ {*}\right) - Q _ {i} ^ {1} \left(s _ {i}, a _ {i: i + k - 1}\right) \geq \Delta_ {i} ^ {k} \left(s _ {i}\right). \tag {9}
$$
Now, consider the random variables  $\hat{Q}_i(s_i,a_i,a_{i+1:k-1})$  and  $\hat{Q}_i(s_i,a_{i:i+k-1}^*)$ . Let  $X_j = \sum_{t=i}^{T}\gamma^{t-i}R(s_t^j,a_t^j)$  be the discounted reward from timestep  $i$  in the  $j$ th episode used to estimate  $\hat{Q}_i(s_i,a_i,a_{i+1:i+k-1})$ ; let  $Y_j$  be the equivalent for estimating  $\hat{Q}_i(s_i a_{i:i+k-1}^*)$ . Then we can write
$$
\hat {Q} _ {i} \left(s _ {i}, a _ {i}, a _ {i + 1: i + k - 1}\right) = \frac {1}{m} \sum_ {j = 1} ^ {m} X _ {j} \quad \hat {Q} _ {i} \left(s _ {i}, a _ {i: i + k - 1} ^ {*}\right) = \frac {1}{m} \sum_ {j = 1} ^ {m} Y _ {j}.
$$
By the definition of the optimal value function  $V^{*}$  and the assumption that all rewards are nonnegative, we have that  $X_{j}, Y_{j} \in [0, V_{i}^{*}(s_{i})]$ . We also know that the expectations of the  $X_{j}$  and  $Y_{j}$  are bounded:
$$
\mathbb {E} [ X _ {j} ] = Q _ {i} ^ {1} (s _ {i}, a _ {i: i + k - 1}) \leq Q _ {i} ^ {k} (s _ {i}, a _ {i}) \leq \max  _ {a _ {i}} Q _ {i} ^ {k} (s _ {i}, a _ {i}).
$$
$$
\mathbb {E} [ Y _ {j} ] = Q _ {i} ^ {1} (s _ {i}, a _ {i: i + k - 1} ^ {*}) = Q _ {i} ^ {k} (s _ {i}, a _ {i} ^ {*}) \leq \max _ {a _ {i}} Q _ {i} ^ {k} (s _ {i}, a _ {i}).
$$
The variance of a random variable with mean  $\mu$  bounded on an interval  $[\alpha, \beta]$  is at most  $(\beta - \mu)(\mu - \alpha)$ . This means we can bound the variance of  $X_{j}$  and  $Y_{j}$  as well:
$$
\operatorname {V a r} (X _ {j}) \leq \max  _ {a _ {i}} Q _ {i} ^ {k} (s _ {i}, a _ {i}) V _ {i} ^ {*} (s _ {i}) \quad \operatorname {V a r} (Y _ {j}) \leq \max  _ {a _ {i}} Q _ {i} ^ {k} (s _ {i}, a _ {i}) V _ {i} ^ {*} (s _ {i}).
$$
Now define
$$
Z = \hat {Q} _ {i} ^ {1} (s _ {i}, a _ {i: i + k - 1} ^ {*}) - \hat {Q} _ {i} ^ {1} (s _ {i}, a _ {i}, a _ {i + 1: i + k - 1}) = \frac {1}{m} \sum_ {j = 1} ^ {m} Y _ {j} - X _ {j} = \frac {1}{m} \sum_ {j = 1} ^ {m} Z _ {j},
$$
where  $Z_{j} = Y_{j} - X_{j}$ . Since  $X_{j}$  and  $Y_{j}$  are independent,
$$
\operatorname {V a r} (Z _ {j}) = \operatorname {V a r} (X _ {j}) + \operatorname {V a r} (Y _ {j}) \leq 2 \max  _ {a _ {i}} Q _ {i} ^ {k} (s _ {i}, a _ {i}) V _ {i} ^ {*} (s _ {i}).
$$
Also define centered versions  $\bar{Z}_j = Z_j - \mathbb{E}[Z_j]$  and  $\bar{Z} = Z - \mathbb{E}[Z]$ . By (9),  $\mathbb{E}[Z] \geq \Delta_i^k(s_i)$ . Furthermore, since  $X_j, Y_j \in [0, V_i^*(s_i)]$ , we also know that  $|Z_j| \leq V_i^*(s_i)$  and  $|\bar{Z}_j| \leq V_i^*(s_i) + \mathbb{E}[Z] \leq 2V_i^*(s_i)$ .
We can now finally apply Bernstein's inequality to bound the probability of one term in (8):
$$
\begin{array}{l} \mathbb {P} \left(\hat {Q} _ {i} \left(s _ {i}, a _ {i}, a _ {i + 1: k - 1}\right) \geq \hat {Q} _ {i} \left(s _ {i}, a _ {i: i + k - 1} ^ {*}\right)\right) \\ = \mathbb {P} (Z \leq 0) \\ = \mathbb {P} \left(\bar {Z} \leq - \mathbb {E} [ Z ]\right) \\ \leq \mathbb {P} \left(\bar {Z} \leq - \Delta_ {i} ^ {k} (s _ {i})\right) \\ \stackrel {\mathrm {(i)}} {\leq} \exp \left\{\frac {- \frac {1}{2} m \Delta_ {i} ^ {k} (s _ {i}) ^ {2}}{\operatorname {V a r} (Z _ {j}) + \frac {2}{3} V _ {i} ^ {*} (s _ {i}) \Delta_ {i} ^ {k} (s _ {i})} \right\} \\ \stackrel {\mathrm {(i i)}} {\leq} \exp \left\{\frac {- \frac {1}{2} m \Delta_ {i} ^ {k} (s _ {i}) ^ {2}}{2 \max _ {a _ {i}} Q _ {i} ^ {k} (s _ {i} , a _ {i}) V _ {i} ^ {*} (s _ {i}) + \frac {2}{3} V _ {i} ^ {*} (s _ {i}) \Delta_ {i} ^ {k} (s _ {i})} \right\} \\ \stackrel {\mathrm {(i i i)}} {\leq} \exp \left\{\frac {- m \Delta_ {i} ^ {k} (s _ {i}) ^ {2}}{6 \max _ {a _ {i}} Q _ {i} ^ {k} (s _ {i} , a _ {i}) V _ {i} ^ {*} (s _ {i})} \right\} \\ \stackrel {\text {(i v)}} {\leq} \exp \left(- \log \left(2 T A ^ {k}\right)\right) \\ = \frac {1}{2 T A ^ {k}}. \tag {10} \\ \end{array}
$$
Here, (i) is a direct application of Bernstein's inequality to the sum  $\bar{Z} = \frac{1}{m}\sum_{j=1}^{m}\bar{Z}_j$ . (ii) uses the bound on  $\operatorname{Var}(Z_j) = \operatorname{Var}(\bar{Z}_j)$  and (iii) uses the fact that  $\Delta_i^k(s_i) \leq \max_{a_i} Q_i^k(s_i, a_i)$  by definition of the gap  $\Delta_i^k$ . Finally, (iv) uses the definition of  $m$ ; since  $s_i \in S_i^{\mathrm{opt}}$  by assumption,
$$
m \geq \log \left(2 T A ^ {k}\right) \max _ {a _ {i} \in \mathcal {A}} \frac {6 Q _ {t} ^ {k} (s _ {i} , a _ {i}) V _ {t} ^ {*} (s _ {i})}{\Delta_ {t} ^ {k} (s _ {i}) ^ {2}}.
$$
Applying (10) to (8) gives
$$
\begin{array}{l} \mathbb {P} \left(\exists a _ {i} \in \arg \max  _ {a _ {i} \in \mathcal {A}} \hat {Q} _ {i} ^ {k} (s _ {i}, a _ {i}): a _ {i} \notin \arg \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} (s _ {i}, a _ {i})\right) \\ \leq \sum_ {a _ {i} \notin \operatorname {a r g} \max  _ {a _ {i} \in \mathcal {A}} Q _ {i} ^ {k} (s _ {i}, a _ {i}), a _ {i + 1: i + k - 1} \in \mathcal {A} ^ {k - 1}} \frac {1}{2 T A ^ {k}} \\ \leq \frac {1}{2 T}, \\ \end{array}
$$
which is the only thing left that is needed to complete the proof.
B Additional theoretical results concerning the effective horizon
In this appendix, we present some additional theoretical results concerning the effective horizon. First, we explore two algorithms—one in the style of policy gradient and one similar to fitted Q-iteration—whose sample complexities can also be bounded by a quantity related to the effective horizon. Then we show conditions under which the effective horizon is small, as well as information-theoretic lower bounds for the sample complexity of RL in terms of the effective horizon.
B.1 PG-GORP and FQI-GORP
The two algorithms we introduce in this section, PG-GORP and FQI-GORP, can be viewed as a bridge between GORP, which we use to define the effective horizon, and PPO and DQN, the deep RL algorithms whose performance we predict using the effective horizon in Section 6. They help to explain why the effective horizon is not only useful for understanding the performance of GORP, but also other RL algorithms.
We will actually give sample complexity bounds on PG-GORP and FQI-GORP in terms of the bound in Theorem 5.4, rather than the effective horizon itself. Supposing that the MDP is  $k$ -QVI-solvable, define
$$
\bar {H} _ {k} = k + \max  _ {t \in [ T ], s \in \mathcal {S} _ {i} ^ {\mathrm {o p t}}, a \in \mathcal {A}} \log_ {A} \left(\frac {Q _ {t} ^ {k} (s , a) V _ {t} ^ {*} (s)}{\Delta_ {t} ^ {k} (s) ^ {2}}\right) + \log_ {A} 6 \log \left(2 T A ^ {k}\right).
$$
Our bounds will also depend on a quantity measuring how far the exploration policy  $\pi^{\mathrm{expl}}$  is from the uniformly random policy  $\pi^{\mathrm{rand}}$ :
$$
\left\| \frac {\pi^ {\mathrm {r a n d}}}{\pi^ {\mathrm {e x p l}}} \right\| _ {\infty} = \max _ {(t, s, a) \in [ T ] \times \mathcal {S} \times \mathcal {A}} \frac {1}{A \pi_ {t} ^ {\mathrm {e x p l}} (a \mid s)}.
$$
$\left\| \frac{\pi^{\mathrm{rand}}}{\pi^{\mathrm{exp}}}\right\|_{\infty} = 1$  in the case when  $\pi^{\mathrm{exp}} = \pi^{\mathrm{rand}}$ , and increases as the smallest probabilities  $\pi^{\mathrm{exp}}$  assigns to actions becomes smaller.
We now introduce the first algorithm, PG-GORP.
Algorithm 2 The PG-GORP algorithm.
1: procedure PG-GORP(m)  
2:  $\pi \gets \pi^{\mathrm{expl}}$ .  
3: for  $i = 1, \dots, T$  do  
4: Sample  $m$  episodes following  $\pi$ .  
5: for  $a \in \mathcal{A}$  do  
6:  $\hat{\nabla}_i(a | s_i) \gets \frac{1}{m} \sum_{j=1}^{m} \frac{\mathbf{1}_{a_i^j=a}}{\pi_i(a | s_i)} \sum_{t=i}^{T} \gamma^{t-i} R(s_t^j, a_t^j)$ .  
7: end for  
8:  $\pi_i(s_i) \gets \arg \max_{a \in \mathcal{A}} \hat{\nabla}_i(a | s_i)$ .  
9: end for  
10: return  $\pi$   
11: end procedure
PG-GORP resembles the REINFORCE algorithm [48], which gave rise to other policy gradient-based algorithms like PPO. At each iteration, Algorithm 2 first samples a number of episodes following its current policy (line 4). Then, it computes an approximate gradient over the policy parameters—in this case, just the action probabilities  $\pi_i(a\mid s_i)$ —via the so-called “policy gradient theorem,” which states
$$
\begin{array}{l} \nabla_ {\pi_ {i} (\cdot | s _ {i})} J (\pi) \approx \frac {1}{m} \sum_ {j = 1} ^ {m} \nabla_ {\pi_ {i} (\cdot | s _ {i})} \log \pi_ {i} (a _ {i} ^ {j} | s _ {i}) \sum_ {t = i} ^ {T} \gamma^ {t - i} R (s _ {t} ^ {j}, a _ {t} ^ {j}) \\ = \frac {1}{m} \sum_ {j = 1} ^ {m} \frac {\nabla_ {\pi_ {i} (\cdot | s _ {i})} \pi_ {i} \left(a _ {i} ^ {j} \mid s _ {i}\right)}{\pi_ {i} \left(a _ {i} ^ {j} \mid s _ {i}\right)} \sum_ {t = i} ^ {T} \gamma^ {t - i} R \left(s _ {t} ^ {j}, a _ {t} ^ {j}\right) \\ = \hat {\nabla} _ {i} (\cdot | s _ {i}). \\ \end{array}
$$
Then, in line 8, Algorithm 2 applies optimization to  $\pi$  based on its estimate of the gradient. In this case, it optimizes until  $\pi_{i}$  assigns all probability to only one action  $\pi_{i}(s_{i})$  at  $s_i$ .
Theorem B.1 (Sample complexity of PG-GORP). Suppose that an MDP is 1-QVI-solvable and that all rewards are nonnegative. Then the sample complexity of PG-GORP is at most
$$
2 T ^ {2} A ^ {\bar {H} _ {1}} \left\| \frac {\pi^ {r a n d}}{\pi^ {e x p l}} \right\| _ {\infty}.
$$
$$
m = A ^ {\bar {H} _ {1}} \left\| \frac {\pi^ {\text {r a n d}}}{\pi^ {\text {e x p l}}} \right\| _ {\infty} = 1 2 A \log (2 T A) \max  _ {t \in [ T ], s \in S _ {i} ^ {\text {o p t}}, a \in \mathcal {A}} \left(\frac {Q _ {t} ^ {k} (s , a) V _ {t} ^ {*} (s)}{\Delta_ {t} ^ {1} (s) ^ {2}}\right) \left\| \frac {\pi^ {\text {r a n d}}}{\pi^ {\text {e x p l}}} \right\| _ {\infty}.
$$