SlowGuess/NeurIPS_2023_OCR huggingface dataset:
function before acting randomly. Second, it can use an "exploration policy"  $\pi^{\mathrm{expl}}$  different from the random policy  $\pi^{\mathrm{rand}}$ . These two generalizations are captured in the following definition. In the definition, we use the notation that a step of Q-value iteration transforms a Q-function  $Q$  to  $Q' = \mathrm{QVI}(Q)$ , where
$$
Q _ {t} ^ {\prime} (s, a) = R _ {t} (s, a) + \max  _ {a ^ {\prime} \in \mathcal {A}} Q _ {t + 1} \left(f (s, a), a ^ {\prime}\right).
$$
Definition 5.1 ( $k$ -QVI-solvable). Given an exploration policy  $\pi^{expl}$  ( $\pi^{expl} = \pi^{rand}$  unless otherwise noted), let  $Q^1 = Q^{\pi^{expl}}$  and  $Q^{i+1} = QVI(Q^i)$  for  $i = 1, \ldots, T-1$ . We say an MDP is  $k$ -QVI-solvable for some  $k \in [T]$  if every policy in  $\Pi(Q^k)$  is optimal.
We will see that running GORP with  $k > 1$  will allow it to find an optimal policy in MDPs that are  $k$ -QVI-solvable. Although the sample complexity of GORP scales with  $A^k$ , we find that nearly all of the environments in BRIDGE are  $k$ -QVI-solvable for very small values of  $k$  (Figure 6).
We now use GORP to define the effective horizon of an MDP. Note that the total number of timesteps sampled by GORP with parameters  $k$  and  $m$  is  $T^2 A^k m = T^2 A^{k + \log_A m}$ . Thus, analogously to how the horizon  $T$  appears in the exponent of the worst-case sample complexity bound  $O(TA^T)$ , we define the effective horizon as the exponent of  $A$  in the sample complexity of GORP:
Definition 5.2 (Effective horizon). Given  $k \in [T]$ , let  $H_{k} = k + \log_{A} m_{k}$ , where  $m_{k}$  is the minimum value of  $m$  needed for Algorithm 1 with parameter  $k$  to return the optimal policy with probability at least  $1/2$ , or  $\infty$  if no value of  $m$  suffices. The effective horizon is  $H = \min_{k} H_{k}$ .
By definition, the sample complexity of GORP can be given using the effective horizon:
Lemma 5.3. The sample complexity of GORP with optimal choices of  $k$  and  $m$  is  $T^2 A^H$ .
As we noted in the introduction, when  $H \ll T$ , as we find is often true in practice, this is far better than the worst-case bound given in Theorem 2.1 which scales with  $A^T$ .
Definition 5.2 does not give a method for actually calculating the effective horizon. It turns out we can bound the effective horizon using a generalized gap notion like those found throughout the RL theory literature. We denote by  $\Delta_t^k$  the gap of the Q-function  $Q^k$  from Definition 5.1, where
$$
\Delta_{t}^{k}(s) = \max_{a\in \mathcal{A}}Q_{t}^{k}(s,a) - \max_{a^{\prime}\not\in\arg \max_{a}Q_{t}^{k}(s,a)}Q_{t}^{k}(s,a^{\prime}).
$$
The following theorem gives bounds on the effective horizon in terms of this gap.
Theorem 5.4. Suppose that an MDP is  $k$ -QVI-solvable and that all rewards are nonnegative, i.e.  $R(s,a) \geq 0$  for all  $s,a$ . Let  $\Delta^k$  denote the gap of the  $Q$ -function  $Q^k$  as defined in Definition 5.1.
(a) Sparse rewards: when only one sequence of actions gives a reward of 1 and all others give 0, the effective horizon  $H = \bar{O}(T)$ .
(b) Dense rewards: when every optimal action gives a reward of 1 and suboptimal actions give no reward, the effective horizon  $H = \tilde{O}(1)$ .
(c) Delayed rewards: when the rewards in (b) are all delayed to the end of the episode, the effective horizon remains  $\hat{O}(1)$ .
(d) For the first 50 timesteps of the Atari game Freeway, we can bound  $H \leq 10.2$ , which is much lower than the horizon  $T = 50$ .
Figure 2: Examples of calculating the effective horizon  $H$  using Theorem 5.4; see Section 5.1 for the details.
$$
H _ {k} \leq k + \max  _ {t \in [ T ], s \in \mathcal {S} _ {i} ^ {o p t}, a \in \mathcal {A}} \log_ {A} \left(\frac {Q _ {t} ^ {k} (s , a) V _ {t} ^ {*} (s)}{\Delta_ {t} ^ {k} (s) ^ {2}}\right) + \log_ {A} 6 \log \left(2 T A ^ {k}\right),
$$
where  $S_{i}^{opt}$  is the set of states visited by some optimal policy at timestep  $i$  and  $V_{t}^{*}(s) = \max_{\pi} V_{t}^{\pi}(s)$  is the optimal value function.
A full proof of Theorem 5.4 is given in Appendix A. Intuitively, the smaller the gap  $\Delta_t^k (s)$ , the more precisely we must estimate the Q-values in GORP in order to pick an optimal action.
The GORP algorithm is very amenable to theoretical analysis because it reduces the problem of finding an optimal policy to the problem of estimating several  $k$ -step Q-values, each of which is a simple mean of i.i.d. random variables. There are endless tail bounds that can be applied to analysis of GORP; we use some of these to obtain even tighter bounds on the effective horizon in Appendix C.
Why should the effective horizon, which is defined in terms of our GORP algorithm, also explain the performance of deep RL algorithms like PPO and DQN which are very different from GORP? In Appendix B, we present two algorithms, PG-GORP and FQI-GORP, which are more similar to PPO and DQN but whose sample complexities can still be bounded with the effective horizon. We also give additional bounds on the effective horizon and lower bounds on sample complexity.
5.1 Examples of the effective horizon
To gain some intuition for the bound in Theorem 5.4, consider the examples in Figure 2. MDP (a) has extremely sparse rewards, with a reward of 1 only given for a single optimal action sequence. However, note that this MDP is still 1-QVI-solvable by Definition 5.1. The maximum of the bound in Theorem 5.4 is at  $t = 1$  with the optimal action, where  $Q_1^1(s, a) = 1 / A^{T-1}$ ,  $V_1^*(s) = 1$ , and  $\Delta_1^1(s) = 1 / A^{T-1}$ . Ignoring logarithmic factors and constants gives  $H \lesssim 1 + \log_A A^{T-1} = T$ . That is, in the case of MDP (a), the effective horizon is no better than the horizon.
Next, consider MDP (b), which has dense rewards of 1 for every optimal action. Again, this MDP is 1-QVI-solvable. The maximum in the bound is obtained at  $t = 1$  and the optimal action with  $Q_1^1(s, a) \leq 2$ ,  $V^*(s) = T$ , and the gap  $\Delta_1^1(s, a) \geq 1$ . Again ignoring logarithmic factors gives in this case  $H \lesssim 1 + \log_A T = \tilde{O}(1)$ . In this case, the effective horizon is much shorter than the horizon, and barely depends on it! This again reflects our intuition that in this case, finding the optimal policy via RL should be much easier.
MDP (c) is similar to MDP (b) except that all rewards are delayed to the end of the episode. In this case, the  $Q$  function is the same as in MDP (b) so the effective horizon remains  $\tilde{O}(1)$ . This may seem counterintuitive since one needs to consider rewards  $T$  timesteps ahead to act optimally. However, the way GORP uses random rollouts to evaluate leaf nodes means that it can implicitly consider rewards quite far in the future even without needing to exhaustively plan that many timesteps ahead.
Finally, consider MDP (d), the first 50 timesteps of the Atari game Freeway, which is included in the BRIDGE dataset. This MDP is also 1-QVI-solvable and the maximum in the bound is obtained in the state shown in Figure 2d at timestep  $t = 10$ . Plugging in the Q values shown in the figure gives  $H \leq 10.2$ , which is far lower than the horizon  $T = 50$ . The low effective horizon reflects how this MDP is much easier than the worst case in practice. Both PPO and DQN are able to solve it with a sample complexity of less than 1.5 million timesteps, while the worst case bound would suggest a
sample complexity greater than  $50 \times 3^{50} / 2 \approx 10^{25}$  timesteps!
Comparison to other bounds Intuitively, why might the effective horizon give better sample complexity bounds than previous works presented in Section 3? The MDP in Figure 2b presents a problem for the covering length and UCB-based bounds, both of which are  $\Omega(A^T)$ . The exponential sample complexity arises because these bounds depend on visiting every state in the MDP during training. In contrast, GORP doesn't need to visit every state to find an optimal policy. The effective horizon of  $\tilde{O}(1)$  for MDP (b) reflects this, showing that our effective horizon-based bounds can actually be much smaller than the state space size, which is on the order of  $A^T$  for MDP (b).
The effective planning window (EPW) does manage to capture the same intuition as the effective horizon in the MDP in Figure 2b: in this case,  $W = 1$ . However, the analysis based on the EPW is unsatisfactory because it entirely ignores rewards beyond the planning window. Thus, in MDP (c) the EPW  $W = T$ , making EPW-based bounds no better than the worst case. In contrast, the effective horizon-based bound remains the same between MDPs (b) and (c), showing that it can account for the ability of RL algorithms to use rewards beyond the window where exhaustive planning is possible.
We now show that sample complexity bounds based on the effective horizon predict the empirical performance of deep RL algorithms far better than other bounds in the literature. For each MDP in the BRIDGE dataset, we run deep RL algorithms to determine their empirical sample complexity. We also use the tabular representations of the MDPs to calculate the effective horizon and other sample complexity bounds for comparison.
Deep RL algorithms We run both PPO and DQN for five million timesteps for each MDP in BRIDGE, and record the empirical sample complexity (see Appendix F for hyperparameters and experiment details). PPO converges to the optimal policy in 95 of the 155 MDPs, and DQN does in 117 of 155. At least one of the two finds the optimal policy in 119 MDPs.
Sample complexity bounds We also compute sample complexity bounds for each MDP in BRIDGE. These include the worst-case bound of  $T A^T$  from Theorem 2.1, the effective-horizon-based bound of  $T^2 A^H$  from Lemma 5.3, as well as three other bounds from the literature, introduced in Section 3 and proved in Appendix D: the UCB-based bound  $SAT$ , the covering-length-based bound  $TL$ , and the effective planning window (EPW)-based bound of  $T^2 A^W$ .
Evaluation metrics To determine which sample complexity bounds best reflect the empirical performance of PPO and DQN, we compute a few summary metrics for each bound. First, we measure the Spearman (rank) correlation between the sample complexity bounds and the empirical sample complexity over environments where the algorithm converged to the optimal policy. The correlation (higher is better) is a useful for measuring how well the bounds can rank the relative difficulty of RL in different MDPs.
Second, we compute the median ratio between the sample complexity bound and the empirical sample complexity for environments where the algorithm converged. The ratio between the bound  $N_{\mathrm{bound}}$  and empirical value  $N_{\mathrm{emp}}$  is calculated as  $\max \{N_{\mathrm{bound}} / N_{\mathrm{emp}}, N_{\mathrm{emp}} / N_{\mathrm{bound}}\}$ . For instance, a median ratio of 10 indicates that half the sample complexity bounds were within a factor of 10 of the empirical sample complexity. Lower values indicate a better bound; this metric is useful for determining whether the sample complexity bounds are vacuous or tight.
Finally, we consider the binary classification task of predicting whether the algorithm will converge at all within five million steps using the sample complexity bounds. That is, we consider simply thresholding each sample complexity bound and predicting that only environments with bounds below the threshold will converge. We compute the area under the ROC curve (AUROC) for this prediction task as well as the accuracy with the optimal threshold. Higher AUROC and accuracy both indicate a better bound.
Results The results of our experiments are shown in Table 2. The effective horizon-based bounds have higher correlation with the empirical sample complexity than the other bounds for both PPO and DQN. While the EPW-based bounds are also reasonably correlated, they are significantly off in absolute terms: the typical bound based on the EPW is 3-4 orders of magnitude off, while the effective horizon yields bounds that are typically within 1-2 orders of magnitude. The UCB-based
Bound	PPO	DQN
Correl.	Median ratio	AUROC	Acc.	Correl.	Median ratio	AUROC	Acc.
Worst-case (T[AT/2])	0.24	7.2 × 1010	0.57	0.63	0.15	5.5 × 1010	0.67	0.76
Covering length (TL)	0.35	6.3 × 106	0.78	0.72	0.27	3.9 × 106	0.86	0.85
EPW (T2AW)	0.69	1.1 × 105	0.78	0.75	0.58	8.0 × 104	0.88	0.85
UCB (SAT)	0.26	20	0.68	0.67	0.31	31	0.67	0.77
Effective horizon (T2AH)	0.81	31	0.92	0.86	0.74	67	0.92	0.86
Other deep RL algorithm	0.77	2.3	0.84	0.85	0.77	2.3	0.86	0.99
GORP empirical	0.79	7.3	0.77	0.82	0.65	11	0.80	0.94
Table 2: Effective horizon-based sample complexity bounds are the most predictive of the real performance of PPO and DQN according to the four metrics we describe in Section 6. The effective horizon bounds are about as good at predicting the sample complexity of PPO and DQN as one algorithm's sample complexity is at predicting the other's.
Bound	PPO	DQN
Correl.	Ratio	Correl.	Ratio
EPW	0.20	2.1	0.70	12
Effective horizon	0.48	2.4	0.35	12
Other bounds	0.00	1.3	0.00	1.9
(a) Reward shaping.
Bound	PPO
Correl.	Ratio
Covering length	-0.36	2.5 × 104
Effective horizon	0.57	2.7
Other bounds	0.00	2.2
(b) Initializing with a policy trained on human data or transferred from similar environments.
Table 3: The effective horizon explains the effects of reward shaping and initializing with a pretrained policy by accurately predicting their effects on the empirical sample complexity of PPO and DQN. Correlation and median ratio are measured between the predicted change in sample complexity and the empirical change. See Section 6 for further discussion.
bounds are somewhat closer to the empirical sample complexity, but are not well correlated; this makes sense since the UCB bounds depend on strategic exploration, while PPO and DQN use random exploration. Finally, the effective horizon bounds are able to more accurately predict whether PPO or DQN will find an optimal policy, as evidenced by the AUROC and accuracy metrics.
As an additional baseline, we also calculate the four evaluation metrics when using the empirical sample complexity of PPO to predict the empirical sample complexity of DQN, or vice-versa, and when using the empirical sample complexity of GORP to predict PPO or DQN's performance (bottom two rows of Table 2). While these are not provable bounds, they provide another point of comparison for each metric. The effective horizon-based bounds correlate about as well with PPO and DQN's sample complexities as they do with each other's. The empirical performance of GORP is typically even closer to that of PPO and DQN than the effective horizon-based bounds.
Reward shaping, human data, and transfer learning In order for RL theory to be useful practically, it should help practitioners make decisions about which techniques to apply in order to improve their algorithms' performance. We show how the effective horizon can be used to explain the effect of three such techniques: reward shaping, using human data, and transfer learning.
Potential-based reward shaping [45] is a classic technique which can speed up the convergence of RL algorithms. It is generally used to transform a sparse-reward MDP like the one in Figure 2a to a dense-reward MDP like in Figure 2b without changing the optimal policy. If the effective horizon is a good measure of the difficulty of RL in an environment, then it should be able to predict whether (and by how much) the sample complexity of practical algorithms changes when reward shaping is applied. We develop 77 reward-shaped versions of the original 33 Minigrid environments and run PPO and DQN. Results in Table 3a show the effective horizon accurately captures the change in sample complexity from using reward shaping. We use similar metrics to those in Table 2: the correlation between the predicted and empirical ratio of the shaped sample complexity to the unshaped sample complexity, and the median ratio between the predicted change and the actual change.
Out of the five bounds we consider, three—worst case, covering length, and UCB—don't even depend on the reward function. The EPW does depend on the reward function and captures some of the effect of reward shaping for DQN, but does worse at predicting the effect on PPO. In comparison, the effective horizon does well for both algorithms, showing that it can accurately capture how reward shaping affects RL performance.
Figure 3: Learning curves for PPO, DQN, and GORP on full-horizon Atari games. We use 5 random seeds for all algorithms. The solid line shows the median return throughout training while the shaded region shows the range of returns over random seeds.
Another tool used to speed up RL is initializing with a pre-trained policy, which is used practically to make RL work on otherwise intractable tasks. Can the effective horizon also predict whether initializing RL with a pre-trained policy will help? We initialize PPO with pre-trained policies for 82 of the MDPs in BRIDGE, then calculate new sample complexity bounds based on using the pre-trained policies as an exploration policy  $\pi^{\mathrm{expl}}$ . Table 3b shows that the effective horizon accurately predicts the change in sample complexity due to using a pre-trained policy. Again, three bounds—worst case, EPW, and UCB—do not depend on the exploration policy at all, while the covering length gives wildly inaccurate predictions for its effect. In contrast, the effective horizon is accurate at predicting the changes in sample complexities when using pre-trained policies.
Long-horizon environments We also perform experiments on full-length Atari games to evaluate the predictive power of the effective horizon in longer-horizon environments. It is intractable to construct tabular representations of these environments and thus we cannot compute instance-dependent sample complexity bounds. However, it is still possible to compare the empirical performance of PPO, DQN, and GORP. If the performance of GORP is close to that of PPO and DQN, then this suggests that the effective horizon, which is defined in terms of GORP, can explain RL performance in these environments as well. Figure 3 compares the learning curves of PPO, DQN, and GORP in deterministic versions of the seven Atari games from Mnih et al. [46] (see Appendix F.1 for details). GORP performs better than both PPO and DQN in two games and better than at least one deep RL algorithm in an additional three games. This provides evidence that the effective horizon is also predictive of RL performance in long-horizon environments.
Overall, our results suggest the effective horizon is a key measure of the difficulty of solving an MDP via reinforcement learning. The intuition behind the effective horizon presented in Section 5 and the empirical evidence in Section 6 both support its importance for better understanding RL.
Limitations While we have presented a thorough theoretical and empirical justification of the effective horizon, there are still some limitations to our analysis. First, we focus on deterministic MDPs with discrete action spaces, leaving the extension to stochastic environments and those with continuous action spaces an open question. Furthermore, the effective horizon is not easily calculable without full access to the MDP's tabular representation. Despite this, it serves as a useful perspective for understanding RL's effectiveness and potential improvement areas. An additional limitation is that the effective horizon cannot capture the performance effects of generalization—the ability to use actions that work well at some states for other similar states. For an example where the effective horizon fails to predict generalization, see Appendix G.1. However, the effective horizon is still quite predictive of deep RL performance even without modeling generalization.
Implications and future work We hope that this paper helps to bring the theoretical and empirical RL communities closer together in pursuit of shared understanding. Theorists can extend our analysis of the effective horizon to new algorithms or explore related properties, using our BRIDGE dataset to ground their investigations by testing assumptions in real environments. Empirical RL researchers can use the effective horizon as a foundation for new algorithms. For instance, Brandfonbrener et al. [47] present an offline RL algorithm similar to GORP with  $k = 1$ ; our theoretical understanding of GORP might provide insights for improving it or developing related algorithms.
We would like to thank Jacob Steinhardt, Kush Bhatia, Lawrence Chan, Paria Rashidinejad, Ruiqi Zhong, and Yaodong Yu for feedback on drafts as well as Jiantao Jiao and Banghua Zhu for helpful discussions.
This work was partially supported by the Open Philanthropy Project, the Office of Naval Research's Young Investigator Program (YIP), and Weill Neurohub. Cassidy Laidlaw is supported by an Open Philanthropy AI Fellowship and a National Defense Science and Engineering Graduate (NDSEG) Fellowship. Stuart Russell is supported by an AI 2050 Senior Fellowship from Schmidt Futures.
[50] Martin Riedmiller. Neural Fitted Q Iteration – First Experiences with a Data Efficient Neural Reinforcement Learning Method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo, editors, Machine Learning: ECML 2005, Lecture Notes in Computer Science, pages 317–328, Berlin, Heidelberg, 2005. Springer. ISBN 978-3-540-31692-3. doi: 10.1007/11564096_32.
[51] Irina Shevtsova. On the absolute constants in the Berry-Esseen type inequalities for identically distributed summands, November 2011. URL http://arxiv.org/abs/1111.6554.arXiv:1111.6554 [math].
[52] Eyal Even-Dar and Yishay Mansour. Learning Rates for Q-learning. Journal of Machine Learning Research, 5(Dec):1-25, 2003. ISSN ISSN 1533-7928. URL https://www.jmlr.org/papers/v5/evendar03a.html.
[53] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The Dependence of Effective Planning Horizon on Model Accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '15, pages 1181-1189, Richland, SC, May 2015. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-3413-6.
[54] Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Miikkulainen. Frame skip is a powerful parameter for learning to play Atari. In *Workshops at the twenty-ninth AAAI conference on artificial intelligence*, 2015.
[55] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for Datasets, December 2021. URL http://arxiv.org/abs/1803.09010. arXiv:1803.09010 [cs].
[56] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, June 2016. URL http://arxiv.org/abs/1606.01540. arXiv:1606.01540 [cs].
[57] Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl S. Muller, Jake A. Whritner, Luxin Zhang, Mary M. Hayhoe, and Dana H. Ballard. Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset, September 2019. URL http://arxiv.org/abs/1903.06754.arXiv:1903.06754 [cs, stat].
[58] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. The Journal of Machine Learning Research, 22(1):12348-12355, 2021. URL https://dl.acm.org/doi/abs/10.5555/3546258.3546526. Publisher: JMLRORG.
[59] Antonin Raffin. RL Baselines3 Zoo, 2020. URL https://github.com/DLR-RM/r1-baselines3-zoo. Publication Title: GitHub repository.
A Proofs of main results
A.1 Proof of Theorem 2.1
Proof. Consider the following RL algorithm:
procedure EXHAUSTIVESEARCH
$$
\mathcal {T} \leftarrow \operatorname {S h u f f l e} \left(\mathcal {A} ^ {T}\right)
$$
$\mathcal{I}$  is an array of size  $[A^T /2]$
for  $i = 1,\dots ,\lceil A^T /2\rceil$  do
run one episode, taking the actions in  $\mathcal{T}[i]$
$$
\mathcal {J} [ i ] \leftarrow \sum_ {t = 1} ^ {T} \gamma^ {t} R \left(s _ {t}, a _ {t}\right)
$$
$$
i ^ {*} \leftarrow \arg \max  _ {i} \mathcal {J} [ i ]
$$
return the policy which takes the actions in  $\mathcal{T}[i^*]$