SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bridging Reinforcement Learning Theory and Practice with the Effective Horizon
Cassidy Laidlaw Stuart Russell Anca Dragan
{cassidy_laidlaw,russell,anca}@cs.berkeley.edu
Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon.
Deep reinforcement learning (RL) has produced impressive results in robotics [1], strategic games [2], and control [3]. However, the same deep RL algorithms that achieve superhuman performance in some environments completely fail to learn in others. Sometimes, using techniques like reward shaping or pre-training help RL, and in other cases they don't. Our goal is to provide a theoretical understanding of why this is—a theoretical analysis that is predictive of practical RL performance.
Unfortunately, there is a large gap between the current theory and practice of RL. Despite RL theorists often focusing on algorithms using strategic exploration (e.g., UCB exploration bonuses; Azar et al. [4], Jin et al. [5]), the most commonly-used deep RL algorithms, which explore randomly, resist such analysis. In fact, theory suggests that RL with random exploration is exponentially hard in the worst case [6], but this is not predictive of practical performance. Some theoretical research has explored instance-dependent bounds, identifying properties of MDPs when random exploration should perform better than this worst case [7, 8]. However, it is not clear whether these properties correlate with when RL algorithms work vs. fail—and our results will reveal that they tend not to.
If the current theory literature cannot explain the empirical performance of deep RL, what can? Ideally, a theory of RL should provably show why deep RL succeeds while using random exploration. It should also be able to predict which environments are harder or easier to solve empirically. Finally, it should be able to explain when and why tools like reward shaping or initializing with a pre-trained policy help make RL perform better.
We present a new theoretical complexity measure for MDPs called the effective horizon that satisfies all of the above criteria. Intuitively, the effective horizon measures approximately how far ahead an algorithm must exhaustively plan in an environment before evaluating leaf nodes with random rollouts.
Figure 1: We introduce the effective horizon, a property of MDPs that controls how difficult RL is. Our analysis is motivated by Greedy Over Random Policy (GORP), a simple Monte Carlo planning algorithm (left) that exhaustively explores action sequences of length  $k$  and then uses  $m$  random rollouts to evaluate each leaf node. The effective horizon combines both  $k$  and  $m$  into a single measure. We prove sample complexity bounds based on the effective horizon that correlate closely with the real performance of PPO, a deep RL algorithm, on our BRIDGE dataset of 155 deterministic MDPs (right).
In order to assess previous bounds and eventually arrive at such a property, we start by creating a new dataset, BRIDGE, of deterministic MDPs from common deep RL benchmarks. A major difficulty with evaluating instance-dependent bounds is that they can't be calculated without tabular representations, so prior work has typically relied on small toy environments for justification. To get a more realistic picture, we choose 155 MDPs across different benchmarks and compute their tabular representations—some with over 100 million states which must be exhaustively explored and stored. This is a massive engineering challenge, but it enables connecting theoretical and empirical results at an unprecedented scale. We focus on deterministic MDPs in BRIDGE and in this paper because they are simpler to analyze but still have many of the interesting properties of stochastic MDPs, like reward sparsity and credit assignment challenges. Many deep RL benchmarks are (nearly) deterministic, so we believe our analysis is highly relevant to practical RL.
Our journey to the effective horizon began with identifying a surprising property that holds in many of the environments in BRIDGE: one can learn to act optimally by acting randomly. More specifically, actions with the highest Q-values under the uniformly random policy also have the highest Q-values under the optimal policy. The random policy is about as far as one can get from the optimal policy, so this property may seem unlikely to hold. However, about two-thirds of the environments in BRIDGE satisfy the property. This proportion rises to four-fifths among environments that PPO [9], a popular deep RL algorithm, can solve efficiently (Table 1). Conversely, when this property does not hold, PPO is more likely to fail than succeed—and when it does succeed, so does simply applying a few steps of lookahead on the Q-function of the random policy (Figure 6). We found it remarkable that, at least in the environments in BRIDGE, modern algorithms seem to boil down to not much more than acting greedily on the random policy Q-values.
The property that it is optimal to act greedily with respect to the random policy's Q-function has important implications for RL theory and practice. Practically, it suggests that very simple algorithms designed to estimate the random policy's Q-function could efficiently find an optimal policy. We introduce such an algorithm, Greedy Over Random Policy (GORP), which also works in the case where one may need to apply a few steps of value iteration to the random policy's Q-function before acting greedily. Empirically, GORP finds an optimal policy in fewer timesteps than DQN (another deep RL algorithm) in more than half the environments in BRIDGE. Theoretically, it is simple to analyze GORP, which consists almost entirely of estimating the random policy's Q-function via a sample average over i.i.d. random rollouts. Since GORP works well empirically and can be easily understood theoretically, we thoroughly analyze it in the hopes of finding sample complexity bounds that can explain the performance of deep RL.
Our analysis of Greedy Over Random Policy leads to a single metric, the effective horizon, that measures the complexity of model-free RL in an MDP. As shown in Figure 1, GORP is an adaptation of a Monte Carlo planning algorithm to the reinforcement learning setup (where the transitions are unknown): it mimics exhaustively planning ahead  $k$  steps and then sampling  $m$  random rollouts from
each leaf node. The effective horizon  $H$  combines the depth  $k$  and number of rollouts  $m$ . We call it the effective horizon because worst-case sample complexity bounds for random exploration are exponential in the horizon  $T$ , while we prove sample complexity bounds exponential only in the effective horizon  $H$ . For most BRIDGE environments,  $H \ll T$ , explaining the efficiency of RL in these MDPs.
In the environments in BRIDGE, we find that the effective horizon-based sample complexity bounds satisfy all our desiderata above for a theory of RL. They are more predictive of the empirical sample complexities of PPO and DQN than several other bounds from the literature across four metrics, including measures of correlation, tightness, and accuracy (Table 2). Furthermore, the effective horizon can predict the effects of both reward shaping (Table 3a) and initializing using a pre-trained policy learned from human data or transferred from a similar environment (Table 3b). In contrast, none of the existing bounds we compare to depend on both the reward function and initial policy; thus, they are unable to explain why reward shaping, human data, and transfer learning can help RL. Although our results focus on deterministic MDPs, we plan to extend our work to stochastic environments in the future and already have some promising results in that direction.
We begin by presenting the reinforcement learning (RL) setting we consider. An RL algorithm acts in a deterministic, tabular, episodic Markov decision process (MDP) with finite horizon. The MDP comprises a set of states  $S$ , a set of actions  $A$ , a horizon  $T \in \mathbb{N}$  and optional discount factor  $\gamma \in [0,1]$ , a start state  $s_1$ , transition function  $f: S \times A \to S$ , and a reward function  $R: S \times A \to \mathbb{R}$ . Throughout the paper we use  $\gamma = 1$  but all our theory applies equally when  $\gamma < 1$ .
An RL agent interacts with the MDP for a number of episodes, starting at a fixed start state  $s_1$ . At each step  $t \in [T]$  of an episode (using the notation  $[n] = \{1, \dots, n\}$ ), the agent observes the state  $s_t$ , picks an action  $a_t$ , receives reward  $R(s_t, a_t)$ , and transitions to the next state  $s_{t+1} = f(s_t, a_t)$ . A policy  $\pi$  is a set of functions  $\pi_1, \ldots, \pi_t : S \to \Delta(\mathcal{A})$ , which defines for each state and timestep a distribution  $\pi_t(a \mid s)$  over actions. If a policy is deterministic at some state, then with slight abuse of notation we denote  $a = \pi_t(s)$  to be the action taken by  $\pi_t$  in state  $s$ .
We denote a policy's Q-function  $Q_{t}^{\pi}:S\times \mathcal{A}\to \mathbb{R}$  and value function  $V_{t}^{\pi}:S\rightarrow \mathbb{R}$  for each  $t\in [T]$ . In this paper, we also use a Q-function which is generalized to sequences of actions. We use the shorthand  $a_{t:t + k}$  to denote the sequence  $a_{t},\ldots ,a_{t + k}$ , and define the action-sequence Q-function as
$$
Q _ {t} ^ {\pi} \left(s _ {t}, a _ {t: t + k}\right) = \mathbb {E} _ {\pi} \left[ \sum_ {t ^ {\prime} = t} ^ {T} \gamma^ {t ^ {\prime} - t} R \left(s _ {t ^ {\prime}}, a _ {t ^ {\prime}}\right) \mid s _ {t}, a _ {t: t + k} \right].
$$
The objective of an RL algorithm is to find an optimal policy  $\pi^{*}$ , which maximizes  $J(\pi) = V_{1}^{\pi}(s_{1})$ , the expected discounted sum of rewards over an episode, also known as the return of the policy  $\pi$ .
Generally, an RL algorithm can be run for any number of timesteps  $n$  (i.e., counting one episode as  $T$  timesteps), returning a policy  $\pi^n$ . We define the sample complexity  $N$  of an RL algorithm as the minimum number of timesteps needed such that the algorithm has at least a 50-50 chance of returning an optimal policy:
$$
N = \min  \left\{n \in \mathbb {N} \mid \mathbb {P} \left(J \left(\pi^ {n}\right) = J ^ {*}\right) \geq 1 / 2 \right\}.
$$
Here, the probability is with respect to any randomness in the algorithm itself. One can estimate the sample complexity  $N$  empirically by running an algorithm several times, calculating the number of samples  $n$  needed to reach the optimal policy during each run, and then taking the median.
The following simple theorem gives upper and lower bounds for the worst-case sample complexity in a deterministic MDP, depending on  $A$  and  $T$ .
Theorem 2.1. There is an RL algorithm which can solve any deterministic MDP with sample complexity  $N \leq T\lceil A^T /2\rceil$ . Conversely, for any RL algorithm and any values of  $T$  and  $A$ , there must be some deterministic MDP for which its sample complexity  $N \geq T(\lceil A^T /2\rceil -1)$ .
All proofs are deferred to Appendix A. In this case, the idea of the proof is quite simple, and will later be useful to motivate our idea of the effective horizon: in an MDP where exactly one sequence of actions leads to a reward, an RL algorithm may have to try almost every sequence of actions to find the optimal policy; there are  $A^T$  such sequences. As we develop sample complexity bounds based on
the effective horizon in Section 5, we can compare them to the worst-case bounds in Theorem 2.1.
Why deterministic MDPs? We focus on deterministic (as opposed to stochastic) MDPs in this study for several reasons. First, analyzing deterministic MDPs avoids the need to consider generalization within RL algorithms. In common stochastic MDPs, one often needs neural-network based policies, whereas in a deterministic MDP one can simply learn a sequence of actions. Since neural network generalization is not well understood even in supervised learning, analyzing generalization in RL is an especially difficult task. Second, deterministic MDPs still display many of the interesting properties of stochastic MDPs. For instance, deterministic MDPs have worst case exponential sample complexity when using naive exploration; environments with dense rewards are easier to solve empirically than those with sparse rewards; credit assignment can be challenging; and there is a wide range of how tractable environments are for deep RL, even for environments with similar horizons, state spaces, and action spaces.
Finally, many common RL benchmark environments are deterministic or nearly-deterministic. For instance, the ALE Atari games used to evaluate DQN [3], Rainbow [10], and MuZero [11] are all deterministic after the first state, which is selected randomly from one of only 30 start states. The widely used DeepMind Control Suite [12] is based on the MuJoCo simulator [13], which is also deterministic given the initial state (some of the environments do use a randomized start state). MiniGrid environments [14], which are commonly used for evaluating exploration [15], environment design [16], and language understanding [17], are also deterministic after the initial state. Thus, our investigation of deterministic environments is highly relevant to common deep RL practice.
Before delving into our contributions, we briefly summarize existing work in theoretical RL and prior sample complexity bounds. Our novel bounds are contrasted with existing ones in Sections 5.1 and 6; for a detailed comparison with full definitions and proofs, please see Appendix D.
Recent RL theoretical results largely focus on strategic exploration using techniques like UCB exploration bonuses [18, 4, 19, 5, 20, 21, 22]. Such bounds suggest RL is tractable for smaller or low-dimensional state spaces. In deterministic MDPs, the UCB-based R-MAX algorithm [23, 18] has sample complexity bounded by  $SAT$ .
Some prior work has focused on sample complexity bounds for random exploration. Liu and Brunskill [7] give bounds based on the covering length  $L$  of an MDP, which is the number of episodes needed to visit all state-action pairs at least once with probability at least  $1/2$  while taking actions at random. This yields a sample complexity bound of  $TL$  for deterministic MDPs. Other work suggests that it may not be necessary to consider rewards all the way to the end of the episode to select an optimal action [24, 25, 8]. One can define a "effective planning window" of  $W$  timesteps ahead that must be considered, resulting in a sample complexity bound of  $T^2 A^W$  for deterministic MDPs. Finally, Dann et al. [6] define a "myopic exploration gap" that controls the sample complexity of using  $\epsilon$ -greedy exploration, a form of naive exploration. However, in Appendix D.4, we demonstrate why their bounds are impractical and often vacuous.
There have been a few prior attempts to bridge the RL theory-practice gap. bsuite [26], MDP playground [27], and SEGAR [28] are collections of environments that are designed to empirically evaluate deep RL algorithms across various axes of environment difficulty. However, they do not provide theoretical explanations for why environments with varying properties are actually easier or harder. Furthermore, their environments are artificially constructed to have understandable properties. In contrast, we aim to find the mathematical reasons that deep RL succeeds or fails in "in-the-wild" environments like Atari and Progen. Conserva and Rauber [29] calculate two regret bounds and compare the bounds to the empirical performance of RL algorithms. However, they consider tabular RL algorithms in simple artificial environments with less than a thousand states, while we experiment with deep RL algorithms on real benchmark environments with tens of millions of states.
Our GORP algorithm and the effective horizon are inspired by rollout and Monte Carlo planning algorithms, which have a long history [30, 31, 32, 33, 24, 34, 35, 36]. These algorithms were used effectively in Backgammon [32], Go [37], and real-time strategy games [38] before the start of deep RL. GORP and related Monte Carlo rollout algorithms are sometimes referred to as "one-step" or "multi-step" lookahead. Bertsekas [39, 40] suggests that one-step lookahead, possibly after a few
steps of value iteration, often leads to fast convergence to optimal policies because it is equivalent to a step of Newton's method for finding a fixed-point of the Bellman equation [41, 42]. Our analysis suggests the success of deep RL is due to similar properties. However, we go beyond previous work by introducing GORP, which approximates a step of policy iteration in model-free RL—a setting where on-line planning approaches are not applicable. Furthermore, unlike previous work in Monte-Carlo planning, we combine our theoretical contributions with extensive empirical analysis to verify that our assumptions hold in common environments.
4 The BRIDGE Dataset
In order to assess how well existing bounds predict practical performance, and gain insight about novel properties of MDPs that could be predictive, we constructed BRIDGE (Bridging the RL Interdisciplinary Divide with Grounded Environments), a dataset of 155 popular deep RL benchmark environments with full tabular representations. One might assume that the ability to calculate the instance-dependent bounds we just presented in Section 3 already exists; however, it turns out that for many real environments even the number of states  $S$  is unknown! This is because a significant engineering effort is required to analyze large-scale environments and calculate their properties.
In BRIDGE, we tackle this problem by computing tabular representations for all the environments using a program that exhaustively enumerates all states,
calculating the reward and transition functions at every state-action pair. We do this for 67 Atari games from the Arcade Learning Environmnt [43], 55 levels from the Progen Benchmark [44], and 33 gridworlds from MiniGrid [14] (Figure 5). The MDPs have state space sizes  $S$  ranging across 7 orders of magnitude from tens to tens of millions, 3 to 18 discrete actions, and horizons  $T$  ranging from 10 to 200, which are limited in some cases to avoid the state space becoming too large. See Appendix E for the full details of the BRIDGE dataset.
A surprisingly common property To motivate the effective horizon, which is introduced in the next section, we describe a property that we find holds in many of the MDPs in BRIDGE. Consider the random policy  $\pi^{\mathrm{rand}}$ , which assigns equal probability to every action in every state, i.e.,  $\pi_t^{\mathrm{rand}}(a|s) = 1 / A$ . We can use dynamic programming on a tabular MDP to calculate the random policy's Q-function  $Q^{\pi^{\mathrm{rand}}}$ . We denote by  $\Pi(Q^{\pi^{\mathrm{rand}}})$  the set of policies which act greedily with respect to this Q-function; that is,
$$
\Pi \left(Q ^ {\pi^ {\text {r a n d}}}\right) = \left\{\pi \mid \forall s, t \quad \pi_ {t} (s) \in \arg \max  _ {a \in \mathcal {A}} Q _ {t} ^ {\pi^ {\text {r a n d}}} (s, a) \right\}.
$$
Perhaps surprisingly, we find that all the policies in  $\Pi(Q^{\pi^{\mathrm{rand}}})$  are optimal in about two-thirds of the MDPs in BRIDGE. This proportion is even higher when considering only the environments where PPO empirically succeeds in finding an optimal policy (Table 1). Thus, it seems that this property may be the key to what makes many of these environments tractable for deep RL.
5 The Effective Horizon
We now theoretically analyze why RL should be tractable in environments where, as we observe in BRIDGE, it is optimal to act greedily with respect to the random policy's Q-function. This leads to a more general measure of an environment's complexity for model-free RL: the effective horizon.
Our analysis centers around a simple algorithm, GORP (Greedy Over Random Policy), shown in Algorithm 1. GORP constructs an optimal policy iteratively; each iteration  $i$  aims to calculate an optimal policy  $\pi_{i}$  for timestep  $t = i$ . In the case where we set  $k = 1$  and  $\pi^{\mathrm{expl}} = \pi^{\mathrm{rand}}$ , GORP can solve environments which have the property we observe in BRIDGE. It does this at each iteration  $i$  by simulating  $m$  random rollouts for each action from the state reached at timestep  $t = i$ . Then, it
Acting greedily with respect to Qπrand is optimal?	PPO finds optimal policy in ≤ 5M timesteps?
Yes	No
Yes	80 MDPs	24 MDPs
No	15 MDPs	36 MDPs
Table 1: The distribution of the MDPs in our BRIDGE dataset according to two criteria: first, whether PPO empirically converges to an optimal policy in 5 million timesteps, and second, whether acting greedily with respect to the Q-function of the random policy is optimal. We find that a surprising number of environments satisfy the latter property, especially when only considering those where PPO succeeds.
averages the  $m$  rollouts' returns to obtain a Monte Carlo estimate of  $Q^{\pi^{\mathrm{rand}}}$  for each action. Finally, it greedily picks the action with the highest estimated Q-value.
Besides taking advantage of the surprising property we found in BRIDGE, GORP has other properties which help us bridge the theory-practice gap in RL. It explores randomly, like common deep RL algorithms, meaning that it can give us insight into why random exploration works much better than the worst-case given in Theorem 2.1. Also, unlike other RL algorithms, it has cleanly separated exploration and learning stages, making it much easier to analyze than algorithms in which exploration and learning are entangled.
Furthermore, GORP is extensible beyond environments satisfying the property we found in BRIDGE. First, it can solve MDPs where one may have to apply a few steps of value iteration to the random policy's Q-
Algorithm 1 The Greedy Over Random Policy (GORP) algorithm, used to motivate the effective horizon.
1: procedure GORP(k,m,πexpl)
2: for i = 1,...,T do
3: for ai:i+k-1 ∈ Ak do
4: sample m episodes following π1,..., πi-1, then actions ai:i+k-1, and finally πexpl.
5: Qi(s_i, ai:i+k-1) ← 1/m ∑j=1m ∑t=i T γt-i R(stj, atj).
6: end for
7: πi(s_i) ← arg maxi∈A max ai+1:i+k-1∈Ak-1 Qi(s_i, ai, ai+1:i+k-1).
8: end for
9: return π
10: end procedure