Title: Bridging RL Theory and Practice with the Effective Horizon
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

RL theory

theory of reinforcement learning

instance-dependent bounds

empirical validation of theory
Abstract: Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the *random* policy also have the highest Q-values under the *optimal* policy—i.e., when it is optimal to act greedily with respect to the random's policy Q function—deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the *effective horizon*, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also show that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon.

Bridging Reinforcement Learning Theory and Practice with the Effective Horizon

Abstract

Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon , which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon .

1 Introduction

Deep reinforcement learning (RL) has produced impressive results in robotics [1], strategic games [2], and control [3]. However, the same deep RL algorithms that achieve superhuman performance in some environments completely fail to learn in others. Sometimes, using techniques like reward shaping or pre-training help RL, and in other cases they don't. Our goal is to provide a theoretical understanding of why this is-a theoretical analysis that is predictive of practical RL performance.

Unfortunately, there is a large gap between the current theory and practice of RL. Despite RL theorists often focusing on algorithms using strategic exploration (e.g., UCB exploration bonuses; Azar et al. [4], Jin et al. [5]), the most commonly-used deep RL algorithms, which explore randomly, resist such analysis. In fact, theory suggests that RL with random exploration is exponentially hard in the worst case [6], but this is not predictive of practical performance. Some theoretical research has explored instance-dependent bounds, identifying properties of MDPs when random exploration should perform better than this worst case [7, 8]. However, it is not clear whether these properties correlate with when RL algorithms work vs. fail-and our results will reveal that they tend not to.

If the current theory literature cannot explain the empirical performance of deep RL, what can? Ideally, a theory of RL should provably show why deep RL succeeds while using random exploration. It should also be able to predict which environments are harder or easier to solve empirically. Finally, it should be able to explain when and why tools like reward shaping or initializing with a pre-trained policy help make RL perform better.

We present a new theoretical complexity measure for MDPs called the effective horizon that satisfies all of the above criteria. Intuitively, the effective horizon measures approximately how far ahead an algorithm must exhaustively plan in an environment before evaluating leaf nodes with random rollouts.

Current state

. . .

. . .

k

timesteps

m

random rollouts

10

2

10

6

10

10

Our bounds

10

3

10

5

10

7

PPO sample complexity

ρ

= 0

.

81

from each leaf node

In order to assess previous bounds and eventually arrive at such a property, we start by creating a new dataset, BRIDGE, of deterministic MDPs from common deep RL benchmarks. A major difficulty with evaluating instance-dependent bounds is that they can't be calculated without tabular representations, so prior work work has typically relied on small toy environments for justification. To get a more realistic picture, we choose 155 MDPs across different benchmarks and compute their tabular representations-some with over 100 million states which must be exhaustively explored and stored. This is a massive engineeering challenge, but it enables connecting theoretical and empirical results at an unprecedented scale. We focus on deterministic MDPs in BRIDGE and in this paper because they are simpler to analyze but still have many of the interesting properties of stochastic MDPs, like reward sparsity and credit assignment challenges. Many deep RL benchmarks are (nearly) deterministic, so we believe our analysis is highly relevent to practical RL.

Our journey to the effective horizon began with identifying a surprising property that holds in many of the environments in BRIDGE: one can learn to act optimally by acting randomly . More specifically, actions with the highest Q-values under the uniformly random policy also have the highest Q-values under the optimal policy. The random policy is about as far as one can get from the optimal policy, so this property may seem unlikely to hold. However, about two-thirds of the environments in BRIDGE satisfy the property. This proportion rises to four-fifths among environments that PPO [9], a popular deep RL algorithm, can solve efficiently (Table 1). Conversely, when this property does not hold, PPO is more likely to fail than succeed-and when it does succeed, so does simply applying a few steps of lookahead on the Q-function of the random policy (Figure 6). We found it remarkable that, at least in the environments in BRIDGE, modern algorithms seem to boil down to not much more than acting greedily on the random policy Q-values.

The property that it is optimal to act greedily with respect to the random policy's Q-function has important implications for RL theory and practice. Practically, it suggests that very simple algorithms designed to estimate the random policy's Q-function could efficiently find an optimal policy. We introduce such an algorithm, Greedy Over Random Policy (GORP), which also works in the case where one may need to apply a few steps of value iteration to the random policy's Q-function before acting greedily. Empirically, GORP finds an optimal policy in fewer timesteps than DQN (another deep RL algorithm) in more than half the environments in BRIDGE. Theoretically, it is simple to analyze GORP, which consists almost entirely of estimating the random policy's Q-function via a sample average over i.i.d. random rollouts. Since GORP works well empirically and can be easily understood theoretically, we thoroughly analyze it in the hopes of finding sample complexity bounds that can explain the performance of deep RL.

Our analysis of Greedy Over Random Policy leads to a single metric, the effective horizon, that measures the complexity of model-free RL in an MDP. As shown in Figure 1, GORP is an adaptation of a Monte Carlo planning algorithm to the reinforcement learning setup (where the transitions are unknown): it mimics exhaustively planning ahead k steps and then sampling m random rollouts from

each leaf node. The effective horizon H combines the depth k and number of rollouts m . We call it the effective horizon because worst-case sample complexity bounds for random exploration are exponential in the horizon T , while we prove sample complexity bounds exponential only in the effective horizon H . For most BRIDGE environments, H ≪ T , explaining the efficiency of RL in these MDPs.

In the environments in BRIDGE, we find that the effective horizon-based sample complexity bounds satisfy all our desiderata above for a theory of RL. They are more predictive of the empirical sample complexities of PPO and DQN than several other bounds from the literature across four metrics, including measures of correlation, tightness, and accuracy (Table 2). Furthermore, the effective horizon can predict the effects of both reward shaping (Table 3a) and initializing using a pre-trained policy learned from human data or transferred from a similar environment (Table 3b). In contrast, none of the existing bounds we compare to depend on both the reward function and initial policy; thus, they are unable to explain why reward shaping, human data, and transfer learning can help RL. Although our results focus on deterministic MDPs, we plan to extend our work to stochastic environments in the future and already have some promising results in that direction.

2 Preliminaries

We begin by presenting the reinforcement learning (RL) setting we consider. An RL algorithm acts in a deterministic, tabular, episodic Markov decision process (MDP) with finite horizon. The MDP comprises a set of states S , a set of actions A , a horizon T ∈ N and optional discount factor γ ∈ [0 , 1] , a start state s 1 , transition function f : S ×A → S , and a reward function R : S ×A → R . Throughout the paper we use γ = 1 but all our theory applies equally when γ < 1 .

An RL agent interacts with the MDP for a number of episodes, starting at a fixed start state s 1 . At each step t ∈ [ T ] of an episode (using the notation [ n ] = { 1 , . . . , n } ), the agent observes the state s t , picks an action a t , receives reward R ( s t , a t ) , and transitions to the next state s t +1 = f ( s t , a t ) . A policy π is a set of functions π 1 , . . . , π t : S → ∆( A ) , which defines for each state and timestep a distribution π t ( a | s ) over actions. If a policy is deterministic at some state, then with slight abuse of notation we denote a = π t ( s ) to be the action taken by π t in state s .

We denote a policy's Q-function Q π t : S × A → R and value function V π t : S → R for each t ∈ [ T ] . In this paper, we also use a Q-function which is generalized to sequences of actions. We use the shorthand a t : t + k to denote the sequence a t , . . . , a t + k , and define the action-sequence Q-function as

The objective of an RL algorithm is to find an optimal policy π ∗ , which maximizes J ( π ) = V π 1 ( s 1 ) , the expected discounted sum of rewards over an episode, also known as the return of the policy π .

Generally, an RL algorithm can be run for any number of timesteps n (i.e., counting one episode as T timesteps), returning a policy π n . We define the sample complexity N of an RL algorithm as the minimum number of timesteps needed such that the algorithm has at least a 50-50 chance of returning an optimal policy:

Here, the probability is with respect to any randomness in the algorithm itself. One can estimate the sample complexity N empirically by running an algorithm several times, calculating the number of samples n needed to reach the optimal policy during each run, and then taking the median.

The following simple theorem gives upper and lower bounds for the worst-case sample complexity in a deterministic MDP, depending on A and T .

Theorem 2.1. There is an RL algorithm which can solve any deterministic MDP with sample complexity N ≤ T ⌈ A T / 2 ⌉ . Conversely, for any RL algorithm and any values of T and A , there must be some deterministic MDP for which its sample complexity N ≥ T ( ⌈ A T / 2 ⌉ -1) .

All proofs are deferred to Appendix A. In this case, the idea of the proof is quite simple, and will later be useful to motivate our idea of the effective horizon: in an MDP where exactly one sequence of actions leads to a reward, an RL algorithm may have to try almost every sequence of actions to find the optimal policy; there are A T such sequences. As we develop sample complexity bounds based on

the effective horizon in Section 5, we can compare them to the worst-case bounds in Theorem 2.1.

Why deterministic MDPs? We focus on deterministic (as opposed to stochastic) MDPs in this study for several reasons. First, analyzing deterministic MDPs avoids the need to consider generalization within RL algorithms. In common stochastic MDPs, one often needs neural-network based policies, whereas in a deterministic MDP one can simply learn a sequence of actions. Since neural network generalization is not well understood even in supervised learning, analyzing generalization in RL is an especially difficult task. Second, deterministic MDPs still display many of the interesting properties of stochastic MDPs. For instance, deterministic MDPs have worst case exponential sample complexity when using naive exploration; environments with dense rewards are easier to solve empirically than those with sparse rewards; credit assignment can be challenging; and there is a wide range of how tractable environments are for deep RL, even for environments with similar horizons, state spaces, and action spaces.

Finally, many common RL benchmark environments are deterministic or nearly-deterministic. For instance, the ALE Atari games used to evaluate DQN [3], Rainbow [10], and MuZero [11] are all deterministic after the first state, which is selected randomly from one of only 30 start states. The widely used DeepMind Control Suite [12] is based on the MuJoCo simulator [13], which is also deterministic given the initial state (some of the environments do use a randomized start state). MiniGrid environments [14], which are commonly used for evaluating exploration [15], environment design [16], and language understanding [17], are also deterministic after the initial state. Thus, our investigation of deterministic environments is highly relevant to common deep RL practice.

3 Related Work

Before delving into our contributions, we briefly summarize existing work in theoretical RL and prior sample complexity bounds. Our novel bounds are contrasted with existing ones in Sections 5.1 and 6; for a detailed comparison with full definitions and proofs, please see Appendix D.

Recent RL theoretical results largely focus on strategic exploration using techniques like UCB exploration bonuses [18, 4, 19, 5, 20, 21, 22]. Such bounds suggest RL is tractable for smaller or low-dimensional state spaces. In deterministic MDPs, the UCB-based R-MAX algorithm [23, 18] has sample complexity bounded by SAT .

Some prior work has focused on sample complexity bounds for random exploration. Liu and Brunskill [7] give bounds based on the covering length L of an MDP, which is the number of episodes needed to visit all state-action pairs at least once with probability at least 1 / 2 while taking actions at random. This yields a sample complexity bound of TL for deterministic MDPs. Other work suggests that it may not be necessary to consider rewards all the way to the end of the episode to select an optimal action [24, 25, 8]. One can define a 'effective planning window' of W timesteps ahead that must be considered, resulting in a sample complexity bound of T 2 A W for deterministic MDPs. Finally, Dann et al. [6] define a 'myopic exploration gap' that controls the sample complexity of using ϵ -greedy exploration, a form of naive exploration. However, in Appendix D.4, we demonstrate why their bounds are impractical and often vacuous.

There have been a few prior attempts to bridge the RL theory-practice gap. bsuite [26], MDP playground [27], and SEGAR [28] are collections of environments that are designed to empirically evaluate deep RL algorithms across various axes of environment difficulty. However, they do not provide theoretical explanations for why environments with varying properties are actually easier or harder. Furthermore, their environments are artificially constructed to have understandable properties. In contrast, we aim to find the mathematical reasons that deep RL succeeds or fails in 'in-the-wild' environments like Atari and Procgen. Conserva and Rauber [29] calculate two regret bounds and compare the bounds to the empirical performance of RL algorithms. However, they consider tabular RL algorithms in simple artificial environments with less than a thousand states, while we experiment with deep RL algorithms on real benchmark environments with tens of millions of states.

Our GORP algorithm and the effective horizon are inspired by rollout and Monte Carlo planning algorithms, which have a long history [30, 31, 32, 33, 24, 34, 35, 36]. These algorithms were used effectively in Backgammon [32], Go [37], and real-time strategy games [38] before the start of deep RL. GORP and related Monte Carlo rollout algorithms are sometimes referred to as 'one-step' or 'multi-step' lookahead. Bertsekas [39, 40] suggests that one-step lookahead, possibly after a few

steps of value iteration, often leads to fast convergence to optimal policies because it is equivalent to a step of Newton's method for finding a fixed-point of the Bellman equation [41, 42]. Our analysis suggests the success of deep RL is due to similar properties. However, we go beyond previous work by introducing GORP, which approximates a step of policy iteration in model-free RL-a setting where on-line planning approaches are not applicable. Furthermore, unlike previous work in Monte-Carlo planning, we combine our theoretical contributions with extensive empirical analysis to verify that our assumptions hold in common environments.

4 The BRIDGE Dataset

In order to assess how well existing bounds predict practical performance, and gain insight about novel properties of MDPs that could be predictive, we constructed BRIDGE (Bridging the RL Interdisciplinary Divide with Grounded Environments), a dataset of 155 popular deep RL benchmark environments with full tabular representations. One might assume that the ability to calculate the instancedependent bounds we just presented in Section 3 already exists; however, it turns out that for many real environments even the number of states S is unknown! This is because a significant engineering effort is required to analyze large-scale environments and calculate their properties.

In BRIDGE, we tackle this problem by computing tabular representations for all the environments using a program that exhaustively enumerates all states,

calculating the reward and transition functions at every state-action pair. We do this for 67 Atari games from the Arcade Learning Enivornment [43], 55 levels from the Procgen Benchmark [44], and 33 gridworlds from MiniGrid [14] (Figure 5). The MDPs have state space sizes S ranging across 7 orders of magnitude from tens to tens of millions, 3 to 18 discrete actions, and horizons T ranging from 10 to 200, which are limited in some cases to avoid the state space becoming too large. See Appendix E for the full details of the BRIDGE dataset.

A surprisingly common property To motivate the effective horizon, which is introduced in the next section, we describe a property that we find holds in many of the MDPs in BRIDGE. Consider the random policy π rand , which assigns equal probability to every action in every state, i.e., π rand t ( a | s ) = 1 /A . We can use dynamic programming on a tabular MDP to calculate the random policy's Q-function Q π rand . We denote by Π( Q π rand ) the set of policies which act greedily with respect to this Q-function; that is,

Perhaps surprisingly, we find that all the policies in Π( Q π rand ) are optimal in about two-thirds of the MDPs in BRIDGE. This proportion is even higher when considering only the environments where PPO empirically succeeds in finding an optimal policy (Table 1). Thus, it seems that this property may be the key to what makes many of these environments tractable for deep RL.

5 The Effective Horizon

We now theoretically analyze why RL should be tractable in environments where, as we observe in BRIDGE, it is optimal to act greedily with respect to the random policy's Q-function. This leads to a more general measure of an environment's complexity for model-free RL: the effective horizon.

Our analysis centers around a simple algorithm, GORP (Greedy Over Random Policy), shown in Algorithm 1. GORP constructs an optimal policy iteratively; each iteration i aims to calculate an optimal policy π i for timestep t = i . In the case where we set k = 1 and π expl = π rand , GORP can solve environments which have the property we observe in BRIDGE. It does this at each iteration i by simulating m random rollouts for each action from the state reached at timestep t = i . Then, it

averages the m rollouts' returns to obtain a Monte Carlo estimate of Q π rand for each action. Finally, it greedily picks the action with the highest estimated Q-value.

Besides taking advantage of the surprising property we found in BRIDGE, GORP has other properties which help us bridge the theory-practice gap in RL. It explores randomly, like common deep RL algorithms, meaning that it can give us insight into why random exploration works much better than the worst-case given in Theorem 2.1. Also, unlike other RL algorithms, it has cleanly separated exploration and learning stages, making it much easier to analyze than algorithms in which exploration and learning are entangled.

Furthermore, GORP is extensible beyond environments satisfying the property we found in BRIDGE. First, it can solve MDPs where one may have to apply a few steps of value iteration to the random policy's Q-

Algorithm 1 The Greedy Over Random Policy (GORP) algorithm, used to motivate the effective horizon.

1:

procedure

GORP(

k, m, π

expl

)

2:

for

i

= 1

, . . . , T

do

3:

for

a

i

:

i

+

k

-

1

∈ A

k

do

4:

sample

m

episodes following

π

1

, . . . , π

i

-

1

,

then actions

a

i

:

i

+

k

-

1

, and finally

π

expl

.

5:

ˆ

Q

i

(

s

i

, a

i

:

i

+

k

-

1

)

1

m

∑

m

j

=1

∑

T

t

=

i

γ

t

-

i

R

(

s

j

t

, a

j

t

)

.

6:

end for

7:

π

i

(

s

i

)

a

i

∈A

a

i

+1:

i

+

k

-

1

∈A

k

-

1

ˆ

Q

i

(

s

i

, a

i

, a

i

+1:

i

+

k

-

1

)

.

8:

end for

9:

return

π

10:

end procedure

function before acting randomly. Second, it can use an 'exploration policy' π expl different from the random policy π rand . These two generalizations are captured in the following definition. In the definition, we use the notation that a step of Q-value iteration transforms a Q-function Q to Q ′ = QVI ( Q ) , where

Definition 5.1 ( k -QVI-solvable) . Given an exploration policy π expl ( π expl = π rand unless otherwise noted), let Q 1 = Q π expl and Q i +1 = QVI ( Q i ) for i = 1 , . . . , T -1 . We say an MDP is k -QVI-solvable for some k ∈ [ T ] if every policy in Π( Q k ) is optimal.

We will see that running GORP with k > 1 will allow it to find an optimal policy in MDPs that are k -QVI-solvable. Although the sample complexity of GORP scales with A k , we find that nearly all of the environments in BRIDGE are k -QVI-solvable for very small values of k (Figure 6).

We now use GORP to define the effective horizon of an MDP. Note that the total number of timesteps sampled by GORP with parameters k and m is T 2 A k m = T 2 A k +log A m . Thus, analogously to how the horizon T appears in the exponent of the worst-case sample complexity bound O ( TA T ) , we define the effective horizon as the exponent of A in the sample complexity of GORP:

Definition 5.2 (Effective horizon) . Given k ∈ [ T ] , let H k = k +log A m k , where m k is the minimum value of m needed for Algorithm 1 with parameter k to return the optimal policy with probability at least 1 / 2 , or ∞ if no value of m suffices. The effective horizon is H = min k H k .

By definition, the sample complexity of GORP can be given using the effective horizon:

Lemma 5.3. The sample complexity of GORP with optimal choices of k and m is T 2 A H .

As we noted in the introduction, when H ≪ T , as we find is often true in practice, this is far better than the worst-case bound given in Theorem 2.1 which scales with A T .

Definition 5.2 does not give a method for actually calculating the effective horizon. It turns out we can bound the effective horizon using a generalized gap notion like those found throughout the RL theory literature. We denote by ∆ k t the gap of the Q-function Q k from Definition 5.1, where

The following theorem gives bounds on the effective horizon in terms of this gap.

Theorem 5.4. Suppose that an MDP is k -QVI-solvable and that all rewards are nonnegative, i.e. R ( s, a ) ≥ 0 for all s, a . Let ∆ k denote the gap of the Q-function Q k as defined in Definition 5.1.

s

1

s

2

R

= 0

. . .

0

s

2

0

. . .

R

= 0

. . .

0

. . .

0

. . .

0

s

T

R

= 1

s

T

0

s

T

0

s

T

0

(a) Sparse rewards: when only one sequence of actions gives a reward of 1 and all others give 0, the effective horizon H = ˜ O ( T ) .

s

1

s

2

R

= 1

. . .

0

s

2

0

. . .

R

= 1

. . .

0

. . .

0

. . .

0

s

T

R

= 1

s

T

0

s

T

0

s

T

0

(b) Dense rewards: when every optimal action gives a reward of 1 and suboptimal actions give no reward, the effective horizon H = ˜ O (1) .

s

1

s

2

R

= 0

s

2

0

. . .

R

= 0

. . .

0

. . .

0

s

T

R

=

T

s

T

T

-

1

s

T

1

s

T

0

(c) Delayed rewards: when the rewards in (b) are all delayed to the end of the episode, the effective horizon remains ˜ O (1) .

Q

1

10

(

s,

NOOP

) =

.

118

Q

1

10

(

s,

UP

) =

.

136

Q

1

10

(

s,

DOWN

) =

.

116

Q

∗

10

(

s,

NOOP

) = 4

Q

∗

10

(

s,

UP

) = 5

Q

∗

(

s,

DOWN

) = 4

10

(d) For the first 50 timesteps of the Atari game Freeway, we can bound H ≤ 10 . 2 , which is much lower than the horizon T = 50 .

Then

where S opt i is the set of states visited by some optimal policy at timestep i and V ∗ t ( s ) = max π V π t ( s ) is the optimal value function.

A full proof of Theorem 5.4 is given in Appendix A. Intuitively, the smaller the gap ∆ k t ( s ) , the more precisely we must estimate the Q-values in GORP in order to pick an optimal action.

The GORP algorithm is very amenable to theoretical analysis because it reduces the problem of finding an optimal policy to the problem of estimating several k -step Q-values, each of which is a simple mean of i.i.d. random variables. There are endless tail bounds that can be applied to analysis of GORP; we use some of these to obtain even tighter bounds on the effective horizon in Appendix C.

Why should the effective horizon, which is defined in terms of our GORP algorithm, also explain the performance of deep RL algorithms like PPO and DQN which are very different from GORP? In Appendix B, we present two algorithms, PG-GORP and FQI-GORP, which are more similar to PPO and DQN but whose sample complexities can still be bounded with the effective horizon. We also give additional bounds on the effective horizon and lower bounds on sample complexity.

5.1 Examples of the effective horizon

To gain some intuition for the bound in Theorem 5.4, consider the examples in Figure 2. MDP (a) has extremely sparse rewards, with a reward of 1 only given for a single optimal action sequence. However, note that this MDP is still 1-QVI-solvable by Definition 5.1. The maximum of the bound in Theorem 5.4 is at t = 1 with the optimal action, where Q 1 1 ( s, a ) = 1 /A T -1 , V ∗ 1 ( s ) = 1 , and ∆ 1 1 ( s ) = 1 /A T -1 . Ignoring logarithmic factors and constants gives H ≲ 1 + log A A T -1 = T . That is, in the case of MDP (a), the effective horizon is no better than the horizon.

Next, consider MDP (b), which has dense rewards of 1 for every optimal action. Again, this MDP is 1-QVI-solvable. The maximum in the bound is obtained at t = 1 and the optimal action with Q 1 1 ( s, a ) ≤ 2 , V ∗ ( s ) = T , and the gap ∆ 1 1 ( s, a ) ≥ 1 . Again ignoring logarithmic factors gives in this case H ≲ 1+log A T = ˜ O (1) . In this case, the effective horizon is much shorter than the horizon, and barely depends on it! This again reflects our intuition that in this case, finding the optimal policy via RL should be much easier.

MDP (c) is similar to MDP (b) except that all rewards are delayed to the end of the episode. In this case, the Q function is the same as in MDP (b) so the effective horizon remains ˜ O (1) . This may seem counterintuitive since one needs to consider rewards T timesteps ahead to act optimally. However, the way GORP uses random rollouts to evaluate leaf nodes means that it can implicitly consider rewards quite far in the future even without needing to exhaustively plan that many timesteps ahead.

Finally, consider MDP (d), the first 50 timesteps of the Atari game Freeway, which is included in the BRIDGE dataset. This MDP is also 1-QVI-solvable and the maximum in the bound is obtained in the state shown in Figure 2d at timestep t = 10 . Plugging in the Q values shown in the figure gives H ≤ 10 . 2 , which is far lower than the horizon T = 50 . The low effective horizon reflects how this MDP is much easier than the worst case in practice. Both PPO and DQN are able to solve it with a sample complexity of less than 1.5 million timesteps, while the worst case bound would suggest a

sample complexity greater than 50 × 3 50 / 2 ≈ 10 25 timesteps!

Comparison to other bounds Intuitively, why might the effective horizon give better sample complexity bounds than previous works presented in Section 3? The MDP in Figure 2b presents a problem for the covering length and UCB-based bounds, both of which are Ω( A T ) . The exponential sample complexity arises because these bounds depend on visiting every state in the MDP during training. In contrast, GORP doesn't need to visit every state to find an optimal policy. The effective horizon of ˜ O (1) for MDP (b) reflects this, showing that our effective horizon-based bounds can actually be much smaller than the state space size, which is on the order of A T for MDP (b).

The effective planning window (EPW) does manage to capture the same intuition as the effective horizon in the MDP in Figure 2b: in this case, W = 1 . However, the analysis based on the EPW is unsatisfactory because it entirely ignores rewards beyond the planning window. Thus, in MDP (c) the EPW W = T , making EPW-based bounds no better than the worst case. In contrast, the effective horizon-based bound remains the same between MDPs (b) and (c), showing that it can account for the ability of RL algorithms to use rewards beyond the window where exhaustive planning is possible.

6 Experiments

We now show that sample complexity bounds based on the effective horizon predict the empirical performance of deep RL algorithms far better than other bounds in the literature. For each MDP in the BRIDGE dataset, we run deep RL algorithms to determine their empirical sample complexity. We also use the tabular representations of the MDPs to calculate the effective horizon and other sample complexity bounds for comparison.

Deep RL algorithms We run both PPO and DQN for five million timesteps for each MDP in BRIDGE, and record the empirical sample complexity (see Appendix F for hyperparameters and experiment details). PPO converges to the optimal policy in 95 of the 155 MDPs, and DQN does in 117 of 155. At least one of the two finds the optimal policy in 119 MDPs.

Sample complexity bounds Wealso compute sample complexity bounds for each MDP in BRIDGE. These include the worst-case bound of TA T from Theorem 2.1, the effective-horizon-based bound of T 2 A H from Lemma 5.3, as well as three other bounds from the literature, introduced in Section 3 and proved in Appendix D: the UCB-based bound SAT , the covering-length-based bound TL , and the effective planning window (EPW)-based bound of T 2 A W .

Evaluation metrics To determine which sample complexity bounds best reflect the empirical performance of PPO and DQN, we compute a few summary metrics for each bound. First, we measure the Spearman (rank) correlation between the sample complexity bounds and the empirical sample complexity over environments where the algorithm converged to the optimal policy. The correlation (higher is better) is a useful for measuring how well the bounds can rank the relative difficulty of RL in different MDPs.

Second, we compute the median ratio between the sample complexity bound and the empirical sample complexity for environments where the algorithm converged. The ratio between the bound N bound and empirical value N emp is calculated as max { N bound /N emp , N emp /N bound } . For instance, a median ratio of 10 indicates that half the sample complexity bounds were within a factor of 10 of the empirical sample complexity. Lower values indicate a better bound; this metric is useful for determining whether the sample complexity bounds are vacuous or tight.

Finally, we consider the binary classification task of predicting whether the algorithm will converge at all within five million steps using the sample complexity bounds. That is, we consider simply thresholding each sample complexity bound and predicting that only environments with bounds below the threshold will converge. We compute the area under the ROC curve (AUROC) for this prediction task as well as the accuracy with the optimal threshold. Higher AUROC and accuracy both indicate a better bound.

Results The results of our experiments are shown in Table 2. The effective horizon-based bounds have higher correlation with the empirical sample complexity than the other bounds for both PPO and DQN. While the EPW-based bounds are also reasonably correlated, they are significantly off in absolute terms: the typical bound based on the EPW is 3-4 orders of magnitude off, while the effective horizon yields bounds that are typically within 1-2 orders of magnitude. The UCB-based

Table 2: Effective horizon-based sample complexity bounds are the most predictive of the real performance of PPO and DQN according to the four metrics we describe in Section 6. The effective horizon bounds are about as good at predicting the sample complexity of PPO and DQN as one algorithm's sample complexity is at predicting the other's.

(a) Reward shaping.

(b) Initializing with a policy trained on human data or transferred from similar environments.

bounds are somewhat closer to the empirical sample complexity, but are not well correlated; this makes sense since the UCB bounds depend on strategic exploration, while PPO and DQN use random exploration. Finally, the effective horizon bounds are able to more accurately predict whether PPO or DQN will find an optimal policy, as evidenced by the AUROC and accuracy metrics.

As an additional baseline, we also calculate the four evaluation metrics when using the empirical sample complexity of PPO to predict the empirical sample complexity of DQN, or vice-versa, and when using the empirical sample complexity of GORP to predict PPO or DQN's performance (bottom two rows of Table 2). While these are not provable bounds, they provide another point of comparison for each metric. The effective horizon-based bounds correlate about as well with PPO and DQN's sample complexities as they do with each other's. The empirical performance of GORP is typically even closer to that of PPO and DQN than the effective horizon-based bounds.

Reward shaping, human data, and transfer learning In order for RL theory to be useful practically, it should help practitioners make decisions about which techniques to apply in order to improve their algorithms' performance. We show how the effective horizon can be used to explain the effect of three such techniques: reward shaping, using human data, and transfer learning.

Potential-based reward shaping [45] is a classic technique which can speed up the convergence of RL algorithms. It is generally used to transform a sparse-reward MDP like the one in Figure 2a to a dense-reward MDP like in Figure 2b without changing the optimal policy. If the effective horizon is a good measure of the difficulty of RL in an environment, then it should be able to predict whether (and by how much) the sample complexity of practical algorithms changes when reward shaping is applied. We develop 77 reward-shaped versions of the original 33 Minigrid environments and run PPO and DQN. Results in Table 3a show the effective horizon accurately captures the change in sample complexity from using reward shaping. We use similar metrics to those in Table 2: the correlation between the predicted and empirical ratio of the shaped sample complexity to the unshaped sample complexity, and the median ratio between the predicted change and the actual change.

Out of the five bounds we consider, three-worst case, covering length, and UCB-don't even depend on the reward function. The EPW does depend on the reward function and captures some of the effect of reward shaping for DQN, but does worse at predicting the effect on PPO. In comparison, the effective horizon does well for both algorithms, showing that it can accurately capture how reward shaping affects RL performance.

0

50M

1k

2k

3k

Beam Rider

0

50M

0

250

Breakout

0

50M

0

1k

Enduro

0

50M

-20

0

20

Pong

0

50M

0

20k

0

50M

0

2k

Seaquest

0

50M

0

2k

Space Invaders

GORP

PPO

DQN

Another tool used to speed up RL is initializing with a pre-trained policy, which is used practically to make RL work on otherwise intractable tasks. Can the effective horizon also predict whether initializing RL with a pre-trained policy will help? We initialize PPO with pre-trained policies for 82 of the MDPs in BRIDGE, then calculate new sample complexity bounds based on using the pre-trained policies as an exploration policy π expl . Table 3b shows that the effective horizon accurately predicts the change in sample complexity due to using a pre-trained policy. Again, three bounds-worst case, EPW, and UCB-do not depend on the exploration policy at all, while the covering length gives wildly inaccurate predictions for its effect. In contrast, the effective horizon is accurate at predicting the changes in sample complexities when using pre-trained policies.

Long-horizon environments We also perform experiments on full-length Atari games to evaluate the predictive power of the effective horizon in longer-horizon environments. It is intractable to construct tabular representations of these environments and thus we cannot compute instance-dependent sample complexity bounds. However, it is still possible to compare the empirical performance of PPO, DQN, and GORP. If the performance of GORP is close to that of PPO and DQN, then this suggests that the effective horizon, which is defined in terms of GORP, can explain RL performance in these environments as well. Figure 3 compares the learning curves of PPO, DQN, and GORP in deterministic versions of the seven Atari games from Mnih et al. [46] (see Appendix F.1 for details). GORP performs better than both PPO and DQN in two games and better than at least one deep RL algorithm in an additional three games. This provides evidence that the effective horizon is also predictive of RL performance in long-horizon environments.

7 Discussion

Overall, our results suggest the effective horizon is a key measure of the difficulty of solving an MDP via reinforcement learning. The intuition behind the effective horizon presented in Section 5 and the empirical evidence in Section 6 both support its importance for better understanding RL.

Limitations While we have presented a thorough theoretical and empirical justification of the effective horizon, there are still some limitations to our analysis. First, we focus on deterministic MDPs with discrete action spaces, leaving the extension to stochastic environments and those with continuous action spaces an open question. Furthermore, the effective horizon is not easily calculable without full access to the MDP's tabular representation. Despite this, it serves as a useful perspective for understanding RL's effectiveness and potential improvement areas. An additional limitation is that the effective horizon cannot capture the performance effects of generalization-the ability to use actions that work well at some states for other similar states. For an example where the effective horizon fails to predict generalization, see Appendix G.1. However, the effective horizon is still quite predictive of deep RL performance even without modeling generalization.

Implications and future work We hope that this paper helps to bring the theoretical and empirical RL communities closer together in pursuit of shared understanding. Theorists can extend our analysis of the effective horizon to new algorithms or explore related properties, using our BRIDGE dataset to ground their investigations by testing assumptions in real environments. Empirical RL researchers can use the effective horizon as a foundation for new algorithms. For instance, Brandfonbrener et al. [47] present an offline RL algorithm similar to GORP with k = 1 ; our theoretical understanding of GORP might provide insights for improving it or developing related algorithms.

Acknowledgments and Disclosure of Funding

We would like to thank Jacob Steinhardt, Kush Bhatia, Lawrence Chan, Paria Rashidinejad, Ruiqi Zhong, and Yaodong Yu for feedback on drafts as well as Jiantao Jiao and Banghua Zhu for helpful discussions.

This work was partially supported by the Open Philanthropy Project, the Office of Naval Researcher's Young Investigator Program (YIP), and Weill Neurohub. Cassidy Laidlaw is supported by an Open Philanthropy AI Fellowship and a National Defense Science and Engineering Graduate (NDSEG) Fellowship. Stuart Russell is supported by an AI 2050 Senior Fellowship from Schmidt Futures.

References

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, January 2016. ISSN 1532-4435.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489, January 2016. ISSN 1476-4687. doi: 10.1038/nature16961. URL https://www.nature.com/articles/nature16961 . Bandiera_abtest: a Cg_type: Nature Research Journals Number: 7587 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Computational science;Computer science;Reward Subject_term_id: computational-science;computer-science;reward.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, February 2015. ISSN 1476-4687. doi: 10.1038/nature14236. URL https://www.nature.com/articles/nature14236 . Bandiera_abtest: a Cg_type: Nature Research Journals Number: 7540 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Computer science Subject_term_id: computer-science.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax Regret Bounds for Reinforcement Learning. arXiv:1703.05449 [cs, stat] , July 2017. URL http://arxiv.org/ abs/1703.05449 . arXiv: 1703.05449.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I. Jordan. Is Q-learning Provably Efficient? arXiv:1807.03765 [cs, math, stat] , July 2018. URL http://arxiv.org/abs/ 1807.03765 . arXiv: 1807.03765.

Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximation. In Proceedings of the 39th International Conference on Machine Learning , pages 4666-4689. PMLR, June 2022. URL https://proceedings.mlr.press/v162/dann22a.html . ISSN: 2640-3498.

Yao Liu and Emma Brunskill. When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithms, April 2019. URL http://arxiv.org/abs/1805.09045 . arXiv:1805.09045 [cs, stat].

Dhruv Malik, Aldo Pacchiano, Vishwak Srinivasan, and Yuanzhi Li. Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearity. In Proceedings of the 38th International Conference on Machine Learning , pages 7412-7422. PMLR, July 2021. URL https://proceedings.mlr.press/v139/malik21c.html . ISSN: 2640-3498.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs] , August 2017. URL http://arxiv. org/abs/1707.06347 . arXiv: 1707.06347.

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning, October 2017. URL http://arxiv.org/abs/1710. 02298 . arXiv:1710.02298 [cs].

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature , 588(7839):604-609, December 2020. ISSN 0028-0836, 1476-4687. doi: 10. 1038/s41586-020-03051-4. URL http://arxiv.org/abs/1911.08265 . arXiv:1911.08265 [cs, stat].

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite, January 2018. URL http://arxiv.org/abs/1801. 00690 . arXiv:1801.00690 [cs].

Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, October 2012. doi: 10.1109/IROS.2012.6386109. ISSN: 2153-0866.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic Gridworld Environment for OpenAI Gym, 2018. URL https://github.com/maximecb/gym-minigrid . Publication Title: GitHub repository.

Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State Entropy Maximization with Random Encoders for Efficient Exploration. In Proceedings of the 38th International Conference on Machine Learning , pages 9443-9454. PMLR, July 2021. URL https://proceedings.mlr.press/v139/seo21a.html . ISSN: 2640-3498.

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design. In Advances in Neural Information Processing Systems , volume 33, pages 13049-13061. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper/2020/hash/985e9a46e10005356bbaf194249f6856-Abstract.html .

Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning, December 2019. URL http://arxiv.org/abs/ 1810.08272 . arXiv:1810.08272 [cs].

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . PhD thesis, University of London, University College London (United Kingdom), 2003.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual Decision Processes with low Bellman rank are PAC-Learnable. In Proceedings of the 34th International Conference on Machine Learning , pages 1704-1713. PMLR, July 2017. URL https://proceedings.mlr.press/v70/jiang17c.html . ISSN: 2640-3498.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably Efficient Reinforcement Learning with Linear Function Approximation. arXiv:1907.05388 [cs, math, stat] , August 2019. URL http://arxiv.org/abs/1907.05388 . arXiv: 1907.05388.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear Classes: A Structural Framework for Provable Generalization in RL. In Proceedings of the 38th International Conference on Machine Learning , pages 2826-2836. PMLR, July 2021. URL https://proceedings.mlr.press/v139/du21a.html . ISSN: 2640-3498.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms. In Advances in Neural Information Processing Systems , volume 34, pages 13406-13418. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 6f5e4e86a87220e5d361ad82f1ebc335-Abstract.html .

Ronen I. Brafman and Moshe Tennenholtz. R-MAX - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research , 3(Oct):213231, 2002. ISSN ISSN 1533-7928. URL https://www.jmlr.org/papers/v3/brafman02a. html .

Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes. Machine Learning , 49(2):193208, November 2002. ISSN 1573-0565. doi: 10.1023/A:1017932429737. URL https: //doi.org/10.1023/A:1017932429737 .

Nan Jiang, Satinder Singh, and Ambuj Tewari. On structural properties of MDPs that bound loss due to shallow planning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence , IJCAI'16, pages 1640-1647, New York, New York, USA, July 2016. AAAI Press. ISBN 978-1-57735-770-4.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour Suite for Reinforcement Learning, February 2020. URL http://arxiv.org/abs/1908.03568 . arXiv:1908.03568 [cs, stat].

Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Jan Ole von Hartz, and Frank Hutter. MDP Playground: A Design and Debug Testbed for Reinforcement Learning, June 2021. URL http://arxiv.org/abs/1909.07750 . arXiv:1909.07750 [cs, stat].

R. Devon Hjelm, Bogdan Mazoure, Florian Golemo, Felipe Frujeri, Mihai Jalobeanu, and Andrey Kolobov. The Sandbox Environment for Generalizable Agent Research (SEGAR), March 2022. URL http://arxiv.org/abs/2203.10351 . arXiv:2203.10351 [cs].

Michelangelo Conserva and Paulo Rauber. Hardness in Markov Decision Processes: Theory and Practice, October 2022. URL http://arxiv.org/abs/2210.13075 . arXiv:2210.13075 [cs].

B. Abramson. Expected-outcome: a general model of static evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 12(2):182-193, February 1990. ISSN 1939-3539. doi: 10.1109/34.44404. URL https:// ieeexplore.ieee.org/abstract/document/44404?casa_token=uj3nSMaKLJcAAAAA: d7LSY9mmrlNL8qhkLN6WEKgrZpmv1pRnvwj0cAolitRmNtbTUFegNNstXuXCEwBnrPayNKBO . Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Bernd Brügmann. Monte Carlo Go. 1993. URL https:// www.semanticscholar.org/paper/Monte-Carlo-Go-Max-Planck/ 5f8b18be86be69077e66377e03198d21d06833f3 .

Gerald Tesauro and Gregory Galperin. On-line Policy Improvement using MonteCarlo Search. In Advances in Neural Information Processing Systems , volume 9. MIT Press, 1996. URL https://proceedings.neurips.cc/paper/1996/hash/ 996009f2374006606f4c0b0fda878af1-Abstract.html .

Dimitri P. Bertsekas, John N. Tsitsiklis, and Cynara Wu. Rollout Algorithms for Combinatorial Optimization. Journal of Heuristics , 3(3):245-262, December 1997. ISSN 1572-9397. doi: 10.1023/A:1009635226865. URL https://doi.org/10.1023/A:1009635226865 .

Hyeong Soo Chang, Michael C. Fu, Jiaqiao Hu, and Steven I. Marcus. An Adaptive Sampling Algorithm for Solving Markov Decision Processes. Operations Research , 53(1):126-139, February 2005. ISSN 0030-364X. doi: 10.1287/opre.1040.0145. URL https://pubsonline. informs.org/doi/10.1287/opre.1040.0145 . Publisher: INFORMS.

Levente Kocsis and Csaba Szepesvári. Bandit Based Monte-Carlo Planning. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning: ECML 2006 , Lecture Notes in Computer Science, pages 282-293, Berlin, Heidelberg, 2006. Springer. ISBN 978-3-540-46056-5. doi: 10.1007/11871842_29.

Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L. M. (Jeroen) Donkers, editors, Computers and Games , Lecture Notes in Computer Science, pages 72-83, Berlin, Heidelberg, 2007. Springer. ISBN 978-3-540-75538-8. doi: 10.1007/978-3-540-75538-8_7.

B. Bouzy and B. Helmstetter. Monte-Carlo Go Developments. In H. Jaap Van Den Herik, Hiroyuki Iida, and Ernst A. Heinz, editors, Advances in Computer Games: Many Games, Many Challenges , IFIP - The International Federation for Information Processing, pages 159-174. Springer US, Boston, MA, 2004. ISBN 978-0-387-35706-5. doi: 10.1007/978-0-387-35706-5_ 11. URL https://doi.org/10.1007/978-0-387-35706-5_11 .

Michael Chung, Michael Buro, and Jonathan Schaeffer. Monte Carlo Planning in RTS Games. January 2005.

Dimitri Bertsekas. Rollout, Policy Iteration, and Distributed Reinforcement Learning . Athena Scientific, Belmont, Massachusetts, first edition edition, August 2020. ISBN 978-1-886529-076.

Dimitri P. Bertsekas. Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control . Athena Scientific, March 2022. ISBN 978-1-886529-17-5.

D. Kleinman. On an iterative technique for Riccati equation computations. IEEE Transactions on Automatic Control , 13(1):114-115, February 1968. ISSN 1558-2523. doi: 10.1109/TAC.1968.1098829. URL https://ieeexplore.ieee.org/abstract/document/1098829?casa_token= tKFm80m5gI4AAAAA:E3oaouksJy-5g-HuySnACwNdxyHpRDk8IbPTssB0B-PK0qZng_ -v9Fsk7qzcyYyYVudFUrI8 . Conference Name: IEEE Transactions on Automatic Control.

Martin L. Puterman and Shelby L. Brumelle. On the Convergence of Policy Iteration in Stationary Dynamic Programming. Mathematics of Operations Research , 4(1):60-69, 1979. ISSN 0364-765X. URL https://www.jstor.org/stable/3689239 . Publisher: INFORMS.

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research , 47: 253-279, June 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL https://www.jair. org/index.php/jair/article/view/10819 .

Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging Procedural Generation to Benchmark Reinforcement Learning, July 2020. URL http://arxiv.org/ abs/1912.01588 . arXiv:1912.01588 [cs, stat].

Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In In Proceedings of the Sixteenth International Conference on Machine Learning , pages 278-287. Morgan Kaufmann, 1999.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning, December 2013. URL http://arxiv.org/abs/1312.5602 . arXiv:1312.5602 [cs].

David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL Without Off-Policy Evaluation, December 2021. URL http://arxiv.org/abs/2106.08909 . arXiv:2106.08909 [cs, stat].

Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. In Richard S. Sutton, editor, Reinforcement Learning , The Springer International Series in Engineering and Computer Science, pages 5-32. Springer US, Boston, MA, 1992. ISBN 978-1-4615-3618-5. doi: 10.1007/978-1-4615-3618-5_2. URL https: //doi.org/10.1007/978-1-4615-3618-5_2 .

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-Based Batch Mode Reinforcement Learning. Journal of Machine Learning Research , 6(18):503-556, 2005. ISSN 1533-7928. URL http://jmlr.org/papers/v6/ernst05a.html .

Martin Riedmiller. Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo, editors, Machine Learning: ECML 2005 , Lecture Notes in Computer Science, pages 317-328, Berlin, Heidelberg, 2005. Springer. ISBN 978-3-540-31692-3. doi: 10.1007/11564096_32.

Irina Shevtsova. On the absolute constants in the Berry-Esseen type inequalities for identically distributed summands, November 2011. URL http://arxiv.org/abs/1111.6554 . arXiv:1111.6554 [math].

Eyal Even-Dar and Yishay Mansour. Learning Rates for Q-learning. Journal of Machine Learning Research , 5(Dec):1-25, 2003. ISSN ISSN 1533-7928. URL https://www.jmlr. org/papers/v5/evendar03a.html .

Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The Dependence of Effective Planning Horizon on Model Accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , AAMAS '15, pages 1181-1189, Richland, SC, May 2015. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-3413-6.

Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Miikkulainen. Frame skip is a powerful parameter for learning to play Atari. In Workshops at the twenty-ninth AAAI conference on artificial intelligence , 2015.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for Datasets, December 2021. URL http://arxiv.org/abs/1803.09010 . arXiv:1803.09010 [cs].

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, June 2016. URL http://arxiv.org/abs/1606. 01540 . arXiv:1606.01540 [cs].

Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl S. Muller, Jake A. Whritner, Luxin Zhang, Mary M. Hayhoe, and Dana H. Ballard. Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset, September 2019. URL http://arxiv.org/abs/1903.06754 . arXiv:1903.06754 [cs, stat].

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. The Journal of Machine Learning Research , 22(1):12348-12355, 2021. URL https://dl.acm.org/doi/ abs/10.5555/3546258.3546526 . Publisher: JMLRORG.

Antonin Raffin. RL Baselines3 Zoo, 2020. URL https://github.com/DLR-RM/ rl-baselines3-zoo . Publication Title: GitHub repository.

Appendix

A Proofs of main results

A.1 Proof of Theorem 2.1

Theorem 2.1. There is an RL algorithm which can solve any deterministic MDP with sample complexity N ≤ T ⌈ A T / 2 ⌉ . Conversely, for any RL algorithm and any values of T and A , there must be some deterministic MDP for which its sample complexity N ≥ T ( ⌈ A T / 2 ⌉ -1) .

Proof. Consider the following RL algorithm:

Since the MDP is deterministic, an RL algorithm only needs to find an optimal sequence of actions. Clearly, there is at least a 1 / 2 chance that some optimal sequence of actions is in the first ⌈ A T / 2 ⌉ elements of T . If this is the case, then EXHAUSTIVESEARCH will return an optimal policy corresponding to that optimal sequence. Since the number of environment timesteps taken by EXHAUSTIVESEARCH is equal to T ⌈ A T / 2 ⌉ , we have that N ≤ T ⌈ A T / 2 ⌉ .

For the converse, fix A and T along with any RL algorithm. Consider a set of states indexed by sequences of actions of length 0 to T :

Then, define a transition function

Now consider A T different MDPs which share the state space S and transition function f , differing only in their reward functions:

That is, each MDP has a single optimal sequence of actions that gives reward 1 on the final timestep; all other rewards are 0.

Let the RL algorithm in question take in some source of randomness z and output a policy π n t ( a | s ; z, M ) after n timesteps in MDP M . Now suppose by way of contradiction the sample complexity of the algorithm is less than T ( ⌈ A T / 2 ⌉-1) in all MDPs in M . By our definition of sample complexity, this means that

Clearly a policy can only be optimal in these MDPs if it is deterministic, so let τ ( z, M ) be the sequence of actions that π T ( ⌈ A T / 2 ⌉-2) t ( a | s ; z, M ) takes if it is optimal, and let it be any suboptimal sequence of actions if the policy is suboptimal. We can rewrite (1) as

Letting Unif ( A T ) define a uniform distribution over action sequences of length T , this implies

which means that there must be some particular z such that

Given that the RL algorithm is now deterministic due to the fixed z , we will prove that the LHS of (2) must be less than or equal to ⌈ A T / 2 ⌉ -1 . This can be shown via induction. For the first episode, the algorithm must take the same actions in every MDP since all MDPs give zero reward until the final action (so there is no way to distinguish them). After the first episode, only one MDP can be distinguished from the others: the one corresponding to the action sequence taken in the first episode, which has reward 1 instead of 0. Thus in the remaining A T -1 MDPs the algorithm must take the same actions in the second episode. Continuing this argument shows that after episode ⌈ A T / 2 ⌉ -2 , the algorithm must still be unable to distinguish between A T -( ⌈ A T / 2 ⌉ -2 ) = ⌊ A T / 2 ⌋ +2 of the MDPs, and so τ ( z, M ) must be the same for all MDPs in this set. Since all of these MDPs have different optimal action sequences, τ ( z, M ) can only be optimal in one of them. Thus τ ( z, M ) must be suboptimal in at least ⌊ A T / 2 ⌋ +1 MDPs, which means the LHS of (2) must be at most A T -( ⌊ A T / 2 ⌋ +1) = ⌈ A T / 2 ⌉ -1 .

Combining this with (2) gives ⌈ A T / 2 ⌉ -1 ≥ A T / 2 , which is a contradiction. Thus, the sample complexity of the RL algorithm must be at least T ( ⌈ A T / 2 ⌉ -1) . ■

A.2 Proof that k = T in the worst case

Lemma A.1. Let Q 1 = Q π rand , Q i +1 = QVI ( Q i ) for i = 2 , . . . , T , and Π( Q i ) be defined as in Section 4. Then for any horizon T and number of actions A ≥ 2 there is an MDP such that no policy in Π( Q i ) is optimal for i < T .

Proof. As in the proof of Theorem 2.1, define an MDP where every action sequence leads to a different state. Pick an arbitrary action sequence a 1: T , and let the reward of taking the final action in that sequence be 1 :

Now take some action a ′ 1 = a 1 , and let the reward for taking that action at the beginning of an episode be 3 / 4 :

Let the rewards for all other state-action pairs be 0. We will show by induction that

The left half of (3) is clearly true for i = 1 , since the random policy will take all action sequences following the t -1 -th optimal action with probability 1 /A T -t , and exactly one of those gives reward 1. The right half is also clear since following a ′ 1 gives immediate reward of 3/4 and then no reward afterwards.

For the inductive step, we begin with the left half of (3); assume it holds for some i . By the definition of Q-value iteration, we have for t < T

and for t = T ,

(i) is because only one action at timestep t +1 can have a Q-value greater than 0, since only one action sequence leads to reward after taking a 1 . For the right half of (3), we have

which is due to no reward being possible after taking a ′ 1 at the first timestep.

Given that (3) holds for i = 1 , . . . , T , it is easy to see that for i < T , any π ∈ Π( Q i ) will take a ′ 1 on the first timestep, since

.

This means that π will be suboptimal, since a 1 is the only optimal action at t = 1

■

̸

A.3 Proof of Lemma 5.3

Lemma 5.3. The sample complexity of GORP with optimal choices of k and m is T 2 A H .

Proof. Recall the definition of effective horizon:

Definition 5.2 (Effective horizon) . Given k ∈ [ T ] , let H k = k +log A m k , where m k is the minimum value of m needed for Algorithm 1 with parameter k to return the optimal policy with probability at least 1 / 2 , or ∞ if no value of m suffices. The effective horizon is H = min k H k .

Let k ∈ arg min k H k . Then by Definition 5.2, GORP (Algorithm 1) with parameters k, m k will converge to an optimal policy with probability at least 1 / 2 . Clearly, Algorithm 1 interacts with the environment for T iterations, each of which require evaluating A k action sequences with m k episodes of T timesteps each, for a total of

timesteps. Thus the sample complexity of GORP satisfies N GORP ≤ T 2 A H .

Now, suppose by way of contradiction that N GORP < T 2 A H . Then this must mean that there are parameters k ′ , m ′ such running GORP with these parameters converges to an optimal policy with probability at least 1 / 2 , and

By Definition 5.2, this means that m k ′ ≤ m ′ , and thus

2 H

which is clearly a contradiction. Thus, it must be that N GORP = T A . ■

A.4 Proof of Theorem 5.4

Theorem 5.4. Suppose that an MDP is k -QVI-solvable and that all rewards are nonnegative, i.e. R ( s, a ) ≥ 0 for all s, a . Let ∆ k denote the gap of the Q-function Q k as defined in Definition 5.1. Then

where S opt i is the set of states visited by some optimal policy at timestep i and V ∗ t ( s ) = max π V π t ( s ) is the optimal value function.

Proof. Let

We will show that GORP (Algorithm 1) converges to the optimal policy with probability at least 1 / 2 given parameters k and m . By Definition 5.2, this means the effective horizon must be at most k +log A m , which gives the bound in the theorem. More precisely, we will show that GORP converges to a policy in Π( Q k ) with probability at least 1 / 2 , which must be optimal because of the assumption that the MDP is k -QVI-solvable.

First, we will show the following relationship between the k -action Q 1 values and Q k :

We prove that (4) holds inductively. For k = 1 , (4) is obviously true. Supposing it holds for k , then

which shows (4) holds for k +1 . (i) holds due to the inductive hypothesis.

Recall that in Algorithm 1, we use ˆ Q i ( s i , a i : i + k -1 ) to denote the estimated Q-value of the k -action sequence a i : i + k -1 . Analogously to (4), define

to be the maximum estimated Q-value of any action sequence that starts with a i . We can rewrite line 7 of Algorithm 1 as

That is, the action that GORP selects for timestep i is chosen from those with the highest values of ˆ Q k i ( s i , a i ) . Suppose we can show that

holds for each i ∈ [ T ] . Then by a union bound,

This implies that π ∈ Π( Q k ) , which is the desired result.

It remains to show that (5) holds. We will actually prove the bound assuming that s i ∈ S opt i . This is still sufficient to imply (6) since one can inductively assume that previous actions are optimal. We can write (5) equivalently as

Let a ∗ i : i + k -1 ∈ arg max a i : i + k -1 ∈A k Q 1 i ( s i , a i : i + k -1 ) be chosen arbitrarily. By (4), this implies that a ∗ ∈ arg max a i ∈A Q k ( s i , a i ) . Then

Consider a single term of the sum in (8). By the definition of the gap and the fact that a i / ∈ arg max a i ∈A Q k i ( s i , a i ) , we know that

Combining this with (4) implies that

Now, consider the random variables ˆ Q i ( s i , a i , a i +1: k -1 ) and ˆ Q i ( s i , a ∗ i : i + k -1 ) . Let X j = ∑ T t = i γ t -i R ( s j t , a j t ) be the discounted reward from timestep i in the j th episode used to estimate ˆ Q i ( s i , a i , a i +1: i + k -1 ) ; let Y j be the equivalent for estimating ˆ Q i ( s i a ∗ i : i + k -1 ) . Then we can write

By the definition of the optimal value function V ∗ and the assumption that all rewards are nonnegative, we have that X j , Y j ∈ [0 , V ∗ i ( s i )] . We also know that the expectations of the X j and Y j are bounded:

The variance of a random variable with mean µ bounded on an interval [ α, β ] is at most ( β -µ )( µ -α ) . This means we can bound the variance of X j and Y j as well:

Now define

where Z j = Y j -X j . Since X j and Y j are independent,

Also define centered versions ¯ Z j = Z j -E [ Z j ] and ¯ Z = Z -E [ Z ] . By (9), E [ Z ] ≥ ∆ k i ( s i ) . Furthermore, since X j , Y j ∈ [0 , V ∗ i ( s i )] , we also know that | Z j | ≤ V ∗ i ( s i ) and | ¯ Z j | ≤ V ∗ i ( s i ) + E [ Z ] ≤ 2 V ∗ i ( s i ) .

We can now finally apply Bernstein's inequality to bound the probability of one term in (8):

Here, (i) is a direct application of Bernstein's inequality to the sum ¯ Z = 1 m ∑ m j =1 ¯ Z j . (ii) uses the bound on Var( Z j ) = Var( ¯ Z j ) and (iii) uses the fact that ∆ k i ( s i ) ≤ max a i Q k i ( s i , a i ) by definition of the gap ∆ k i . Finally, (iv) uses the definition of m ; since s i ∈ S opt i by assumption,

Applying (10) to (8) gives

which is the only thing left that is needed to complete the proof.

■

B Additional theoretical results concerning the effective horizon

In this appendix, we present some additional theoretical results concerning the effective horizon. First, we explore two algorithms-one in the style of policy gradient and one similar to fitted Q-iterationwhose sample complexities can also be bounded by a quantity related to the effective horizon. Then we show conditions under which the effective horizon is small, as well as information-theoretic lower bounds for the sample complexity of RL in terms of the effective horizon.

B.1 PG-GORP and FQI-GORP

The two algorithms we introduce in this section, PG-GORP and FQI-GORP, can be viewed as a bridge between GORP, which we use to define the effective horizon, and PPO and DQN, the deep RL algorithms whose performance we predict using the effective horizon in Section 6. They help to explain why the effective horizon is not only useful for understanding the performance of GORP, but also other RL algorithms.

We will actually give sample complexity bounds on PG-GORP and FQI-GORP in terms of the bound in Theorem 5.4, rather than the effective horizon itself. Supposing that the MDP is k -QVI-solvable, define

Our bounds will also depend on a quantity measuring how far the exploration policy π expl is from the uniformly random policy π rand :

∥ ∥ ∥ π rand π expl ∥ ∥ ∥ ∞ = 1 in the case when π expl = π rand , and increases as the smallest probabilities π expl assigns to actions becomes smaller.

We now introduce the first algorithm, PG-GORP.

Algorithm 2 The PG-GORP algorithm.

PG-GORP resembles the REINFORCE algorithm [48], which gave rise to other policy gradient-based algorithms like PPO. At each iteration, Algorithm 2 first samples a number of episodes following its current policy (line 4). Then, it computes a an approximate gradient over the policy parameters-in this case, just the action probabilities π i ( a | s i ) -via the so-called 'policy gradient theorem,' which states

Then, in line 8, Algorithm 2 applies optimization to π based on its estimate of the gradient. In this case, it optimizes until π i assigns all probability to only one action π i ( s i ) at s i .

Theorem B.1 (Sample complexity of PG-GORP) . Suppose that an MDP is 1-QVI-solvable and that all rewards are nonnegative. Then the sample complexity of PG-GORP is at most

Proof. Let

We will show that Algorithm 2 converges with probability at least 1 / 2 with this choice of parameter, giving the sample complexity bound in the theorem since the algorithm clearly samples T 2 m total timesteps from the environment.

Similarly to the proof of Theorem 5.4, we will prove this by showing that with probability at least 1 -1 / (2 T ) , at each iteration π i ( s i ) ∈ arg max a ∈A Q π expl i ( s i , a ) . This gives π ∈ Π( Q 1 ) , which by 1-QVI-solvability means π must be optimal.

Consider the i th iteration of Algorithm 2. Define for each a ∈ A and j ∈ [ m ] the random variable

First, can see that

since π i = π expl i until line 8.

Second, we have

Finally, using same the reasoning as in the proof of Theorem 2, we can bound the variance of X j ( a ) :

We now apply Bernstein's inequality to

for each a ∈ A . If a ∈ arg max a ∈A Q π expl i ( s i , a ) , we apply a lower tail bound:

If a / ∈ arg max a ∈A Q π expl i ( s i , a ) , we apply an identical upper tail bound:

These tail bounds hold simultaneously for all actions with probability at least 1 -1 2 T . Furthermore, assuming they hold and using the definition of the gap ∆ 1 i ( s i ) , it must be that

which is enough to show that π i ( s i ) ∈ arg max a ∈A Q π expl i ( s i , a ) with probability at least 1 -1 2 T and thus prove the theorem. ■

Now that we have seen that PG-GORP enjoys similar sample complexity bounds to GORP in the common case that an MDP is 1-QVI-solvable, we introduce FQI-GORP. FQI-GORP derives its name from fitted Q-iteration (FQI), which was originally proposed by Ernst et al. [49]. DQN was inspired by neural FQI [50], so FQI-GORP provides a natural connection to DQN.

Algorithm 3 The FQI-GORP algorithm.

1: procedure FQI-GORP( k, m )

2: for i = 1 , . . . , T do

3: Sample A k m episodes, following π for timesteps 1 to i -1 and then π expl .

5: for t = i + k -2 , . . . , i do

7: end for

8: π i ( s i ) ← arg max a ∈A ˆ Q k i ( s i , a ) .

9: end for

10: return π

11: end procedure

FQI-GORP iteratively constructs a series of Q-functions at each iteration by minimizing a meansquared temporal difference error loss, similar to FQI and DQN.

Theorem B.2 (Sample complexity of FQI-GORP) . Suppose that an MDP is k -QVI-solvable and that all rewards are nonnegative. Then the sample complexity of FQI-GORP is at most

Proof. Let

We will show that FQI-GORP with parameters k and m will return an optimal policy with probability at least 1 / 2 . Since FQI-GORP samples a number of timesteps from the environment equal to T 2 A k m , this will prove the bound in the theorem.

Consider the i th iteration of Algorithm 2. We will show that with probability at least 1 -1 / (2 T ) , for every s i + k -1 ∈ S reachable at timestep i + k -1 starting from s i , and for every action a i + k -1

To prove (11), it is first helpful to write an explicit formula fitted Q-value, assuming that the loss in line 4 of Algorithm 3 is minimized:

That is, the Q-value is a simple average of several reward-to-go values, each of which has expectation Q 1 i + k -1 ( s, a ) . The probability of reaching some reachable state-action pair ( s, a ) at timestep i + k -1

must be at least A -k ∥ ∥ ∥ π rand π expl ∥ ∥ ∥ -k ∞ . Thus, we can bound the sample size below via a concentration inequality for binomial random variables:

If the sample size is at least 12 log (2 TA ) max a i ∈A Q k i ( s i ,a i ) V ∗ i ( s i ) ∆ 1 i ( s i ) 2 , then Bernstein's inequality (as applied in the proofs of Theorems 5.4 and B.1) gives

Thus, taking a union bound, we have that the probability (11) does not hold for some reachable state-action pair at timestep i + k -1 must be at most 1 / (2 T ) , since there can be at most A k such pairs.

We will now show by induction that given that (11) holds for all reachable state-action pairs,

holds for t = i + k -1 , . . . , i for all reachable state-action pairs at t . The base case of t = i + k -1 is already taken care of, so we only need to show the inductive step.

Assume (12) holds for all reachable state-action pairs at t +1 . We can write an explicit formula for the fitted Q-values at timestep t , given that the loss function on line 6 of Algorithm 3 is minimized:

Given that (12) holds for each pair ( f ( s, a ) , a ′ ) , it is now easy to see that (12) must hold for t as well. By induction (12) must hold for t = i with probability at least 1 -1 / (2 T ) . Given that it holds, and by definition of the gap, this implies that π i ( s i ) ∈ arg max a ∈A Q k i ( s i , a ) . Thus with probability at least 1 / 2 , π ∈ Π( Q k ) . By the assumption that the MDP is k -QVI solvable, π must be optimal with probability at least 1 / 2 . ■

B.2 Goal MDPs

Now, we will prove bounds on the effective horizon in one particular class of MDPs: goal MDPs.

Definition B.3 (Goal MDP) . An MDP is considered a goal MDP if there is some set of goal states S goal which are absorbing, i.e., f ( s, a ) = s for every s ∈ S goal, and furthermore the reward function is of the form

That is, in a goal MDP reward is only received for reaching some set of goal states; the total episode reward is 1 if a goal state is reached and 0 otherwise. As an example, all of the Minigrid environments in BRIDGE are goal MDPs. We can show the following bound on the effective horizon in goal MDPs.

Theorem B.4 (The effective horizon in goal MDPs) . Suppose that π expl ( a | s ) > 0 for all s ∈ S and a ∈ A . Then any goal MDP is 1-QVI-solvable. Furthermore, suppose that there is some p > 0 such that, for all timesteps t ∈ [ T ] and all state-action pairs s t , a t at that timestep from which a goal state can be reached,

Then the effective horizon can be bounded as

Before we see the proof, note that Theorem B.4 agrees with our intuition that it should be harder to find an optimal policy for a goal MDP when it is less likely that the exploration policy reaches the goal, i.e., when p is smaller.

For instance, consider the MDP in Figure 2a. In this MDP, the minimum probability of reaching the goal with the random exploration policy after taking some action is exponentially small: p = 1 /A T -1 . Applying Theorem B.4 gives a bound of H ≤ T +log A log(2 T ) .

In contrast, consider the Minigrid gridworld in Figure 4 from BRIDGE. Here we can bound p ≈ 0 . 00137 , which gives H ≤ 1 + log A (1 /p ) + log A log(2 T ) = 1 + log 3 729 + log 3 log(200) ≤ 7 + 1 . 52 = 8 . 52 ≪ T = 100 . The techniques in Appendix C give a much tighter bound of H ≤ 1 . 64 .

Proof. We can assume that V ∗ 1 ( s 1 ) = 1 , since otherwise every trajectory in the MDP gives reward 0 and thus the effective horizon is trivially bounded. First, we will show that

This is enough to imply the MDP is 1-QVI-solvable, since any policy in Π( Q π expl ) must take only actions with Q π expl t ( s, a ) > 0 , which must be optimal according to (14).

̸

To show that (14) holds, first consider the ⇒ implication. Assume Q π expl t ( s, a ) > 0 and by way of contradiction suppose that Q ∗ t ( s, a ) = 1 , which means that Q ∗ t ( s, a ) = 0 . Clearly this cannot happen since this would imply Q π expl t ( s, a ) ≤ 0 . Now, consider the ⇐ direction. If Q ∗ t ( s, a ) = 1 , then there must be some sequence of actions starting with a which leads from s to a goal state. By assumption, π expl assigns positive probability to each action in this sequence. Thus Q π expl t ( s, a ) = P π expl ( s T ∈ S goal | s, a ) > 0 .

Next, we will prove the bound on H from (13). From (14), we can see that Algorithm 1 (GORP) will return an optimal policy for the MDP as long as at each iteration i it picks some action a i with Q π expl i ( s i , a i ) > 0 . In turn, this will happen as long as ˆ Q 1 i ( s i , a i ) > 0 for some such a i , since ˆ Q 1 i ( s i , a ) must be 0 for any suboptimal a .

Thus, we need only show that

holds for each i when m ≥ log(2 T ) /p . This will allow us to conclude via a union bound that Algorithm 1 will find an optimal policy with probability at least 1 / 2 in this case, which gives the desired bound on the effective horizon.

To show (15), consider iteration i of Algorithm 1 and let a i ∈ A such that Q π rand i ( s i , a i ) > 0 . We can assume that such an action exists as long as Algorithm 1 has succeeded in iterations previous to i . Let X j = ∑ T t = i γ t -i R ( s j t , a j t ) be the reward-to-go from the j th episode sampled to evaluate ˆ Q 1 i ( s i , a i ) . By the definition of a goal MDP, each X j ( a i ) ∈ { 0 , 1 } . Furthermore, since ˆ Q 1 i ( s i , a i ) =

1 ∑ m X j , then ˆ Q 1 ( s i , a ) > 0 as long as some X j = 1 . This implies

the bound previously proposed in (15) which we argued gives the desired bound. (i) uses the assumption that Q π expl i ( s i , a i ) = P ( X j = 1) ≥ p . ■

B.3 Lower bounds

Next, we will show that there are MDPs in which exponential dependence on the effective horizon is unavoidable. That is, in some cases there is an information-theoretic lower bound on the sample complexity of RL proportional to A H .

Theorem B.5. Fix T ≥ 1 , A ≥ 2 , and H ∈ [ T ] . Then for any RL algorithm, there is an MDP with A actions, horizon T , and effective horizon at most H for which the algorithm's sample complexity is at least

Note that this matches the upper bound on the sample complexity of GORP given in Lemma 5.3 up to a factor of roughly 2 H . When H = T , it exactly agrees with the lower bound given in Theorem 2.1.

Proof. The proof uses MDPs with the same state space and transition function as in Theorem 2.1. Define A T such MDPs which differ only in their reward functions:

That is, each MDP has a single optimal sequence of actions that gives reward 1 every H timesteps.

By the same argument as in the proof of Theorem 2.1, for any RL algorithm there must be some MDP in M such that after interacting with the environment for less than ⌈ A H / 2 ⌉ -1 episodes, the algorithm cannot with probability at least 1 / 2 identify the optimal actions for timesteps t = 1 to t = H . We can repeat this line of reasoning for timesteps t = H + 1 to t = 2 H , and so on for a total of ⌊ T/H ⌋ steps, to show that with less than ⌊ T/H ⌋ ( ⌈ A H / 2 ⌉ -1 ) episodes, there must be some MDP in M whose optimal action sequence cannot be identified with probability greater than (1 / 2) ⌊ T/H ⌋ ≤ 1 / 2 . Thus, the sample complexity of the RL algorithm on this MDP must be at least

which is the desired bound.

It only remains to be shown that the effective horizon of the MDPs in M is actually H . To see why, consider running Algorithm 1 with k = H and m = 1 . That is, at each iteration i , GORP will try all H -length action sequences followed by actions from π expl . Then, it will pick the action sequence with the highest empirical reward-to-go. From the definition of the MDPs in M , all action sequences starting with a suboptimal action must have empirical reward-to-go of 0. Furthermore, at least one H -length action sequence starting with an optimal action must get reward-to-go of at least 1. Thus, GORP will with probability 1 choose an optimal action at each timestep. This means that the effective horizon must be at most H +log A 1 = H . ■

C Tighter bounds on the effective horizon

In Theorem 5.4, we obtained bounds on the effective horizon and thus on the sample complexity of GORP. However, we find that the bounds given by Theorem 5.4 are often very loose compared to the empirical performance of GORP due to two factors. First, Theorem 5.4 requires considering the worst case of Q k t ( s, a ) V ∗ t ( s ) / ∆ k t ( s ) 2 over all optimal states. However, in many MDPs in our dataset, there are optimal states with extremely small gaps that in practice are almost never reached by GORP. When Theorem 5.4 is applied, these states make the sample complexity bounds very large despite GORP working well empirically. Second, Theorem 5.4 uses asymptotically tight techniques for bounding the sample complexity that can be quite loose for small sample sizes. Below, we describe the algorithm we use to provably bound the sample complexity of GORP (and thus the effective horizon) that gives much tighter results.

Consider the GORP algorithm as given in Algorithm 1. Let a t denote the random variable corresponding to the action ultimately chosen by the algorithm for timestep t . Let s t denote the state reached by actions a 1 , . . . , a t -1 . Denote by P m the probability measure given by running the algorithm with parameter m . We would like to bound the probability that the algorithm does not achieve the optimal return in the MDP. Let this event by denoted as

where A ∗ t ( s ) denotes the set of optimal actions in state s at timestep t , i.e.

It is straightforward to see from Algorithm 1 that it requires T 2 A k m timesteps of interaction the environment. Thus the sample complexity of the algorithm is

Clearly, we can upper bound the sample complexity using any m such that the probability of failure is bounded as P m ( E ) < 1 / 2 . To do so, for each value of k , we perform a binary search over values of m from 1 to 10 100 . For each possible m , we calculate an upper bound on P m ( E ) . If the upper bound is below 1 / 2 , we then search below m ; if it is greater, we search above m . When the search has converged to a relative precision of 1 / 100 , we output T 2 A k m as the sample complexity and H k = k +log A m as the effective horizon for that particular value of k .

C.1 Upper bounding P m ( E )

We upper bound the failure probability P m ( E ) recursively. Let O t denote the event that all actions taken before t have been optimal, i.e.,

To begin the recursion, note that at the final timestep,

We will use two recursion rules: one from next states to state-action pairs and one from state-action pairs to states. The first rule is

and the second (for t < T ) is

We apply these results recursively from t = T, . . . , 1 to finally obtain P ( E | O 1 , s 1 ) = P ( E ) .

The remaining difficulty is calculating P ( a t | s t ) . Recall from Algorithm 1 that a t is chosen as the first action of a k -action sequence

where ˆ Q t ( s t , a t : t + k -1 ) is the empirical mean return-to-go from m episodes starting in state s t and taking actions a t , . . . , a t + k -1 followed by actions sampled from the exploration policy. To simplify notation, let ⃗ a t denote a t : t + k -1 . We use various inequalities, described in detail below, to bound the probability that a particular k -action sequence is chosen:

Letting p ( ⃗ a t ) denote the actual probability an action sequence is chosen, we can rewrite (16) to

Given the bounds on p ( ⃗ a t ) (i.e., p and p ), we formulate a linear program with the bounds as constraints, plus the constraint that ∑ ⃗ a t ∈A k p ( ⃗ a t ) = 1 , with the objective of maximizing (18). Solving this gives an upper bound on P ( E | O t , s t ) . We can then propagate this bound recursively using (17) to bound P ( E ) .

Calculating p and p

We use up to four methods to bound p ( ⃗ a t ) , and pick the one which gives the lowest conditional failure probability after solving the linear program described above. As previously, let D t ( s t , ⃗ a t ) denote the distribution of the reward-to-go starting in s t and taking actions a t , . . . , a t + k -1 followed by actions sampled from the exploration policy. Thus we can write

Most of the following methods use the following decomposition:

for some sequence -∞ = q 0 ≤ q 1 ≤ . . . ≤ q N = ∞ . Here, (i) uses the fact that the random variables ˆ Q t ( s t , ⃗ a t ) across all action sequences ⃗ a t ∈ A k are independent. (ii) is a lower bound on the integral via a Riemann sum.

Alternatively, suppose we know that the CDF of ˆ Q t ( s t , ⃗ a t ) is bounded by

where F Z ( x ) is the continuous CDF of some random variable Z . Then

by a similar argument. We also have equivalent bounds in the other direction:

where the CDF of Z is greater than or equal to that of ˆ Q t ( s t , ⃗ a t ) . We use N = 100 when using these bounds.

Binomial bounds In the case where for all ⃗ a t ∈ A k , the distribution D t ( s t , ⃗ a t ) has mass on only 0 and some other value C , we have

This case occurs in many environments that are goal-based, i.e. where the agent gets reward only for reaching some goal and then the episode ends. We find that it significantly improves the sample complexity bounds in those environments. Without loss of generality, we may assume C = 1 . We then apply (19) and (21) to obtain p ( ⃗ a t ) and p ( ⃗ a t ) . We let q 0 , . . . , q N be set such that

using either the exact inverse CDF of the binomial distribution for small m or a normal approximation for large m . Then, we can calculate all terms in the bounds (19) and (21) using the binomial CDF. Since the CDF is more expensive to calculate for larger m , we only use the binomial-based bounds when m ≤ 10 6 and k = 1 .

Berry-Esseen bounds For this type of bound, we calculate the variance σ 2 = Var( D t ( s t , ⃗ a t )) and third absolute moment

for each ⃗ a t ∈ A k . Then by the Berry-Esseen theorem [51], we have that

The resulting upper and lower bounds on the CDFs of ˆ Q t ( s t , ⃗ a t ) for all ⃗ a t ∈ A k can be used in (20) and (22) to calculate p ( ⃗ a t ) and p ( ⃗ a t ) . Since this bound requires order N evaluations of the normal CDF and inverse CDF, which is somewhat expensive, we only use it when A k ≤ 100 .

Bernstein bounds Similarly to the Berry-Esseen bounds, Bernstein's inequality can be used to bound the CDF of ˆ Q t ( s t , ⃗ a t ) , and is sometimes superior to Berry-Esseen for large m due to giving tail bounds that decay exponentially rather than quadratically. In particular, suppose D t ( s t , ⃗ a t ) is supported on the interval [ α, β ] ; we can compute these bounds via value iteration. Then

Bernstein's inequality gives the following bounds on the CDF of ˆ Q t ( s t , ⃗ a t ) :

Similarly to the Berry-Esseen bounds, we use these in (20) and (22) to calculate p ( ⃗ a t ) and p ( ⃗ a t ) when A k ≤ 100 .

Bennett bounds The final method we use to calculate p ( ⃗ a t ) and p ( ⃗ a t ) is computationally cheaper than the others, so we can use it no matter the size of A k . As in the Bernstein bounds, we calculate

the interval support and bound the variance of each D t ( s t , ⃗ a t ) . We then let u be the arithematic mean of the highest action sequence Q value and the second-highest, i.e.

Then, we for each action sequence with less-than-highest Q-values, i.e. for each ⃗ a t / ∈ arg max ⃗ a t ∈A k Q t ( s t , ⃗ a t ) , we calculate the upper bound

Each of the tail bounds in (23) can be upper bounded using Bennett's inequality to obtain p ( ⃗ a t ) . We let p ( ⃗ a t ) = 1 if ⃗ a t ∈ arg max ⃗ a t ∈A k Q t ( s t , ⃗ a t ) and we set p ( ⃗ a t ) = 0 for all ⃗ a t ∈ A k .

D Previously proposed sample complexity bounds

In this appendix, we give proofs of sample complexity bounds based on properties previously proposed in the RL theory literature. We also compare these bounds to our effective horizon-based bounds in examples that showcase their failure modes.

D.1 Upper confidence bounds (UCB) and strategic exploration

A central problem in RL is exploration: how to efficiently reach enough states in an MDP in order to identify the optimal policy. One common way of approaching exploration is with upper-confidence bounds (UCB), which originated in the bandit literature. Algorithms using UCBs generally choose actions based on the current best estimate of that action's value plus an exploration 'bonus' that incentivizes exploration of little-seen states. Examples in the RL literature include Kakade [18], Azar et al. [4], Jiang et al. [19], Jin et al. [5, 20], Du et al. [21], Jin et al. [22]. Generally, these UCB algorithms achieve minimax sample complexity in terms of some measure of the 'size' of the state space-either the number of states S [4], or quantities like the Bellman-Eluder dimension [22].

A very simple UCB-type algorithm, R-MAX [23, 18], achieves a sample complexity bounded by SAT in deterministic, tabular MDPs. It depends on knowing the maximum reward at any state-action pair in the MDP, R max = max ( s,a ) ∈S×A R ( s, a ) . It also requires access to a computational oracle that can calculate an optimal policy for any transition function f and reward function R , for instance via value iteration.

1: procedure R-MAX

2: initialize ˆ f ( s, a ) ← s for all ( s, a ) ∈ S × A

3: initialize ˆ R ( s, a ) ← R max for all ( s, a ) ∈ S × A

4: for j = 1 , . . . , SA do

5: for t = 1 , . . . , T do

6: take an action a in the current state s according to the optimal policy for ˆ f and ˆ R

7: ˆ R ( s, a ) ← the observed reward

8: ˆ f ( s, a ) ← the observed next state s ′

9: end for

10: end for

11: return an optimal policy for ˆ f and ˆ R

12: end procedure

This is the version of R-MAX for deterministic MDPs; there is a more complex version for stochastic MDPs. The exploration bonuses in R-MAX are simply the initialization of ˆ R to the maximum possible reward. This ensures that an optimal value function computed from ˆ f and ˆ R is always an upper bound on the true optimal value function. The following result shows that R-MAX finds the optimal policy, and thus proves that its sample complexity is at most SAT .

Theorem D.1. R-MAX returns an optimal policy.

Proof. Let ˆ V ∗ and ˆ Q ∗ be the optimal value function and Q-function under ˆ f and ˆ R . We will begin by showing that at any point in the algorithm, V ∗ t ( s ) ≤ ˆ V ∗ t ( s ) and Q ∗ t ( s, a ) ≤ ˆ Q ∗ t ( s, a ) for all ( t, s, a ) ∈ [ T ] ×S × A .

The proof is via induction on t from T to 1 . To begin, clearly ˆ Q ∗ T ( s, a ) ∈ { Q ∗ T ( s, a ) , R max } , so the bound holds for ˆ Q ∗ T . Now, suppose that at some t ∈ [ T ] and all ( s, a ) ∈ S ×A , Q ∗ t ( s, a ) ≤ ˆ Q ∗ t ( s, a ) . Then

so the bound holds for ˆ V ∗ t as well. Finally, say that at some t ∈ [ T -1] and for all s ∈ S , V ∗ t +1 ( s ) ≤ ˆ V ∗ t +1 ( s ) . To show this implies Q ∗ t ( s, a ) ≤ ˆ Q ∗ t ( s, a ) for any ( s, a ) ∈ S × A , consider two cases. First, if ( s, a ) has been seen by the algorithm, then

Otherwise, if ( s, a ) has not been seen, then ˆ R ( s, a ) = R max and ˆ f ( s, a ) = s . In this case,

By induction we see that V ∗ t ( s ) ≤ ˆ V ∗ t ( s ) and Q ∗ t ( s, a ) ≤ ˆ Q ∗ t ( s, a ) for all ( t, s, a ) ∈ [ T ] ×S × A .

Next, we will prove that any optimal policy π for ˆ f and ˆ R must either (a) be optimal for f and R or (b) reach a previously unseen state-action pair. In particular, we will show that if V π 1 ( s 1 ) < ˆ V ∗ 1 ( s 1 ) , then π must reach a previously unseen state-action pair. Otherwise, V π 1 ( s 1 ) ≥ ˆ V ∗ 1 ( s 1 ) ≥ V ∗ 1 ( s 1 ) , showing that π is optimal for f and R .

We will again work inductively starting from the last timestep. First, suppose that Q π T ( s, a ) < ˆ Q ∗ T ( s, a ) for some ( s, a ) ∈ S × A . This is equivalent to R ( s, a ) < ˆ R ( s, a ) , which clearly means that ( s, a ) cannot have been explored.

Now, suppose that at some timestep t ∈ [ T ] , we know that for any ( s, a ) ∈ S × A , Q π t ( s, a ) < ˆ Q ∗ t ( s, a ) implies that π must explore some new state-action pair at or after timestep t starting in ( s, a ) . If V π t ( s ) < ˆ V ∗ t ( s ) for some s ∈ S , then

By assumption this means π must explore some new state-action pair at or after timestep t , since it takes an action a satisfying Q π t ( s, a ) < ˆ Q ∗ t ( s, a ) .

Finally, suppose that for some t ∈ [ T ] , we know that for any s ∈ S , V π t +1 ( s ) < ˆ V ∗ t +1 ( s ) implies that π must explore some new state-action pair at or after timestep t +1 starting in s . Suppose for some ( s, a ) ∈ S × A that Q π t ( s, a ) < ˆ Q ∗ t ( s, a ) . Then

which implies that either R ( s, a ) < ˆ R ( s, a ) or V π t +1 ( f ( s, a )) < V π t +1 ( ˆ f ( s, a )) . In the first case, ( s, a ) must be unexplored. In the second case, either ( s, a ) is unexplored, or

In any of these cases, π must explore a new state-action pair either in this timestep or in the future starting from ( s, a ) .

Inductively, this shows that V π 1 ( s 1 ) < ˆ V ∗ 1 ( s 1 ) implies that π must reach a previously unseen state-action pair. We will show that this property implies that R-MAX must return an optimal policy.

In particular, note that after the j th loop iteration in R-MAX, it must either have an optimal policy or have explored at least j of the state-action pairs in the MDP. This is a simple consequence of the above property: at each iteration, either the policy used by R-MAX must be optimal or it must explore at least one additional state-action pair. This means that after all the SA loop iterations, R-MAX will either have an optimal policy or have explored all the state-action pairs, in which case it will also have an optimal policy. ■

D.2 Covering length

The covering length of an MDP was originally proposed by Even-Dar and Mansour [52] and later used by Liu and Brunskill [7] to prove sample complexity bounds on RL algorithms which use random exploration. Liu and Brunskill [7] show various bounds on the covering length using graph-theoretic notions. While they focus on discounted infinite-horizon MDPs, we use a version of covering length adapted to finite-horizon MDPs, similar to that used by Dann et al. [6].

Definition D.2 (Covering length) . The covering length L of an MDP under an exploration policy π expl is the number of episodes needed until all state-action pairs have been visited with probability at least 1 / 2 .

One can easily show sample complexity bounds based on the covering length.

Theorem D.3 (Covering length sample complexity bound) . There is an RL algorithm which can solve any MDP with sample complexity TL given an exploration policy π expl , where L is the covering length of π expl .

Proof. Consider the following RL algorithm:

1: procedure COVERINGLENGTHRL( π expl , L )

2: collect a dataset of L episodes, sampling actions according to π expl

3: record ˆ R ( s, a ) and ˆ f ( s, a ) for all state-action pairs seen in the dataset

4: define ˆ R ( s, a ) and ˆ f ( s, a ) arbitrarily for state-action pairs not seen in the dataset

5: run value iteration using ˆ R and ˆ f to obtain a policy π

6: return π

7: end procedure

By the definition of covering length, with probability at least 1 / 2 the algorithm should produce ˆ R ( s, a ) = R ( s, a ) and ˆ f ( s, a ) = f ( s, a ) for all ( s, a ) ∈ S × A . In this case, π will be an optimal policy. Thus, COVERINGLENGTHRL returns an optimal policy with probability at least 1 / 2 while interacting with the environment for TL timesteps. This means the sample complexity of COVERINGLENGTHRL is at most TL . ■

To bound the covering length for MDPs in the BRIDGE dataset, we make use of the following result.

Lemma D.4 (Bounds on the covering length) . Define the occupancy measure µ of π expl as

Suppose that for every state-action pair ( s, a ) , there is some timestep t when µ t ( s, a ) > 0 . Then

We calculate µ min = min ( s,a ) ∈S×A max t ∈ [ T ] µ t ( s, a ) for each MDP in BRIDGE and use the upper bound from Lemma D.4 to obtain a sample complexity bound of T log(2 SAT ) /µ min. Since ∑ t ∈ [ T ] µ t ( s, a ) ≤ T max t ∈ [ T ] µ t ( s, a ) the upper and lower bounds in Lemma D.4 agree up to a factor of T log(2 SA ) / log(2) , so this is reasonably tight. In fact, in 122 of the 155 MDPs in BRIDGE, min ( s,a ) ∈S×A ∑ t ∈ [ T ] µ t ( s, a ) = min ( s,a ) ∈S×A max t ∈ [ T ] µ t ( s, a ) , making the upper and lower bounds tight up to only logarithmic factors.

Proof. Let P m be a probability measure corresponding to sampling m episodes following π expl . Let

E j t ( s, a ) denote the event that the j th episode has ( s t , a t ) = ( s, a ) . Let E t ( s, a ) be the event that E j t ( s, a ) occurs in at least one one of those episodes, i.e.

Finally, let C be the event defined by

That is, C is when every state-action pair has been seen at some timestep in at least one episode. We can thus equivalently define L = min { m | P m ( C ) ≥ 1 / 2 } .

We will now start by showing the upper bound of L ≤ ⌈ log(2 SA ) /p ⌉ . Let m = ⌈ log(2 SA ) /p ⌉ . We can write

which proves the upper bound.

To show the lower bound, take any ( s, a ) ∈ arg min ( s,a ) ∈S×A ∑ t ∈ [ T ] µ t ( s, a ) . First, suppose that ∑ t ∈ [ T ] µ t ( s, a ) ≥ log(2) . Then the lower bound in the lemma must be at most 1 / 2 , which is clearly true. Thus, we may subsequently assume ∑ t ∈ [ T ] µ t ( s, a ) < log(2) .

where (i) uses the fact that 1 -x ≥ exp( -2 x ) for x ∈ [0 , log(2)] . This shows that with probability greater than 1 / 2 , not all state-action pairs will be seen in m episodes, and thus establishes that L > m , which is the desired bound. ■

D.3 Effective planning window (EPW)

Perhaps the closest existing concepts to our effective horizon are various notions of 'effective planning window.' This generally refers to tree-based planning algorithms which only consider action sequences of some length W from the current state, rather than considering action sequences all the way until the end of the MDP. For instance, Kearns et al. [24] show that in discounted MDPs, one need only plan to some ϵ -horizon to obtain an ϵ -optimal policy. Jiang et al. [53] build on this and show that one may want to use a different discount factor for planning than the one that is used for evaluation. Malik et al. [8] also introduce a notion of effective planning window based on the number of timesteps one must look ahead in an MDP to avoid terminal states.

We do not directly apply any of these previous results to our setting. Since we are concerned primarily with finite-horizon undiscounted MDPs, it does not make much sense to apply a discount factor as in Kearns et al. [24] and Jiang et al. [53]. We find that the assumptions in Malik et al. [8] are quite unusual and do not really hold in any of the environments in BRIDGE. In particular, the analysis in Malik et al. [8] requires that a trajectory through an MDP is either optimal or ends early in a terminal state.

Instead of directly using these results, we define a notion of effective planning window based on the length of action sequences one must consider in an MDP while ignoring any rewards after the sequence.

Definition D.5 (Effective planning window) . Define Q 1 t ( s, a ) = R ( s, a ) for all ( t, s, a ) ∈ [ T ] ×S×A and let Q i = QVI ( Q i -1 ) for i = 2 , . . . , T . The effective planning window of an MDP is the minimum W ∈ [ T ] such that all policies in Π( Q W ) are optimal.

Note that the effective planning window bears significant similarity to the k -QVI-solvability property from Definition 5.1. However, Q 1 is defined as equal to the reward function for the EPW, while in Definition 5.1 it is equal to Q π expl .

The EPW also results in sample complexity bounds of T 2 A W very similar to those of T 2 A H for the effective horizon. However, we find empirically that H < W in 82% of the MDPs in BRIDGE, making the effective horizon-based bounds generally tighter.

Theorem D.6. For any MDP with effective planning window W , there is an RL algorithm whose sample complexity is at most T 2 A W .

Proof. We will use the following algorithm:

1: procedure PLANOVERWINDOW( W ) 2: for i = 1 , . . . , T do 3: for a i : i + W -1 ∈ A k do 4: Sample an episode following π 1 , . . . , π i -1 , then actions a i : i + W -1 , and then arbitrary actions. 5: ˆ R i ( s i , a i : i + W -1 ) ← ∑ i + W -1 t = i R ( s t , a t ) . 6: end for 7: π i ( s i ) ← arg max a i ∈A max a i +1: i + W -1 ∈A k -1 ˆ R i ( s i , a i : i + W -1 ) . 8: end for 9: return π

10: end procedure

Again, this algorithm is quite similar to GORP (Algorithm 1) except that it only samples a single episode per action sequence and it ignores rewards beyond the planning window. Clearly, PLANOVERWINDOW will take T 2 A W steps in the environment. Thus, to bound the sample complexity, we only need to show it returns an optimal policy with probability at least 1 / 2 .

To prove this, we will show that

Based on line 7 of the algorithm, this is enough to show that π ∈ Π( Q W ) , and thus that π must be optimal by Definition D.5.

To prove (24), we will first show by induction that Q j t ( s t , a t ) = max a t +1: t + j -1 ∈A j -1 ∑ t + j -1 t ′ = t R ( s t ′ , a t ′ ) , where s t ′ +1 = f ( s t ′ , a t ′ ) for t ′ = i, . . . , i + W -2 . The base case when j = 1 is by definition: Q 1 t ( s t , a t ) = R ( s t , a t ) . For the inductive step, assume the formula holds for j and note that

Next, note that by the way ˆ R is constructed, ˆ R ( s i , a i : i + W -1 ) = ∑ i + W -1 t = i R ( s t , a t ) , where s t +1 = f ( s t , a t ) for t = i, . . . , i + W -2 . Thus combining this with the formula proved by induction, (24) clearly holds and the proof is complete. ■

D.4 Other bounds

One other work that derives sample complexity bounds for RL with random exploration is Dann et al. [6]. They define an algorithm which maintains at all times a current best policy π , and acts according to this policy but with some exploration noise, e.g., via an ϵ -greedy policy expl ϵ ( π ) . They introduce the notion of a 'myopic exploration gap,' which is defined as

This gap is shown to generalize the notion of covering length as well as various others from the literature. However, we find that it is not so useful in many environments in BRIDGE.

The problem we find is illustrated in the MDP below:

s

1

s

2

R

= 1

. . .

0

s

2

0

. . .

R

= 1

. . .

0

. . .

0

. . .

0

s

T

R

= 1

s

T

0

s

T

0

s

T

T

-

1

In this MDP, one need simply follow the actions which give rewards of 1 to achieve the optimal return of T . One can show H = W = 1 in this MDP, which give identical sample complexities of T 2 A .

However, the difficulty with the myopic exploration gap is that the analysis in Dann et al. [6] cannot rule out the policy π which takes all actions to the right (achieving return T -1 ) from being chosen at some point while running their RL algorithm. If this happens, then the only way to find a better policy is to completely switch to the policy π ′ which takes all left actions (achieving return T ). This implies that α is maximized when c = A T , leading to α = (1 /A ) T/ 2 . Since the sample complexity bounds in Dann et al. [6] are O (1 /α 2 ) , this gives a bound proportional to A T , which is no better than the worst case.

Thus, whenever there are 'distracting' rewards, no matter how distant, as in this case, the theory from Dann et al. [6] cannot give good sample complexity bounds. There is also no easy way calculate α directly for an arbitrary environment. For these reasons, we do not include their bounds in our experiments.

E Dataset details

In this appendix, we give a detailed explanation of how we chose the MDPs in BRIDGE and how we constructed their tabular representations. See Figure 5 for an overview of BRIDGE.

E.1 Environments

We limited the horizon of environments for BRIDGE to some T ∈ { 10 , 15 , 20 , 30 , 50 , 70 , 100 , 200 } , depending on the environment, in order to avoid the state space becoming intractably large. We use subscripts to denote the horizon to which an environment is limited. For instance, PONG 50 refers to the Atari game Pong limited to 50 timesteps.

Frameskip We carefully used frameskip for each environment. Frameskip is a standard practice in Atari [54] in which each action taken in the environment is played for a certain number of frames; the agent only receives the next state after all these frames have completed. We use unusually high frameskips in order to capture episodes with longer wall-clock times in a small number of environment timesteps. The frameskip values we use are listed in

67 Atari

games

55 Procgen

levels

33 Minigrid

gridworlds

The BRIDGE dataset

10

2

10

5

10

8

Num. states

S

0

25

Num. MDPs

5 1015

Num. actions

A

0

50

10 100 200

Horizon

T

0

50

Table 4. For most Atari games, we use a frameskip of 30, corresponding to taking 2 actions per second. The frameskips for Procgen environments vary; we chose ones that tended to align with how long it took the agent to perform various low-level tasks in the environment like moving one space. We did not use frameskip for Minigrid.

Table 4: Frameskip values used for the MDPs in BRIDGE.

Atari games For each of the 57 Atari games in the Arcade Learning Environment (ALE) benchmark [43], we attempted to construct tabular representations for each horizon T ∈ { 10 , 20 , 30 , 50 , 70 , 100 , 200 } . However, we excluded environments once the state space exceeded 100 million states. We kept multiple horizon-limited versions of games, i.e., BRIDGE contains PONG 10 , PONG 20 , PONG 30 , etc. For some games, even T = 10 produced too many states, so we did not include these at all. We use the minimal action sets for each MDP rather than all 18 possible Atari actions.

We made one exception to these procedures for Montezuma's Revenge, as it is an environment well-known for being difficult to explore, so we wanted to make sure to include it in BRIDGE. We found that with T = 10 , there was not enough time to get any reward, and with T = 20 there were too many states. We found that using T = 15 and a frameskip of 24 did allow an agent to receive reward, so we used this version in BRIDGE.

We made a couple other modifications to the standard Atari setup. First, we limited agents to one life: as soon as a life is lost, the episode ends. Second, in SKIING 10 , we added an additional 200 frames of NOOP actions after the 10 timesteps ( = 300 frames) in each episode. This is necessary to correctly reflect the reward incentives in SKIING with longer horizons.

Finally, we scaled the rewards for many Atari games to make the reward scale more uniform across different games. Often, when deep RL is applied to Atari, rewards are clipped to [ -1 , 1] to avoid instability. However, the MDP with clipped rewards may have a different optimal policy than the unclipped MDP. Thus, instead of clipping, we use scaling. We generally choose the scale factor based on the multiples of points received in the game: for instance, in ATLANTIS, rewards are always received in multiples of 100 so we scale by 1 / 100 . Table 5 lists the reward scaling factors for all games where we apply scaling.

Table 5: Factors by which Atari games rewards are scaled by in BRIDGE, for those where we apply reward scaling.

Procgen levels The Procgen benchmark [44] consists of 16 games. For each game, one can generate an arbitrary number of random levels, each of which is identified by a seed. Furthermore, each game has an 'easy' and 'hard' difficulty, each with different levels, and some have an additional 'exploration' level which presents a particularly difficult exploration challenge.

While the benchmark is designed to measure generalization of RL agents trained on some number of levels to unseen levels, we use each level as a separate MDP. For each game, we attempted to construct an MDP for the easy levels with seeds 0 , 1 , and 2 , the hard level with seed 0 , and the exploration level if it exists for that environment. We denote MAZE E1 30 to be the easy level with seed 1 for the MAZE game limited to T = 30 timesteps; MAZE H0 30 is the analogous hard level with seed 0 and MAZE EX 30 is the exploration level.

We generally increased the horizon for each game to the highest value in { 10 , 20 , 30 , 40 , 50 , 70 , 100 , 200 } before the number of states was greater than 100 million. The horizon values we ultimately chose can be seen in the table in Appendix G.3.

Minigrid gridworlds Minigrid [14] is an extensible framework for building gridworlds. We considered all the pre-built gridworlds included in Minigrid for inclusion in BRIDGE except for those requiring natural language observations for specifying the task to be completed. We also excluded gridworlds with more than 1 million states, since for technical reasons we were unable to parallelize the construction of tabular MDPs for Minigrid. For gridworlds with randomized start states, we chose the start state with seed 0. We use T = 100 for all the gridworlds.

E.2 Constructing tabular representations

For each of the environments described above, we wrote a program to compute a full tabular representation of the transition function f and reward function R . Our program uses a search procedure to iteratively explore every state-action pair. We keep a queue of states that need to be explored, which at first is just the initial state. In parallel, a number of worker threads take states from this queue. After popping a state, a worker thread sets the environment to that state and then takes a previously unexplored action, storing the resulting next state and reward. If there are still unexplored actions in the current state, it adds it back to the queue. If the next state is not terminal and has not had all its actions, it can continue this process.

While the search procedure for exhaustively enumerating states is conceptually simple, we experienced difficulties implementing it efficiently due to the massive scale of some of the MDPs in BRIDGE. For instance, the full state of the Atari simulator used in the ALE is about 10-12 KB of data. Storing 100 million states by themselves would thus require over one terabye of memory! We avoided this problem by aggressively compressing state data using dictionary compression. Other challenges included efficiently parallelizing the data structures we used to store the queue of states, the transition function, and the reward function. Our final implementation is able to explore more than 20,000 state-action pairs per second in PONG while running on 64 cores.

Once we have enumerated all states and actions that can be reached in the given horizon, we also apply a consolidation step to reduce the number of states. Often, the internal representation of states in the Atari and Procgen environments includes extra or superfluous data, which leads to duplicate states in our tabular representation. We repeatedly consolidate states that (a) have the same screen, (b) have the same rewards for each action, and (c) lead to the same next states for each action. When no more states can be consolidated, we store the resulting transition and reward functions.

We excluded any MDPs for which every sequence of actions results in the same total reward, since these are uninteresting from an RL perspective.

E.3 Reward shaping

For each Minigrid environment, we constructed one or more versions with shaped rewards for our experiments on the effects of reward shaping. We used three potential functions for shaping:

Φ dist ( s ) : the negative distance from the state to the nearest goal. Distance is measured as the minimum number of moves needed to reach the goal, assuming there are no obstacles in the way.

Φ doors ( s ) : the number of doors that are open.

Φ pickup ( s ) : the number of objects that have been picked up at least once.

For one or more potential functions Φ , we augment each reward R ( s, a ) with Φ( f ( s, a )) -Φ( s ) . The potential functions are chosen to incentivize useful behavior in the environments: moving towards goals, picking up objects like keys that could be helpful, and opening doors to reach more parts of the gridworld.

For each Minigrid MDP, we use all potential functions that apply to that MDP. For instance, if an MDP does not have any doors, we do not use Φ doors. We also apply the combination of Φ doors and Φ pickup if both are applicable to an MDP.

When analysing the reward shaping results, we only include MDPs for which PPO/DQN converged on both the unshaped and shaped versions.

E.4 Datasheet for BRIDGE

We provide a datasheet, as proposed by Gebru et al. [55], for the BRIDGE dataset.

E.4.1 Motivation

For what purpose was the dataset created? We have described the purpose extensively in the paper: we aim to bridge the theory-practice gap in RL. BRIDGE allows this by providing tabular representations of popular deep RL benchmarks such that instance-dependent bounds can be

calculated and compared to empirical RL performance.

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Not specified for the double-blind reviewing process.

Who funded the creation of the dataset? Also not specified for the double-blind reviewing process.

Any other comments? No.

E.4.2 Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The instances are Markov Decision Processes (MDPs).

How many instances are there in total (of each type, if appropriate)? There are 155 MDPs in BRIDGE. They include 67 MDPs based on Atari games from the Arcade Learning Environment [43], 55 MDPs based on Procgen games [44], and 33 MDPs based on MiniGrid gridworlds [14].

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? The MDPs in BRIDGE are based on a small subset of the many environments that are used for empirically evaluating RL algorithms. We aimed to cover a range of the most popular environments. To make our analysis possible, we excluded environments that were not deterministic or did not have discrete action spaces. We also reduced the horizon of many of the environments to make it tractable to compute their tabular representations.

What data does each instance consist of? For each MDP, we provide the following data:

A transition function and a reward function, which are represented as a matrix with an entry for each state-action pair in the MDP.

A corresponding gym environment [56] that can be used to train policies for the MDP with various RL algorithms.

Properties of the MDP that are calculated from its tabular representation, including the effective planning window, bounds on the effective horizon, bounds on the covering length, etc.

Results of running RL algorithms (PPO, DQN, and GORP) on the MDP. This includes the empirical sample complexity as well as various metrics logged during training.

For MiniGrid MDPs, there are additional versions of the MDP with shaped reward functions (see Appendix E.3) which also include all of the above data.

For Atari and Procgen MDPs, there is additionally a non-uniform exploration policy (see Appendix F.2). For Atari games, this is trained via behavior cloning from the Atari-HEAD [57] dataset; for Procgen games, it is trained on other Procgen levels. We include the above data recalculated using the non-uniform exploration policy in place of the uniformly random exploration policy.

Is there a label or target associated with each instance? In this paper, we aim to bound and/or estimate the empirical sample complexity of RL algorithms, so these could be considered targets for each instance.

Is any information missing from individual instances? There is no information missing.

Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? No.

Are there recommended data splits (e.g., training, development/validation, testing)? No.

Are there any errors, sources of noise, or redundancies in the dataset? We do not believe there are errors or sources of noise in the dataset. The tabular representations of the MDPs have been carefully tested for correspondence with the environments they are based on. There is some redundancy, as many Atari games are represented more than once with varying horizons.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is mostly self-contained, except that the gym

environments rely on external libraries. There are archival versions of these available through package managers like PyPI.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? No.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.

E.4.3 Collection process

How was the data associated with each instance acquired? The data was collected using open-source implementations of each environment.

What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? As described in Appendix E.2, we developed a software tool to construct the tabular representations of the MDPs in BRIDGE. We validated the correctness of the tabular MDPs through extensive testing to ensure they corresponded exactly with the gym implementations of the environments.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? The MDPs in BRIDGE were selected from three collections of commonly used RL environments: the Arcade Learning Environment, ProcGen, and MiniGrid. We chose these three collections to represent a broad set of deterministic environments with discrete action spaces. Within each collection, the environments were further filtered based on the criteria described in Appendix E.1.

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Only the authors were involved in the data collection process.

Over what timeframe was the data collected? The dataset was assembled between February 2022 and January 2023. The RL environments from which the MDPs in BRIDGE were constructed were created prior to this; see the cited works for each collection of environments for more details.

Were any ethical review processes conducted (e.g., by an institutional review board)? No.

E.4.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Yes, various preprocessing and analysis was done. See Appendix E.2 for details.

Was the 'raw' data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? Yes, this is included with the dataset.

Is the software that was used to preprocess/clean/label the data available? Yes, this is available with the rest of our code.

Any other comments? No.

E.4.5 Uses

Has the dataset been used for any tasks already? The dataset has thus far only been used to validate our theory of the effective horizon in this paper.

Is there a repository that links to any or all papers or systems that use the dataset? There is not. However, we will require that any uses of the dataset cite this paper, allowing one to use tools like Semantic Scholar or Google Scholar to find other papers which use the BRIDGE dataset.

What (other) tasks could the dataset be used for? We hope that the BRIDGE dataset is used for further efforts to bridge the theory-practice gap in RL. The dataset could be used to identify other properties or assumptions that hold in common environments, or to calculate instance-dependent sample complexity bounds and compare them to the empirical sample complexity of RL algorithms.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? As we have already mentioned, BRIDGE is restricted to deterministic MDPs with discrete action spaces and relatively short horizons. This could mean that analyses of the dataset like ours do not generalize to the broader space of RL environments that may have continuous action spaces, stochastic transitions, and/or long horizons. We have included some experiments, like those in Appendix F.1, to show that our theory of the effective horizon generalizes beyond the MDPs in BRIDGE. We encourage others to do the same and we hope to address some of these limitations in the future with extensions to BRIDGE.

Are there tasks for which the dataset should not be used? We do not foresee any particular tasks for which the dataset should not be used.

Any other comments? No.

E.4.6 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, we will distribute the dataset publicly.

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? We are still finalizing the method through which the dataset will be distributed.

When will the dataset be distributed? We plan to make the dataset public in May or June 2023.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? It will be distributed under CC-BY-4.0.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? The Atari ROMs used to construct the Atari MDPs in BRIDGE are copyrighted by the original creators of the games. However, they are widely used throughout the reinforcement learning literature and to our knowledge the copyright holders have not complained about this. Since we are not legal experts, we do not know if releasing our dataset violates their copyright, but we do not believe that we are harming them since the tabular representations in BRIDGE are only useful for research purposes and cannot be used to play the games in any meaningful way.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

Any other comments? No.

E.4.7 Maintenance

Who will be supporting/hosting/maintaining the dataset? We (the authors) will support and maintain the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Redacted for double-blind review.

Is there an erratum? We will record reports of any errors in the dataset and release new versions with descriptions of what was fixed as necessary.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? We will release new versions of the dataset to correct any reported errors as described above. We may also expand the dataset in the future with more MDPs or new kinds of MDPs, such as stochastic or continuous-action-space MDPs. Any updates will be communicated through the service we use to host the dataset (TBD).

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? No.

Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers. We hope to find a host for the dataset that will retain older versions of the dataset. We only plan to maintain the latest version of the dataset, however. We will note this policy in the

dataset's description.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? There is no predefined mechanism to contribute to the dataset, but we will consider external contributions on a case-by-case basis. We encourage others to extend and build on the dataset.

Any other comments? No.

F Experiment details

In this appendix, we describe details of the experiments from Section 6. In particular, we describe how we calculate the empirical sample complexity of PPO and DQN.

We use the implementations of PPO and DQN from Stable-Baselines3 (SB3) [58]. For the network archictures, we use convolutional neural nets (CNNs) for Atari and Procgen and a fully-connected network for Minigrid. The network architectures are the default CNN and fully-connected architectures chosen by SB3. We used the below hyperparameters for PPO and DQN, which are mainly taken from the tuned Atari hyperparameters in the RL Baselines3 Zoo [59] repository.

We used a discount rate of γ = 1 for all environments and algorithms except for PPO and DQN in MiniGrid environments, where we used γ = 0 . 99 . We found that the performance of PPO and DQN significantly degraded when we used γ = 1 for MiniGrid.

PPO We use the following hyperparameters for PPO:

Table 6: Hyperparameters we use for PPO.

For each environment, we try rollout lengths of 128 and 1,280, as we find this is the most sensitive hyperparameter to tune.

DQN We use the following hyperparameters for DQN:

Table 7: Hyperparameters we use for DQN.

We try decaying the ϵ value for ϵ -greedy over the course of either 500 thousand or 5 million timesteps, as we found this was the most sensitive hyperparameter to tune for DQN.

Calculating the empirical sample complexity In order to compute the empirical sample complexities of PPO and DQN, throughout training we run evaluation episodes and see if the algorithms have discovered an optimal policy yet. During the evaluation, PPO policies take actions according to the argmax over the probabilities they assign to each action, rather than sampling as during training episodes. DQN takes actions greedily with respect to its current Q-function (i.e., with ϵ = 0 ). If the total episode reward during the evaluation is the optimal return, then we terminate the training run and record the total number of timesteps interacted with the environment as the empirical sample complexity. We take the median sample complexity over 5 random seeds and then the minimum over all hyperparameter settings to get the final empirical sample complexity.

We found that in some environments PPO achieved optimal reward during almost all the training episodes but none of the evaluation episodes. This can happen if a policy does not assign the highest probability to an optimal action in some states but can make up for this by being very likely overall to obtain the highest possible total reward. Thus, if more than half of the training episodes during an iteration achieve the optimal return, we also count this as converging to an optimal policy for the purposes of calculating the empirical sample complexity.

F.1 GORP vs. deep RL algorithms over longer horizons

To show that GORP is not just effective over short horizons, we ran additional experiments comparing GORP, PPO, and DQN in more typical Atari benchmark environments with frameskip 4 and a horizon of T = 27 , 000 (corresponding to a maximum of 30 minutes of gameplay). Similarly to the environments in BRIDGE, we limit agents to a single life and make the environments entirely deterministic. The hyperparameters for PPO and DQN are identical to those given in Tables 6 and 7 except for the following changes: we train for 50 million timesteps; we use a discount rate of γ = 0 . 99 ; and, we set an entropy coefficient of 0.01 for PPO.

We use a training batch size of 10 4 for PPO and decay ϵ for DQN over the course of the first 5 million timesteps. For GORP, we use k = 1 and tune m for each environment.

F.2 Exploration policies

For the experiments in Section 6 we needed pre-trained policies to initialize PPO with; here, we describe the details of how we trained them. We used two different training methods: one for Atari environments and one for Procgen environments.

In the Atari environments, we trained policies via behavior cloning (BC), i.e., supervised learning, from human data in the Atari-HEAD dataset [57]. We resampled the actions and processed the screen images from the dataset to align with the frameskip and observation preprocessing of our Atari environments. We trained a BC policy on each environment for 400 batches of 500 timesteps each. We used Adam with a learning rate of 10 -3 . We also added an entropy bonus to the loss function with

a weight of 0 . 1 to avoid the BC policy assigning very little weight to some actions. Our theoretical results in Section B.1 suggest that this should improve the sample complexity. Since not all Atari games are included in Atari-HEAD, we only used a subset for the experiments in Section 6.

In the Procgen environments, we pre-trained policies on a set of levels not included in BRIDGE. In particular, we trained a policy with PPO on 500 easy levels for 25 million timesteps, which in very similar to the methodology in Cobbe et al. [44]. We also use an entropy bonus in PPO with weight 0 . 1 for the same reason as above.

Once we have the pre-trained policies, we compute tabular representations of them on the corresponding MDPs in BRIDGE. To do so, we feed the observations for every state in the MDP through the pre-trained policy network and record the resulting action distribution. This allows us to compute Q π expl and thus obtain bounds on the effective horizon when using the pre-trained exploration policy.

F.3 Computational resources

For deep RL experiments, we used a mix of A100, A4000, and A6000 GPUs from Nvidia. We ran the algorithms either on separate GPUs or sometimes we ran multiple random seeds simultaneously on the same hardware. We used 1-8 CPU threads to run the RL environments. Using this setup, PPO and DQN generally took 2-8 hours to complete 5 million timesteps of training. We used early stopping when the algorithms found an optimal policy before 5 million timesteps, so the amount of compute per experiment was often less than this.

For constructing and analyzing the tabular MDPs in BRIDGE, we used up to 128 CPU threads and 500 GB of memory. The amount of time necessary to construct and analyze the MDPs ranged from less than a minute to around 5 days.

G Additional experiment results

Here, we present additional results from the experiments in Section 6.

G.1 Example of the effective horizon failing to predict generalization

As we described in the discussion, the effective horizon cannot model generalization across different states. For instance, in PONG-30 (Pong limited to 30 timesteps/15 seconds), the effective horizon gives a sample complexity of roughly 5 billion timesteps and empirically GORP takes over 80 million timesteps to converge to an optimal policy (Appendix G.3). However, both PPO and DQN converge in under 500,000 environment steps. We hypothesize this is because they are able to generalize the skill of hitting the ball across the multiple rounds of the Pong game, which the effective horizon cannot capture because it considers learning separately at every timestep.

G.2 Additional plots

1

2

3

4

5

6

7

8

9

10

Minimum value of

k

for which MDP is

k

-QVI-solvable

0

25

50

75

100

Number of MDPs

PPO sample complexity

>

5

M

PPO sample complexity

5

M

10

25

10

50

10

75

10

3

10

4

10

5

10

6

>

5

M

PPO

10

8

10

16

10

24

10

8

10

16

10

24

10

4

10

7

10

10

10

4

10

7

10

10

10

25

10

50

10

75

Worst-case

(

T

⌈

A

T

/

2

⌉

)

10

4

10

5

10

6

>

5

M

DQN

10

8

10

16

10

24

Covering length

(

TL

)

10

8

10

16

10

24

EPW

(

T

2

A

W

)

10

4

10

7

10

10

UCB

(

SAT

)

10

4

10

7

10

10

Effective horizon

(

T

2

A

H

)

Sample complexity bounds

Empirical sample complexity

Atari

Procgen

Minigrid

10

-

8

10

-

2

10

4

10

-

2

10

-

1

10

0

PPO

10

-

1

10

0

10

1

10

-

8

10

-

2

10

4

EPW

10

-

1

10

0

10

1

DQN

10

-

1

10

0

10

1

Effective horizon

Change in empirical samp. complex. (ratio)

Minigrid

Change in sample complexity bounds (ratio)

10

0

10

6

10

12

Covering length

10

-

2

10

0

10

2

PPO

10

-

2

10

0

10

2

Effective horizon

Change in sample complexity bounds (ratio)

Change in empirical

samp. complex. (ratio)

Atari

Procgen

G.3 List of MDPs in BRIDGE with statistics

The following table lists all the MDPs in BRIDGE, along with various properties: the number of states S , number of actions A , horizon T , minimum k for which the MDP is k -QVI-solvable, a bound on the effective horizon using the techniques in Appendix C, a bound on the covering length L using Lemma D.4, and the effective planning window W .

G.4 Table of bounds and empirical sample complexities

This table lists all the sample complexity bounds we calculate for each MDP in BRIDGE along with the empirical sample complexities of PPO, DQN, and GORP.

G.5 Table of returns

G.6 Learning curves for BRIDGE MDPs

0

5M

100

150

ALIEN

10

0

5M

50

100

AMIDAR

20

0

5M

50

100

ASSAULT

10

0

5M

200

400

ASTERIX

10

0

5M

0

250

ASTEROIDS

10

0

50k

100

200

ATLANTIS

10

0

500k

500

1000

ATLANTIS

20

0

5M

0

2500

ATLANTIS

30

0

5M

0

5000

ATLANTIS

40

0

5M

0

5000

ATLANTIS

50

0

5M

0

10000

ATLANTIS

70

0

2.5M

0

25

BANKHEIST

10

0

1M

0

2000

BATTLEZONE

10

0

5M

50

100

BEAMRIDER

20

0

2.5M

0

5

BOWLING

30

0

2.5M

0

2

BREAKOUT

10

0

5M

0

.

0

2

.

5

BREAKOUT

20

0

5M

0

5

BREAKOUT

30

0

5M

0

10

BREAKOUT

40

0

5M

0

10

BREAKOUT

50

0

5M

0

20

BREAKOUT

70

0

5M

0

25

BREAKOUT

100

0

5M

0

50

BREAKOUT

200

0

5M

1000

2000

CENTIPEDE

10

0

5M

250

500

CHOPPERCOMMAND

10

0

20k

200

400

CRAZYCLIMBER

20

0

2.5M

250

500

750

CRAZYCLIMBER

30

0

500k

25

50

DEMONATTACK

10

0

5M

0

5

ENDURO

10

0

5M

0

5

10

0

100k

0

1

FREEWAY

10

0

2.5M

0

2

FREEWAY

20

0

5M

0

.

0

2

.

5

FREEWAY

30

0

2.5M

0

5

FREEWAY

40

0

5M

0

5

FREEWAY

50

0

5M

0

5

FREEWAY

70

0

5M

0

10

FREEWAY

100

0

5M

0

25

FREEWAY

200

0

250k

25

50

FROSTBITE

10

0

500k

0

20

GOPHER

30

0

5M

50

100

150

GOPHER

40

0

25k

0

50

HERO

10

0

100k

0

1

ICEHOCKEY

10

0

2.5M

0

200

KANGAROO

20

0

5M

0

500

KANGAROO

30

0

5M

0

100

MONTEZUMAREVENGE

15

0

5M

200

400

MSPACMAN

20

0

5M

50

100

150

NAMETHISGAME

20

0

5M

100

200

PHOENIX

10

0

50k

-

2

-

1

PONG

20

0

5M

-

4

-

2

PONG

30

0

5M

-

5

0

PONG

40

0

5M

-

5

0

PONG

50

0

5M

-

10

0

PONG

70

0

5M

-

10

0

PONG

100

0

50k

0

100

PRIVATEEYE

10

0

5M

200

400

QBERT

10

0

2.5M

250

500

QBERT

20

0

5M

0

500

ROADRUNNER

10

0

1M

0

20

SEAQUEST

10

0

5M

-

30000

-

20000

-

10000

SKIING

10

0

50k

20

40

SPACEINVADERS

10

0

5M

0

1

TENNIS

10

0

250k

0

200

TIMEPILOT

10

0

500k

0

20

TUTANKHAM

10

0

5M

0

2500

VIDEOPINBALL

10

0

5M

0

500

WIZARDOFWOR

20

0

5M

2

4

BIGFISH

E0

10

0

1M

1

2

3

BIGFISH

E1

10

0

5M

2

.

5

5

.

0

7

.

5

BIGFISH

E2

10

0

5M

0

.

0

2

.

5

BIGFISH

H0

10

0

5M

0

.

25

0

.

50

0

.

75

CHASER E0

20

0

5M

0

.

25

0

.

50

0

.

75

CHASER E1

20

0

5M

0

.

25

0

.

50

0

.

75

CHASER E2

20

0

5M

0

.

50

0

.

75

CHASER H0

20

0

100k

0

2

CLIMBER E0

10

0

500k

0

2

10

0

5M

0

10

10

0

5M

0

2

10

0

1M

0

10

COINRUN E0

10

0

1M

0

10

COINRUN E1

10

0

2.5M

0

10

COINRUN E2

10

0

1M

0

10

COINRUN H0

10

0

200k

0

10

DODGEBALL E0

10

0

5M

0

5

DODGEBALL E1

10

0

250k

0

5

DODGEBALL E2

10

0

5M

0

5

DODGEBALL H0

10

0

5M

0

5

FRUITBOT E0

40

0

5M

0

5

FRUITBOT E1

40

0

25k

-

5

0

FRUITBOT E2

40

0

5M

0

10

FRUITBOT H0

40

0

5M

0

10

HEIST E1

10

0

5M

0

10

10

0

5k

0

10

JUMPER E0

20

0

2.5M

5

10

JUMPER E1

20

0

500k

0

10

JUMPER E2

20

0

5M

0

10

JUMPER EX

20

0

5M

0

10

LEAPER E1

20

0

2.5M

0

10

LEAPER E2

20

0

5M

0

10

LEAPER H0

20

0

5M

0

10

LEAPER EX

20

0

100k

5

10

MAZE E0

30

0

5M

0

10

MAZE E1

30

0

1M

0

10

MAZE E2

30

0

5M

0

10

MAZE H0

30

0

5M

0

10

MAZE EX

100

0

1M

0

1

MINER E0

10

0

5M

0

2

MINER E1

10

0

10k

0

1

MINER E2

10

0

1M

0

.

0

2

.

5

MINER H0

10

0

5M

0

10

NINJA E0

10

0

2M

0

10

NINJA E1

10

0

5M

0

10

NINJA E2

10

0

5M

0

10

NINJA H0

10

0

2M

0

1

PLUNDER E0

10

0

20k

0

1

PLUNDER E1

10

0

200k

0

1

PLUNDER E2

10

0

25k

0

1

PLUNDER H0

10

0

5M

0

10

STARPILOT E0

10

0

250k

0

5

STARPILOT E1

10

0

5M

0

5

STARPILOT E2

10

0

5M

0

5

STARPILOT H0

10

0

50k

0

.

5

1

.

0

EMPTY-5X5

0

500k

0

.

5

1

.

0

EMPTY-6X6

0

5M

0

1

EMPTY-8X8

0

5M

0

1

EMPTY-16X16

0

2.5M

0

1

DOORKEY-5X5

0

5M

0

1

0

5M

0

1

DOORKEY-8X8

0

5M

0

1

DOORKEY-16X16

0

100k

0

1

MULTIROOM-N2-S4

0

5M

0

1

MULTIROOM-N4-S5

0

5M

0

1

MULTIROOM-N6

0

2.5M

0

1

KEYCORRIDORS3R1

0

2.5M

0

1

KEYCORRIDORS3R2

0

5M

0

1

KEYCORRIDORS3R3

0

5M

0

1

KEYCORRIDORS4R3

0

5M

0

1

UNLOCK

0

5M

0

1

UNLOCKPICKUP

0

5M

0

1

BLOCKEDUNLOCKPICKUP

0

5M

0

1

OBSTRUCTEDMAZE-1DL

0

5M

0

1

OBSTRUCTEDMAZE-1DLH

0

5M

0

1

OBSTRUCTEDMAZE-1DLHB

0

5M

0

1

FOURROOMS

0

5M

0

1

LAVACROSSINGS9N1

0

5M

0

1

LAVACROSSINGS9N2

0

2.5M

0

1

LAVACROSSINGS9N3

0

5M

0

1

LAVACROSSINGS11N5

0

2M

0

1

SIMPLECROSSINGS9N1

0

2.5M

0

1

SIMPLECROSSINGS9N2

0

2.5M

0

1

SIMPLECROSSINGS9N3

0

5M

0

1

SIMPLECROSSINGS11N5

0

100k

0

1

LAVAGAPS5

0

5M

0

1

LAVAGAPS6

0

500k

0

1

LAVAGAPS7

GORP

PPO

DQN

Optimal return

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to improve the theoretical understanding of RL. The contributions of the paper are as follows:\n1. The authors curated a dataset called BRIDGE consisting of 155 deterministic MDPs along with their tabular representations\n1. The authors found that, for a given deterministic MDP, deep RL is more likely to succeed **if the policy acting greedily w.r.t to the random policy's Q function (which can be obtained via dynamic programming) is optimal**.\n1. Motivated by the property in the last bullet point, the authors propose a complexity measurement metric called **effective horizon** that roughly measures the number of steps of look-ahead search needed to identify the next optimal action.\n   * The authors show effective horizon is more reflective of PPO and DQN's empirical performance than prior complexity bound metrics\n\nSTRENGTHS:\nThe paper proposes a highly useful benchmark of deterministic MDPs, which is helpful for more theoretical analysis instead of relying merely on results on black-box algorithms. The finding that deep RL is more likely to succeed if the policy acting greedily w.r.t to the random policy's Q function (which can be obtained via dynamic programming) is optimal is extraordinary and quantitatively verified. Furthermore, the effective horizon bound is highly efficient and predictive of the agent's performance, which is useful for future theoretical studies. Last but not least, the paper is well written.\n\nWEAKNESSES:\nThe paper is well-written, and it is good to see that the authors have released the training code. However, I do have a concern about the PPO training results with Atari. \n\nFigure 5 on page 44 shows the training results of Atari with 50M steps (200M frames). However, the scores obtained by PPO are considerably lower than those from popular RL libraries. Below is a table summarizing the performance difference between the results reported in this paper compared to the results reported for `openai/baselines` in the [openrlbenchmark project](https://github.com/openrlbenchmark/openrlbenchmark/blob/8a1248b63daef6b4bab78467a2d7356038b87042/README.md#compare-cleanrls-ppo-with-openaibaseliness-ppo2-on-atari-games). **The PPO in this paper used 5x more samples and produced only 1/3 to 1/15 of the scores of  `openai/baselines`.** While the settings are clearly different in some ways (e.g., I noticed the authors did not use frame stacking in the Atari preprocessing), the performance discrepancy is large enough that I think warrants further investigation. \n\n| Environment | this paper's RLLib PPO @ 50M steps | `openai/baselines`' PPO @ 10M steps |\n| --- | --- | --- |\n| Beam Rider | ~700 | 2835.71 ± 387.92 |\n| Breakout | ~30 | 405.73 ± 11.47 |\n| Enduro | ~200 | 986.69 ± 25.28 |\n| Pong | 20 | 20.45 ± 0.81 |\n| Qbert | ~1000 | 15228.25 ± 920.95 |\n| Seaquest | ~400 | 1518.33 ± 400.35 |\n| Space Invaders | ~200 | 1019.75 ± 49.08  |\n\nPPO has many implementation details ([Engstrom, Ilyas, et al., 2020](https://openreview.net/forum?id=r1etN1rtPB); [Andrychowicz, et al., 2021](https://openreview.net/forum?id=nIAxjsniDzg); [Huang et al., 2022](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)) which may cause subtle issues. For sanity checks, maybe it's worthwhile conducting a PPO RLLib experiment with SB3's [Atari wrappers](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/atari_wrappers.py) for 10M steps. \n\nIf the performance is on par with  `openai/baselines` PPO results, then the PPO results of this paper can be attributed to different environment preprocessing. If PPO RLLib significantly underperforms, it could mean that PPO RLLib is not a proper PPO implementation, and the PPO results of this paper should be re-done with SB3's PPO, `openai/baselines`' PPO, or CleanRL's PPO, all of which are more well-benchmarked. \n\n\n\n\nP.S. According to https://github.com/openrlbenchmark/openrlbenchmark's documentation, you can retrieve the data above by running \n\n\n```bash\npip install openrlbenchmark==0.2.0\nwandb login\npython -m openrlbenchmark.rlops \\\n    --filters '?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return' \\\n        'baselines-ppo2-cnn?cl=openai/baselines PPO' \\\n    --filters '?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean' \\\n        'ppo?cl=SB3 ppo' \\\n        'dqn?cl=SB3 dqn' \\\n    --filters '?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return' \\\n        'ppo_atari_envpool_xla_jax_truncation?cl=CleanRL PPO (JAX)' \\\n    --env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4  EnduroNoFrameskip-v4 PongNoFrameskip-v4  QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \\\n    --env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4  EnduroNoFrameskip-v4 PongNoFrameskip-v4  QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \\\n    --env-ids BeamRider-v5 Breakout-v5  Enduro-v5 Pong-v5  Qbert-v5 Seaquest-v5 SpaceInvaders-v5  \\\n    --no-check-empty-runs \\\n    --pc.ncols 4 \\\n    --pc.ncols-legend 4 \\\n    --rliable \\\n    --rc.score_normalization_method atari \\\n    --rc.normalized_score_threshold 8.0 \\\n    --rc.sample_efficiency_plots \\\n    --rc.sample_efficiency_and_walltime_efficiency_method Median \\\n    --rc.performance_profile_plots  \\\n    --rc.aggregate_metrics_plots  \\\n    --output-filename figures/0compare \\\n    --scan-history\n```\n\n\n|                             | openai/baselines PPO   | SB3 ppo            | SB3 dqn           | CleanRL PPO (JAX)   |\n|:----------------------------|:-----------------------|:-------------------|:------------------|:--------------------|\n| BeamRiderNoFrameskip-v4     | 2835.71 ± 387.92       | 3844.13 ± 266.71   | 5774.61 ± 408.73  | 3133.78 ± 293.02    |\n| BreakoutNoFrameskip-v4      | 405.73 ± 11.47         | 307.39 ± 118.27    | 251.67 ± 15.92    | 465.90 ± 14.30      |\n| EnduroNoFrameskip-v4        | 986.69 ± 25.28         | 745.19 ± 139.98    | 493.08 ± 69.14    | 1178.37 ± 90.34     |\n| PongNoFrameskip-v4          | 20.45 ± 0.81           | 20.69 ± 0.33       | 20.44 ± 0.17      | 20.62 ± 0.18        |\n| QbertNoFrameskip-v4         | 15228.25 ± 920.95      | 13167.47 ± 1066.11 | 11502.93 ± 487.54 | 17318.50 ± 385.80   |\n| SeaquestNoFrameskip-v4      | 1518.33 ± 400.35       | 1538.04 ± 389.55   | 2936.30 ± 941.92  | 1203.30 ± 364.07    |\n| SpaceInvadersNoFrameskip-v4 | 1019.75 ± 49.08        | 911.16 ± 96.51     | 778.57 ± 46.76    | 1164.26 ± 176.96    |\n\nQUESTIONS:\nThe finding on the \"surprisingly common property\" is very interesting to me. I would be interested in more content on it. For example, how did you find this property? What is the motivation for finding the Q values of the random policy? Is there an example of how $\\prod(Q^{\\pi^{rand}})$ is constructed for a simple MDP?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new complexity metric, denoted as effective horizon, for deterministic MDPs. It is defined as the number of full value iteration updates starting from an initial policy, plus a logarithmic factor for concentration bounds, needed to reach an optimal policy. A corresponding algorithm, GORP, is also suitably defined. The bound is calculated for a set of 155 deterministic MDPs, and compared with empirical sample complexity of PPO and DQN, and is shown to correlate decently well compared to other sample complexity bounds. It is also shown how the effective horizon can be applied to shaped rewards or different initial policies to get a better problem-dependent estimate of sample complexity.\n\nSTRENGTHS:\nThere are extensive results and analyses in the paper and appendix, with good comparisons to other sample complexity measures. The task of converting many large MDPs to tabular representations and computing bounds can be very useful for future research on tighter theoretical bounds.\n\nThe proposed measure of effective horizon does seem to have good correlation with empirical algorithms, and intuitively makes sense, as the number of value iteration steps can correlate with the number of gradient updates to a value network.\n\nThe biggest strength of effective horizon is that it is policy and reward dependent, which makes it much easier to apply (as the paper points out) to shaped rewards and better initialized policies.\n\nWEAKNESSES:\nThere are many references to tables or figures in the appendix without marking them as being in the appendix, which can be confusing.\n\nAs mentioned in the paper, there are several limitations such as being only for deterministic MDPs, and hard to compute in general for large MDPs. Nevertheless, the initial results do demonstrate the viability of this type of bound, which future work can take and build upon.\n\nAnother downside is that the effective horizon can be hard to apply to algorithms with additional exploration built-in, such as using an intrinsic reward. This is due to the fact that the intrinsic reward is non-stationary and changes over time, so the effective horizon would only give a bound for a specific snapshot of intrinsic reward, rather than the entire training process. How to handle non-stationarity could be an interesting future direction.\n\nSome concerns about the experiments are brought up in the questions section below.\n\nQUESTIONS:\nHow do you define optimal policy across your environments since your episodes are relatively short? Many Atari games go on for thousands if not hundreds of thousands of timesteps, so what does it mean to reach optimality?\n\nFor the empirical GORP results in table 2, what value of k and m were used? Did you sweep over k to find the best k? Or use the smallest k that reaches optimal?\n\nFor table 2, for the horizon T, were you using the horizon of the tabular representation (i.e. after frameskip)? If so, there is a degree of arbitrary choice here, as many of the Atari environments had large frameskips of 30, which greatly reduces the T needed to see reward. If frameskips were smaller, such as 4, and horizon increased accordingly to accommodate, then all theoretical bounds would be even more orders of magnitude off from empirical sample complexity.\n\nLIMITATIONS:\nAre addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the question of what finding an explanation for the performance of deep RL algorithms. To accomplish this, the authors propose the notion of effective horizon. It is found that this notion is better able to answer questions about sample complexity and effect of reward shaping or policy initialization in RL than all prior theoretical bounds.\n\nSTRENGTHS:\nStrengths:\n1. The first great part about the paper is the writing. Reading the paper is easy and flows well. \n2. The authors present a surprising finding - on a large fraction of environment just evaluating random policy with one step of policy improvement leads to optimal policy. For the rest if the environments, a few steps of value iteration usually leads to optimal policy. This is in contrast to previous ideas and theory in RL which are unable to explain when RL fails in certain environments and when it will work. \n3. Authors propose a new method GORP (Greedy over random policies), which performs a few step of policy improvements over a random policy and is shown to be a competitor to PPO and DQN algorithms even for long horizon.\n4. The experiments demonstrate that the theoretical bounds for GORP neatly explains empirical bounds obtained by DQN/PPO on complex environments by measuring quantities like correlation, medium ratio, etc. \n5. The analysis in the paper is also the first to explain the effect of reward shaping and initializing RL with a policy using the GORP bound.  The effective horizon bound has a correlation of around 0.4 in both of these settings (reward shaping and policy initialization) which is more than other bounds which are agnostic to reward or do not consider the effect of policy past the horizon.\n\nWEAKNESSES:\nWeaknesses:\n1. Unlike prior bounds using UCB and EPW, GORP seems to require oracle information to compute the effective horizon. Can the authors comment on the implication of this in the paper? For example, we can’t use GORP bounds to compute sample efficiency without having an algorithm that solves the task first. \n2. It is claimed that GORP achieves an optimal policy in almost all the environment when a effective horizon > 1 is used (Line 77), but the experiments for this seem to be missing. Learning curves and final performance along with effective horizon used would be important to support this claim.\n3. Novelty of GORP: It would be important to do a more thorough literature review with prior works in the area which proposed an algorithm very similar to GORP in order to clarify the novelty there:\na. MCTS\nb. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Silver et al\nc. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Schrittwieser et al\nd. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, Lowrey et al\ne. Learning Off-Policy with Online Planning, Sikchi et al\nf. Model-Based Offline Planning, Argenson et al\n\nQUESTIONS:\n1. Does GORP with a particular effective horizon outperform DQN/PPO? This experiment seem to be missing in Figure 5 and figure 10 where k is fixed to be 1.\n2. How is line 7 in Algorithm 1 performed in practice? Is the whole list of possible actions enumerated to find the best one? If so, does the method assume known model of the environment? In that case, why does comparing to only model-free methods make sense for sample complexity bounds?\n\nLIMITATIONS:\nAuthors address the limitation in Section 7 of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims at addressing the gap between RL theory and deep RL practice. \nThe central finding of the paper is a property that correlates well with the performance of PPO and DQN algorithms on a set of 155 environments (such as Atari, Procgen, and MiniGrid).\nThe property looks at whether the argmax action of the action-value function under the random policy coincides with the argmax action of the optimal action-value function. \nInspired by the property, the authors propose a complexity measure called the effective horizon that corresponds to the number of lookahead search steps needed to identify an optimal action, given that the leaf nodes are evaluated under the random policy.\nBounds that utilize the measure correlate with empirical sample complexity of PPO on the collection of 155 games (r = 0.79 for PPO and r = 0.65 for DQN).\nLastly, the paper demonstrates the predictive power of the effective horizon when using reward shaping and pre-training.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow.\n- Motivation is solid: the paper makes a step certainly in the right direction as the gap between RL theory and practice is tremendous.\n- The personal experience of the reviewer with the submission started from \"That's surprising\", followed by \"It can't be true\", finally reaching \"Ah, it makes sense\", implying that the paper offers an interesting and non-trivial way to think about RL. One of the take-aways for the reviewer is that if the policy evaluation is done perfectly, there might be a need only for just a few (sometimes zero) policy improvement steps.\n\nThe reviewer is inclined to recommend the paper for acceptance because it is likely to generate a fruitful discussion in the community.\n\nWEAKNESSES:\nThere might be certain weaknesses that the reviewer didn't notice but as of now there are none that feel worthwhile mentioning; having said that, the reviewer has limited experience with complexity bounds in RL and maybe more experienced RL theory person will be able to identify the limitations of the analysis.\nThe reviewer is looking forward for the discussion phase.\n\nQUESTIONS:\nCould you quantify the agreement between the (exact) action-value function corresponding to a random policy and a Q-network with randomly initialized parameters? If the agreement is high, it'd somewhat imply that for many domains the weights are initialized at a point that is almost an optimal solution; such a finding would be even more surprising.\n\nThe reviewer would suggest to take a look at the Policy Churn paper [1] that observes that the argmax of the learned action-value function in DQN-based methods changes in 10% of the states after each parameter update. The observation might be related to the high predictive power of the measure the authors observe.\n\n[1] Schaul, Tom, André Barreto, John Quan, and Georg Ostrovski. \"The phenomenon of policy churn.\" Advances in Neural Information Processing Systems 35 (2022): 2537-2549.\n\nLIMITATIONS:\nPerhaps the main limitation is that the effective horizon could be used for analysis purposes only; from the practical viewpoint, it is unclear how for a large MDP estimate efficiently the effective horizon (and thus obtain the sample complexity for the MDP).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}