Title: value: Bridging RL Theory and Practice with the Effective Horizon
Authors: value: Cassidy Laidlaw
Stuart Russell
Anca Dragan
Venue: NeurIPS 2023
Keywords: value: reinforcement learning
RL theory
theory of reinforcement learning
instance-dependent bounds
empirical validation of theory

Abstract: value: Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the *random* policy also have the highest Q-values under the *optimal* policy—i.e., when it is optimal to act greedily with respect to the random's policy Q function—deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the *effective horizon*, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also show that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon.


Review discussion:
The paper proposes a highly useful benchmark of deterministic MDPs, which is helpful for more theoretical analysis instead of relying merely on results on black-box algorithms. The finding that deep RL is more likely to succeed if the policy acting greedily w.r.t to the random policy's Q function (which can be obtained via dynamic programming) is optimal is extraordinary and quantitatively verified.
Furthermore, the effective horizon bound is highly efficient and predictive of the agent's performance, which is useful for future theoretical studies.
Last but not least, the paper is well written.
The paper is well-written, and it is good to see that the authors have released the training code.
However, I do have a concern about the PPO training results with Atari.
Figure 5 on page 44 shows the training results of Atari with 50M steps (200M frames).
However, the scores obtained by PPO are considerably lower than those from popular RL libraries. Below is a table summarizing the performance difference between the results reported in this paper compared to the results reported for `openai/baselines` in the [openrlbenchmark project](https://github.com/openrlbenchmark/openrlbenchmark/blob/8a1248b63daef6b4bab78467a2d7356038b87042/README.md#compare-cleanrls-ppo-with-openaibaseliness-ppo2-on-atari-games). **The PPO in this paper used 5x more samples and produced only 1/3 to 1/15 of the scores of  `openai/baselines`.** While the settings are clearly different in some ways (e.g., I noticed the authors did not use frame stacking in the Atari preprocessing), the performance discrepancy is large enough that I think warrants further investigation.
| Environment | this paper's RLLib PPO @ 50M steps | `openai/baselines`' PPO @ 10M steps |
| --- | --- | --- |
| Beam Rider | ~700 | 2835.71 ± 387.92 |
| Breakout | ~30 | 405.73 ± 11.47 |
| Enduro | ~200 | 986.69 ± 25.28 |
| Pong | 20 | 20.45 ± 0.81 |
| Qbert | ~1000 | 15228.25 ± 920.95 |
| Seaquest | ~400 | 1518.33 ± 400.35 |
| Space Invaders | ~200 | 1019.75 ± 49.08  |
PPO has many implementation details ([Engstrom, Ilyas, et al., 2020](https://openreview.net/forum?id=r1etN1rtPB); [Andrychowicz, et al., 2021](https://openreview.net/forum?id=nIAxjsniDzg); [Huang et al., 2022](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)) which may cause subtle issues. For sanity checks, maybe it's worthwhile conducting a PPO RLLib experiment with SB3's [Atari wrappers](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/atari_wrappers.py) for 10M steps.
If the performance is on par with  `openai/baselines` PPO results, then the PPO results of this paper can be attributed to different environment preprocessing. If PPO RLLib significantly underperforms, it could mean that PPO RLLib is not a proper PPO implementation, and the PPO results of this paper should be re-done with SB3's PPO, `openai/baselines`' PPO, or CleanRL's PPO, all of which are more well-benchmarked.
P.S. According to https://github.com/openrlbenchmark/openrlbenchmark's documentation, you can retrieve the data above by running
```bash
pip install openrlbenchmark==0.2.0
wandb login
python -m openrlbenchmark.rlops \
--filters '?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return' \
'baselines-ppo2-cnn?cl=openai/baselines PPO' \
--filters '?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean' \
'ppo?cl=SB3 ppo' \
'dqn?cl=SB3 dqn' \
--filters '?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return' \
'ppo_atari_envpool_xla_jax_truncation?cl=CleanRL PPO (JAX)' \
--env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4  EnduroNoFrameskip-v4 PongNoFrameskip-v4  QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \
--env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4  EnduroNoFrameskip-v4 PongNoFrameskip-v4  QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \
--env-ids BeamRider-v5 Breakout-v5  Enduro-v5 Pong-v5  Qbert-v5 Seaquest-v5 SpaceInvaders-v5  \
--no-check-empty-runs \
--pc.ncols 4 \
--pc.ncols-legend 4 \
--rliable \
--rc.score_normalization_method atari \
--rc.normalized_score_threshold 8.0 \
--rc.sample_efficiency_plots \
--rc.sample_efficiency_and_walltime_efficiency_method Median \
--rc.performance_profile_plots  \
--rc.aggregate_metrics_plots  \
--output-filename figures/0compare \
--scan-history
|                             | openai/baselines PPO   | SB3 ppo            | SB3 dqn           | CleanRL PPO (JAX)   |
|:----------------------------|:-----------------------|:-------------------|:------------------|:--------------------|
| BeamRiderNoFrameskip-v4     | 2835.71 ± 387.92       | 3844.13 ± 266.71   | 5774.61 ± 408.73  | 3133.78 ± 293.02    |
| BreakoutNoFrameskip-v4      | 405.73 ± 11.47         | 307.39 ± 118.27    | 251.67 ± 15.92    | 465.90 ± 14.30      |
| EnduroNoFrameskip-v4        | 986.69 ± 25.28         | 745.19 ± 139.98    | 493.08 ± 69.14    | 1178.37 ± 90.34     |
| PongNoFrameskip-v4          | 20.45 ± 0.81           | 20.69 ± 0.33       | 20.44 ± 0.17      | 20.62 ± 0.18        |
| QbertNoFrameskip-v4         | 15228.25 ± 920.95      | 13167.47 ± 1066.11 | 11502.93 ± 487.54 | 17318.50 ± 385.80   |
| SeaquestNoFrameskip-v4      | 1518.33 ± 400.35       | 1538.04 ± 389.55   | 2936.30 ± 941.92  | 1203.30 ± 364.07    |
| SpaceInvadersNoFrameskip-v4 | 1019.75 ± 49.08        | 911.16 ± 96.51     | 778.57 ± 46.76    | 1164.26 ± 176.96    |
The finding on the "surprisingly common property" is very interesting to me. I would be interested in more content on it.
For example, how did you find this property? What is the motivation for finding the Q values of the random policy? Is there an example of how $\prod(Q^{\pi^{rand}})$ is constructed for a simple MDP?
Yes.
No ethics review needed.
We thank the reviewer for their careful and comprehensive review. We are glad that they found our results on acting greedily with respect to the random policy’s Q-function “extraordinary and quantitatively verified.” Below are responses to questions and comments.
**Our PPO results vs. benchmarks:** We appreciate the reviewer’s thorough comparison of our PPO results in Appendix H. 1 to previous benchmarks. We believe that the difference in returns is because *we limit agents to a single life* in the Atari games, as described in Appendix F. 1. We do this in order to align the Atari games more precisely with the theoretical framework for episodic RL, where episodes always start in states drawn from the same distribution. In typical Atari training, episodes are truncated when a life is lost but then a new episode is started in the state following the lost life.
This actually makes the starting state distribution policy-dependent, which is quite different from the theoretical RL setting. Note that this also introduces a difference between training, when Atari agents’ episodes are truncated on loss of life, versus testing, when the agents’ rewards are summed over all lives. To avoid these problems, we simply end episodes entirely after one life during both training and testing.
We believe that this accounts for the difference between our results and those from other RL libraries. RLlib’s PPO implementation has been benchmarked [here](https://docs.ray.io/en/latest/rllib/rllib-algorithms.html#ppo) in several Atari environments and performs quite similarly to SB3, so we do not think that the RLlib implementation is the reason for the difference in results. Most of these environments have 3-5 lives, which could explain why the reward is often 3-5x lower for our setting.
We also plan to rerun our experiments with the SB3 implementations of PPO and DQN to make sure that they are robust to the exact implementations of the RL algorithms. We will include these results in the final paper if accepted.
**More information about the surprisingly common property:** One of the motivations for examining the Q-values of the random policy was an analysis of policy gradient algorithms.
When initialized with the random policy, the policy gradient can be exactly expressed in terms of the Q-values of the random policy:
$$\\nabla\_\\theta J(\\theta) = \\mathbb{E}\_{\\pi^\\text{rand}}\\left[ Q^{\\pi^\\text{rand}}(s, a) \\nabla\_\\theta \\pi\_\\theta(a \\mid s) \\right]$$
Thus, one might expect policy gradient algorithms to (at least initially) move towards a policy which chooses actions with the highest Q-values under the random policy. This motivated us to examine the performance of simply acting greedily with respect to the random policy’s Q-function.
To calculate $\\Pi(Q^{\\pi^\\text{rand}})$ for an environment, we first run value iteration on the tabular representation to precisely calculate the Q-function $Q^{\\pi^\\text{rand}}$ at every state-action pair.
Then, we can easily compute the set of policies which are greedy according to that Q-function as policies which take one of the actions with the highest Q-value at each state.
For instance, suppose at the first state there are three actions with Q-values of 1, 1, and ½.
Then a policy which takes either of the first two actions is greedy with respect to the Q-function. Often, there are multiple actions with the highest Q-value, since in many environments multiple actions may have the same effect.
I thanks the author for the detailed reply.
> We also plan to rerun our experiments with the SB3 implementations of PPO and DQN to make sure that they are robust to the exact implementations of the RL algorithms. We will include these results in the final paper if accepted.
That sounds like a good plan, and thanks for entertaining this thought. I really like the writing and ideas of the paper and think having the SB3 experiments can make the paper more solid.
> We believe that the difference in returns is because we limit agents to a single life in the Atari games, as described in Appendix F.1.
This is a perfectly reasonable explanation. My reservation is that DRL is brittle, and there is a non-zero chance that
1. you experiment with SB3's PPO and DQN and find the claims are no longer supported by the experiments, or
2. you align the Atari and hyperparameter setting with the RLlib's codebase and find RLlib's PPO still underperforms, which causes a reproducibility issue, making it difficult to attribute and interpret the paper's claims.
To give an example of the second issue, [Luo et al., 2020](https://arxiv.org/pdf/1912.00167.pdf) claim their proposed approach IMPACT outperforms IMPALA because their IMPACT gets ~550 score in Breakout whereas their IMPALA baseline gets ~500 score.
However, the [original IMPALA](https://arxiv.org/pdf/1802.01561.pdf) reported 640.43 for IMPALA (shallow) and 787.34 for IMPALA (deep). This is clearly a reproducibility issue on their IMPALA, and it's unclear whether IMPACT can still outperform a more aligned IMPALA baseline.
> RLlib’s PPO implementation has been benchmarked here (https://docs.ray.io/en/latest/rllib/rllib-algorithms.html#ppo) in several Atari environments and performs quite similarly to SB3, so we do not think that the RLlib implementation is the reason for the difference in results.
Thanks for sharing the RLlib's benchmark results.
However, the benchmark results are concerning for several reasons.
1. The benchmark results are almost 5 years old (benchmarked on 2018-09 (https://github.com/ray-project/rl-experiments#2018-09)). I could not find a benchmark on their more recent codebase.
2. The benchmark results are not sufficient (only tested on 4 envs);
also, it appeared only to use 1 random seed.
3. The benchmark showed RLlib's PPO underperforms `openai/baselines`' PPO, SB3 ppo, and CleanRL PPO (JAX) in **4 out of the 4 tasks** reported.
|                             | openai/baselines PPO   | SB3 ppo            | CleanRL PPO (JAX)   | RLlib's PPO @ 10M steps |
|:----------------------------|:-----------------------|:-------------------|:--------------------|:------------------------|
| BeamRiderNoFrameskip-v4     | 2835.71 ± 387.92       | 3844.13 ± 266.71   | 3133.78 ± 293.02    | 2807.00           |
| BreakoutNoFrameskip-v4      | 405.73 ± 11.47         | 307.39 ± 118.27    | 465.90 ± 14.30      | 104.00            |
| EnduroNoFrameskip-v4        | 986.69 ± 25.28         | 745.19 ± 139.98    | 1178.37 ± 90.34     | -                |
| PongNoFrameskip-v4          | 20.45 ± 0.81           | 20.69 ± 0.33       | 20.62 ± 0.18        | -                 |
| QbertNoFrameskip-v4         | 15228.25 ± 920.95      | 13167.47 ± 1066.11 | 17318.50 ± 385.80   | 11085 |
| SeaquestNoFrameskip-v4      | 1518.33 ± 400.35       | 1538.04 ± 389.55   | 1203.30 ± 364.07    | -                |
| SpaceInvadersNoFrameskip-v4 | 1019.75 ± 49.08        | 911.16 ± 96.51     | 1164.26 ± 176.96    | 671.00            |
Thank you for your response to our rebuttal. We agree that deep RL is brittle and that it is important to make sure that our main results are robust to implementation details and hyperparameters.
Since the rebuttal period, we have now been able to re-run our main experiments with the Stable Baselines 3 implementations of PPO and DQN.
That is, we have trained PPO and DQN agents for 5 million steps across 5 random seeds on all the environments in BRIDGE using the Atari hyperparameters from the `rl-zoo` repository [here](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams). We did tune the hyperparameters somewhat to each environment—we tuned $\text{n\\_steps} \in \\{128, 1280\\}$ for PPO and tuned $\text{exploration\\_fraction} \in \\{0.1, 1\\}$ for DQN.
Here are our main results from Table 2 when using the results from the SB3 implementations instead of RLlib:
| Bound                                  | PPO: Correl. | Median ratio         | AUROC    | Acc.     | DQN: Correl. | Median ratio         | AUROC    | Acc.     |
|----------------------------------------|-------------|----------------------|----------|----------|-------------|----------------------|----------|----------|
| Worst-case ($T \lceil A^T / 2 \rceil$) | 0.18        | $1.3 \times 10^{10}$ | 0.62     | 0.61     | 0.18        | $5.5 \times 10^{10}$ | 0.67     | 0.75     |
| Covering length ($T L$)                | 0.29        | $4.5 \times 10^{6}$  | 0.78     | 0.73     | 0.31        | $3.9 \times 10^{6}$  | 0.81     | 0.83     |
| EPW ($T^2 A^W$)                        | 0.67        | $4.7 \times 10^{4}$  | 0.80     | 0.73     | 0.61        | $8.0 \times 10^{4}$  | 0.87     | 0.84     |
| UCB ($S A T$)                          | 0.22        | **20**               | 0.67     | 0.65     | 0.34        | **31**               | 0.64     | 0.76     |
| Effective horizon ($T^2 A^H$)          | **0.80**    | **20**               | **0.91** | **0.85** | **0.75**    | 60                   | **0.90** | **0.85** |
| *Other algorithm*                      | 0.75        | 2.0                  | 0.84     | 0.83     | 0.75        | 2.0                  | 0.74     | 0.98     |
| *GORP*                                 | 0.77        | 5.7                  | 0.76     | 0.79     | 0.65        | 9.4                  | 0.81     | 0.93     |
We find that the main conclusions of the paper are unchanged when using the SB3 implementations. The effective horizon-based bounds still correlate the best out of any of the provable bounds with the empirical performance of deep RL algorithms (in fact, the correlation is slightly better with SB3 than RLlib). The UCB bounds are a bit closer in median ratio to the empirical performance of DQN, but the correlation is so poor that we believe the effective horizon remains the best explanation for deep RL's performance.
Comparing directly between the RLlib and SB3 implementations of PPO, we found that:
* In 37% of environments, the SB3 implementation outperforms the RLlib one (i.e., either the final return is higher or it achieves the optimal return in fewer timesteps).
* In 23% of environments, the SB3 implementation performs equally to the RLlib one (i.e., they both achieve the same final return, and if it is optimal, then they achieve it in the same number of timesteps).
* In 40% of environments, the SB3 implementation performs worse than the RLlib one.
We also found a Spearman correlation of 0.94 between the empirical sample complexities of the two implementations. This suggests that the RLlib and SB3 implementations are relatively similar.
We have not yet had the chance to re-run the long-horizon Atari experiments using SB3 (Appendix H.1).
However, the fact that we were able to closely replicate our original results using the SB3 implementations—showing that the effective horizon closely correlates with deep RL performance—strongly suggests that our results are robust to the exact implementations of the RL algorithms.
Thanks for running these experiments. I have updated my ratings correspondingly.
There are extensive results and analyses in the paper and appendix, with good comparisons to other sample complexity measures. The task of converting many large MDPs to tabular representations and computing bounds can be very useful for future research on tighter theoretical bounds.
The proposed measure of effective horizon does seem to have good correlation with empirical algorithms, and intuitively makes sense, as the number of value iteration steps can correlate with the number of gradient updates to a value network.
The biggest strength of effective horizon is that it is policy and reward dependent, which makes it much easier to apply (as the paper points out) to shaped rewards and better initialized policies.
There are many references to tables or figures in the appendix without marking them as being in the appendix, which can be confusing.
As mentioned in the paper, there are several limitations such as being only for deterministic MDPs, and hard to compute in general for large MDPs.
Nevertheless, the initial results do demonstrate the viability of this type of bound, which future work can take and build upon.
Another downside is that the effective horizon can be hard to apply to algorithms with additional exploration built-in, such as using an intrinsic reward. This is due to the fact that the intrinsic reward is non-stationary and changes over time, so the effective horizon would only give a bound for a specific snapshot of intrinsic reward, rather than the entire training process. How to handle non-stationarity could be an interesting future direction.
Some concerns about the experiments are brought up in the questions section below.
How do you define optimal policy across your environments since your episodes are relatively short? Many Atari games go on for thousands if not hundreds of thousands of timesteps, so what does it mean to reach optimality?
For the empirical GORP results in table 2, what value of k and m were used? Did you sweep over k to find the best k? Or use the smallest k that reaches optimal?
For table 2, for the horizon T, were you using the horizon of the tabular representation (i.e. after frameskip)? If so, there is a degree of arbitrary choice here, as many of the Atari environments had large frameskips of 30, which greatly reduces the T needed to see reward. If frameskips were smaller, such as 4, and horizon increased accordingly to accommodate, then all theoretical bounds would be even more orders of magnitude off from empirical sample complexity.
Are addressed in the paper.
No ethics review needed.
We thank the reviewer for their comprehensive review. We appreciate that they found our results “extensive” and that they think the tabular representations in our BRIDGE dataset “can be very useful for future research on tighter theoretical bounds.” Below are responses to individual comments and questions.
**Optimal policy in short-horizon environments:** As the reviewer noted, the environments in the BRIDGE dataset have relatively short horizons compared to many typical RL benchmarks. We found that it was intractable to compute the tabular representations for long-horizon environments since the number of states reachable over long horizons was too large.
Thus, we use horizon-limited environments for most of the results in the paper. We define the optimal policy as the policy which achieves the highest return over the limited horizon $T$ of the environment.
Note that we run the deep RL algorithms and GORP in the horizon-limited environments too such that they are exactly equivalent to the tabular representations.
Thus, the maximum possible return achievable by PPO, DQN, and GORP is exactly the same as the return of the optimal policy in the tabular representation. We only use the normal Atari settings (frameskip=4, horizon=27,000) for the experiments in Appendix H.1. We will clarify this in the paper.
**Values of $k$ for empirical GORP sample complexity:** For the results in Table 2, we swept over values of $k$ and chose the value with the smallest sample complexity.
However, even if we restrict GORP to using $k = 1$, we find a Spearman correlation of 0.69 for PPO and 0.69 for DQN between the GORP sample complexity and the deep RL sample complexity. This is still about as well as PPO or DQN correlate with each other, indicating that GORP is quite similar to PPO/DQN even when $k$ is not tuned. We will include these results in the final version of the paper if accepted.
**Values of horizon $T$ in Table 2:** We use the horizons for the horizon-limited versions of these environments. As described above, these are the versions that PPO, DQN, and GORP are trained in as well, so there is no arbitrary selection of horizon—it is based on the environment that the RL algorithms are interacting with. If we made the frameskip smaller, it might change the theoretical bounds, but it could also significantly change the empirical sample complexity of PPO and DQN since they will also need to interact with the environment for more timesteps to see rewards.
Our results in Appendix H.1 show that GORP is also quite similar to PPO/DQN over much longer horizons like those normally used for benchmarking deep RL algorithms. Since the effective horizon is based on GORP, we also expect that the effective horizon-based bounds would closely reflect deep RL performance with longer horizons and smaller frameskip (although we can’t test this exactly since it is intractable to compute tabular representations of the long-horizon environments).
**References to tables and figures in the appendix:** We thank the reviewer for pointing out that these references are confusing as they are now. We will make it more clear when references are made to the appendix.
**Application to algorithms with additional exploration strategies:** We agree that this is a limitation of our analysis and will mention it in the paper as a direction for future work.
Thank you for your response! I like the paper a lot and my score remains unchanged.
Strengths:
1. The first great part about the paper is the writing. Reading the paper is easy and flows well.
2. The authors present a surprising finding - on a large fraction of environment just evaluating random policy with one step of policy improvement leads to optimal policy. For the rest if the environments, a few steps of value iteration usually leads to optimal policy. This is in contrast to previous ideas and theory in RL which are unable to explain when RL fails in certain environments and when it will work.
3. Authors propose a new method GORP (Greedy over random policies), which performs a few step of policy improvements over a random policy and is shown to be a competitor to PPO and DQN algorithms even for long horizon.
4. The experiments demonstrate that the theoretical bounds for GORP neatly explains empirical bounds obtained by DQN/PPO on complex environments by measuring quantities like correlation, medium ratio, etc.
5. The analysis in the paper is also the first to explain the effect of reward shaping and initializing RL with a policy using the GORP bound.  The effective horizon bound has a correlation of around 0.4 in both of these settings (reward shaping and policy initialization) which is more than other bounds which are agnostic to reward or do not consider the effect of policy past the horizon.
Weaknesses:
1.
Unlike prior bounds using UCB and EPW, GORP seems to require oracle information to compute the effective horizon. Can the authors comment on the implication of this in the paper?
For example, we can’t use GORP bounds to compute sample efficiency without having an algorithm that solves the task first.
2. It is claimed that GORP achieves an optimal policy in almost all the environment when a effective horizon > 1 is used (Line 77), but the experiments for this seem to be missing. Learning curves and final performance along with effective horizon used would be important to support this claim.
3. Novelty of GORP: It would be important to do a more thorough literature review with prior works in the area which proposed an algorithm very similar to GORP in order to clarify the novelty there:
a. MCTS
b. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Silver et al
c. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Schrittwieser et al
d. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, Lowrey et al
e. Learning Off-Policy with Online Planning, Sikchi et al
f. Model-Based Offline Planning, Argenson et al
1. Does GORP with a particular effective horizon outperform DQN/PPO? This experiment seem to be missing in Figure 5 and figure 10 where k is fixed to be 1.
2. How is line 7 in Algorithm 1 performed in practice? Is the whole list of possible actions enumerated to find the best one? If so, does the method assume known model of the environment? In that case, why does comparing to only model-free methods make sense for sample complexity bounds?
Authors address the limitation in Section 7 of the paper.
No ethics review needed.
We thank the reviewer for their thorough and thoughtful review. Below are responses to questions and comments raised in the review.
**Effective horizon vs $k$:** First, we noticed that there may have been some confusion between the value of $k$ for which an MDP is $k$-QVI-solvable versus the effective horizon $H$. $k$ relates to the number of steps of value iteration that need to be applied to the random policy’s Q-function before one can act greedily to solve the MDP (Definition 5.1). The effective horizon (Definition 5.2) combines $k$ with $m_k$, which is the number of random rollouts collected for each action sequence in GORP (see Figure 1 or Algorithm 1). We will clarify the difference between these two values in the paper.
**Requiring oracle information to compute the effective horizon:** While it is true that one needs access to the tabular representation of the MDP to compute the effective horizon, in practice this is true for UCB and EPW bounds too. UCB bounds depend on the number of states in an environment, which is often unknown. The number of states can sometimes be upper-bounded, but so can the effective horizon (since $H \leq T$ always). Exploring every state to count them may be as hard or harder than solving the MDP. EPW bounds also require access to the optimal policies for an MDP, since one needs to know which actions are optimal to know if those actions can be found by only considering rewards in the planning window. We will clarify this in the paper.
**Claim that GORP achieves an optimal policy in many environments:** We refer the reviewer to Appendix H in the supplementary material for the full results of our experiments on GORP. Appendix H.5 lists the effective horizon for each environment; Appendix H.7 lists the final reward achieved by each algorithm, including GORP; and Appendix H.8 has learning curves for PPO, DQN, and GORP in all environments.
**Novelty of GORP:** We thank the reviewer for the references to related work on Monte Carlo planning. We note that GORP may be similar to Monte Carlo planning algorithms, but is novel because it is actually *not* a planning algorithm—instead, it is a model-free RL algorithm. It does not use a learned or specified world model, unlike all the given references. One of our key contributions with GORP is to show that model-free RL algorithms like PPO and DQN may implicitly be doing something similar to Monte Carlo planning despite appearing quite different.
**Does GORP with a particular effective horizon outperform PPO/DQN:** Again, we note that $k$ is different from the effective horizon $H$.
However, we will include results for GORP constrained to $k=1$ in the final paper.
For instance, GORP with $k=1$ has a lower sample complexity than DQN in 67 of the 99 environments in BRIDGE where DQN converges; with any $k$, this rises to 81 of 99. GORP with $k = 1$ also outperforms PPO in 39 of 98 environments where PPO converges, compared to 56 with any value of $k$.
Finally, note that we compared GORP against PPO and DQN on long-horizon Atari games in Figure 5; we used only $k = 1$ for those experiments (see Line 1571).
**Line 7 in Algorithm 1:** In Line 7 of the GORP algorithm, the argmax action is chosen based on the estimated Q values for each $k$-action sequence, which are calculated inside the loop on Line 5. Each loop consists of several rollouts following the existing learned policy up to timestep $i - 1$, then following a particular action sequence $a_{i : i + k - 1}$ for timesteps $i$ to $i + k - 1$, and then acting randomly; the mean of the rewards across rollouts is computed and saved as the approximate Q-value $\\hat{Q}\_i(s\_i, a\_{i : i + k - 1})$  for that action sequence. On Line 7, the sequence with the highest estimated Q-value is found and its first action is selected as the policy’s action for timestep $i$. None of these steps uses a model, so the algorithm is model-free.
Thanks for the clarification and answering the questions. It would be great if you can incorporate parts of the response above to make clear (1) differences from model-based methods mentioned above and how you bridge the gap from them to dqn/ppo (2) stress that effective horizon is not just k, possibly giving an intuition for the same.
Thanks for your response. We will definitely incorporate explanations of both points into the paper. We appreciate the feedback to make our exposition more clear.
- The paper is well-written and easy to follow.
- Motivation is solid: the paper makes a step certainly in the right direction as the gap between RL theory and practice is tremendous.
- The personal experience of the reviewer with the submission started from "That's surprising", followed by "It can't be true", finally reaching "Ah, it makes sense", implying that the paper offers an interesting and non-trivial way to think about RL. One of the take-aways for the reviewer is that if the policy evaluation is done perfectly, there might be a need only for just a few (sometimes zero) policy improvement steps.
The reviewer is inclined to recommend the paper for acceptance because it is likely to generate a fruitful discussion in the community.
There might be certain weaknesses that the reviewer didn't notice but as of now there are none that feel worthwhile mentioning;
having said that, the reviewer has limited experience with complexity bounds in RL and maybe more experienced RL theory person will be able to identify the limitations of the analysis.
The reviewer is looking forward for the discussion phase.
Could you quantify the agreement between the (exact) action-value function corresponding to a random policy and a Q-network with randomly initialized parameters? If the agreement is high, it'd somewhat imply that for many domains the weights are initialized at a point that is almost an optimal solution; such a finding would be even more surprising.
The reviewer would suggest to take a look at the Policy Churn paper [1] that observes that the argmax of the learned action-value function in DQN-based methods changes in 10% of the states after each parameter update. The observation might be related to the high predictive power of the measure the authors observe.
[1] Schaul, Tom, André Barreto, John Quan, and Georg Ostrovski. "The phenomenon of policy churn." Advances in Neural Information Processing Systems 35 (2022): 2537-2549.
Perhaps the main limitation is that the effective horizon could be used for analysis purposes only; from the practical viewpoint, it is unclear how for a large MDP estimate efficiently the effective horizon (and thus obtain the sample complexity for the MDP).
No ethics review needed.
We thank the reviewer for their careful and thoughtful review. We appreciate that the reviewer found that our work offered “an interesting and non-trivial way to think about RL.”
**Difference between random policy’s Q-function and randomly initialized Q network:** In this work, we only focus on the Q-function of the random policy; we never analyze the Q function corresponding to a Q network with randomly initialized parameters. We would not expect a randomly initialized Q network to provide any useful information about which actions to take. We will clarify this point in the paper.
The Policy Churn paper is quite interesting and could help explain properties of deep RL algorithms related to their use of neural networks.
However, in this paper we show that (in deterministic environments) deep RL performs quite similarly to GORP, an algorithm which does not use neural networks at all. This suggests that one can gain significant understanding of deep RL without understanding its neural network components. We hope that future work like the Policy Churn paper will further explore how neural networks interact with deep RL.
**Limitations on the practical use of the effective horizon:** We agree that in many cases it is intractable to explicitly calculate the effective horizon. We argue in the Limitations section on line 408 that “despite this, [the effective horizon] serves as a useful perspective for understanding RL’s effectiveness and potential improvement areas.” We hope that practitioners can gain an intuitive understanding of the effective horizon from our paper that helps them use deep RL more effectively.
Thank you for the comments! I stick to my score and recommend the paper for acceptance.

Meta-review:
role: PC
data: comment: value: The reviewers unanimously agree that this is a very interesting work that takes a refreshing perspective on how and why deep RL works, and combines strong empirical evidence with theoretical explanation.

Decision: Accept (oral)