Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning
Authors: Daniel Palenicek
Michael Lutter
Joao Carvalho
Jan Peters
Venue: ICLR 2023 poster (2023)
Keywords: Model-based Reinforcement Learning
Value Expansion

Abstract: Model-based reinforcement learning is one approach to increase sample efficiency. However, the accuracy of the dynamics model and the resulting compounding error over modelled trajectories are commonly regarded as key limitations. A natural question to ask is: How much more sample efficiency can be gained by improving the learned dynamics models? Our paper empirically answers this question for the class of model-based value expansion methods in continuous control problems. Value expansion methods should benefit from increased model accuracy by enabling longer rollout horizons and better value function approximations. Our empirical study, which leverages oracle dynamics models to avoid compounding model errors, shows that (1) longer horizons increase sample efficiency, but the gain in improvement decreases with each additional expansion step, and (2) the increased model accuracy only marginally increases the sample efficiency compared to learned models with identical horizons. Therefore, longer horizons and increased model accuracy yield diminishing returns in terms of sample efficiency. These improvements in sample efficiency are particularly disappointing when compared to model-free value expansion methods. Even though they introduce no computational overhead, we find their performance to be on-par with model-based value expansion methods. Therefore, we conclude that the limitation of model-based value expansion methods is not the model accuracy of the learned models. While higher model accuracy is beneficial, our experiments show that even a perfect model will not provide an un-rivaled sample efficiency but that the bottleneck lies elsewhere.

Published as a conference paper at ICLR 2023
DIMINISHING RETURN OF VALUE EXPANSION METH- ODS IN MODEL-BASED REINFORCEMENT LEARNING
Daniel Palenicek
Michael Lutter
João Carvalho
Jan Peters 1234
ABSTRACT
Model-based reinforcement learning is one approach to increase sample effi- ciency. However, the accuracy of the dynamics model and the resulting com- pounding error over modelled trajectories are commonly regarded as key limita- tions. A natural question to ask is: How much more sample efficiency gained by improving the learned dynamics models? Our paper empirically an- swers this question for the class of model-based value expansion methods in con- tinuous control problems. Value expansion methods should benefit from increased model aceuracy by enabling longer rollout horizons and better value function ap- proximations. Our empirical study, which leverages oracle dynamics models to avoid compounding model errors, shows that (1) longer horizons increase sample efficiency, but the gain in improvement decreases with each additional expansion step, and (2) the increased model accuracy only marginally increases the sample efficiency compared to learned models with identical horizons. Therefore, longer horizons and increased model accuracy yield diminishing returns in terms of sam- ple efficiency. These improvements in sample efficiency are particularly disap- pointing when compared to model-free value expansion methods. Even though they introduce no computational overhead, we find their performance to be on-par with model- based value expansion methods. Therefore, we conclude that the lim- itation of model-basee learned models. While higher model accuracy is beneficial, our experiments show that a perfect model will not provide un-rivalled sample efficiency but that the bottleneck lies elsewhere.
be
value expansion methods is not the model accuracy of the
INTRODUCTION
Insufficient sample efficiency is central issue that prevents reinforcement learning (RL) agents from learning in physical environments. Especially in applications like robotics, samples from the real system are particularly scarce and expensive to acquire due to the high cost of operating robots. A technique that has proven to substantially enhance sample efficiency is model-based reinforcement learning (Deisenroth et al. 2013). In model-based RL, model of the system dynamics is usually learned from data, which is subsequently used for planning (Chua et al.) 2018 Hafner et al. 2019) or for policy learning (Sutton 1990, Janner et al., 2019).
Over the years, the model-based RL community has identified several ways of applying (learned) dynamics models in the RL framework. [Sutton s (1990) DynaQ framework uses the model for data augmentation, where model rollouts are started from real environment states coming from replay buffer. The collected data is afterwards given to a model-free RL agent. Recently, various improvements have been proposed to the original DynaQ algorithm, such using ensemble neural network models and short rollout horizons (Janner al. 2019 Lai et al. (020)), and improving the synthetic data generation with model predictive control (Morgan al. (021).
Feinberg et al. (0118) proposed an alternative way of incorporating a dynamics model into the RL framework. Their model-basee value expansion (MVE) algorithm unrolls the dynamics model and
Intelligent Autonomous Systems, Technical University of Darmstadt, daniel,palenicek @ teu-daammttddt.de
AL. Hochschulstr. 10, 64293 Darmstadt, Germany. 3German Research Center for AT (DFKI), Research Department: Systems AT for Robot Learning.
*Centre for Cognitive Science, Hochschulstr. 10, 64293 Darmstadt, Germany.

---

Published as a conference paper at ICLR 2023
discounts along the modelled trajectory to approximate better targets for value function learning. Subsequent works were primarily concerned with adaptively setting the rollout horizon based on some (indirect) measure of the modelling error, e.g.. model uncertainty (Buckman et al.] [018]] Abbas et al. 2020), a reconstruction loss (Wang et al. 2020) or approximating the local model error through temporal difference learning (Xiao et al. (019))
Lastly, using backpropagation through time (BPTT) (Werbos, 1990), dynamics models have been applied to the policy improvement step to assist in computing better policy gradients. [Deisenothh & Rasmussen (01T)) use Gaussian process regression (Rasmussen & Williams, (006) and moment matching to find closed-form solution for the gradient of the trajectory loss function. Stochastic value gradients (SVG) (Heess et al.) (015)) uses model in combination with the reparametrization trick (Kingma & Welling, 2014) to propagate gradients along real environment trajectories. Others leverage the model's 's differentiability to directly differentiate through model trajectories (Byravan et al. 2020, Amos et al. 202T). On a more abstract level, MVE- and SVG-type algorithms are very similar. Both learn dynamies model and use it for H-step trajectories (where / is the number of modelled time steps) to better approximate the quantity of interest - the next state Q-function in the case of SVG and the Q- Q-function for MVE. These value expansion methods all assume that longer rollout horizons will improve learning if the model is sufficiently accurate.
The common opinion is that learning models with less prediction error may further improve the sam- ple efficiency of RI. This argument is based on the fact that the single-step approximation error can become a substantial problem when using learned dynamics models for long trajectory rollouts. Mi- nor modelling errors can accumulate quickly when multi-step trajectories are built by bootstrapping successive model predictions. This problem is known as compounding model error. Furthermore, most bounds in model-based RL are usually dependent on the model error (Feinberg et al.) 2018).
and improvement guarantees assume model errors converging to zero. In practice, rollout horizons are often kept short to avoid significant compounding model error build-up (Janner et al., 2019). Intuitively, using longer model horizons has the potential to exploit the benefits of the model even more. For this reason, immense research efforts have been put into building and learning better dynamics models. We can diffcrentiate between purely engincered (white-box) models and learned (black-box) models (Nguyen-Tuooe over black-box models, as they are more interpretable and, thus, more predictable. However, most real-world robotics problems are too complex model analytically, and one has to retreat to learn- ing (black-box) dynamics models (Chua et al.) 2018: Janner et al. 20200.. Recently, authors have proposed neural network models that use physics-based inductive biases, also known as grey-box models (Lutter et al. 2019a b. Greydanus et al. 2019).
& Peters 2011). White-box models offer many advantages
While research has focused on improving model quality, a question that has received little attention yet is: Is model-based reinforcement learning even limited by the model quality? For Dyna-style data augmentation algorithms, the answer is yes, as these methods treat model- and real-environment samples the same. For value expansion methods, i.e., MVE- and SVG-type algorithms, the answer unclear. Better models would enable longer horizons due to the reduced compounding model error and improve the value function approximations. However, the impact of both on the sample efficiency remains unclear. In this paper, we empirically address this question for value expansion methods. Using the true dynamics model, we empirically show that the sample efficiency does not increase significantly even when a perfect model is used. We find that increasing the rollout horizon with oracle dynamics as well as improving the value function approximation using oracle model compared to a learned model at the same rollout horizon yields diminishing returns in improving sample efficiency. With the phrase diminishing returns, we refer its definition in economics (Case) & Fair 1999). In the context of value expansion methods in model-based RL, we mean that the marginal utility of better models on sample efficiency significantly decreases as the models improve in accuracy. These gains in sample efficiency of model-based value expansion are especially disap- pointing compared to model-free value expansion methods, e.g., Retrace (Munos et al. 2016). While the model-free methods introduce no computational overhead, the performance is on- an-par compared to model-based value expansion methods. When comparing the sample efficiency of different hori- zons for both model- -ased and model-free value expansion, one can clearly see that improvement in sample efficiency at best decreases with each additional model step along a modelled trajectory. Sometimes the overall performance even decreases for longer horizons in some environments.
2

---

Published as a conference paper at ICLR 2023
Contributions. In this paper, we empirically address the question of whether the sample efficiency of value expansion methods is limited by the dynamics model quality. We summarize our contri- butions follows. (1) Using the oracle dynamics model, we empirically show that the sample efficiency of value expansion methods does not increase significantly over a learned model. (2) We find that increasing the rollout horizon yields diminishing returns in improving sample efficiency, even with oracle dynamics. (3) We show that off-policy Retrace is very strong baseline, which adds virtually no computational overhead. (4) Our experiments show that for the critic expansion, the benefits of using a learned model does not appear to justify the added computational complexity.
H-STEP POLICY EVALUATION AND IMPROVEMENT
In this section, we introduce the necessary background and components of value expansion.
Reinforcement Learning. Consider discrete-time Markov Decision Process (MDP) (Puterman 2014], defined by the tuple (S,A,P, RR p. 7) with state space S, action space A, transition prob- ability 8t+1 ~ PP-1|s [8t, de), reward function r2 = R(s,, initial state distribution 8o ~ / and discount factor E [0, 1). At each time step / the agent interacts with the environment according to its policy и. Wherever is clear, we will make use of the tick notation and a' to refer to the next state and action at time t + 1. A trajectory T (so, do $1, ai,..... ST) is sequence of states and actions. The objective of an RL agent is to find an optimal policy that maximizes the expected sum of discounted rewards x² arg max, E. 1E>= r'r1]. Often æ is parametrized and, in the model-free case, is common to optimize its parameters with on-policy (Sutton et al.) 1999 Peters
& Schaal 2008 Peters et al. 2010 Schulman et al. 2015 2017: Palenicek| 2021] off-policy gra-
dient methods Degris et al.
2012 Silver et al. 2014 Lillicap et al., 2016 Fujimoto et al. 2018
2018). In model-based RL an approximate model P of the true dynamics is learned. and optionally an approximate reward function R. This model is then used for data generation (Sut- ton 1990; Janner al. 2019 Cowen-Rivers al. 2022), planning (Chua et al., 2018 |Hafner al., 2019 Lutter et al. 20222222 a b. Schneider et al. 2022) or stochastic optimization (Deisenroth & Ras-
Haarnoja et al.
mussen 2011
Heess al. 2015 Clavera et al. 2020 Amos et al., 202T).
Maximum-Fntrop Reinforcement Learning (MaxEnt RI.) augments the MDP's reward r, with policy entropy term, which prevents collapsing to a deterministic policy, thus implicitly enforc- ing continuing exploration (Zicbart et al.) 2008 Fox et al.) 2016 Haarnoja et al.) 2017). The (soft) action-value function (²*(s, = E,, 1'==o 7' a log π (o.) 80 5, ag al com- putes the expected sum of discounted soft-rewards following a policy æ. The goal is to maximize Jn Engep 1 - a log (ace(s)) [s)] w.r.t. т. Haarnoja al. (2018) have proposed the Soft Actor-Critic (SAC) algorithm to solve this objective. The SAC actor and critic losses are defined as
(r;
(D) = EEnD 1-V (s)) = E.DD a>T(111)) [a la - (s,,a)),
(I)
(D) = )'
7V" Q" (1++)"())-
(2)
where D is a dataset of previously collected transitions (replay buffer), and the next state value target V V²('') (s') = Ex'~<(119)) [Q= d') a log (a') s')]. Note, ty ( (s') is a target value. Thus, do not differentiate through as is common in deep RL. (Riedmiller 2005 Mnih et al. 2013).
Value Expansion. Through the recursive definition of the value- and action-value functions (Bell- man 1957). they can be approximated by rolling out the model for H time steps. We refer to this (single sample) estimator as the Hl step value expansion, defined as
Q" as) ((s,)) a) D 7t (r. a ((gl(()))))) + " (sH)
(3)
with (s, = (= a) and (s) = Ea--(-1s) [QT a) - a [a)s)) For = this estimator reduces to the function approximation used in SAC. It is important to note that these trajectories are on- oonpolicy. In practice, the SAC implementation uses a double Q f-function (Fujimoto) et al. 2018: Haarnoja et al. 2018), but to keep the notation simple we do not include it here. We further evaluate the expectation for the next state value function with a single sample, i.c. (=((') Qa (s', a') log (a''s8 with a' ~ o(( 8'), done in Haarnoja et al. (2018).
3

---

Published as a conference paper at ICLR 2023
Actor Expansion.
The H- H-spep value expansion can now be used in the actor update by incorpo- 3 in the actor loss (Equation !
rating Equation
J (D) Eg a--n(1ss) |a log क (als) (5 (s,a)),
which resembles the class of SVG algorithms (Heess al. 2015 Byravan et al.) 2020 Amos al.) 2021 Clavera et al., 2020). The difference to the original SVG formulation is the addition of the entropy term. Note that, for H 0 this definition reduces to the regular SAC actor loss (Equation I We will refer to the H H-step value expansion for the actor update as actor expansion (AE).
Critic Expansion. Incorporating value expansion in the policy evaluation step results in TD-A style (Sutton & Barto 2018, Schulman et al. (016) or Model-Based Value Expansion (MVE) meth-
ods [Wang et al. 2020
Buckman et al. 2018). The corresponding critic loss is defined as
JØ (D) E ((a,,))) a'-))(1ss) [r+ ( a') a log ((a'ss)) (s)
(5)
Similar to actor expansion, for H = 0 the critic update reduces to the SAC critic update (Fquation 2 We will refer to the H -tep value expansion for the critic update as critic expansion (CE).
Extension to Off-Policy Trajectories. We can naturally extend the action-value expansion in Equation 3 with the Retrace formulation (Munos et al.] (016), which allows to use off-policy trajectories collected with policy y * π (e.g. stored in replay buffer). The H-step Retrace state-action value expansion (single sample) estimator is defined as
Q (s, a) = a) + EE' 7t (co + +a-1V(s)) - aQ((s, ai)) + +7" CH-VV(ss)s. (6)
with the importance sampling weight e = II À Amin (1, (((s))//((a8s))))) and co = 1. We set 1 for the remainder of this paper. And by definition Qperaace (s, a) == Q7 (s. The Retrace extension has the desirable property that = 1 for on-policy trajectories (i.e. = T), and in that case Quéfaace reduces to the H-step action-value target Q" (Equation see Appendix for detailed derivation). Furthermore, for H/ 0, it reduces to the SAC target. The second desirable property is that trajectories from the real environment or dynamics model can be used. Hence, we use the Retrace formulation to evaluate all combinations of CE and AE with on- and off off-policy data and real and model-gen -eder-geeer data. Since Retrace encompasses all the variants we introduced, for readability, will from now on overload notation and define Q" a) == Q (s)
natural combination is to use actor and critic expansion simultaneously. However, we analyze each method separately to better understand their issues. Table I.. provides overview of related value expansion literature and displays further information about their usage of actor and critic expansion and on- and off- off-policy data.
EMPIRICAL STUDY
In this experimental section, we investigate the initial research question, how much will more curate dynamics models increase sample efficiency? In our experimental setup we replace the learned dynamics model with oracle model the environment's simulator which is the most accurate model that could be learned. In this scenario, no model error exists, which lets us exam- ine the impact of the compounding model error or rather, the lack of it. In these experiments, we specifically exclude data augmentation methods, where it is trivial to realize that they can arbi- trarily benefit from a perfect dynamics model. We conduct a series of empirical studies across the introduced value expansion algorithms on five standard continuous control benchmark tasks: Invert- edPendulum, Cartpole SwingUp, Hopper, Walker2d, and Halfcheetah] ! These range from simple to complex, with the latter three including contact forces, often resulting in discontinuous dynamics.
In the following subsections, we first present the premise of this paper - the two types of dimin- ishing returns of value expansion methods (Section B. Given the oracle model, we rule the existence of a compounding model error and, therefore, possible impacts on the learning perfor- mance. Next, turn to the generated H-step target values. We show that the increasing target
'The code is available at: http:://iitta.ccmmm/mmmaaaaaalll expasino

---

Published as a conference paper at ICLR 2023
InvertedPendulum
Cartpole
Walker2d
Halfcheetah
162
1.5
363
1.0
Robuu
1.5
Picuurn
0.5
0.0
RETRACE
1.0
Robum
1.5
Recuum
0.5
0.0
1.0
Rotum
0.25
0.50
0.75
1.00
0.0
0.5
1.0
0.25
0.50
0.75
1,00
0:25
0.50
0.75
1,00
0.25
0.50
0.75
1.00
Eme Steps
164
Ery Stepps
145
Try Steps
105
Ene Steps
les
Enr Steps
106
SAC
H-10
H-20
Hw30
Figure 1: Shows the diminishing return of critic expansion and actor expansion methods using mul- tiple rollout horizons H for SAC agent. CE does not benefit noticeably from horizons larger than 5 in most cases. There is clear trend that CE methods benefit from better models since the neural network model performs mostly on par with the oracle model. Model-free Retrace performances comparably. For AE (rows & 5), larger horizons can be detrimental, even with an oracle model. We plot the average episode undiscounted return mean (solid line) and standard deviation (shaded area) against the number of real environment interaction steps for random seeds. Some AE runs for larger H terminate early due to exploding gradients (See Section 3.3
variance caused by longer rollout horizons is not linked to the diminishing returns (Section Fi- nally, Section 3.3 look into the critic and actor gradients. We implement the learned model as a stochastic neural network ensemble, similar to PETS and MBPO (See Appendix A.3 for details).
3.1 DIMINISHING RETURNS ON CONTINUOUS CONTROL TASKS
Figure D shows the learning curves of the different value expansion methods for multiple rollout horizons and model types based SAC agent. Each model-based value expansion method, i.c., critic expansion (CE) and actor expansion (AE) is shown with an oracle and a learned model. The rollout horizons range from very short to long E [0. 1, 3, 10. 20, 30), where
reduces to vanilla SAC. Larger horizons H >>> 30 are not considered as the computational time complexity be- comes unfeasible (see Table A.3). 3. In addition comparing model-basee value expansion methods, Figure [ also shows learning curves of the model-free value expansion method Retrace.
Diminishing Return of Rollout Horizons. Within short horizons, less than 5 steps, we con- sistently observe that single increase in the rollout horizon leads increased performance in most cases. However, the rate of improvement quickly decreases with each additional expansion step (see Figure D. Close all variants of SAC reach their peak performance for more than 5 steps. the sample efficiency and obtained reward do not increase anymore and can even decrease. We refer to this phenomenon as the diminishing return rollout horizon. SAC-CE (1st & 2nd rows) performs better than SAC-AE (4th & 5th rows) for both oracle and learned dynamics models. While for shorter horizons, the performance is comparable, the performance SAC-AE degrades significantly for longer horizons. For SAC-AE, learning becomes unstable, and good policy can be learned for most systems using longer horizons. Thus, increasing rollout horizons does not yield unrivalled sample efficiency gains. As diminishing returns w.r.t. rollout horizon observed for the oracle and learned dynamics model, the reduced performance of the longer rollout horizon cannot be explained by the compounding model errors hypothesised by prior work.
5. For
5

---

Published as a conference paper at ICLR 2023
Diminishing Return of Model Accuracy. When directly comparing the performance of the learned and the oracle dynamics models, the learning curves are very similar. The learning per- formance of the learned model is only slightly reduced compared to the oracle model, which is expected. However, the degraded performance is barely visible for most systems when comparing the learning curves. This trend can be observed across most environments, algorithms and hori- zons. Therefore, the reduced model accuracy and the subsequent compounding model error do not appear to significantly impact the sample efficiency of the presented value expansion methods. We conclude that one cannot expect large gains in sample efficiency with value expansion methods by further investing developing more accurate dynamics models. This is not to say that poor dynam- ics models do not harm the learning performance of value expansion methods. Rather, we observe that overcoming the small errors of current models towards oracle dynamics will, at best, result in small improvements for value expansion methods on the continuous control tasks studied.
Model-free Value Expansion. When comparing the model-based value expansion methods to the model-free value expansion methods (Figure [ 3rd row), the sample efficiency of the model- based variant is only marginally better. Similar to the model-based variants, the model-free Retrace algorithm increases the sample efficiency for shorter horizons but decreases in performance for longer horizons. This minor difference between both approaches becomes especially surprising as the model-based value expansion methods introduce a large computational overhead. While the model-free variant only utilizes the collected experience, the model-based variant requires learning parametric model and unrolling this model at each update of the critic and the actor. Empirically the model-free variant is up to 15x faster in wall-clock time compared to the model-based value expansion at H = 5. Therefore, model-free value expansion methods are very strong baseline to model-base value expansion methods, especially when considering the computational overhead.
3.2 INCREASING VARIANCE OF Q"(ssa) DOES NOT EXPLAIN DIMINISHING RETURNS
One explanation for the diminishing returns could be the increasing variance of the Q-function tar- gets as SAC estimates Q" (s, a) by rolling out model with a stochastic policy. This unrolling can be seen traversing Markov reward process for H steps, for which it is well known that the target estimator's ' variance increases linearly with H, similar to the REINFORCE CE policy gradient algorithm (Pilug 1996 Carvalho al. (021). We also observed the increase in variance of the Q- function targets empirically when increasing the horizon. The experiments highlighting the increase in variance described in Appendix A.4 To test whether the increased variance of the value ex- pansion targets explains the diminishing returns, we conduct the following two experiments. First, perform the same experiments as in Se.Tion[3.]Jwitt DDPG. As DDPG uses deterministic pol- icy and the oracle dynamics are deterministic, increasing the horizon does not increase the variance of the Q-function targets. Second, we use variance reduction when computing the targets with SAC by averaging over multiple particles per target.
Deterministic Policies. Figure 2 shows the learning curves of DDPG for the same systems and horizons as Figure [ Since both the oracle dynamics and the policy are deterministic, increasing the horizon does not increase the variance of the Q-function targets. DDPG shows the same diminishing returns as SAC. When increasing the horizon, the gain in sample efficiency decreases with each ad- ditional step. Longer horizons are not consistently better than shorter horizons. For longer horizons, the obtained reward frequently collapses, similar to SAC. Therefore, the increasing variance in the Q-function targets cannot explain the diminishing returns.
Variance Reduction through Particles. To reduce the Q-function target variance when using SAC, we average the computed target Q" a) over multiple particles per target estimation. For SAC-CE, we used 30 particles and 10 particles for SAC-AE. The corresponding learning curves are shown in Figure D] Comparing the experiments with variance reduction with the prior results from Figure [] there is no qualitative difference in training performance. Increasing the horizons by one additional step does not necessarily yield better sample efficiency. As in the previous ex- periments, for longer horizons, the learning curves frequently collapse, and the performance dete- riorates even with the oracle dynamics model and averaging over multiple particles. Therefore, the increasing variance of the SAC Q-function targets does not explain the diminishing returns of value expansion methods.

---

Published as a conference paper at ICLR 2023
InveteePendullu
Cartpole
Walker2d
Halfcheetah
162
1.5
363
1.0
Robum
1.5
Petuurn
0.5
0.0
1.0
Robum
1.5
Retur
0.0
1.0
Rouum
0.25
0.50
0.75
1.00
0.0
0.5
1.0
0.25
0.50
0.75
1,00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Eie Steps
164
Bry Staps
145
Try Steps
105
Ene Steps
105
Enr Steps
106
DDPS
H--3
H-10
H-20
H-30
Figure 2: Similar Figure 1 this plot shows the diminishing return of critic expansion and actor expansion methods using multiple rollout horizons H but for a DDPG agent. We notice the same behaviours, where CE does not benefit noticeably from larger horizons in most cases. There is also no clear trend that CE methods benefit from better models since the neural network model per- forms mostly on par with the oracle model. Again, model-free Retrace performance comparable. Rows 4 & 5 show that for AE, larger horizons can be detrimental for DDPG well, even with an oracle model. We plot the average episode undiscounted return mean (solid line) and standard deviation (shaded area) against the number of real environment interaction steps for 9 random seeds. Some AE runs for larger terminate early due exploding gradients (See Section 3.3
InvertedRendulum
Cartpole
Hopper
Walker2d
Halfcheetah
1e3
1.5
SAC-GE (orale))
0.5
0.0
1e2
le2
1.5
1e3
1e3
1=3
SACAEE
(orarie)
1.0
0.5
0.0
1.0
0.0
0.5
10
0.5
0.5
1.0
0.5
1.
Enw Steps led
Env Steps
le5
Eny Steps
le6
Erw Steps 1e6
Env Steps
1e6
H-10
H=20
Figure 3: Variance reduction through particles for SAC-CE and SAC-AE experiments. Each Q- function target is averaged multiple particles to reduce the variance (30 particles for SAC- CE and 10 for SAC-AE). Compared to experiments from Figure [] we see that variance reduction gives the same qualitative training performance. Therefore, the increased variance of the Q-function targets is not the root cause of the diminishing returns in value expansion.
3.3 IMPACT THE GRADIENTS
Lastly, analyze the gradient statistics during training to observe whether the gradient deteriorates for longer horizons. The gradient variance for a subset environments and different horizons is shown in Figure 4 In the Appendix, the gradient mean and variance shown for all environments,

---

Published as a conference paper at ICLR 2023
Cartpole (oracle)
Cartpole (NN)
Halfcheetah (oracle)
Halfcheetah (NN)
SAC CE Cmtic
Grad Skdd
1.0
10"
SAC-AF Actor
Giad Std
10.2
0.0
0.5
1.0
0.0
0.5
1.0
0.5
0.5
1.0
Eny Steps
le5
Ery Steps
le5
Erv Steps
Env Steps
SAC
H=10
H=20
Figure 4: Critic and actor gradient's standard deviation for Cartpole and Halfcheetah and different rollout horizons with both an oracle dynamics model and learned dynamies model. (a) We observe that the SAC-CE critic gradients show low variance. (b) We see that the SAC-AE actor gradient variance explodes for Halfcheetah with longer horizons. this Figure, we clip values 10 which becomes necessary for visualizing longer horizons on the Halfcheetah due to exploding gradients.
including both SAC and DDPG. See Figure A.3 & A.4 for all SAC experiments and Figure A.5 & A.6 for the DDPG experiments.
Critic Expansion Gradients. The first row of Figure shows the SAC-CE critic gradient's stan- dard deviation (averaged over a replay buffer batch), over all dimensions of the Q-function parame- ter space, along the training procedure of SAC-CE. For all environments and horizons, the gradient mean and standard deviation appear within reasonable orders of magnitude (also compare Figure A.3 in the Appendix). Although, as expected, we observe a slight increase in the standard deviation with an increasing horizon for both the oracle and the learned neural network model. The mean is around zero (10-9) and the standard deviation between 10-3 and 10-1 Therefore, for both SAC-CE and DDPG-CE there is no clear signal in the gradient's 's mean standard deviation that links to the di- minishing returns of CE methods. The little difference w.r.t. horizon length expected because the critic gradient VqJ of the loss (Equation 5 is only differentiating through Q in the squared loss since Q (s', a') is assumed to be fixed target, which is a common assumption in most RI. algorithms EErst et al. 2005 Riedmiller 2005 Mnih et al. 2013).
Actor Expansion Gradients. The second row of Figure 4 shows the actor gradient's standard deviation (averaged over a replay buffer batch), over all dimensions of the policy parameter space, along the training process for SAC-AE. Contrary to CE, AE backpropagates through the whole rollout trajectory compute the gradient w.r.t. the policy parameters. This type of recursive com- putational graph is well known to be susceptible to vanishing or exploding gradients (Pascanu et al., 2013). Comparing rows four and five of Figure [ observe that a better dynamics model does not directly translate into faster learning or higher rewards since both the oracle and neural network model show the same effect lower returns for larger horizons. For SAC-AE we partially corre- late these results with the actor gradients in Figure 4 For example, the standard deviation of the Halfcheetah actor gradients explodes for horizons larger or equal to 3 for the oracle and 20 for the leamned models. This problem of exploding gradients can also be seen in the Hopper environment using learned dynamics (Appendix Figure A.4). This increase in variance can be correlated with the lower returns as SAC-AE frequently fails to solve the tasks for long horizons. A similar phe- nomenon can be seen in the DDPG-AE experiments as well, where the policy and oracle dynamics are deterministic. Therefore, increasing the horizon must be done with care even if better model does not suffer from compounding model errors. For AE, building a very accurate differentiable model that predicts into the future might not be worth it due gradient instability. For AE, it might only be possible to expand for a few steps before requiring additional methods to handle gradient variance (Parmas al. 2018).
8

---

Published as a conference paper at ICLR 2023
4 CONCLUSION & FUTURE WORK
In this paper. we answer the question of whether dynamics model errors are the main limitation for improving the sample efficiency of model-based value expansion methods. Our evaluations focus on value expansion methods as for model-based data augmentation (i.e. Dyna-style RL algorithms) approaches clearly benefit from perfect model. By using oracle dynamics model and thus avoiding model errors altogether, our experiments show that for infinite horizon, continuous state- and action space problems with deterministic dynamics, the benefits of better dynamics models for value-expansion methods yield diminishing returns. More specifically, we observe the diminishing return of model accuracy and diminishing return of rollout horizon. (1) More accurate dynamics models do not significantly increase the sample efficiency of model- value expansion methods, as the sample efficiency of the oracle dynamics model and learned dynamics models are comparable. (2) Increasing the rollout horizons by an additional step yields decreasing gains in sample efficiency at best. The gains sample efficiency quickly plateau for longer horizons, even for oracle mod- els, without compounding model error. For some environments, longer horizons even hurt sample efficiency. Furthermore, we observe that model-free value expansion delivers on-par performance compared to model-based methods without adding any additional computational overhead. There- fore making the model-free value expansion methods a very strong baseline. Overall, our results suggest that the computational overhead of model-based value expansion often outweighs the ben- efits of model-based value expansion methods, compared to model-free value expansion methods, especially in cases where computational resources are limited.
ACKNOWLEDGMENTS
This work was funded by the Hessian Ministry of Science and the Arts (HMWK) through the projects "The Third Wave Artificial Intelligence - 3AI and Hessian.AL. Calculations for this research were conducted the Lichtenberg high peefoperffomance computer of the TU Darmstadt. João Carvalho is funded by the German Federal Ministry of Education and Research (project IKIDA, 01IS20045).
REFERENCES
Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective dyna-style planning under limited model capacity. In International Conference on Machine Learning. 2020.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, 2021.
Richard Bellman. Dynamic Programming. Princeton University Press, I edition, 1957.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances Neural Information Processing Systems, 2018.
Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Ne- unert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gra- dients: Model-based policy optimization with tranferable latent dynamics models. In Conference on Robot Learning. 2020.
João Carvalho, Davide Tateo, Fabio Muratore, and Jan Peters. An empirical analysis of measure- valued derivatives for policy gradients. In International Joint Conference on Neural Networks, 2021.
K.E. Case and R.C. Fair. Principles of Economics. Prentice Hall, 1999.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, 2018.
Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmenter actor-critic: Backpropagating through paths. In International Conference Learning Representations, 2020.

---

Published as a conference paper at ICLR 2023
Alexander Cowen-Rivee Daniel Palenicek, Vincent Moens, Mohammed Amin Abdullah, Aivar Sootla, Jun Wang, and Haitham Bou-Ammar. Samba: Safe model-based & active reinforcement learning. Machine Learning. 111(1):173-20) 203, 2022.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. In International Conference on Machine Learning, 2012.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach policy search. In International Conference on Machine Learning. 2011.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1-141, 2013.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, In Inter- national Conference Machine Learning, 2018
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the Noise in Reinforcement Learning via Soft Updates. In Conference on Uncertainty in Artificial Intelligence, 2016.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL http:
//github. cotp:oool//oo///aa/./aa/a///
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, 2018
Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, 2019.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv: /a/../:/229.000 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Michael Janner, Igor Mordatch, and Sergey Levine. Gamma-models Generative temporal difference learning for infinite-horizon prediction. In Advances in Neural Information Processing Systems, 2020.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In Internationnl Confer- ence on Learning Representations, 2014.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, 2020.
10

---

Published as a conference paper at ICLR 2023
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference Learning Representations, 2016.
Michael Lutter, Kim Listmann, and Jan Peters. Deep lagrangian networks for end-to-end learning of energy-based control for under-actuated systems. In International Conference on Intelligent Robots and Systems, 2019a.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2019b.
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan, Gabriel Dulac-Aroldd, Piotr Trochim, Nicolas Heess, Josh Merel, and Yuval Tassa. Learning dynamics models for model predictive agents. arXiv preprint arXiv:2109. 14311, 2021a.
Michael Lutter, Johannes Silberbauer, Joe Watson, and Jan Peters. Differentiable physics models for real-world offline model-based reinforcement learning. In International Conference Roboties and Automation, 2021b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5002. 2013.
Andrew S Morgan, Daljeet Nandha, Georgia Chalvatzaki, Carlo D D'Eramo, Aaron M Dollar, and Jan Peters. Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning. In International Conference on Robotics and Automation, 2021.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
Duy Ngunen-Tuong and Jan Peters. Model learning for robot control: survey. Cognitive Process- ing, 12(4):319-340, 2011.
Daniel Palenicek. A survey on constraining policy updates using the kl divergence. Reinforcement Learning Algorithms: Analysis and Applications, pp. 49-57, 2021.
Paavo Parmas, Carl Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy
search robust to the curse of chaos. In International Conference Machine Learning, 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, 2013.
Jan Peters and Stefan Schaal. Natural actor-critie, Neurocomputing, 2008.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI Con- ference on Artificial Intelligence, 2010.
Georg Ch. Pflug. Optimizution of Stochastic Models. Springer, 1996.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Carl E Rasmussen and Christopher Williams. Gaussian processes for machine learning. The MIT Press, 2006.
Martin A. Riedmiller. Neural fitted Q iteration - first experiences with a data efficient neural rein- forcement learning method. In European Conference Machine Learning. 2005.
Tim Schneider, Boris Belousov, Georgia Chalvatzaki, Diego Romeres, Devesh K Jha, and Jan Peters. Active exploration for robotic manipulation. In International Conference Intelligent Robots and Systems, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.

---

Published as a conference paper at ICLR 2023
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Con- ference Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014.
Richard S Sutton. Integrated architectures for learning. planning, and reacting based approximat- ing dynamic programming. Machine Learning, 1990.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, 1999.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Junjie Wang, Qichao Zhang, Dongbin Zhao, Mengehen Zhao, and Jianye Hao. Dynamic hori- zon value estimation for model-based reinforcement learning. arXiv preprint (il::2009009993.. 2020.
P.J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 1851011 1990. doi: 1091/095..5337.
Chenjum Xiao, Yifan Wu, Chen Ma, Dale Schuurmans, and Martin Müller. Learning to combat compounding-erron in model-based reinforcement learning. arXiv preprint arXiv:/912. 1206, 2019.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Conference Artificial Intelligence, 2008
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
RELATED WORK
Table 1.1: Overview of key related literature. We categorize model-based RI. algorithms using value expansion in the actor and/or critic update, and further note which ones use on-policy trajectories resulting from a dynamics model and which use off- trajectories from replay buffer.
Expansion
Policy
Algorithm
Actor Critic
On
Off
SVG (Heess et al. 2015)
IVG (Byravan et af. 2020
SAC-SVG (Amos ct al. 2021
x
X
MAAC (Clavera et al. 2020)
X
MVE (Feinberg et al. 2018)
X
STEVE (Buckman et al. 2018
Retrace (Munos et al. 2016)
X
X
Dreamer
(Hatner et al.
2020
X
X
A.2 FROM RETRACE TO ON-POLICY H-STEP VALUE EXPANSION
In this section, we show how Retrace reduces the H-step value expansion target (Equation 3 in the on-policy case, with À 1. First, we perform a simple rewrite from the original Retrace formulation in Munos et al. (2016) such that it more in line with the notation this paper. Note that in Retrace, the rollouts are generated with policy μ.. While the value function V corresponds to the current policy т.
Q (50, ag) Q(s0: ao) + Eassy
Σ T' (R(ss,)))))))) + +VV (V(41)))) Q(s,, ai)) 1=
80: ao
= =((o, 00) -Raggug
Σ (R(s,, Q(ss,
+
80: ao
1=
H1
H+1
= ={{o0 on) + -Earm
Σ (R((((() - Q(., az))
+
Σ 7'04-1 V (8e)
80, ap
1=
= Q(80, on) + -Eary
7 Rooo an) (((0000)))) ao1)
H
1 (a((ss)))))) - Q(si, + Lva-Va
+ Earaa
+ Eaana
en V=
= Q(so: an) + -R(so, ao) Q(so:
+ +aara
(a(((s)))))) -
+ +1 H V (8H+1)
R((so 00) Bayag
Σ
+ + 11 CH V V (8u+1)
13

---

Published as a conference paper at ICLR 2023
Now, when À = 1 the importance sampling weights all become ea 1 and get
((00, ao) + Baray
1' (R(st, (s:) Qss,, at)) + "
H (R((s,,, + Ea (Q(s), (+) - log [ Q(5e, ae))
= R((o, an) + Engap
H
11
R(so, an) +
Σ (B,, (R(s,, Ba (((s, āe]]
Eas= la log (ā,]ss))] - En a,)l) + " ($n44))
Since тT the expectations over Q are identical and it further reduces to
Rsss, 0o) +
(BassT|R
(ogg (
= R(ss, 0o) + Earw
Σ (R((s) 0z) log ((;)((a))s)))) + (+
Finally, we that in the on-policy case with 1, the Retrace target reduces the regular SAC H-step value expansion target from Equation 3
A.3 EXPERIMENT DETAILS
This section provides detailed information regarding policy and Q-function networks, as well as the learned dynamics model. Further, include hyperparameters of the training procedure.
Policy Representation. The policy is represented by a neural network. consists of two hidden layers with 256 neurons each and ReLU activations. The network outputs mean vector and per- dimension standard deviation vector which are then used to parameterize a diagonal multivariate Gaussian.
Q-function Representation. To represent the Q-function we use double Q network. Each net- work has two hidden layers with 256 neurons each and ReLU activations and outputs a single sealar Q value. The minimum of the two Q networks is taken in critic and actor targets.
Learned Dynamics Model. duced in Janner et al. (2019). is ensemble of neural networks with 4 hidden layers with 256 neurons each and ReLU activations. To generate prediction, one of the networks from the ensem- ble is sampled uniformly at random. The selected network then predicts mean and log variance over the change in state. This change is added to the current state to get the prediction for the next state.
The learned dynamics model a reimplementation of the one intro-
Hyperparameters for Training. For a fair comparison, hyperparameters are shared across SAC and DDPG for the different expansion methods. In Table A.2 we report the hyperparameters across all experiments.
Computation Times. We report of the computation times for the reader get a better grasp of the computational complexity. Even though our GPU-only implementation using Brax (Freeman al.) (022)) is much more efficient than having a CPU-based simulator and transferring data back and forth to the GPU, the computational complexity of rolling out the physics simulator in the inner training loop, is still significant. This relates complexity in computation time as well as VRAM requirements. Table A.3 shows the computation times on an NVIDIA A100 GPU.
14

---

Published as a conference paper at ICLR 2023
Table A.2: Hyperparameters used for SAC and DDPG training. Note, that DDPG does not use the learning rate due to deterministic policy.
Inverted Pendulum
Parameter
Cartpole
Hopper
Walker2d
HalfCheetah
policy learning rate critic learning rate a learning rate target network т
3e 34
3e-4
5e 5
0.005
Retrace À
number parallel envs
128
min replay size
512
216
minibatch size
256
512
discount y
0.95
0.99
0.95
action repeat
Table A.3: Computation times of the training with oracle dynamics. The computational overhead dominates for short horizons already.
1
5
20
30
InvertedPendulumm
SAC-CE
6m
8m
10m
12m
18m
28m
42m
SAC-AE
6m
14m
30m
42m
1h
2h
3h
Retrace
6m
6m
7m
8m
10m
14m
18m
Hopper & Walker2d
SAC-CE
20m
1h20m
2h30m
5h30m
10h30m
20h20m
30h20m
SAC-AE
20m
9h 40m
35h
55h
114h
167h
278h
Retrace
20m
20m
22m
23m
27m
30m
40m
HalfCheetah
SAC-CE
40m
3h15m
5h 40m
13h
25h
49h
77h
SAC-AE
40m
30h
62h
100h
168h
Retrace
40m
41m
45m
50m
52m
1h
1h30m
15

---

Published as a conference paper at ICLR 2023
A.4 TARGET DISTRIBUTIONS AND INCREASING VARIANCE
As mentioned in the main paper, SAC estimates Qf (8, a) by rolling out a model with a stochas- tic policy. This be seen as traversing Markov reward process for H steps, for which is well known that the target estimator's variance increases linearly with H. At the same time, the value expansion targets with oracle dynamics should model the true return distribution bet- ter. In order to confirm this, we compare the true return distribution, approximated by particles of infinite horizon Monte Carlo returns, to the targets produced by different rollout horizons us- ing D = E,,anD Dw [Q (s, { p=).pp Q (s, a)} m))p=....p)] where Dw is the Wasserstein distance (Villani 2009), P 100 particles per target (to model the distribution over targets). (8) is the true Monte Carlo return to compare against (for computational reasons we use rollout horizon of 300 steps). Finally, we evaluate the expectation of this estimator over 104 samples (s,a) a) ~ D drawn from the replay buffer. We repeat this analysis for all horizons and 25 checkpoints along the vanilla SAC training. Figure A. 2nd row shows that with increasing horizons DW decreases i.e., as expected, longer rollouts do indeed capture the true return dis- tribution better. The 3rd and 4th row of Figure A.1 depict the expectation of the target values and the corresponding variance De [ (s, p]]. The expected values of the target distribution Q# are very similar for different horizons. This observation means that the approximated Q" -function already captures the true return very well and, therefore, model-based expansion only marginally improves the tar- gets in expectation. The 4th row shows, however, that the variance over targets increases by orders of magnitude with the increasing horizon (darker lines have larger variance).
particles over replay buffer batch Es,a~ [B[L (
InvertedPendulumm
Cartpole
Hopper
Walker2d
Halfcheetah
1+2
1+2
1+3
1+3
1ed
1.0
Return
2
0.5
1 10 12/0-10
100
10
101
10-1
10F
101
10P
10c
100
10a
100
104
10a
101
10"
10
100
10-1
10%
10P
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Eny Step5
365
Er Steps
3eG
Eny Steps
167
Env Steps
107
167
SAC
HaO
Ho3
H
Ho:00
H-20
H0
H=000
Figure A.I: Analysis of H Hstep targets for multiple checkpoints along the training process. The first row shows the undiscounted episodic return of SAC along training for seed. The checkpoints along this single training run are used for the analyses in the following rows. The second row shows the Wasserstein distance between the sample-based target distribution predicted by the H- step estimator and the true target distribution represented by Monte Carlo samples. Rows three and four display the mean and variance over particles of the sample-based target distributions. The mean targets for different horizons are roughly equal, while the variance increases with the horizon.
16

---

Published as a conference paper at ICLR 2023
A.5 ANALYSIS OF DISCOUNT FACTOR
Another hypothesis for the source of the diminishing return could be the discount factor 7, Fig- ure A.2 shows a line of initial experiments with ablation over different discount factors the Inverted environment. The phenomenon of diminishing returns is obvious in all experi- ments no matter the discount factor. We also do not see correlation between the diminishing return and the discount factor in these experiments. This leads us to believe that the discount factor is not the source of diminishing returns. Further experiments and formal investigation are left future work.
y= 0.9
Y = 0.95
V = 0.99
y= 1.0
1e2
1e2
1e2
1e2
1.5
SAC-CE (aracl))
1.0
0.25
0.75
1.00
0.25
0.50
1,00
0.25
0.25
0.50
Enw Stops 164
Env Steps
1o4
Env Steps
164
Env Steps
104
H=D H-1
H
H-10
H-20
Figure A.2: Ablation of different discount factors y on the InvertedPendullm. The discount factor does not appear to influence the diminishing returns and, therefore, the discount factor does not seem to be the source of diminishing returns.
17

---

Published as a conference paper at ICLR 2023
A.6 GRADIENT EXPERIMENTS
Figures A.3 A.4
A.5 and A.6 show critic and actor gradients' mean and standard deviation for the
different expansion methods of SAC and DDPG, respectively.
A.6.1
SAC CRITIC EXPANSION GRADIENTS
Inverted PeRDUM0
1e-3
Hw0
H-1
1--2
1.0
0.5
0.5
1.0
0.5
10
0.5
0.5
1.0
0.5
1.0
0.5
let
1ed
1nd
Invertes
5.0
2.5
Insarted BETRACE
0.5
0.0
11.5
Mo.ko
1.0
0.0
0.5
0.5
1.0
1.0.00
0.5
1.0.00
1m5
165
1w5
165
1w-1
Cansore
1.0
0.5
0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0.00
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
0.5
105
165
165
165
165
165
1e-2
1.0
0.0
0.5
0.0
0.5
1.0 0.0
0.5
1.0.0.
0.5
1.0 0.0
0.5
1.0
0.5
1.0 0.0
0.5
1.0
165
165
145
165
145
10-1
Hopper
0.5
0.5
1.0
0.5
1.0
0.5
0.5
1.0
0.5
1.0
0.5
1et
1eb.
1e6
1e6
16-1
1.0
Les
1.0
1.0
0.5
1.0
1.0
0.5
1sb
16b
1
1wb
1wb
14h
Valker
0.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1e-2
106
106
Le5
106
RETRACE
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
Hatthhotah
145
1wb
146
1w5
1w5
14h
-2
1--1
0.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
tarchodan
1et.
1e6
1e6
1e6
1eB
1e6
118
1.0
0.5
1.0
1.0
5.0
2.5
leb
Steps
Steps Steps
gradiont
gradient
Figure A.3: These plots show the critic's gradients mean and standard deviation of SAC-CE for all environments and all rollout horizons. The values are in reasonable orders of magnitude and, therefore, the diminishing returns in SAC-CE cannot be explained via the critic gradients.
18

---

Published as a conference paper at ICLR 2023
A.6.2 SAC ACTOR EXPANSION GRADIENTS
H=0
H=1
H=3
H=5
H=
H=
H=30
1e-2
Inwected
Penoulan
0.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
0.5
1.0
lel
1el
lel
1e-3
Inverted
Pendilum
0.5
0.5
1.0
0.5
1.0
0.5
1.0
9.5
1.0
0.5
1.0
Cartbols
-1.0
1.0.0.0
0.5
1.0
0.5
1,0.0.0
1.0 0.0
0.5
1.0
165
165
Carpols
0.5
0.5
1o-2
1.0
1.0
mosall
1.02
Hospor
0.0
-2.5
35.0"
0.5
HoRthooan
1c6
1.63
1.00
1.0
1.0
Hallchstin
Steps
Steps
165
gradient mean
grodiamt stdl
Figure A.4: These plots show the actor's gradients mean and standard deviation of SAC-AE for all environments and all rollout horizons. For readability, clip large gradients 102 (Halfchee- tah environment). Even though the gradients' mean have reasonable values (close to zero), except for the Halfcheetah environment, the standard deviation shows spikes and abnormal values for creasing rollout horizons, e.g., note the spikes in the standard deviation in the Hopper (NN model) experiments for horizons 20 and 30 (row last two columns), which go up the clipped value of 102. These increase in variance can be correlated with the lower return in Figure (row 5, col 3).
19

---

Published as a conference paper at ICLR 2023
A.6.3 DDPG CRITIC EXPANSION GRADIENTS
1e-5 H=0
H=1
H=3
H=5
H=10
H=
H=30
16-3
Insadee Pendiam
2.5
1.0
0.5
1.0
P Pardaku
WAP
Inverted BETRACE
0.5
0.5
1.0
1e-45
1ed
1.4.
1md
1nd
1ed
1st
0.0
0.5
0.5
1.0000
1.0.0.0
0.5
1.0 0.0
0.5
1.0
1.0.0 0.0
1e.55
1u5
1u5
1w5
165 1e-2
Carpooe (NN
0.0
0.5
1.0 0.0
0.5
1.0000
0.5
0.0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0.00
0.5
1.0
1e5
165
1e5
165
14h
1x-2
BETRACK
0.0
0.5
0.0.0
0.5
1.000
0.5
1.0.0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
165
1e5
1e5
165
165
165
0.5
1.0
0.5
1.0
0.5
1.0
0.5
Hospor moddel)
BETRACE
0.5
1.0
1.0
1m--5
1e5
1e
0.5
1.0
0.5
1.0
0.5
1ub
1
1w5
1ab
0.5
0.5
1.0
0.5
1.0
0.5
0.5
1.0
0.5
1.0
0.5
1.0
1e6
1e6
160
1ed
1e6
1e5
146
161
RETRICE
0.5
2.5
1.0
0.5
1.0
0.5
1.0
1.0
0.5
1.0
0.5
1.0
1e5
166
166
166
1e6
1e5
1e5
1g.0)
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
RETRACE
1.0
1.0
1e6
Steps 166
1e6
166
Steps
Steps
166
gradient
gradient
Figure A.5: These plots show the critic's gradients mean and standard deviation of DDPG-CE for all environments and all rollout horizons. The values are in reasonable orders of magnitude and, therefore, the diminishing returns in DDPG-CE cannot be explained via the critic gradients.
20

---

Published as a conference paper at ICLR 2023
A.6.4 DDPG ACTOR EXPANSION GRADIENTS
1e-5
H=0
H=1
H=3
H=55
H=10
H=
H=30
1.0
Insected
Penoula
0.5
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
0.5
1.0
Inverted
Pendilmm
lel
1e1
1ed
1et
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
1.0
0.5
0.5
1,0.0.0
1.0
0.5
165
Carpol
Hasp
1.0
Hospor
Wa Wallerzd
HoRthooan
1,63
Hallchssin
Steps
gradient mean
urad grodianttooo
Figure A.6: These plots show the actor's gradients mean and standard deviation of DDPG-AE for all environments and all rollout horizons. For readability, clip large gradients to 102 (Halfcheetah environment).
21


Reviews:
[{"review_id": "qixPH65RhJ", "reviewer": "Reviewer_vr7K", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors conduct an extensive empirical study using simulated control tasks on a family of model-based RL algorithms to answer the following question: how much more sample efficiency can be gained by improving the learned dynamics models. The results from the experiments are contrary to the common belief that regards the accuracy of the dynamics model as a limitation of model-based approaches. The experimental results show that learned models provide on-par improvements compared to perfect models and the improvements diminishes with increasing horizon. Furthermore, the off-policy method RETRACE that does not rely on dynamics models and thus has better computational complexity archives similar sample efficiency.  \n", "summary_of_the_review": "Due to the slight lack in significance as discussed above, ", "strength_and_weaknesses": "The results from the paper are extensive and are relevant to the model-based RL community. And the observations that a learned model achieves efficiency similar to a perfect model and RETRACE is a strong baseline for model-based algorithms are interesting. \n\nHowever, I think the empirical results are kind of expected. The authors focused on Q function estimation using a single trajectory trajectory (Eq (3)), which is known to suffer from large variance when horizon increases. Many existing methods are designed to specifically tackle this problem [1][2]. In the experiments, the authors can explore this direction by reducing the variance in the true and learned dynamics model and using these variance reduction methods under the value expansion framework. Furthermore, the phenomenon of diminishing returns can be simply attributed to the exponential decay due to the reward discount factor.\n\n\nWould the authors please clarify the sentence on In page 7, “we show that the critic gradients are probably not the source of the diminishing returns.”?\n\nTypo: second line in Introduction, samples one -> sample on\n\n[1] Grathwohl, Will, et al. \"Backpropagation through the void: Optimizing control variates for black-box gradient estimation.\" arXiv preprint arXiv:1711.00123 (2017).\n\n[2] Cheng, Ching-An, Xinyan Yan, and Byron Boots. \"Trajectory-wise control variates for variance reduction in policy gradient methods.\" Conference on Robot Learning. PMLR, 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FD_5IZeTpF", "reviewer": "Reviewer_5jBe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyses model-based RL methods and studies the impact of the model-based rollout horizon H. The main finding of the paper is that longer horizon do not guarantee better results (they have diminishing returns, and may even be detrimental). This is even true for oracle models (perfect models!), so it is not caused by compounding model errors.\n\nHalf the experiments focus on identifying the above effect (larger H is not always better), and the other half of the experiments try to find why longer horizons do not always perform well. No answer is found for that last question, and the paper concludes that future research needs to focus on why longer horizons do not benefit model-based RL.", "summary_of_the_review": "Clear information that needs to be available to the RL community. There is only a minor problem of whether model-based RL is limited in some way, or there is a bug somewhere.", "strength_and_weaknesses": "Strengths:\n\n- The paper contains information that clearly needs to be known by the RL community. Recent algorithms, such as Dreamer and MuZero, leverage models and do planning/rollouts in them. It is important to know that there is possibly a problem in RL that prevents these models, as good as they may be, from benefiting the agent.\n\nSmall weakness:\n\n- Some directions have been pursued as to why longer horizons are not always beneficial, but these directions did not bear fruit. The paper therefore identifies a problem without a direction for solving it.\n- The experiments are based on SAC and DDPG, with their losses modified to include model-based rollouts (in ways explained in the paper). It would have been very interesting to also see experiments with existing implementations of SOTA algorithms, such as DreamerV2, and with plugging the oracle model in them. This would have allowed to be certain that the diminishing return of H does not come from an implementation error in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9ILi2Q74EUD", "reviewer": "Reviewer_Q2WM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper has a clear focus, aiming to address the question \"is model accuracy a bottleneck to value-expansion methods in RL?\" The paper approaches this by studying the performance of two algorithm families, SAC and DDPG, in several continuous control tasks, using both a learnt dynamics model, and an oracle (exact) dynamics model.\n\n\n\n--- POST-REBUTTAL\n\nI thank the authors for their detailed response.\n\nMy initial review mentioned that this is a potentially impactful contribution that aims to address an important question in model-based reinforcement learning. I highlighted several reservations with the initial draft, including: (i) lack of architecture descriptions/hyperparameters/general concerns around reproducibility, (ii) the potential for further experiments to strengthen the evidence for the main hypothesis claimed in the paper, and (iii) the specificity of the findings to the agents/environments investigated in the paper.\n\nIn their response, the authors have included more qualifications in the paper regarding (iii).\n\nThe inclusion of the further DDPG experiments is a good step towards checking the bias-variance explanation of their findings with regard to (ii) as well; this definitely strengthens the paper in my view. I think there is still scope for further experimentation here, but this shouldn't be a barrier to acceptance in my view. In particular, the authors mentioned that evaluating the bias-variance trade-off further, or the impact of an inaccurate value function, by extending the roll-out horizon beyond H=30, was computationally intractable. The inclusion of the run-time table in the appendix makes this clear, and I think adds useful context to the experiments carried out in the main paper. However, using a smaller discount factor would make H=30 sufficient to essentially eliminate the bias from the inaccurate value function, and this could have been an interesting additional experiment to run.\n\nRegarding (i), the authors have included more details regarding agent hyperparameters, and the neural-network-based dynamics model. I think this is now at the level of detail where reproducibility is realistic. The paper could still be strengthened by including more details about any hyperparameter sweeps carried out e.g. in training the NN dynamics model.\n\nAside from these larger points, the authors' response also clarified many of the more minor queries I initially raised.\n\nBased on the factors above, I have increased my rating for the paper, and would argue for it be accepted.\n\nI think there is still scope for the clarity of the paper to further improved. One aspect in particular that came up in discussions with other reviewers and the meta-reviewer is that the messaging of the core take-aways of the paper are perhaps a bit unclear. In some sense, it is arguably not surprising that there are diminishing returns from increasing the rollout horizon (once H >> 1/(1 - gamma), for example, the bias of the value function has essentially been eliminated). In contrast, the observation that moving from a learnt model to an oracle model has limited impact on algorithm performance seems like a much more impactful take-away.", "summary_of_the_review": "In my view, this is an interesting paper that tackles an important question. I think it has the potential to be impactful in the RL community, but I currently see several issues that prevent me from recommending acceptance, including (i) the lack of clarity in the way the results are communicated in Section 3.3, which are crucial to the main findings of the paper, (ii) a discussion of the generality of the findings, and (iii) full experimental details. I would be open to potentially adjusting my rating in response to the authors' rebuttal.", "strength_and_weaknesses": "This is an empirical paper, which seeks to address a common belief in model-based RL: that model accuracy is a bottleneck to performance in value-expansion methods in RL, prohibiting us from doing longer-horizon expansions. The singularity of the paper's focus is a strength, and I found the work to be clearly presented. Papers such as this have the potential to be highly useful to the community, in carefully examining commonly-held beliefs, and impacting the practice of model-based RL in the future. However, I do have several comments on areas where I believe the paper currently falls short, which is the basis for my current rating. I would be open to revising this rating based on the authors' response.\n\n_Scope_\n\nI appreciated the inclusion of several different agents, and several environments. While it is always possible to include more comparisons in a paper such as this, and a line needs to be drawn somewhere, I think there are a few important comparisons which, even if not included in the paper, should be discussed.\n\nDeterministic policies. Although the paper considers the DDPG agent in Figure 1, this agent does not seem to be investigated further in the paper. However, I think analyzing DDPG gradients in a similar way to the SAC analysis would be an interesting additional data point for the paper. If the DDPG policy and model are deterministic, this removes all stochasticity from the gradients, and therefore potentially provides a more nuanced answer to the question: \"do diminishing returns stem from increased gradient variance\", which is not currently answered conclusively in the paper.\n\nError in bootstrapped value function. As far as I can tell, all bootstrapped value functions used are those learnt by the SAC agents. Since one of the main hypotheses as to what drives the change in performance as a function H is a bias/variance trade-off for gradient estimation, it would have been interesting to see the effects of more/less accurate bootstrap value estimates on performance.\n\nOther types of environment. The paper focuses on deterministic, continuous control tasks. I would not necessarily expect experiments beyond this in a conference paper, but some discussion as to whether the authors expect the findings to generalise beyond this would be very valuable. For example, could stochastic transitions affect the bias/variance trade-off and lead to very different performance as a function of H.\n\nFurther values of H. The largest value of H considered in the paper is 30. With a discount factor of 0.99, this still attaches a weight of 0.99^30 = 0.74 to the bootstrapped value function. It would have been interesting to see results with value expansion closer to Monte Carlo simulation (i.e., with a value of H so that gamma^H is reasonably close to 0).\n\nTo emphasize, I don't think it's necessary to run further experiments relating to each of these comparisons (although doing so would obviously increase the impact of the paper, and the weight of any conclusions drawn), but I think discussion, and possible qualification, of the paper's findings with these comparisons in mind would strengthen the paper, and make it more useful for the community in general.\n\n\n\nSome additional comments for each section are provided below\n\n_Section 2_\n\nIn the definition of Markov decision processes given here, some non-canonical choices are made: state/action spaces as subsets of R^n, transitions having probability density functions (which is actually not satisfied by the (deterministic) continuous control tasks considered later in the paper). How important are these assumptions?\n\nEqn (2): Should there be a stop gradient around V^pi(s') in this case?\n\nJust below Eqn (6), the authors claim that when mu = pi, the RETRACE estimate reduces to the H-step action-value target. This is not true, because c_{t-1}V_t - c_t Q_t, which appears in Eqn (6), does not evaluate to 0.\n\n_Section 3.2_\n\nI am unsure exactly what is intended to be taken away from the Wasserstein comparison. It is clear that the distance should go to 0 as H increases, but it is not clear to me why these Wasserstein distances are of direct importance to studying the source of diminishing returns.\n\nThe comparison of estimator variance as H varies feels more related to the topic of the paper, but further discussion would be valuable here: while variance increases with H, is the increase in variance correlated with the extent of diminishing returns observed?\n\n_Section 3.3_\n\nI found the results of this section very difficult to interpret. As far as I can see, the authors do not define the mean and standard deviation of the gradients here (is this on a per-coordinate basis?) Additionally, the scale for the gradient means the y-axis seems too large, so that the curves essentially look like the line y=0 in most panels.\n\nThe authors reach the conclusion that gradient variance may contribute to the poor performance of large horizons in actor-expansion methods, but is not likely to be the cause of diminishing returns in critic-expansion methods. More discussion of these results would strengthen the paper (what makes the variance so catastrophic in some environments like HalfCheetah, and not in others)? If gradient variance doesn't explain diminishing returns for critic-expansion methods, what alternative hypotheses should be investigated further?\n\n_Model-free vs. model-based comparison_\n\nOne of the main takeaways that the authors present is that the model-free expansion based on RETRACE is competitive to model-based methods. I agree with this conclusion when the number of learner steps is taken to be equal in both cases, but presumably one of the main advantages of using model-based expansion is that if the computational budget is available, more updates can be performed by sampling new experience from the model (while RETRACE is constrained just to use collected trajectories). This aspect of the comparison seems to be absent from the paper currently, and I think should at least be mentioned as another possible reason to prefer model-based value expansion over model-free methods.\n\n_Other comments_\n\np1: \"samples one\" -> \"samples from\"\nThere are a few instances of citations to recent papers for well established concepts e.g. Deisenroth et al. (2013) for model-based RL, and (Peyré & Cuturi) (2019) for the Wasserstein distance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "16TtOdvaY-n", "reviewer": "Reviewer_kpea", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of how model accuracy relates to sample efficiency a class of model-based reinforcement learning algorithms, that is, model-based value expansion algorithms. The paper conducts experiments to show that the improvement of using the oracle dynamics model over a learned dynamics model is not too much, and there is a phenomenon of diminishing return for both, which means short horizons suffice to achieve the best sample efficiency, while model-free baselines are only slightly worse. Thus, the authors conclude that model-based value expansion algorithms are not limited by the accuracy of the learned dynamics model. ", "summary_of_the_review": "My major concerns with this paper are that the variance of the $Q$ value estimation is not taken into consideration. The writing can be more precise and less confusing. I don't recommend acceptance in the current form but am glad to increase my score if my concerns can be addressed. \n\n--- \nUpdate: As the authors provide more experiments on the bias-variance trade-off, addressing my major concern, I'd like to increase the score to 6. ", "strength_and_weaknesses": "Strength:\n\n- The preliminary section covers many components and greatly helps the reader to understand the background. The paper is well organized. \n- The problem this paper studies is indeed important. Diagonizing the bottleneck for model-based reinforcement learning algorithms can be quite difficult. \n\nWeakness:\n\n- There are a lot of missing details in Section 2, which makes it difficult to understand the exact algorithm this paper studies:\n  - How is $V$-function in Eq (3) evaluated? Is it a different network than $Q$, as in [1], or simply sampling an action $a' \\sim \\pi(\\cdot| s')$ and using $Q^\\pi(s', a') - \\alpha \\log \\pi(a' | s')$ as the result, as in [2]? More importantly, how is $V$-function evaluated in Eq (6)? \n  - The Retrace target $Q^H_\\text{retrace}$ doesn't encompass Equation (3): it doesn't take the entropy of the policy into consideration. \n  - If $\\lambda \\neq 1$, $Q^H_\\text{retrace} \\neq Q^H$ and the paper doesn't provide the used $\\lambda$. \n- The paper only rules out the possibility of compounding error, but there are a lot of other factors which might affect the efficiency of model-based expansion. An important topic in estimating the value functions is how the bias-variance trade-off is taken into consideration. As I didn't see the paper makes effort to reduce the variance of $Q$ value estimation, I assume this paper only considers Monte-Carlo estimation, which is known to suffer from high variance. Retrace, which utilizes $V$-function to reduce variance and serves as a baseline in this paper, is another method to estimate the $Q$ values. One even simpler baseline is to average the $Q$ values over a few trajectories starting from the same $(s, a)$, given that the samples from the model are \"free\". Consider an extreme scenario, where we use the exact expectation of $Q^{\\pi, H}(s, a)$ (so that the variance is reduced to 0) to do action-expansion or value-expansion. If the phenomenon of diminishing returns still exists in this scenario, I'll be convinced. The conclusion that I can draw from the paper is that with Monte-Carlo estimation of $Q$ values, an oracle dynamics model with a longer horizon might hurt. \n- Even in the case we're using an oracle dynamics model, the diminishing return phenomenon is not surprising either. One can expect that the distribution of $Q^H(s, a)$ converges to $Q^\\infty(s, a)$ at some rate when $H$ goes to infinity. (Indeed, this is what Figure 2 wants to show.) So even with a perfect $Q$ estimator with zero variance, the diminishing return may still exist. It will surprise me if the performance will drop, though. \n- Maybe experiments show that 100 particles suffice to evaluate the Wasserstein distance between $Q^H(s, a)$ and $Q^\\infty(s, a)$. Please also report the mean and variance of $Q^\\infty(s, a)$ and $Q^H(s, a)$ so that the readers can get a sense of what the Wasserstein distance means. \n- The author \"conclude that the limitation of model-based value expansion methods is not so much the model accuracy of the learned models.\" However, in Figure 1.a,  SAC-CE (oracle model) performs significantly better than SAC-CE (NN model) in HalfCheetah, so the accuracy of the learned models **can** be a problem and can be an even greater problem than the choice of horizon under certain scenarios. What I can agree with is that it's not the only limitation. \n- This paper doesn't have any theory and doesn't find the root causes for the phenomenon either. \n\n\n\n[1]: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja. etc.\n[2]: Soft Actor-Critic Algorithms and Applications, Haarnoja. etc. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster