Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
Authors: Chenjun Xiao
Yifan Wu
Chen Ma
Dale Schuurmans
Martin Müller
Venue:  (2020)
Keywords: reinforcement learning
model-based RL

Abstract: Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the model errors get infeasibly large. In this paper, we investigate techniques for choosing the planning horizon on a state-dependent basis, where a state's planning horizon is determined by the maximum cumulative model error around that state. We demonstrate that these state-dependent model errors can be learned with Temporal Difference methods, based on a novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy,  significantly improving the efficiency of policy learning compared to model-based and model-free baselines.


Under review as a conference paper at ICLR 2020
LEARNING TO COMBAT COMPOUNDING-ERROO IN MODEL-BASER REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is in- accurate. An algorithm should ideally be able to trust an imperfect model over reasonably long planning horizon, and only rely model-free updates when the model errors get infeasibly large. In this paper. we investigate techniques for choosing the planning horizon on a state-dependent basis, where state's planning horizon is determined by the maximum cumulative model eπгоr around that state. We demonstrate that these state-dependent model errors can learned with Tem- poral Difference methods, based on novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy, significantly improving the efficiency of policy learning compared to model-based and model-free baselines.
INTRODUCTION
Model-free reinforcement learning (RI) aims to learn an effective behavior policy directly from interaction with black-box environment. This approach has recently achieved great success, particularly in game playing (Mnih et al. [015 Moravčík et al. 2017 Silver et al. 2016)] 2017). Unfortunately, model-free RL techniques deployment infeasible whenever data collection expensive. A key challenge remains to improve the sample efficiency of general purpose RL methods.
hampered by poor sample effiiency, which makes their
By contrast, model-based RL attempts to learn a model of an environment from direct experience collected during training. A learned model can be either directly combined with planning algorithm (Hafner et al.) 2018 Sutton 1990), or applied to improve the target values for model-free RI. (Buckman et al. 20188 Feinberg al. 2018). Model-based RL is often thought be more sample efficient than model-Iree approaches (Sutton & Barto 2018). Recent theoretical work confirms this intuition by showing that there exist environments where model-based approaches can be exponentially more sample efficient than any model-free approach (Sun et al. 2018)).
The performance of model-based RL heavily relies on the quality of the model a learning agent can acquire. When an accurate model is given or can be learned with relatively little experience, model-based RI. can be significantly more data efficient than model-free approaches. However, in noisy and complex environments, learning accurate model can be challenge. In such cases, model errors can compound and render the model useless for planning, which can lead to catastrophic failure of model-based RL. Although having accurate model in complex environment can be unrealistic, it is sometimes possible to obtain a model that is accurate in local subsets of the state space. For example in robotic control tasks, environment interaction can be much easier to model than the dynamics of interaction with other objects. In such cases, if pure model-based approach would fail, one might still expect to gain advantage over model-based approaches by exploiting the accurate parts of the model.
dynamics that do not consider external
One potential advantage of model-based RL is that a longer planning horizon can be considered by rolling out the model for multiple steps. Ideally, with an imperfect model, one would like principled approach for adapting the planning horizon at different states, in order to overcome the compounding error problem of model- based RL. When the model has large error around some states, the learning agent should trust the model less by using a small planning horizon.

---

Under review as a conference paper at ICLR 2020
On the other hand, for states where the model near optimal, a large planning horizon should be adopted. To implement this idea, pro- vide a few key observations that are essential propose: First, charac- terize the error in a multi-step learning target under an approximate model as multi-step dis- counted cumulative model error. Second, we show how the cumulative model error for dif- ferent planning horizons can be learned based on TD-learning. Finally, we introduce Adap- live Model-based Value Expansion (AdaMVE), an extension to Model-basee Value Expansion (MVE) Feinberg et al. (2018) that adaptively selects planning horizons for each state, based on the learned accumulative model errors. To illustrate, Fig- D provides an example of how AdaMVE works in FourRoom gridworld maze with an imperfect model. This example shows that A can successfully adapt the plan- ning horizon for different subsets of the state space, and significantly outperforms both MVE and model-free baselines. We evaluate our method on both gridworld mazes with different imperfect models as well as continuous control tasks. The experimental results suggest that the adaptive horizon algorithm can significantly alleviate the compounding error problem model-based RL, while exploiting its advantage in terms of sample efficiency.
for the methods
16
www
(a) Adaptive Horizons
(b) Policy Learning
Figure 1: in FourRoom with an imperfect model. The model is perfect in three rooms while totally wrong in the left bottom room. Non-adaptive MVE diverges due to the large model errors (right). In contrast, AdaMVE is able to adapt the planning horizon at different state (see (a), darker color means longer planning horizon). outperforming both the model- based and model-free baselines.
Illustration of adaptive planning horizon
2 BACKGROUND
Consider a Markov Decision Process (MDP) Sutton Barto 2018 M (S, where S is the state space, A is the action space, P(- a) Is the transition probability distribution function, R(s, a) is the reward function and 7 is the discount factor. We also assume that each state is represented by feature vector E Rdd The goal is to find policy TT-ss) that maximizes the cumulative discounted reward starting from any state E S. Let p*((ss) denote the induced transition distribution for policy æ. For later convenience, we also introduce the notion of multi-step transition distributions as PT,, where P7 denotes the distribution over the state space after rolling out pr for t steps starting from state s. For example. PT (-1s) is the Dirac delta function at s and PT (-|s) (-[s). We use R (s) to denote the expected reward at state s when following policy ie. R (a)))()s) [R(s, a)). The state value function defined by
(*(s) = - r== E'E PT (s) [R (sc))
The action-value function (a.k.a. Q-function) can be written as
Q (s a) = R(s + ( P(
(s')s
The optimal policy is define as the policy N that maximizes (*))) at all states a E S. Such policy always exists (Sutton& & Barto 2018).
Model-based reinforcement learning approaches explicitly make use of dynamics model p P of the environment to compute the optimal policy, while model-free approaches learn the optimal policy without explicitly modeling (e.g. directly learning the action-value functions from samples). Throughout the paper we assume that the reward function R is known to the agent, hence "model" refers to an estimated transition dynamics
2.1 MULTI-STEP MODEL-BASED VALUE
One potential advantage of learning model is to compute multi-step target value by iteratively rolling out the model. that is, to take the predicted state of the model and feed in again as the state input, projecting sample state time steps later, and so Formally, for any policy n, given
2

---

Under review as a conference paper at ICLR 2020
planning horizon H, reference (target) value function V, and an approximate (e.g. learned) model P, the H H-step model-based value is defined as
H-1
P.H
(8)
Pp y S
(R + " PD~Pp Pg) [V(#))]
(1)
1=0
This value can be integrated with model-free methods in different ways based on the policy For example in AlphaGo Zero, the H-step optimal lookahead policy कir argmax, V P. (s) combined with proper exploration strategy is used as the behavior policy of the learning agent, where Ti is approximated by Monte Carlo Tree Search (Silver et al., 2017).
Model-based value expansion (MVE) is another example of utilizing objective (I) (Buckman et al. 2018; Feinberg et al. 2018). MVE applies the learning agent's current policy as the rollout policy to (8),
obtain (z), which is used as the update target value for TT Learning. For example in Q-learning
(Watkins & Dayan 1992), given sampled transition (s, r, r')') and target Q-value function (s,,a), one can replace the target value V(s') = max, Q a) with the multi-step estimate V (5') or a mixture of such estimates with different values of H. The rollout policy æ can be greedy with respect to Q. Note that when H 0, there no rollout hence Vs'( Vs'),, which recovers the model-free update target.
When the model is perfect, MVE can reduce the biases of the targets, leading to improved performance model-free methods (Feinberg al. 2018). However, the major limitation of MVE is that the rollout horizon H needs to be tuned in task-specific manner: in complex environment where the model difficult to learn, smaller rollout horizon usually performs better than a larger one. To overcome this drawback, Buckman et al. propose stochastic ensemble value expansion (STEVE), which applies stochastic ensembles both over multiple models and rollout horizons to choose the best H dynamically (Buckman et al. 2018)
2.2 WASSERSTEIN DISTANCE
The Wasserstein distance a distance metric between two distributions. Its dual form is defined as follows (Villani 2008):
W ((p.
sup Egep [g(2)] Egnu [9(2)]
(2)
1 $1
where |ااو| the Lipschitz constant of function g. If both p and g are Dirac delta functions (i.e. de-
terministic) denoted by 6z, and 6z, respectively, ((p, q) 1| 2g is just the Euclidean distance.
If only one of the distributions is deterministic, c.g. q 62,, then W (p. q) Ex~p [[] 1 2g ll2]
This can be checked directly from the primal form observing that g(2) =| = achieves the superimum in the dual form Villani 2008).
LEARNING MULTI-STEP MODEL ERROR
To exploit the advantage multi-step value estimation while alleviating the negative impact using an imperfect model, we propose to adapt planning horizons H such that the H -teeep expanded value using the approximate model is close to the one obtained using the true model. Specifically, for policy π, consider the h-step model-basee basel value error for an approximate model P defined by
E(hlx,s,V,P) V (s) VP
We aim to select an appropriate planning horizon for state s based on this error. Since the accuracy of an approximate model could vary in different subspace of S, selecting planning horizons in a state dependent way is particularly desirable in practice.
Given state exactly computing error 3 is unfeasible model-based based value due to the inaccessibility of the true model P. To overcome this problem, we propose practical algorithm to learn the value expansion error for each state approximately, based on the observation that the value expansion error defined in (3) can characterized by the discounted accumulative model error. The following theorem states such connection.
cannot directly compute the multi-step
3

---

Under review as a conference paper at ICLR 2020
Theorem 1. Given any policy n. approximate model P, and reference value function V, for planning horizon H we have
((H = (s) (T) S K
H (T (s))
(4)
where W (s) = -Eawx(-1a)
VT (W a)] with W (s. ((((sss a), P(-ss, a)) being the Wasserstein
distance and K = supp all possible horizons.
is the maximum Lipschitzness of the estimated value function over
The H-step discounted cumulative model error (RHS of ( can be viewed as finite-horizon RL objective. We can define a new MDP to learn this error, MM AS,A, P. W, 7), where the state and action space, the transition function, and the discount factor remain unchanged from the original problem, the W-reward function W(s.0) a) = W (PP(-ls, a), P(-|s, a)) defines the "reward" of the MDP. With Mg we define new state-value function named h-step state model error function, which measures the expected cumulative model error if the agent starts in state s and follows some policy x,
Ex (b, 1)) = { Etr (B (s+)
0 {
h 0
(5)
h > 0
According to 4). (s. h) is an upper bound of the h-step model-based value error 3 constant. Similarly, the ht-step action model error is defined by
up to
{
0
h = 0
8xH h*
W(s, + 7 (
h > 0
(6)
Note that {s, h) Eaws (s, h a)) by the definition. Learning the /t-sepp model-based value error under some policy π now becomes traditional policy evaluation problem in RL with a different reward function. In this paper we use mode-free methods to learn the model error function.
3.1 LEARNING h-STEP CUMULATIVE MODEL ERROR
We now introduce principled method to learn the cumulative model error based on finite horizon Bellman updates. Since MR,s only differs from the original MDP in the reward function, we can directly learn the h-step model error using the transition data sampled from a relay buffer. As discussed in Section 2.2 when either the ground-truth or the approximate transition is deterministic, the W- reward is just the expected Euclidean norm between the real and predicted next state. For stochastic transitions, W-reward can be approximated by learning the g function in ( as suggested in the Wasserstein GAN (Arjovsky et al. 2017). Therefore, given a sampled transition data (st, Qt. Pt, St+1), we can directly compute the W-reward for learning. In addition, since the policy π used for computing mode-based value is non-stationary during learning, which may or may not be available at the time of evaluating the value expansion errors, we measure the value expansion error using a reference policy म. The choice of will be discussed later.
We use TD-learning to train the model error function. In particular, given data sample (s, a, (,,,, a,s,s,, E is updated by minimizing the one step Bellman error
1 { [ 7E(''' a', - - E(s, a)) E 2
min
where a' is selected using the policy म. and E is the target model error value function. It is important to note that this learning process can be combined with any model-based algorithm where replay buffer is used. Also, our method does not introduce additional sample complexity, since the data used to train can be sampled from the replay buffer.
Choice of the Reference Policy. Since the policy used for value expansion is changing during the learning process and the cumulative model error is policy dependent, it is expensive to retrain the model error for the current policy at every step. Thus, choose the reference policy मे to "prepare for the future". There are several possibilities but we consider three that allow model error to be efficiently learned with samples from the replay buffer.
4

---

Under review as a conference paper at ICLR 2020
1. The conservative reference policy argmax, E(h/T, 8, V. targets the maximum model error. When using the learned model error to decide a proper planning horizon, this policy allows us to consider the worst
model error, making the selected horizon more robust in practice. To learn the model error under this policy, we can use a' argmax, E(s',a, a h 1) when doing the update (7). This can be viewed as an extension of Q-learning (Watkins & Dayan 1992 to learn the maximum model error.
2. The greedy reference policy selects argmax, ((s, a), where Q is the current target Q-value function of the learning agent. This policy tries to measure the model error under the learning agent's 's current behavior.
3. The replay buffer reference policy selects an action which occurred in the replay buffer at 8. This policy can be considered as a mixture policy of previous agent's 's behaviors. As we only care about the "value function" E* (s. h) and every sampled transition (s, a, r, s') can be viewed as "on-policy" under the replay buffer policy, we can directly estimate E (s,s, h)) by on-policy TD learning: ming z (s, a) + - 1) - h)
3.2 ADAPTIVE PLANNING HORIZON USING MODEL ERROR
In this section, we introduce the Adaptive Model-based Value Expansion (AdaMVE) algorithm, as an example of using learned multi-step model error function adapt planning horizon for different states. Instead of applying a fixed horizon tuned for different environment when computing ① as in MVE, AdaMVE attempts adapt the rollout horizon for different states according to model error function learned as described in the previous section. We suppose that the learner's behavior policy æ determined by Q-values. For discrete domain, π argmax (s,, a). For a continuous domain, π is trained to approximate the greedy policy over Q(s, a) as in DDPG (Lillicrap et al. 2015).
In AdaMVE, we aim to find a proper planning horizon H (s) E [0, Hmax] for any state s E S I For state S, all rollout horizons h E [0, Hmmx]. Although & can be viewed as a good proxy for & in 4.. it is still diffieult to directly find an appropriate planning horizon by setting hard threshold, due to the unknown Lipschitz constant K in Theorem I To resolve this issue, instead of setting a hard threshold to get a maximum rollout horizon H (s), we use a soft weighted combination over all horizons h S Hmax- For horizon h with higher model error É(s, h) set smaller weight wh. More specifically, we define the weights according to a "softmax policy"
the learned model error function & to produce the model error É h) for
w((ls) exp (-((s,))/T}
where T is temperature parameter. AdaMVE uses this policy as its weighting function mix the expansion values (I) of different rollout horizons,
Haus
T P. Huss
(8)
Σ w(Nls) (s). h=0
To give scalar measure of the "planning horizon" when using soft combination, for each state we define its weighted average horizon as (s))) = EHen wh)ss) # h, which is the expected rollout
horizon under "softmax policy".. When the model error is zero everywhere, whh[s) = 11/(Hms [Hmax + 1) and V (s) is the average value estimates of all horizon. The average planing horizon is H(s) = p.
Hmnx/
/2. /2. When the model error is infinitely large everywhere, (0)ss) = 1 and (his) 0 for all thus no rollout is being considered. The average planing horizon is [ss) = 0.
The combined value estimate (9) is used as the learning target to update Q(s, Specifically, at each training step, AdaMVE samples a batch of data (s, a, r, 8') from replay buffer B. For each s' the batch, on-policy Hmax -see rollout is computed using the model and the learning agent's current policy т. For each data (s, a, r, s'), we update Q by minimizing ] (r + V P. (s') Q(s, where a target Q- D-value is used to compute the value at the end state of each rollout. Pseudocode of AdaMVE is provided in the Appendix.
Hmus max is set by computational consideration in practice. Our proposed method is valid for any large Hma-
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Previous work model-based reinforcement learning can be divided in two categories: using the model for planning in low-dimensional state spaces, and combining the benefits of model-based and model-free approaches. For the first category. Gal et al. (Gal et al. 2016) combine the PILCO algorithm (Deisenroth & Rasmussen 201T) with neural dynamic model. Hafner al. (Hatner et al., 2018) propose to learn a latent dynamic model and choose actions through online planning with the latent model. For the second category, Weber al. use imaginary rollouts generated by the dynamic model for policy learning (Racanière et al., 2017). Gu et al. propose to augment imaginary rollout data to the experience replay buffer and show that this can accelerate model-free learning (Gul et al., 2016). In this paper, we use MVE (Feinberg et al.) 2018) as the haseline algorithm to show the effectiveness of our adaptive planning horizon algorithm. But it important to note that our method can be combined with any of the model-based methods discussed above.
The compounding error phenomenon of model-based RL is previously discussed (Asadi et al. 2018 Jiang et al. 2015 Talvitie 2017 Wang al. 2019). Surprisingly, relatively little work has been done to solve this problem. Buckman et al. propose STEVE, which uses stochastic ensemble of models and planning horizons to relax the error caused by using only model with fixed planning horizon (Buckman et al.) 2018). The ensemble with lowest variance is used as the learning target. In comparison, our proposed method directly handles the compounding error by adaptively selecting planning horizons based on a learned model error function. In addition to using a different design idea, our approach has both the model and the model error as single function, which is far less computationally expensive than STEVE, that learns multiple models in an ensemble. It is also worth noting that the model in our approach can be either hand designed, pretrained from another task, or learned online.
5 EXPERIMENTS
We conduct experiments on both gridworld and continuous control environments. For the gridworld environment, implement adaptive value expansion (AdaMVE) based DQN Mnih et al. (2015) and compare to the vanilla DQN and its non-adaptive value expansion variant (MVE). For continuous control, we implement AdaMVE with DDPG Lillicrap et al. (2015) and compare the vanilla DDPG and MVE. We perform single update to the policy for all methods in comparison at each environmental step.
5.1 EXPERIMENTS ON GRIDWORLD
Visualizing the Adaptive Horizon. We first evaluate the learned adap- tive planning horizon by visualizing in gridworld with predefined im- perfect model. If our method works correctly, we should observe a large horizon at states where the model is accurate, but small horizon for those states where the model deviates a lot from the true environment dynamics.
+
We FourRoom gridworld maze (Fig. 2 Each room has 9 cells. The agent's 's objective is to find the goal position (in red) starting from random initial position. There five actions: left, right, up, down and stay. The maximum length of each episode 50. After each episode the agent restarts a random position. The reward is 1 when hitting the goal and 0 in the other positions. A state is represented by the (I, y) coordinate. planning horizon using the following models:
Figure 2: FourRoom Env
We evaluate the adaptive
Oracle model. The transition function behaves exactly the same as the true environment.
3Room model. The transition function is true in three rooms, but completely wrong in the left bottom room. For a given state and action in this room, the model simply produces a randomly sampled next state from all possible positions.
NoWall model. This model ignores the existence of the wall. For example, given a state at one side of a wall, if the agent takes the action towards the wall, this model will predict the position overlapping with the wall as the next state, but in the true environment the agent will just stop at the current position.

---

Under review as a conference paper at ICLR 2020
(a) 3Room CSRV
(b) 3Room Greedy
(c) 3Room Replay
(d) NoWall CSRV
(e) NoWall Greedy
(I) NoWall Replay
Oracle
(h) 3Room
NoWall
Experiment results FourRoom. (f)-(f) Visualization of learned planning horizon for all states. For each state, the average horizon (s) weighted by (8) is presented. We use Hmax = thus (s) <= Hmax/ / 2.5 from definition. Our method successfully adapt the planning horizon when the model is imperfect. (g)-(i) Policy learning performance with different models. The shaded area shows the standard error. Results clearly show that AdaMVE significantly outperforms MVE when the model is imperfect (no wall model and 3room model).
Figure 3:
We evaluate the three reference policies (conservative, greedy and replay buffer) discussed in Sec- tion B.1, denoted by csrv, greedy and replay respectively. The results are visualized in Fig. (a)- For each state (position),, we show its weighted average horizon (s), as defined Section 3.2 The
results clearly show the effectiveness of the proposed method in finding appropriate planning
horizon for different parts of the state space. example, in the 3room model, our method can adopt zero planning horizon for states in the left bottom room where the model has large error.
Policy Learning Performance in FourRoom. We compare AdaMVE with MVE and DQN using the three models described above. Results are presented in Fig. (g)-(i). Each data point is averaged over 5 runs. Each run evaluated after every 2000 environmental steps by computing the mean total episode reward across 10 episodes. When the oracle model is available, the model error is zero everywhere, hence AdaM performs exactly the same with MVE. Both algorithms outperforms DQN, which confirms that the model-based value expansion targets lead improved performance. However, when the model is noisy, MVE diverges due to model errors. In contrast, AdaMVE still converges, and does so significantly faster than DQN. This because AdaMVE can adapt the rollout horizon for states where the model has large error as shown in the visualization. For example, when using the 3room model, AdaMVE only performs the model-free updates for states in the bottom left room. In contrast, at these states MVE still trusts the multi-step value expansion target, which has large error that causes diverge.
5.2 CONTINUOUS CONTROL
We experiment on continuous control tasks to further verify the benefit of adaptive rollout horizons.
We first use Mujoco Tooorov et al. 2012) to create 3D mazes and learn to control a PointMass agent to navigate to goal area in the maze, starting from a random location, as shown in Fig a-c). We also test on two Mujoco control problems in OpenAl Gym (Brockman al.) 2016); HalfCheetah and Swimmer. All results are based on different runs. Each data point in the plots Is evaluated by 200 test episodes for PointMass Navigation, and 50 test episodes for Mujoco control problems.
7

---

Under review as a conference paper at ICLR 2020
283
120
(a) PointNoWall
(b) PointRoom (PR) (c) PointMaze (PM)
(d) PR Pretrain
(c) PM Pretrain
Diy (ua))
(1) PR Online P
(g) PM Online P
(h) HalfCheetah
Swimmer
Figure 4: Results of continuous control. (a)-(c) PointMass Navigation example environments. (d)-(e) PointMass Navigation policy learning performance using pretrained model. (f)-(g) PointMass Navi- gation policy learning performance using online learned model. (h)-(i) Policy learning performance on HalfCheetah and Swimmer. AdaMVE outperforms both DDPG and MVE with pretrained model. With online learned model, vanilla AdaMVE does not improve performance over the baselines. AML.AdaMVE outperforms all the baseline algorithms with an online learned model.
Results with Pretrained Model. We first evaluate the adaptive method using pretrained model on the PointMass Navigation problem. We pretrain model by executing a uniform-random policy in PointNoWai (Fig (a)). This pretrained model is then used an imperfect model in PointRoom (PR. Fig 4 (b)) and PointMaze (PM, Fig 4 (c)) without further training. This model is supposed to be good at modeling local motions while bad modeling interactions with the wall. For AdaMVE, we use the replay reference policy and H Hma. = 5. MVE applies a fixed rollout horizon 5. As shown in Fig (d)- AdaMVE outperforms both MVE and DDPG, which further justifies the benefits of selecting rollout horizons in an adaptive way.
We next conduct experiments with an online learned model. The model is updated by one gradient step at each environment step. We observe that in this setting it hard to achieve competitive performance by directly learning the model online. To fix this problem, propose selective model learning: for a batch B of data (s, s,s') that are used for learning the model, we rank the data according to the model error function É(,, ham)), and use percent of the data that have small model errors learn the model. By using this selective model learning approach, we hope to learn partially accurate model that only focus the dynamics which are easy to be learned. Our adaptive planning method can still benefit from such model since able to learn where the model has large errors and only adopt small planning horizon at those states. We also note that using the model error function E is different with directly computing the model error for each data in B. since É is learned by a reference policy and thus model learning. Another advantage of using & is that we can tune the haml parameter, in which case we try identify states whose nearby regions have large model error.
Results with Online Learned Model.
provide more stable guidance for robust
We denote AdaMVE with selective model learning by the replay reference policy and Hmx 3 for both AdaMVE and SmI-AdaMVE, We tune haml from {1, 2) and use x = 50 for selective model learning. For MVE, we tune the rollout horizon H from (1,3,5) 5) and report the best result. Surprisingly, we find that H 1 gives the best results in all test domains. We also compare our methods with STEVE, the state-of-the-art model- basedl-based value expansion method (Buckman et al. 2018). our implementation of STEVE, we use value functions, H = 3, and 3 independently online learned models to create ensembles. Results are presented in Fig 4 (f-(i.. With an online learned model, vanilla AdaMVE does not improve performance over the baselines, due to the difficulty to catch the error of an online updated model. However, by relaxing the model learning procedure using selective model learning, AdaMAE outperforms all the baselines with online learned model. Importantly, our proposed method has both the model and the model error as a single function, which is far less computationally expensive than the stochastic adaptive method STEVE, but show significantly better performance in practice.
In all test domains, we use
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
We present a principled method to learn model errors by TD-learnin in model-based reinforcement learning. Based on the learned model errors, an adaptive approach to select state-dependent planning horizons introduced. Our proposed algorithm, AdaMVE, combines model-based and model-free reinforcement learning by adaptively selecting the rollout horizons in model-based value expansion. Empirical results shows that AdaMVE (i) successfully adapts the planning horizons according the local correctness of the model, (ii) outperforms model-free, non-adaptive and stochastic-adapen model-based baselines.
For the future work, we would like to combine our adaptive planning horizon method in other model-based RL approaches such as model predictive control and Monte Carlo tree search. Another future direction is how to learn a (partially correct) model online. In the experiments, we observe that simply fitting a neural network with minibatch and training 1.2 losses is not good enough and a better model learning method is needed. Our proposed selective model learning method is preliminary attempt to solve this problem. We believe learning a reasonably good model online in complex high dimensional control tasks is important problem and deserves thorough future studies.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.078: 2017.
Wasserstein
gan.
arXiv preprint
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model- based reinforce- ment learning. In International Conference on Machine Learning, pp. 264-273, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.013: 0154600001155, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pp. 8224 8234, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465-472, 2011.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1582-1599 -1911 2018.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data- Eafa-Efffcieett Machine Learning workshop, ICML, volume 4, 2016.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference Machine Learning, pp. 2829-2838 2016.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1011111/11..1.. 2018.
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon model accuracy. In Proceedings the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1181-1189. International Foundation for Autonomous Agents and Multiagent Systems, 2015.

---

Under review as a conference paper at ICLR 2020
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.009911 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence heads-up no-limit poker. Science, 356(6337):508-51 2017.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, (29(778)44444 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal. and John Langford. Model-based reinforcement learning in contextual decision processes. arXiv preprint arXh::1811.08544 2018.
Richard S Sutton, Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pp. 216-224. Elsevier, 1990.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Erik Talvitie. Self-correcting models for model-based reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning, arXiv preprint
2019.
Christopher JCH Watkins and Peter Dayan. Q-learning Machine learning, 8(3-4):279-29) 292, 1992.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.I PSEUDOCODE
We provide the pseudocode for AdaMVE. We provide the pseudocode for discrete setting (based on DQN), and the model eтror learned by the replay buffer reference policy. This code can be easily extend to continuous setting (based on DDPG) and model error learning with the other two reference policies as discussed in Section 3.1
Algorithm 1 Adaptive Mode-based Value Expansion
Input: maximum rollout horizon Hmx
Initialize the replay buffer B to capacity N Initialize the approximate model P. Initialize action value function Qe (and/or the policy funtion ma if applicable) Initialize state model error function Eg with maximum rollout horizon Hmax for t to T do
Sample transitions using the e-greedy policy and store transitions in B (If learn model) Sample transitions from B to learn the model Pu Sample transitions from B to learn the model error & Sample a batch of transitions (s, a, r, s') from B For each s' in the batch get (s', ft) for
[0, Hmax] For each s' compute the multi-step value expansion target for each h E [0, max]
Compute the target value V P.Hno (s') according to (9)
Update 9 using model free RL by using V H.H...
(s') as the target value
Update target network parameters for both policy training and model error training, 0 and Ø. with exponential decay
end for

---

Under review as a conference paper at ICLR 2020
A.2 PROOF OF THEOREM 1
Proof. For any S h & H, define Un to be the H step value expansion that rolls out the true model P for the first h steps and the approximate model P for the remaining H h steps:
h-1 Un
H-1
Y E
P P7 (1s) RR (st)) +
Σ 7 , (14)))
RR (st)
2 pp o (11)) [[s)]]
(10)
where P P/ denotes the distribution over states after rolling out h steps with P and t h steps with P, i.e.
P (-(s)) Σ
S'ES
From the definition of Un we know Uo 1 P1 (s) and UH
Vp.H (s). Hence have
H-1
(s) (8) VP Un - Un
Σ
(Un - Un+1)
h=0
To analyze the difference between Un and Uht1, we rearrange the terms in 10 in two different ways:
B PT [R (sx) + 7 "B (T(())) VT (
Un
(11)
Un- = Σ y (P(le (++) + 7 N+IR ProP
V (0
(12)
Now applying to Un and Un+1. we can bound Un Un+1 by
E PP( [R (s:)] + h+1
h P [V ((n+nn)
2'E (-10)) [12 (st)] - h+ W (1s) [V (n
h Eyg Po (1)s))aaa BeNP(lsann V [ - -EewP((onaa)) [Vp - [6]]
Pp.H-- |1, w Be Pla
sup E. ( [0(s'] BggxP((eenaa)) [ 1
"K Esn=PT(1s) [W (sn)]
(13)
The bound
(33 also holds for the opposite direction Un41 Un. Therefore
H-1
H-1
[7, (s) (s) S
Σ
Un, Until K
Σ
h+ E My (t(ss) [W*(s
h=0
h=0
which concludes the proof.
12

---

Under review as a conference paper at ICLR 2020
A.3 EXTRA EXPERIMENTS ON FOURROOM
Comparison with MVE using other planning horizon. In Fig. we compare AdaMVE with MVE
using a fixed horizon H In this section we provide experiment results compared with MVE_ h1 (fixed horizon H = 1) and M3EJ33 (fixed horizon H = 3). AdaMVE still applies Hmas = 5. AdaMVE clearly outperforms MVE-h1 and MVE-h3 on both domains in terms of sample efficiency and final performance.
40
30-
AUUIVV LATY
ANUNNE
AOUMVE greedy
ANUNEE geedy
molly
repley
traniing
(a) 3Room
(b) No Wall
Figure 5: We compare AdaMVE with MVE. h1 (fixed planning horizon 1) and MVE h3 (fixed planning horizon 3). AdaMVE outperforms both methods on both domains terms of sample efficiency and final performance.
Transferring Adaptivety to Different Tasks If the environment dynamics is fixed, the model error learned in one task can be directly transferred to different task. To illustrate this, we create new task named FourRoom2, by changing the goal position in FourRoom from (15,15) to (2,18). In FourRoom2, we first train model error function in FourRoom, then directly apply the pre-trained model error for AdaMVE, We include DQN and AdaMVE that learns from scratch in the new tasks the baseline algorithms for comparison. Results shown in Fig. AdaMVE with the transferred model error denoted by T-AdaMVE.
40
40
35-
75-
30-
30
25-
20
Ssanct roday
10
D01
DOS
18000 training steps
traning stops
(a) 3Room
(b) No Wall
Figure 6: Evaluation of model error transformation.
Recall that the conservative policy tries maximize the model error to provide robust estimation. Hence, the model error learned by this policy in FourRoom should still be effective for FourRoom2. The result clearly confirms this intuition: with the pre-trained model error, AdaMVE. cerrs learns much faster than the one learning from scratch. In comparison, the model error learned by the greedy policy and replay policy less useful for transfer between the tasks. When the wall model is used, the transferred model eтror even brings negative effects. This is due to the fact that both the greedy policy and replay policy are based on the learning agent's 's behavior during training and therefore would change in different tasks.
13

---

Under review as a conference paper at ICLR 2020
A.4 EXPERIMENT DETAILS
Table T lists the parameters used in gridworld experiments. Table 2 lists the parameters used in Mujoco Navigation. For selecting the mixing temperature т, in Gridworld we pick the best one (0.01) between 0.01 and 0.001. For Mujoco environments we use = 0.01 without further tuning. In Swimmer, use the double-Q trick to provide better performance (Fujimoto al. 2018). For selective model learning, the reported results use sml = 2 for PointRoom, and haml = for other domains.
Table I: Parameters used in gridworld experiments.
Parameter
Values
c-greedy
0.2
discount (7)
0.98
T
0.01
batch size
128
optimizer (all networks)
Adam
learning rate (q network)
0.001
learning rate (model error network)
0.0001
replay buffer initial size
2000
replay buffer size
10
target network update interval
target network mixing coefficient
0.001
Q network for policy learning Q network for model error nonlinearity
(200, 200, 200)
(200, 200, 200)
ReLU
Maximum rollout steps Hmax
5
Table 2: Parameters used in Mujoco Navigation.
Parameter
Values
action noise for exploration discount (7)
(.1.0.
0.99
0.01
batch size
128
optimizer (all networks)
Adam
learning rate (all networks)
0.0001
A network weight decay (DDPG)
1e-6
Q network weight decay (DDPG) replay buffer initial size replay buffer size
le-3
20000
106
target netowrk update interval
target network mixing coefficient
0.001
Q network for both model error and policy learning
(300, 300)
A network for both model error and policy learning nonlinearity
(300, 300)
ReLU
Maximum rollout steps H
14

---

Under review as a conference paper at ICLR 2020
A.5 SUPPORTING RESULTS
We provide supporting results in this section. Figure ( (a) and (b) show the weighted average horizon H during training in PointRoom and PointMaze with pretrained model. The maximum horizon Max is set to 5. We can see that H over different states has very high variance, which is a sign of successful adaptation since the pretrained model is wrong the states that are next the wall (shorter rollout horizon) while being accurate at the states that are away from the wall (longer rollout horizon). PointMass has shorter overall rollout horizons than PointRoom because PointMaze has an extra wall and thus smaller set of states where the pretrained model is accurate, hence enjoys less benefit of using AdaMVE over DDPG than in PointRoom.
2.5
2.5
planning herizon
2.0
planning herizon
2.0
1.5
1.5
1.0
1.0
0.5
0.0
MVE
MVE
AdaMVE
0.0
AdaMVE
0.0
1.5
0.0
0.5
1.5
2.0
training steps (1e5)
training steps (1e5)
(a) (s) PointRoom
(b) I(s)) PointMaze
Figure 7: Supporting Results. When Hamx = 5, the maximum of weighted average horizon H 2.5 according to the definition.
15


Reviews:
[{"review_id": "ryxG7NUmcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#rebuttal responses\n \nThe authors' reply does not convince me, and I still think the paper has some problems:\n(1) I do not believe that the cumulative model-error can not be learned efficiently;\n(2) Experimental results are weak as some baselines do not converge! \n\nThus I keep my rating as reject.\n\n#review \nThis paper proposes a new adaptive model-based value-expansion method, AdaMVE, that decides the planning horizon of the learned model by learning the model-error. The model-error is learned by temporal difference methods.\nExperimental results show that AdaMVE beats MVE, STEVE, and DDPG in several environments.  \n\nOverall the paper is well written.  The paper proposes an interesting question: how to adaptively change the planning horizon based on the state-dependent model-error? Firstly, The authors upper bound the cumulative target error by the cumulative model-error.  Then the cumulative model-error is learned by the temporal difference method. With the learned cumulative model-error function over different rollout steps, the planning horizon is decided by a softmax policy.  \n\nHowever, I do not think that learning an upper bound of the target error helps to determine the value of H, as there is no justification that the gap between the target error and the model error is small theoretically.  I also doubt that the cumulative model-error can be learned without large loss, as there are no plots of W in this paper. \n\nThe authors claim that it is expensive to retrain the model error for the current policy at every step, thus they use some reference policy. I think it is ok, but I want to see the results of AdaMVE using the updated current policy, or updating the Q function before improving the policy. Adding a figure showing the change of H in the training helps to motivate this paper.\n\nFinally, AdaMVE is only compared in two MuJoCo environments. Baselines in other environments are only trained in 1e5 steps, thus the experimental results are not convincing.\n\nI am happy to change my opinion on this paper if authors give better motivation and the detail of the learning.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlGiJetcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Rebutal Respons:\n\nPlanning Horizon:\n- I agree small horizons can speed up learning. However, if we want to drastically reduce the sample complexity, we need longer planning horizons. Therefore, we should be looking further in this direction but this is out of scope for this paper.\n\nModel Learning:\n- Good idea to update the conclusion and treat the online model learning as future work.\n\nFigure 3:\nYou could plot axes a - f in a single row and remove the colorbar for each individual plot. Just do one colorbar for all plots. This would also improve the comparability of the different methods. Currently, the color-coding is different for each axis, which is bad practice. Axes g - i can be reshaped in a separate figure. Furthermore, figure 2 is not necessary as the 4 room environment is depicted in Fig. 3 a - f and and one could reference these axes. \n\n=> I keep my rating as weak accept.\n\n# Review:\nSummary:  \nThe paper proposes an adaptive scheme to adapt the horizon of the value function update. Therefore, the sample efficiency should be increased and the value function should be learned faster. \n\nConclusion: \nThe problem of learning good policies from partially correct models is very interesting and important. The proposed approach is technically sound and reasonable. The experiments highlight the qualitative as well as quantitative performance. Furthermore, the quantitative performance is compared to state-of-the-art methods. I cannot comment on the related work as I am not familiar with the baselines MVE & STEVE. \n\nMy Main Concerns are:\n\n- The roll-out horizon is 3-5 timesteps. This horizon is really short especially for problems with strong non-linearities and high sampling frequencies. For such systems 5 timesteps would only correspond to 0.5s (10Hz) or 0.05s (100Hz) and I am uncertain whether these short horizons really help for such problems. \n\n- The specialized online model learning algorithm seems quite hacky. It feels like it was introduced last minute to make it work. The overall question of how should we learn a model optimally is super important and should be addressed within a separate paper (and more thoroughly). I would even propose to remove the online learning section from this paper as it is just too hacky and without relation to prior work or context. \n\n- Could the authors please update figure 3 as the figure has too much whitespace. This whitespace could be used to enlarge the individual axis when the axis are rearranged.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygwEZxFYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning to Combat Compounding-Error in Model-Based Reinforcement Learning\n\nThe authors study the problem of choosing the appropriate planning horizon in model-based reinforcement learning. We want to have a balance between the efficiency of planning into the future and while avoiding the compounding error brought by the learnt model.\nAnd the authors propose to learn an error estimation model that is a function of current state, policy and horizon by estimating the compounding error as a new reward using policy evaluation.\nPros -\nI think the novelty in this paper is most appealing for me to vote for a clear acceptance. The growth of model-based reinforcement learning is in a big need of a framework to work with the planning horizon.\nThe proposed method is a very elegant way of estimating the errors and choose the appropriate horizon.\nStill, I think there is a lot of room for improvement, as summarized below.\n\nCons\n- One thing that concerns me is that the maximum horizon H looks very small. A max horizon of 3, 5 or even 10 is still much smaller than the value we usually use in planning based reinforcement learning (MPC in your case). A common choice can be up to 30 or 50 for PETS based algorithms [1].\n\nDoes that indicate that the proposed framework is not stable or robust when the horizon is big? Or maybe that scale of horizon is not common in Steve and thus not tried in the experimental section?\n\n- Experiments\nFor the high dimensional control tasks, only Half-Cheetah and Swimmer was tried. It will generalize the conclusion if the results of other benchmarking environments are tested, including complex tasks such as Humanoid, and easier tasks such as Cart-pole.\n\n- Algorithms\nThe used baselines in the paper are generally not considered to be state-of-the-art. I guess it is sort-of fair comparison since you add the proposed component on top of the baselines. But extending the current state-of-the-art such as TD3 [2] / PETS\n\nI also suggest adding this benchmarking paper into the related work, which empirically studies the compounding error in different state-of-the-art model-free and model-based reinforcement algorithms.\n\n[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).\n[2] Fujimoto, Scott, Herke van Hoof, and David Meger. \"Addressing function approximation error in actor-critic methods.\" arXiv preprint arXiv:1802.09477 (2018).\n[3] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel and Jimmy Ba. “Benchmarking Model-Based Reinforcement Learning.” ArXiv abs/1907.02057 (2019): n. pag.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews: R (R3), WA (R2), A (R1). AC has read the reviews, rebuttal and paper. AC is concerned about the short planning horizon, which seems like a major issue: (i) as R1 notes, most MPC algorithms use much longer horizons as they find it helps performance and (ii) the claim of the approach to be able to pick the planning horizon is moot if its dynamic range is small.  Overall, the paper is very borderline. The idea is interesting but without addressing longer horizons, the contribution is limited. Under guidance from the PCs, the AC feels that the paper just falls below the acceptance threshold and thus cannot be accepted unfortunately. The work is definitely interesting however and should be revised for a future submission. 


Decision: Reject