Model-Augmented Actor-Critic: Backpropagating through Paths
Authors: Ignasi Clavera
Yao Fu
Pieter Abbeel
Venue:  (2020)
Keywords: reinforcement learning
model-based
actor-critic
pathwise

Abstract: Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Under review as a conference paper at ICLR 2020
MODEL-AUGMENTED CRTICC BACKPROPAGATING THROUGH PATHS
Anonymous authors Paper under double-bling review
ABSTRACT
Current model- reinforcement learning approaches the model simply learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how make more effective use of the model by exploiting its differentiability. We construct a policy optimization algo- rithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value funetion. We show that our approach (i) is consistently more sample efficient than existing state-of-the-arr model-basee algorithms, (ii) matches the asymptotic performance of model- algorithms, and (iii) scales to long horizons, regime where typically past model-based approaches have struggled.
INTRODUCTION
Model-based reinforcement learning (RL) offers the potential be general-purposs. tool for learning complex policies while being sample efficient. When learning in real-world physical systems, data collection can be an arduous process. Contrary model-free methods, model-based approaches are appealing due to their comparatively fast learning. By first learning the dynamics of the system in supervised learning way, can exploit off-policy data. Then, model-based methods use the model to derive controllers from it either parametric controllers (Luo et al. 2019 BBckkman et al. 2018 Janner
al. 2019) or non-parametric controllers (Nagabandi et al. 2017| Chua et al. 2018).
Current model-based methods learn with an order of magnitude less data than their model-free counterparts while achieving the same asymptotic convergence. Tools like ensembles, probabilistic models, planning over shorter horizons, and meta-learning have been used to achieved such perfor- mance (Kurutach et al. 2018 Chua et al. 2018 Clavera et al.) 2018). However, the model usage in all of these methods is the same: simple data augmentation. They the learned model black-box simulator generating samples from it. In high-dimensional environments or environments that require longer planning, substantial sampling is needed to provide meaningful signal for the policy. Can we further exploit our learned models?
In this work, we propose estimate the policy gradient by backpropagating its gradient through the model using the pathwise derivative estimator. Since the learned model is differentiable, one can link together the model, reward function, and policy to obtain an analytic expression for the gradient of the returns with respect to the policy. By computing the gradient in this manner, we obtain an expressive signal that allows rapid policy learning. We avoid the instabilities that often result from mack-propagating through long horizons by using terminal Q-function. This scheme fully exploits the learned model without harming the learning stability seen in previous approaches (Kurutach) al., 2018 Heess et al., 2015). The horizon which apply the terminal Q-function acts hyperparameter between model-free (when fully relying on the Q-function) and model-based (when using a longer horizon) of our algorithm.
The main contribution of this work is a model- model-based method that significantly reduces the sample complexity compared to state-of-the the-thh- model-based algorithms (Janner et al.) [0199] Buckman et al.) 2018). For instance, we achieve a 10k return in the half-cheetah environment in just 50 trajectories. We theoretically justify our optimization objective and derive the monotonic improvement of our

---

Under review as a conference paper at ICLR 2020
learned policy in terms of the Q-function and the model error. Furtermore, we experimentally analyze the theoretical derivations. Finally, we pinpoint the importance of our objective by ablating all the components of our algorithm. The results reported in four model- benchmarking environments (Wang et al.] 2019] Todorov et al. 2012). The low sample complexity and high performance of our method carry high promise towards learning directly on real robots.
RELATED WORK
Model-Based Reinforcement Learning. Learned dynamies models offer the possibility to reduce sample complexity while maintaining the asymptotic performance. For instance, the models can act as a learned simulator on which a model-free policy is trained on (Kurutach ct al. 2018| Luo et al.
2019 Janner al. 2019). The model can also be used to improve the target value estimates (Feinberg et al. 2018 or to provide additional context to a policy (Du & Narasimhan) 2019). Contrary to these methods, our approach uses the model in different way: exploit the fact that the learned simulator is differentiable and optimize the policy with the analytical gradient. Long term predictions suffer from a compounding error effect in the model, resulting in unrealistic predictions. In such cases, the policy tends to overfit to the deficiencies of the model, which translates to poor performance in the real environment: this problem known as model-bias (Deisenroth & Rasmussen 201T). The model-bias problem has motivated work that uses meta- weta-lcarning (Clavera et al. [218)) interpolation between different horizon predictions (Buckman et af., 2018 Janner et al. 2019) 1919, and interpolating between model and real data (Kalweit & Boedecker, 2017). To prevent model-bias, we exploit the model for a short horizon and use a terminal value function to model the rest of the trajectory. Finally, since our approach returns a stochastic policy, dynamics model, and value function could use model-predictive control (MPC) for better performance at test time, similar to (Lowrey et al. 2018)] Hong al.) 2019). MPC methods (Nagabandi et al.) (017) have shown to be very effective when the uncertainty of the dynamics is modelled (Chua et al. 2018 Wang & Ba 2019).
Differentable Planning. Previous work has used backpropagate through learned models to obtain the optimal sequences of actions. For instance, [evine Levine & Abbeel (2014) learn linear local models and obtain the optimal sequences of actions, which is then distilled into neural network policy. The planning can be incorporated into the neural network architecture (Okada et al. [007]] Tamar et al.
differentiable function (Pereira et al. 2018 8 Amos et al. 2018). Planning sequences of actions, even when doing model-predictinng control (MPC). does not scale well to nigh-dimensional, complex domains Jaanerr et al. (2019). Our method, instead learns neural network policy in an actor-critic fashion aided with a learned model. our study, we evaluate the benefit of carrying out MPC on top of our learned policy at test time, Section 5.4 The results suggest that the policy captures the optimal sequence of action, and re-planning does not result in significant benefits.
2016 Srinivas al. 2018 Karkus et al.) 2019) or formulated
Policy Gradient Estimation. The reinforcement learning objective involves computing the gradient of an expectation (Schulman et al. 2015a). By using Gaussian processes (Deisenroth & Rasmussen 2011), it is possible to compute the expectation analytically. However, when learning expressive parametric non-linear dynamical models and policies, such closed form solutions do not exist. The gradient is then estimated using Monte-Carlo methods (Mohamed et al. 2019). In the context of model-based RL, previous approaches mostly made use of the score-function or REINFORCE estimator (Peters& & Schaal 2006 Kurutach et al. 2018). However, this estimator has high variance and extensive sampling is needed, which hampers its applicability in high-dimensional environments. In this work, we make use of the pathwise derivative estimator (Mohamed et al., 2019). Similar to our approach, Heees et al. (2015) uses this estimator in the context of model-based RL. However, they just make use of real-works trajectories that introduces the need of a likelihood ratio term for the model predictions, which in turn increases the variance of the gradient estimate. Instead, we entirely rely on the predictions of the model, removing the need of likelihood ratio terms.
Actor-Critic Methods. Actor-critic methods alternate between policy evaluation, computing the value function for the policy; and policy improvement using such value function (Sutton & Barto
1998. Barto et al. 1983). Actor-critic methods can be classified between on-policy and off-policy.
On- On-policy methods tend to be more stable, but at the cost of sample efficiency (Sutton 1991 Mnih
et al. 2016). On the other hand, off-policy methods offer better sample complexity (Lillcrap et al.
2015). Recent work has significantly stabilized and improved the performance of off-policy methods
using maximum-entropy objectives (Haarnoja al. 2018a) and multiple value functions (Fujimoto)
2

---

Under review as a conference paper at ICLR 2020
et al. 2018 Our method combines the benefit of both. By using the learned model we can have learning that resembles an on-policy method while still being off off-po.icy.
3 BACKGROUND
In this section, we present the reinforcement learning problem, two different lines of algorithms that tackle it, and summary on Monte-Carlo gradient estimators.
3.1 REINFORCHMENT LEARNING
discrete-time finite Markov decision process (MIP) M is defined by the tuple (S, A. Pop 7)- Here, S is the set of states, A the action space, 8t-1 ~ [8t, a4) the transition distribution, r S IR is reward function, po : R+ represents the initial state distribution, y the discount factor, and I' is the horizon of the process. We define the return as the sum of rewards r(s,, ax) along a trajectory T = (Sg. au..... -T- -1- Sr)) The goal of reinforcement learning is find a policy S x A + I that maximizes the expected return, i.e., max J(0) maxę E E, r'r(st, dz)]-
Actor-Critic, In actor-critic methods, learn function Q (critic) that approximates the expected return conditioned on state s and action a, E,>, 7'r(st, (s)(ss = s, dg a). Then, the learned Q-function is used optimize a policy (actor). Usually, the Q-function is learned by iteratively minimizing the Bellman residual:
JQ (Q(({(QQ,,)))) (s,, a.) (r(s,, 04) + 1.
The above method is referred as one-step Q-learning, and while naive implementation often results in unstable behaviour, recent methods have succeeded in stabilizing the Q-function training (Fujimooo
al. 2018). The actor then can be trained to maximize the learned Q function J, Q(ss π (s)) The benefit of this form of actor-critic method is that it can be applied in off-policy fashion, sampling random mini-batches transitions from an experience replay buffer (Lin 1992).
Model-Based RL. Model-based methods, contrary to model-free RI, learn the transition dis- tribution from experience. Typically, this carried out by learning a parametric function ap- proximator fØs known as a dynamics model. We define the state predicted by the dynam- ics model as $1+1, i.e., $441 fa(st,a)) The models are trained via maximum likelihood: max (Ø) max Bllog (allog((S,,) ((,r+118,,,, a:)
3.2 MONTE-CARLA GRADIENT ESTIMATORS
In order optimize the reinforcement learning objective, it is needed to take the gradient of an expectation. In general, it is not possible to compute the exact expectation so Monte- gradient estimators are used instead. These are mainly categorized into three classes: the pathwise score function, and measure-values gradient estimator (Mohamed et al., 2019). In this work, we use the pathwise gradient estimator, which is also known as the re- re-araameterization trick (Kingma & Welling 2013). This estimator is derived from the law of the unconscious statistician (LOTUSS) (Grimmett & Stirzaker 2001)
E [2) = Epp
Here, we have stated that we can compute the expectation random variable without knowing its distribution, if we know its corresponding sampling path and base distribution. A common case, and the one used in this manuscript, parameterizes Gaussian distribution: x pe = NN oz). which is equivalent to μ4 Ede for A(0 1).
4 POLICY GRADIENT VIA
PATHWISE DERIVATIVE
Exploiting the full capability of learned models has the potential to enable complex and high- dimensional real robotics tasks while maintaining low sample complexity. Our approach, model- augmented actor-critic (MAAC), exploits the learned model by computing the analytic gradient
3

---

Under review as a conference paper at ICLR 2020
of the returns with respect to the policy. In contrast to sample-based methods, which one can think of as providing directional derivatives in trajectory space, MAAC computes the full gradient, providing strong learning signal for policy learning, which further decreases the sample complexity. In the following, we present our policy optimiza- tion scheme and describe the full algorithm.
0
4.1 MODEL-AUGMENTED ACTOR-CRIIE OBJECTIVE
a2
aH
Among model-free methods, actor-critic meth- ods have shown superior performance in terms of sample efficiency and asymptotic perfor- mance (Haarnoja et al. 2018a). However, their sample efficiency remains worse than model- based approaches, and fully off-policy methods still show instabilities comparing to on-policy al- gorithms (Mnih et al. 2016). Here, propose a modification of the Q-function parametriza- tion by using the model predictions on the first time-steps after the action is taken. Specifically, do policy optimization by maximizing the following objective:
S0
S1
S2
SH
r2
Q
Figure I: Stochastic computation graph of the proposed objective J.. The stochastic nodes represented by circles and the deterministic ones by squares.
H
J (0) = E
Σ 7r((s)) + + ((((HHH)))))
whereby, 9 ~ f(st, ax) and ar (st). Note that under the true dynamics and Q-function, this objective is the same as the RL objective. Contrary to previous reinforcement learning methods, optimize this objective by back pockppropagttion through time. Since the learned dynamics model and policy are parameterized as Gaussian distributions, we can make use of the pathwise derivative estimator compute the gradient, resulting in an objective that captures uncertainty while presenting low variance. The computational graph of the proposed objective is shown in Figure]] [
While the proposed objective resembles n-step bootstrap (Sutton & Barto 1998), our model usage fundamentally differs from previous approaches. First, we do not compromise between being off- policy and stability. Typically, n-step bootstrap either on-policy, which harms the sample complexity, or its gradient estimation uses likelihood ratios, which presents large variance and results in unstable learning (Heess et al.] (015). Second, we obtain a strong learning signal by backpropagating the gradient of the policy across multiple steps using the pathwise derivative estimator, instead of the REINFORCE estimator (Mohamed et al. 2019: Peters Schaal 2006). And finally, we prevent the exploding and vanishing gradients effect inherent to back-propagation through time by the means of the terminal Q-function (Kurutach et al. 2018).
The horizon H/ in our proposed objective allows to trade off between the accuracy of our learned model and the accuracy of our learned Q-function. Hence, controls the degree to which our algorithm is model-based or well model-free. If would end up with a model-free update; for И = oc, the objective results a shooting objective. Note that we will perform policy optimization by taking derivatives of the objective, hence we require accuracy on the derivatives of the objective and not on its value. The following lemma provides a bound on the gradient error in terms of the error on the derivatives of the model, the Q-function, and the horizon H.
not to trust our model at all (H = 0),
Lemma 4.1 (Gradient Error). Let J and Q be the learned approximation of the dynamics and O-function Q, respectively. Assume that and Q have Lgj 2-Lisschitz continuous gradient and f and ff have L /r/22-ipcchitt continuous gradient. Let E = max, ||V f(s,, à,) - f as l2) be the error on the model derivatives and cq V à - VQ(HH, aH) the error on the Q-function derivative. Then the error on the gradient between the learned objective and the true objective can be bounded by:
E [[Ve Volalla]
-Voklll s Sci(Her + (H)EE
Proof. See Appendix.
4

---

Under review as a conference paper at ICLR 2020
The result in Lemmal 4.1 stipulates the error of the policy gradient in terms of the maximum error in the model derivatives and the error in the Q derivatives. The functions 01 and C2 are functions of the horizon and depend on the Lipschitz constants of the model and the Q-function. Note that are just interested in the relation between both sources of error, since the gradient magnitude will be scaled by the learning rate, by the optimizer, when applying it to the weights.
4.2 MONOTONIC IMPROVEMENT
In the previous section, we presented our objective and the error it incurs in the policy gradient with respect to approximation error in the model and the Q function. However, the error on the gradient indicative of the effect of the desired metric: the average return. Here, we quantify the effect of the modeling error on the return. First, we will bound the KL-divergence between the policies resulting from taking the gradient with the true objective and the approximated one. Then we will bound the performance in terms of the KL.
Lemma 4.2 (Total Variation Bound). Under the assumptions of the Lemma 4.1 let - -0, 20 Va
be the parameters resulting from taking a gradient step on the exact objective, and ê = 0, + a a the parameters resulting from taking a gradient step on approximated objective, where a E Rt Then the following bound on the total variation distance holds
max Dyv(TTolTTg) S acslerci(H)) + cQC)(H))
Proof. See Appendix.
The previous lemma results in bound the distance between the policies originated from taking a gradient step using the true dynamics and Q-function, and using its learned counterparts. Now, we can derive a similar result from Kakade & Langford (2002) to bound the difference in average returns.
Theorem 4.1 (Monotonic Improvement). Under the assumptions of the Lemma 4.1 be 0' and Ô as defined in Lemma and assuming that the reward is bounded by Tmax- Then the average return of the T satisfies
2ar
(0) 2 (0) 1
aca +
Proof. See Appendix.
Hence, provide explicit lower bounds of improvement in terms of model error and function error. Theorem 4. extends previous work of monotonic improvement for model-free policies (Schul man et al. 201 5b Kakade & Langford 2002), to the model-based and actor critic up by taking the error on the learned functions into account. From this bound one could, in principle, derive the optimal horizon H that minimizes the gradient error. However, in practice, approximation errors are hard to determine and we treat H/ as an extra hyper-parameter. In section 5.2, we experimentally analyze the error on the gradient for different estimators and values of H.
4.3 ALGORITHM
Based on the previous sections, we develop a new algorithm that explicitly optimizes the model- augmented actor-critic (MAAC) objective. The overall algorithm is divided into three main steps: model learning, policy optimization, and Q-function learning.
Model learning. In order to prevent overfitting and overcome model bias (Deisenroth & Rasmussen 2011), we use a bootstrap ensemble of dynamics models {fe...... f models parameterizes the mean and the covariance a Gaussian distribution with diagonal covariance. The bootstrap ensemble captures the epistemic uncertainty, uncertainty due to the limited capacity data, while the probabilistic models are able to capture the aleatoric uncertainty (Chua et al., 2018)) inherent uncertainty of the environment. We denote by fo the transitions dynamics resulting from Φr. where U ~ U[M] is uniform random variable on {1,..., M). The dynamics models are trained via maximum likelihood with early stopping on validation
f x J. Each of the dynamics
5

---

Under review as a conference paper at ICLR 2020
Algorithm MAAC
1: Initialize the policy Te. model fo. Qu, Denv + 0, and the model dataset modell +
2: repeat
3: Sample trajectories from the real environment with policy TT. Add them to Denv.
4: for ........ 11 do
5: - Φ - V, y using data from Denv-
6: end for
7: Sample trajectrrocc trajectories T from fø. Add them to moddll
8:
< moodel D for 1.... G2 do
9:
Update &- + 3x JoJ= using data from D 11: Update ab 1/- AQ V (1bb) using data from D
10:
12:
end for
until the policy performs well in the real environment
14:
return Optimal parameters 0*
Policy Optimization. We extend the MAAC objective with an entropy bonus 20186), and perform policy learning by maximizing
HHarnoja et al.
#-
( EE
r'(()) + + Qu + +
t=
where 8141 ~ (3r, az). 90 ~ D, a We learn the policy by using the pathwise derivative of the model through H steps and the Q-function by sampling multiple trajectories from the same šo- Hence, we learn maximum entropy policy using pathwise derivative of the model through H steps and the Q-function. We compute the expectation by sampling multiple actions and states from the policy and learned dynamics, respectively.
Q-function Learning. In practice, we train two Q-functions (Fujimoto et al. 2018) since it has been experimentally proven to yield better results. We train both Q functions by minimizing the Bellman error (Section 3.1:
Jo( = (((Qu( at) (r(st, + 10 de+ (44))) 1)))
Similar to (Janner et al. 2019). we minimize the Bellman residual on states previously visited and imagined states obtained from unrolling the learned model. Finally, the value targets are obtained in same fashion horizon for the expansion. In doing so, maximally make use of the model by not only using it for the policy gradient step, but also for training the Q-function.
Stochastic Ensemble Value Expansion (Buckman et al. 2018), using H as
Our method, MAAC, iterates between collecting samples from the environment, model training, policy optimization, and Q-function learning. A practical implementation of our method described in Algorithm I First, obtain trajectories from the real environment using the latest policy available. Those samples are appended to a replay buffer Denv. on which the dynamics models are trained until convergence. The third step is to collect imaginary data from the models: collect k-step transitions by unrolling the latest policy from a randomly sampled state on the replay buffer. The imaginary data constitutes the Dmootet, which together with the replay buffer, is used to learn the Q-function and train the policy.
Our algorithm consolidates the insights built through the course of this paper, while the same time making maximal use of recently developed actor-critic and model-based methods. All in all, it consistently outperforms previous model-basee and actor-critic methods.
5
RESULTS
Our experimental evaluation aims to examine the following questions: 1) How does MAAC compares against state-of-the-art model-based and model-free methods? 2) Does the gradient error correlate

---

Under review as a conference paper at ICLR 2020
with the derived bound?, 3) Which are the key components of its performance?, and 4) Does it benefit from planning at test time?
In order to answer the posed questions, evaluate approach model-basee continuous control benchmark tasks in the MuJoCo simulator (Todorov et al. 2012 Wang et al. 2019).
5.1 COMPARISON AGAINST OFATE-OF-THE-ART
We compare our method sample complexity and asymptotic performance against lfte-of-theean model-free (MF) and model-based (MB) baselines. Specifically, we compare against the model- soft actor-critic (SAC) (Haarnoja et al. 2018a), which is an off-policy algorithm that has been proven to be sample efficient and performant: as well as two state-of-the-an based pollcy-opttmizztion (MBPO) aanner al. 2019) and stochastic ensemble value expansion (STEVE) (Buckman et al., 2018 The original STEVE algorithm builds on top of the model-free algorithm DDPG (Lillicrap et al. 2015), however this algorithm is outperformed by SAC. In order to remove confounding effects of the underlying model-free algorithm, we have implemented the STEVE algorithm on top of SAC. We also add SVG(I) (Heess al.) 2015) comparison, which similar to our method uses the derivative of dynamic models to learn the policy.
model-based baselines: model-
The results, shown in Fig. 2 highlight the strength of MAAC in terms of performance and sample complexity. MAAC scales to higher dimensional tasks while maintaining its sample efficiency and asymptotic performance.
all the four environments, our method learns faster than previous MB and MF methods. We are able to learn near-optimal policies in the half-cheetah environment in just over 50 rollouts, while previous model-based methods need least the double amount of data. Furthermore, in complex environments, such ant, MAAC achieves near-optimal performance within 150 rollouts while other take orders magnitudes more data.
AntEnv
HaifCheetahEnv
Average Return
4000
10000
3000
7500
5000
2500
1000
0
0.00
0.25
0.50
0.75
1.00
1.25
1,50
0.0
0.2
0.4
0.6
0.8
Time-steps
1e5
Time-steps
1e5
HopperEnv
Walker2dEnv
Average Return
2000
1000
-2000
2000
0
1
2
3
5
0.0
0.2
0.4
0.6
0.8
1.0
stme-steps
1e4
Time-steps
1e5
svg
mbpo
steve
sac
ours
Figure 2: Comparison against state-of thef-the-n art model-free and model-based baselines in four different MuJoCo environments. Our method, MAAC, attains better sample efficiency and asymptotic performance than previous approaches. The gap in performance between MAAC and previous work increases the environments increase complexity.
5.2 GRADIENT ERROR
Here, we investigate how the bounds obtained relate to the empirical performance. In particular, we study the effect of the horizon of the model predictions the gradient enror. In order to do so,
7

---

Under review as a conference paper at ICLR 2020
construct double integrator environment; since the transitions linear and the cost is quadratic for a linear policy, we can obtain the analytic gradient of the expect return.
Figure B depicts the L1 error of the MAAC ob- jective for different values of the horizon H as well what would be the error using the true dynamics. As expected, using the true dynam- ies yields lower gradient етгоr since the only source comes from the learned Q-function that is weighted down by "" The results using learned dynamics correlate with our assumptions and the
L1 Error of the Gradient
70
65
60
derived bounds: the error from the learned dy- namics is lower than the one in the Q-funtion, but scales poorly with the horizon. For short horizons the eπror decreases as we increase the horizon. However, large horizons is detrimental since it magnifies the error on the models.
Error
55
50
45
40
30.0
0.5
1.0
1.5
2.0
2.5
5.3 ABLATIONS
Time-steps
1e3
In order to investigate the importance of each of the components of our overall algorithm, we carry out an ablation test. Specifically, we test three different components: 1) not using the model to train the policy, i.c., set = 0, 2) not using the STEVE targets for training the critic, and 3) using single sample estimate of the path-wise derivative.
H=0
H=3
H=2
H= 10
H=5
true dynamics(H=5)
Figure 3: L1 error on the policy gradient when using the proposed objective for different values of the horizon H well as the error obtained when using the true dynamics. The results correlate with the assumption error the learned dynamics is lower than the error in the Q-function, as well as they correlate with the derived bounds.
The ablation test is shown in Figure 4 The test underpins the importance of backpropagating through the model: setting H to be O inflicts severe drop in the algorithm performance. On the other hand, using the STEVE targets results in slightly more stable training, but it does not have significant effect. Finally, while single sample estimates can used simple environments, they are not accurate enough in higher dimensional environments such as ant.
AntEnv
HalfCheetthEEn
HopperEniv
Walker2dFnv
Average Neeuun
2500
:0000
2000
8000
1000
0000
4000
-1000
1000
2000
1000
-2000
8.0
0.2
0.6
0.5
0.4
9.5
08
Time-stops
Time-steps
Time-steps
Time-steps
H-0
single sample
maac(ours)
Figure 4: Ablation test of method. We test the importance of several components of our method: not using the model train the policy (JI 0), using the STEVE targets for training the Q-function (-STEVE), and using single sample estimate the pathwise derivative. Using the model is component that affects most the performance, highlighting the importance of our derived estimator.
5.4 MODEL PREDICTIVE CONTROL
One of the key benefits of methods that combine model-basee reinforcement learning and actor-critic methods that the optimization procedure results a stochastic policy, a dynamics model and a O-function. Hence, have all the components for, at test time, refine the action selection by the
8

---

Under review as a conference paper at ICLR 2020
means of model predictive control (MPC). Here, investigate the improvement in performance of planning at test time. Specifically, we use the cross-entropy method with our stochastic policy as initial distributions. The results, shown in Table 2 show benefits in online planning in complex domains; however, its improvement gains are more timid in easier domains, showing that the learned policy has already interiorized the optimal behaviour.
AntEnv
HalfCheetahEnv
Hopperfinv
Walker2dEnv
MAAC+MPC
3.97e3 + 1.48c3
1.09e4 + 94.5
2.8c3 + 11
1.76e3 + 78
MAAC
3.06e3 + 1.45e3
1.07e4 253
2.77c3 E 3.31
1.61e3 = 404
Table 1: Performance at test time with (maac+mpe) and without (maac) planning of the converged policy using the MAAC objective.
CONCLUSION
In this work, we present model-augmented actor-critic, MAAC, a reinforcement learning algorithm that makes use of learned model by using the pathwise derivative across future timesteps. We prevent instabilities arisen from backpropagation through time by the means of a terminal value function. The objective is theoretically analyzed in terms of the model and value error, and derive a policy improvement expression with respect to those terms. Our algorithm that builds on top of MAAC is able achieve superior performance and sample efficiency than state-of-the-ar model-based and model-free reinforcement learning algorithms. For future work, it would be enticing deploy the presented algorithm on real-robotic agent.
REFERENCES
Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and J. Zico Kolter. Differentiable mpe for end-to-end planning and control, 2018.
A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5): 834-846, Sep. 1983. doi: 10.1 1LL1100/TSM....13333.1111
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. CoRR, abs/1 006607 2018. URL
org/ http//bss/11077 00111111077770111
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint 1885
2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. CoRR, abs/1809.00 809.05214, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-II), pp- 465 472, 2011.
Yilun Du and Karthik Narasimhan. Task-agnostic dynamics priors for reinforcemmn learning. CoRR, abs/1905. 09981, 2019.
04819
04819
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Masel-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 0902 2018.

---

Under review as a conference paper at ICLR 2020
G.R. Grimmett and D.R. Stirzaker. Probability and random processes, volume 80. Oxford university press, 2001.
xzSt2XK2ONKJ :chhoo/loooissooa.....aaaaaaaaaa scholar google. hrt.op/........oooooaaaaaaaaaaaa _st==,,
Sact=citationscdee
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-pooicy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv:1801.0129 8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, 11..5182,, 2018b.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
Zhang-Wei Hong, Joni Pajarinen, and Jan Peters. Model-based lookahead reinforcement learning. ArXiv, abs/1900.00011, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. CoRR, abs/1996.0822) 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In IN PROC. 19TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING, pp. 267-274, 2002.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-drit imagination for continuous deep rein- forcement learning. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.). Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 195-206. PMLR, 13-15 Nov 2017.
Péter Karkus, Xiao Ma, David Hsu, Leslie Pack Kaelbling, Wee Sun Lee, and Tomás Jozaao-Pérez Differentiable algorithm networks for composable robot learning. CoRR, htss/1905.11600 2019.
/tattp://ariivi org/
1el.11600 602
Diederik P Kingma and Max Welling. 1rXiv:13/2.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 1802. 2018.
Sergey Levine and Picter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 107 1071-1079, 2014.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (rXxvv:509.02977 2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning. planning and teaching. Machine Learning, ((3):293-321, May 1992. ISSN 1573-0565. doi: N0.1007/BF00992699
Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. CoRR, abs/18 1110118111 2018
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. ICLR, 2019.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver. and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp. 1928-1937, 2016.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. 2019.
10

---

Under review as a conference paper at ICLR 2020
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for model--ased deep reinforcement learning with model-free fine-tuning. arXiv preprint 0tto::708.025906 2017.
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-end differen- tiable optimal control, 2017.
Marcus Pereira, David D. Fan, Gabriel Nakajima An, and Evangelos A. Theodorou. Mpe-inspired neural network policies for sequential decision making. CoRR, abs/1802.05803, 2018. URL
http:// /ttt:://axxi..oo wttpp//wwwrg/ab/////22.0000 05800
J. Peters and S. Schaal. Policy gradient methods for roboties. In 2006 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2219-2225, Oct 2006. doi: 10.1 10.1009ROS00006 282564.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. CoRR, abs/1506.05254, 2015a. URL http::/arxivv.rgg 0tte/1506.052551
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897, 2015b.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint arXiv: 1804000645 2018.
Richard S Sutton. Planning by incremental dynamic programming, In Machine Learning Proceedings 1991, pp. 553-357. Elsevier, 1991.
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, edition, 1998. ISBN 0262193981.
Aviv Tamar, Sergey Levine, and Pieter Abbeel. Value iteration networks. CoRR, abs/ abss/1602.02867, 2016. IRLpbttp///axxtt
abs 1602. C2867
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. CoRR, abs/1906.08649, 2019. URL http://aaxxx oow.aiiiiiiii.iooooooooooooooooe http//aas.//..00.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, abs/1 0at///90700057777 2019.

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Here we prove the lemmas and theorems stated in the manuscript.
A.1 PROOF OF LEMMA 4.1
Let (0) and J. (0) be the expected return of the policy vo under objective and under the RI. objective, respectively. Then, we can write the MSE of the gradient as
al VoJa E
M) + Von" Q)112
- EBllVo - M) BlI (Q (Q))lla
whereby, M M - - 7r((s;) and M o 7
We will denote as the gradient w.r.t the inputs of network, xt = (8t, az) and & (8,, āz): where à ~ ae (8z). Notice that since / and ı are Lipschitz and their gradient Lipschitz as well, we have that Veă, K', where depends on the Lipschitz constants of the model and the policy. Without loss of generality, we assume that K is larger than 1. Now, we can bound the error on the Q as
[V - 12 =IVQV 8h VQV o 1ll
IIVQQ VQ)V 0R VQ(VOOH VoZH)l2
IVVQ- VQ la VorH \ VaTHl12
$ <<Ql| + -
SeOK" + LQQ VerHl22
Now, we will bound the term Vo$ Vost+1122
|| V
Vosttille 2
V V + V Voo. v.fVos: vafVoall
S VV. v. 0z: 2|2 + |V V..iVon - VoJVollll
Seyl|V + L, Vos Vos V Voail + ef Voû.le
S < + + Voš; - Vas + V
V (L, + L,
Ves,122
Hence, applying this recursion obtain that
111 - Se L-1 K
VV - Vot++1ll2 S of
where L = Ly + Liplan Then, the error in the gradient in the previous term is bounded by
I
[vo(Q - QQ1la S EQK# LQES L-1 KH
In order to bound the model term we need first bound the rewards since
VV M)ll2 <
( r(s:))ll
t=0
Similar to the previous bounds, bound now each reward term by
le+1 K
(v(((s))) - r(s)))ll I L,
L 6-1
With this result we bound the total error in models
H-1
r
Leg ((KII) -1 (TK) -
- - Milla <
Lg 1-1
Kt
(L-1)
7KL-1
7K
t=
12

---

Under review as a conference paper at ICLR 2020
Then, the gradient error has the form
(L-1) 7KI nk 7k x -1) +ro(7K)" + +Iqes --1
Leg (()KL)* -1 (TK)*
L
B[IVVoJ+(0) (O)la))
A.2 PROOF OF LEMMA 4.2
The total variation distance can be bounded by the KL-divergence using the Pinsker's inequality
DKL DKI(FelTT))
D -
2
Then if assume third order smoothness policy, by the Fisher information metric theorem then
(xz(Fo)(Tg)) alle 211 + o @11))
Given that ||0 - 01|2 - all VoJ, Vo ll2 for small enough step the following inequality holds
(MI(TOllTT) s a a E((eso 2H))
Combining this bound with the Pinsker inequality
(rv(Toll=g) s a a + +
A.3 PROOF OF THEOREM
Given the bound on the total variation distance, we can now make use of the monotonic improvement theorem to establish an improvement bound in terms of the gradient error. Let (0) and Ja (Ō) be the expected return of the policy ma and Ta under the true dynamics. Let p and e be the discounted state marginal for the policy To and Tg. respectively
(0) J.(
Σ (()) - ((s)Tgr(s,
9,a
< Tmax
Σ -
8,0
2r max
max
(a(a)))
(a(ss)
2r max Drv
Then, combining the results from Lemma 4.2 we obtain the desired bound.
A.4 ABLATIONS
In order to show the significance of each component of MAAC, we conducted more ablation studies. The results shown in Figure 5 Here, analyze the effect of training the Q-function with data coming from just the real environment, not learning a maximum entropy policy, and increasing the batch size instead of increasing the amount of samples estimate the value function.
13

---

Under review as a conference paper at ICLR 2020
AntÊnv
HalfChestahEEnv
HopperEnn
Walker2dFnv
20000
Average Netur
1000
2000
4000
1000
1000
2000
-2000
3000
800
0.25
0,50 0.75
100
125
1,30
0.2 0.4
0.6
08
Time-stops
Time-steps
steesstops
Time-stops
data
eopentrppy
5x batch
Figure 5: We further test the significance of some components of our method: not the dynamics to generate data, and only real data sampled from environments to train policy and Q-functions (real data). remove entropy from optimization objects (no entropy), using a single sample estimate pathwise derivative but increase the batch size accordingly (5x batch size). Considering entropy and using dynamic models to augment data set are both very important.
A.5 EXECUTION TIME COMPARISON
Iteration (s)
Training Model (s)
Optimization (s)
MBPO Iteration (s)
HalfChectahEnv
1312
486
738
708
HopperEnv
845
209
517
723
Table 2: This table shows the time that different parts of MAAC need to train for one iteration after
6000 time steps, averaged across seeds. We also add the time needed for MBPO for one iteration here for comparison.
14


Reviews:
[{"review_id": "S1e3iVZltS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After seeing the clarifications made to address the other reviewers and my own reservations, I lean towards accepting this paper.  It is a simple yet novel way to incorporate a model, and I appreciate the thorough results with multiple baselines and additional ablations to show importance of each component of the method.\n\n-------------------------------------\n\nThe authors suggest back-propagating through a learned dynamics model and provide a derivation on monotonic improvement of the proposed objective. They show increased sample efficiency, asymptotic performance matching model-free methods, and ability to scale to long horizons. They provide bounds on the error of the gradient when using the learned model and Q function and the total variation distance between policies trained using true dynamics versus the learned dynamics model. This leads to a theorem showing monotonic improvement of the policy under their algorithm, MAAC.\n\nDecision: Weak Accept. The work contains both theoretical and empirical results on back-propagating through a learned dynamics model compared to other model-based and model-free methods. However, I'm unsure about the novelty of the method itself. How is this different from other planning through back propagation methods? This should be an additional section in related work. As examples, there is Universal Planning Networks [1], Differentiable MPC [2], and Path Integral Networks [3], [4] present a way to differentiate through path integral optimal control. I will increase my score if these concerns are addressed.\n\nNits:\nMissing C in \"Contrary to these methods\" in Related Work section\nConclusion: analized -> analyzed\n\n[1] Universal Planning Networks - Srinivas et al. 2018\n[2] Differentiable MPC for End-to-end Planning and Control - Amos et al. 2018\n[3] Path integral networks: End-to-end differentiable optimal control - Okada et al. 2017\n[4] Mpc-inspired neural network policies for sequential decision making - Pereira et al. 2018", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklOqV3pKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents MAAC, a model-based reinforcement learning algorithm which makes use of path-wise derivative to optimize the policy. The learned Q function is used to estimate the terminal rewards and reduce instability and is trained by a STEVE-like style and Clipped Double Q. The dynamics model is learned by the same method in PETS. The policy is trained by directly computing the gradient to improve model-boosted Q. The theory shows that if learned model and learned Q function have similar derivative as the real one, the improvement of policy can be lower bounded. Experiments show that MAAC achieves the state-of-the-art and can be further improved by adding MPC. \n\n\nThe paper is well-written in general. The emperical results are very good and are claimed to be the new state-of-the-art. \n\n\nQuestions:\n\n1. Could you please elaborate how J_Q(phi) is calculated? I found one in preliminary but I suppose the one used in Algorithm 1 is different as the paper states that MAAC also uses the model to train Q. \n2. How is H(pi_theta) estimated? What's underlying state distribution of H(pi_theta)? \n3. The objective of policy network is similar to SAC's objective, but the optimization is different. SAC optimizes the objective by minimizing the KL divergence between pi and exp(Q). Do you have any comparison between BPTT and KL divergence minimization? \n4. How do you \"use the cross-entropy method with our stochastic policy as our initial distributions\" in MPC? \n5. Does MAAC work for Humanoid? \n6. How many samples are used to estimate the expectation in J_pi(theta)? For the single sample experiment, what will happen if the batch size is increased? \n7. In Algorithm 1, how does \"Sample trajectories T from \\hat{f}_\\phi\" work? \n8. To my understanding, when H = 0, the policy optimization doesn't use the model. Assuming Q function optimization uses the same H, the only usage of learned dynamics model is that pi and Q are trained in a different state distribution (D vs D_env). The ablation study shows that in this case, MAAC still outperforms SAC. So is it possible that D_model provides data augmentation? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gzAjBWqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#rebuttal responses\nThanks for the clarification!  However, I will keep my original score for these reasons:\n(1) Only 3 random seeds are used for each environment, which is not convincing as the variance of MAAC is large in some figures.\n(2) Baselines are only trained with 10^5 steps and do not converge. Thus it not fair to say that MAAC matches the asymptotic performance of model-free algorithms.\n\n#review\nThis paper constructs a model-based policy optimization algorithm (MAAC) that uses the pathwise derivative of the learned model and policy across future timesteps. The terminal value function is used to improve stability.  The theoretical guarantee of the error of the model-based gradient is presented. Experimental results show that MAAC outperforms SAC, STEVE, and MBPO on four environments in terms of the sample efficiency.\n\nThe experimental results are strong and I appreciate the plots of the gradient error in a simple task, shown in Figure 3. But I want to see the comparison of the final performance of each algorithm in these environments, and I doubt that the baselines do not converge.\n\nHowever, the paper is badly written. First of all, the authors claim that the pathwise derivate method is applied to optimize the objective function. But the detail of the method is missing. Secondly, I can not follow the procedure of how Q function is learned in the MAAC algorithm. Thirdly, Theorem 4.1 gives a performance improvement bound of the new policy w.r.t J_pi without the entropy term. But the MAAC algorithm optimizes the objective with the policy entropy term.\n\nAlso, MAAC applies many techniques in other papers. The paper does not clearly show the advantage of each component:\n(1) I want to see the experimental results of MAAC optimizing the objective without the entropy.  \n(2) The SVG(1) algorithm should be compared as a baseline as the SVG(1) also uses the gradient of the learned model to optimize the policy.\n(3) The policy and the Q function are optimized on both the real samples and the generated samples. I want to see the ablation study or justification on whether training on real samples or both samples.\n\nQuestions:\n1. How many independent runs are used in experiments?\n2. Does the computation of the pathwise derivate method cost much time？ Is MAAC much slower than SAC?\n\nI am happy to increase my score if the authors justify these questions.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel model-based reinforcement learning algorithm. The key difference with previous approaches is that the authors use gradients through the learned model. They present theoretical results on error bounds for their approach and a monotonic improvement theorem. In the small sample regime, they show improved performance over previous approaches.

After the revisions, reviewers raised a few concerns:
The results are only for 100,000 steps, which does not support the claim that the models achieves the same asymptotic performance as model – free algorithms would.
The results would be stronger as the experiments were run with more than 3 random seats.
In the revised version of the text, it's unclear if the authors are using target networks.

Overall, I think the paper introduces some interesting ideas and shows improved performance over existing approaches. I recommend acceptance on the condition that the authors tone down their claims or back them up with empirical evidence. Currently, I don't see evidence for the claim that the method achieves similar asymptotic performance to model free algorithms or the claim that their approach allows for longer horizons than previous approaches.
Decision: Accept (Poster)