Gradient Information Matters in Policy Optimization by Back-propagating through Model
Authors: Chongchong Li
Yue Wang
Wei Chen
Yuting Liu
Zhi-Ming Ma
Tie-Yan Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Model-based RL
Policy Optimization

Abstract: Model-based reinforcement learning provides an efficient mechanism to find the optimal policy by interacting with the learned environment. In addition to treating the learned environment like a black-box simulator, a more effective way to use the model is to exploit its differentiability. Such methods require the gradient information of the learned environment model when calculating the policy gradient. However, since the error of gradient is not considered in the model learning phase, there is no guarantee for the model's accuracy. To address this problem, we first analyze the convergence rate for the policy optimization methods when the policy gradient is calculated using the learned environment model. The theoretical results show that the model gradient error matters in the policy optimization phrase. Then we propose a two-model-based learning method to control the prediction error and the gradient error. We separate the different roles of these two models at the model learning phase and coordinate them at the policy optimization phase. After proposing the method, we introduce the directional derivative projection policy optimization (DDPPO) algorithm as a practical implementation to find the optimal policy. Finally, we empirically demonstrate the proposed algorithm has better sample efficiency when achieving a comparable or better performance on benchmark continuous control tasks.

Published as a conference paper at ICLR 2022
GRADIENT INFORMATION MATTERS IN POLICY OPTI- MIZATION BY BACK-PROPAGATING THROUGH MODEL
Chongchong Li : Yue Wang : Wei Chen 31, Yuting Liu Zhi-Ming Ma 4 & Tie-Yan Liu'
Beijing Jiaotong University
(18118002, ytlitlll jbjju..a edu. 2 Microsoft Research Asia
(yuwang5, Shyyliu@ee emicroofff... crosoft. com
Institute of Computing Technology, Chinese Academy of Sciences chenwei20228 2eeittttt...te ac. en
4 Academy of Mathematics and Systems Science, Chinese Academy of Sciences ma zm3 ant an.cn on
ABSTRACT
Model-based reinforcement learning provides efficient mechanism to find the optimal policy by interacting with the learned environment. In addition to treat- ing the learned environment like black-box simulator, use the model is to exploit its differentiability. Such methods require the gra- dient information of the learned environment model when calculating the policy gradient. However, since the error of gradient is not considered in the model learning phase, there no guarantee for the model's accuracy. To address this problem, we first analyze the convergence rate for the policy optimization meth- ods when the policy gradient calculated using the learned environment model. The theoretical results show that the model gradient error matters in the policy optimization phrase. Then we propose a two-model-based learning method control the prediction error and the gradient error. We separate the different roles these two models at the model learning phase and coordinate them at the pol- icy optimization phase. After proposing the method, derivative projection policy optimization (DDPPO) algorithm as a practical im- plementation to find the optimal policy. Finally, empirically demonstrate the proposed algorithm has better sample efficiency when achieving a comparable or better performance benchmark continuous control tasks. Codes available at httpss//gihbbbccm///CCaa//l/pp
effective way
introduce the directional
INTRODUCTION
Reinforcement learning (RL) is powerful technique for solving the sequential decision making
problems (Li 2018 Sutton & Barto 1998). Recent work on model-based RL (Nagabandi et al. 2018 Luo al. 2018; Kurutach
2018; Wang al.) 2019 Janner et al., 2019 Pan et al. 2020 has shown the power of first learning the environment model and then using it to do the policy optimization. Several methods are proposed to achieve the goal of getting similar performance by using fewer data, such as ensembles (Kurutach et al. 2018), probabilistic models (Chua et al.) 2018). and meta-learning (Clavera al. 2018).
In addition to treating the learned environment as black-box simulator, more effective way of using the model is to exploit its differentiability (Heess ct al.] 2015 Clavera et al. 2019 D' Oro & Jaškowski 2020 Amos et al. 2021), which is mainly focused this paper. To get the policy updating direction, these methods compute the policy gradient directly by back-propagating through the model. Therefore, the model gradient is used in the calculation and its error will influence the accuracy of the policy gradient. However, since the traditional model learning only aims to get the accurate prediction for the next state and the reward, there is guarantee for the accuracy of the
"This work was done when Chongchong Li was interning at MSRA. Corresponning author.

---

Published as a conference paper at ICLR 2022
model gradient. In other words, the algorithm requires the accurate model gradient, but we only learn to decrease the prediction enror which results in an objective mismatch.
In this paper, to address these problems, first theoretically analyze the problem and then propose our solution based on the theoretical results. First of all, we present the convergence rate analysis for the policy optimization algorithms in which the policy gradient is calculated using the learned environment model. By taking the model gradient error into account, we can see that the gradients of the transition and reward models matter in the policy optimization. Specifically, the bias of the estimated policy gradient used to update the policy is not only introduced by the prediction error of the learned model but also introduced by the gradient error of the learned model. Furthermore, the policy gradient bias due to the different types of model error will finally influence the convergence rate the policy optimization process.
Then, inspired by the theoretical results, we propose the two-model-based learning method. Ac- cording to the policy gradient bias and convergence rate analysis, in order to optimize the policy efficiently and accurately, and small gradient error. Therefore, we propose to set separate models for different purposes of usage. In the model learning phase, the prediction model aims to reduce the prediction error, and the gradient model focuses on minimizing gradient error. In the policy optimization phase, we will use the prediction model to rollout the data and use the gradient model to calculate the policy gradient.
need the learned environment model both with small prediction error
To make the proposed method applicable, we introduce the directional derivative projection policy optimization (DDPPO) algorithm. Our first goal is use data to estimate the gradient or Jacobian matrix for the environment model and use the estimator to learn the model's gradient explicitly. The challenge is that the state and action are usually with high dimensions and directly estimating the gradient or Jacobian matrix using data is intractable. Thus, we first estimate the directional derivative using data and then project the model's gradient or Jacobian matrix into these directions. Minimizing the error between the estimated directional derivative and the projection value, we can learn the model's gradient. Secondly, after learning the environment model with a more accurate gradient, we can leverage two-model- based learning method to do the policy optimization.
Finally, we conduct experiments on the simple environments and the benchmark MuloCo continu- ous control environments. The experimental results verify our theoretical findings and demonstrate the effectiveness of the proposed method.
Our main contribution can be summarized as follows:
1. We theoretically depict how the different model errors influence the convergence rate of the model-based policy optimization algorithm. The result shows that the gradient error of the model indeed matters in the convergence of the policy optimization.
2. We propose the two-model-basee learning method and the practical DDPPO algorithm which learns and uses two environment models (prediction model used for rollout and gradient model used to provide the gradient information) for the model-based policy optimization.
Empirically, we can achieve better sample efficiency in the MuJoCo continuous control tasks than state-of-the-ard model-bases and model-free methods.
2 PRELIMINARIES
Reinforcement Learning: We consider a discrete-time Markov decision process (MDP) M, de- fined by tuple (S,
where S and A are the state and action spaces, respectively. Here, f St++1 = f(st, (t, is the transition distribution, r : & r(st, at) is reward function, Po is the initial state distribution and y represents the discount factor. We define the return (J) as the expected sum of discounted rewards along trajectory (R Eee 7'r(s,, (r))s,,,)). The goal of reinforcement learning is to find policy a that maximizes the expected sum of discounted rewards, i.e.,
oo
max J (() = max Eng R = max Bag
Σ (r))s))))
(I)
Model-based RL is characterized by learning the transition model using data collected from inter- action with the environment. Typically, we use parametric function So to denote the transition
2

---

Published as a conference paper at ICLR 2022
model trained by a neural network. The state predicted by the learned model is defined as št+1 and 5141 fo- Similarly, we use a parametric function vo to denote the reward model. Generally use M to denote the true model and M the learned model. Then $1+1, Fr ~ Mo(si, at).
The model is trained by supervised learning. typically, via maximum likelihood:
Jan (8p) log MM ($141, Palst, a.)
(2)
where (st, des Tt, 9r+1) are data collected from interaction with the environment.
Policy Optimization According to the Policy Gradient Calculated Using Model: After we learn the model, we can use the model to rollout trajectories and use the data to calculate the policy gra- dient for the purpose of policy optimization. Specifically, denote the policy parameters as Ø, we can calculate the return J using generated data as (Ø) = 1- >'re]. where $t41, ~ M (3r, āe) and T (&E). To optimize the objective, one ods such as A3C (Mnih et al.] 2016). PPO (Schulman et al.) 2017). SAC (Haarnoja et al. 2018).
the traditional policy optimization meth-
Beyond that, many previous works argue that we can use the model to calculate the policy gradient by pack-propggating through the model (Heess et al. 20155 Clavera et al. 2019: Amos et al. 2021 leveraging the re- ce-aarametrrization trick (Kingma & Welling 2014).
Specifically, since we have learned the environment model, we can calculate the gradient of the
objective w.r.t. & by chain rule as shown in Equation in which the gradients need to be calculated using the learned model.
dr, drs dag d
day day dscii da;
dJ
dJ dat
dre
DC
dra dsu d8 dsu
dac d$
E
E
dه
dae de
das
dst+1 dae
The expectation is taken randomness from random policy and transition. To optimize the policy, we can update the policy parameters in the gradient direction. Since we don't have the true gradient, use learned model estimate the policy gradient denoted AR dø
ar is learning rate.
dR dø
Φ141 tb +
(4)
Notations We use fraction notation d to represent the gradient calculated by taking gradient
T on the true model and use the "hat" version d: to represent the gradient calculated on the learned model. We use the "tilde" notation to represent the state, action or reward generated using the learned model and the without-tilde notation to represent those variables generated using the true model. For example, the notation de++1 denotes the gradient function value calculated by taking des
gradient for the output of the transition model s' (s' = =(ss e)) with respect to its input given input value equals $t. We use the notation Ey to represent the transition model prediction error
which means, for V 8, a, we have E [[((s.u) u) [(s, a)) ee. We use the notation e) to represent
the gradient error of the transition model. For V s, a, we have
ds'
ds'
ds
ds'
df(s, a)
df ds
df(s.
df df(s
+
E
= E
+
S
(5)
ds
ds
da
da
ds
da
da
Similarly, we use the notation ef to represent the gradient error of the reward model.
3 RELATED WORK
Due to the space limitation, we clarify several highly related works in this section and put the de- tailed related work into the Appendix A Overall speaking, most of the model-based policy opti- mization algorithms aim to calculate the policy gradient accurately and efficiently. These works can be divided into two categories, i.e., how to learn the model and how use the model.
Learn the better model: Chua et al. (0018) design an ensemble- buseed neural network to learn the true model. Zhang et al. 2021al design the dropout mechanisms improve the robustness of the
3

---

Published as a conference paper at ICLR 2022
learned model. Zhang al. (2020b) design the auto-regressive network structure improve the accuracy of the model prediction.
Consider the decision-making phase when learning the model: These works are always called Decision-aware model learning (DAML). D'Oro et al. (2020) use policy gradient to get the re- weighted model prediction error for each sample. |aaahmand al. (2017) use the value function to get the re-weighted model prediction error for each sample.
Use the learned model to argument data: Janner al. (2019) use the learned model to rollout the trajectories and use both the generated and the true trajectories to train the SAC algorithm. Yu et al.
(2020) [ee al. (2200) extend the similar idea to the offline model-based RL setting.
Leee
To calculate the policy gradient by back-propas propaaaaagg through the model: SVG (Heess et al.
2015) presents framework for learning continuous control policies using pao--propggating through the model. More recent works (Clavera et 2019 Amos et al. (202T) extend the idea by combining the learned Q-function and the SOTA policy optimization algorithms and thus achieve better perfor- mance. MAGE (D' Oro & Jaskowski, 2020) computes the policy gradient using value gradient and proposes to reduce the value gradient through TD learning.
Different from the above methods, we further consider the influence of the model gradient error with respect to the convergence of the policy optimization. We will use multiple function approximators to separate different requirements for the model at the learning and using phase. Moreover, we can leverage the insight inside of these related works to better design our model-based algorithms.
4 MODEL-BASED POLICY OPTIMIZATION BY CONSIDERING THE MODEL GRADIENT ERROR
In this section, will analyze the influence of the model error in the policy optimization process. First of all, for the policy optimization algorithms that using policy gradient calculated according the learned model, we theoretically show that both the prediction the learned model matter for the policy gradient bias and the convergence rate. Secondly, based on the theoretical findings, we propose our method to learn the accurate model for policy optimization. Thirdly, we show the implementation details of our method and propose the DDPPO algorithm.
and the gradient error of
4.1 CONVERGENCE RATE FOR MODEL-BASE POLICY OPTIMIZATION
In this section, we will prove the finite sample bound for the model-based policy optimization algo- rithms. Due to the space limitation, we put the full proof into the Appendix
Due to the space limitation, we simply put the assumptions that is necessary for understanding the theoretical results here and put the detailed assumptions into Appendix C
The transition model, the reward model and the policy function are all Lipschitz continuous. The state space and the reward are both bounded. For fixed policy, the stationary distribution exists and has the uniform ergodicity property:
Assumption 1.
drv (P(se+r E -18e --)===))pp0( ())))())) S "mp" VT 2 VsES.
(6)
First of all, we analyze the bias of the policy gradient introduced by the model error. Recall the policy gradient formulation (Equation(3) in the section 2 In the model-based setting, need to use
the learned model to calculate the gradient value dr. drs dsg., da:il ) in Equation Since the
'da" dau "2841' da.
model error is unavoidable, it is necessary for us to analyze the bias of the policy gradient introduced by the model error.
Theorem 1, Suppose assumptions hold. The policy gradient bias es can be bounded as
dR-
} + Crez + Gse
EM
=
ds
dip
(7)
where Co, Cr, Cs are three constants related to the Lipschitz constants of the transition model and the reward model and
Remark: The above theorem shows that the difference between the policy gradient calculated using two different models can be upper bounded by the difference of the gradient of the reward modeley.
4

---

Published as a conference paper at ICLR 2022
the gradient of the transition modeley, and the prediction of the transition modeley- The result suggests that gradient error matters in estimating the policy gradient. Now, one natural conjecture is that the gradient error will influence the convergence of the policy optimization process. The next theorem depicts the influence rigorously.
Secondly, we analyze the convergence rate for the policy gradient algorithms. Since it is always a non-convex optimization problem, the commonly used measure is the speed of the gradient norm converges to zero. Recall the update rule (Equation[4). We update the policy parameters by following the estimated policy gradient direction. Intuitively, the model error will influence the convergence. and X holds. L, is the Lipschitz constant of the objective function for the parameter @ and 1% is the smoothness constant of the objective function for the parameter Φ. Suppose the policy parameters are updated as shown in Equation 4 and the learning rate is set as Cz+, We have
Theorem 2. Suppose assumptions 3
d.J der
2 Σ at
20 2raOL3 Σ al (1323++1312)) +22.mpp + (8)
T
min 0CTET
E
The above theorem shows the rate of the model-based policy gradient method converges its stationary point. This setting is commonly used in the previous papers that analyze the con- vergence rate for the actor critic method with function approximation and non-convex optimization (Wu et al. 2020: Kumar et al., 2019 Qiu et al.) 202T Ghadimi & Lan 2013). The results show that
Remark:
if we set the learning rate a, that satisfies the condition limy +00
E Eis 01
0, the policy optimiza-
tion process will converge to near its stationary point with rate O
As an example, if we
EE
set the learning rate as dr V we can get the convergence with rate O
VT
The bias is introduced by the policy gradient bias and the Markov noise. The bias in- troduced by the Markov noise can be arbitrarily small if we set sufficiently large which doesn't 't change the main order of the convergence rate. The bias introduced by the policy gradient bias is analyzed in Theorem According to the theoretical results, reduce policy gradient bias, we need to control the model prediction eтrоr and the model gradient error in the model learning phase.
Remark:
4.2 TWO-MODEL- -BASE LEARNING POLICY OPTIMIZATION
The theorem in the previous subsection indicates that both the model prediction error and the model gradient error are crucial for policy optimization. In this subsection, we will discuss how to learn and use the model taking both errors into account. Specifically, our approach repeatedly alternates between collecting samples from the environment, training the model, and optimizing the policy.
Model Learning: Different from the classical model-based methods, we design a two-model learning method to achieve the goal. We call these two models prediction models and the gradient models respectively. Our goal is to use the prediction model to predict the next state and the reward and to use the gradient model calculate the policy gradient. Training of the prediction model is commonly through supervised learning, e.g., maximum likelihood with early stopping on a valida- tion set (Janner al. 2019; Clavera al.) (019). Training of the gradient model is more complex. For the purpose of reducing the gradient error, we need to use the data collected from the environ- ment to estimate the gradient of the environment model and construct the mean square gradient error for the parameter optimization.
Policy Optimization: To update the policy, we first generate trajectories by unrolling the latest policy on the prediction model. Then, we calculate the gradient of the objective by back proppaaoaggg through the gradient model. Finally, we update the parameters in the policy by following the policy gradient direction.
4.3 DIRECTIONAL DERIVATIVE PROJECTION POLICY OPTIMIZATION: A PRACTICAL IMPLEMENTATION
In this subsection, we propose Directional Derivative Projection Policy Optimization (DDPPO) as an implementation of our proposed two-model based learning method. The key question how to use the data collected from the environment to estimate the gradient or Jacobian of the environment
5

---

Published as a conference paper at ICLR 2022
Algorithm 1 Directional Derivative Projection Policy Optimization
1:
Initialize the policy T.. predictive model Mž, gradient model ME, value function Q Initialize environment replay buffer Denv, model replay buffer D Poodel
2:
3: repeat
4: Take action in real environment with policy To. and add transition (8, a, 5') to Denv
5: Train predictive model M2 and update Op by maximizing Jxa=(0p)
6: Train gradient model M22 and update Og by maximizing (0g)
7:
for Nrottouts steps do
8: Sample 8, uniformly from Dene
9:
Perform k-step model rollout on Mo, starting from with policy To
10:
Add samples to Dmndel
11:
end for
12:
D + Denv U Dmodel
13:
for Nupdate Q steps do
14:
Update using data form D: v 6- v do (V))
15:
end for
16:
for
Nupdate Paticy steps do Sample trajectories of length H with policy and predictive model My Update & using sampled trajectories and gradient model Mg, & + + An Jz)) end for
17:
18:
19:
20:
until The policy performs well in real environment M
21:
return Optimal policy Rg*
model. Note that the dimension of the Jacobian of the learned environment model is the product of the state's dimension and the action's 's dimension, which is usually large. Directly estimate the gradient or Jacobian matrix is intractable. For model-based RL, we cannot estimate gradients by perturbing one dimension of a state or action and estimate the partial derivative, because we can only sample data from the environment, and this is contrary to the purpose of efficiency. Besides these, we will also discuss how to design two models to rollout and caleulate gradients respectively when optimizing the policy. The overall algorithm pseudo-code is shown in Algorithm | More details for explaining the pseudo-code can be found in Appendix B.1
Model Learning: Prediction loss and the gradient loss.
The prediction model MP is trained following Janner et al. 2019) via maximum likelihood (Fqua- tion 2) To improve the ability of models to portray complex environment, we use a bootstrap ensem-
ble of models ,,,, Meg) which is consistent with Janner et al. (2019): ; Clavera et al. (2019).
Different from the prediction model, the gradient model M/9 aims to better characterize the gradient of the environment instead of an accurate prediction. Hence, we hope to trade-off the original maximum likelihood objective by a gradient loss term that constraints the Jacobian of the learned model. Although the model's gradient or Jacobian matrix is difficult to estimate directly on account of the high dimensions, the directional derivative can be estimated by finite difference using data sampled. Meanwhile, the directional derivative of the learned model can be calculated by projecting the model's gradient or Jacobian matrix into a certain direction.
Hence in this paper, we use the error between the estimated directional derivative and the projec- tion value to constrain the learned model's gradient. Note that for an input x = (s, a), and the corresponding output (x) with dimension d, the projection of the Jacobian along a vector i.e., (x) - I can be estimated by finite difference, M()+ Therefore, we use the finite
||a|| difference as a target to constraint the Jacobian of the gradient model. Since the finite difference converges to the directional derivative when the norm of u goes to zero, the data point æ + v should be close to x. So in this paper, we sample the nearest n data points of x to calculate n directional derivatives in different directions and use the finite difference to constraint the Jacobian of the learned model. Sampling the nearest points is for a more accurate estimation of the directional derivatives since the finite difference is closer to the true directional derivative the two inputs are closer. Then, the gradient loss item can be constructed as follows:
R
M() da (M():) M (2

---

Published as a conference paper at ICLR 2022
InvertedPendlllum
Hopper
Walker2d
12.00
4000
10000
2000
800
2000
Awwaae
6.00
2000
400
1000
2.00
10k
13k
23k
50k
75k
1004
23k
50k
100k
HalfCheetah
Ant
Humanoid
6000
5000
20rmge Rebur.
5000
Rebm
6000
4000
3000
3000
40.00
2000
2000
20:00
1000
1000
%
Stops
Stops
Stops
Ours MBPO
STEVE
SVG
MAAC
MAGE
SAC
Figure 1: Learning curves of our algorithm and baselines six different MuJoCo environments. Solid curves
express the mean of five trials. Shaded regions correspond to standard deviation among five trials. Our method has better sample efficiency than the model-based baselines. The improvement of performance over previous algorithms is large on complex environments, i.e, Ant and Humanoid.
where U2 denotes directions x; I. and D; the nearest n points of in the buffer. Intuitively, the gradient at æ can be constrained if there are enough points close enough to æ. And a relatively small gradient loss item means the learned model should have a gradient such that its first-order approximation is consistent with the given data. By leveraging the gradient loss term, the gradient mode M9, is trained via a multi-objective loss:
Joo (0,) E [ogg
ax)] + w R.
(10)
Note that here we don't 't prefer to learn gradient model using gradient loss term only. We can just use the data in the buffer to estimate few of the directional derivatives since the points sampled should make the estimation correct. So here we also use the prediction loss constrain the hypothesis space of the learned model to prevent large bias local optimal. Finally, the trained gradient model and the prediction model will be used for policy optimization.
be close to the
Model Usage: Policy optimization by baopppropgg through model. In our DDPPO algo- rithm, to train the policy, the objective function we use is constructed by following SAC (Haarnoja eet al. 2018), since SAC achieve outstanding performance. The policy is Iearned by rolling out the prediction model for H steps and maximizing the following objective function:
(+( = E
(F - a log To (arl3))) + Σ
( ""(Q(in an) a log To (o((a(())))
(11)
where 8.41 MP 4p- 50 random sampled from the environment buffer D Denes a ~ To. a is the tem- perature hyperparameter in SAC. Note that the policy can be learned by computing the derivative of the objective since the learned prediction model MP is differentiable. However, in our approach, we replace the gradient T2....M(' a²) by aa MM a,) calculate the full gradient, believing in that the gradient model can capture the true gradient better than the prediction model.
As for the Q-value function, it can be trained by minimizing the Bellman residual. practice, we follow SAC (Haarnoja et al., (018)) that trains two Q-functions to mitigate positive bias which is known to degrade the performance of value- methods (Hasselt, 20100; Fujimoto et al. 2018), and make use of target Q-function which has been shown to stabilize training (Mnih et al. 2015).
5 EXPERIMENT RESULTS
The goal of our experimental evaluation is to answer the following questions: (1) How does our algorithm perform on benchmark reinforcement learning tasks, compared to state-of-the-ar model- based and model-free algorithms?
(2) Does it matter to explicitly learn the gradient of the model by using the gradient loss? (3) How does the sensitivity of our algorithm when changing the hyper- parameters and w that are related to the gradient model learning? (4) Why build two models? To answer these questions, we evaluate our approach on model- based continuous control benchmarks in the MuloCo (odorov et al. 2012). Experimental details can be found in Appendix B.2
7

---

Published as a conference paper at ICLR 2022
Pendiuumm
LOR
Pendulum
LGR
-690
-00
-233
Kags
with gradient loss
thout gradientt loss
gradient orror
prodictbee aror
The left two figures show learning curves of models with and without gradient loss on two simple environments, Data used to train each model is collected by a random policy. Solid curves express the mean of gradient error of five trials. Dotted lines express the mean of prediction error during the training of the model. Shaded regions correspond to standard deviation among five trials. The model with gradient loss has lower gradient error than the model without gradient loss. The improvement is more pronounced in nonlinear cases, i.e., Pendulum environment. The right figures show the overall performance of using two different types of model, which yields the importance of considering the gradient loss.
Figure 2:
5.1 COMPARISON WITH STATE-OF-THE-
To answer the first question, compare method against state-of-the-art model-based and model-free methods in terms of sample complexity and asymptotic performance. Model-free base- line includes SAC (Haarnoja al.; 2018), which is widely accepted baseline. For model-based methods, we select MBPO (Janner et al. 2019) and STEVE (Buckman et al., 2019) as our base- line which both use short-horizon model-baset rollouts. We also compare to model-based methods SVG (Heess et al. (215)), which firstly uses the derivative of models to learn the policy, and MAAC, (Clavera al. (019)) similar to SVG, however entirely relies on the predictions of the model, remov- ing the need for likelihood ratio terms. MAGE (D' Oro & Jaskowski 2020) is added to comparison since this method focuses on the gradient of the Q-function.
The learning curves for all methods, shown in Figure I highlight the strength of our method in terms of performance and sample complexity. In all the MuloCo simulator environments, especially in two higher-dimensiee dimensional tasks, Ant and Humanoid, our method learns faster and has better efficiency than previous model-bast based and model-free methods. Note that MAGE learns fastest on Humanoid, and the trick of this algorithm can also be applied to our method. However, even without the help of tricks in MAGE, we can still achieve better asymptotic properties. We get return around 5400 in the Humanoid environment using just over 100 thousand interactions with the environment, while around three times as many interactions are needed for MBPO to achieve the same performance.
5.2 GRADIENT ERROR ANALYSIS
In this subsection, we answer the second question, i.c., does it matter to explicitly learn the gradient of the model by using the gradient loss? Firstly, we aim to analyze the difference between the gradient of the true model and the learned model. We collect experience using a random policy and train the model with and without gradient loss respectively. Then we calculate the MSE error of the gradient of the model. Figure shows how the gradient error and the prediction error changes during training of the model on two simple environments that the true gradient is analytical available. From the results in Figure we can see that the prediction error for both models decreases to zero while the gradient eтror of the model without gradient loss keeps larger than that of the model with gradient loss. The learning curves highlight the strength of adding a gradient loss to the model in terms of high prediction accuracy and accurate gradient estimation under the simple case.
5.3 ABLATION STUDY
To answer the third question, i.c., how does the sensitivity of our algorithm when changing the hyper- parameters, and to show the results in comprehensive way, we use a cim-dimessional environment, Hopper, and a nigh-dimensional environment, Ant, as our representative and then conduct two sets of experiments them. Figure 3 shows the results. The left column shows how the efficiency of our algorithm changes by tuning n.. The results demonstrate that, regardless of the value n., our method is better than our variant without gradient loss. The middle column shows the results for tuning w. The environment
results indicate that the weight recommended to be smaller for a higher-dimensionn and the performance is not sensitive to the number of directional derivatives used to construct the gradient loss, even only one directional derivative used can also help to improve the performance.
8

---

Published as a conference paper at ICLR 2022
Number of points: Hopper
Welght: Hopper
Method: Happer
4000
wkhout gradiant
4000
With gradinn loes
4000
ourtin=1: Lt
cunslmm 23
without gradiant l6ss
3000
oursin=51
Average Tatum
3000
Ret.m
3000
singte model
2000
MAAC
2000
MAAE
2000
1000
1000
1000
25k
75k
50k
75k
25k
30k
75k
Steps
Number of points: Ant
Weight: Ant
Method: Ant
6000
withot gradiant loss
purstnwai 1I
60000
Without pridenn locs
curt/memi 1)
6000
without gradiant loss
Jusennge Retum
pursin=25)
cursime O)
HAAC
Avrraga
2000
MAAC
4000
MAAC
4000
2000
2000
2000
00
0
0
Stops
Stops
Steps
Figure 3: Left: This figure indicates our method is not sensitive to the number of directional derivatives used to construct the gradient loss. Middle: The performances of our method with different w, the coefficient in front of the gradient loss. The weight recommended to be smaller for ligher-dimeenionaa environment. Right: Our method outperforms the variant (without gradient loss) which underpins the importance of considering the gradient error when training the model. And the higher performances than the variant (single model) that only uses one model highlight the strength of the two-model-based structure.
Hopper
Ant
4000
-uzasre mame
durti eparama -2:00
044.4 upaar
Basolinale para
3000
Aeargg 207
2300
4000
2300
3000
2000
500
1000
2.5k
50k
75k
25k
5.1c
75K
200k
325K
Sters
Figure 4: Our method performs better than baselines with similar number of parameters which supports that the improvement is not actually from the additional parameters. Results for all six domains are shown in Figure iin Appendix [B.
Now we answer the last question, i.e., why build two models. To investigate the importance of the gradient loss and role of two-model-based. we remove the gradient loss in our method and retrain the policy (without gradient loss) to see the importance of considering the gradient loss. To see the impact of training two models for prediction and gradient calculation respectively, we design a variant of our method that trains only one model with gradient loss used for both purposes (single model). The experimental results are shown in the right column of Figure The results underpin the importance of considering the gradient error when training the model if back-propagating through it. On the other hand, the results highlight the strength of the wo-model-based structure. We also draw the prediction eтror of both models in our algorithm and the results are shown in Appendix B33 The prediction error of the prediction model is lower than that of the gradient model, which further validates the idea of two-model- -based learning. To address the concern that the improvement of the performance may be caused by more parameters, we design a variant of our model that trains model with double parameters, however without gradient loss. The experimental results shown in Figure 4 that yields the importance of considering gradient information.
CONCLUSIONS AND FUTURE WORK
In this paper, we theoretically amalyze how the model gradient error and the model prediction error influence the policy gradient bias and the convergence rate of model-based policy optimization. The theoretical results motivate us design wwo-model-based learning method for policy optimization. We design the learning and using methods for the two models respectively by taking the model gra- dient information into account and propose DDPPO algorithms practical implementation. Our algorithm can achieve superior sample efficiency than state-of of-the-art model- -ased reinforcement leamnigg algorithms on challenging high- diggn-iinnsional continuous control tasks. For future work, it is enticing to design better gradient estimator in order to optimize the gradient model more efficiently. The idea about learning the model according to its usage instead of only considering its prediction error is also general enough to be applied to other problems beyond reinforcement learning.

---

Published as a conference paper at ICLR 2022
REFERENCES
Romina Abachi, Mohammad Ghavamzadeh, and Amir-massoud Farahmand. Policy-Aware Model Learning for Policy Gradient Methods. arXiv:2003.00033 [cs]. January 2021. URL http:
llaxxvvcrr/abba. wttpr///aaa//.....//o// 2003. 00030
arXiv: 2003.00030.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, pp. 6-20. PMLR, May 2021. URL http::/proceediinge mlr www.ppess//v14)
amos21a html
ISSN: 2640-3498.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lec. Sample-
Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. arXiv: /007://80000660 [cs, stat), June 2019. htRp: httt::://arrii..cc//aa////////// 110pp//1017700165555
arXiv: 1807.01675.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Reinforcement Learning in Handful of Trials using Probabilistic Dynamies Mod-
Deep
els. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httpss://pppers wttp:ssswspssppppeeaa.eeaaa wttpsc/aapp////1///ss//tt// 324558857799905112825555-----
html
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abheel. Model-Based Reinforcement Learning via Meta-Policy Optimization. In Conference on Robot Learming, pp. 617-629. PMLR, October 2018. httpt/pppp/p...eciiinno........r
v87/claveralBa. html,
ISSN: 2640-3498.
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-Augmented Actor-Critic: Backpropagating
through Paths. In International Conference on Learming Representations, September 2019. URL
https://
/ttpp:://openreview wwt.eeew.eeeeeee..e.e..eeeeeee.e hew.:rrwwwee..ettffooolledd---e
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient Model-Based Re- inforcement Learning through Optimistic Policy Search and Planning. In Advances in Neural Information Processing Systems, volume 33, pp. 14156-14170. Curran As-
sociates, Inc., 2020. URL httos://papers. nips. cc pccppaperr/22//haaa.... $3669988334452242252222773333-23
Wojeiech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pas- canu. Sobolev Training for Neural Networks. In Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL https:/tt: /papprss///pppp..ppp/////s/s nips. nipp.cc/paper 2017/hash/75Ba066 5111 Aoo htm
Pierluca D Oro and Wojciech Jaškowski. How to Learn Useful Critic? Model-based Action- Gramient-Estimator Policy Optimization. In Advances Neural Information Processing Systems, volume 33, pp. 313-324. Curran Associates, Inc., 2020. URL -ag
nips wwppsccc... cc/ Abstract.
Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: a model-based and data-efficient ap- proach to policy search. In Proceedings of the 28th International Conference International Conference on Machine Learning. ICML'11, pp. 465 465-472, Madison, WI, USA, June 2011. Om- nipress. ISBN 978 378-1-4503-0619-5
Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient-Aware Model-Based Policy Search. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))3801 3808, April 2020. ISSN 2374-3468. doi: (0.1609/aaii..34i000 534104.5791. aaai wttp://aaaaaaa..........o.. http:///w.....aaaaaaaaaaaaaaaaaa
URL / 04.
5799
Number:
H. Drucker and Y. Le Cun. Improving generalization performance using double backpropagation.
IEEE Transactions on Neural Networks, 3(6)9991-997, November 1992. ISSN 1941-0093. doi: 10.1109/72. 165600. Conference Name: IEEE Transactions Neural Networks.
10

---

Published as a conference paper at ICLR 2022
Amir-Massout Farahmand, Andre Barreto, and Daniel Nikovski. Value-Aware Loss Function for Model-based Reinforcement Learning. Artificial Intelligence and Statistics, pp. 1486-1494.
PMLR. April 2017. URL hatp://proceedings. mlr /ttecceediingg....reeeeee/////// httaa////aaaaaaaaaaaaaaaa
html
ISSN: 2640-3498.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning.
1eXivv1803000100 [cs, stat], February 2018. URL http:://aaxxii..oorabb/////a//
00101 arXiv: 1803.00101.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learn-
ing. pp. 1587-1596. PMLR, July 2018. URL htt.hhssss/pocceeediiiiiee http:/oceeeenoooss........ssa/a
m press/v8C/
fujimotol8a.in html
ISSN: 2640-3498.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth methhds for nonconvex stochas- tic programming. SIAM Journal on Optimization, 23(4):2341 -2368, 2013.
methods
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor In Interna- tional Conference on Machine Learning, pp. 1861-1870. PMLR, July 2018. URL https:
//proceedings.mit press wtteasssvvvv//e http//////aaaaoaaaaaaaaa
html, ISSN: 2640-3498.
Hado Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems, vol- ume 23. Curran Associates, Inc., 2010. URL sttpss:/pproceed.nng hwwp://owco.ee.ooooeueor....o..o neurips.ccr/////eeoo
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yu- val Tassa. Learning Continuous Control Policies by Stochastic Value Gradients. In Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015.
URL httos::/papers wttppps.c/ppppr///5////ae. [4510000355422000034355511r00-22
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model- Based Policy Optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019, URL URLp::t::://ppperrs nips.. pttppp.cccppppprr2/////aaasa
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning. In Proceedings of the 1st Annual Conference on Robot Learning, pp. 195 206. PMLR, October 2017. URL https: http:///oocceenng..m.em wwt.:sss.....pp.eeeeeeemm ISSN: 2640-3498.
kalweitl 17. h
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future. In International Conference Learning Representations, September 2018. URL htttps://openrevieet
net htt.:a/neeeetooommcceceeeeeeeeee
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv: 14 [cs, stat], May 2014. htLp: /atraaiiivv/..////////aaa//aaavl wwt.:/o////bb/s/eee 6114 arXiv: 1312.6114.
Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actor- critic method for reinforcement learning with function approximation. arXiv preprint 1XXiv:990.008/22 2019.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-Ensemble Trust-Region Policy Optimization. In International Conference on Learning Representations, February 2018. URL http:ts:://peeeeeeeeeee net wtwp:/eeeeeeooooiddddd-ite
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learn- ing: what are the crucial model properties and which model to choose? In International Confer-
ence on Learning Representations, 2021. URL https:/ httt:::://ppezevview hwt.:oruufffee..e/ forum?id
p5cy1694568

---

Published as a conference paper at ICLR 2022
Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representatiinss 2020.
Yuxi Li. Deep Reinforcement Learning. rrXiv:/810006333 [cs, stat), October 2018. URL http: //arxiv.org/abs////./
06339
arXiv: 1810.06339.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. In International Conference on Learning Representations, January 2016. URL https: //openreviee ew. net.
d=t__080-82
Angelo Gregório Lovatto, Thiago Pereira Bueno, Denis Mauá, and Leliane Nunes de Barros. Decision-Awar Model Learning for Actor-Critic Methods: When Theory Does Not Meet Prac- tice. October 2020.
net/ (nt
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Al- gorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guar- antees. In International Conference on Learning Representations, September 2018. URL
Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Er- mon. Calibrated Model-Based Deep Reinforcement Learning. In International Conference Machine Learning, pp. 4314-4323, PMLR, May 2019. URL httpth:sspppceeeeeeeeeee htttps://procceeingss. mll..
html
ISSN: 2640-3498.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540::52-533. February 2015. ISSN 1476-4687, doi: 10.1008//ature1400 URL Number: 7540 Publisher: Na-
https://wwt htpp:://www.nature nature. com/ wctp///riicclle/ttture222333 ture Publishing Group.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In International Conference on Machine Learning, pp. 1928-1937. PMLR, June 2016. URL http::/ www.w.ww........tm...t..m.. pwwps/eesss.... wattesss/44/mmiilie
m.r.
html ISSN: 1938-7228.
Nirbhay Modhe, Harish Kamath, Dhruv Batra, and Ashwin Kalyan. Model-Advantage Optimization for Model-Based Reinforcement Learning. arXiv:21066 14080 fcs, stat), 2021. URL http:
.0tp://2000...000000 arXiv: 14080.
14080
Anusha Nagahandi, Gregory Kahn, Ronald Fearing, and Sergey Levine Neural Network Dy- namics for Model-Based Deep Reinforcement Learning with Model-Free Fine- Tine-Tuning. In IEEE International Conference on Robotics and Automation, pp. 7559-7566, May 2018. doi: 10.1109/
Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-Aware Prediction: Learning to Model What Matters. In International Conference Machine Learning, pp. 7207-7219. PMLR, November 2020. htllhtt::///oceeeeeee 3498.
.mr. ott.
html
ISSN: 2640-
Yurii Nesterov et al. Lectures on convex optimization, volume 137, Springer, 2018.
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the Model When Is Confident: Masked Model-based Actor-Critic. In Advances in Neural Information Processing Systems, volume 33, pp. 10537-10546. Curran Associates, Inc., 2020. URL https:://apprre nip.. Soger/202200 http://///../aaate33ecceeaaaeaa.
- .html
Shuang Qiu, Zhuoran Yang. Jieping Ye, and Zhaoran Wang. On finite-time convergence of actor- critic algorithm. IEFE Journal Selected Areas in Information Theory, 2(2):652-6644 2021.
12

---

Published as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06344 [cs], August 2017. URL
arXiv: 1707.06347.
Yuda Song and Wen Sun. PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration. In International Conference on Machine Learning, pp. 9801-9811, I. PMLR, July 2021. URL 2640-3498.
www.p..re.........m..m.e. wtt..peeee//1111//g/gee2 httmlll..mmmm html
ISSN:
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cam- bridge, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, October 2012. doi: 10.1 100.///O...002..63666666
ISSN: 2153-0866.
Manan Tomar, Amy Zhang, Roberto Calandra, Matthew E. Taylor, and Joelle Pineau. Model- Invariant State Abstractions for Model-Based Reinforcement Learning. arXiv:2 102.0200000 [cs], June 2021. URL/tttp://axxttt
org/ abs 2200. 09850 arXiv: 2102.09850.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking Model-Based Reinforce- org/
ment Learning. arXiv: 1907.02900 [es, stat], July 2019. URL
1907.02057
arXiv: 1907.02057.
Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, and Tie- Yan Liu. Finite sample analysis of the gtd policy evaluation algorithms in markov setting. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5510-5519 2017.
Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite time analysis of two time-scale actor critic methods. ArXiv, abs/2005.01350, 2020.
Yao Yao, Li Xiao, Zhicheng An, Wanpeng Zhang, and Dijun Luo. Sample Efficient Reinforcement Learning via FodeleEnseeblle Exploration and Exploitation. arXiv:2107.01825 [es], July 2021. arXiv: 2107.01825
URL htt.hhtpp//arxivv.rrrrg wwwp://////aaapaaooooo.ooossae/a 210p: 018.511855
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 13::14129-11 14142, 2020.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning. In International Conference Artificial Intelligence and Statistics, pp. 4023. PMLR, March 2021a. URL ISSN: 2640-3498.
ngs. mll.
press/v1 1ttpss//v13///aaag2211 html
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6): 3586-3612, 2020a.
Michael Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2020b.
Wanpeng Zhang, Xi Xiao, Yao Yao, Mingzhe Chen, and Dijun Luo. MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning. arXiv:2108, 01295 [cs), August 2021b. URL arXiv: 2128.01255.
wtt.vvoorga/bbb//////b// 01295
13

---

Published as a conference paper at ICLR 2022
A RELATED WORK
Reinforcement learning (RL) is a powerful technique for solving the sequential decision making problems (Li) 2018 Sutton & Barto 1998). The sequential decision-making problem is always formulated as the Markov decision process (MDP) framework. The agent uses policy to determine its action according to the current state of the environment, interacts with the environment, and gets the immediate reward from the environment as feedback. Then, the environment changes to the next state. The goal for reinforcement learning is to find the optimal policy that can achieve the highest expectation of the summation of the future reward. Recent work on model-based RL (Nagabandi cct al. 2018 Luo et al.] 2018 Kurutach et al. 2018:) Wang et al.] 2019 Janner et al.) 2019 Pan et a 2020 has shown the power of first learning the environment model and then use do the policy optimization. These methods all achieve similar performance by using fewer data compared to their
model-free counterparts.
Model-free and Model-based Methods: Model-free approaches, as a general-purpose tool for learning complex policies (Mnih et al., 2015; Lillicrap al. 2016; Haarnoja et al.] 2018), has the problem of low-efficiency (Janner et al. 2019), which limits the application in real-world physical systems where data collection can be an arduous process. Model-based Reinforcement Learning attempts to reduce sample complexity while maintaining the asymptotic performance. The learned model can be viewed as a black-box simulator and then used for training model-free policy (Naga-
bandi et al. 2018 Luo et al.) 2018 Kurutach et al., 2018 Wang et al. [019 Janner et al. [2019] Pan ct al. 2020). Tools like ensembles (Kurutach et al. 2018), probabilistic models (Chua et al. 2018), and meta-learning (Clavera et al., (0118) are widely used in model-based RL. A major problem of model-based RL is model-bias (Deisenroth & Rasmussen 2011) caused by a compounding error effect of long-term predictions. To address this problem, popular approach is to use interpolation between different horizon predictions (Buckman et al. [019)] Janner et al.) 2019) and interpolating
between model and real data (Kalweit & Boedecker) 2017). Different from the traditional model based method, we argue to learn the model beyond the accurate prediction and to use the model more efficiently by calculating the policy gradient analytically through the learned model.
Improved Model-based Methods: Several issues have been studied to achieve better performance in model-based RL. Malik et al. (2019) explore which uncertainties are needed for model-based RL. Further research implements a masking mechanism based on the model's s uncertainty estimation to decide whether the model should be used or not (Pan al.] 2020). Dropout mechanisms are also applied to model-based RI. algorithms to improve the robustness while maintaining high sampling efficiency (Zhang et al. (022b). Efficient exploration also considered in model-based RL. (Curi et al. 2020: Song & Sun 2021 Yao et al. (021). For model learning. Ke et al.] (2018) use latent variables to carry future information that improves long-term predictions, Tomar et al. (202I) use model-invariance state abstraction to improve generalization. Kégl et al. (2021) firstly design met- rics to evaluate the various popular generative models when using them on the control problem.
Zhang et al. (2021a) shows that tuning of hyperparameters dynamically improves the performance compared to using static hyperparameters which are kept fix for the whole training. This operation can be seen as simple treatment of model uncertainty. Considering that we also need the predic- tion model at the model learning and using phase, we can leverage the useful ideas inside of these methods help us to build a robust prediction model and use efficiently.
model learning (DAML) (Lovatto et al. 2020 Nair al. 2020 Abachi et al., 202T, D'Oro et al. 2020 Farahmand et al., 2017 Modhe et al. 2021) considers the problem that how we can learn the model that can be better leveraged by the policy optimization or Q learning. Droro et al. (2020) using policy gradient to re-weight the model prediction error for each sample. Farahmand et al. (2017) use the value function to re-weight the model prediction error for each sample. Please note that, although the main goal about learning the model by considering the follow-up decision making problems seems very similar, our ideas and methods are totally different from DAML. DAML is still the traditional rollout-bace model-based method with re-weighted prediction loss.
Decision-aware Model Dearning:Decision--nee
Differentiable Model-based Method: A more effective way of using the model is to exploit its differentiability. SVG (Heess et al.) 2015) presents framework for learning continuous control policies using back prok-propggating through the model. However, SVG just uses real trajectories lever- aging likelihood ratio term which in turn increases the variance of the gradient estimate. Recently,
14

---

Published as a conference paper at ICLR 2022
Amos et al. (021)) extend SVG further leveraging the soft update (Haarnoja et al. 2018) and model- based value expansion (Feinberg et al.) 2018). Different from SVG, MAAC (Clavera et al., 2019) entirely relies on the predictions of the model, removing the need for likelihood ratio terms, and achieves better performance than MBPO leveraging model-based value expansion for updating pol- icy and Q-function. The plan horizon in these methods allows trade-off between the accuracy of the learned model and the accuracy of the learned Q-function. In addition to back- proppaaaggg through the path, MAGE (D' Oro & Jaškowski 2020) uses the differentiability of the learned model to compute gradient targets in temporal difference learning. Our method is different from these methods. Different from these methods, our main contribution is that we propose use two models to learn the model prediction and the model gradient respectively. By explicitly construct the gradi- ent estimator, our gradient model can produce more accurate policy gradient and thus benefits for the policy optimization.
Outside of RL: Several algorithms outside of RL. adopt gradient penalties into the loss function when training neural network, known as double backpropagation (Drucker & Le Cun 1992). Sobolev training (Czarnecki et al.] 2017) is most related to this topic, which tries to learn both the value and gradient of a target function during supervised training. However, Sobolev training requires the ground -trndhttrthhhh gradient available. In the model--ased reinforcement learning setting, we do not know the ground-truth gradient.
B ALGORITHM DETAILS
B.1 IMPLEMENTATION OF DDPPO
B.1.1 DIRECTIONAL. DERIVATIVE
The directional derivative of function with respect to a vector v point (e.g., position) may be denoted by ( that is a limit: V.f(x) = lirn--0 ((othw)_f(a) If the function f is differentiable at x then the directional derivative exists along any vector and we have: (=)
h
ff(x). f.- Note that if f is a scalar function and the dimension of x is high, then ff(x) is a vector and difficult to estimate, however (x) is scalar, and easy to estimate by finieeddifferenc. This intuition helps us to design the constraint on the gradient of the learned model using data samples.
B.1.2 BACK- PRCCPGPRPAG THROUGH GRADIENT MODEL
In our method, we use the prediction model to rollout, however use the gradient model to calculate the gradient when performing policy optimization. This procedure is like SVG (Heess et al. (015)) that gets predictions using the true environment however gets the gradient using a learned model. Note that reparameterization and important sampling weights are needed for this method. In paper, we use the learned model as a deterministic model that only uses the mean outputted as the prediction. Now we show an example of calculating the gradient. When predicting, we first receive a state s and use the policy to get action a, then using the prediction model to get the next state and reward s', r. Note that we need to memory the computation graph for getting the action. Then we use the gradient model to calculate the gradient by feeding the (s, a) to it and paoo-propngttion. Combined with the memorized computing graph we can get the final policy gradient.
B.1.3 IMPLEMENTATION
Our algorithm can be divided into three main parts: model learning, policy optimization, and Q- function learning. And our approach alternates between collecting samples from the environment, training the model, updating Q-function, and policy optimization. The overall algorithm pseudo- code is demonstrated in Algorithm
Firstly, we collect data using the current policy and add it into the environment replay buffer Deno. Then the prediction model is trained using data from Dene- The gradient model is also trained with the multi-objective loss. The third step is to perform & step model rollout on the prediction model ME with the current policy, where is increased over time which proposed by Janner et al. (2019) to achieve better performance. Those samples are appended to the replay buffer Dmodeet together with samples from Denrs used to update the Q-function. To update the policy, firstly, obtain
15

---

Published as a conference paper at ICLR 2022
Table 1: Hyperparameter settings for DDPPO results shown in Figure] T InvertedPendulum
Environment Name epochs
Hopper
Walker2D
HalfCheetah
Ant
Humanoid
15
100
100
100
150
150
environment steps /epoch
1000
ensemble size
7
G
/environment step
10
G2 Jenvironment step
T0
H
3
2
3
n
10
25
25
w
10
50
0.1
1.0
0.1
0.1
imaginary trajectories by unrolling the latest policy on the prediction model Mp, for H steps from randomly sampled state in Dene- Then update policy by calculating the gradient of the objective in equation 11 while using the gradient model M, instead of M
In our approach, apply the soft Q-update (Haarnoja et al. 2018) which minimizes the following objective:
= (Q(or. (r(st, az) + (8L+1 ax41) - a
2
(12)
The temperature hyperparameter is automatically tuned by adjusting the expected entropy over the visited states to match target value (Haarnoja et al. 2018) with the following objective:
Ja E a log Th (ar, st) a?].
(13)
where is the target entropy.
B.2 EXPERIMENT SETTINGS
B.2.1 ENVIRONMENT SETTINGS
We evaluate our approach on six continuous control benchmark tasks MuJoCo (Todorov et al., 2012) simulator in our experiments: InvertedPendulum- Hopper-v2, Walker2d-v2, HalfCheetah- v2, Ant-v2 and Humanoid-v2. Standard full-length versions of these tasks are used. Note that the Ant and Humanoid environments truncated observations which is consistent with MBPO Janner et all 2019). The code for building Ant and Humanoid environment provided by Janner al. 2019
B.2.2
HYPERPARAMETEE SETTINGS
Hopper
Table D shows the hyperparameters used for DDPPO to
achieve results shown in Figure ! G1 is the number of up-
3500
dates for policy per environment step, G2 is the number of updates for critic per environment step. is the rollout length for updating policy. is the number of directional derivatives used to construct the gradient loss. w is the coefficient in front of the gradient loss when updating the gradient model. Note that H is used to trade-off' between the accuracy of the learned model and the accuracy of the learned Q-function. As shown in the right figure. the results by setting small H is close the results of MBPO, while larger H may cause worse results. We try different steps and propose 2 for Hopper and for other environments. We employ 5000 (200 for the vertedPendulum environments) warmup steps of interaction with the environment before starting to update the actor and the critie. The network structure of both the prediction model and the gradient model is the same
3000
2500
2000
1500
ours(H=1)
1000
ours(H=2)
500
ours(H=3)
MBPO
0
25k
50k
75k
100k
Steps
16

---

Published as a conference paper at ICLR 2022
MBPO. Both models are trained after every 250 steps of interaction with the environment. Every time after model training. we perform rollout for k steps and add the collected data to the buffer used for updating the critic. k is increased with the number of interactions. For InvertedPendullm, Ant, HalfCheetah, and Walker2D & keeps fixed. For Hopper k increases from 1 to 15 between epoch 25 epoch 155. For Humanoid & increases from to 25 between epoch 25 to epoch 155. The policy and Q-value function are updated 10 times per environment step which is less than the number of MBPO. For updating the policy we rollout on the prediction model for I/ steps with batch size 256.
B.2.3 EXPERIMENT DETAILS FOR BASELINES
For MBPO, directly the reported number given by Janner al. 2019 For SAC we the codes and hyperparameters available For MAGE, we use the codes available" and hyperparameters from the author. For MAAC, the model structure and hyperparameters such learning rate the same as MBPO, and the extra hyperparameter, H, we try different steps and set 2 for hopper and 3 for others which is consistent with ours. For other baselines, we use the codes or results from Wang et al. (2019 5 Note that the number of layers and the number of nodes of Q value functions are all the same for SAC, MBPO, MAAC, and ours. The number of layers and the number of nodes of the learned model are all the same for MBPO, MAAC, and ours. Finally, we keep unimportant hyper- parameters of our method as consistent as possible to SAC or MBPO to keep it as fair as possible.
B.3 MORE RESULTS FOR ABLATIONS
The results in Figure 3 show the importance of considering the gradient error when training the model if paok-propagating through it and highlight the designed two-model-based learning proce- dure. We extend the experiments in Section 5.3 to more environments. The results in [ggure/]shoon the performance of our method compared to the variants of our method. The results in Figure show that the prediction models in our algorithm have lower prediction error than the gradient mod- els, which confirm our conjecture that the prediction and gradient calculation should be done by the corresponding model.
InvertedPendullum
Hopper
Walker2d
4000
Return
Average Return
3000
Average Return
3000
2000
2000
400
200
1000
1000
10k
15k
25k
50k
75
100k
%
25k
75k
100k
Staps HalfCheetah
Steps Humanoid
Ant
Aver age Refurn
10000
Average Deturn
6000
Average Beturn
5000
8000
4000
6000
4000
3000
2000
2000
2000
1000
25k
50k Staps
75k
100k
0
25k
S0k
75k
100k 125k 150k
%
25k
50k
125k 150k Steps
75k
100k
Stops
ours
without gradient loss
single model
Figure 5: The impact of adding the gradient loss and the importance of two-model-base learning.
17

---

Published as a conference paper at ICLR 2022
InvertedPendulum
Hopper
Walker2d
Prodictive error on validation
0.0006
Sredicivee errer on validaton
Predictive on vallidatio
1.0
0.0005
0.025
0.8
0.0004
0.020
0.6
0.015
0.010
0.4
0.0001
0.005
0.2
0.0000
0.000
5000
10000
15000
20000
25000
50000
75000
100000
25000
50000
75000
Env Steps
Env Steps
Env Stops
HalfCheetah
Ant
Humanoid
Predi Privictivv ctive error validation
1.0
Predi ctive errer on validation
0.200
Predi ctive arrer on valldation
0.5
0.175
0.150
0.6
0.125
0.3
0.4
0.100
0.2
0.2
0.075
08000
25000
50000 75000
100000
15000
20000
25000
50000
100000
150000
Env Steps
Enw Steps
Env Steps
gradient madel
prediction model
Figure 6: Prediction error of the prediction model and the gradient model on validation during the training of the agent. Directly adding the gradient loss item to the model training loss may influence the learning for prediction. This may be the reason why the performance of our variants (single model) is not even as good as using the model without consideration of the gradient loss for some environments, such as Hopper, HalfCheetah, and Humanoid.
InvertedPendulum
Hopper
Walker2d
4000
1000
Average Return
Average Return
3000
Average Return
800
3000
600
2000
2000
400
200
5K
10k
15k
0
25k
50k
75k
100k
%
25k
50k
75k
100k
Steps HalfCheetah
Steps
Steps
Ant
Humanoid
12000
6000
5000
Average Return
Return
Return
8000
4000
6000
4000
3000
4000
2000
2000
1000
50k
75k
100k
%
25k
50k
75k
100k
125k 150k
0
25k
50k
75k
100k
125k
150k
Steps
Staps
Steps
ofrs(# -2M)
baseline param -1M)
baseline -2M)
Figure 7: The higher performances than the variant with similar number of parameters support that the improvement our method is not actually from the additional capacity.
C PROOFS OF THE THEOREM
In this section, we present the proof of the main theorems in section 3.
18

---

Published as a conference paper at ICLR 2022
First of all, summarize the necessary assumptions here. These assumptions commonly used for analyzing reinforcement learning algorithms.
Assumption (Stationary).
For vt1
< tz and she sh denote = t2 t1 there exist L1(T) < oo,
l2 t1. such that E
12
S LI(T). And Li(T) can be uniformly upper bounded is the 2-norm of the matrix of which the element ij is:
by L1 < co. Here, |E
+ (6) 10-00,,,16 52, (1) 2,
where st(i) is the i-th element in the vector
Assumption 3 (Bounded region). There exist a finite D < oc such that: (Wr et al. 2020
Wang
et al. 2017
1ss - s2ll DD for V81, 82 E.S
(14)
Assumption (Lipschitz). The transition model and the reward model are both Lipschitz continu- ous with Lipschitz constant I.g and Tor respectively (Clavera et al. 2019). addition, we need the derivative of the transition model and the reward model to be Lipschitz continuous with Lipschitz constant Lifa and Lyv.
Assumption (Uniform ergodicity). For a fixed policy no Te(-)s)), denote μi as the stationary dis-
tribution induced by the policy and the transition probability measure P o.). Then there exists
m < oc, Cu < co, and p E (0, 1) such that: Wu et al. 2020- Zhang et al. 2020a Qiu et al. 2027)
drv (P(s+++ E -|s -1) s), S mp Vr 2 0, Vs E S.
(15)
Assumption (Lipschitz policy). Let (a)a)s) be policy parameterized by 8. there exist constants such that for all given state s and action a we have: (Wu et al.) 22020) Clavera et al. 2019 Oro & Jaskowski 2020
du dTT((als) dø
S ToT.
(16)
Assumption (Step-size). The step-size 04 is non-increasing and non-negative (Wang et al. Wi al. 2020).
2017
Assumption 8 (Bounded reward). The reward is bounded in the region [0, mmael Clavera et al.
2019 Wu et al. 2020).
Here, we show some lemmas that is important for the proof.
1-1441
Lemma 1. Suppose assumptions hold, denote the Co = min(
T-2, --2,) for we have
1 - se S Coes
(17)
Lemma 2. Denote C₁ =LL, O2 LC L7 We have
dst
dst
dst
C f Caes
8
E
E
(18)
dso
dsp
dso
Lemma 3. If constent 0 < < 1, for arbitrary constant C, we have:
Σ - ?
(1 (1
(19)
L and C4 = I.I.yy Collla + CoL.o then we (1-7)3,03
Lemma 4. Denote constants C3 = I-
(1-7)3
have:
dR
d
dRM
E
< Cze + +C4e + Cses
(20)
ds
ds
ds
19

---

Published as a conference paper at ICLR 2022
Lemma 5. Considering Markov process M with stationary distribution p. Data & sampled from the M. There are two functions of parameter o: G( Et) and g( where
g( G E)-
(21)
Denote the upper bound for the function G as L1. Denote the Lipschitz constant for the function G as L2
Φ = de -a,G((be.E)
(22)
For all 0 < T < oo, we can proef that:
T
Σ an (o(())) G(rr, Ex) - s(dr)) 1=1
S 2To0l
Σ TTT +
Σ
TrLpmpp
(23)
t=r+1
t=1
Lemma Denote J as the upper bound of the expected sum of discounted rewards J. Then
U3 <
Tman
Lemma 7. There exist two constants L, and L3 such that for all 01, O2 we have
[| J [(b)] Ls || 2221
(24)
dJ dJ
S 13 11 - Φ221
(25)
dé depu
Lemma 8. There exist constants LR such that for all Φ1, 822 we have
B|[R(11) - R(Φ)))l S Lx 1l 22ll
(26)
(27)
Lemma 9, Suppose the function is smooth with smoothness constant L²j
dJ
T.y 11 Φ2 |l
J( ž J(ő1) +
. ②n)
(28)
dōi
Lemma 10. yf and VI S v + 22. then we have
x S 2y 22
(29)
C.I PROOF OF THEOREM
Overall speaking, the main idea behind the proof of Theorem [ can be summarized as follows.
Considering that to calculate a gradient value of a function output w.r.t its input, we need to specify the function and the input value. Therefore, as in the Equation 3 both the function( and f) to be taking gradient and the input value 5; are different from the ground-truth. So we need to first bound the error of the prediction state(Lemma I and then bound the error of the gradient of the learned model true state (Lemma
Firstly, we define some notations, denote Co = LyLnt,,CsLay, Cy = -L,,CCtthnLe Cs Cob.o CoL.e Lh L, b, Cor now we can prove the theorem by using previous
1-7
I-
lemmas:
dRM
dRM do
(30)
06
dr.
dri
0
dR
dō
dō,
dR ds441
da; dØ
+ Σ
x'
(31)
dø
de
d3r41 dā,
dó
dst+1 da,
1=0
1=0
Cuen + +c7) + +
7
8
7'
dR
+ d#
dR 8
ds
L
(32)
ds
dāg
da
t=0
< Cocy + +G) + 1 1-
1
?
L (Cac
Cady Cscs) + (1 7)
?
(T Cocf + +c))
Ln Lo (L
(33)
=Coef + OTef + +Ceg.
(34)
20

---

Published as a conference paper at ICLR 2022
The first equation is the definition. The second inequality follows the assumption 2 the right term.. The third term follows the lemma
This theorem is fit for the case that the Lipschitz constant Ly 1. For the dynamics that the gradient is always larger than 1, it is unrealistic and uncontrollable. For the dynamics that the gradient is always less than theorem and the assumptions are enough. For the last case, it needs a stronger assumption directly on || szll-
Remark:
C.2 PROOF OF THE THEOREM
First of all, we briefly summarize the proof idea here. We first decompose the difference of objective function brought by parameter updating into two terms according to its smooth assunpiion(Euuttion The first term an inner product term. The inter product term can be further decompose into
35
two terms(Equation (36). One is about the model error and the other one is about the non-i.id. data. We can carefully bound each terms and then combine them all together to proof the theorem. Readers can refer to the related papers Wang et al. (2017) Wu et al. (2020) Ghadimi & Lan (2013)
to get more insight of the proof.
According to the lemma
9 have:
dJ dR déi" dØ
LJ 2
((b41) J(oi) at
1
(35)
where dn is the estimated policy gradient calculated using learned model and used to update the 2o
policy parameter from to @c+1.
Denote hi
dR
AR de
hi
JR
We can further decompose the value improvement as
((d0. mt)
(
dJ
12 |1
J(b+1)) J(dr) + az
(36)
doi
According to the
inequality we have
dJ
E
2
E
E 11n
2
E
(Coe + Crey + Cscg).
(37)
dø
do:
dØ.
Rearrange the term have
dJ de
dJ
dR
dJ
dJ
E
E
(38)
do do,
do
dep
d
Note that the first term in the right hand side is 0 if the data is LLD. sampled from the stationary distribution. If the data is sampled from the markov process, the data non-i.idd. we can further bound the first term using the Lemma
dJ dRM dJ
T
Σ
2
2TL
a
1 + Σ
ortumm=)
(39)
doi
do, dø:
The second term is
dJ
d.
dJ
E
=
B
(40)
do:
der
Noticing that
dR
Φ1+1 - 6:
a2
(41)
dé,
21

---

Published as a conference paper at ICLR 2022
Combine Equations 37
39
40
41
get
a a4 B1 S ((0+11) - - +a, + Bs.
(42)
where B1
dJ ddi
?
B2 2TL + +E Ei=1 ot
[[3I9T L.tRR*
+ Et omp' B3
2
Coef + C7EY - Cgey, B1 = (+dd) dipe"
AR
dd dbi
Bs = Lzaî
Notice B1, Br, Bs are functions
dip
of I.
Taking summarizing on both side, we have
T
Σ a Σ
BBBs/BB
s ((0T+1)) J(do) + Σ a Bs,
(43)
1== a
Σ7 1=
a₁ B
2U, +
B2
(44)
2t=1 a1
2t=
1=
We have
Di= Bs)
2 + Bg
B3
+
(45)
Σt=l a4
Σt= at
Therefore,
Bs 5
28, + Bg
B
min (VB. B1
+
(46)
ISKTT
Σ 04
4
According to the lemma
10 We prove that
4U, + +B2
min S
B1
B3.
(47)
Σ
C.3 PROOF OF THE USEFUL LEMMAS
Proof of Lemmal 7 According to the transition model definition and its Lipschitz continuous sumption, on the one hand, since the states are bounded in the range with diameter D, we can see that || 55.-sp sell < D. On the other hand.
11(+1 L
|| -slll S scy + Lg || s2-1l1
Ex.
(48)
Proof of the Lemma
dzc
døi
dsi
ds
1
E
(49)
d
dšu
ds;_1
dso
dā;
ds;
d3,
E
II
II
(50)
iml
ds
d3,
ds;-
1
dā
daj
de;
ds;
dsc
dsi
)
(()
døg
<B
11
+
ds3-
dz;-1
dsi-1
ds;-1
ds;-1
dā.-1
(51)
Li(e/ Coey
(52)
StLfeg
(53)
22

---

Published as a conference paper at ICLR 2022
Proof of the Lemma Et= (1 1)
Let = E ty'C, then can get that ys Ei=o 2i=0tyttl t So,
(1 -)8 = Ennr' = -2.C. At last we can get
78
C-
Proof of the Lemma
dRM
dRM
) =
00
di
dē:
dr, dsc
E
E
E
(54)
ds
ds
dže 80
ds, 50
1=)
1=
8
dšg
( dr
dr
dsi
76
8
+
6
(55)
d$u
dsc
ds
dsp
[i(et + Lya Coer) +r (Cief + C2er))
(56)
1=0
(L1 + L,e + Lr
Σ 7 (Ciez + C2e1))
(57)
1=0
1 (Li(e) Coe))
1
(1- eg +
7
7
CoLysoLee
(58)
(1 (1-
-C3e Ceez + Cseg.
(59)
Proof of the Lemma
On (o(d)), G()r, g(bb))
(60)
Ore ((a(0)), G(dr, G(dr, + (g(()), ((orr g(())))
(61)
We first bound the first term in the right hand side.
au (1g(0)), G( Ex) -
(62)
T
at (g(0)) G(dr. +11 at (g(d)), G(dr,&++))
Σ T-
(63)
+1
Σ
(g(()), G( G(
(64)
(=++1
T
<2rag +1
Σ
at (g(d)), G(dr, - G(dett)))))
(65)
t=
T
1
<2rOO +
Σ
aill
(o(()), G(on, &(,b+
(66)
1m7+1
k=
H--1
< <2TOO +
Σ
LeLell
Σ
(g(()), ou - +1+1)1l
(67)
1=7+1
k=
T
< <TrOOLi+ +
Σ L,LLLTT.
(68)
1=7+1
(69)
23

---

Published as a conference paper at ICLR 2022
Now, we will bound the second term in equation in expectation.
E
Σ ae ((g())), ((t,,)++)))))))))))))+))) - s(oh))))
-
(70)
an ((r())), -(()))
(71)
arbill G(or, EEFFF - mlE)) dEll
(72)
aeLidTv(P(EHtr E u(F)),
(73)
asTamp'
(74)
Proof of the Lemma
20
J E
7 r:
Tmar
UJ.
(75)
1 -
Proof of the Lemma 7 The first inequality comes from Lemma B.4 in Wu et al. (2020). The second
inequality comes from Lemma 3.2 in Zhang et (2020a).
Proof of the Lemma
dRM
DC
dr ds:
20
dr dst
20
E
E
E
L --9
L, (76)
ds
dsc so
dec su
t=
t=0
t=0
dRM
dr
dR ds da;
B
E
+
(77)
dΦ
d
dsst1 da, dØ
1=0
1=0
1
L Lu +
E7
7'
dseti dar dō) )
dR
dsc+1 daz
< 1 1-9
by Lo
?
Li Lab (78)
1
(1 (1-
-LR-
(79)
Proof of the Lemma
The conclusion comes from the Equation 4.4.9 in Nesterov al. (2018).
Proof of the Lemma 10
Since vx and v + /2 are both positive, then have S (VD V2)²
i.c, I S y + + 2 yz note that 2/1zz S y + z, then we have x S 2y + 2z.
24


Reviews:
[{"review_id": "ojO-Rkw_GL7", "reviewer": "Reviewer_TLT1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the insight that method can use the model's predictions and gradients and thus we should be aware of model gradient errors. However in practice, we do not know the gradients on the trajectories that we attain and equation (12) requires computing the n-nearest points of $x$ in the replay buffer to estimate the model gradient. This seems potentially intractable in practice for large replay buffers, and seems like it will not always coincide with the true model gradient if the buffer is sparsely filled. Is the scalability here the reason why the methods were not run for more timesteps?\n\nThe experimental results in Figure 1 show the sample efficiency and show an improvement over the relevant baselines, using 5 trials. The most relevant baseline here is to MAAC, as Eq 14 is set up very similarly. I especially like the ablation in Figure 3(right)/Figure 4 that ablates against the model not using the separate gradients.\n\nOne slight concern I have here is that **some of the empirical gain may come from modifying the base model-based algorithm rather than adding the gradient information.** I see the value being optimized in Eq 14 as nearly identical to the one optimized in MAAC (S4.1 there), but some of the baseline results (without the gradient loss) in Figure 4 are better than the MAAC baseline in Figure 1 (especially the cheetah/humanoid).\n\nMy largest concern with the paper is that the end of Section 5.1 says that the results of ~5k reward attain \"near-optimal policies in the Humanoid environment\" (for running) despite papers such as the [SAC-SVG](https://arxiv.org/abs/2008.12775) attaining a reward of ~9-10k in the same environment. This paper does not provide a video of the learned policy, and furthermore, it is possible for the humanoid agent to remain stationary (i.e. standing still) the entire episode and attain a reward of 5k (because the [base alive bonus is 5 reward/timestep](https://github.com/openai/gym/blob/bb81e141ea7ae67ce339109095841592e8231185/gym/envs/mujoco/humanoid.py#L34) and the episode runs for a maximum length of 1000 timesteps.) **I think this is extremely misleading and should not be accepted for publication.** I request for the authors to clarify this point and provide a video of the agent learned on this task. I am very willing to further discuss this point and increase my score on this paper if this is properly addressed.", "summary_of_the_paper": "This paper considers learning a model for reinforcement learning and proposes to also learn the gradients of the model in addition to just the predictions. They approximate the model gradient using the nearest samples from the replay buffer in (12) to learn the model gradient (13), which they use to replace the model gradients in the value expansion in (14). They evaluate on MuJoCo environments (Fig 1) and show some key ablations (Fig 3) on the number of samples used for the gradient estimates, and the weight term in (12), and combined/separate models.", "summary_of_the_review": "This is an insightful paper on modeling the gradients that contains an extensive empirical evaluation. My main concern is in the presentation of the humanoid results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jhgqV0bAemo", "reviewer": "Reviewer_nkhB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am recommending to accept this paper because its algorithm construction is well-reasoned, the theory appears to be novel and provides appropriate support for the proposed algorithm, and the empirical results provide some minor support for the claims. Overall, I find that this paper contributes an important perspective on the role of using differentiable models, and the need to control the error of first-order information (gradient model) instead of only zeroth-order information (prediction model) when using differentiable models.\n\nI remain on the fence, however, because (W1) the empirical results provide only marginal support for the proposed algorithm and (W2) there is a disconnect between the paper's stated goals and the novelty the paper _actually_ provides, leading to neither being done particularly well.\n\n---\n\n**Details for W1:**\n\n While the proposed algorithm is the only algorithm to consistently perform well across all 6 benchmark domains, the paper does not provide sufficient information about the baselines to understand if this experiment is fair. A couple of immediate concerns come to mind:\n\n1. How are the hyperparameters selected for the baseline algorithms?\n2. Because some baselines are model-based, some are model-free, and many baselines use a different number of models, is the comparison between baselines fair in terms of compute usage and number of free parameters? The proposed algorithm uses an additional model to encode the world model gradient, suggesting that it has more learnable parameters than any other baseline. Do these extra parameters account for the performance difference alone, or is the performance difference actually due to decreased bias in the policy gradient?\n\nThe use of only 5 random seeds is also concerning, especially considering the very high variance exhibited in many of the reported results. Are any of these results statistically significant? Do any statistical significance tests allow for the use of 5 samples? Does this imply that the reported results could simply be due to random chance?\n\n**Details for W2:**\n\nLargely due to the empirical concerns stated above, it seems very challenging to support the claim that the proposed algorithm achieves state-of-the-art results. Consider additionally that only a small subset of the standard Mujoco domains are used significantly limits the scope of the SOTA claims that can be made. Proposing a novel algorithm which uses another NN-worth of free parameters over baselines, then claiming SOTA does beg the question whether SOTA was achieved only through more parameters.\n\nHowever, I think the real value provided by this paper is further insight into the need to control the gradient error when using differentiable models. The paper performs two interesting ablation studies, one showing that the proposed algorithm does successfully control this error term, and the other showing the impact of controlling this error on the overall performance. However, neither of these ablations are complete; especially the second. \n\nFigure 3 shows that adding the gradient loss term to control this bias has negligible impact on the performance of the proposed algorithm across all three manipulations (number of directional derivatives sampled, different weightings for the added loss term, and different methods of controlling the gradient error). Considering further that two of the six domains were cherry-picked for reporting in the main body of the paper, suggests a post-hoc maximization bias occurred while selecting these results as well. In order to well-support the theory proposed in this paper, I believe the most important set of experiments is actually the ablations. These should be significantly expanded and further insights should be drawn about the role of controlling the gradient error term. In order to make room for this expansion, the SOTA experiments can be largely treated as a demonstration that the proposed method scales well and is competitive, with considerably less focus placed on Figure 1.", "summary_of_the_paper": "This paper seeks to improve performance of model-based policy optimization methods by taking advantage of the world-model's differentiability. It starts by showing theoretically that error in estimating the model's gradient contributes to bias in the learned policy. The paper then proposes directly optimizing the jacobian of the world model using a sample-based strategy in order to directly control this bias term.", "summary_of_the_review": "**Edit during discussion phase:**\n\nI have increased my overall score from 6 -> 8 based on the ongoing discussion. I have also increased my \"empirical novelty\" score from 2 -> 3 because I believe the new ablations and clarifications yield deeper insight into the impact of using first-order information to constrain transition models in RL.\n\n---\n\n(Copied from above Main Review)\n\nI am recommending to accept this paper because its algorithm construction is well-reasoned, the theory appears to be novel and provides appropriate support for the proposed algorithm, and the empirical results provide some minor support for the claims. Overall, I find that this paper contributes an important perspective on the role of using differentiable models, and the need to control the error of first-order information (gradient model) instead of only zeroth-order information (prediction model) when using differentiable models.\n\nI remain on the fence, however, because (W1) the empirical results provide only marginal support for the proposed algorithm and (W2) there is a disconnect between the paper's stated goals and the novelty the paper _actually_ provides, leading to neither being done particularly well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cX5g0QzFf2O", "reviewer": "Reviewer_8zAV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## UPDATE DURING REBUTTAL\n\nTo AC & reviewers & authors - I think the authors did make substantial improvements to the paper and it is now easier to understand. Given the prevalence of this problem in MBRL I do think this is a contribution to help the field understand how best to learn dynamics models for control. Though, the empirical questions make my recommendation very borderline. My idea is that it is closer to a weak accept now, but it will be hard to fight for this paper's acceptance without a little more understanding of why it works.\n\n## General Comments\n- **technical merit**: the core idea the authors to improve model-based reinforcement learning are well motivated. The proposed solution is interesting and novel. Though, they seem to not be citing a core paper motivating their work: Lambert, Nathan, et al. \"Objective Mismatch in Model-based Reinforcement Learning.\" Learning for Dynamics and Control. PMLR, 2020. \n- **experimental quality**: the numerical results are impressive, but I don't see enough explanation or intuitions for them. Especially when continuing to browse the ablations in the appendix, I think the experiments need to be presented with more understanding for the results to be reproduced.\n- **writing quality**: the writing of this paper could be substantially improved. The writing has many typos and the structure is not very clear.\n- In a couple reads I never understood the part on the directional derivative as a scalar. Can the authors help me understand this? I acknowledge this may be common background that I am missing, so it may be a useful addition to the appendix.\n- If combining the proofs with DDPPO is too difficult, maybe this should be two separate papers.\n\n## Comments By Section(s)\n\n### 1 Introduction\n- This section was a bit too broad, which made it hard to get the point of the paper across. The author's do not need to describe everything about RL here or the history of MBRL (related works do that, and end up being repetitive)\n- This sentence is crucial, but both worded weirdly and not emphasized enough in the section \"Note that there is a big difference between these methods ... when calculating the policy gradient in these methods\"\n- The paper could do better at clarifying the contributions of the mathematics and the algorithm. The authors discuss a method and then a new algorithm. Maybe this can be combined into one point, but it is hard to follow and see where the authors are going. The contributions help this, but by that point there are more questions than answers.\n- the second half is very repetitive. Trimming this will make the paper easier to follow and add space to other sections that need it. \n\n### 2 Preliminaries\n- The paragraph introducing equation 4 is confusing. If it is a \"preliminary\", which paper should I look at to know more? Please cite that so I can learn it / follow the paper.\n\n### 3 Related Works\n- This paper could be good to include in the related works. Kégl, Balázs, Gabriel Hurtado, and Albert Thomas. \"Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?.\" ICLR 2021.\n- the formatting here is really forced and deteriorates the reading experience.\n- The related works seem dismissive of the past work. It is clear this paper is different, but why are past works important?\n\n### 4 Model-based Policy Optimization by Considering the Model Gradient Error\n\n#### 4.1 CONVERGENCE RATE FOR MODEL-BASED POLICY OPTIMIZATION\n- in the intro, this sentence makes me think there will be a general explanation of how this can be used for algorithms, but it seems to be lacking \"for the policy optimization algorithms in what the policy gradient is calculated using the learned environment model\"\n- This section needs substantially more explanation to support the math. Some assumptions can be moved to the appendix unless it is explained in the text why they are crucial and new. \n- Are there related works that hint to these formulations? They are big step and hard for me to follow. Knowing this, the authors should spend more time making them \"land\". \n- I think there is a typo in the remark after theorem 2 -- the learning rate alpha_t should be 1/sqrt(T) rather than sqrt(t)?\n\n#### 4.2 TWO-MODEL-BASED LEARNING POLICY OPTIMIZATION\n- very repetitive. Can remove some here or in intro.\n\n#### 4.3 DIRECTIONAL DERIVATIVE PROJECTION POLICY OPTIMIZATION: A PRACTICAL IMPLEMENTATION\n- I believe fairly strongly the algorithm should be in the main body of the paper, as it is sufficiently different from past. work. \n- Otherwise, a lot of this section is again repetitive. Please separate the repeated parts of the paper to where they should be. Intro is for framing, preliminaries for background / common math, and the new stuff in your method :). For example, training a model with MSE / MLE is described multiple times and can be done with only a citation.\n- this sentence is interesting and potentially crucial to the result \"So in this paper, we sample the nearest n data points of x to calculate n directional derivatives in n different directions and use the finite difference to co\". It would be interesting to hear more intuition on why this works.\n- Figure 1 says \"trials\" - how many each, and how many for other algorithms?\n- A system diagram could make it clearer how the algorithm works.\n\n### 5 Experiments\n- Figure 2 seems to be in too simple of environments to validate the claims totally. The strong performance of humanoid is a big piece of the results, how do these intuitions translate to higher dimensional tasks?\n- What happens if the gradient model is used for state rollouts? This could be an interesting way to address the objective mismatch issue.\n- which implementation is used for the compared algorithms? \n\n- Why were Hopper and Ant chosen for Figure 3? Were these sweeps tried on other environments? Are the results consistent. At least commenting on these lines and or why the trends would break down are important to people considering using the method.\n- Hard to tell if the two model part is needed by the results? The impressive humanoid results are diluted by me when considering it's not the key variables the author's propose contributing to success. Why is there little gain from the gradient-trained model when adding the accuracy model? The labels on figure 4/5 could be more precise. One model is not the most clear.\n\n----\n\n## Grammatical comments / nits\n- abstract \"We\" -> we\n- intro weird phrasings \"decision-making problem is always formulated as\", \"agent uses policy to determine\" -> determines, \"the rollout-based model-based methods that the model gradient is crucial\" -> where the model gradient is..., \"we can see that the transition and reward model gradient matter in\"\n- intro confusing \"Therefore, we propose to separate different requirements for the model into different models\"\n- \"w.r.t\" missing period before eqn. 3\n- \"the expectation is taking over\" -> \"taken over\" after eqn. 3\n- related works \"policy gradient to re-weighted... re-weighted\" weird, also missing period at the end of this sentence ... \"sample Use the learned\"\n- section 4 \"that using the policy gradient\" that are using?\n- should use latex references for section names, referred to \"Preliminary section\" many times with different capitalization. \n- wrong \" in 4.1 \"tilde\" notation, \"hat\"\n- theorem 2 remark: \"gradient method converge to its...\" -> converges \n- in 4.2 \"between collecting samples from the environment, training the model, and optimize the policy\" --> optimizing\n- in 4.3 \"Hence in this paper, we use the error between the estimated directional derivative and the projection value to constraint the learned model’s gradient\" --> constrain\n- \"Model Usage\" section: \"we use the objective function that constructed by following\" --> is constructed, \" , \" space before comma later\n\n----\n## Other comments / ideas (not included in scoring of paper)\n- in the appendix, the authors include hyper parameters. Do you all have an intuition for why the higher dimensional tasks need fewer neighbor points, H, to calculate the derivative? As the density drops, I thought more points may be needed?\n- There is a really interesting correlation in figures 4 and 5 in the appendix that is not explained, but could be a great intuition to support the paper. The environments where the one-model approach fails to perform as well seem like they could have a bigger difference in the model errors? I am looking at the Hopper env in figure 4 and 5. Though, the ant does not back this up as much.\n", "summary_of_the_paper": "The authors present a new approach for model-based reinforcement learning by training two models and using them separately. One model prioritizes accurate transitions and another has a weighting term for value-gradient, which they combine in a new algorithm directional derivative projection policy optimization.\n", "summary_of_the_review": "The paper proposes a novel solution to a well known problem in model-based RL. Though, the paper is not well written so it is hard to follow, even for an expert in the area. In its current state, I do not recommend acceptance because of its writing / presentation / lack of conveyed intuition, but these are problems that can be fixed in a moderate revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MEKP33GblxP", "reviewer": "Reviewer_9ZXt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The method presented is quite original and well-positioned among existing methods for model learning.\n\n**Significance:** The problem in object is of interest for the RL community.\n\n**Rigour:** The theoretical results are not of particular benefit for the practical algorithm. The theoretical results and experimental protocols seem sound, although the core change proposed by the method does not provide very large improvements over baselines.\n\n**Strengths**\n\n- Directly learning the Jacobians of the dynamics is a very appealing goal, given the resurgence in popularity of value gradient methods (i.e., methods which compute the policy gradient by direct differentiation through a learned model).\n- The presented approach is well-positioned in the literature as a *decision-aware* approach for model learning.\n- The theoretical results presented in Section 4, despite not being novel in essence, are quite interesting.\n- Both the proposed loss function for model learning and the idea of having two models for trajectory generation and gradient computation are, as far as I know, new and simple to implement.\n\n**Major Concerns**\n\n- The loss function in Equation 12 requires to select the nearest neighbours of the current data point. With the usual implementation for the replay buffer, this adds a significant computational burden to the already expensive model-based policy optimization algorithm. Is the method directly facing this issue? What is the actual computational complexity of the algorithm?\n- It seems, especially from Figure 4 in Appendix, that a large part of the additional performance of the proposed method comes from the two-models component and not from the gradient learning one. Moreover, there is the risk that much of the improvement provided by the use of two models actually comes from the additional capacity. A good experiment should control for the number of parameters to check whether the performance boost only comes from the additional parameters, and not the particular architecture in use. Thus, I would have expected to see an experiment in which the number of parameters used by the two models in total is the same as the one used by the baseline with a model alone.\n- To double check more robustly whether the proposed objective for gradient estimation is effective per se, it would be extremely beneficial to have a simple, independent, experiment, in a supervised learning problem.\n- The mechanism used for backpropagating through a model after generating a trajectory from another one is unclear. To the best of my knowledge, the only technique for achieving something similar while using stochastic models is the one used in SVG (Heess et al., 2015) to differentiate through trajectories of real experience via reparameterization. This should be clearly explained in the paper.\n- In Figure 3, a more appropriate baseline to show should be MAAC, which seems to be the core algorithm upon which the approach is based, given its backpropagation through the learned model.\n\n**Minor Concerns**\n\n- I believe it would be more clear to state the assumptions before stating the two theoretical results, e.g., in the background section.\n- It could be beneficial to explain the loss function in Equation 12 in terms of a Taylor approximation: the learned model should have a gradient such that its first-order approximation is consistent with the given data. Is this a valid interpretation?\n- Section 4 should contain an explanation of why the gradient loss in Equation 12 alone would be insufficient for training a model. Are both of them used to avoid degenerate solutions only, or is there a deeper motivation?", "summary_of_the_paper": "Many recent model-based reinforcement learning algorithms exploit the differentiability of the learned model by computing gradients through it. Nonetheless, the standard approach for model learning is to minimize a loss on the prediction of the next environment state which, when the underlying dynamics is complex, does not guarantee that also the resulting gradient information will be accurate. The paper proposes a method for model learning which explicitly tries to estimate the Jacobians of the underlying dynamics, by leveraging a loss function on the directional derivative and a nearest neighbour approximation. Additionally, the paper proposes to use a model learned with a combination of this new loss and a prediction loss for backpropagation only, while training a traditional model for the generation of trajectories.", "summary_of_the_review": "There are some issues in the experimental results and in the justification of some moving parts of the algorithm, but the paper proposes a simple and seemingly sound solution for a very important problem in model-based reinforcement learning. I thus lean towards recommending to accept it.\n\n---\nThe authors addressed my main concerns. Thus, I now recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers model-based RL, and focuses on approaches that benefit from the differentiability of the model in order to compute the policy gradient. It theoretically shows that the error in the gradient of the model w.r.t. its input appears in an upper bound of the error in the policy gradient computing using the learned model. Motivated by this, it suggests a MBRL approach that learns two models, one of them minimizes the next-state prediction error (as commonly done) and the other minimizes a combination of prediction error and the gradient error. 
The paper empirically studies the method through extensive experiments.

Reviewers are generally positive about this work. They believe that the paper is insightful and the method is original. At first, there were some important concerns raised by the reviewers, but the authors revised their paper in the discussion period, and it appears that the reviewers are all satisfied now. I also read the paper during the rebuttal phase, and I should say that I have some concerns myself, especially on the theory part of the paper. Given that the authors did not have an opportunity to answer my questions, I do not put much weight on my concerns (and I believe most of them can be addressed with some clarifications). Considering the positive response of reviewers and promising results, I am going to recommend **acceptance** of this paper.

I strongly encourage the authors to consider the comments by reviewers, as well as the following ones, in the revision of their paper.


**Comments**

1) The true dynamics $f$ is defined as a stochastic one, i.e., $s_{t+1} = f(s_t, a_t, \epsilon_t)$ (just before Eq. 1), and similarly for the learned model. Here $\epsilon_t$ is the noise causing the stochasticity of the model. But later, when the errors on the model and its gradient are introduced (i.e., $\epsilon_f$ and $\epsilon_f^g$), the role of stochasticity becomes unclear.
For example, we have
$\|| \tilde{f}(s,a) - f(s,a) \||  \leq \epsilon_f$.

What happened to the noise term?

The same is true for Eq. (5). The next-state s' (either according to the true dynamics or the learned model) is random. In that case, it is not obvious how to interpret Eq. (5). Is it the error of the expected gradient of the next state? Or is it something else?

In case the dynamics is assumed to be deterministic, this should be clarified early in the paper.

2) The upper bound in Theorem 1 might be vacuous if the Lipschitz constant $L_f$ of the model is larger than 1.
To see this, consider Lemma 1. The constant $C_0$ is $\min [D/\epsilon_f, (1-L_f^{t+1})/(1 - L_f)]$.
If $L_f$ is larger than 1, for large enough t, the term $(1-L_f^{t+1})/(1 - L_f)$ blows up and $C_0$ becomes $D/\epsilon_f$. Therefore, the upper bound of Lemma 1 becomes $D$. Here $D$ is the diameter of the state space, which is assumed to be bounded.

This carries to in the next lemmas. In Lemma 4, $C_5$ would be of the same order as $C_0$ (multiplied by an extra $L_1 L_f / (1 - \gamma) )$, so the upper bound of this lemma becomes proportional to D too.

The $C_0$'s appearance continues in the proof of Theorem 1, in which $C_8$ is proportional to $C_0$ and $C_5$. So, $C_8$ is also become proportional to $D/\epsilon_f$. When we have $C_8 \epsilon_f$ in Eq. (34), we get a constant term $D$.
A similar dependence appears in the proof of Theorem 2, where B_3 is proportional to $C_8 \epsilon_f$, which can be as large as $D$. And in Eq. (47), we have $B_3^2$. So the upper bound in Eq. (47), which seems to the be upper bound of Theorem 2, is proportional to $D^2$. This means that if $L_f$ is larger than one, the upper bound does not go to zero, no matter how small the model error $\epsilon_f$ is (unless it is actually zero). This makes the bound meaningless.

This might be unavoidable. I am not sure about it at the moment. But it definitely requires a discussion.

3) Assumption 2 has a term in the form of $E[\frac{s_{t_2}}{ s_{t_1}} ]$ (I have simplified the form). The states $s_{t_2}$ and $s_{t_1}$ are vectors in general. How is the division defined here?

4) Please improve the clarify of the proofs. For example, in Lemma 2 it seems that a negative sign is missing in Eq. (49). Also how do we get Eq. (50) and Eq. (52)? (I couldn't easily verify them).

5) I believe the "periodicity property" used in Assumption 1 should be "ergodicity property".

6) The paper still has a lot of typos, e.g., "To optimize the objective, One can ..." (P3), "argument data" (instead of augmented) (p4), "Superpose" (p5), "funcrion" (p6).
Decision: Accept (Poster)