Making Better Decision by Directly Planning in Continuous Control
Authors: Jinhua Zhu
Yue Wang
Lijun Wu
Tao Qin
Wengang Zhou
Tie-Yan Liu
Houqiang Li
Venue: ICLR 2023 poster (2023)
Keywords: Model-based Reinforcement Learning
Reinforcement Learning
Planning
Policy Optimization

Abstract: By properly utilizing the learned environment model, model-based reinforcement learning methods can improve the sample efficiency for decision-making problems. Beyond using the learned environment model to train a policy, the success of MCTS-based methods shows that directly incorporating the learned environment model as a planner to make decisions might be more effective. However, when action space is of high dimension and continuous, directly planning according to the learned model is costly and non-trivial. Because of two challenges: (1) the infinite number of candidate actions and (2) the temporal dependency between actions in different timesteps. To address these challenges, inspired by Differential Dynamic Programming (DDP) in optimal control theory, we design a novel Policy Optimization with Model Planning (POMP) algorithm, which incorporates a carefully designed Deep Differential Dynamic Programming (D3P) planner into the model-based RL framework. In D3P planner, (1) to effectively plan in the continuous action space, we construct a locally quadratic programming problem that uses a gradient-based optimization process to replace search. (2) To take the temporal dependency of actions at different timesteps into account, we leverage the updated and latest actions of previous timesteps (i.e., step $1, \cdots, h-1$) to update the action of the current step (i.e., step $h$), instead of updating all actions simultaneously. We theoretically prove the convergence rate for our D3P planner and analyze the effect of the feedback term. In practice, to effectively apply the neural network based D3P planner in reinforcement learning, we leverage the policy network to initialize the action sequence and keep the action update conservative in the planning process. Experiments demonstrate that POMP consistently improves sample efficiency on widely used continuous control tasks. Our code is released at https://github.com/POMP-D3P/POMP-D3P. 

Published as a conference paper at ICLR 2023
MAKING BETTER DECISION BY DIRECTLY PLANNING IN CONTINUOUS CONTROL
Jinhua Zhu', Yue Wang", Lijum Wu', Tao Qin', Wengang Zhou', Tie-Yan Liu², Houqiang Li* !University of Science and Technology of China;
Microsoft Research AI4Science
tteslazhu@me ustc. edu. en, {zhwg, lingg edu. cn 2 "[yuwang5, li juwa, taoqin, ty] u wwwoomii..tcooommmmmmmoomoo
ABSTRACT
By properly utilizing the learned environment model, model-based reinforcement learning methods can improve the sample efficiency for decision-making prob- lems. Beyond using the learned environment model train a policy, the success MCMSS-bass methods shows that directly incorporating the learned environ- ment model as planner to make decisions might be more effective. However, when action space of high dimension and continuous, directly planning accord- ing to the learned model is costly and non-trivial. Because of two challenges: (I) the infinite number of candidate actions and (2) the temporal dependency between actions in different timesteps. To address these challenges, inspired by Differen- tial Dynamic Programming (DDP) in optimal control theory, we design a novel Policy Optimization with Model Planning (POMP) algorithm, which incorporates a carefully designed Deep Differential Dynamic Programming (D3P) planner into the model-based RI. framework. In D3P planner, (1) to effectively plan the continuous action space, we construct a locally quadratic programming problem that uses a gradient-based optimization process to replace search. (2) To take the temporal dependency of actions at different timesteps into account, we leverage the updated and latest actions of previous timesteps (i.e., step 1, ... h 1) to update the action of the current step (i.e., step h), instead of updating all actions simultaneously. We theoretically prove the convergence rate for our D3P plan- and analyze the effect of the feedback term. In practice, to effectively apply the neural network based D3P planner reinforcement learning, we leverage the policy network to initialize the action sequence and keep the action update conser- vative in the planning process. Experiments demonstrate that POMP consistently improves sample efficiency on widely used continuous control tasks. Our code released at attttps: /ttp:////ttttt//////// com. wttpttub.coo//OMM-D////PPP/ DD3P
INTRODUCTION
Model-based reinforcement learning (RL) (Janner et al. 2019a Yu ct al. 2020 Schrittwieser et al. 2020 Halner al. 202T has shown its promise to be general- purposse tool for solving sequential Docision-making problems. Different from model-fre free RL algorithms (Mnih et al. 2015 Haarnoja al., 2018). for which the controller directly learns a complex policy from real off-policy data, basel RI methods first learn predictive model about the unknown dynamics and then lever- age the learned model help the policy learning. With several key innovations (Janner et al. 2019a] Clavera et al., 2011)), model-based RL algorithms have shown outstanding data efficiency and perfor- mance compared to their model-free free counterparts, which make it possible to be applied in real-world physical systems when data collection is arduous and time-consuming (Moerland et al. 2020
There are mainly two directions to leverage the learned model model-based RL, though not mutu- ally exclusive. In the first class, the models play auxiliary role to only affect the decision-maie -aking by helping the policy learning (Janner et al.) 2019b; Clavera et al. 2019). In the second class, the model is used to sample pathwise trajectory and then score this sampled actions (Schrittwieser al.

---

Published as a conference paper at ICLR 2023
2020). Our work falls into the second class to directly use the model as planner (rather than only help the policy learning). Some recent papers (Dong et al. 2020 Hubert et al., 2021;] Hansen et al, 2022b) have started walking in this direction, and they have shown some cases support the mo- tivation behind it. For example, in some scenarios (Dong et al. 2020), the policy might be very complex while the model is relatively simple to be learned.
These idea is easy to be implemented in the discrete action space where MCTS is powerful to do the planning by searching (Silver al. 2016 2017 [Schrittwieser al., 2020; Hubert al. [022]. However, when the action space is continuous, the tree-based search method can not be applied trivially. There are two key challenges. (1) Continuous and nigh-dimensional actions imply that the number of candidate actions is infinite. (2yThe temporal dependency between actions implies that the action update in previous timesteps can influence the later actions. Thus, trajectory optimization in continuous action space is still challenge and lacks enough investigation.
To address the above challenges, in this paper, we propose Policy Optimization with Model Plan- ning (POMP) algorithm in the model-based RL framework, in which a novel Deep Differentiable Dynamic Programming (D3P) planner is designed. Since model-based RL is closely related to the optimal control theory, the high efficiency of differential dynamic programming (DDP) (Pantoja 1988 Tassa et al. 2012) algorithm in optimal control theory inspires us to design an algorithm about dynamic programming. However, since the DDP requires a known model and a high computational cost, applying the DDP algorithm to DRL. is nontrivial.
The D3P planner aims optimize the action sequence in the trajectory. The key innovation in D3P is that we leverage first-order Taylor expansion of the optimal Bellman equation to get the action update signal efficiently, which intuitively exploits the differentiability of the learned model. We can theoretically prove the convergence rate of D3P under mild assumptions. Specifically, (1) D3P uses the first-order Taylor expansion of the optimal Bellman equation but still constructs a local quadratic objective function. Thus. by leveraging the analytic formulation of the minimizer of the quadratic function, D3P can efficiently get the local optimal action. (2) Besides, feedback term is introduced in D3P with the help of the Bellman equation. In this way, D3P updates the action in current step by considering the action update in previous timesteps during planning. Note that D3P is a alug-and-plas algorithm without introducing extra parameters.
When we integrate the D3P planner into our POMP algorithm under the model-based RL framework, the practical challenge is that the neural network- learned model is always highly nonlinear and with limited generalization ability. Hence the planning process may be misled when the initialization is bad or the action is out-of-distributtin. Therefore, to provide the initialization of the action before planning and provide a conservative term at the planning to admit the conservation principle, in order to keep the small error of the learned model along the planning process. Overall speaking, our POMP algorithm integrates the learned model, the critic, and the policy closely make better decisions.
propose to leverage the learned policy
For evaluation, we conduct several experiments on the benchmark MuJoCo continuous control tasks. The results show our proposed method can significantly improve the sample efficiency and asymp- totic performance. Besides, comprehensive ablation studies are also performed to verify the neces- sity and effectiveness of our proposed D3P planner.
The contributions of our work are summarized as follows: (1) We theoretically derive the D3P plan- ner and prove its convergence rate. (2) We design POMP algorithm, which refines the actions the trajectory with the D3P planner in an efficient way. (3) Extensive experimental results demonstrate the superiority of our method in terms of both sample efficiency and asymptotic performance.
2 RELATED WORK
The full version of the related work is in Appendix A we briefly introduce several highly related works here. In general, model-based RL for solving decision- makiing problems can be divided into three perspectives: model learning, policy learning, and decision-makinn making. Moreover, optimal control theory also concerns the decision-making problem and is deeply related to model-based RL.
Model learning: How to learn a good model to support decision decision-making is crucial in bodel-based RL. There are two main aspects of the work: the model structure designing (Chua et al. 2018 Zhang
2

---

Published as a conference paper at ICLR 2023
et al. 2021
2020 Hafner et al.
2021 Chen et al. 2022) and the loss designing ① O'Oro al. 2020 2017 Li et al. 2021).
Farahmand et al.
Policy learning: Two methods are always used learn the policy by using the learned model. One is to serve the learned model as a black-box simulator to generate the data (Janner et all 2019bb Yu et al. 2020 Lee al., 2020). Another way is to use the learned model to calculate the policy gradient (Heess et al.) 2015b] (015))/Cleveraa et al. [019]] Amos et al. 2021).
need to generate the actions that achieve our goal. Many of the model-based RL methods make the decision by using the learned policy solely (Hafner al. 202T). Similar our paper, some works also try make decisions by using the learned model. but the majority only focus on the discrete action space. The well-known MCTS method achieves a lot of success. For example, the well-known Alpha Zero (Silver et al. 2017). MuZero (Schrittwieser et al.) 2020). There are only few works that study the continuous action space, such as the Continuous UCT (Couêtoux et al., (01T), the sampled MuZero (Hubert et al.) 2021), the TreePI (Springenberg et al. (020), and the TD-MPC (Hansen al. 2022a).
Decision-making: When making the decision,
Optimal control theory: Beyond deep RL, optimal control also considers the decision-making problem but rather relies on the known and continuous transition model. In modern optimal control, Model Predictive Control (MPC) (Camacho Alba 2013) framework is always adopted when the environment is highly non-linear. In MPC, the action is planned during the execution by using the model, and such a procedure is called trajectory optimization. Plenty of previous works (Byravan et al. 2021: Chua al. 2018, Pinneri et al., 2021 Nagabandi et al., 2020) use MPC framework to solve the continuous control tasks, but most of them are based on zero-order or sample-bases method do the planning. The most relevant works are DDP (Murray & Yakowitz 1984). iLQR (Li & Todorov] 2004). and iLQG (Todorov & Li, 2005 Tassa et al. 2012). We discuss the detailed differences between our method and these methods in Appendix
Since our planning algorithm relies on the learned model and learned policy, we build our algorithm based on these works on model learning and policy learning. Our POMP algorithm tries solve a more challenging task compared to the related work on decision-making: efficiently optimize the trajectory in continuous action space when the environment model unknown. Different from our works, the MPC with DDP as trajectory optimizer from optimal control theory requires the known environment model, and also requires the hessian matrix for online optimization from scratch.
3 PRELIMINARIES
Reinforcement Learning. We consider discrete-time Markov Decision Process (MDP) M, de- fined by the tuple f,r, where X is the state space, A is the action space, f Tt+1 = fzr, at) is the transition model, r : X' x A + E is the reward function, 7 the discount factor. We denote the future discounted return at time as R. = Ei' -re, and Reinforcement Learn- ing (RL) aims to find policy To X' x A + + that can maximize the expected return J. where maxg (0) = max Ex, R = max Er, xc E -r ar))
Bellman Equation. We define the optimal value function V*(x) = max E[R (z, = x]). The optimal value function obeys an important identity known as the Bellman optimality equation V*(z)
maxa, EE r( at (xt = x) + "V*((++)) The idea behind this equation is that if we know the r(xe, a+) for any ae and next step value function (2((++1 for any 8141, we can recursively select the action a, which maximizes r ag (r, = x) +>/++*(*//00/ (1141)) Similarly, we can denote the optimal action-value function Q" a) = m 22 = x, G = a), and it obeys a similar Bellman optimility equation Q" a) max B ((1 aela; 0, a) + +" 0ct1)).
Model-based RL. Model-based RL method distinguishes itself from model-free counterparts by using the data to learn transition model. Following Janner et al.] (2019a) and Clavera et al.) (2019), we use parametric neural networks to approximate the transition function, reward func- tion, policy function and Q-value function with the following objective function to be optimized H-1 at)], (ww E [[oorr(re (12,, J=( E EH 7"r(x1, 0t) +
(vb) = E. [log
3

---

Published as a conference paper at ICLR 2023
y QUHH [h} and JQ [2 ai) (r + (01411 ((1)))12), respectively. In Jx),), truncate the trajectory in horizon H to avoid long time model rollout.
Notations. For one-dimensional state and action case, denote the partial differentiation of function by using its output with subseripts, e.g., T2 - Or(a,a)
- dr(a.a)
fz - Øf(2.)) da
dx ro
da
fa @f(z,)) Qu aQ(z,a) and Qa A @Q(x.a), See Appendix On
B
for the multi-dimension case.
PLANNING IN CONTINUOUS ACTION SPACE
In this section, we present our POMP algorithm and the D3P planner in detail. First, we derive the D3P planner which relies on the Bellman equation. Then, we theoretically prove its convergence property. Finally, we show how to effectively apply D3P planner in our POMP algorithm in RI..
4.1 DEEP DIFFERENTIAL DYNAMIC PROGRAMMING
In this subsection, we will theoretically derive the D3P planner and prove its convergence property. There are mainly two challenges in continuous action space planning: (1) the infinite number of candidate actions, and (2) the temporal dependency between actions in different timesteps.
Here, we briefly introduce the main idea of our D3P planner to solve the above challenges. We first define an objective function and formulate it as an optimization problem based on the Bellman equation. Then, we convert it to a local optimization problem and approximate the objective function via Taylor expansion. To avoid the computation of the hessian matrix, we use the first-order Taylor expansion construct quadratic function. Since the analytical solution of quadratic function is easy to get, we can efficiently get the local optimal action sequence and thus overcome the challenge (1) to some extent. To get over challenge (2), introduce feedback term into the objective function to depict the state change induced by the action update in prior timesteps. By considering the feedback term that explicitly involves the information of prior action updates, we can correct the action update in time. The remaining question is whether the D3P planner can indeed optimize the original objective after we make several approximations when deriving the algorithm. Through theoretical analysis, we show that the convergence rate of the proposed algorithm can be guaranteed.
We now introduce how we derive the D3P planner. For clarification, we use the finite horizon MDP as a proof of concept setting. The state and action are one-dimensional variables. The infinite horizon MDP with multi-dimensional state and action can be derived similarly and we put in Ap- pendix D Recall the goal of RL methods, our planning algorithm aims to find the action sequences H
(ai ...,...000 .n that can maximize the value function V(n,, 1) maxa1,"0a 10 Ek=1 r(an, an). where Th+1 = f(z,, on).
Due to challenge (1), such optimal action sequence is in general hard to find. Hence D3P planner treats this optimal action sequence searching problem as an optimization problem that lever- ages the optimal Bellman equation to formulate the following objective function,
V( h) max (rax, on) + ([ an), h + 1)]- ay
Since the reward function and the transition function is unknown, we will use neural network to approximate them. However, the optimization problem is highly non-convex. Thus, we consider an auxiliary goal that is to find the local optimal a + Pa in the neighbourhood of current action a to improve the action from a to a + òa. Denote QITn, an) r(zn, an) + (ff an), h + 1), our goal can be re-expressed as daz arg maxáa [Q(Is, an + 5a)].
To accelerate the optimization process, D3P planner constructs a quadratic objective function to get the local optimal action analytically. Specifically, we propose to use the first-order Taylor expansion to avoid computing the hessian matrix. However, the first-order Taylor expansion can not lead to quadratic objective function directly, hence we first seek a surrogate objective function D(I, a) & (Q(z, a) Vmaz)? where Vinaz is constant and set to larger than the upper bond of Q(z, a). is easy to check that arg mins, D(x, da) A arg max Q a + ga).
For challenge (2), intuitively, after updating the action a; in prior timestep. state xe+11 will change and we should update the action ac+1 accordingly. Such manner is often called "feedback". To
4

---

Published as a conference paper at ICLR 2023
Algorithm Deep Differential Dynamic Programming (D3P)
Require: Initial action sequences {=r initial state æ1, iteration number Na. valid horizon H1 maximum expected improvement Vmaa Q(x a).
1: for = 1..................... ,H do
Initialize the trajectory.
2: Calculate r; (I,, az), Ts+1 fz)
3: end for
4: for 1,, Na do
Optimize the trajectory. Backward process.
5: Calculate Qr ah): Qa an)
6: for jor jo -..... do
7: Calculate Ta. rw, far fa-
8: Calculate Qa. Qu k K, Ve using Equation
4
and
9:
end for
0. for j - 1, H do
Forward process.
11:
Calculate du, using Equation 3 and aj + a, + daz.
13: Calculate 2++1 < fz ay). and 0zs+1 2j+1 Iz.
14:
end for
15:
end for
16:
return The last best action 01.
achieve the feedback control, now consider Q(x + 6z, + da), in which òæ represents the state change due to the prior action update. Applying first-order Taylor expansion for the Q function in D function get quadratic function of òo(recall the notations in Preliminary)
(x + dar, da) (Q( a) Qu (3r, a)da + Qe a)ōx )
we now get the optimal action update da* as a function of the feedback ox, denote ky (=(2a.en)
Q(wa.en) --vaa and KA, (zk,)an)) Qa (ms.onn)
daj = = Kλōτy = Q( an) - Viaaz Qa(xn,
Qa an Qu(ann and Szh.
(3)
The remaining part is how to calculate the Qz, a), Qu (z, a) in the update rule,
Qa(En, ax) (a an) (f ax), 1) fa an), Qx (xn, an)) = r. (vh, a4) + (= an), 1) f2(2,, an).
(4)
(5)
By leveraging the differentiable model including the reward and transition function, only the gradient of value function (f o,), h 11++1, is hard to calculate. We use the Bellman equation and Taylor expansion once again to calculate (f an). h 1). Putting da,, into Bellman equation (1) and using Taylor expansion,
V(n + 8xn, h)- = Q(Ix + òσy,, an + +
(6)
Q( an) + Qz an dorn + Qa an)oaj
(7)
= (Q(#n, an) Qa an jkn) + (Q=(In, an) Qa(xn, ap K₁ )8x
(8)
zero-ordar term
first-order term
We can now use the coefficient of the first-order term in Taylor expansion of V(x + Sxn, h) to calculate the Vz
V8 Q2 an) Qa Kn)
The whole D3P planner is shown in Algorithm method is applied in the deterministic environment, but our D3P planner be easily extended to the stochastic environment with reparameterization tricks (such as normal distribution noise in Kingma & Welling (2013). Since we adopt some approximation the derivation of the algorithm, we need some convergence guarantee.
Noting that the current presentation of our
Theorem 1. Let (In, an J..,,,,....1 denote the current state and action in a sequence of length T. Let {a/ an dan b=1,.-.11 H denote the actions updated once by D3P planner. Under mild
5

---

Published as a conference paper at ICLR 2023
assumptions, can prove that for h E {1,... H). there exist constant C and B such that
H
H
1/// - afll S C
Σ
lak- - afll + B
_ ak -
(10)
where proportional to the Lipschitz (denoted L1) and smoothness (denoted L2) constant of the transition function and reward function C = OLL,, L2). B proportional to the scale of the second order derivation of the transition and reward function B =O(fea San Faz: Taas Tax: Tzx).
The above theorem shows that we can choose good initialization point for the planning process, we can guarantee the asymptotic convergence of the planning process. For the finite sample case, the convergence rate least linear convergence. If the second derivative of the transition function is near zero (B is sufficient small), the convergence rate is near quadratic convergence. The intuition shown in Lemma 2 In this situation, the 2nd order derivative D can be approximated by the multiplication of the 1st order derivative of Q and thus of f and For example Daa = QaQa
We further analyze the influence of the feedback term in terms of the convergence rate.
Corollary 1. !f we do not consider the feedback term (òx 0), the convergence rate is llol - aj S C | -aa-aal"
aa - ax Il+ Qa(*s mb))
Ei=h-1 fu az) [f ai)6a; + C'aa.]
(a(
The corollary shows that if we do not consider the temporal dependency between actions in different timestep, in other words 8x 0, the convergence rate will be slower than Equation (12) with an extra error term. The intuition is, since we are optimizing the action sequence along a trajectory, the action update will change the trajectory. Given our objective is a function of state and action, the different states will lead to the different optimal actions. Therefore, we do not consider the state change due to the action update in the previous timesteps, the action update direction will not be toward the true gradient direction. Besides, the influence is proportional to the magnitude of the state change which is determined by the system property (fz, fa) and previous action update 8a,
4.2 POLICY OPTIMIZATION WITH MODEL PLANNING: PRACTICAL IMPLEMENTATION
In this subsection, we show how we apply our D3P planner to the deep RL framework. Since the D3P planner is a Blug-and-plle play algorithm, compared to the traditional model-based RL algorithm like MAAC (Clavera et al. 2019). only the decision-making parts are different. The POMP algorithm is summarized in Appendix B Note that D3P planner module does not introduce any additional neural networks. All network structure, including model, critic, and policy are the same as MAAC (Clavera et al. 2019) and MBPO (Janner et al., 2019b).
One key problem that needs to be resolved before applying the D3P planner how to avoid mis- leading planning due to the limited generalization ability of the learned model. Such a problem can not be ignored as long as the ground- model is unknown, which can only be learned by data with function approximation. We consider two components in the algorithm to alleviate the effect of the model error: the initialization strategy and the conservative planner objective.
For the initialization strategy, we propose to use the policy network and learned model to initialize the aate-actionn trajectory. That is, the initial action used by D3P planner the output of the learned policy. The motivations are as follows. (1) Since the policy is trained to maximize the return-to- go as general model- RI,, the proposed action would be reasonable and competitive, which is better than random initialization. (2) Since the data used to train policy is sampled from the replay buffer, the action outputted by the policy network should lead to small model prediction error.
For the conservative planner objective, constraining the actions outputted by D3P planner near the training data can keep the model prediction error small and provide an additional regularization for the planner. Specifically, since the policy output multivariate Gaussian, we can easily calculate the log- log-likelihood logP(2;,, aa) for a given state action pair. The log ligeliicl is used as an auxiliary reward, and we add it to the output of the reward function when doing planning in the evaluation phase. Specifically, we add an additional reward at the first step, and the optimization H-2 r(æn, an) + Q 0++H-1) + Please note that we only use this conservative term
objective of D3P becomes Je(fa,..... a++ H-1)) Di*H- a logP((,, a.). where a is during evaluation, as we want to encourage exploration when training.

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
In this section, we aim to answer the following questions: (1) Compared to state-of-the-art methods, how does our method perform on benchmark continuous control tasks? (2) planning necessary to make a better decision in continuous control? (3) Is our D3P planner advantageous in continuous control? (4) How the learned model quality affects decision-making? (5) Does our D3P efficiently optimize the trajectory quality? (6) Is the policy network necessary in our framework? To answer the above questions, we evaluate our method on continuous control benchmark tasks in the MuJoCo simulator (Todorov et al.] 2012). Our method is built on top of MAAC(Clavera et al. 2019), which means the procedure of model learning, policy optimization, and the corresponding hyper- parameters are the same as MAAC. More details are left in Appendix
C.3
Due to space limitation,
we leave the detailed description of the baseline methods in Appendix
5.1
COMPARISONS WITH EXISTING METHODS
InvertedPandulum
Hopper
Walker2d
4030
5000
1000
Ratuur
3030
Anerage Retumm
4000
800
-3000
600
2000
2000
4.00
200
0
10c
50k
100k
0
25k
s0k
100k
Steps Cheetah
Stepp
Steps
Ant
Humanoid
5000
12500
6000
5000
10300
Retur
4000
Aawrgge
7500
Amaaag
4000
3000
2000
1000
25k
53k
75k
10ok
25k
S0k:
75k
100L 125K 150k
%
25k
50k
75K
100k 125K
150K
Steps
Staps
Steps
SAC
SVG
STEVE
MAGE
MBPO
MAAC
OURS
Figure 1:
Learning curves of our method and other baseline methods on six continuous control tasks. The solid lines represent the mean of 10 (for our methody/5 (for other baseline methods) trails with different seeds, and the shaded regions correspond to STD among trials. Our method achieves the best results among these strong model-free and model-based reinforcement learning methods.
To answer the first question, we compare our method with six SOTA baseline methods, and the results are shown in Fig. [ Specifically, matter on asymptotic performance on the sample efficiency, our method shows a significant performance improvement against MAAC, of which our method is built top, on all six tasks. Moreover, on two control tasks with high- dinensiinnii action space, Ant and Humanoid, the improvement of our method are more obvious. In general, our method achieves better performance than all other model-based and model-free baseline methods, which demonstrates the effectiveness and generality of our method. Note that in humanoid task, MAGE achieves better sample efficiency than ours in early training phase, but our method achieves a better final result than MAGE and MAGE is worse than our method on all other tasks.
5.2 ABLATION STUDIES
In this section, we conduct several ablation experiments to answer questions (2) (6) posted before and show the necessity and effectiveness of the proposed components in our method.
Is planning necessary to make a better decision in continuous control? We design experiments to verify the effectiveness of two possible ways to make better decision: (1) Using the model to do planning and (2) Increasing the N₂ in Algorithm 2 which is the number of update times of the policy net after collect data from the real environment, and then relying on the policy to make the decision. Here we increase N, from 10 (in MAAC original implementation) to {20, 50, 100) to whether increasing the update times of the policy could help policy optimization, and the results
7

---

Published as a conference paper at ICLR 2023
Walker2d
Cheetah
Ant
5000
14000
2000
12000
4000
6400
Avwrage bekur
Antrage huture
10000
3000
4000
2000
6000
2000
4000
2000
2000
25k
50k
73k
10CK
25k
50k
75k
100k
25k
50k
75k
100K
125K
150K
Steps
Steps
Steps
MAAC (N₂ = 10)
MAAC (N2 20)
MAAC (N₂ = 50)
MAAC (N,- 100)
OURS
Figure 2:
Ablation about the update times Np of policy in each iteration. We can see that increasing Np cannot help policy optimization.
As shown in the figure, N₂ = 10 in the original MAAC is a rather good choice, and increasing v even would harm the policy optimization. However, our method, which uses the learned model as a planner could consistently improve the policy.
are presented in Figure 2
Is our D3P planner advantageous in continuous control? D3P planner considers the temporal dependency and constructs local quadratic objective function to optimize the initial trajectory pro- posed by the policy network. To validate the advantage of our method, we replace the D3P planner in our method with an SGD-like planner, which directly optimizes the action sequence with gradient ascend; a random- mhootig planner (Press et al., 2007). which randomly samples some actions in the entire action space and then scores these actions according to the reward and critic function; a cross-entropy method (CEM) planner (Rubinstein & Kroese, 2004, Hansen et al., 2022a). which adaptively and iteratively adjusts the sampling distribution in sophisticated manner. Noting that we only change the planner in all these variants, and keep the model and policy learning unchanged for a fair comparison. The results are shown in Figure 3 and we can see that SGD-like planner (denoted by POMP with SGD planner) performs similarly to policy network (denoted by MAAC) and the improvement over policy (MAAC) is limited. Our method (denoted by POMP with D3P planner) is more effective than SGD-like planner. Moreover, the gaps between our method and the CEM planner (denoted by CEM), the mando-shootinn planner (denoted by Random-shooting clearly show the efficiency of the first-order method (compared to the zero-order method).
our method optimizes the tra- jectory via planning in a learned environment model, key part is to see how the learned model quality affects the planning results. To answer this question, we pick 4 types of the learned model with different amount of training data ( thhe more training data, the better the quality of the learned model). Then we cluster the policy network according to their performance into 6 groups Finally, combine the different quality models with each policy group see the average performance improvement after we applying the D3P planner on the learned model and policy. First, for each
How the learned model quality affect decision-making?
Walker2d
Cheetah
5000
14000
7000
12000
4000
6000
Antrage Retum
Betum
3000
4000
2000
6000
3000
4000
2000
1000
%
25k
50k
75k
100k
23k
50k Staps
75k
100k
%
25k
50k
75k
100k
125k
150k
Staps
Steps
MAAC
POMP with SGD planner
CEM
Random-Shooting
POMP with D3P planner
Figure 3: SGD-like planner, CEM planner, and a random-shooting planner, the results show the advantage of our D3P planner.
Ablation studies about D3P planner. We replace the D3P planner in our method with a
8

---

Published as a conference paper at ICLR 2023
9930 L05 10ns
improrsmans
400
Impropement
2300
by Qualit 10T 1015 1o-1.
(traming number)
1e) Triining Steps
Figure 4: (a) The improvement of applying learned model with different training steps on policy with different quality. "Improvement" means the evaluation return using our D3P planner to sub- tract the return that without our D3P planner. "Policy quality" means the average episode return of the policy when applying the policy in the environment, and "ik~(i + 1)k" denotes the policy cluster whose average return lies in i + 1)k. "Model i denotes the learned model which is trained using ik data. (b) The improvement of different iteration number Na in D3P (Line 4 in Algorithm []. "Model quality" means the number of training data used to train the model, and "ik~jk" denotes the learned model with ikwijk training data. (c) Ablation about the policy usage in our method, "RAND" denotes POMP with randomly initialized trajectory rather than a policy generated trajectory in D3P. "Na = i" denotes POMP with iteration number and "Na = i w/o cons" denotes POMP with iteration number i and without the conservative term when evaluation.
model and each policy, we evaluate the average return using 10 trajectories. Then, we cluster the learned model and policy according to their training data and the average return and then calculate the average performance improvement in each cluster. From the result shown in Figure4aay (1) the improvement of the model trained on only 10k train data is similar to those of models trained by more data (except 5k>~6 is slightly worse), which means it is enough to use early stage model in our D3P planner; (2) our D3P planner could consistently improve the performance the decision made by policy network directly, especially in early and middle stage.
Does our D3P efficiently optimize the trajectory quality? Similarly, we cluster the learned model according their used training data, and combine it with a fixed policy (with an average return about 4k) and see the impact of different iteration numbers Na used in our D3P planner. From the results shown in Figure bb) (1) the performance improvements increase as we use more iteration numbers, which shows the effectiveness of our method; (2) the improvements are almost the same after Na >= 6. and we do not need more iterations, which demonstrate the efficiency of our method; (3) the results also show that the early stage model is enough for our D3P planner.
Is the policy network necessary in our framework? There are two usages for the policy network in our D3P planner: (1) initialize the trajectory to be optimized, (2) add a conservative term as an auxiliary reward during evaluation. We conduct an ablation experiment to verify the necessity of the policy network in our method, and the results are shown in Figure [c). First, when we use trajectory randomly generated rather than proposed by a policy network, the D3P failed to find any meaningful action (denoted by "RAND"), which proves the importance of trajectory initialization. Second, as we increase the iteration number in D3P planner, the performance with our conservative term is consistently better than those without it, especially the later stage when the policy network is near optimal. This means the generality of the learned model is limited when we use a large iteration number Na. and we need constrain the optimization space of the method.
CONCLUSIONS AND FUTURE WORK
In this work, we first derived the D3P planner which is effective and efficient for continuous control and proved its convergence rate. Then, we proposed the POMP algorithm, which leverages our D3P planner in practical model-based RL framework. Extensive experiments and ablation studies on benchmark continuous control tasks demonstrate the effectiveness of our method and show the benefit of utilizing the model planning in continuous control. For future work, given the model uncertainty can effectively trade-off the exploration and exploitation, how to properly estimate and incorporate the uncertainty of the learned model into planning is meaningful topic.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by NSFC under Contract 61836011, and in part by the Fundamental Research Funds for the Central Universities under contract WK3490000007..
REFERENCES
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, pp. 6-20. PMLR. May 2021. URL BtL/http:/// httt:://prroeeedings. mlr wttp://gs........ssssss.sl.. wwwps/veessll.ese
amos2la html
ISSN: 2640-3498.
Yuichiro Aoyama, George Boutselis, Akash Patel, and Evangelos A Theodorou. Constrained differ- ential dynamic programming revisited. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 9738-9744, IEEE, 2021.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide la- longo, Yuval Tassa, Jost Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, et Evaluating model-based planning and planner amortization for continuous control. arXiv preprint arXiv:2 10.03333, 2021.
Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.094811 2022.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in Handful of Trials using Probabilistic Dynamics Mod- els. In Advances in Neural Information Processing Systems, volume 31. Curran
Associates. Inc., 2018. URL lattss:/ppapers wwtp:/pppppppppp...e.ppppp httpsc/apppp//////aaa...
Abstrrcccbbrract..... html
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-Augmented Actor-Critic: Backpropagating through Paths. In International Conference on Learming Representations, September 2019. URL
atttps://pperreiite wet.ieewwweeeeeeeeeeeeeeee.eeeee wtte://ooummmiids111222
Adrien Couëtoux, Jean- Bantisee Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bon- nard. Continuous upper confidence trees. In International conference learning and intelligent optimization, pp. 433-445. Springer, 2011.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International Conference Machine Learning, pp- 2627-2637. PMLR, 2020.
Pierluca D'Oro and Wojciech Jaskowski. How to learn a useful critic? model-based action- -rriinng estimator policy optimization. Advances in Neural Information Processing Systems, 33:313 2020.
Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient-Aware Model-Based Policy Search. Proceedings of the AAAI Conference Artificial Intelligence, 34(04) April 2020. ISSN 2374-3468. doi: 10.16099amii.v34/// 3399.,5779.
URL https://ojs..ooo aaai http:://s..saaaaa.cr/eaaaaaaaaar 04.
Number:
Amir-Massout Farahmand, Andre Barreto, and Daniel Nikovski. Value-Aware Loss Function for Model-based Reinforcement Learning. In Artificial Intelligence and Statistics, pp. 1496-1494.
PMLR. April 2017. URL [attp::/proceedingg/ mlr. wwwpppresseeeeeee http::eediiiaaai..meeeeeeeeeeese
html ISSN: 2640-3498.
10

---

Published as a conference paper at ICLR 2023
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018.
Danijar Hafner. Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL
htttss://ppeneewe ew net. Forummmmm www.oorummddooaabbooooa d=0oabwyZb0u
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04923 2022a.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8387-8406. 8406. PMLR, 17-23 Jul 2022b. URL httph:: mmr www.mmw.....ppp..mm.mm..... aaress/v162/aaneeee
html
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015a.
Nicolas Heess, Gregory Wayne, David Silver. Timothy Lillicrap, Tom Erez, and Yu- val Tassa. Learning Continuous Control Policies by Stochastic Value Gradients. In Advances Neural Information Processing Systems, volume 28. Curran Asso- ciates. Inc., 2015b. URL httos: nips. CC/ pttpp/ape//2///as///// Abstract abtraaaasss...aaaeeaaaaaaaa....a .tmll
Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 4476-4486. PMLR, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019a.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model- Based Policy Optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b. URL htwp::ttt////t /ttpsss:///pppps...m nips cc pttpppppee///1//aaaat. 03000099907700000000133333304130
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint
arXiv: 1412.6991 2014.
Diederik Kingma and Max Welling. Auto-cncoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representations, 2020.
Chongchong Li, Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, and Tie- Yan Liu. Gradient in- formation matters in policy optimization by back-propagall through model. In International Conference on Learning Representations, 2021.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (7), pp. 222-229. Citeseer, 2004.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 ((7754)):529-533 2015.
Thomas M Moerland, Joost Brockens, and Catholijn M Jonker. Mlddel-baeel reinforcement learn ing: A survey. arXiv preprint arXiv:2006. 127 2020.

---

Published as a conference paper at ICLR 2023
DM Murray and SJ Yakowitz. Differential dynamic programming and newton's method for discrete optimal control problems. Journal of Optimization Theory and Applications, 43(3):395 414, 1984.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
F. De O. Pantoja. Differential dynamic programming and newton's method. International Journal of Control, 47(5))1539-1553, 1988.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference Robot Learning, pp. 1049-1065. PMLR, 2021.
William H Press, Saul A Teukolsky, William I Vetterling, and Brian P Flannery. Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press, 2007.
Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: unified approach to com- binatorial optimization, Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004.
Julian Schritwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839) 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 -59993544999--- 2017.
Jost Tobias Springenberg, Nicolas Heess, Daniel Mankowitz, Josh Merel, Arunkumar Byravan, Ab- bas Abdolmaleki, Jackie Kay, Jonas Degrave, Julian Schrittwieser, Yuval Tassa, et al. Local search for policy iteration in continuous control. arXiv preprint arXiv:2010.05544. 2020.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906-4913. IEEE, 2012.
Emanuel Todorov and Weiwei Li. A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pp. 300-306. IEEE, 2005.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142 2020.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics, pp. 4015-4023. PMLR, March 2021. URL https:: pttpsss/pproeeeaggg..oo ISSN: 2640-3498.
press/ v 113vvv1100222222 1n. html
Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2023
A RELATED WORK
Model-based RL methods for solving decision-making problems focus on three key perspectives: how to learn the model? how to use the learned model to learn the policy? And how to make the decision using the learned model and policy? Besides, decision-making that relies on the model is also investigated in the optimal control theory field which is deeply related to model-based RL.
Model learning: How to learn good model to support decision-making is crucial problem in model-based RL. There are two main aspects of the work: the model structure designing and the loss designing. For model structure designing, ensemble-based model (Chua et al. 2018), dropout
mechanisms (Zhang et al. 2021), auto-regressive structure (Zhang et (000), stochastic hidden model (Hafner et al. 2021), ) and transformer based model (Chen et all.)2202 are always considered
to improve the model robustness and prediction accuracy. For loss designing, decision awareness
OD'Oro et al.] 2020: Farahmand et al., (017) and gradient awareness (Li et al. 2021) always
considered to reduce the gap between model learning and model utilization.
Two methods are always used learn the policy by using the learned model. One is to serve the learned model as a black-box simulator to generate the data. Janner et al. (2019b) representing work of this line. Yu al. (2020). Lee al.] (2020) also follow such a manner by extending it to offline-RL RL setting. Another way is to use the learned model to calculate the policy gradient. Heees et al. (2015)) presents an algorithm to calculate the policy gradient by back- propagating through the model. Clavera et al. (2019) and Amoss et al. (2021) share similar methods but use promising actor and critic learning strategy to achieve better performance.
Policy learning:
When making the decision, we need to generate the actions that can achieve our goal. Most of the model-based RL methods make the decision by using the learned policy solely (Janner et al. 2019b; Yu et al. 2020; Clavera et al. 2019 Hafner et al., (021). Similar to our paper, some works also try to make decisions by using the learned model, but the majority only focus on the discrete action space. For example, the well-known Alpha Zero system (Silver et al.) (017) uses MCTS to derive the action by using the known model. In MuZero and (Schrittwieser et al.) 2020), the authors propose to a learned model combined with MCTS planner to achieve significant performances in a broad range of tasks within discrete action space. There are only a few works that study the continuous action space. Couêtoox al. (201 I) extends the MCTS framework to continuous action space but also needs to know the real model and handle the model. In Huberi al.) (202T), the author proposed a sampled MuZero algorithm to handle the complex action space by planning over sampled actions. In Hansen et al. (2022a), the authors propose to learn a value function that can be used as long term return in the Cross-Entropy (CE) method for planning.
Decision-making:
Optimal control: Beyond deep RL, optimal control also considers the decision-making problem but rather relies on the known and continuous transition model. In modern optimal control theory, Model Predictive Control (MPC) (Camacho & Alba 2013) framework is always adopted when the environment highly non-linear. In MPC, the action is planned during the execution by using the model, and such procedure is called trajectory optimization. There plenty of previous works that use the MPC framework to solve continuous control tasks. For example, Byravan et al. (202I) proposes to use campling-based MPC for high-dimensional continuous control tasks with learned models and a learned policy as a proposal distribution. PPinneri et al.] (202T) proposes an improved version of the Cross-Entropy Method for efficient planning. Nagabandi et al. (2020) proposes PDDM method that uses gradient-free planner algorithm combined with online MPC method to learn flexible contact-rich dexterous manipulation skills.
Differential Dynamical Programming: The most relevant works are DDP (Murray & Yakowitz, 1984]. iLQR (Li & Todorovi 2004). and iLQG (Tassa al.) 2012). Differentiable Dynamic Pro- gramming (DDP) (Tassa et al. (2012) employs the Bellman equation structure (Murray & Yakowitz, 1984 Pantoja, 1988 Aoyama et al. 2021) and has fast convergence property. becomes more and more popular in the control field. iLQR (Li & Todorov| 2004). and iLQQ (Tassa et al.) 2012) Todorov & Li 2005) are two variants of the DDP. In iLQR and iLQG, the second-order derivative of the environment model is ignored (set as zero). Therefore, iLQR and iLQG are more computa- tionally efficient compared to the original DDP method. Since both iLQG and our D3P planner are motivated by DDP, they look similar naturally. But our method has several key differences com- pared with theirs, and these differences are well-designed to incorporate the neural network model. (1) DDP, il QLQ, and il QG are both pure planning algorithms that require known environment
13

---

Published as a conference paper at ICLR 2023
model. (2) Computing the second-order derivative of the neural network based model is compu- tationally costly (Hessian matrix). In our method, we only rely on the first-order derivative of the model. (3) The previous methods the second-order Talyor expansion of the Q-value function to handle the local optimization problem. But it is hard to guarantee that the hessian matrix is a nega- tive definite matrix, which is necessary condition for convergence. Here, we construct auxiliary target functiin D and use a first-order Talyor expansion for the Q function inside of the D function guarantee the non-positive definite matrix.
B POMP ALGORITHM
In this subsection, we present the details of POMP algorithm. Overall speaking, POMP algorithm learn three components: model, critic, and actor with the neural network function approximator, and leverage the D3P planner module integrate all three components make better decision.
The POMP algorithm runs as follows. We first learn the model, the policy, and the critic using pre-given or random-polle seciloo--ooicy-geeeeeee
data. Then, we leverage the D3P planner generate actions based on the model, the critic, and the policy network to interact with the environment and add these data to the true replay buffer. Next, we will use the data from the true replay buffer to train the model. We also generate fake data by using the learned model and add these data to the fake replay buffer. After that, we will sample the data from both real buffer and fake buffer to train the critic and policy. We will repeat the training process until certain convergence conditions are satisfied. When doing planning and rollout with the learned model to generate fake data, we follow the method used by Janner al. (2019a); Clavera et al. (2019) to truncate the trajectory and use Q-function to approximate the return after the truncation. When updating the policy, we calculate the policy gradient by pack-propagating through the model which is inspired by Clavera et al.) (2019). When updating the critic, we follow the SAC (Haarnoja et al. 2018) to construct two Q-functions with two target Q-functions and apply the soft Q-update.
Algorithm 2 POMP
Require: Policy update times Np. total interaction number N, model train frequency k. 1: Initialize the learnable model fø, the reward function Tw. the policy network ne, the critic Qo. replay buffer D.m 0, fake replay buffer D Dmndet 0.
2: for 1,
do
3: Initialize the action sequence using policy net TT. and learned model fe-
4: Interact with real environment Eyeat using D3P planner (Algorithm and add the transition
Dem--
5: if mod k == 0 then
6: repeat
7:
ar Vwrr using data from Dam. until The learnable model and reward function converge.
Update vi vh of V.JJ, w f-
8:
9:
end if
10: Sample transitions with fu, and add them to D moddel
11:
D < PD U D model
12:
for 1, Np do
13:
Update 0 0 ar VoJr using data from D. Update e + Φ aq V VoJQ using data from D.
14:
end for
16: end for
return Optimal parameters v w", and 6²
C EXPERIMENT SETUP
C.I IMPLEMENTATION DETAILS
How to set Viar- Q(2 a)? In our D3P method, we introduce constant Vmar and set it larger than the upper bound of (1 a). However, we can not know the true value of the upper bound Q(x a), and setting too large small Vmaz not perfect for planning. In our implementation, fist define
14

---

Published as a conference paper at ICLR 2023
InvertedPendulum
Hopper
Walker2d
Cheetah
Ant
Humanoid
Training Steps
15000
100000
100000
100000
150000
150000
Batch Size
256
256
256
256
256
256
Learning Rate
3e
3e 4
3e
3e
3e
3c
Horizon N
3
10
10
10
10
Na
10
10
10
10
10
10
Model train freq &
250
250
250
250
250
250
Ensemble Size
7
7
Maximum Vaaz (xx,a)
50
20
10
20
20
50
Table 1: The detailed hyper hyper-parannnn parameters in our experiments.
a maximum expected improvement and then grid search Vmar - ((x a) == {exp
log ( x i)li
1, K} to get the best Vmaw according to our planning objective function. Please note that the grid search are implemented in parallel.
C.2 DESCRIPTIONS OF OUR EXPERIMENT ENVIRONMENTS
Following prior model-based RL work, we conduct our experiments on classical continuous con- trol tasks from MuJoco (Todorov al. 2012), and the descriptions of these environments are sum- marized as follows!
Inverted Pendulum: This environment involves a cart that can be moved linearly, with a pole fixed on it at end and having another end free. The cart be pushed left right, and the goal to balance the pole on the top of the cart by applying forces on the cart. The action space dimension and state space dimension are and 4, respectively.
2. Hopper: The hopper
two-dimensional one-legged figure that consists of four main body parts the torso at the top, the thigh in the middle, the leg in the bottom, and single foot on which the entire body rests. The goal to make hops that move in the forward (right) direction by applying torques the three hinges connecting the four body parts. The action space dimension and state space dimension are and1 11, respectively.
3. Walker2D: The walker a -wo-dimensional two-legged figure that consists four main body parts single torso at the top (with the two legs splitting after the torso), two thighs in the middle below the torso, two legs in the bottom below the thighs, and two feet attached the legs which the entire body rests. The goal is make coordinate both sets of feet, legs, and thighs to move in the forward (right) direction by applying torques on the six hinges connecting the six body parts. The action space dimension and state space dimension are 6 and 17, respectively.
4. Half Cheetah:
The HalfCheetah is 2-dimensional robot consisting of 9 links and 8 joints connecting them (including two paws). The goal is to apply a torque on the joints to make the cheetah run forward (right) fast as possible, with a positive reward allocated based on the distance moved forward and a negative reward allocated for moving backward. The torso and head of the cheetah are fixed, and the torque 6 joints over the front and back thighs (connecting to the torso), shins (connecting to the thighs), and feet (connecting the shins). The action space dimension and state space dimension are 6 and 17, respectively.
only be applied on the other
5. Ant: The ant is a 3D robot consisting of one torso (free rotational body) with four legs attached to with each leg having two links. The goal is to coordinate the four legs to move in the forward (right) direction by applying torques on the eight hinges connecting the two links of each leg and the torso (nine parts and eight hinges). The action space dimension and state space dimension are 8 and 27, respectively.
6. Humanoid: The 3D bipedal robot is designed to simulate human. It has torso (ab- domen) with a pair of legs and arms. The legs each consist of two links, and so do the arms
Please refer to https: wwwpwwwwmmmmmmmm.. iibrarrrr...o wwwp:/deeeeeeveeeeeereeoeeeee
for details.
15

---

Published as a conference paper at ICLR 2023
(representing the knees and elbows respectively). The goal of the environment is to walk forward as fast as possible without falling over. The action space dimension and state space dimension are 17 and 376, respectively.
C.3 EXPERIMENTAL DETAILS
In our method, for a fair comparison, except the D3P planning, we keep the model learning pol- icy learning, and Q-function learning to be the same as prior work (Janner et al., 2019b; Clavera et al., 2019). Specifically, the learnable prediction model is parameterized by an ensemble of individual 5-layer MI.Ps, and is trained by Adam optimizer (Kingma Bag 2014) with all history transition data from replay buffer after certain hundreds of timesteps (the timesteps vary depend- ing on the specific task); after each interaction with the environment, the policy is optimized using the pathwise derivative of the imagined trajectory produced by the learned model and the learned policy; the Q-function is learned by minimizing the TD-error for each history data saved in replay buffer and imagined data from learned model and policy function. The detailed hyper-paramette are summarized in Table | and refer to Janner et al. (2019b); [Clavera et al. (2019) for more details.
(Claver
Noting that our planner is built upon the framework of MBPO and MAAC. Therefore, the sample efficiency of our method is comparable with MBPO and MAAC which also used the same state augmentation strategy. So, the improvement of the sample efficiency is not relevant to the state augmentation strategy.
C.4 THE DESCRIPTION OF BASELINE METHODS
To show the effectiveness of our algorithm, we compare our method on six classical continuous control tasks against the following state-of-the the-artt model-free and model-based RL algorithms: (i)
Soft Actor-Critic (SAC) (Haarnoja et al.) 2018 a popular off-policy actor-critic RL algorithm based on maximum entropy RL framework; (ii) SVG(I) (Heess et al. (0155a), which first uses dynamics derivatives model-based RL; (iii) STochastic Ensemble Value Expansion (STEVE) method (Buckman et al., 2018). which utilizes the learned models only when the uncertainty of the learned model is not too high; (iv) Model-based
policy optimization (MAGE) method Oro & Jaskowski 2020), which computes gradient targets in temporal differ- ence learning by backpropagating through the learned dynamics; (v) Model-Based Policy Optimiza- tion (MBPO) method (Janner al. 2019b), which shows that using short model-generated rollouts Actor-Critic (MAAC) (Clavera et al., 2019) method, which exploits the learned model by computing the analytic gradient of the returns with respect to the policy.
branched from real data could benefit model-based algorithms; (vi) Model-Augmentt
D MORE EXPERIMENTAL RESULTS
D.1 STUDIES ON THE ROBUSTNESS OF OUR METHOD.
We test the sensitivity of our method when we change the hyperparameter used in the training phase. The ablation studies about iteration number Na used in our training phase and the maximum expected improvement Viaaz - Qxx,, a) (which we denote by A) are shown in Figure Figure respectively. We can that our method consistently outperforms MAAC, and the dhyerr-aramee choice is not much sensitive to our method.
D.2 COMPARISON WITH CONTINUOUS MUZERO
MuZero (Schrittwieser al. 2020 is successful model-based RL method for discrete action tasks, which carefully trades off the exploitation and exploration. To compare these tree-based methods with our gradient-based method, we conduct a comparison with MuZero combined with Continuous UCT (Progressive Widening (Couêtoux et al. 2011 in our experiments We gird search several important typer-parameters for the continuous MuZero variant, and the detailed
[
are summarized in Table 2
The results are shown in Figure 7 From this figure, we can see that as
We use
commonly used public
code
https://ge thub. www.//eree----uvuaeeee
muzero- genera htteeraa/teeeeoottiiiiee
16

---

Published as a conference paper at ICLR 2023
Walker2d
Cheetah
5000
14000
12000
4000
Auerage Baturn
Retur
10000
3000
2000
9000
4000
2000
1000
2000
1000
%
75k
100K
%
100K
%
50k
100K
POMP with N2
POMP with Ng = 10
POMP with Ng = 20
MAAC
Figure 5: Ablation studies on hyperparameter iteration number Na in training.
Walker2d
Cheetah
Ant
14000
7000
4000
Beeurrs
Average Return
Auerage Returr
1000
3000
4000
2000
2000
1000
25
30k Staps
75k
10tk
%
25k
50k
75k
100k
%
25k
50k
75K
200K
125k
150k
Steps
Staps
with A
with A 20
POMP with &
POMP with 4-10
MAAC
Figure 6: Ablation studies on hyperparameter A.
the dimension increases, the gap between our method and the continuous MuZero variant is more and more obvious, which shows the advantage of our method. This also implies that employing Muzero in continuous domain effectively non trivialii.l. Since UCT a principled way to do the exploration in the discrete domain, combining it with our D3P planner for continuous domain will be an interesting research direction the future.
pyper-parameter
Values
a in Progressive widening
{0.3, 0.4, 0.6, 0.7, 0.8 {1.0, 1.25, 1.5, 2.0
0.6,
C in UCB
Simulation step / in MCTS
{ 64, 128, 256, 512)
Table 2: We grid search several important hyper-parametesd for the continuous MuZero variant.
D.3 STUDIES ON THE PLANNING HORIZON
We fix the planning horizon H to be the same as those in MAAC Clavera et al. (2019), since they have systematically studied this hyper-parameter in Section 5.2 of their paper: the gradient error scales poorly with the horizon, and large horizons are detrimental since it magnifies the error on the models. We also add ablation study to show how the planning horizon influence the performance of our method in Figure g The results are consistent with prior work Janner et al. (20196); Clavera et al. (2019).
17

---

Published as a conference paper at ICLR 2023
Swimmer
Hopper
100
4000
5000
3500
90
4000
3000
Nuerage urtum
60
2500
3000
40
2500
20
1000
0
50k
100K
%
50k
100k
Continuous MuZero
POMP
Figure 7: The comparison of continuous MuZero variant with our method. The dimension of action space for Swimmer, Hopper and Walker2d are 2, 3, and 4, respectively. We can see that as the dimension increases, the gap between of method and the continuous MuZero variant are more obvious, which shown the advantage of our method.
Ant
MAAC
H 1
H=
Average Retur
5000
==
4000
=
H 10
3000
2000
1000
%
25k
50k
75k Steps
100k
125k
150k
Figure 8: The studies on planning horizon H.
D.4 PLOTTING RESULTS OF DIFFERENT RANDOM SEEDS
Since all the RL literature compare different methods by plotting the mean and standard deviation in their paper, we follow the common practice in our paper. Besides, we also provide the individual run curve in Figure D Obviously, we plot individual runs for each method, it will be messy and unclear for visualization.
Walker2d
Cheetah
Ant
5000
7000
12000
6000
10000
5000
Average
2000
getraige
8000
4000
2000
3000
4000
2000
1000
2000
1000
%
25k
50k Steps
75k
100k
00
25k
50k Steps
75k
100k
25k
50k
75k
100k
125k
150k
Steps
Figure 9: The individual 10 runs of our method.
18

---

Published as a conference paper at ICLR 2023
D.5 THE IMPACT OF THE NUMBER OF EXPERIMENT RUNS
We have shown the performance of our method with 10 seeds and plotted the mean curve and shaded region with deviation in Figure (the individual 10 runs are also shown in Figure 9 One may still wonder whether the limited number of runs would influence the experimental results. Thus, run each task with another 20 more seeds (30 seeds, totally), and show the results in Figure []] Comparing the results of 30 seeds with the results of 10 seeds (shown in Figure TOp, we can see that the impact of the number of experiment runs is limited to our method, which does not alter our experimental conclusion. Last, as the RL committee always shows the results with the mean and deviation values, we acknowledge that more runs of each task are needed to show robust and consistent experimental results for RL algorithms.
Hopper
Walker2d
Cheetah
12000
4000
3000
Average Return
Average Return
Average Raturn
10000
2000
8000
2000
2000
6000
4000
1000
100k
%
%
Steps
Steps MAGE
Steps
SAC
SVG
STEVE
MBPO
MAAC
OURS
Figure 10: The experimental results with seeds of our method.
Hopper
Walker2d
Cheetah
4000
14000
12000
4000
3000
Average Retum
Average Retur
Average Retum
10000
3000
8000
2000
2000
6000
1000
%
%
%
Steps
Steps
Steps
SAC
SVG
STEVE
MAGE
MBPO
MAAC
OURS
Figure 11: The experimental results with 30 seeds of our method.
VECTOR FORM OF OUR D3P PLANNER
For brevity and clear clarification, we treat the action and state as one-dimensional scalars in our main paper. Here we provide the vector form of the derivation the D3P algorithm.
Consider the state and action are both multi-dimensional vector with dimension dx and d.. The transition function is now mapping: Rd + Rdx the reward function now a mapping: Rd - R.. In this situation, fa is the Jacobin matrix of shape dy da. whose (i, j)-th entry is fa., day"
dl, Similarly fx is the Jacobin matrix of shape dx dx. whose (i. j)-th entry is fx., dxj"
d/s
19

---

Published as a conference paper at ICLR 2023
ra is the Jacobin matrix of shape x da whose (1. j)-th entry is rau,
dr. day"
is the Jacobin matrix
of shape 1 dx, whose (1, j)-th entry is Tx1; dx,
dr
The objective function of our D3P planner is
V(x, h) max[r(Xx, an) + ((f(x an), h + 1)].
(11)
Denote Q(x,, an) = r(xh, an) + (x(x,,, an) + 1), our goal can be re-expressed as
San = arg max [Q(x,, an + -a)]]
(12)
We seek surrogate objective function (xx a) 2 (Q(x. a) Vmma)). and we then apply first-order Taylor expansion for the Q function ((x, a) (xx,a). a),
D(x, a + 5a) = (Q(x, a) + Qa a)óa Vmaa)²
(13)
So, the optimal action update is da" (Q((x,) Van ( (x,
Then we introduce a feedback term 5x. denote
k (Q(x, a) (Qa (x, ax) a)) Q0 (x,a);
(14)
K (Q (x, Q, a)) Q2 (x, a) a) (x,
where the shape of ls is d, x and the shape of K is du dz. The update rule of the action given by:
Ša = kk Kôxx.
(15)
The update rule of (x an) and Qx (Xn, an)
Qx an) = In [h: an) + (x an), 1 1) fa(x,; an);
(16)
Qx (xn, an) Tx (Xn, an) + (x an), 1 fx(xx, an),
and we can calculate Vx by
V = Qx(xn, Oa(xb,a))K
(17)
F PROOF OF THEOREM
In this section, present the proof of the Theorem 1 and Corollary in Section 4. First of all, summarize the necessary assumptions here.
The transition f(x, a) and reward function r(x, a) are both continuous and with continuous first and second order derivative. The first and second order derivative bounded by L1 and L2 respectively.
Assumption 1.
ll + llfall + || + || l1 $ L1
(18)
f + Ilfzall + Ilfaall || + || ||rzal| + ||Fall| S L2
(19)
Assumption 2. The variables Qa calculated in the iteration of D3P are always non-zero.
F.1 PROOF OF THE THEOREM
Overall speaking, will use the mathematical induction method to prove the theorem. We will first prove the convergence rate given the trajectory length H = 2. Then, we assume the theorem is true in trajectory with length H = I, and prove it still holds in trajectory with length H +++ 1.
In the proof, we denote the trajectory length as H. and denote the location in the trajectory using h where h E 21,2, H). We denote the action in h after update d', where ah on dap- We denote the optimal action as af, where a; arg max,, r(2f,, az) + Vf a,), h + 1) where th fa o and xi 21.
20

---

Published as a conference paper at ICLR 2023
In the proof, we will use A with subscript like A1 to denote some formulation for simplicity and will give its expression in before we use it. We will use B with subscript like B1 to denote the term related to the error due using the first-order derivative to approximate the second order derivative. We will use C with subscript like C1 to denote the general constant.
Before the proof, we first recall the update rule of the D3P planner.
dap kip Kndwn (zn, ap) Viaa
Qa(an and
(20)
(Jb; un)
Qa(wn, an)
Qa an) (ah an) + Va( h + 1)fa (1h, an), Qa an) = (zh an) + V2 (2 h+ + 1)f² (Ih, an),
(21)
(22)
Va( an) Qn an) Qa (2h, a Qu an) Qa(an, an) (oh
(23)
Qa(an. an)
First of all, we consider the case when trajectory length H=2, We calculate the error of aj and al in
terms of
d22 a2 02 02 5a²
(24)
(02 02) Vina
Qx(12, 02) dx
02 2
(25)
Qu(22, 02)
Qa (12)
(12, (2) - a₂) Qa(Ea, 02) (Q(22202) Vmaz) Qa Q2) a2
Q3 (I2, a2)
(26)
Denote D(12, a2) (((22, a2) Vmma)) ) Given the H 2, have Q(22, 42) r((22 02).
Therefore, we have (Taa 02) (Q(x2, 02) Vman) = Da(12, a2). Also, Qa 05) = 0. accord- ing the definition of at,
By using lemma I we have that
Da(12, = (2(2),a2) - (D)
(27)
= .%'' Dua t(a, a2)) a²)
- + Dax t(x² x2), x2) x²)
(28)
Denote A Dan (22; am - (a)))(22))a2-2)1 and A
= fo Dan (x - 1x2 - -x2), x2)
25 dtt and consider the first and second equation 26 have
(z (2) a²) Qa a2) (Q(x2, 02) - Vman)
(29)
02) a²) D (2(22,00) 0z)
(30)
= -2, a²) A A2.
(31)
We first consider the A₁ term.
Denote h1(2, a) = Q²(x, - Daa(I,
Denote B =
fo hy ( 1(a5 a2))
21

---

Published as a conference paper at ICLR 2023
22 02) (22 - az) -3)- A1
(32)
2 (22) 02) / Dua - 02)) dt
=(a2- -22)
(33)
= (a2 - (2)
Q a2) - (12 - -(a; 02)) dt
(34)
SIla2 اiلl
Q2 a2) Dua a2 t(a a2))dt
(35)
SI|o2 uill S.
(22222 02) C2 02 Mu2 - a2)) + Q (x22 - -1(2 a2)) - Daa (22, 02 I(ag - a2))dl
(36)
{
-))((aaa))ttt az)dt+ + S/¹ hi (22, - -t(ag a2))dt
$1102 - ažll
L2(1 - -
(37)
L2 B1 a2 - --il-
<I1a2
(38)
2
Now, we will consider the Ag term and the third term in equation 26 Qa Qz, Denote Bg fo ha(23 - t(z3 222) at)
Denote h2(2, a) = Dox (2, a)-
A2 Qu(22, Q2) a2)
(39)
= / Dar - t(2; 22). -22 - Q2(22. (2222
(40)
/ Dur(ze- - [(2 - 22)) 02) - =2) (2
(222 x2))
(41)
S Qo(a3- t(xz x2). Q2) (z) - a2) z3)dt
(42)
Qa(20, 02) Q0 (22 02) (r22 - x2) + / hel2z t(xb x2) -2) (43)
- =-Qo 02) a2) (2 x55)
Qa 02) 02) (a's -2))
(44)
L3L1
(I2 - xz)² + /' h2(23 t(0} - x2) 2)) (2) - zz)dt
+ 2
(45)
(I2 x2)² + / ho(23 - ((x5 - (2=22),000 02)
QQ((12 (2) (n (12, a2)
(2) 252) +
x22 dt
2
(46)
T 21
SL ||a - -aill +
2 ||a - + B2lla1 aill-
(47)
Summarize the conclusion, we can prove now
da a < ((02 - ag)² + (a2 - ag) + (a) - aa)² + (a) - aj))
(48)
The next task is to prove the af at s =0(1a; at)² + (a'z 0ÿ)² + (al aj) + (a/ aW) Similarly, we have
d - ajj = =a + dai - aj
(49)
= aa - at + Q(21, 01) Vmna
(50)
Qa
(Q8 (1),
-ai) (1)m1) Vina))
(51)
22

---

Published as a conference paper at ICLR 2023
We have Qu (x1, ai) = 0 according to the definition of ai.
Using Lemma
denote B3 = So h (z t(a) ai))dt. we have
(1(21, - aï) + Qa(211 (Q)( a1) - Vmav)
(52)
-a1 ai) I. (2( Dan (11, t(aî - a1 ai))dt
(53)
<||ai ajl 12 + Balla1 - -aill-
(54)
2
Thus, have
L2 |a - aill + B3 lal - -iil
| aill -
(55)
2 (z) (21, a1)
2
|| aللl s
L22
222 |0 - agll 2 + B₁ l02 - azll
(56)
03 (22, a2)
2
L2
BaLj (2(21,
+ 20% 01)
+ B1 - aail+ +
+ B2 ||a1 - -ail]
(57)
For simplicity, we can write
[1o4 - - & C(I1a1 aill + +|a2 az1²) + B(Iaa - ajil² + ||az oīll)
for h = 1,2. (58)
Up to here, prove the theorem is true in trajectory with horizon H 2.
Now, using induction method, suppose the theorem is true for // = - 1. The induction hypothesis
means that for the following problem (denote P(I 1)), there exist two constant C and B, such
that for vh E 21,2,...4 1}, we have ||ah az |l S C CE || - aj B B-
k=
lax 0j ||-
min Σ Ir(zk, ax)
(59)
01
Th+1 f(ok, k == 1, 1,. 1-- 2.
(60)
What we need to prove is for the new problem with H I (denote as p(l)), the theorem still holds.
The main idea is to merge the reward function in last two timesteps into one, and then prove the 8a-i is the same the one in the problem P(( 1) which is denoted da;-1. Then, according to the update rule, for < l - 1, the da, daz, also holds. For h = I, the theorem be proved using the exact the same process as we prove al - az in H = 2. Combining all these conclusions, we can then prove the theorem holds for the problem p(!) and thus the proof finished.
Here we show how can we construct a new reward function by merge two reward function. Denote Q2 (m) (af -xt) where æ Qalar.a)
R(21-1,9-1) r(#1-1. r(z,, a Q(1%. V Q.(2).a))
f(
In new problem, the update for action
sa = -1-1 K1-1821-1-
(61)
h- Q(11-1 V Vinz)
(62)
Qa(21-1, 1)
Q (x(
Ki-1 Q((11-1,01-1)
(63)
23

---

Published as a conference paper at ICLR 2023
(()))) (1 (zm, (1
(64)
Qs Qa 01)
-2((111, Ta (z), 01)
Qa (2), (1)
(65)
= =ra(01-1 Ta (2), 01) Q2(21, ai)
Qu( (10.
(66)
(0(111111-1) = (a (ft.
(67)
= (= Ta (z, 0)) Qo(2,, ai) So bat
(a() a
(68)
= =r((11-1,0-11 ra (a at) (o(2), 01)
Qu (a (1, fo 1
(69)
be easily verified, da;-1 80:-1.
F.2 PROOF OF THE COROLLARY I
If we do not consider the feedback term, K = 0. The new update rule will be
8ap kn Q(Th, un) Viax
(70)
Qa an)
(a an) = Ta (Th, an) + Va (#R+1, h + 1)fa(ann an)
(71)
(Tn, an) = Tz (1t, an) + Vg (x + 1)fa(Th))) an)
(72)
Va (I an) Qu(In,an))
(73)
According the proof of Theorem
for Vh.
H
H
Qs Qo an) Qa(an, x/ 2b
an - at C1
aap at + C2
Σ
1lax - atll +
(74)
an)
1=1
k=1
"1
Qu ax) Qu an)
<C C1
Σ
|au atll + C2
Σ
| uill
(75)
k=1
k=1
where 2/1 = f(1h111, a1 + 8ap-1), 2b f fIn-11, ap-1). Using Taylor expansion, there exist a
constant C¹' such that
1)) T/h Th =Sa (c
xn-1) Sa (1
C((ah-11 xn-1)² (ah-1 an-1)
(76)
1/h-111 - -h-1 S we can ignore the error the first-order Taylor expansion,
1/ FA
[fa(zs, a;)da; + C(ba)) Σ 1 1=h-1
(77)
And the corollary can be proved.
Lemma 1. Denote the function f(x) have continues derivative. Denote the first order derivative of function (xx as fx(x). Then we have
f(x2) f(x1) J. fa(Ø1 t(xi -2))) x ddt.
(78)
24

---

Published as a conference paper at ICLR 2023
For multi-variable function f(x, y). we have
f(z2, v2) - f(11,81)= I. fa - 1(x1 x2), - x1) + f2(22. y1 - t(yi - y2))(12 - yı dit
(79)
Proof of the Lemma it is easy to verify that
7
We first prove the single- version. Denote g(t) = f( - t(x1 12)),
f(z2) - f(x1) = g(1) (()).
(80)
According to the fundamental theorem of calculus, we have
dg(() dt
((1) .(0)
(81)
dt
(x) df(21
t(x1 - -x22)
(82)
dl
dx1 t(x1 - x22) d(I1 - t(x1 x22) dt
(83)
dr
dt
fa(x1 uxx - xx2)) - x )dl.
(84)
Then, prove the multi-variable version. Denote g(t) f(z1 t(x1 x2). 81
t(y1 y2)).
f(x2, 02) - f(11, 91)
(85)
=g(1) -g(
(86)
!'' dt(t) dt
(87)
df (x1 1(x1 - 12), 1/1 - 1ly1 122) dt
(88)
dt
df(
1(x1 - x2)) d(I1 t(x1 x2)) df f(x1 - x2))
d(y1
t(yi - V22)
dt
dr
dl
dy
dl.
(89)
('' fa +(x11 -22) x1) + fy 11 - t(y1 [22) di)dt.
(90)
Lemma 2. Denote (2r, a) = 1 (Q(x,, (0(xraa) a) Vmnax), denote (1(x,a) = Daa Daa(or, a) - (2,
h2( Dax Qa Qz a), have
h Qaa (Q(1 a) - Vmax): )
(91)
h = Qax (Q(z, a) - Vmaz))
(92)
Proof of Lemma
d²D(z, d (1()(z))) Vmaa)) d d(Qa(r, ()(((r a) - Vmaz))
Dax = dadr
(93)
dadr
dr
= (z, (I, a) + Qaz ( Vman)
(94)
Similarly, we can prove that
Dan Qua + Qua (Q(T) Vmna).
(95)
25


Reviews:
[{"review_id": "pzwa-QciDH_", "reviewer": "Reviewer_RrZx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel model-based RL algorithm for continuous control tasks that leverages a planner inspired by the Differential Dynamic Programming (DDP) approach to generate actions during environment interactions. The planner, Deep DDP or D3P, computes a locally quadratic approximation of the planning objective leveraging the Bellman equation and computes a \"delta\" action update on top of the current update by optimizing this approximation. This is combined with a term that accounts for the feedback from prior action updates to correct the current action update. Additionally, actions are initialized using a learned policy network, and the transition model, critic and reward function used for D3P are also jointly learned. The transition model and rewards are trained in a supervised manner on collected data (both from random interactions & learned policy/planner output); the policy and critic are trained on a combination of real interactions and imagined data from the model. The approach is tested on six continuous control tasks using the Mujoco simulator and shows good results compared to baselines; several ablations are also provided.", "summary_of_the_review": "Overall, this paper presents a simplification of the DDP algorithm to leverage neural networks for planning in continous control; this is combined with an RL approach for data generation and shows good performance across the tested tasks. Several strengths and weaknesses were called out; in particular additional ablations are needed to better quantify the strengths of the approach and the clarity of the text can be improved. ", "strength_and_weaknesses": "(+) The approach presents a promising algorithm for planning in continuous control spaces; the initial results in particular look promising on the tested tasks. \n\n(+) The ablations provide good insight into the properties of the proposed method. I particularly like the sweep over model quality and proposal distributions in Fig. 4\n\n(-) The clarity is a bit lacking, particularly it would help if the notation were simplified and more details are provided. Specifically details on the implementation are lacking: The approach seems to be using model ensembles, what are these used for? How are they used in the context of the planning algorithm? How are the gradients of the critic, model etc. needed for planning computed?\n\n(-) More details on the tasks should be provided. Particularly the different embodiments are listed but it is not mentioned what the actual tasks are, is it walking/running/standing in place?\n\n(-) The idea of feedback to handle the temporal nature of planning is not new. Approaches such as iLQR [1] and iLQG [2] had a feedback term. These papers should be cited and would be good to mention what the differences are compared to these -- in fact the major difference to me seems to be a simplification of the gradient computation from these methods & use of NN models. Please clarify this explicitly.\n\n(-) It is surprising (and a bit disappointing) that such a small planning horizon (H <= 4) is sufficient for the proposed approach. Does the approach work with longer horizons or does it fail due to model approximation errors? What about a simple 1-step lookahead (H=1)? An ablation of the horizon is crucial to understanding the strength of the approach.\n\n(-) Another ablation that is crucial is the choice of the planner. Fig. 3 shows an ablation replacing D3P with SGD but in general simple derivative based planners are poor choices for planning with neural networks. Is it instead possible to replace it with a simple sampling based planner, e.g. random shooting? For the short horizons (H <= 4) considered in the paper even such planners might work. It would be helpful to quantify this as currently it is not obvious if the improvement is due to the ability to do lookahead search (which can be achieved with many planners) vs the specific use of D3P. \n\n[1] Li, Weiwei, and Emanuel Todorov. \"Iterative linear quadratic regulator design for nonlinear biological movement systems.\" ICINCO (1). 2004.\n\n[2] Todorov, Emanuel, and Weiwei Li. \"A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems.\" Proceedings of the 2005, American Control Conference, 2005.. IEEE, 2005.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pwskO3w1R9", "reviewer": "Reviewer_UNLu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new algorithm called Policy Optimization with Model Planning (POMP), for continuous control problems that incorporates a model-based planner derived from differential dynamic programming (DDP). DDP cannot be practically applied directly as a planner since it has a high computational cost and requires a known model. Thus, as a practical approximation, the paper presents Deep Differentiable Dynamic Programming (D3P), and proves its convergence. Experimental results show that POMP, which contains D3P as the planner, outperforms some model-free and model-based baselines on some simulated robotics continuous control tasks.", "summary_of_the_review": "The paper presents a new model-based planner for continuous control problems that might be of interest to RL practitioners. However, some issues with clarity, and the presentation and discussion of the experimental results take away from the overall impact of this paper. Therefore, I argue to reject.\n\nThe paper could be improved by increasing the clarity throughout, and carefully expanding the discussion of the experimental results.\n", "strength_and_weaknesses": "The main strength of this paper is the introduction of a novel algorithm based on DDP for planning in continuous control problems.\n\nThe main weakness of this paper is the presentation and discussion of the experimental results. \n\nFirst, the experimental results compare the mean computed from 5 runs per algorithm. 5 samples is usually not enough to draw meaningful conclusions about the differences between algorithms. 10 runs would be better, and 30 would be ideal.\n\nThe shaded region in the plots is stated to be the standard deviation. Is this assuming the distribution of the learning curves is normal? Is this actually the case?\n\nWith only 5 runs it might actually be clearer just to plot all the runs individually instead of the mean. This would give the reader a better idea of the distribution and relative performance.\n\nIn the section “Is planning necessary to make a better decision in continuous control?” a new parameter N_p is introduced without any reference in the main text. Is this a parameter of MAAC? What does it mean? This seems to be central to understanding this section.\n\nThe section “How the learned model quality affect decision-making?” talks about how the amount of training data used to train the model affects planning utility. What is this model? How is it trained?\n\nThe text is missing some information to make Figure 4 more understandable. First plots 4.a and 4.b show “Improvements” on the y-axis. What is this? How is it computed. In 4.a what is policy quality? How is it measured? In 4.b what is model quality? How is it measured? Are they averages of something? A single sample?\n\nI think there is some potentially interesting content in the ablation experiments, but in its current form it is hard to understand. It deserves a more thorough explanation and improved clarity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3_hVG5-dZ50", "reviewer": "Reviewer_GhwF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a planner solving a locally quadratic program to improve the action selection during the rollouts. The better action selection should improve the learning performance and enable the agent to achieve a higher reward with fewer samples. The experiments show this improved sample efficiency and the agent learns faster to obtain higher rewards. \n", "summary_of_the_review": "The execution and presentation of this research project are very good. One can follow the motivation, approach, and results. Very few papers achieve this clarity. However, the paper is also very far from perfect. Missing out on the related work on zero-order MPC with learned models is quite bad. Not clearly stating the differences between the iLQR optimization and the sloppy vector math significantly reduces the quality of the paper. Further ablation study comparing the locally quadratic online optimization to zero order optimization would be great to come closer to answering gradient based vs sample-based MPC with learned models question. Therefore, the details of the work can be significantly improved and make this paper from okayish to excellent. ", "strength_and_weaknesses": "Thanks for this enjoyable review. The paper is very very well written. One can easily follow the paper and understand the bigger picture. Such a clear presentation of the ideas and results is very rare. Furthermore, the experimental evaluation is great. The paper first shows the performance in terms of rewards and conducts ablation studies answering specific questions. From a research project execution point of view, the paper is an A+. The research idea/question is ok but not super innovative, which is fine. Therefore, the paper is a very good contribution to ICLR. However, there are also quite some shortcomings, which prevent this paper from being an excellent paper instead of \"only\" a good paper. \n\nThe paper unfortunately misses a large part of the model-based RL literature. Many papers have already used an MPC approach that optimizes an action sequence WITH A LEARNED MODEL online to obtain better actions. There are initial versions that do not rely on a policy, e.g. Pets (Chua et. al.), PDDM (Nagabandi et. al), iCEM (Pinneri et. al.), and later versions that distill the samples into a policy using any deep RL algorithm and use the learned policy as a prior e.g., EVALUATING MODEL-BASED PLANNING AND PLANNER AMORTIZATION FOR CONTINUOUS CONTROL (Byravan et. al.). These are just a few exemplary papers and there are 10-30 papers on MPC with learned models from various groups. Somehow, this paper forgets to address this line of research.\n\nOne interesting point of all of these approaches is that they rely on zero-order optimization to solve the local optimization problem. A frequently discussed question was; Why is everybody using in-efficient zero-order optimization methods? Wouldn't gradient-based approaches be more efficient? One common hypothesis was, that deep networks are good for forward prediction but the gradients are not great. Therefore, it is believed the backprop through time literature never took off in deep RL. However, this paper uses a gradient-based optimization to solve the MPC problem. It also performs an ablation study of the locally quadratic problem to the naive SGD backprop to time. So it is very close to giving more insights into this question. Therefore, it would be great to perform more ablation studies on whether the locally quadratic approach performs better than the zero-order online optimization with learned models and include this line of research within the paper. \n\nFurther minor points: \n\n* `In most of the model-based RL algorithms, the learned predictive models always play an auxiliary role to only affect the decision-making by helping the policy learning.`\nGiven that there are many papers doing MPC with learned models/rewards, this statement is wrong. \n\n * `differential dynamic programming (DDP) (DE O. PANTOJA, 1988; Tassa et al., 2012) algorithm` The capitalization of the citation is wrong.\n\n*  `optimal bellman equation`  Bellman has to be capitalized.\n\n* Equation 4 and the following are sloppy. The equations treat vectors as scalars and easily divide through vectors, i.e., Q_a, Q_x. This section should be rewritten to properly treat vectors and remove the ambiguity of what these equations mean. \n\n* The locally quadratic DDP optimization looks very close to identical to iLQR from Tassa et. al.. The paper needs to add either a section to highlight the differences to the iLQR optimization approach. When the only difference is the learned deep dynamics models, the paper needs to explicitly state: \"We use iLQR with learned models for MPC\" and not slap a new name on an existing technique, i.e., Deep Differential Dynamic Programming (D3P).\n\n*  `Please note that we only use this conservative term during evaluation, as we want to encourage exploration when training`\nDoes this statement implicitly mean, if we include the conservative term in the exploration it is too little exploration and our learning curve is below the baselines? If you use the greedy algorithm in your evaluation to plot the learning curve, do you also use the deterministic policies of your baselines during the evaluation? \n\n* The MBPO state augmentation to push sample efficiency is never really mentioned in the main paper but only shown in algorithm 2 in the appendix. It would be great to make this more explicit in the main paper. Furthermore, it would be great to perform an ablation study without this sample augmentation to see how the pure DDP MPC increases sample efficiency.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2GloRwoQOIE", "reviewer": "Reviewer_Pekh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new model-based RL framework for continuous control in deterministic environments. The authors propose Policy Optimization with Model Planning (POMP) an algorithm that uses Differential Dynamic Programming (DDP) as the planner module. A specific instantiation of DDP is introduced which uses (D3P) which uses neural networks as differentiable models and first order taylor approximation to improve the computational efficiency of the planner. The method is shown to converge to the optimal sequence of actions, given a fixed learned model. Extensive experimental evaluations are performed, together with an ablation study that show the effectiveness of the proposed method compared with  state-of-the-art methods", "summary_of_the_review": "Overall I am positive towards the paper. The author propose a novel original method, that works well in practice and that has convergence guarantees under some mild assumptions. It is unclear though how the proposed method performs compared to MuZero for continuous actions, which is an important baseline for the setting, indeed both methods were cited repeatedly in the paper. \nI would increase my score if this comparison is performed, or alternatively if a valid reason for omitting this comparison is given.", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written and easy to follow\n- A theoretical analysis of the proposed method is shown, with relative convergence proof\n- An extensive empirical evaluation is performed showing the effectiveness of the method\n\nWeaknesses:\n- The authors should better express the limitations of the method. They should clearly state that the method is meant to be used in deterministic environments as this is heavily used in the definition of their planner.\n- They should further clarify that the convergence proof applied only to the planner with a fixed (probably wrong) learned transition model and reward, i.e. D3P recovers the optimal action of a wrong model. It should be made clear that no convergence guarantee is given on the whole algorithm, i.e. no guarantees on the performance of the learned policy compared to the optimal policy of the MDP\n- The empirical evaluation is thorough but important baselines are missing in my opinion. I would have expected to see a comparison with MuZero combined with Continuous  UCT (Progresstive Widening). This because the model and reward learning procedure of POMP is very similar to the one of MuZero. Moreover, continuous UCT, like D3P is known to converge to the optimal sequence of actions given a fixed environment model. With the current empirical evaluation, It is unclear if POMP would outperform MuZero + Continuous UCT, which is a fairly important baseline for model-based RL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster