Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel model-based RL algorithm for continuous control tasks that leverages a planner inspired by the Differential Dynamic Programming (DDP) approach to generate actions during environment interactions. The planner, Deep DDP or D3P, computes a locally quadratic approximation of the planning objective leveraging the Bellman equation and computes a "delta" action update on top of the current update by optimizing this approximation. This is combined with a term that accounts for the feedback from prior action updates to correct the current action update. Additionally, actions are initialized using a learned policy network, and the transition model, critic and reward function used for D3P are also jointly learned. The transition model and rewards are trained in a supervised manner on collected data (both from random interactions & learned policy/planner output); the policy and critic are trained on a combination of real interactions and imagined data from the model. The approach is tested on six continuous control tasks using the Mujoco simulator and shows good results compared to baselines; several ablations are also provided.
strength_and_weaknesses: (+) The approach presents a promising algorithm for planning in continuous control spaces; the initial results in particular look promising on the tested tasks. 

(+) The ablations provide good insight into the properties of the proposed method. I particularly like the sweep over model quality and proposal distributions in Fig. 4

(-) The clarity is a bit lacking, particularly it would help if the notation were simplified and more details are provided. Specifically details on the implementation are lacking: The approach seems to be using model ensembles, what are these used for? How are they used in the context of the planning algorithm? How are the gradients of the critic, model etc. needed for planning computed?

(-) More details on the tasks should be provided. Particularly the different embodiments are listed but it is not mentioned what the actual tasks are, is it walking/running/standing in place?

(-) The idea of feedback to handle the temporal nature of planning is not new. Approaches such as iLQR [1] and iLQG [2] had a feedback term. These papers should be cited and would be good to mention what the differences are compared to these -- in fact the major difference to me seems to be a simplification of the gradient computation from these methods & use of NN models. Please clarify this explicitly.

(-) It is surprising (and a bit disappointing) that such a small planning horizon (H <= 4) is sufficient for the proposed approach. Does the approach work with longer horizons or does it fail due to model approximation errors? What about a simple 1-step lookahead (H=1)? An ablation of the horizon is crucial to understanding the strength of the approach.

(-) Another ablation that is crucial is the choice of the planner. Fig. 3 shows an ablation replacing D3P with SGD but in general simple derivative based planners are poor choices for planning with neural networks. Is it instead possible to replace it with a simple sampling based planner, e.g. random shooting? For the short horizons (H <= 4) considered in the paper even such planners might work. It would be helpful to quantify this as currently it is not obvious if the improvement is due to the ability to do lookahead search (which can be achieved with many planners) vs the specific use of D3P. 

[1] Li, Weiwei, and Emanuel Todorov. "Iterative linear quadratic regulator design for nonlinear biological movement systems." ICINCO (1). 2004.

[2] Todorov, Emanuel, and Weiwei Li. "A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems." Proceedings of the 2005, American Control Conference, 2005.. IEEE, 2005.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and the experiments and ablations are nicely done. As mentioned above, it would help clarity if the notation for the theoretical part is improved and additional details regarding the implementation, tasks etc are provided. Two additional relevant citations; [1] - an application of MCTS style planning to continuous control tasks, and [2] a related paper that uses sampling based MPC for high-dimensional continuous control tasks with learned models and a learned policy as a proposal distribution. 

Quality & Novelty: The approach is built on top of prior work and specifically adds a Deep-DDP planner on top of existing work to showcase the strengths of model-based planning. Some of the presented contributions have been shown in prior work (iLQR/iLQG) as well and in some sense the approach seems to be a simplification of such methods to leverage NN based models. The presented results look quite promising and the approach can be quite useful if the strengths can be better quantified and the novelty made clear in the context of prior work.

Reproducibility: A link to an implementation on github is provided with the submission which helps with reproducibility.

[1] Springenberg, Jost Tobias, et al. "Local search for policy iteration in continuous control." arXiv preprint arXiv:2010.05545 (2020).

[2] Byravan, Arunkumar, et al. "Evaluating model-based planning and planner amortization for continuous control." arXiv preprint arXiv:2110.03363 (2021).
summary_of_the_review: Overall, this paper presents a simplification of the DDP algorithm to leverage neural networks for planning in continous control; this is combined with an RL approach for data generation and shows good performance across the tested tasks. Several strengths and weaknesses were called out; in particular additional ablations are needed to better quantify the strengths of the approach and the clarity of the text can be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new algorithm called Policy Optimization with Model Planning (POMP), for continuous control problems that incorporates a model-based planner derived from differential dynamic programming (DDP). DDP cannot be practically applied directly as a planner since it has a high computational cost and requires a known model. Thus, as a practical approximation, the paper presents Deep Differentiable Dynamic Programming (D3P), and proves its convergence. Experimental results show that POMP, which contains D3P as the planner, outperforms some model-free and model-based baselines on some simulated robotics continuous control tasks.
strength_and_weaknesses: The main strength of this paper is the introduction of a novel algorithm based on DDP for planning in continuous control problems.

The main weakness of this paper is the presentation and discussion of the experimental results. 

First, the experimental results compare the mean computed from 5 runs per algorithm. 5 samples is usually not enough to draw meaningful conclusions about the differences between algorithms. 10 runs would be better, and 30 would be ideal.

The shaded region in the plots is stated to be the standard deviation. Is this assuming the distribution of the learning curves is normal? Is this actually the case?

With only 5 runs it might actually be clearer just to plot all the runs individually instead of the mean. This would give the reader a better idea of the distribution and relative performance.

In the section “Is planning necessary to make a better decision in continuous control?” a new parameter N_p is introduced without any reference in the main text. Is this a parameter of MAAC? What does it mean? This seems to be central to understanding this section.

The section “How the learned model quality affect decision-making?” talks about how the amount of training data used to train the model affects planning utility. What is this model? How is it trained?

The text is missing some information to make Figure 4 more understandable. First plots 4.a and 4.b show “Improvements” on the y-axis. What is this? How is it computed. In 4.a what is policy quality? How is it measured? In 4.b what is model quality? How is it measured? Are they averages of something? A single sample?

I think there is some potentially interesting content in the ablation experiments, but in its current form it is hard to understand. It deserves a more thorough explanation and improved clarity.

clarity,_quality,_novelty_and_reproducibility: The text is generally readable, but there are some issues with clarity that are distracting.

For example, the abstract states that one of the problems with planning in continuous control is that there is a “temporal dependency between actions in different timesteps.” It is not clear to me what this means or why it is true. This could be clarified. And later the text states that “the temporal dependency between actions implies that the action update in previous timesteps can influence the later actions.” I think this might be saying the same thing, but it needs some clarification as well.

Some descriptions of how the algorithm components fit together are hard to parse. For example, in the introduction, to help with potentially poor initializations of the planning process, the paper proposes to “to leverage the learned policy to provide the initialization of the action before planning and provide a conservative term at the planning to admit the conservation principle, in order to keep the small error of the learned model along the planning process.” I don’t understand what this is trying to say.

A description of the POMP algorithm is missing from the main text. How is this system actually used to help select actions at runtime? This is not clear to me from the text.

(Also see weaknesses above.)

The work presented appears novel.

The results in the paper could likely be reproduced using all of the information provided in the appendix, but I’m not sure if enough details exist in the main text for the results to be reproducible.

summary_of_the_review: The paper presents a new model-based planner for continuous control problems that might be of interest to RL practitioners. However, some issues with clarity, and the presentation and discussion of the experimental results take away from the overall impact of this paper. Therefore, I argue to reject.

The paper could be improved by increasing the clarity throughout, and carefully expanding the discussion of the experimental results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to use a planner solving a locally quadratic program to improve the action selection during the rollouts. The better action selection should improve the learning performance and enable the agent to achieve a higher reward with fewer samples. The experiments show this improved sample efficiency and the agent learns faster to obtain higher rewards. 

strength_and_weaknesses: Thanks for this enjoyable review. The paper is very very well written. One can easily follow the paper and understand the bigger picture. Such a clear presentation of the ideas and results is very rare. Furthermore, the experimental evaluation is great. The paper first shows the performance in terms of rewards and conducts ablation studies answering specific questions. From a research project execution point of view, the paper is an A+. The research idea/question is ok but not super innovative, which is fine. Therefore, the paper is a very good contribution to ICLR. However, there are also quite some shortcomings, which prevent this paper from being an excellent paper instead of "only" a good paper. 

The paper unfortunately misses a large part of the model-based RL literature. Many papers have already used an MPC approach that optimizes an action sequence WITH A LEARNED MODEL online to obtain better actions. There are initial versions that do not rely on a policy, e.g. Pets (Chua et. al.), PDDM (Nagabandi et. al), iCEM (Pinneri et. al.), and later versions that distill the samples into a policy using any deep RL algorithm and use the learned policy as a prior e.g., EVALUATING MODEL-BASED PLANNING AND PLANNER AMORTIZATION FOR CONTINUOUS CONTROL (Byravan et. al.). These are just a few exemplary papers and there are 10-30 papers on MPC with learned models from various groups. Somehow, this paper forgets to address this line of research.

One interesting point of all of these approaches is that they rely on zero-order optimization to solve the local optimization problem. A frequently discussed question was; Why is everybody using in-efficient zero-order optimization methods? Wouldn't gradient-based approaches be more efficient? One common hypothesis was, that deep networks are good for forward prediction but the gradients are not great. Therefore, it is believed the backprop through time literature never took off in deep RL. However, this paper uses a gradient-based optimization to solve the MPC problem. It also performs an ablation study of the locally quadratic problem to the naive SGD backprop to time. So it is very close to giving more insights into this question. Therefore, it would be great to perform more ablation studies on whether the locally quadratic approach performs better than the zero-order online optimization with learned models and include this line of research within the paper. 

Further minor points: 

* `In most of the model-based RL algorithms, the learned predictive models always play an auxiliary role to only affect the decision-making by helping the policy learning.`
Given that there are many papers doing MPC with learned models/rewards, this statement is wrong. 

 * `differential dynamic programming (DDP) (DE O. PANTOJA, 1988; Tassa et al., 2012) algorithm` The capitalization of the citation is wrong.

*  `optimal bellman equation`  Bellman has to be capitalized.

* Equation 4 and the following are sloppy. The equations treat vectors as scalars and easily divide through vectors, i.e., Q_a, Q_x. This section should be rewritten to properly treat vectors and remove the ambiguity of what these equations mean. 

* The locally quadratic DDP optimization looks very close to identical to iLQR from Tassa et. al.. The paper needs to add either a section to highlight the differences to the iLQR optimization approach. When the only difference is the learned deep dynamics models, the paper needs to explicitly state: "We use iLQR with learned models for MPC" and not slap a new name on an existing technique, i.e., Deep Differential Dynamic Programming (D3P).

*  `Please note that we only use this conservative term during evaluation, as we want to encourage exploration when training`
Does this statement implicitly mean, if we include the conservative term in the exploration it is too little exploration and our learning curve is below the baselines? If you use the greedy algorithm in your evaluation to plot the learning curve, do you also use the deterministic policies of your baselines during the evaluation? 

* The MBPO state augmentation to push sample efficiency is never really mentioned in the main paper but only shown in algorithm 2 in the appendix. It would be great to make this more explicit in the main paper. Furthermore, it would be great to perform an ablation study without this sample augmentation to see how the pure DDP MPC increases sample efficiency.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: The execution and presentation of this research project are very good. One can follow the motivation, approach, and results. Very few papers achieve this clarity. However, the paper is also very far from perfect. Missing out on the related work on zero-order MPC with learned models is quite bad. Not clearly stating the differences between the iLQR optimization and the sloppy vector math significantly reduces the quality of the paper. Further ablation study comparing the locally quadratic online optimization to zero order optimization would be great to come closer to answering gradient based vs sample-based MPC with learned models question. Therefore, the details of the work can be significantly improved and make this paper from okayish to excellent.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new model-based RL framework for continuous control in deterministic environments. The authors propose Policy Optimization with Model Planning (POMP) an algorithm that uses Differential Dynamic Programming (DDP) as the planner module. A specific instantiation of DDP is introduced which uses (D3P) which uses neural networks as differentiable models and first order taylor approximation to improve the computational efficiency of the planner. The method is shown to converge to the optimal sequence of actions, given a fixed learned model. Extensive experimental evaluations are performed, together with an ablation study that show the effectiveness of the proposed method compared with  state-of-the-art methods
strength_and_weaknesses: Strengths:
- The paper is clearly written and easy to follow
- A theoretical analysis of the proposed method is shown, with relative convergence proof
- An extensive empirical evaluation is performed showing the effectiveness of the method

Weaknesses:
- The authors should better express the limitations of the method. They should clearly state that the method is meant to be used in deterministic environments as this is heavily used in the definition of their planner.
- They should further clarify that the convergence proof applied only to the planner with a fixed (probably wrong) learned transition model and reward, i.e. D3P recovers the optimal action of a wrong model. It should be made clear that no convergence guarantee is given on the whole algorithm, i.e. no guarantees on the performance of the learned policy compared to the optimal policy of the MDP
- The empirical evaluation is thorough but important baselines are missing in my opinion. I would have expected to see a comparison with MuZero combined with Continuous  UCT (Progresstive Widening). This because the model and reward learning procedure of POMP is very similar to the one of MuZero. Moreover, continuous UCT, like D3P is known to converge to the optimal sequence of actions given a fixed environment model. With the current empirical evaluation, It is unclear if POMP would outperform MuZero + Continuous UCT, which is a fairly important baseline for model-based RL.
clarity,_quality,_novelty_and_reproducibility: Quality:
The paper presents a novel model-based RL method, with some analysis on the convergence guarantees of one of their components
Clarity
The paper is clearly written and easy to follow
Originality:
The paper present a novel and original method
Reproducibility:
The method is straightforward and easy to implement, the authors provide pseudocode, training and architectural details as well as the source code. Reproducibility should not be an issue even though I did not run the source code myself.
summary_of_the_review: Overall I am positive towards the paper. The author propose a novel original method, that works well in practice and that has convergence guarantees under some mild assumptions. It is unclear though how the proposed method performs compared to MuZero for continuous actions, which is an important baseline for the setting, indeed both methods were cited repeatedly in the paper. 
I would increase my score if this comparison is performed, or alternatively if a valid reason for omitting this comparison is given.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a gradient-based planning algorithm, D3P, designed specifically to be used in the context of deep learning for continuous control. D3P is leveraged in a deep learning agent called POMP and shows improvements over several baselines on Mujoco continuous control tasks.

The reviewers cited several strengths of the paper, including the algorithm being promising and the experimental evaluation being thorough. In the original submission, the reviewers flagged some issues with clarity, lack of comparisons to sample-based baselines like Random Shooting/CEM or search baselines like MCTS, some missing discussion of prior literature, and a number of other smaller issues. The authors provided a very thorough rebuttal including all requested baselines as well as additional discussion and clarifications in the paper. The reviewers were overall satisfied with these revisions and ultimately found this to be a strong paper worth presenting at ICLR. I therefore recommend acceptance.
justification_for_why_not_higher_score: The reviewers felt the approach, while solid and well-evaluated, was not particularly innovative. One of the reviewers also felt there remained some clarity issues even after the rebuttal. 
justification_for_why_not_lower_score: The reviewers all agreed that this was a good paper with an interesting method and impressive results.
Decision: Accept: poster