Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a methodology that improves exploration for uncertainty-aware dynamics models using probabilistic ensembles of neural networks in CEM-MPC planning (zero-order trajectory optimization approach) by providing an explicit separation between aleatoric and epistemic uncertainty of the learned dynamics model. The approach follows closely the methodology of previous work, PETS [6], yet improves upon it by  (i) adding an autoregressive component in the dynamics models for forward prediction, (ii) introducing an entropy-driven approach to measure epistemic and aleatoric uncertainty in the ensemble of neural nets, (iii) uses the estimated aleatoric uncertainty to penalize an MPC cost function while adding epistemic bonuses estimated from the epistemic uncertainty. Further, due to the chosen zero-order optimization technique, safety constraints can also be considered by formulating them as probabilities of safety constraints violations and adding them as penalty to the cost function. The resulting algorithm, RAZER, is compared to PETS in 4 continuous state-action space experiments in simulated environments of increasing complexity. Results clearly show that the proposed methodological improvements (RAZER) outperform PETS in terms of exploration/state-space coverage, uncovering safe solutions in regions with little training data guided by the epistemic bonus and safety constraints.
strengths_and_weaknesses: Strengths: 

-Principled methodology to uncover both epistemic and aleatoric uncertainties from learning dynamics models. While this is currently applied to zero-order trajectory optimization problems, such explicit separation can be used in other tasks involving predictions of uncertain dynamics systems.

-Consideration of safety constraints as a form of uncertainty in the cost function is a clever way of dealing with constraint satisfaction without explicitly having to guarantee constraints in the MPC optimizer.

-While the modifications to PETS might seem incremental, I believe the results are compelling as they show clearly improved exploration and state-space coverage over baseline.

Weaknesses:

-PETS uses the learned models within an CEM-MPC planner too. Here, it is claimed that an efficient implementation of an CEM-MPC planner from [31] was used. What is the main advantage/difference? Where is this efficiency reflected in the evaluation of results? Is [31] just a faster implementation of CEM-MPC? If so, how fast? Is PETS being implemented with this CEM-MPC planner too or with the original version? Analysis of such computational complexity is missing.

-While the inclusion of safety constraints as penalties in the cost is a nice trick to include safety in planning the type of constraints allowable, basically bounds on the states [a,b] where a<b might be too restrictive. Is there a way to include other types of constraints in the approach, such as polytopic constraints, which are often found in robotics problems involving contacts and safety? 

-Including the constraints as cost in the CEM-MPC is not the only way to guarantee the satisfaction of safety constraints. There are other approaches that have done this, see relevant paper “Constrained Cross-Entropy Method for Safe Reinforcement Learning” from Wen and Topcu. It could be beneficial to discuss the differences with newer constrained-based approaches and how the current methodology could be extended to adopt them.

-Could you provide an intuition or rule-of-thumb as to how to define and leverage the weights in the cost function for the different penalties/bonuses? For example, what’s the resulting behavior if both aleatoric penalty and epistemic bonus are high and safety constraint is low, or vice-versa or any combination of the three. In the current submission, we are only given analyzes of either the change in weights of the aleatoric penalty or epimestic bonus or increase in safety constraint probability threshold, independently.

-As mentioned in the text, satisfying constraints reduces tracking accuracy, how can this be alleviated? 

summary_of_recommendation: This paper presents interesting insights into the modeling of different types of uncertainties with neural networks and their use for trajectory-optimization. The work builds upon a previously introduced framework, yet the proposed improvements seem to provide compelling results, specifically in the improvements of state-space exploration while remaining within safe bounds. Further analysis of specific claims and decision choices can be beneficial (see weaknesses).
issues: See Weaknesses.
 

Minor Errors:

-Page 5, Line 194 “we define the an aleatoric”

-Page 6, line 235 “risk-avers planning”

-Page 7, line 243 “the somewhat save lower”->safe

-Page 7, Line 271 “from scratch”?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for separating aleatoric and epistemic uncertainties in learned dynamic models of noisy robotic systems. The paper also proposes a method for using these learned models in risk-averse model-based planning. The methods are demonstrated on a variety of simulation examples, and outperform the existing PETS method.
strengths_and_weaknesses: STRENGTHS
- The main strength of this paper is Sec. 3.4: the separation of aleatoric and epistemic uncertainties from noisy data.
- The experimental results seem convincing that the method works.

WEAKNESSES
- I think the idea of a 1-step Gaussian nonlinear dynamics model is really odd; it seems to totally ignore the fact that we can and should model the parts of the dynamics that we actually know. I guess the method works, but it would be great to have a much clearer justification of why this Gaussian ensemble is a good model.
- The experiment section is written in a very confusing way. It's unclear that the experiments are thorough (e.g., when RAZER has a 96% success rate, I have no idea how many trials were conducted), though some information is related through the x-axis labels of the figures.
summary_of_recommendation: I think there is a lot of implementation information missing from this paper that would make it much easier to assess whether or not the method is practical and interesting; basically, the paper is pretty unclear about what it does. I think that, if the authors are able to clarify my concerns in the "weaknesses" section above, and the "issues" section below, then I'd be happy to accept this paper.

Final review/summary: The authors have addressed my concerns and the paper seems stronger now. I already felt "accept" was in order, so I think my current recommendation stands.
issues: I've included both positive notes that I took while reading as well as issues that I think could be improved.

Abstract
- short and clear, I like it

Introduction and Related Work
- Off the top of my head, I don't know what "zero-order" MPC is. To me, "first-order" means gradient descent for trajectory optimization, and "second-order" means also using a Hessian; so zero-order must mean a sampling-based method with no gradient information? Might be helpful to clarify this for less familiar readers.
- I'm very curious to see how this work performs uncertainty propagation.

Method
- I'm curious if this paper will make any statements about how much data is needed to fit $f$.
- The hypothesis that "accounting for uncertainty... is essential to develop risk-averse policies" feels tautological. Risk-aversion IS the process of accounting for uncertainty. That is, this statement strikes me as more like a definition than a hypothesis.
- The discussion of picking apart aleatoric and epistemic uncertainty is really interesting.
- The notation $\mathbb{X}$ has not been defined -- presumably this is supposed to be the whole state space.
- The simplification to enable computation seems drastic, since it's a subset of $\mathbb{R}^2$! The state can only be 2-D? That seems to contradict the Fetch arm and HalfCheetah examples in Fig. 1. I think the unsafe set needs to be described much more clearly; I guess this is supposed to mean that there can be many unsafe sets, and each one is a 2-D box? It would be clearer to just say the constraint set is (a union of) compact box/es in the state space.
- The fun acronyms for the proposed method (PETSUS,RAZER) should be stated in the introduction to prime the reader for these terms later on.

Experiments
- The first mention of PETS in this section should include a citation
- I guess the epistemic uncertainty cost weight in (10) is on some sort of schedule, starting out high and decreasing over time? If this isn't the case, it would be good to explain how and why RAZER transitions from exploration to exploitation.
- How long did training take? How many times was each method run in each environment? Is the proposed method able to run both training and testing in real time?
- The descriptions in Sec. 4.2 are a bit incoherent (some run-on sentences, unclear explanations of results).
- The figures are much more clear than the text, though!

Discussion
- It would be good to title this section as "Conclusion"
- This section is very clear and highlights the contribution of the proposed method!
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is proposing a risk-aware CEM-based MPC method. The main machinery is extending the PETS model with direct consideration of the uncertainty terms (both epistemic and aleatoric) in the loss function. Although the proposed method is general in a sense it can be applied to an arbitrary zeroth-order optimizer, only CEM is considered in the experiments.
strengths_and_weaknesses: The proposed method is technically sound. The consideration of uncertainties is critical for learning and authors do a good job in defining technically sensible formulations of uncertainty.

The application area of safety-critical MPC is a great match to the venue and also an important problem.

The Paper is well-written with some minor issues.

One major issue I see is the lack of ablation studies. It is not clear to me whether the separation of different uncertainties actually helps or not. Somewhat related, there are lots of penalty weights that need to be defined as hyper-parameters and they seem to impact the performance significantly. I believe this would warrant a discussion on how these should be chosen as well as an empirical analysis of the sensitivity to these penalty weights.

Minor but important: I think the notation is not entirely consistent. Frak and Cal fonts are used in a rather inconsistent manner. I don't know what frak font represents in this paper.
summary_of_recommendation: The paper is proposing an interesting algorithm and it is technically sound. I think it is a rather straightforward extension of PETS but it is an important and required extension. My only concern is the lack of ablation study and sensitivity analysis for penalty terms.  

----

I thank the authors for their rebuttal. It answers my concerns. I believe the paper is definitely interesting and can be accepted. I will keep the weak accept score
issues: - Ablation study for penalty weights

- Sensitivity analysis for penalty weights

- More consistent notation
review_title: null

Meta-review: This paper proposes an approach for incorporating risk in zero-order (i.e., derivative-free) trajectory optimization. In particular, the method uses an ensemble of stochastic neural networks in order to estimate aleatoric and epistemic uncertainty in the robot dynamics model. The approach seeks to be optimistic in the face of epistemic uncertainty and pessimistic in the face of aleatoric uncertainty. The resulting model predictive control scheme is demonstrated on different robot control examples. 

Strengths:
+ The reviewers agree that the key idea of separating aleatoric uncertainty and epistemic uncertainty and dealing with these in different ways is a promising one. 
+ The reviewers generally agree that the experimental evaluation in the paper is thorough and convincing (with one point of concern noted below).
+ The reviewers agree that the paper is well written and clear overall. 

Weaknesses:
- The reviewers highlighted the fact that hyperparameters in the approach may not be entirely straightforward to choose and that the paper should provide rules of thumb on how to choose them. In addition, further details on the experimental evaluation would be helpful (see Reviewer pSNT and RZzg’s comments). 
- One concern with the experimental evaluation is the lack of thorough ablation studies. Given that the primary hypothesis of the paper is that epistemic and aleatoric uncertainty should be separately dealt with, the experimental evaluation should thoroughly demonstrate this via ablation studies. 

Suggestions:
I urge the authors to consider the reviewers’ feedback  and the primary points highlighted above. The main suggestions are to: (i) add further details on the approach, hyperparameters, and experimental evaluation (in the places suggested by the reviewers), and (ii) address the point regarding ablations.

----- Post rebuttal -----

In the rebuttal period, the authors have addressed the primary concerns reviewers had raised by (i) adding more details on algorithmic choices, and (ii) clarifying the role of taking into account epistemic and aleatoric uncertainty. The reviewers are now in agreement that the paper makes a good contribution and should be accepted.

Decision: Accept (Poster)