Risk-Averse Zero-Order Trajectory Optimization
Authors: Marin Vlastelica
Sebastian Blaes
Cristina Pinneri
Georg Martius
Venue: CoRL2021 Poster (2021)
Keywords: CEM
data-driven MPC
uncertainty
model-based RL

Abstract: We introduce a simple but effective method for managing risk in zero-order trajectory optimization that involves probabilistic safety constraints and balancing of optimism in the face of epistemic uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble of stochastic neural networks. Various experiments indicate that the separation of uncertainties is essential to performing well with data-driven MPC approaches in uncertain and safety-critical control environments.

Risk-Averse Zero-Order Trajectory Optimization
Marin Vlastelica's, Sebastian Blaes",, Cristina Pinneri and Georg Martius'
Autonomous Learning Group, Max Planck Institute for Intelligent Systems, Tübingen, Germany 2 Department of Computer Science, ETH Zurich and Max Planck ETH Center for Learning Systems (mvlastelica, sblaes, epinneri, gmartius y @ tuubingen. mpg de
We introduce a simple but effecttve method for managing risk in zero- order trajectory optimization that involves probabilistic safety constraints and balancing of optimism in the face of epistemic uncertainty and pessimism in the face of aleatoric uncertainty of an ensemble of stochastic neural networks. We empirically validate that the separation of uncertainties is essential to performing well with data-driven MPC approaches in uncertain and safety-critical control environments.
Abstract:
Keywords:
zero-order trajectory optimization, CEM, MPC, aleatoric, epistemic, safety, uncertainty, model learning
Introduction
Data-driven approaches to sequential decision-mann maasionn-maingggg are becoming increasingly popular [1, 2, 3, 4]. They hold the promise of reducing the number of prior assumptions about the system that are imposed by traditional approaches that are based on nominal models.
Such approaches come in several different flavors [5]. Model-free approaches attempt to extract closed-loop control policies directly from data, while model--aseed approaches rely on learned model of the dynamics to either generate novel data to extract a policy or to be used in a model-predictin control fashion (MPC). This work belongs to the latter line of work.
Model-based methods have several advantages over pure model-free approaches. Firstly, humans tend to have better intuition how incorporate prior knowledge into model rather than into policy or value function. Secondly, most model-free policies are bounded to a specific task, while models are task-agnostic and can be applied for optimizing arbitrary cost functions, given sufficient exploration.
Nevertheless, learning models for control comes with certain caveats. Traditional MPC methods require the model and cost function to permit a closed-form solution which restricts the function class prohibitively. Alternatively, gradient-based based iterative optimization can be employed, which allows for a larger class of functions but typically fails to yield satisfactory solutions for complicated function approximators such as deep neural network models. In addition, calculating first-order or even second-order information for trajectory optimization tends to be computationally costly, which makes hard to meet the time constraints of real-world settings. This motivates the usage of zero-order methods, i.c gradient-free or sample-basee based, such as the Cross-entropy Method (CEM) that do not rely on gradient information but are efficiently parallelizable.
Many methods relying on a learned model and zero-order trajectory optimizers have been proposed [6, 7, 8). but all share the same problem: compounding of errors through auto-regressive model prediction. This naturally brings us to the question of how can we effectively manage model errors and uncertainty to be more data-efficient and safe. Arguably, this is of the main obstacles to applying data-driven model-based methods to the real world, e.g. to robotics settings.
In this work, we introduce risk-averse zero-order trajectory optimization method (RAZER) for managing errors and uncertainty in zero-order MPC and test it on challenging scenarios (Fig. 1). We argue that it is essential to differentiate between the two types of uncertainty in the model-predictis setting: the alcatoric uncertainty arising from inherent noise in the system and epistemic uncertainty arising from the lack of knowledge [9, 10]. We measure these uncertainties by making of probabilistic ensembles with trajectory sampling similar to PETS [6]. Our contributions can be
"equal contribution
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

RAZER
(a) Noisy-Feree FetchPickAnnPllcee
(b) Solo8- LeanOvvrObjeee
(c) BridgeMaze
Figure I: Environments considered for uncertainty-awart planning. Code and videos are available at
https: //marmaaiisss//e lab. ..b.gitiin github......... ioiii////o L/RRAER///////
summarized as follows: (i) method for separation of uncertainties in probabilistic ensembles (termed PETSUS); (ii) efficient use of aleatoric and epistemic uncertainty in model- zero-order trajectory optimizers; (iii) simple but practical approach to probabilistic safety constraints in zero-order MPC.
Related Work
Uncertainty Estimation. In the typical model-based reinforcement learning (MBRL) setting, the true transition dynamics function is modeled through approximator. Impressive results have been achieved by both parametric models [11, 12, 13, 14], such as neural networks, and nonparametric models [15, 16, 17, 18), such Gaussian Processes (GP). The latter inspired seminal work the incorporation of the dynamics model's s uncertainty for long-term planning [18, 19]. However, their usability is limited to low-data, low-dimensional regimes with smooth dynamics [20, 21], which is not ideal for robotics applications. Alternative parametric approaches include ensembling of deep neural networks, used both in the MBRL community [6, 22], and outside [23, 24]. In particular, ensembles of probabilistic neural networks established state-of the---rtttttttte results [6]. but focus mainly on estimating the expected cost and disregard the underlying uncertainties. In comparison, we propose treatment of the resulting uncertainties of the ensemble model.
Zero-order MPC. The learned model can be used for policy search like in PILCO [25, 18, 19, 26] or for online model-predictive control (MPC) [27, 28, 6]. In this work, we do planning in an MPC fashion and employ zero-order method trajectory optimizer, since they have shown be less likely to get stuck in local minima and make an explicit treatment of the uncertainty in the cost possible. Specifically, consider sample-efficient implementation of the Cross-Entropy method (29, 30] introduced in [31].
Safe MPC. Separating the sources of uncertainty is of particular importance for applications directly affecting humans' safcty, as self-driving cars, elderly care systems, or in general any application that involves physical interaction between agents and humans. Disentangling epistemic from alcatoric uncertainty allows for separate optimization objectives per definition. Extensive research uncertainty decomposition has been done in the Bayesian setting and the context of safe policy search [32, 33, 34, 35], MPC planning [36, 37, 38], and distributional RL [39, 40]. On the other hand, state-of- soffthe-art baseline for ensemble learning like PETS [6], despite estimating uncertainty, only optimizes for the expected cost during action evaluation. Our work aims filling this gap by explicitly integrating the propagated uncertainty information the zero-order MPC planner.
the two, as they represent semantically different
3 Method
Our approach concerns itself with the efficient usage of uncertainties in zero-order trajectory opti- mization and therefore generally applicable to such optimizers. We are interested in modeling noisy system dynamics Ft-1 fI1, u4 w(x,, u4) where f is nonlinear function, 2, the observation vector, u, applied control input and w(x,, 4,) noise term sampled from an arbitrary distribution.
Consequently, in the absence of prior knowledge about the function f. the system needs to be modeled by a complex function approximator such as neural network. Furthermore, we are interested in managing uncertainties based on our fitted model, which is erroncous. To this end,

---

use stochastic ensembles of size K, where the output of each model & (Iz, uz) are parameters of normal distribution depending on input observation 2, and control ut. As by-product, our auto- regressive model prediction based on sequence of control inputs u becomes a predictive distribution over trajectories r (rD, tog 21, 11 .i..::::: 1h" (are: u) == p(T Ta, u; () where 9 denotes the parameters of the ensemble. For convenience, from this point onward we will differentiate between multiple usages of s². We denote with & the distribution p( Ae Te, Wr:++ At-1; over states at time step I + At, aa the distribution over the Gaussian parameter outputs p(0++ Att (Tt, uutt At- 0) at time step 1 + At of the planner.
3.1 Planning and Control
To validate our hypothesis that accounting for uncertainty in the environment and model prediction is essential to develop risk-avens policies, we use the Cross-Entropy Method (CEM) with improvements suggested in Pinneri et al. [31]. Accordingly, at each time step we sample a finite number of control sequences for finite horizon H from an isotropic Gaussian prior distribution which we evaluate geward-moder and the cost function. The sampling
from the state 24 using an ruto-segressive distribution is refitted in multiple rounds based on the best performing trajectories. After this optimization step. the first action of the mean of the fitted Gaussian distribution is executed. Since this approach utilizes predictive model for a finite horizon at each time step, it naturally falls into the category MPC methods.
Although we use CEM, our approach of managing uncertainty can generically be applied to other zero-order trajectory optimizers such as MPPI [28] by modification of the trajectory cost function.
3.2 The Problem of Uncertainty Estimation
Since we have a stochastic model of the dynamics, at the model prediction time step we observe a distribution over potential outcomes. Indeed, since our model outputs are parameters of a Gaussian distribution, with auto-regressive predictions we end up with a distribution over possible Gaussians for certain time step t.
Given a sampled action sequence u and the initial state z, we observe a distribution over trajectories ur. To efficiently sample from the trajectory distribution di, we use the technique introduced by Chua et al. [6] (PETS) which involves prediction particles that are sampled from the probabilistic models and randomly mixed between ensemble members at each prediction step. In this way, the sampled trajectories are used to perform a Monte Carlo estimate of the expected trajectory cost Ey-ag (c(T))- However, this does not take the properties of o into account, which might be high-entropp distribution and may lead to very risky and unsafe behavior. In this work, we alleviate this by looking at the properties of vh7 i.e. different kinds of uncertainties arising from the predictive distribution.
3.3 Learned Dynamics Model
We learn dynamics model fo that approximates the true system dynamics Zt+1 fær, Me, w(x,, 4z)). As a model class, we use an ensemble of neural networks with stochastic outputs as in Chua al. [6]. Each model k. parameterizes a multivariate Gaussian distribution with diagonal covariance, f (Iz. u.) = A (( z4 + n2 (Z,, Mz), E/ u4) where 14(,)) and (.))) are model functions outputting the respective parameters.
Iteratively, while interacting with the environment, we collect dataset of transitions D and train each model & in the ensemble by the following negative log-likelihood loss on the Gaussian outputs:
[( k) = Ez B | log (#2415 + n& (æe, ux), ΣS (Zt, ux))
(1)
In addition, we several regularization terms to make the model training more stable. We provide more details on this in Suppl. A.
3.4 Separation of Uncertainties
In the realm of parametric estimators, two uncertainties are of particular interest. Aleatoric uncertainty the kind that is irreducible and results from inherent noise of the system, e.g. sensor noises in robots. On the other hand, we have epistemic uncertainty resulting from lack of data or knowledge which is
3

---

reducible. This begs the question, how can we separate these uncertainties given auto ruttoregressive dynamics model fo? The way that we efficiently sample from 30 by mixing sampled prediction particles. This process is illustrated by the red lines in Fig. 2.
Simple model prediction disagreement is not good measure for aleatoric uncertainty since it can be entangled with epistemic uncertainty. Given our how we model the system dynamics, we measure aleatoric uncertainty entropy of the predicted normal distributions across ensemble members. More concretely, given sampled particle state Tr, we define the estimated aleatoric uncertainty for an ensemble member associated to particle b at time step as:
24 (3r [aa, uz) = Ha.va. (z)
Where Ar, 5, is the output distribution of ensemble model based inputs ëG, ue- Since in the end are interested in the aleatoric uncertainty incurred from applying the action sequence u from initial state Ty, the quantity of interest for is the expected aleatoric uncertainty for time slice
2l. (1((56 u.))
(3)
Intuitively, because we only have access to the ensemble for sampling, we take time-slice in the sampled trajectories from vे and compute the output entropies. Moreover, since we assume a Gaussian 1-step predictive distribution this is expectation over differential Gaussian entropy. An alternative way of computation which we also explore this work is calculating the expected particle variance for time slice I + of the prediction horizon:
B Σ
Var B
2 (Zc..
(4)
NN
h=1
particle
Note that Σ& (0 Uz) outputs the co- variance of the prediction at + 1. For estimating the epistemic uncertainty, one would be tempted to look at the dis- agreement between ensemble models in parameter space Var [0), but this is completely satisfying, since neural networks tend to be over-paree prer--armmttizeed and variance within the ensemble still may exist albeit the optimum has been reached by all ensemble models. An alternative would be to calculate the Fisher information metric I == Var[V, log L(x141 1 uz]] where L denotes the likelihood function, but this tends to be expensive to compute.
Rean
Figure 2: Probabilistic Ensembles with Trajectory Sampling and Uncertainty Separation (PETSUS)
Given the assumption of local Gaussianity, the true epistemic uncertainty for this case is the predictive entropy over the Gaussian parameters & at time step t + h.
E(2t, Uc:+Ar-1) Hogg (0 Fe, U1++Ar-1)
(5)
is easy to verify that this quantity O given perfect predictions of the model. Note that, because of auto-regressive predictions of nonlinear model, this is very difficult object to handle. Nevertheless, since our predictive distribution p(z|a,, Uy; i is parametrized by model output, we may utilize disagreement in O; to approximate C. To get correct estimations, we need to propagate mean predictions 전 in addition to the particles as illustrated as the yellow lines in Fig. We quantify epistemic uncertainty ensemble disagreement time step t:
Var (œ++1) Var (nt (Te, ut) + Var (",, ut)
where Var is the empirical variance over the = 1... K ensembles.
3.5 Probabilistic Safety Constraints
When applying data-driven control algorithms to real systems, safety is of utmost importance. In the realm of zero-order optimization, safety constraints can be easily introduced by putting infinite

---

cost constraint-violating trajectories. Nevertheless, we are dealing with erroneous stochastic nonlinear models which lead nontrivial predictive distributions of future states, based on the control sequence u. For this reason, we want to control the risk of violating the safety constraints that we, as practitioners, are willing to tolerate. we denote the observation space as X. given a violation set C C X, we define the probability of the control sequence ux to enter the violation set at time t ++ Af as p(æ E C It, u) Jaec (/ (x Zt, u. In practice, efficiently, since our distribution Var nontrivial as a result of nonlinear propagation of uncertainty, Furthermore, the violation set C might not have the structure necessary to allow an efficient solution to the integral, in which case one needs to resort Monte Carlo estimation.
hard to compute this integral
To simplify computation and gain speed, we consider box violation sets resulting in each dimension of x: being constrained to be outside of [a, bj E {a, bl a, b E TR?, < b). By performing moment matching by a Gaussian in each time-slice whe the probability of ending up in state at time step t ++ At is given by integrating (1;; 1r+ Σ(+De), where μ and Σ are estimated by Monte Carlo sampling. If we further assume a diagonal covariance Σ, this integral can be deconstructed into d univariate Gaussian integrals. which can be computed fast and in closed form. Hence, the probability of a constraint violation happening at time step is defined by:
p(a E Tt, u)
II
/ i= ZEC
N(x'; N(d'i
3.6 Implementing Risk-Averse ZERo-Order Trajectory Optimization (RAZER)
We assume the task definition is provided by the cost c(12; u.. For trajectory optimization, start from a state 2, and predict with an action sequence u the future development of the trajectory т. Along this trajectory, we want compute single cost term which is conveniently defined as the expected cost of all particles T summed over the planning horizon H:
H
dx u)
Σ
1
Σ
c(a c( 4+4+)).
(8)
The optimizer, in our case CEM, will optimize the action sequence u to minimize the cost in probabilistic sense, i.e. p(u x) exp(-3 (the higher the more likely it finds the global optimum). To make the planner uncertainty- aware, we need make sure avoids unpredictable parts of the state space by making them less likely. Using the aleatoric uncertainty provided by PETSUS Eq. 4, we define the alcatoric penalty as
where 3 reflects the strength of the optimizer
H Σ
Og (It, u = Wa
Vart
where wax > 0 is weighting constant. The larger the aleatoric uncertainty, the higher the cost.
To guide the exploration states where the model has epistemic uncertainty Eq. (due lack of data), we use an epistemic bonus:
C u)) = We Σ
Var
(10)
At=1
where we > 0 is a weighting constant. To be able to operate on real system, the most important part is adhere to safety constraints. As formulated in Eq. 7, the predicted safety violations need to be uncertainty aware, independent of the source of uncertainty. We integrate this into the planning method by adding:
ce (Zi, u. =
Σ A1=1
Ip(anAi ()) 8]
(11)
where [.] is Iverson bracket. and wes is either large penalty Crax O to disable safety. An alternative for implementing safety constraints into CEM is by changing the ranking function [41]. The overall algorithm used in model-predictive control fashion is outlined in Suppl. B.
5

---

-0.0 (PETS)
0.005
-0.01
-0.05
1.0
State Coverage
0.0
0.6
0.4
0.0
Steps
(a) State space exploration over time depending epistemic bonus (we).
(h) State space coverage with
(c) State space coverage with 0.05.
We O.
Figure 3: Active learning setting: The epistemic bonus allows RAZER seek states for which only little training data exists (a,c). Means and standard deviations for (a) were computed over 5 runs. PETS overfits a particular solution (b). In (b) and (c), the brightness of the dots is proportional to the time when they were first encountered.
4 Experiments
We study our uncertainty-awar planner in continuous state and action space environments and compare to naively optimizing the particle- saseeee-aeedddee estimate of the expected cost similarly to Chua et al. [6]. We start by giving a description of the environments.
BridgeMaze This toy environment (see Fig. Ic) was specifically designed to study the different aspects of uncertainty independently. The agent (blue cube) starts on the left platform and has to reach the goal platform on the right. To reach the goal platform, the agent has to move over one of three bridges without falling into the lava. The upper bridge is safeguarded by walls; hence, is the safest path to the goal but also the longest. The lower bridge has no walls and therefore is more dangerous for an unskilled agent cross but the path shorter. The middle bridge the shortest path to the goal. However, randomly appearing strong winds perpendicular to the bridge might cause the agent to fall off the bridge with some probability, making this bridge dangerous.
Noisy- HalfCheetah This environment is based on HalfCheetah-r3 from the OpenAl Gym toolkit. We introduce aleatoric uncertainty the system by adding Gaussian noise ~ ((,, o²) to the actions when the forward velocity is above 6. The action noise translates into non- -an-Gsssiin and potentially very complicated state space noise distribution that makes the control problem very challenging.
Noisy-FetchPick. PdPaaee Based the FetchhiikAndPPaceev/ gym environment. Additive action noise is applied to the gripper so that its grip on the box might become tighter or looser. The noise is applied for z-positions < 0.8 which is illustrated in Fig. 1a by blue line causing the agent drop the box with high probability if tries to lift the box too early.
In this robotic environment, the task of quadrupedal robot [42] is to stand up and lean forward to reach target position (purple markers need reach green dots in Fig. 1b) without hitting object visualized by the red cube representing the unsafe zone. The robot starts in laying position as shown in the inset of Fig. 1b. As in the Noisy-HalfCheette environment, Gaussian action noise is applied to mimic real-world perturbances.
Solo8-Leat
4.1 Algorithmic Choices and Training Details
For model model-predictive planning we use the CEM implementation from Pinneri et al. [31]. Further details about hyperparameters can be found in Suppl. A.2. For planning. we the same architecture for the ensemble of probabilistic models, both RAZER and in PETS. The only difference that in RAZER we also forward propagate the mean state predictions in addition the sampled state predictions. Further details can be found in Suppl. A.I.
4.2 Active Learning for Model Improvement
model uncertainties are used for risk-averse planning, they are only meaningful the model has the right training data. Only from good data can the parameters of the approximate noise model be learned correctly. case of too little data, the agent might avoid parts the state space due to an overestimation of the model uncertainties. On the other hand, the agent might enter unsafe regions

---

1.0
RAZER PETS
0.8
0.5
0.6
Drepping
0.4
0.2
0-2
0.0
0.0
0.05
0.12
0.14
0.16
PETS
0.1
0.5
1.0
2.0
5.0
Aleatoric Penalty
Gripper Noise
(a) BridgeMaze success depending wa for 50
(b) Dropping rate in Notxy-FeachPickAndPPar for
100 runs.
Figure 4: Risk-averse planning the face of aleatoric uncertainty yields higher success rates in noisy environments. For (b) we use ground truth models and a fixed aleatoric penalty weight wgg.
for which the uncertainties are underestimated, By adding the epistemic bonus to our cost, the planner can actively seek states with high epistemic uncertainty, i.e. for which only little training data exists.
Figure 4a shows this active data gathering process for the BridgeMaze environment. PETS finds one particular solution the problem reaching the goal platform. It chooses the path over the safer, lower bridge rather than the dangerous middle path and the longer path via the upper bridge (Fig. 3b). Once, one solution is found, the model overfits space. This is also reflected in the plateauing of the red curve in Fig. 3a.
without exploring any other parts of the state
In comparison, RAZER actively explores larger and larger parts of the state space with increasing weight of the epistemic bonus (Fig. 3a). RAZER not only finds the easy solution found by PETS but also extensively explores other parts of the state space (Fig. 3c). To not get stuck the middle bridge during exploration due to the inherent noise, it is important to separate between epistemic and aleatoric uncertainties. Only the former should be used for exploration. With enough data, our model can correctly capture the uncertainties of these states resulting in the epistemic uncertainty approaching zero.
4.3 Risk-Averse Planning
Once a good model is learned, it can be used for safe planning. What differentiates RAZER from PETS is that it makes explicit use uncertainty estimates while the latter uncertainties only enter planning by taking the mean the particle costs and not differentiating between different sources of uncertainty.
BridgeMaze. Figure 4a shows the success rate of PETS and RAZER in the BridgeMaze. In both cases, we use the same model that was trained from data collected during a training run with 0.05. Hence, the model saw enough training data from all parts of the state space. The noise in the environment is tuned such that there is a chance to cross the bridge without falling. While in Fig. 3b PETS avoided this path because of an overestimation of the state's 's value due to lack
R5Z
RAZER@20%
RAZER050%
PETS
Avg. Velocity
Violations
800
60
600
400
20
200
50
25
100
125
150
Avg.
Training Iterations
Training Iteration
0.05
0.5
1.0
(a) With alcatoric penalty (10 runs).
(b) With safety constraints.
(c) With safety constraints.
Figure 5: Noisy-HaffCheetah environment (task lengths 300 steps) with learned models.At 150 iterations we have seen only 45k points. (a) Performance under noisy actions. By applying the aleatoric penalty, RAZER can navigate the uncertainties better - leading to higher returns faster. (b) Safety violations above a certain body height (simulating low ceiling) for different values of In (c) the number of violations is averaged over the last 50 iterations (summed over 10 rollouts).

---

120
Safety Violations
100
80
80-
Tracking
40
60
20
0.01
0.95
0.01
0.95
1.0
PETS
PETS
Delta
Delta
Figure 6: Safe planning vs. task-oriented planning in the Solo8- Lee---LLeOOOerrrie environment with noisy actions. Left: number of safety violations for different values of á (Eq. 11). Right: enforcing safety constraints causes slight reduction in tracking accuracy due to the fixed planning budget and competing objectives of task and safety costs.
of training data and sometimes sees a chance to cross the bridge. However, these attempts are very likely to fail because of stronger winds that occur randomly, resulting in success rate of only 58%. RAZER does not rely on sampling for the aleatoric part and can thus avoid risk. With a higher penalty constant the success rate increases up 96% but only long the agent is willing to take a risk all. For large values of wi the agent becomes so conservative that it only moves slowly (decreasing reward in Fig. 4a).
environment when models are learned from scratch? Without aleatoric penalty, the planner is optimistic. Risky situations are only detected if a failing particle is sampled. Thus, the noise is mostly neglected and the robot increases its velocity, gets destabilized, and ends up slower than with the aleatoric penalty (Fig. 5a),
Noisy-Halle HalfCheetah. How does RAZER perform on the
FetchPickAndPllce In this environment, a 7-DoF robot arm should bring the box to target position - starting and target positions are at the opposite sides of the table. The shortest path to lift the box and move in straight line the target. However, with noise applied to the gripper action, there is certain probability to drop the box along the way. When penalizing aleatoric uncertainty, this is avoided and also fewer trajectory samples are "wasted" high-entropic regions, as presented in Fig. la. Figure 4b shows the number of times the box is dropped on the table depending on the aleatoric penalty. RAZER adopts a cautious behavior, preferring slide the box on the table and lifting it only in the area without action noise, maintaining dropping rate lower than 20%, even when considerable noise is applied.
4.4 Planning with External Safety Constraints
Noisy-Half HalfCheetah: We consider a safety constraint on the height of the body above ground simulating narrow passage. Figure 5b shows the number of safety violations. Note that PETS has the same penalty cost for hard violations.
In this experiment, the robot has move to two target points with its front and rear of the trunk while avoiding entering a specified rectangular area (fragile object). The front feet are fixed. To track the points, the robot has lean forward, such that it can lose balance due to noisy actions. contrast to PETS, RAZER successfully manages to satisfy the safety constraints almost always as shown in Fig. 6. However, satisfying the safety constraint comes with the cost of reduced tracking accuracy.
Solo8-L Bolo8-LeanOL LeanOverObject..
Conclusion
In this work, we have provided methodology to separate uncertainties in stochastic ensemble models (PETSUS) which can be used as a tool to build risk-averse model-based planners that are also data-efficient and enforce safety through probabilistic safety constraints (RAZER). This type of risk-averseness can be achieved by a simple modification of the cost function form of uncertainty penalties in zero-order trajectory optimizers.
Furthermore, the separation uncertainties allows us to proper exploration via epistemic bonus which benefits generalization of the model.As future work, it would be of interest to see this approach applied to a proper transfer learning setting from simulations to real systems, where risk-averseness combined with exploratory behavior is crucial for efficient learning and safe operation.

---

Acknowledgments
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S 1809BB. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Marin Vlastelica and Sebastian Blaes, and the Max Planck ETH Center for Learning Systems for supporting Cristina Pinneri.
References
[1] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani. Data Efficient Reinforce- ment Learning for Legged Robots. Conference on Robot Learning (CoRL), 2019.
[2] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[3] A. S. Polydoros and I.. Nalpantidis. Survey of model-based reinforcement learning: Applications on robotics. Journal of Intelligent & Robotic Systems, 86(2):153-113 2017.
[4] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Gracpel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-699 2020.
[5] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11)) 2013.
[6] K. Chua, R. Calandra, R. McAllister and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31 of NeurIPS. Curran Associates, Inc., 2018.
[7] T. Wang and J. Ba. Exploring model- planning with policy networks. In International Conference on Learning Representations (ICLR), 2020.
[8] G. Williams, A. Aldrich, and E. Thcodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, aibs/1509.01.111 149, 2015.
[9] S. C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering System Safery, 54(2):217-223, 1996. ISSN 0951-8320. httptt/s//ooi.ooo///.0000///////
URL https:
/tt.:///000011120000000000 / / w/wwwws. www.wscieee.. sciencedi rect wwt.///ccccc....... http:////cceeeee/aattaaaaa/aaa//
[10] A. D. Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Struc- tural Safety, 31(2): 105-112, 2009. ISSN 0167-4730. hoittp:://oi.ogg//..001//sssssss
2008.06 3008.06.000 020. URL https: http::/wwwwww.eeeeee.eeeeee www.sceeieeeeccec.ccomom www.://enneeeeeeee.oeeeeeeeeeeee com//sceence/attiiie///i...
[11] I. Lenz, R. A. Knepper, and A. Saxena. Deepmpe: Learning deep latent features for model predictive control. Robotics: Science and Systems. Rome, Italy, 2015.
[12] J. Fu, S. Levine, and Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In 2016 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 4019-4026 IEEE, 2016.
[13] Y. Gal, R. McAllister, and C. E. Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficien Machine Learning workshop, ICML, volume page 25, 2016.
[14] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555 2565. PMLR, 2019.
[15] J. Kocijan, R. Murray-Smith, C. E. Rasmussen, and A. Girard. Gaussian process model based predictive control. Proceedings of the 2004 American control conference, volume 3, pages 2214-2219. IEEE, 2004.

---

[16] D. Nguyen-Tuong, J. Peters, and M. Seeger. Local gaussian process regression for real time online model learning and control. In Proceedings of the 21st International Conference on Neural Information Processing Systems, pages 1193-1200, 2008
[17] A. Grancharova, J. Kocijan, and T. A. Johansen. Explicit stochastic predictive control of combustion plants based on gaussian process models. Automatica, 44(6):1621-1631, 2008.
[18] M. P. Deisenroth. D. Fox, and C. E. Rasmussen. Gaussian processes for data- learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37(2): 408 108-423, 2013.
[19] Kamthe and M. Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In International Conference on Artificial Intelligence and Statistics, pages 1701-1710. PMIR, 2018.
[20] C. Rasmussen and M. Kuss. Gaussian processes in reinforcement learning. In NIPS, 2003.
[21] C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. Adaptive Computa- tion and Machine Learning. MIT Press, Cambridge, MA, USA, Jan. 2006.
[22] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018. URL https: / openreview net/ / forum? -dd=S-iinb-Wi
[23] Osband, C. Blundell, A. Pritzel, and B. V. Roy. In Proceedings of the 30th International Conference on Neural Information Processing Sys- tems, NIPS' 16, page 4033-4041, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838811
Deep exploration via bootstrapped dqn.
[24] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Red Hook, NY, USA, 2017.
Curran Associates Inc. ISBN 9781510860964.
[25] M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, ICML, 2011.
[26] S. Curi, F. Berkenkamp, and A. Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. Pruc. Neural Information Processing Systems (NeurIPS), December 2020.
[27] M. Morari and J. Lee. Model predictive control: past, present and future. Computers & Chemical Engineering, (3(4-5):667-682 1999.
[28] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference Robotics and Automation (ICRA), pages 1714-1721. IEEE, 2017.
[29] R. Rubinstein and W. Davidson. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1999.
[30] Z. Botev, D. Kroese, Rubinstein, and P. L' L'Ecuyer. Chapter The Cross-Entropy Method for Optimization, volume 31, pages 35-59. Elsevier, 2013.
[31] C. Pinneri, S. Sawant, S. Blaes, Achterhold, J. Stueckler, M. Rolinek, and Martius. Sample- efficient cross-entropy method for real-time planning. In Conference on Robot Learning 2020, 2020. URL http:::/cortlcceee
github. ic/ cor12020 211220200pppe__21111 paper_217/_
[32] O. Mihatsch and R. Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2): 267-290, 2002.
[33] Gareia and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
10

---

[34] S. D.epeweg. J. M. Hernandez-Lobato F. Doshi-Velez, and S. Udluft. Learning and policy search stochastic dynamical systems with bayesian neural networks. In International Conference on Learning Representations (ICLR), 2017.
[35] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference Machine Learning, pages 1184-1193, PMLR, 2018.
[36] E. Arruda, M. J. Mathew, M. Kopicki, M. Mistry, M. Azad, and J. Wyatt. Uncertainty averse pushing with model predictive path integral control. 2017 IEEE-RAS 17th International Conference Humanoid Robotics (Humanoids), pages 497-502, 2017.
[37] K. Lee, G. N. An, V. Zakharov, and E. A. Theodorou. Perceptual attention-based predictive control. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 220- 232. PMLR, 30 Oct-01 Nov 2020. URL http: http//ppoceeeengeeee/ee/e//e/.se mmr. press/v1c0/ lee20b. html.
[38] Abraham, A. Handa, N. Ratliff, K. Lowrey, T. D. Murphey, and D. Fox. Model-based generalization under parameter uncertainty using path integral control. IEEE Robotics and Automation Letters, 5(2):2864 2871, April 2020.
[39] W. Clements, B.-M. Robaglia, B. V. Delft, R. B. Slaoui, and S. Toth. Estimating risk and uncertainty in deep reinforcement learning. Workshop Uncertainty & Robustness in Deep Learning Workshop at the International Conference on Machine Learning (ICML UDL), 2020.
[40] J. Zhang and P. Weng. Safe distributional reinforcement learning. ArXiv, abs/2102. 2021.
[41] M. Wen and U. Topcu. Constrained cross-entropy method for safe reinforcement learn- ing. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL httph:tt////ccc https: http::/:////////eee///ee.e neurips cc/papprr/201//ELL.. 9a1 1alffeeb9a1922ell744eeeeeob3 685.6668644666 62661126222881666663333300066661 wwwpeeoeeooopppp.r.ee
[42] F. Grimminger, A. Meduri. M. Khadiv, J. Viereck, M. Wüthrich, M. Naveau, V. Berenz, S. Heim, Widmaier, T. Flayols, et al. An open cortue-controlled modular robot architecture for legged locomotion research. IEEE Robotics and Automation Letters, 5(2):3650-365 2020.


Reviews:
[{"review_id": "rtZ43PBYqtE", "reviewer": "Reviewer_pSNt", "summary": "This paper proposes a method for separating aleatoric and epistemic uncertainties in learned dynamic models of noisy robotic systems. The paper also proposes a method for using these learned models in risk-averse model-based planning. The methods are demonstrated on a variety of simulation examples, and outperform the existing PETS method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n- The main strength of this paper is Sec. 3.4: the separation of aleatoric and epistemic uncertainties from noisy data.\n- The experimental results seem convincing that the method works.\n\nWEAKNESSES\n- I think the idea of a 1-step Gaussian nonlinear dynamics model is really odd; it seems to totally ignore the fact that we can and should model the parts of the dynamics that we actually know. I guess the method works, but it would be great to have a much clearer justification of why this Gaussian ensemble is a good model.\n- The experiment section is written in a very confusing way. It's unclear that the experiments are thorough (e.g., when RAZER has a 96% success rate, I have no idea how many trials were conducted), though some information is related through the x-axis labels of the figures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think there is a lot of implementation information missing from this paper that would make it much easier to assess whether or not the method is practical and interesting; basically, the paper is pretty unclear about what it does. I think that, if the authors are able to clarify my concerns in the \"weaknesses\" section above, and the \"issues\" section below, then I'd be happy to accept this paper.\n\nFinal review/summary: The authors have addressed my concerns and the paper seems stronger now. I already felt \"accept\" was in order, so I think my current recommendation stands.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I've included both positive notes that I took while reading as well as issues that I think could be improved.\n\nAbstract\n- short and clear, I like it\n\nIntroduction and Related Work\n- Off the top of my head, I don't know what \"zero-order\" MPC is. To me, \"first-order\" means gradient descent for trajectory optimization, and \"second-order\" means also using a Hessian; so zero-order must mean a sampling-based method with no gradient information? Might be helpful to clarify this for less familiar readers.\n- I'm very curious to see how this work performs uncertainty propagation.\n\nMethod\n- I'm curious if this paper will make any statements about how much data is needed to fit $f$.\n- The hypothesis that \"accounting for uncertainty... is essential to develop risk-averse policies\" feels tautological. Risk-aversion IS the process of accounting for uncertainty. That is, this statement strikes me as more like a definition than a hypothesis.\n- The discussion of picking apart aleatoric and epistemic uncertainty is really interesting.\n- The notation $\\mathbb{X}$ has not been defined -- presumably this is supposed to be the whole state space.\n- The simplification to enable computation seems drastic, since it's a subset of $\\mathbb{R}^2$! The state can only be 2-D? That seems to contradict the Fetch arm and HalfCheetah examples in Fig. 1. I think the unsafe set needs to be described much more clearly; I guess this is supposed to mean that there can be many unsafe sets, and each one is a 2-D box? It would be clearer to just say the constraint set is (a union of) compact box/es in the state space.\n- The fun acronyms for the proposed method (PETSUS,RAZER) should be stated in the introduction to prime the reader for these terms later on.\n\nExperiments\n- The first mention of PETS in this section should include a citation\n- I guess the epistemic uncertainty cost weight in (10) is on some sort of schedule, starting out high and decreasing over time? If this isn't the case, it would be good to explain how and why RAZER transitions from exploration to exploitation.\n- How long did training take? How many times was each method run in each environment? Is the proposed method able to run both training and testing in real time?\n- The descriptions in Sec. 4.2 are a bit incoherent (some run-on sentences, unclear explanations of results).\n- The figures are much more clear than the text, though!\n\nDiscussion\n- It would be good to title this section as \"Conclusion\"\n- This section is very clear and highlights the contribution of the proposed method!"}}, {"review_id": "jOucL-PxmBA", "reviewer": "Reviewer_Rk2o", "summary": "The paper is proposing a risk-aware CEM-based MPC method. The main machinery is extending the PETS model with direct consideration of the uncertainty terms (both epistemic and aleatoric) in the loss function. Although the proposed method is general in a sense it can be applied to an arbitrary zeroth-order optimizer, only CEM is considered in the experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is technically sound. The consideration of uncertainties is critical for learning and authors do a good job in defining technically sensible formulations of uncertainty.\n\nThe application area of safety-critical MPC is a great match to the venue and also an important problem.\n\nThe Paper is well-written with some minor issues.\n\nOne major issue I see is the lack of ablation studies. It is not clear to me whether the separation of different uncertainties actually helps or not. Somewhat related, there are lots of penalty weights that need to be defined as hyper-parameters and they seem to impact the performance significantly. I believe this would warrant a discussion on how these should be chosen as well as an empirical analysis of the sensitivity to these penalty weights.\n\nMinor but important: I think the notation is not entirely consistent. Frak and Cal fonts are used in a rather inconsistent manner. I don't know what frak font represents in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is proposing an interesting algorithm and it is technically sound. I think it is a rather straightforward extension of PETS but it is an important and required extension. My only concern is the lack of ablation study and sensitivity analysis for penalty terms.  \n\n----\n\nI thank the authors for their rebuttal. It answers my concerns. I believe the paper is definitely interesting and can be accepted. I will keep the weak accept score", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Ablation study for penalty weights\n\n- Sensitivity analysis for penalty weights\n\n- More consistent notation\n", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "0hd34ZVuo6g", "reviewer": "Reviewer_RZzg", "summary": "This paper introduces a methodology that improves exploration for uncertainty-aware dynamics models using probabilistic ensembles of neural networks in CEM-MPC planning (zero-order trajectory optimization approach) by providing an explicit separation between aleatoric and epistemic uncertainty of the learned dynamics model. The approach follows closely the methodology of previous work, PETS [6], yet improves upon it by  (i) adding an autoregressive component in the dynamics models for forward prediction, (ii) introducing an entropy-driven approach to measure epistemic and aleatoric uncertainty in the ensemble of neural nets, (iii) uses the estimated aleatoric uncertainty to penalize an MPC cost function while adding epistemic bonuses estimated from the epistemic uncertainty. Further, due to the chosen zero-order optimization technique, safety constraints can also be considered by formulating them as probabilities of safety constraints violations and adding them as penalty to the cost function. The resulting algorithm, RAZER, is compared to PETS in 4 continuous state-action space experiments in simulated environments of increasing complexity. Results clearly show that the proposed methodological improvements (RAZER) outperform PETS in terms of exploration/state-space coverage, uncovering safe solutions in regions with little training data guided by the epistemic bonus and safety constraints.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n-Principled methodology to uncover both epistemic and aleatoric uncertainties from learning dynamics models. While this is currently applied to zero-order trajectory optimization problems, such explicit separation can be used in other tasks involving predictions of uncertain dynamics systems.\n\n-Consideration of safety constraints as a form of uncertainty in the cost function is a clever way of dealing with constraint satisfaction without explicitly having to guarantee constraints in the MPC optimizer.\n\n-While the modifications to PETS might seem incremental, I believe the results are compelling as they show clearly improved exploration and state-space coverage over baseline.\n\nWeaknesses:\n\n-PETS uses the learned models within an CEM-MPC planner too. Here, it is claimed that an efficient implementation of an CEM-MPC planner from [31] was used. What is the main advantage/difference? Where is this efficiency reflected in the evaluation of results? Is [31] just a faster implementation of CEM-MPC? If so, how fast? Is PETS being implemented with this CEM-MPC planner too or with the original version? Analysis of such computational complexity is missing.\n\n-While the inclusion of safety constraints as penalties in the cost is a nice trick to include safety in planning the type of constraints allowable, basically bounds on the states [a,b] where a<b might be too restrictive. Is there a way to include other types of constraints in the approach, such as polytopic constraints, which are often found in robotics problems involving contacts and safety? \n\n-Including the constraints as cost in the CEM-MPC is not the only way to guarantee the satisfaction of safety constraints. There are other approaches that have done this, see relevant paper “Constrained Cross-Entropy Method for Safe Reinforcement Learning” from Wen and Topcu. It could be beneficial to discuss the differences with newer constrained-based approaches and how the current methodology could be extended to adopt them.\n\n-Could you provide an intuition or rule-of-thumb as to how to define and leverage the weights in the cost function for the different penalties/bonuses? For example, what’s the resulting behavior if both aleatoric penalty and epistemic bonus are high and safety constraint is low, or vice-versa or any combination of the three. In the current submission, we are only given analyzes of either the change in weights of the aleatoric penalty or epimestic bonus or increase in safety constraint probability threshold, independently.\n\n-As mentioned in the text, satisfying constraints reduces tracking accuracy, how can this be alleviated? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents interesting insights into the modeling of different types of uncertainties with neural networks and their use for trajectory-optimization. The work builds upon a previously introduced framework, yet the proposed improvements seem to provide compelling results, specifically in the improvements of state-space exploration while remaining within safe bounds. Further analysis of specific claims and decision choices can be beneficial (see weaknesses).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "See Weaknesses.\n \n\nMinor Errors:\n\n-Page 5, Line 194 “we define the an aleatoric”\n\n-Page 6, line 235 “risk-avers planning”\n\n-Page 7, line 243 “the somewhat save lower”->safe\n\n-Page 7, Line 271 “from scratch”?\n"}}]

Author rebuttal:

Meta-review: This paper proposes an approach for incorporating risk in zero-order (i.e., derivative-free) trajectory optimization. In particular, the method uses an ensemble of stochastic neural networks in order to estimate aleatoric and epistemic uncertainty in the robot dynamics model. The approach seeks to be optimistic in the face of epistemic uncertainty and pessimistic in the face of aleatoric uncertainty. The resulting model predictive control scheme is demonstrated on different robot control examples. 

Strengths:
+ The reviewers agree that the key idea of separating aleatoric uncertainty and epistemic uncertainty and dealing with these in different ways is a promising one. 
+ The reviewers generally agree that the experimental evaluation in the paper is thorough and convincing (with one point of concern noted below).
+ The reviewers agree that the paper is well written and clear overall. 

Weaknesses:
- The reviewers highlighted the fact that hyperparameters in the approach may not be entirely straightforward to choose and that the paper should provide rules of thumb on how to choose them. In addition, further details on the experimental evaluation would be helpful (see Reviewer pSNT and RZzg’s comments). 
- One concern with the experimental evaluation is the lack of thorough ablation studies. Given that the primary hypothesis of the paper is that epistemic and aleatoric uncertainty should be separately dealt with, the experimental evaluation should thoroughly demonstrate this via ablation studies. 

Suggestions:
I urge the authors to consider the reviewers’ feedback  and the primary points highlighted above. The main suggestions are to: (i) add further details on the approach, hyperparameters, and experimental evaluation (in the places suggested by the reviewers), and (ii) address the point regarding ablations.

----- Post rebuttal -----

In the rebuttal period, the authors have addressed the primary concerns reviewers had raised by (i) adding more details on algorithmic choices, and (ii) clarifying the role of taking into account epistemic and aleatoric uncertainty. The reviewers are now in agreement that the paper makes a good contribution and should be accepted.

Decision: Accept (Poster)