Policy Optimization In the Face of Uncertainty
Authors: Tung-Long Vuong
Han Nguyen
Hai Pham
Kenneth Tran
Venue:  (2020)
Keywords: Reinforcement Learning
Model-based Reinforcement Learning

Abstract: Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In this paper, we propose a novel policy optimization framework using an uncertainty-aware objective function to handle those issues. In this framework, the agent simultaneously learns an uncertainty-aware dynamics model and optimizes the policy according to these learned models. Under this framework, the objective function can represented end-to-end as a single computational graph, which allows seamless policy gradient computation via backpropagation through the models. In addition to being theoretically sound, our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines.

Under review as a conference paper at ICLR 2020
POLICY OPTIMIZATION IN THE FACE OF UNCER- TAINTY
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnera- ble to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In this paper, we propose novel policy optimization framework using an uncertainty-awanl
objective function to handle those issues. In this framework, the agent simultaneously learns uncertainty- aware dynamics model and optimizes the policy according to these learned mod- els. Under this framework, the objective function can represented end-to-end a single computational graph, which allows seamless policy gradient computa- tion via backpropagation through the models. In addition to being theoretically sound, our approach shows promising results on challenging continuous con- trol benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines.
INTRODUCTION
Popular reinforcement learning (RL) algorithms are divided into two main paradigms: model-free (MFRL) and model-based (MBRL) types. While achieving good asymtotic performances in many high dimensional problems (Mnih al.] 2015) (iilerr et al.) [017 Schulman et al., 2017: Hessel et al. 2018 Espeholt et al., 2018), MFRL methods suffer from high sample complexity since they learn state/state-action values only from rewards and do not explicitly exploit the rich information underlying the transition dynamics data. On the contrary, MBRL. approaches, by trying model the transition dynamics that are in turn used for planning without having to frequently interacting with real systems, are known to have sample efficiency and thus possess more practicability (Deisenroth
ct al. 2013 Finn et al. 2016 Ebert et al.) 2018 Sutton & Barto 2018
Kaiser et al. 2019).
Current MBRL methods, however, still have limitations because the accuracy of the learned dynam- ics model is usually not satisfied, especially in complex environments (Zhang et al.) 2018 Lowrey let al. 2018). The model error and its compounding effect when planning, i.e. small bias in the model can lead to a highly erroneous value function estimate and a strongly- policy, make MBRL less competitive in terms of asymptotic performance than MFRL for many non-trivial tasks. Numerous attempts have been made to tackle with this model bias problem but none of them have been really successful, such as using Gaussian Process (GP) (Deisenroth Ras- mussen 2011: Gal & Ghahramani 2016), Bayesian Neural Networks (Gal et al. 2016 Depeweg al."" 2016al Kamthe Deisenroth 2017). and Emsembling (Kurutach al. 2018 Clavera al.,
suboptimal
2018
Another limitation of many existing MBRL methods is that they rely on the model predictive con- trol (MPC) framework (Garcia et al. 1989). While being commonly used, MPC has serveral draw- backs (Atkeson & Schaal, 1997 Thananjeyan et al. 2019). First, each step requires solving a high- dimensional optimization problem and thus is computationally prohibitive for applications requiring either real-tima or low-latency reaction such as autonomous driving. Second, the policy is only im- plicit via solving the mentioned optimization problem. Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an exist- ing tatter-tha--random policy.

---

Under review as a conference paper at ICLR 2020
Contributions. To address those challenges of MBRL, we propose new framework called Pol- icy Optimization with Uncertainty-awe aware Model (POUM) that is able to optimize in the face of uncertainty, Our policy optimization is based on Policy Gradient, which has been widely adopted in MFRI. (Lillicrap et al. 2013] Schulman et al., 2017 Haarnoja et al.] 2018). However, in POUM, the objective function, utility function, is formulated around the uncertainty-aware dynamics model. This utility function takes into account both the mean and the variance of the value function esti- mate. This helps reducing the model bias while effectively approximating true objective, which is the value function of the policy. For experiments, we demonstrate the advantages of POUM over state-of-the -hat (SoTA) methods on various RL tasks given training from scratch and all the envi- ronments are unaltered, and also investigate on how much risk is tolerable in those tasks. And last, POUM can be represented end-to-end single computation graph, which greatly facilitates the training.
2 RELATED WORK
Initial successes of MBRL in continuous control achieved promising results by learning control policies trained on models of local dynamics using linear parametric approx-
Traditional MBRL.
imators (Abbeel et al., 2007 Levine & Koltun| 2013). Alternative methods such as Deisenroth
& Rasmussen (201 Levine & Koltun (2013) incorporated non-parametric probabilistic GPs to capture model uncertainty during policy planning and evaluation. While these methods enhance data efficiency in oow-dimesiional tasks, their applications in more challenging domains such as environments involving non-contact dynamics and high-dimensional control remain limited by the inflexibility of their temporally local structure and intractable inference time. Our approach, on the contrary, pushes the uncertainty modeling to the objective function and not anywhere else in the ar- chitecture. Plus, the fact that this objective is designed to propagate all the way to the value function makes it versatile in capturing uncertainty. What is more, all core components are constructed by neural networks gives ing asymptotically high performance compared to MFRL methods and, at the same time, retaining data efficiency in those complex domains.
solution more power in dealing with high-dimensional tasks, thus acquir-
Deep Neural Networks (DNNs). Recently, there has been a revived interest in using DNNs to learn predictive models of environments from data, drawing inspiration from ideas in the early lit- erature on this MBRL field, mainly because the large representational capacity enables them as suitable function approximators for complex environments, especially that involve images or videos (Ebert et al. 2018 Kaiser et al. 2019). However, additional care has to be usually taken to avoid model bias, a situation where the DNNs overfit in the early stages of learning, resulting in inaccurate models. For example, [eeeewe al. (2016)) modeled Bayesian type of DNNs to capture uncer- tainty in transition dynamics. In another approach, Nagabandi et al.] (2017) combined a learned dynamics network with MPC initialize the policy network to accelerate learning in model-free deep RL. Chua et al. (2018) extended this idea by introducing a bootstrapped ensemble of proba- bilistic DNNs to model predictive uncertainty of the learned networks and demonstrating that a pure modell-assed approach can attain the asymptotic performance of MFRL counterparts. However, the use of MPC to define a policy leads to poor run-time execution and hard to transfer policy across tasks. On the contrary, our framework is much simpler in that we do not employ any extra method to model the dynamics uncertainty into DNNs that are already complicated itself with numerous architectures and hyperparameters, but instead formulate single, new uncertainty objective for end-to-end optimization.
Ensemble. Another group of work leveraged the learned ensemble of dynamics models to train a policy network. Kurutach et al. (2018) learned stochastic policy via trust-region policy optimiza- tion, and Clavera et al. (2018) casted the policy gradient as a meta-learning adaptation step with respect to each member of the ensemble. [Buckman et al. (2018) proposed an algorithm to learn a weighted combination of roll-outs of different horizon lengths, which dynamically interpolates between model-based and model-free learning based on the uncertainty in the model predictions. To our knowledge, this is the closest work in aside from ours, which learns a reward function in addition to the dynamics function. But none of the aforementioned work propagates the uncertainty all the way to the value function and uses the concept of utility function to balance risk and return as in model.
2

---

Under review as a conference paper at ICLR 2020
Finally, ensemble of DNNs also provide a straightforward technique to obtain reliable estimates of predictive uncertainty [Lakshminarayanann et al. (017) and has been integrated with bootstrap to guide exploration in MFRL (Osband et al. 2016 Jannrr et al. 2019). While many of the approaches mentioned in this section employ bootstrap to train an ensemble of models, we note that their im- plementations comprise of reconstructing bootstrap datasets every training iteration, which effec- tively trains every single data sample and thus diminishes the advantage on uncertainty quantification achieved through bootstrapping. Except for novel objective formulation, our model is different in that, to maintain online bootstrapped datasets across ensembles, it adds each incoming data sam- ple dataset according to a Poisson probability distribution (Park et al.) 2007] Qin et al. 2013). thereby guaranteeing asymptotically consistent those datasets.
3 UNCERTAINTY-AWARE MODEL-BASED POLICY OPTIMIZATION
3.1 BACKGROUND
Consider discrete-time Markov Decision Process (MDP) defined by tuple M = [S, A, f.r, in which is a state space, A is an action space, f S x A - S is a deterministic (or probabilistic) transition function, factor. We define the return as sum of the rewards r (8t, 0z) = r (St, (*x)) for / = 0...., T for induced by a policy
S A + IR is deterministic reward function, and 7 E (0. 1) is discount
00,-..; the whole trajectory (so, 00 ...y 8T,
+ A and discounted by 7. Here T' E Z4 is task horizon, which may take a value of oo for non-episodic environments. The goal of RL find an optimal policy x² to maximize the expected return
ar) ,T,rr)
J(T)
= E (80)]
(1)
sp0~S"
where the value function is defined as (*a(s) Et= 7 r(st, (s:)), and the state transition is
814-1 = f(,,, (s:)), with so being randomly chosen from the distribution of s E S. Then if the dynamics function f and the reward function r are given, solving Equation I can be done using the
Calculus of Variations (Young, 2000) or using Policy Gradient (Sutton et al. 20000 when the control function is parameterized or finite dimensional.
In RL, however, f and are often unknown and hence Equation ! becomes a blackbox optimization problem with an unknown objective function. Following the Bayesian approach commonly used in the blackbox optimization literature (Shahriari et al., 2015), we propose to solve this problem by iteratively learning probabilistic estimate V of V from data and optimizing the policy according to this approximate model, as detailed in the next section.
3.2 FORMULATION OF UNCERTAINTY- AWARE OPTIMIZATION OBJECTIVE
It is worth noting that any unbiased method would model (TT) as probabilistic estimate, i.e. (T) would be distribution (as opposed point estimate) for a given policy ग. Optimizing stochastic objective is, however, not well-defined. Our solution is to transform V into a deterministic utility function that reflects subjective measure balancing the risk and return. Following Markowitz [952); Sato et al. ( (200T): | Gareia & Fernández (2015). we propose a risk-sensitive objective criterion using a linear combination of the mean and the standard deviation of ((T).. Formally stated, our objective criterion, which we also call the utility function, now becomes
(s))
EE som-s
(V(T)())) + +exa (V(r)<n))]
where μ and a are respectively the mean and the standard deviation of (s)) and is a constant that represents the subjective risk preference of the learning agent. A positive risk preference infers that the agent is adventurous while a negative risk preference indicates that the agent has a safe exploration strategy. To our best knowledge, this uncertainty-aware model-based objective function has not been used in the RL literature.
3.3 EMPIRICAL ESTIMATE OF VALUE FUNCTION
Section 3.2
provides a general framework for policy optimization under uncertainty, assuming the availability of the estimation model ((Tz) of the true value function V(z). In this section,
3

---

Under review as a conference paper at ICLR 2020
describe how to estimate (xx with model-based approach. The main idea is to approximate the functions {f. r) with probabilistic parametric models (f. and fully propagate the estimated uncertainty when planning under each policy ı from initial state 8u- The value function estimate V can be formulated as
T-1 E (T(),))()))))
((T)()))=
..... T 1. Next, describe how efficiently model ( with well-calibrated uncertainty and rollout technique that allows the uncertainty to be faithfully propagated into (xx)
where &o so and 84+1 = f(e, (8x)) for t o,....
3.3.1 BOOTSTRAP SETUP FOR MODEL LEARNING
Following the traditional bootstrap methodology, the empirical model function f is represented as (fo, (st, an) $++1)=11 For simplicity of implementation, as deterministic and rely on the ensemble as the sole mechanism for quantifying and propagating uncertainty. Each bootstrapped model fox. which is parameterized by Φx. learns to minimize the L2 one-step prediction loss over the respective bootstrapped dataset Dy:
model each bootstrap replica
min E ((ss 1| For (se, az)1 11
The training dataset D, from which the bootstrapped datasets {Tx are sampled, stores the transitions which the agent has experienced. Since each model observes its own subset of the real data samples, the predictions across the ensemble remain sufficiently diverse the early stages of the learning and will then converge to their true values as the error of the individual networks decreases.
In addition to model estimation and unlike many other model-based approaches, we also learn the reward function along the same design of classical MBRL algorithms Sutton (199I). But in POUM, use deterministic model (also parameterized by DNN) for the reward function to simplify the policy evaluation.
3.3.2 BOOTSTRAP ROLLOUT
In this section, we describe how to propagate the estimates with uncertainty from the dynamics model to evaluate policy We represent our policy To - A as neural network parame- terized by 0. Note that we choose to represent our policy as deterministic. We argue that while all estimation models, including that of the dynamics and of the value function, need be stochastic (i.e. uncertainty-aware the policy does not need to be. The policy is not an estimator and determin- istic policy simply means that the agent consistent when taking an action, no matter how uncertain it may know about the world.
Given a deterministic policy my and an initial state 8q E D, we can estimate the distribution of (so) by simulating тy through each each bootstrapped dynamics model. And since each boot- strap model is an independent approximator of the dynamics function, by expanding the value func- tion via these dynamics approximators, we eventually obtain independent estimates of that value function. Finally, those separate and independent trajectories collectively form an ensemble estima- tor of V.
In practice, we sample these trajectories with a finite horizon H < expand the value function estimation for very long horizon due to few reasons. First, DNNs training becomes harder when the depth increases. Second, despite our best effort to control the uncertainty, still do not have guarantee that uncertainty modeling is perfectly calibrated, which in turn may be problematic if the planning horizon too large. Finally, policy learning time proportional the rollout horizon.
still challenge to
4

---

Under review as a conference paper at ICLR 2020
3.4 POLICY GRADIENT
Based on Equation 2
the optimization target to optimize based on policy gradient method is:
arg max (() = Es-uS [Ug[s)]
(5)
where Up(s) ((Vo(s)) +cx (Vo(s)). Using the ensemble method and the rollout technique described above, we can naturally compute u (V)(ss) and (V) for a given policy To and for a given state 8. Therefore, the policy me be updated using the SGD variance of it.
Importantly, in terms of implementation, it is worth noting that the aforementioned rollout method also allows for easily expressing (()) in Equation 5 as single computational graph of A. This makes it straightforward to compute the policy gradient VaUn(s) using automatic differentiation, a feature provided by default in most popular deep learning toolkits.
ALGORITHM SUMMARY
Algorithm Policy Optimization with Uncertainty-aware Model (POUM)
1: Initialize training dataset D, bootstrapped datasets (TD: parameterized bootstrapped mod-
els [ a parameterized reward model ig. and a parameterized deterministic policy ma-
2: while not done do
3: Step in the environment, collect new data point (s, a, 8', r) and push into D,
4:
Sample from D and push data into the bootstrapped replay buffers: for each member 3th in the ensemble, add 2; ~ Poisson(1) copies of that data point to D;
5:
Update (f)) f=1 on D; and f D using SGD,
6:
Evaluate (s) and (s)) by simulating through the learned models J. and fo. 7: Update To using SGD with the policy gradient being backpropagated on E, [s)( through
the learned models.
8: end while
We summarize framework POUM in Algorithm aand later in this section, will also highlight some important details in our implementation.
4.1 DYNAMICS MODEL LEARNING WITH ONLINE BOOTSTRAP
As discussed in Section there are several prior attempts learn uncertainty-wware dynamics models such as GPs, Bayesian neural networks (NNs), dropout NNs and ensemble of NNs. In this work, however, we employ an ensemble of bootstrapped DNNs. Bootstrap and statistical approach for uncertainty quantification. Furthermore, as will be also later explained in Section 3.4 this ensembling approach also gives rise to easy gradient computation.
generic, principled
4.1.1 ONLINE BOOTSTRAP FOR TRAINING DATA
Bootstrap learning is often studied in the context of batch learning. However, since our agent updates its empirical model F after each physical step for the best possible sample efficiency, we follow an online bootstrapping method by sampling from Poisson distribution (Oza 2005 Qin et al.) 2013). This is a very effective online approximation to batch bootstrapping. and can be easily done by this simple rule: bootstrapping dataset D with n examples means sampling n examples from DB with replacement. In detail, each example i will appear z; times in the bootstrapped sample where z, is random variable whose distribution is Binommn, 1/n) because during resampling, the 5th example will have n chances to be picked, each with probability 1/n. This Binom (n. 1/n) distribution converges to Poisson (1) when n + oc. Therefore, for each new data point, this method adds 2 copies of that data point to the bootstrapped dataset D2g where 2 is sampled from Poisson(1).
Online off-ppoiccy learning. Except for the initialization step (we may initialize the models with batch training from off-policy data), model learning is an online learning process. For each time
5

---

Under review as a conference paper at ICLR 2020
step, the learning cost stays constant and does not grow over time, which is required for lifelong learning. Despite being online, the learning is off- aff-policy because we maintain bootstrapped replay buffer for each model in the ensemble. For each model update, sample minibatch of training data from the respective replay buffer. In addition, as mentioned, the models can also be initialized from existing data even before the policy optimization starts.
4.1.2 LINEARLY- SAMPLING FROM BOOTSTRAPPED TRAINING DATA
Since our replay buffers are accumulated online, naive uniformly sampling strategy would lead to early data being sampled more frequently than the later ones. We thus propose a linearly weighted random sampling scheme to mitigate this early-data bias issue. In this sampling scheme, example hthh is randomly sampled with weight i, i.. higher weights for the fresher examples in each online update step. Despite its simplicity, this scheme plays important role in data bias removal, as shown in Appendix A.
5
EXPERIMENT
Our experiments are designed to help 1) compare our POUM framework with other SoTA ap- proaches and 2) investigate the impact of the risk factor in our utility function on guiding agents.
5.1 COMPARISON TO BASELINE ALGORITHMS
Experimental Design. We evaluate the performance of our POUM algorithm on four continuous control tasks including: one classic control task (Pendulum-vO) and three other tasks in the MuJoCo simulator (Todorov et al. 2012) from OpenAl Gym (Brockman et al. 2016). is important to note that, we keep the default configurations prodived by OpenAl Gym (See Appendix (A.2. and also does not assume access to the reward function as some recent works in model-based reinforcement
learning (Chua al. 2018 Clavera al. 2018 Kurutach et 2018).
For the baselines, we compare POUM to the following SoTA algorithms designed for continuous control: MBPO (Janner al. 2019). DDPG (Lillicrap et al.) (015)), SAC (Haarooja et al. 2018), STEVE (Buckman et al. 2018). 8). For each one of them, we evaluate the learned policy after every episode. 'The evaluation is done by running the current policy on 20 random episodes and then computing the average return over them.
shows that POUM has a sample efficiency compared to the baseline algo- rithms across wide range of environments. Furthermore, it also has the asymptotic performance competitive to or even better than that of the model-free counterparts. Note that, there are horizontal parts at the beginning of evaluation curves in some algorithms and environments, that because these algorithms take random exploration at the beginning of training (as their default configuration) to initialize dynamics. For simple environments: Pendulum-vO, Reacher-v2, Push-v2, our POUM can get a good performance without initialized dynamics]
Results. Figure 5.1
However, Figure 5. HalfCheetah-v2 is sensitive to random seeds. We hypothesize that this is due to the impact of risk- preference value policy optimization framework and our strategy of aggressive online learning and linearly weighted batch sampling. The ablation study below validates our current analysis on these hypotheses.
5.1 also shows that the performance of POUM in more complex environment like
5.2 ABLATION STUDY
To obtain a better understanding about the role of the subjective risk preference in the utility function, we conduct an ablation study on the parameter e that controls this risk factor (Equation and make the following observations.
As illustrated in Figure 5.2 as the first observation, complete zero risk is not a good choice. This behavior is expected because risk means uncertainty is quantified properly, leading to agents
MBPOO failed to attain good performance for Reacher-v2 and Pusher-v2, regardless our best effort to produce the results based on the authors' official repository.

---

Under review as a conference paper at ICLR 2020
HaltChootah-122
Pundulum-VC
8800
6800
240
-100
400
2050
1030
120
325
250
175
204
22
FelaChor
STEVT
OCPS
-20
-10
15
ateps (lwc))
stegs (3wc))
Figure 1: Average return of POUM model over different randomly selected random seeds com- pared with SoTA appoaches. Solid lines indicate the mean and shaded areas indicate one standard deviation. POUM beats all other solutions environments tested, except for HalfCheetah- where has a competitive performances compared MBPO.
12000
10000
6000
panpy
2200
(1e3]
Figure 2: POUM with different subjective risk preference values HalfCheetah-v2 environment. The moderate risk yields the best return while too much risk (either too high or too low) will harm the agent in getting good results. We the same settings of HalfCheetah-v2 environment as presented in Appendix A.2.3 excluding risk tisk-preference values
could not learn well to model the dynamics, well as optimize efficient policy. In another observation, POUM performs best with the risks c = -1, follows by c = -2 and c = 3, while gets worse and worse both directions, the risk factor goes either higher lower. This phenomenon is because the risk factor controls the scale of standard deviation, and hence the variance. Too low or too high risks, consequently, imply too much variance which is not favorable in many cases because

---

Under review as a conference paper at ICLR 2020
it indudes agents to explore more aggressively and hence suffer more potential failures, while not exploiting current, safer experiences. Finally and interestingly, as the scale of the risk changes, both directions not behaving the same. In particular, POUM gets bad results with positive risk- preference value, and even can not learn with high positive value. That is because in current work, we fixed the subjective risk preference value and at the beginning of learning process, the dynamics models are unstable and high variance, with high positive risk-preference values, policy learning strange decisions. In contrast, with negative fisk-preference, our utility function work lower confidence bound that keep policy in a safe region. The figure indicates that with lower negative risk-predt prefereee value, the learning curve more stable. However, lower risk prkfereeeeeeeeeee value means that less exploration, and results in lower final reward.
6 DISCUSSION AND CONCLUSION
In summary, this paper proposed new approach in MBRL in which we developed novel objec- tive function that balances the mean and variance in the estimation of the value function, which is induced by the model. Our experiments suggest that POUM algorithm asymptotic performance of model-free methods in challenging continuous control tasks and com- pared to other SoTA approaches, it does so in much fewer samples. We further demonstrate that the model bias issue in model-basee uncertainty quantification, by guiding agents with subjective risk factor. Unlike other methods, quantifying and controlling the uncertainty with novel uncertainty objective function, and without any complex designs for other components an advantage, of being simple yet efficient, compared with others.
only can achieve the
RL can be dealt with effectively with principled and careful
Nonetheless, we acknowledge that our current implementation for POUM still has several limi- tations, such as high variance in the empirical performance, which still depends on many hyper- parameters (plan horizon, risk sensitivity, and all hyper-parameters associated with neural networks training techniques) and even depends random seeds. It is, however, worth noting that these traits are not unique to our methods. In spite of this limitation, the results indicate that if implemented properly, MBRL methods can be both sample efficient and have better asymptotic performances than the MFRL counterparts on challenging tasks. In addition, by explicitly representing both the dynamics model and the policy, POUM enables transfer learning, not just for the world (dynamics) model but also for the policy.
To up, identify that sample efficiency, off-policy learning, and transferability are three essary, albeit not sufficient, properties for real-world reinforcement learning. We claim that our method meets these criteria and hence is a step towards real-world reinforcement learning.
REFERENCES
Pieter Abbecl, Adam Coates, Morgan Quigley, and Andrew Y Ng. An application of reinforcement learning to aerobatic helicopter flight. In Advances in neural information processing systems, pp. 1-8, 2007.
Christopher G Atkeson and Stefan Schaal. Learning tasks from a single demonstration. In Proceed- ings of International Conference on Robotics and Automation, volume 2, pp. 1706-1712. IEEE, 1997.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.0/555 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lec. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural 2018.
Information Processing Systems, pp.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4765,, 2018.
8

---

Under review as a conference paper at ICLR 2020
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-police optimization. arXiv preprint arXiv:1809.05214. 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-basee and data efficieff approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp- 465-472, 2011.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, al. A survey policy search for roboties. Foundations and Trends in Robotics, 2(1-2):1-142, 2013.
Stefan Depeweg, José Miguel Hernández-Lobato Finale Doshi-Veles and Steffen Udluft. Learning and policy search in stochastic dynamical systems with hayesian neural networks. arXiv preprint arXiv: /605.07177 201
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprini arXiv:1605.07127 2016b.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-bases robotic control. arXiv preprint 182.00081 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted letor-learner architectures. arXiv preprint arXiv: 10:::8800 000..000611111 2018.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Picter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yarin Gal. Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, ICML, volume 4, 2016.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: theory and prac- tice-a survey, Automatica, 25(3)3335-34, 1989.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 17X1v:1801.01290, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint
08253, 2019.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: / /903 /903.00374 2019.
Sanket Kamthe and Mare Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. arXiv preprint arXiv: urxiv::/0060 1706.06466. 2017. 2017.

---

Under review as a conference paper at ICLR 2020
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.105330 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Sergey Levine and Vladlen Koltun. Guided policy search. International Conference on Machine Learning. pp. 1-9, 2013.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0291 020.9.0299,.2222 2015.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offinn: Efficient learning and exploration via model-based control. arXiv preprint 1rXiv:181.001844 2018.
Harry Markowitz. Portfolio selection. The journal finance, (1):777-11, 1952.
1952.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 51897540):529, 2015.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. CoRR, abs/1708.02566, 2017. URL
org/ abs / abs/1708. 708. 02596
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Nikunj C Oza. Online bagging and boosting. In 2005 IEEE international conference on systems, man and cybernetics, volume 3, pp. 2340-2345. leee, 2005.
Byung-Hoon Park, George Ostrouchov, and Nagiza F Samatova. Sampling streaming data with replacement. Computational Statistics & Data Analysis, 52(2):750-762, 2007.
Zhen Qin, Vaclav Petricek, Nikos Karampatziakis, Lihong Li, and John Langford. Efficient online bootstrapping for large scale learning. arXiv preprint 1312.533 2013.
Makoto Sato, Hajime Kimura, and Shibenobu Kobayashi. Td algorithm for the variance of return and mean-variance reinforcement learning. Transactions of the Japanese Society for Artificial Intelligence, 16(3):353-362, 2001.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.067777 2017.
Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
David Silver, Julian Schritwwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems, pp. 1057-1063, 2000.
10

---

Under review as a conference paper at ICLR 2020
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E Gonzalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Extending deep model pre- dictive control with safety augmented value estimation from demonstrations. arXiv preprint arXiv:/905. 13402, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Laurence Chisholm Young. Lecture the calculus of variations and optimal control theory, volume 304. American Mathematical Soc., 2000.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint arXiv: 0000000008,00 2018.
A APPENDIX
A.I WHY LINEARLY WEIGHTED RANDOM SAMPLING IS A FAIRER SAMPLING SCHEME
Consider the following online learning process: for each time step, we need to randomly sample an example from the accumulating dataset. Suppose that time t, each example 7th is randomly sampled with weight w(t, i). Note that at each time i, we have total of / examples in the dataset. Then the probability of that example being sampled is
it,
kt,
If we use uniformly random sampling then the expected number of times an example ith gets se- lected until time is
C
Hence, for all t, for i > j. Cj is larger than Cj by E. Now, if we use a linearly weighted random sampling scheme, in which w(t, then the expected number of times example ith gets selected until time is
2i
i k+1
C
-2-27
kk +
We can see that at time 6, C is still larger than C₁ for < j but by weighting recent examples more in each online update step, reduce the overall early-data bias.
A.2
EXPERIMENTAL SETTINGS
A.2.1
ENVIRONMENTS
Table Description of the environment used for testing
Environment
State dimension
Action dimension
Task horizon
Reacher-v2
11
50
Pusher-v2
23
27
100
Pendulum-v0
3
200
HalfCheetah-v2
23
1000
A.2.2 NETWORK ARCHITECTURE
A.2.3
SETTINGS
11

---

Under review as a conference paper at ICLR 2020
Table 2: Network layer configurations
Environment
bootstrap model:
Reward model: i'
Policy: ma
Reacher-v2
13, 256, 128, 11] 300 1024, 512, 256, 23]
13, 256, 128, 1]
11, 128, 128, 2
Pusher-v2
30, 1024, 512, 256, 1]
23, 512, 256, 7
Pendulum-v0
4, 64, 6
4, 64, 64, 1]
3,8.
HalfCheetah-v2
23, 1024, 512, 256, 17]
(23, 1024, 512, 256, 1]
[17, 512, 256, 6)
Network format:
input size, hidden layers size, output size]
Table 3: settings
Hype-parameter
Pendulum-v0 Reacher-v2
Pusher-v2
HalfCheetah-v2
Bootstrap size
16
16
16
Plan horizon
20
30
35
20
Risk preference
-0.25
-0.25
0
Policy updates
4
16
(per environment step)
Dynamics update (per environment step)
6
6
12


Reviews:
[{"review_id": "ryl2cBrqYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe main contribution of this work is introducing the uncertainty-aware value function prediction into model-based RL, which can be used to balance the risk and return empirically. \n\nMethodology\nThis work uses a linear combination of the mean and standard deviation of value function to capture the uncertainty in learning state value function. \n\nIt is not clear how to convert the objective function from Eq 2 (expectation over the initial state) to Eq 5 (expectation over all states). Those two objectives are not equal.\n\nIt is not clear how does the uncertainty in model prediction (dynamics and reward function)\ncan be alleviated through the proposed method, as claimed in the introduction. \nIt seems the novelty part lies in considering the uncertainty of value function estimation. \nHow does this relate to solving the limitation of model predictive control? \n\nWhat is the objective function for learning reward function r_\\phi?\n\n\nExperimental results:\nThe experiments are not sufficient to demonstrate the effectiveness of the proposed method. \nIt would be more convincing to compare the proposed method with a few more model-based approaches on more tasks. The results of MBPO is better\nthe proposed POUM in one of two tasks. The performance of\nMBPO on Reacher-v2 and Pusher-v2 is missing? \n\n\nWriting:\nThis paper has many typos and the presentation is not very clear.\n- Section 4.1 \"in Section 3.4,\" \n- Last paragraph in P3: convergence convergence analysis \n- \"shows that POUM has a sample efficiency compared\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "ryeftDnCFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Introduction\nThe motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.” \n\nThere has been work on learning a parametric policy from MPC. Therefore, you can extract a parametric policy from the optimization that MPC performs. The statement “Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy” is not true. The MPC will transfer better to other tasks that have the same dynamics, since it is not task specific. Given the learned dynamics model and the reward function you can act optimally in any new task as long as the learned dynamics are valid.\n\n# Related work\nMy main concern with the related work section is that there a lot of literature on risk sensitive and optimism in the face of uncertainty (which is a subset of your method when c>0) in control, bandits, and some on *reinforcement learning* that has been neglected. \n\n# Uncertainty-Aware Model-Based Policy Optimization\nAs said before, risk-sensitive in reinforcement learning has been done before and there’s even more work on control and bandits. For instance in [1] (page 5, paragraph (b)) has the same equation and they discuss the effect of the constant being negative or positive. More recent work has also used similar formulations [2].\nThis section mostly contains previous work, e.g., bootstrap rollout, policy gradient, using a deterministic policy ([3, 4, 5]). One thing that it’s still not clear from reading the paper, are you backpropagating the through the dynamics model, are you using a policy gradient method (REINFORCE, TRPO, PPO,… )?\n\n# Algorithm Summary\nOne of the novelties introduced is the fact that the data used in each model comes from sampling a Poisson variable. However, this is not ablated in the results sections. Is it necessary? [6] Claims that there’s no need to use different data for the learned models.\n\n# Experiment\nThe experiment section lacks from more complex environments, in this case the most complex is half-cheetah. Furthermore, given that 3 of the tasks are short horizon tasks you should probably also compare against model-based methods that build on top of policy gradients (e.g., [6]). \n\nIt seems that some choices in the algorithm are not ablated: 1) use of poisson, 2) use of deterministic vs stochastic policy, 3) Is there a single risk that works across environments? Which environments are risk prone/adverse? 4) How about having c ~ N(0, 1), effectively modelling V as a gaussian?\n\n-----------------------------------\n\nOverall, the paper is not mature enough to be accepted: there is not enough novelty, and the results lack of novelty, enough delta in performance from prior work, and have high variance.\n\n------------------------------------\n\nMinor/Typos:\nFirst paragraph: “trying model the transition”\nWhat does it mean that the accuracy is not satisfied?\nWhy the related work on deep model-based reinforcement learning is called Deep Neural Networks?\n3.2 third paragraph: “Next we provide a convergence convergence …”\n3.3.2, first paragraph: “no matter how uncertain it may know about the world”\nWhy the axis in the results section mean different things?\n\n\n\n[1] Risk-sensitive Reinforcement Learning. Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer. \n[2] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch\n[3] Continuous control with deep reinforcement learning. Timothy P. Lillicrap et. al.\n[4] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine\n[5] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee.\n[6] Model-Ensemble Trust-Region Policy Optimization. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Byg_BlmItS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I enjoyed this paper overall, and I think the idea is a good one. However there remain significant issues with the paper that preclude me giving a good score. Firstly, there is almost no discussion of the environment model. Anyone who has worked on Model Based RL will tell you that the details here are crucial. This deserves a full discussion, and a comparison to other methods in the literature.\n\nNext, the experimental results really aren't convincing. The dependence on random seeds is worrying, and isn't as common in model free algorithms as you claim, which are mostly robust to seeds (the good ones at least). The fact that the best policy is risk *averse* is very strange, since these are estimates combining both the epistemic and aleatoric uncertainty (which is somewhat unfortunate), which means being risk averse would lead the agent to not explore. That is very worrying and makes me think that something very strange is going on with the models. In fact since the policy is deterministic and the environment / rewards are practically speaking deterministic, the uncertainty here is actually mostly epistemic and so a c < 0 means the agent is disincentivized from exploring.\n\nThere should be more discussion about the fact that the policy is deterministic. Is this merely to make estimating V^pi easier?\n\n\"Next, we provide a convergence convergence analysis and show that maximizing this utility function\nU(π) is equivalent to maximizing the unknown value function V (π).\"\n\nWord convergence appears twice in a row, but more importantly this is totally missing! Where is the analysis?\n\nIn the algorithm you write:\n\"Update {fb} and rˆφ using SGD\"\nBut on what data? Presumably sampled from D but this isn't mentioned.\n\nIs it the case that the policy is updated *only* using the model based rollouts? I.e., the reward signal is never used in the policy gradient but only used to train the models? If so, this seems quite fragile and I would like to see a comparison of different approaches here.\n\nTable 2 is unreadable and needs to be explained.\n\nIt would appear that you are missing a reference to the very relevant UBE paper, which also deals explicitly with the uncertainty of the value function estimates: https://arxiv.org/abs/1709.05380\nIn fact I would be curious to see any way that these two approaches could be combined (though that would be follow up work).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this work is introducing the uncertainty-aware value function prediction into model-based RL, which can be used to balance the risk and return empirically. 

The reviewers generally agree that this paper addresses an interesting problem, but there are some concerns that remain (see reviewer comments). 

I also want to highlight that in terms of empirical results, it is insufficient to present results for 3 different random seeds. To highlight any kind of robustness, I suggest *at least* 10-20 different random seeds; otherwise the findings can/will be misleading. 
Decision: Reject