Title: POLICY OPTIMIZATION IN THE FACE OF UNCER-TAINTY

Abstract: Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In this paper, we propose a novel policy optimization framework using an uncertainty-aware objective function to handle those issues. In this framework, the agent simultaneously learns an uncertaintyaware dynamics model and optimizes the policy according to these learned models. Under this framework, the objective function can represented end-to-end as a single computational graph, which allows seamless policy gradient computation via backpropagation through the models. In addition to being theoretically sound, our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines.

1. INTRODUCTION:

Popular reinforcement learning (RL) algorithms are divided into two main paradigms: model-free (MFRL) and model-based (MBRL) types. While achieving good asymtotic performances in many high dimensional problems (Mnih et al., 2015; Silver et al., 2017; Schulman et al., 2017; Hessel et al., 2018; Espeholt et al., 2018) , MFRL methods suffer from high sample complexity since they learn state/state-action values only from rewards and do not explicitly exploit the rich information underlying the transition dynamics data. On the contrary, MBRL approaches, by trying model the transition dynamics that are in turn used for planning without having to frequently interacting with real systems, are known to have sample efficiency and thus possess more practicability (Deisenroth et al., 2013; Finn et al., 2016; Ebert et al., 2018; Sutton & Barto, 2018; Kaiser et al., 2019) .

Current MBRL methods, however, still have limitations because the accuracy of the learned dynamics model is usually not satisfied, especially in complex environments (Zhang et al., 2018; Lowrey et al., 2018) . The model error and its compounding effect when planning, i.e. a small bias in the model can lead to a highly erroneous value function estimate and a strongly-biased suboptimal policy, make MBRL less competitive in terms of asymptotic performance than MFRL for many non-trivial tasks. Numerous attempts have been made to tackle with this model bias problem but none of them have been really successful, such as using Gaussian Process (GP) (Deisenroth & Rasmussen, 2011; Gal & Ghahramani, 2016) , Bayesian Neural Networks (Gal et al., 2016; Depeweg et al., 2016a; Kamthe & Deisenroth, 2017) , and Emsembling (Kurutach et al., 2018; Clavera et al., 2018) .

Another limitation of many existing MBRL methods is that they rely on the model predictive control (MPC) framework (Garcia et al., 1989) . While being commonly used, MPC has serveral drawbacks (Atkeson & Schaal, 1997; Thananjeyan et al., 2019) . First, each step requires solving a highdimensional optimization problem and thus is computationally prohibitive for applications requiring either real-time or low-latency reaction such as autonomous driving. Second, the policy is only implicit via solving the mentioned optimization problem. Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy.

Contributions. To address those challenges of MBRL, we propose a new framework called Policy Optimization with Uncertainty-aware Model (POUM) that is able to optimize in the face of uncertainty. Our policy optimization is based on Policy Gradient, which has been widely adopted in MFRL (Lillicrap et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018) . However, in POUM, the objective function, a utility function, is formulated around the uncertainty-aware dynamics model. This utility function takes into account both the mean and the variance of the value function estimate. This helps reducing the model bias while effectively approximating true objective, which is the value function of the policy. For experiments, we demonstrate the advantages of POUM over state-of-the-art (SoTA) methods on various RL tasks given training from scratch and all the environments are unaltered, and also investigate on how much risk is tolerable in those tasks. And last, POUM can be represented end-to-end in a single computation graph, which greatly facilitates the training.

2. RELATED WORK:

Traditional MBRL. Initial successes of MBRL in continuous control achieved promising results by learning control policies trained on models of local dynamics using linear parametric approximators (Abbeel et al., 2007; Levine & Koltun, 2013) . Alternative methods such as Deisenroth & Rasmussen (2011) ; Levine & Koltun (2013) incorporated non-parametric probabilistic GPs to capture model uncertainty during policy planning and evaluation. While these methods enhance data efficiency in low-dimensional tasks, their applications in more challenging domains such as environments involving non-contact dynamics and high-dimensional control remain limited by the inflexibility of their temporally local structure and intractable inference time. Our approach, on the contrary, pushes the uncertainty modeling to the objective function and not anywhere else in the architecture. Plus, the fact that this objective is designed to propagate all the way to the value function makes it versatile in capturing uncertainty. What is more, all core components are constructed by neural networks gives our solution more power in dealing with high-dimensional tasks, thus acquiring asymptotically high performance compared to MFRL methods and, at the same time, retaining data efficiency in those complex domains.

Deep Neural Networks (DNNs). Recently, there has been a revived interest in using DNNs to learn predictive models of environments from data, drawing inspiration from ideas in the early literature on this MBRL field, mainly because the large representational capacity enables them as suitable function approximators for complex environments, especially that involve images or videos (Ebert et al., 2018; Kaiser et al., 2019) . However, additional care has to be usually taken to avoid model bias, a situation where the DNNs overfit in the early stages of learning, resulting in inaccurate models. For example, Depeweg et al. (2016b) modeled a Bayesian type of DNNs to capture uncertainty in transition dynamics. In another approach, Nagabandi et al. (2017) combined a learned dynamics network with MPC to initialize the policy network to accelerate learning in model-free deep RL. Chua et al. (2018) extended this idea by introducing a bootstrapped ensemble of probabilistic DNNs to model predictive uncertainty of the learned networks and demonstrating that a pure model-based approach can attain the asymptotic performance of MFRL counterparts. However, the use of MPC to define a policy leads to poor run-time execution and hard to transfer policy across tasks. On the contrary, our framework is much simpler in that we do not employ any extra method to model the dynamics uncertainty into DNNs that are already complicated itself with numerous architectures and hyperparameters, but instead formulate a single, new uncertainty-aware objective for end-to-end optimization.

Ensemble. Another group of work leveraged the learned ensemble of dynamics models to train a policy network. Kurutach et al. (2018) learned a stochastic policy via trust-region policy optimization, and Clavera et al. (2018) casted the policy gradient as a meta-learning adaptation step with respect to each member of the ensemble. Buckman et al. (2018) proposed an algorithm to learn a weighted combination of roll-outs of different horizon lengths, which dynamically interpolates between model-based and model-free learning based on the uncertainty in the model predictions. To our knowledge, this is the closest work in aside from ours, which learns a reward function in addition to the dynamics function. But none of the aforementioned work propagates the uncertainty all the way to the value function and uses the concept of utility function to balance risk and return as in our model.

Finally, ensemble of DNNs also provide a straightforward technique to obtain reliable estimates of predictive uncertainty (Lakshminarayanan et al., 2017) and has been integrated with bootstrap to guide exploration in MFRL (Osband et al., 2016; Janner et al., 2019) . While many of the approaches mentioned in this section employ bootstrap to train an ensemble of models, we note that their implementations comprise of reconstructing bootstrap datasets at every training iteration, which effectively trains every single data sample and thus diminishes the advantage on uncertainty quantification achieved through bootstrapping. Except for a novel objective formulation, our model is different in that, to maintain online bootstrapped datasets across ensembles, it adds each incoming data sample to a dataset according to a Poisson probability distribution (Park et al., 2007; Qin et al., 2013) , thereby guaranteeing asymptotically consistent those datasets.

3. UNCERTAINTY-AWARE MODEL-BASED POLICY OPTIMIZATION:

3.1 BACKGROUND Consider a discrete-time Markov Decision Process (MDP) defined by a tuple M = {S, A, f, r, γ}, in which S is a state space, A is an action space, f : S × A → S is a deterministic (or probabilistic) transition function, r : S × A → R is a deterministic reward function, and γ ∈ (0, 1) is a discount factor. We define the return as sum of the rewards r (s t , a t ) = r (s t , π(s t )) for t = 0, . . . , T for the whole trajectory (s 0 , a 0 , ..., s T , a T ) induced by a policy π : S → A and discounted by γ. Here T ∈ Z + is a task horizon, which may take a value of ∞ for non-episodic environments. The goal of RL is to find an optimal policy π to maximize the expected return

EQUATION

where the value function is defined as

V π (s 0 ) = T -1 t=0 γ t r(s t , π(s t ))

, and the state transition is s t+1 = f (s t , π(s t )), with s 0 being randomly chosen from the distribution of s ∈ S. Then if the dynamics function f and the reward function r are given, solving Equation 1 can be done using the Calculus of Variations (Young, 2000) or using Policy Gradient (Sutton et al., 2000) when the control function is parameterized or is finite dimensional.

In RL, however, f and r are often unknown and hence Equation 1 becomes a blackbox optimization problem with an unknown objective function. Following the Bayesian approach commonly used in the blackbox optimization literature (Shahriari et al., 2015) , we propose to solve this problem by iteratively learning a probabilistic estimate V of V from data and optimizing the policy according to this approximate model, as detailed in the next section.

3.2. FORMULATION OF UNCERTAINTY-AWARE OPTIMIZATION OBJECTIVE:

It is worth noting that any unbiased method would model V (π) as a probabilistic estimate, i.e. V (π) would be a distribution (as opposed to a point estimate) for a given policy π. Optimizing a stochastic objective is, however, not well-defined. Our solution is to transform V into a deterministic utility function that reflects a subjective measure balancing the risk and return. Following Markowitz (1952) ; Sato et al. (2001) ; Garcıa & Fernández (2015) , we propose a risk-sensitive objective criterion using a linear combination of the mean and the standard deviation of V (π). Formally stated, our objective criterion, which we also call the utility function, now becomes

EQUATION

where µ and σ are respectively the mean and the standard deviation of V (π)(s 0 ), and c is a constant that represents the subjective risk preference of the learning agent. A positive risk preference infers that the agent is adventurous while a negative risk preference indicates that the agent has a safe exploration strategy. To our best knowledge, this uncertainty-aware model-based objective function has not been used in the RL literature.

3.3. EMPIRICAL ESTIMATE OF VALUE FUNCTION:

Section 3.2 provides a general framework for policy optimization under uncertainty, assuming the availability of the estimation model V (π) of the true value function V (π). In this section, we describe how to estimate V (π) with a model-based approach. The main idea is to approximate the functions {f, r} with probabilistic parametric models { f , r} and fully propagate the estimated uncertainty when planning under each policy π from an initial state s 0 . The value function estimate V can be formulated as

EQUATION

where ŝ0 = s 0 and ŝt+1 = f (ŝ t , π(ŝ t )) for t = 0, . . . , T -1. Next, we describe how to efficiently model {f, r} with well-calibrated uncertainty and a rollout technique that allows the uncertainty to be faithfully propagated into V (π).

3.3.1. BOOTSTRAP SETUP FOR MODEL LEARNING:

Following the traditional bootstrap methodology, the empirical model function f is represented as

{ f φ k (s t , a t ) → s t+1 } B k=1 .

For simplicity of implementation, we model each bootstrap replica as deterministic and rely on the ensemble as the sole mechanism for quantifying and propagating uncertainty. Each bootstrapped model f φ k , which is parameterized by φ k , learns to minimize the L2 one-step prediction loss over the respective bootstrapped dataset D k :

EQUATION

The training dataset D, from which the bootstrapped datasets {D k } B k=1 are sampled, stores the transitions on which the agent has experienced. Since each model observes its own subset of the real data samples, the predictions across the ensemble remain sufficiently diverse in the early stages of the learning and will then converge to their true values as the error of the individual networks decreases.

In addition to model estimation and unlike many other model-based approaches, we also learn the reward function along the same design of classical MBRL algorithms Sutton (1991) . But in POUM, we use a deterministic model (also parameterized by a DNN) for the reward function to simplify the policy evaluation.

3.3.2. BOOTSTRAP ROLLOUT:

In this section, we describe how to propagate the estimates with uncertainty from the dynamics model to evaluate a policy π. We represent our policy π θ : S → A as a neural network parameterized by θ . Note that we choose to represent our policy as deterministic. We argue that while all estimation models, including that of the dynamics and of the value function, need to be stochastic (i.e. uncertainty-aware), the policy does not need to be. The policy is not an estimator and deterministic policy simply means that the agent is consistent when taking an action, no matter how uncertain it may know about the world.

Given a deterministic policy π θ and an initial state s 0 ∈ D, we can estimate the distribution of V (π)(s 0 ) by simulating π θ through each each bootstrapped dynamics model. And since each bootstrap model is an independent approximator of the dynamics function, by expanding the value function via these dynamics approximators, we eventually obtain independent estimates of that value function. Finally, those separate and independent trajectories collectively form an ensemble estimator of V .

In practice, we sample these trajectories with a finite horizon H < T . It is still a challenge to expand the value function estimation for a very long horizon due to a few reasons. First, DNNs training becomes harder when the depth increases. Second, despite our best effort to control the uncertainty, we still do not have a guarantee that our uncertainty modeling is perfectly calibrated, which in turn may be problematic if the planning horizon is too large. Finally, policy learning time is proportional to the rollout horizon.

3.4. POLICY GRADIENT:

Based on Equation 2, the optimization target to optimize based on policy gradient method is:

EQUATION

where U θ (s) = µ( V θ (s)) + c × σ( V θ (s)). Using the ensemble method and the rollout technique described above, we can naturally compute µ( V θ (s)) and σ( V θ (s)) for a given policy π θ and for a given state s. Therefore, the policy π θ can be updated using the SGD or a variance of it.

Importantly, in terms of implementation, it is worth noting that the aforementioned rollout method also allows for easily expressing U (θ) in Equation 5as a single computational graph of θ. This makes it straightforward to compute the policy gradient ∇ θ U θ (s) using automatic differentiation, a feature provided by default in most popular deep learning toolkits. • Step in the environment, collect new data point (s, a, s , r) and push into D, 4:

4. ALGORITHM SUMMARY:

• Sample from D and push data into the bootstrapped replay buffers: for each member i th in the ensemble, add z i ∼ P oisson(1) copies of that data point to D i , 5:

• Update { f i } B

i=1 on D i and rφ on D using SGD, 6:

• Evaluate V θ (s) and U θ (s) by simulating through the learned models { f i } B i=1 and r φ , 7:

• Update π θ using SGD with the policy gradient being backpropagated on E s [U θ (s)] through the learned models. 8: end while We summarize our framework POUM in Algorithm 1 and later in this section, we will also highlight some important details in our implementation.

4.1. DYNAMICS MODEL LEARNING WITH ONLINE BOOTSTRAP:

As discussed in Section 1, there are several prior attempts to learn uncertainty-aware dynamics models such as GPs, Bayesian neural networks (NNs), dropout NNs and ensemble of NNs. In this work, however, we employ an ensemble of bootstrapped DNNs. Bootstrap is a generic, principled and statistical approach for uncertainty quantification. Furthermore, as will be also later explained in Section 3.4, this ensembling approach also gives rise to easy gradient computation.

4.1.1. ONLINE BOOTSTRAP FOR TRAINING DATA:

Bootstrap learning is often studied in the context of batch learning. However, since our agent updates its empirical model F after each physical step for the best possible sample efficiency, we follow an online bootstrapping method by sampling from Poisson distribution (Oza, 2005; Qin et al., 2013) . This is a very effective online approximation to batch bootstrapping, and can be easily done by this simple rule: bootstrapping a dataset D with n examples means sampling n examples from D with replacement. In detail, each example i will appear z i times in the bootstrapped sample where z i is a random variable whose distribution is Binom(n, 1/n) because during resampling, the i th example will have n chances to be picked, each with probability 1/n. This Binom(n, 1/n) distribution converges to P oisson(1) when n → ∞. Therefore, for each new data point, this method adds z k copies of that data point to the bootstrapped dataset D k , where z k is sampled from a P oisson(1).

Online off-policy learning. Except for the initialization step (we may initialize the models with batch training from off-policy data), our model learning is an online learning process. For each time step, the learning cost stays constant and does not grow over time, which is required for lifelong learning. Despite being online, the learning is off-policy because we maintain a bootstrapped replay buffer for each model in the ensemble. For each model update, we sample a minibatch of training data from the respective replay buffer. In addition, as mentioned, the models can also be initialized from existing data even before the policy optimization starts.

4.1.2. LINEARLY-WEIGHTED SAMPLING FROM BOOTSTRAPPED TRAINING DATA:

Since our replay buffers are accumulated online, a naive uniformly sampling strategy would lead to early data being sampled more frequently than the later ones. We thus propose a linearly weighted random sampling scheme to mitigate this early-data bias issue. In this sampling scheme, example i th is randomly sampled with weight i, i.e. higher weights for the fresher examples in each online update step. Despite its simplicity, this scheme plays an important role in data bias removal, as shown in Appendix A.1.

5. EXPERIMENT:

Our experiments are designed to help 1) compare our POUM framework with other SoTA approaches and 2) investigate the impact of the risk factor in our utility function on guiding agents.

5.1. COMPARISON TO BASELINE ALGORITHMS:

Experimental Design. We evaluate the performance of our POUM algorithm on four continuous control tasks including: one classic control task (Pendulum-v0) and three other tasks in the MuJoCo simulator (Todorov et al., 2012) from OpenAI Gym (Brockman et al., 2016) . It is important to note that, we keep the default configurations prodived by OpenAI Gym (See Appendix A.2.1) and also does not assume access to the reward function as some recent works in model-based reinforcement learning (Chua et al., 2018; Clavera et al., 2018; Kurutach et al., 2018) .

For the baselines, we compare POUM to the following SoTA algorithms designed for continuous control: MBPO (Janner et al., 2019 ), DDPG (Lillicrap et al., 2015) , SAC (Haarnoja et al., 2018) , STEVE (Buckman et al., 2018) . For each one of them, we evaluate the learned policy after every episode. The evaluation is done by running the current policy on 20 random episodes and then computing the average return over them.

Results. Figure 5 .1 shows that POUM has a sample efficiency compared to the baseline algorithms across a wide range of environments. Furthermore, it also has the asymptotic performance competitive to or even better than that of the model-free counterparts. Note that, there are horizontal parts at the beginning of evaluation curves in some algorithms and environments, that because these algorithms take random exploration at the beginning of training (as their default configuration) to initialize dynamics. For simple environments: Pendulum-v0, Reacher-v2, Push-v2, our POUM can get a good performance without initialized dynamics1 . However, Figure 5 .1 also shows that the performance of POUM in more complex environment like HalfCheetah-v2 is sensitive to random seeds. We hypothesize that this is due to the impact of riskpreference value on policy optimization framework and our strategy of aggressive online learning and linearly weighted batch sampling. The ablation study below validates our current analysis on these hypotheses.

5.2. ABLATION STUDY:

To obtain a better understanding about the role of the subjective risk preference in the utility function, we conduct an ablation study on the parameter c that controls this risk factor (Equation 2) and make the following observations. As illustrated in Figure 5 .2, as the first observation, complete zero risk is not a good choice. This behavior is expected because no risk means no uncertainty is quantified properly, leading to agents The moderate risk yields the best return while too much risk (either too high or too low) will harm the agent in getting good results. We use the same settings of HalfCheetah-v2 environment as presented in Appendix A.2.3 excluding risk-preference values could not learn well to model the dynamics, as well as optimize an efficient policy. In another observation, POUM performs best with the risks c = -1, follows by c = -2 and c = -3, while it gets worse and worse at both directions, the risk factor goes either higher or lower. This phenomenon is because the risk factor controls the scale of standard deviation, and hence the variance. Too low or too high risks, consequently, imply too much variance which is not favorable in many cases because it indudes agents to explore more aggressively and hence suffer more potential failures, while not exploiting current, safer experiences. Finally and interestingly, as the scale of the risk changes, both directions are not behaving the same. In particular, POUM gets bad results with positive riskpreference value, and even can not learn with high positive value. That is because in current work, we use fixed the subjective risk preference value and at the beginning of learning process, the dynamics models are unstable and high variance, with high positive risk-preference values, policy learning strange decisions. In contrast, with negative risk-preference, our utility function work as lower confidence bound that keep policy in a safe region. The figure indicates that with lower negative risk-preference value, the learning curve is more stable. However, lower risk-preference value means that less exploration, and results in lower final reward.

6. DISCUSSION AND CONCLUSION:

In summary, this paper proposed a new approach in MBRL in which we developed a novel objective function that balances the mean and variance in the estimation of the value function, which is induced by the model. Our experiments suggest that our POUM algorithm not only can achieve the asymptotic performance of model-free methods in challenging continuous control tasks and compared to other SoTA approaches, it does so in much fewer samples. We further demonstrate that the model bias issue in model-based RL can be dealt with effectively with principled and careful uncertainty quantification, by guiding agents with a subjective risk factor. Unlike other methods, quantifying and controlling the uncertainty with a novel uncertainty-aware objective function, and without any complex designs for other components is an advantage, of being simple yet efficient, compared with others.

Nonetheless, we acknowledge that our current implementation for POUM still has several limitations, such as high variance in the empirical performance, which still depends on many hyperparameters (plan horizon, risk sensitivity, and all hyper-parameters associated with neural networks training techniques) and even depends on random seeds. It is, however, worth noting that these traits are not unique to our methods. In spite of this limitation, the results indicate that if implemented properly, MBRL methods can be both sample efficient and have better asymptotic performances than the MFRL counterparts on challenging tasks. In addition, by explicitly representing both the dynamics model and the policy, POUM enables transfer learning, not just for the world (dynamics) model but also for the policy.

To sum up, we identify that sample efficiency, off-policy learning, and transferability are three necessary, albeit not sufficient, properties for real-world reinforcement learning. We claim that our method meets these criteria and hence is a step towards real-world reinforcement learning. .

If we use uniformly random sampling then the expected number of times an example i th gets selected until time t is

C t i = t k=i 1 k .

Hence, for all t, for i > j, C t i is larger than C t j by j k=i 1 k . Now, if we use a linearly weighted random sampling scheme, in which w(t, i) = i, then the expected number of times an example i th gets selected until time t is

C t i = t k=i 2i k(k + 1) = 2 t k=i i k - i k + 1 = 2 -2 i t .

We can see that at time t, C t i is still larger than C t j for i < j but by weighting recent examples more in each online update step, we reduce the overall early-data bias. [13, 256, 128, 11] [13, 256, 128, 1] [11, 128, 128, 2] Pusher-v2 [30, 1024 , 512, 256, 23] [30, 1024 , 512, 256, 1] [23, 512, 256, 7] Pendulum-v0 [4, 64, 64, 3] [4, 64, 64, 1] [3, 8, 1] HalfCheetah-v2 [23, 1024 , 512, 256, 17] [23, 1024, 512, 256, 1] [17, 512, 256, 6] Network format:

A.2 EXPERIMENTAL SETTINGS:



A.2.1 ENVIRONMENTS:

[ input size, hidden layers size, output size ] 

:

MBPO failed to attain a good performance for Reacher-v2 and Pusher-v2, regardless our best effort to produce the results based on the authors' official repository.

Figures and Tables:

Figure 1: Average return of POUM model over 3 different randomly selected random seeds compared with SoTA appoaches. Solid lines indicate the mean and shaded areas indicate one standard deviation. POUM beats all other solutions on environments tested, except for HalfCheetah-v2 where it has a competitive performances compared to MBPO.

Figure 2: POUM with different subjective risk preference values on HalfCheetah-v2 environment. The moderate risk yields the best return while too much risk (either too high or too low) will harm the agent in getting good results. We use the same settings of HalfCheetah-v2 environment as presented in Appendix A.2.3 excluding risk-preference values

Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E Gonzalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Extending deep model predictive control with safety augmented value estimation from demonstrations. arXiv preprint arXiv:1905.13402, 2019. Lecture on the calculus of variations and optimal control theory, volume 304. American Mathematical Soc., 2000. Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint arXiv:1808.09105, 2018. A APPENDIX A.1 WHY LINEARLY WEIGHTED RANDOM SAMPLING IS A FAIRER SAMPLING SCHEMEConsider the following online learning process: for each time step, we need to randomly sample an example from the accumulating dataset. Suppose that at time t, each example i th is randomly sampled with weight w(t, i). Note that at each time t, we have a total of t examples in the dataset.Then the probability of that example being sampled is w(t, i)

Table content: <table><tr><td>Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.</td></tr><tr><td>In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.</td></tr><tr><td>IEEE, 2012.</td></tr><tr><td>Laurence Chisholm Young.</td></tr></table>

Description of the environment used for testing Environment State dimension Action dimension Task horizon

Table content: <table><tr><td>Reacher-v2</td><td>11</td><td>2</td><td>50</td></tr><tr><td>Pusher-v2</td><td>23</td><td>27</td><td>100</td></tr><tr><td>Pendulum-v0</td><td>3</td><td>1</td><td>200</td></tr><tr><td>HalfCheetah-v2</td><td>23</td><td>6</td><td>1000</td></tr><tr><td>A.2.2 NETWORK ARCHITECTURE</td><td/><td/><td/></tr><tr><td colspan="2">A.2.3 HYPER-PARAMETER SETTINGS</td><td/><td/></tr></table>

Network layer configurations Environment bootstrap model: f i Reward model: rφ Policy: π θ

Table content: <table><tr><td>Reacher-v2</td></tr></table>

Hyper-parameter settings

Table content: <table><tr><td>Hype-parameter</td><td colspan="4">Pendulum-v0 Reacher-v2 Pusher-v2 HalfCheetah-v2</td></tr><tr><td>Bootstrap size</td><td>16</td><td>8</td><td>16</td><td>16</td></tr><tr><td>Plan horizon</td><td>20</td><td>30</td><td>35</td><td>20</td></tr><tr><td>Risk preference</td><td>-0.25</td><td>-0.25</td><td>0</td><td>-1</td></tr><tr><td>Policy updates</td><td>8</td><td>4</td><td>16</td><td>2</td></tr><tr><td>(per environment step)</td><td/><td/><td/><td/></tr><tr><td>Dynamics update</td><td>8</td><td>6</td><td>6</td><td>6</td></tr><tr><td>(per environment step)</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
# Introduction The motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.” There has been work on learning a parametric policy from MPC. Therefore, you can extract a parametric policy from the optimization that MPC performs. The statement “Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy” is not true. The MPC will transfer better to other tasks that have the same dynamics, since it is not task specific. Given the learned dynamics model and the reward function you can act optimally in any new task as long as the learned dynamics are valid. # Related work My main concern with the related work section is that there a lot of literature on risk sensitive and optimism in the face of uncertainty (which is a subset of your method when c>0) in control, bandits, and some on *reinforcement learning* that has been neglected. # Uncertainty-Aware Model-Based Policy Optimization As said before, risk-sensitive in reinforcement learning has been done before and there’s even more work on control and bandits. For instance in [1] (page 5, paragraph (b)) has the same equation and they discuss the effect of the constant being negative or positive. More recent work has also used similar formulations [2]. This section mostly contains previous work, e.g., bootstrap rollout, policy gradient, using a deterministic policy ([3, 4, 5]). One thing that it’s still not clear from reading the paper, are you backpropagating the through the dynamics model, are you using a policy gradient method (REINFORCE, TRPO, PPO,… )? # Algorithm Summary One of the novelties introduced is the fact that the data used in each model comes from sampling a Poisson variable. However, this is not ablated in the results sections. Is it necessary? [6] Claims that there’s no need to use different data for the learned models. # Experiment The experiment section lacks from more complex environments, in this case the most complex is half-cheetah. Furthermore, given that 3 of the tasks are short horizon tasks you should probably also compare against model-based methods that build on top of policy gradients (e.g., [6]). It seems that some choices in the algorithm are not ablated: 1) use of poisson, 2) use of deterministic vs stochastic policy, 3) Is there a single risk that works across environments? Which environments are risk prone/adverse? 4) How about having c ~ N(0, 1), effectively modelling V as a gaussian? ----------------------------------- Overall, the paper is not mature enough to be accepted: there is not enough novelty, and the results lack of novelty, enough delta in performance from prior work, and have high variance. ------------------------------------ Minor/Typos: First paragraph: “trying model the transition” What does it mean that the accuracy is not satisfied? Why the related work on deep model-based reinforcement learning is called Deep Neural Networks? 3.2 third paragraph: “Next we provide a convergence convergence …” 3.3.2, first paragraph: “no matter how uncertain it may know about the world” Why the axis in the results section mean different things? [1] Risk-sensitive Reinforcement Learning. Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer. [2] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch [3] Continuous control with deep reinforcement learning. Timothy P. Lillicrap et. al. [4] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine [5] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee. [6] Model-Ensemble Trust-Region Policy Optimization. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel.