Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: # Introduction
The motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.” 

There has been work on learning a parametric policy from MPC. Therefore, you can extract a parametric policy from the optimization that MPC performs. The statement “Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy” is not true. The MPC will transfer better to other tasks that have the same dynamics, since it is not task specific. Given the learned dynamics model and the reward function you can act optimally in any new task as long as the learned dynamics are valid.

# Related work
My main concern with the related work section is that there a lot of literature on risk sensitive and optimism in the face of uncertainty (which is a subset of your method when c>0) in control, bandits, and some on *reinforcement learning* that has been neglected. 

# Uncertainty-Aware Model-Based Policy Optimization
As said before, risk-sensitive in reinforcement learning has been done before and there’s even more work on control and bandits. For instance in [1] (page 5, paragraph (b)) has the same equation and they discuss the effect of the constant being negative or positive. More recent work has also used similar formulations [2].
This section mostly contains previous work, e.g., bootstrap rollout, policy gradient, using a deterministic policy ([3, 4, 5]). One thing that it’s still not clear from reading the paper, are you backpropagating the through the dynamics model, are you using a policy gradient method (REINFORCE, TRPO, PPO,… )?

# Algorithm Summary
One of the novelties introduced is the fact that the data used in each model comes from sampling a Poisson variable. However, this is not ablated in the results sections. Is it necessary? [6] Claims that there’s no need to use different data for the learned models.

# Experiment
The experiment section lacks from more complex environments, in this case the most complex is half-cheetah. Furthermore, given that 3 of the tasks are short horizon tasks you should probably also compare against model-based methods that build on top of policy gradients (e.g., [6]). 

It seems that some choices in the algorithm are not ablated: 1) use of poisson, 2) use of deterministic vs stochastic policy, 3) Is there a single risk that works across environments? Which environments are risk prone/adverse? 4) How about having c ~ N(0, 1), effectively modelling V as a gaussian?

-----------------------------------

Overall, the paper is not mature enough to be accepted: there is not enough novelty, and the results lack of novelty, enough delta in performance from prior work, and have high variance.

------------------------------------

Minor/Typos:
First paragraph: “trying model the transition”
What does it mean that the accuracy is not satisfied?
Why the related work on deep model-based reinforcement learning is called Deep Neural Networks?
3.2 third paragraph: “Next we provide a convergence convergence …”
3.3.2, first paragraph: “no matter how uncertain it may know about the world”
Why the axis in the results section mean different things?



[1] Risk-sensitive Reinforcement Learning. Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer. 
[2] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch
[3] Continuous control with deep reinforcement learning. Timothy P. Lillicrap et. al.
[4] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine
[5] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee.
[6] Model-Ensemble Trust-Region Policy Optimization. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
The main contribution of this work is introducing the uncertainty-aware value function prediction into model-based RL, which can be used to balance the risk and return empirically. 

Methodology
This work uses a linear combination of the mean and standard deviation of value function to capture the uncertainty in learning state value function. 

It is not clear how to convert the objective function from Eq 2 (expectation over the initial state) to Eq 5 (expectation over all states). Those two objectives are not equal.

It is not clear how does the uncertainty in model prediction (dynamics and reward function)
can be alleviated through the proposed method, as claimed in the introduction. 
It seems the novelty part lies in considering the uncertainty of value function estimation. 
How does this relate to solving the limitation of model predictive control? 

What is the objective function for learning reward function r_\phi?


Experimental results:
The experiments are not sufficient to demonstrate the effectiveness of the proposed method. 
It would be more convincing to compare the proposed method with a few more model-based approaches on more tasks. The results of MBPO is better
the proposed POUM in one of two tasks. The performance of
MBPO on Reacher-v2 and Pusher-v2 is missing? 


Writing:
This paper has many typos and the presentation is not very clear.
- Section 4.1 "in Section 3.4," 
- Last paragraph in P3: convergence convergence analysis 
- "shows that POUM has a sample efficiency compared"

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: I enjoyed this paper overall, and I think the idea is a good one. However there remain significant issues with the paper that preclude me giving a good score. Firstly, there is almost no discussion of the environment model. Anyone who has worked on Model Based RL will tell you that the details here are crucial. This deserves a full discussion, and a comparison to other methods in the literature.

Next, the experimental results really aren't convincing. The dependence on random seeds is worrying, and isn't as common in model free algorithms as you claim, which are mostly robust to seeds (the good ones at least). The fact that the best policy is risk *averse* is very strange, since these are estimates combining both the epistemic and aleatoric uncertainty (which is somewhat unfortunate), which means being risk averse would lead the agent to not explore. That is very worrying and makes me think that something very strange is going on with the models. In fact since the policy is deterministic and the environment / rewards are practically speaking deterministic, the uncertainty here is actually mostly epistemic and so a c < 0 means the agent is disincentivized from exploring.

There should be more discussion about the fact that the policy is deterministic. Is this merely to make estimating V^pi easier?

"Next, we provide a convergence convergence analysis and show that maximizing this utility function
U(π) is equivalent to maximizing the unknown value function V (π)."

Word convergence appears twice in a row, but more importantly this is totally missing! Where is the analysis?

In the algorithm you write:
"Update {fb} and rˆφ using SGD"
But on what data? Presumably sampled from D but this isn't mentioned.

Is it the case that the policy is updated *only* using the model based rollouts? I.e., the reward signal is never used in the policy gradient but only used to train the models? If so, this seems quite fragile and I would like to see a comparison of different approaches here.

Table 2 is unreadable and needs to be explained.

It would appear that you are missing a reference to the very relevant UBE paper, which also deals explicitly with the uncertainty of the value function estimates: https://arxiv.org/abs/1709.05380
In fact I would be curious to see any way that these two approaches could be combined (though that would be follow up work).

review_title: Official Blind Review #2

Meta-review: The main contribution of this work is introducing the uncertainty-aware value function prediction into model-based RL, which can be used to balance the risk and return empirically. 

The reviewers generally agree that this paper addresses an interesting problem, but there are some concerns that remain (see reviewer comments). 

I also want to highlight that in terms of empirical results, it is insufficient to present results for 3 different random seeds. To highlight any kind of robustness, I suggest *at least* 10-20 different random seeds; otherwise the findings can/will be misleading. 
Decision: Reject