Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary
This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).

Review
The core contribution of the paper is an extension to STEVE that uses an idea from risk-averse RL of biasing the underlying estimators away from high-variance predictions, and adds a dynamic weight to that bias.

Strengths
- The addition of a risk-aversion term to STEVE is a good contribution to the literature on safe RL. While it may be possible to criticize this contribution as somewhat trivial, I am disinclined to do so, as finding simple ideas that are effective is the hallmark of an important contribution, in my opinion.
- Under the assumption that the baselines are fair, the empirical results show substantial improvements for a good selection of challenging tasks along three metrics: initial efficiency, asymptotic performance, and avoidance of negative outcomes.
- The paper provides a careful and detailed section on the relevant preliminaries, giving precise notation that is expanded step-by-step from basic actor-critic approaches all the way through the description of RAVE. (Although see my comment below about possibly making the notation more concise.)
- Overall, the paper’s presentation is clear and natural. (Although see my comment below about a strong need for proofreading.)

Weaknesses
- The paper opens with a claim that MBRL has worse performance than MFRL in environments with “noisy environments and long trajectories”. However, recent work [1] has shown that MBRL can outperform MFRL on many of the tasks considered in this paper using far fewer environment steps, even when training directly from pixels.
- The experiments are all based on observations of the true state vectors rather than pixels, which dramatically simplifies working with imagination rollouts, as the dimensionality is so much lower. This hides the high computational cost and modeling challenges involved in using imagination-based MBRL or hybrid RL approaches in real-world environments, and (in my opinion) is a major shortcoming of this line of research (not only of this paper). Since the DDPG family of algorithms doesn’t have to do rollouts, they have a strong advantage over this type of approach on more realistic settings where the true state is unknown and must be inferred from high-dimensional observations. If experiments are not going to be done using pixels, the discussion should at least mention the trade-offs involved between the proposed algorithm and the baselines in that setting.
- The core baseline of the paper is DDPG, which is unnecessarily weak. D4PG [2] came out a year and a half ago and has the same high-level properties as DDPG, but outperforms it on all of the Mujoco tasks considered in this paper. Additionally, it includes some of the ideas presented in this work, but in a model-free setting, including a distributional treatment and multi-step rollouts, so it is easy to imagine that some of the experimental gains presented here would be erased when using it as a baseline.
- The use of ensembling is another confounder in the experiments. Ensembling models almost always yields an improvement, so any technique that relies on some form of ensembling needs to additionally demonstrate that the gains presented are not solely due to ensembling. In this case, the experiments with STEVE and RAVE should minimally be performed with different numbers of models in their ensembles.
- (Minor) Similarly, the prediction horizon can have a large impact, and only STEVE and RAVE consider prediction horizons other than 1. Showing how the horizon affects performance would help make the comparisons more fair.
- Figure 1 shows a bias problem in some current approaches in a clear toy problem, but does not show whether RAVE addresses that problem. Even though this section precedes the presentation of RAVE, it is necessary to show that the proposed solution actually helps on the toy problem.
- There are a number of problems with the proposal for adaptively computing the alpha parameter. In general, such adaptive hyperparameters add a great deal of complexity to hyperparameter tuning, so such suggestions should be either strongly motivated by theory or by empirical results. Neither seems to be the case here.
  - Only one proposal for adaptive alpha (equation 13) is considered. Its justification is plausible, but it would be more convincing if other dynamic approaches were considered in the experiments. For example:
    - alpha(env_step) = alpha * env_step / max_env_step
    - alpha(s_t,a_t) = alpha * min{1.0, 1 / (Z * ||E[fhat(s_t,a_t)] - s_t+1||^2)}
    - The opposite of the proposed approach, where alpha starts high and gets lower over the course of training.
  - Indeed, a potentially useful quantity during evaluation, when s_t+1 is unknown, would be based on the variance of the predicted next state, rather than the difference of the expectation of the predicted next state and the true next state. E.g.:
    - alpha(s_t,a_t) = alpha / Var[fhat(s_t,a_t)]
    This formulation says that the precision of the prediction determines the confidence of the model, which is also intuitively reasonable (to me, anyway) and doesn’t rely on knowing the future.
  - (Minor) The adaptive variant appears to be only slightly better than alpha=0.5, but requires two hyperparameters that are unspecified in the main body of the paper — alpha and Z. The appendix lists those parameters, but no discussion is made on how much tuning was done to determine that pair of parameters.
  - (Minor) A plot of how the dynamic value of alpha changes during training would be useful, at least in the appendix.

Recommendation
In light of my comments above, I cannot currently recommend the acceptance of this paper at ICLR. However, I think that the core idea is likely to hold up under more careful experimental comparisons. If the authors submit a revised draft that addresses the substance of my concerns, I would be very likely to increase my rating. In particular, I would like to see much more careful experimental treatment of the idea, so that readers can have high confidence about the circumstances where RAVE is likely to be a good choice.

[1] Hafner et al., “Learning Latent Dynics for Planning from Pixels”. ICML 2019. https://arxiv.org/abs/1811.04551
[2] Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients”. ICLR 2018. https://arxiv.org/abs/1804.08617


Other Comments and Questions
- This paper needs a lot of proofreading. A few examples of errors that should be fixed before publication:
  - “equation. 2”: This should read “Equation 2”, “equation 2”, “Eq. 2” or “eq. 2”. The period in the last two options indicates that letters have been elided. The mistake of using a period where no elision has happened occurs throughout the paper.
  - “prophesy”: This is not the correct word. Just say “prediction”.
  - “image”: This is often used when the correct word would be either “imagine” or “imagination”.
  There are many other errors that could be fixed easily with the help of a native English speaker.
- The mathematical notation in sections 3 through 5 is precise, but it is also a bit heavy. Consider whether there would be any ambiguity added if, for example, {\hat Q}^DVE_{\zeta_s,\zeta_r,\zeta_d,\theta’,\phi’} were instead notated {\hat Q}^DVE_{\zeta,\theta’,\phi’}.
- Figure 2 adds nothing of value to the paper and should be removed.
- Consider comparing on DMControl, which is the same set of tasks as Mujoco, but the scores are standardized such that each task has a maximum reward of 1000 per episode.
- Why does DDPG get worse over time on the RoboSchool tasks? Without a clear explanation, it looks like a bug, and a bug like that calls into question the rest of the DDPG results as well.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper considers the problem of high function approximation errors facing the stochastic environments when trying to combining model-based reinforcement learning (RL) with model-free RL. The paper begins by showing that previous methods like model-based value expansion (MVE) and stochastic ensemble value expansion (STEVE) can perform even worse than the pure model-free DDPG algorithm in a rather noisy environment. Following the ideas of MVE and STEVE, It then proposes a risk averse value expansion (RAVE) to replace the target Q function in the actor-critic algorithm, which is built upon an ensemble of probabilistic models (PE) and adopt the lower confidence bound as a surrogate of the target value as in the risk-sensitive RL. A simple yet intuitive approach for adaptively selecting the confidence bound \alpha is also proposed. The experiments show that RAVE does improve over the state-of-the-art algorithms in several different environments, with a better draw-down control. In general, this paper is well-written and the idea of RAVE is novel as far as I know. But since I'm not very familiar with the specific literature of combing model-based and model-free RL, and since the idea of RAVE is relatively straightforward (but admittedly practically powerful and theoretically interesting), I choose to give a conservative accept to account the possibility that some existing works have followed very similar approaches. 

Some minor comments: 1) What is the loss of the negative log-likelihood in (4) for? 2) The authors may want to explain clearly what is the variance indicating in (8) more clearly, although one can guess that it is closely related to (5). 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this work , the authors combine model-based value expansion(MVE) with
model-free reinforcement learning and also take into account the high-order stochastic characteristics
of the environment to make the value expansion algorithm in modeling error propagation of dynamics risk-averse. 
They propose a novel Hybrid-RL method, namely the Risk Averse Value Expansion(RAVE), that uses an ensemble of probabilistic dynamics models to generate imaginative rollouts and to model risk aversion
of risks by estimating the lower confidence bound of the ensemble. They showed that by taking the risk (in terms of variance) of the dynamic propagation error into the account, RAVE can achieve comparable performance with other state-of-the-art baselines in benchmark experiments including MuJoCo and robo-school. Also they showed that RAVE reduced the variance of the return and thus prevented catastrophic consequences such as falling.

I found this work interesting as the authors try to take the uncertainty of probabilistic dynamics model into account for estimating the value function and its confidence bounds in model-free RL. Utilizing such confidence interval of values, they can have a way of doing exploration while being risk-averse. While similar approaches of modeling the mean and variance of Q functions can be found in some existing work, such as boot-strapped DQN (Osband'16), and efficient exploration via Bayesian DQN (Azizzadenesheli'18), none of these work model the variance of the Q-function using the error propagation of the dynamics model. Through extensive experiments, it has also demonstrated the effectiveness of RAVE in terms of achieving good performance, approximation error in value estimation, as well as robustness to failure in standard mujoco benchmarks, which provides readers some detailed understanding on how this risk-averse uncertainty modeling/propagation technique helps in exploration. However, while i found this idea interesting, it appears to me that the current work is still quite preliminary (without theoretical concentration inequality based variance bounds for guiding exploration), and empirically, it would be great to compare this method with the aforementioned bootstrapping/bayesian based approaches. My other major comment is on claiming the proposed method to be "risk-averse", because in RL, risk-averse methods are commonly known to not only optimize the expected return but also to provide guarantees to other moments of the return, such as variance of CVaR. However, while the method used variance of the expected value (due to the error propagation of the dynamics) for exploration, I am not seeing the risk-averse optimization criteria being studied here. Therefore calling the RL method risk-averse might be a little mis-leading.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a novel deep reinforcement learning algorithm at the intersection of model-based and model-free reinforcement learning: Risk Averse Value Expansion (RAVE). Overall, this work represents a significant but incremental step forwards for this "hybrid"-RL class of algorithms. However, the paper itself has significant weaknesses in its writing, analysis, and presentation of ideas.

The main strength of this work is its empirical section. The proposed algorithm is fairly compared against many relevant baselines on a variety of continuous control tasks, on both the Mujoco and Roboschool simulators, and demonstrates significant performance increases across all of them. The visualization in Figure 4 is interesting, and provides good insight into the issues with DDPG and STEVE, and the reasons for the success of RAME. Based on these results, the authors have convinced that RAME is a state-of-the-art algorithm.

However, in spite of its strong performance on benchmarks, I believe that this paper needs a significant overhaul/rewrite before publication.

One major criticism I have is on the authors' treatment of the different types of uncertainty. The introductory sections spend a large amount of time on the differences between aleatoric and epistemic uncertainty, and various other related concepts. But when it comes to the actual new algorithm, that the authors only barely touched upon the (very important) point that V[Q^DVE] is actually a *mix* of epistemic and aleatoric uncertainties. When minimizing this term, it's not clear whether epistemic or aleatoric uncertainty is actually being reduced. (Based on the experiments, which report only expected values, not CVaR or anything of the sort, it seems the authors care only about reducing epistemic uncertainty; if so, it's unclear why they choose to minimize a term which includes aleatoric uncertainty too.) A more principled understanding of this quantity seems essential to this line of work. Similarly, "risk" typically refers to aleatoric uncertainty, but the risk-senstive component of RAME computes the confidence lower bound w.r.t. a mix of aleatoric and epistemic uncertainty. Calling this algorithm "risk sensitive" is likely to generate confusion, in my opinion.

A few other points on presentation of ideas:
- Switching from a deterministic to a stochastic model is a trivial extension of MVE/STEVE, and way too much time is spent on this point (even going so far as to name a new algorithm!). Section 4 contains no new insight, beyond "deterministic models can be bad in stochastic environments", which is obvious. Consider re-thinking your experiments for this section to help readers better understand *what* goes wrong.
- In my opinion, RAME is as much a successor to TD3 as it is to STEVE. Taking the lower outcome of an ensemble of size 2 is equivalent to applying a penalty based on the stdev of the outcome distribution, with α=1. The introduction of the paper should be re-written to clearly point out that that RAME ~= TD3 + STEVE.
- I'd also like to see a lot more ablations, on at least a few environments. There are at least four factors that need to be teased apart: 1) deterministic vs stochastic model 2) lower confidence bound penalty 3) adaptive α for LCBP 4) STEVE reweighting. Does RAME require all of these elements to perform well? How do these elements interact?

Some feedback on the writing:
- There are various small factual errors. For example, in the very first paragraph, the Dyna algorithm is attributed to Kuturach et al, instead of Sutton (1990). (The algorithm from Kuturach is ME-TRPO.)
- The notation is not very good; there are lots of symbols flying around everywhere, and it makes the ideas (which are fundamentally very simple) a bit difficult to parse. For example, naming each parameter-set individually, everywhere, is unnecessary.
- There are some strange non-sequiturs and overall lack of flow.

I think that this work has a lot of potential, and am especially impressed by the empirical results. I recommend rejection in its current form, but hope to see a revised version of this work appear at a conference in the near future.
review_title: Official Blind Review #4

Meta-review: The authors propose to extend model-based/model-free hybrid methods (e.g., MVE, STEVE) to stochastic environments. They use an ensemble of probabilistic models to model the environment and use a lower confidence bound of the estimate to avoid risk. They found that their proposed method yields state-of-the-art performance over previous methods.

The valid concerns by Reviewers 1 & 4 were not addressed by the authors and although the authors responded to Reviewer 3, they did not revise the paper to address their concerns. The ideas and results in this paper are interesting, but without addressing the valid concerns raised by reviewers, I cannot recommend acceptance.
Decision: Reject