Title: RISK AVERSE VALUE EXPANSION FOR SAMPLE EFFI-CIENT AND ROBUST POLICY LEARNING

Abstract: Model-based Reinforcement Learning(RL) has shown great advantage in sampleefficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.

1. INTRODUCTION:

In contrast to the tremendous progress made by model-free reinforcement learning algorithms in the domain of games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019) , poor sample efficiency has risen up as a great challenge to RL, especially when interacting with the real world. Toward this challenge, a promising direction is to integrate the dynamics model to enhance the sample efficiency of the learning process (Sutton, 1991; Calandra et al., 2016; Kalweit & Boedecker, 2017; Oh et al., 2017; Racanière et al., 2017) . However, classic model-based reinforcement learning(MBRL) methods tend to lag behind the model-free methods(MFRL) asymptotically, especially in cases of noisy environments and long trajectories. The hybrid combination of MFRL and MBRL(Hybrid-RL for short) has attracted much attention due to this reason. A lot of efforts has been devoted to this field, including the dyna algorithm (Kurutach et al., 2018) , model-based value expansion (Feinberg et al., 2018) , I2A (Weber et al., 2017) , etc.

The robustness of the learned policy is another concern in RL. For stochastic environments, policy can be vulnerable to tiny disturbance and occasionally drop into catastrophic consequences. In MFRL, off-policy RL(such as DQN, DDPG) typically suffers from such problems, which in the end leads to instability in the performance including sudden drop in the rewards. To solve such problem, risk sensitive MFRL not only maximize the expected return, but also try to reduce those catastrophic outcomes (Garcıa & Fernández, 2015; Dabney et al., 2018a; Pan et al., 2019) . For MBRL and Hybrid-RL, without modeling the uncertainty in the environment(especially for continuous states and actions), it often leads to higher function approximation errors and poorer performances. It is proposed that complete modeling of uncertainty in transition can obviously improve the performance (Chua et al., 2018) , however, reducing risks in MBRL and Hybrid-RL has not been sufficiently studied yet.

In order to achieve both sample efficiency and robustness at the same time, we propose a new Hybrid-RL method more capable of solving stochastic and risky environments. The proposed method, namely Risk Averse Value Expansion(RAVE), is an extension of the model-based value expansion(MVE) (Feinberg et al., 2018) and stochastic ensemble value expansion(STEVE) (Buckman et al., 2018) . We systematically analyse the approximation errors of different methods in stochastic environments. We borrow ideas from the uncertainty modeling ( Chua et al. (2018) ) and risk averse reinforcement learning. The probabilistic ensemble environment model is used, which captures not only the variance in estimation(also called epistemic uncertainty), but also stochastic transition nature of the environment(also called aleatoric uncertainty). Utilizing the ensemble of estimations, we further adopt a dynamic confidence lower bound of the target value function to make the policy more risk-sensitive. We compare RAVE with prior MFRL and Hybrid-RL baselines, showing that RAVE not only yields SOTA expected performance, but also facilitates the robustness of the policy.

2. RELATED WORKS:

The model-based value expansion(MVE) (Feinberg et al., 2018 ) is a Hybrid-RL algorithm. Unlike typical MFRL such as DQN that uses only 1 step bootstrapping, MVE uses the imagination rollouts of length H to predict the target value. The assistance of environment model can greatly improve the sample efficiency at the start, but the precision of long term inference becomes limited asymptotically. In order to properly balance the contribution of the value expansion of different horizons, stochastic ensemble value expansion(STEVE) (Buckman et al., 2018) adopts an interpolation of value expansion of different horizon. The accuracy of the expansion is estimated through the ensemble of environment models as well as value functions. Ensemble of environment models also models the uncertainty to some extent, however, ensemble of deterministic model captures mainly epistemic uncertainty instead of stochastic transitions (Chua et al., 2018) .

The uncertainty or the function approximation error is typically divided into three classes (Geman et al., 1992) : the noise exists in the objective environment, e.g., the stochastic transitions, which is also called aleatoric uncertainty (Chua et al., 2018) . The model bias is the error produced by the limited expressive power of the approximating function, which is measured by the expectation of ground truth and the prediction of the model, in case that infinite training data is provided. The variance is the uncertainty brought by insufficient training data, which is also called epistemic uncertainty. Dabney et al. (2018b) discuss the epistemic and aleatoric uncertainty in their work and focus on the latter one to improve the distributional RL. Recent work suggests that ensemble of probabilistic model(PE) is considered as more thorough modeling of uncertainty (Chua et al., 2018) , while simply aggregate deterministic model captures only variance or epistemic uncertainty. The stochastic transition is more related to the noise(or aleatoric uncertainty), and the epistemic uncertainty is usually of interest to many works in terms of exploitation&exploration (Pathak et al., 2017; Schmidhuber, 2010; Oudeyer & Kaplan, 2009) . Other works adopt ensemble of deterministic value function for exploration (Osband et al., 2016; Buckman et al., 2018) .

Risks in RL typically refer to the inherent uncertainty of the environment and the fact that policy may perform poorly in some cases (Garcıa & Fernández, 2015) . Risk sensitive learning requires not only maximization of expected rewards, but also lower variances and risks in performance. Toward this object, some works adopt the variance of the return (Sato et al., 2001; Pan et al., 2019; Reddy et al., 2019) , or the worst-case outcome (Heger, 1994; Gaskett, 2003) in either policy learning (Pan et al., 2019; Reddy et al., 2019) , exploration (Smirnova et al., 2019) , or distributional value estimates (Dabney et al., 2018a ). An interesting issue in risk reduction is that reduction of risks is typically found to be conflicting with exploration and exploitation that try to maximize the reward in the long run. Authors in (Pan et al., 2019) introduce two adversarial agents(risk aversion and longterm reward seeking) that act in combination to solve this problem. Still, it remains quite tricky and empiristic to trade-off between risk-sensitive and risk-seeking(exploration) in RL. In this paper, we propose a dynamic confidence bound for this purpose.

A number of prior works have studied function approximation error that leads to overestimation and sub-optimal solution in MFRL. Double DQN(DDQN) (Van Hasselt et al., 2016) improves over DQN through disentangling the target value function and the target policy that pursues maximum value. In TD3 (Fujimoto et al., 2018) the authors suggest that systematic overestimation of value function also exists in actor-critic MFRL. They use an ensemble of two value functions, with the minimum estimate being used as the target value. Selecting the lower value estimation is similar to using uncertainty or lower confidence bound which is adopted by the other risk sensitive methods (Pan et al., 2019) , though they claimed different motivations.

3. PRELIMINARIES:



3.1. ACTOR-CRITIC MODEL-FREE REINFORCEMENT LEARNING:

The Markov Decision Processes(MDP) is used to describe the process of an agent interacting with the environment. The agent selects the action a t ∈ A at each time step t. After executing the action, it receives a new observation s t+1 ∈ S and a feedback r t ∈ R from the environment. As we focus mainly on the environments of continuous action, we denote the parametric deterministic policy that the agent uses to decide its action as a t = µ θ (s t ). Typically we add Gaussian exploration noise on top of the deterministic policy, such that we have a stochastic behavioral policy π θ,σ : S ×A → R. It is calculated as π θ,σ (s t , a t ) = p N (a t |µ θ (s t ), σ 2 ), where p N (x|m, σ 2 ) represents the probability density at x in a Gaussian distribution N (m, σ 2 ). As the interaction process continues, the agent generates a trajectory τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , ...) following the policy π θ,σ . For finite horizon MDP, we use the indicator d : S → {0, 1} to mark whether the episode is terminated. The objective of RL is to find the optimal policy π * to maximize the expected discounted sum of rewards along the trajectory. The value performing the action a with policy π at the state s is defined by

Q π (s, a) = E s0=s,a0=a,τ ∼π ∞ t=0 γ t r t ,

where 0 < γ < 1 is the discount factor. The value iteration in model-free RL tries to approximate the optimal value Q π * with a parametric value function Qφ by minimizing the Temporal Difference(TD) error, where φ is the parameter to be optimized. The TD error between the estimates of Q-value and the corresponding target values is shown in equation. 1, where φ is a delayed copy of the parameter φ, and a ∼ π θ , with θ being a delayed copy of θ (Lillicrap et al., 2015) .

EQUATION

To optimize the deterministic policy function in a continuous action space, deep deterministic policy gradient(DDPG) (Lillicrap et al., 2015) maximizes the value function (or minimizes the negative value function) under the policy µ θ with respect to parameter θ, shown in equation. 2.

L θ = -E τ [ t Qφ (s t , µ θ (s t ))]

(2)

3.2. ENVIRONMENT MODELING:

To model the environment in continuous space, an environment model is typically composed of three individual mapping functions: fr,ζr : 

S × A × S → R,

and this process might go on to generate a complete imagined trajectory of [s t , a t , rt , ŝt+1 , ...].

The neural network is commonly used as an environment model due to its powerful express ability.

To optimize the parameter ζ we need to minimize the mean square error(or the cross entropy) of the prediction and the ground truth, given the trajectories τ under the behavioral policy.

3.3. UNCERTAINTY AWARE PREDICTION:

The deterministic model approximates the expectation only. As we mentioned in the previous sections, overestimation is attributed to the error in function approximation. Following Chua et al. (2018) , we briefly review different uncertainty modeling techniques.

Probabilistic models output a distribution (e.g., mean and variance of a Gaussian distribution) instead of an expectation. We take the reward component of the environment model as an example, the probabilistic model is written as r ∼ N ( fr,ζr , σ2 r,ζr ), and the loss function is the negative log likelihood(equation. 4). 

L ζr = -E τ log p N (

We define the operator Êx and Vx as the average and the variance on the ensemble of xes respectively. As proposed by (Chua et al., 2018) , the variance σ2 in equation. 4 mainly captures the aleatoric uncertainty, and the variance V mainly captures the epistemic uncertainty. 

EQUATION

3.5. STOCHASTIC ENSEMBLE VALUE EXPANSION:

Selecting proper horizon H for value expansion is important to achieve high sample efficiency and asymptotic accuracy at the same time. Though the increase of H brings increasing prophecy, the asymptotic accuracy is sacrificed due to the increasing reliance on the environment model.

In STEVE (Buckman et al., 2018) , interpolating the value expansions QMVE ζr,ζs,ζ d ,θ ,φ ,H of different H ∈ [0, H max ] is proposed. The weight for each horizon is decided by the inverse of its variance, which is calculated by estimating an ensemble of values switching the combination of parameters in environment model f and value function Qφ . Through our notation, STEVE can be written as equation. 8. We apply the learning methods including DDPG, MVE, STEVE to this environment, and plot the changes of estimate values at the starting point(see fig. 5 ).

QSTEVE (r:

The results show that, in the deterministic environment, the Q-values estimated by all methods converge to the ground-truth asymptotically in such a simple environment. However, after adding the noise, previous MFRL and Hybrid-RL methods show various level of overestimation. The authors of (Feinberg et al., 2018) have claimed that value expansion improves the quality of estimated values, but MVE and STEVE actually give even worse prediction than model-free DDPG in the stochastic environment. A potential explanation is that the overall overestimation comes from the unavoidable imprecision of the estimator (Thrun & Schwartz, 1993; Fujimoto et al., 2018) , but Hybrid-RL also suffers from the approximation error of the dynamics model. When using a deterministic environment model, the predictive transition of both environments would be identical, because the deterministic dynamics model tends to approximate the expectation of next states(e.g, fs,ζs (s t = 0, a t > 0) = 1.0, fs,ζs (s t = 1.0, a t > 0) = 2.0). This would result in the same value estimation for k = 0 and k = 1 for both value expansion methods, but the ground truth of Q-values are different in these two environments. As a result, the deterministic environment introduces additional approximation error, leading to more severe overestimation. 

We apply the distributional expansion starting from r t , s t+1 to acquire the trajectory τζr,ζs,ζ d ,θ ,H (r t , s t+1 ) = (r t , s t+1 , ãt+1 , rt+1 , st+2 , ..., st+H+1 , ãt+H+1 ), based on which we write DVE as equation. 10.

EQUATION

We then keep track of an ensemble of the combination of the parameters {ζ r , ζ s , ζ d , φ }. For each group of parameters we use an ensemble of N parameters, which gives us 4N parameters in all.

Then we select a random combination of four integers {i, j, k, l}, which gives the parameter combination of {ζ r,i , ζ s,j , ζ d,k , φ l }. By switching the combination of integers we acquire an ensemble of DVE estimation. Then we count the average and the variance on the ensemble of DVE, and by subtracting a certain proportion(α) of the standard variance, we acquire a lower bound of DVE estimation. We call this estimation of value function the α-confidence lower bound(α-CLB), written as equation. 11. 

Qα:

Subtraction of variances is commonly used in risk-sensitive RL (Sato & Kobayashi, 2000; Pan et al., 2019; Reddy et al., 2019) . The motivation is straight forward, we try to suppress the utility of the high-variance trajectories, in order to avoid possible risks. However, the α here is left undecided. We will come to this problem in the next part. 

While adopting the lower confidence bound may introduce the bias of underestimation, it makes the policy less preferable to actions with large variance of return.

5.2. ADAPTIVE CONFIDENCE BOUND:

An unsolved problem in RAVE is to select proper α. The requirement of risk aversion and exploration is somehow competing: risk aversion seek to minimize the variance, while exploration searches states with higher variance. To satisfy both requirements, previous work proposed two competing agents, and each will make decision for a short amount of time (Pan et al. (2019) ). Here we propose another solution to this problem. We argue that the agent needs to aggressively explore at the beginning, and it should get more risk sensitive as the model converges. A key indicator of this is the epistemic uncertainty. The epistemic uncertainty measures how well our model get to know the state space. In MBRL and Hybrid-RL, there is a common technique to easily monitor the epistemic uncertainty, by evaluating the ability of the learned environment model to predict the consequence of its own actions (Pathak et al., 2017) .

Following this motivation, we set the confidence bound factor to be related to its current state and action, denoted as α(s, a). We want α(s, a) to be larger when the environment model could perfectly predict the state to get more risk sensitive, and smaller when the prediction is noisy to allow more exploration. We have

EQUATION

)

where Z is a scaling factor for the prediction error. With a little abuse of notations, we use α here to represent a constant hyperparameter, and α(s, a) is the factor that is actually used in α-CLB. α(s t , a t ) picks the value near zero at first, and gradually increases to α with the learning process.

6. EXPERIMENTS AND ANALYSIS:

We evaluate RAVE on continuous control environments using the MuJoCo physics simulator (Todorov et al., 2012) . The baselines includes the model-free DDPG and STEVE that currently yields the SOTA Hybrid-RL performance in MuJoCo. We also align our performance with the SOTA MFRL methods including twin delayed deep deterministic (TD3) policy gradient algorithm (Fujimoto et al., 2018) , soft actror-critic(SAC) algorithm (Haarnoja et al., 2018) , and proximal policy optimization(PPO) (Schulman et al., 2017) , using the implementation provided by the authors. To further demonstrate the robustness in complex environments, we also evaluate RAVE on OpenAI's Roboschool (Klimov & Schulman, 2017) , where STEVE has shown a large improvement than the other baselines.We detail hyper-parameters and the implementation in the supplementary materials.

6.1. EXPERIMENTAL RESULTS:

We carried out experiments on eight environments shown in fig. 3 . Among the compared methods, PPO is the only on-policy updating method, which has very poor sample-efficiency compared with either STEVE or off-policy MFRL methods, as PPO needs a large batch size to learn stably (Haarnoja et al., 2018) . DDPG achieves quite high performance in HalfCheetah-v1 and Swimmer-v1, but fails on almost all the other environments, especially on challenging environments such as Humanoid. In Hopper-v1 and Walker2d-v1, STEVE can not compare with TD3 and SAC, which yields quite good performance in many environments. However, RAVE performed favorably in most environments in both asymptotic performance and the rising speed(meaning sample efficiency), except for HalfCheetah-v1 and Swimmer-v1, where DDPG has already achieved satisfying performance and the margin between DDPG and Hybrid-RL is not that large.

6.2. ANALYSIS:

Distribution of Value Function Approximation. To investigate whether the proposed method predicts value function more precisely, we plot the of the predicted values Q against the ground truth values of Hopper-v1 in fig. 4 . The ground truth value here is calculated by directly adding the rewards of the left trajectory, thus it is more like a monte carlo sampling from ground truth distribution, which is quite noisy. To better demonstrate the distribution of points, we draw the confidence ellipses representing the density. The points are extracted from the model at environment steps of 1M. In DDPG and STEVE, the predicted value and ground truth aligned poorly with the ground truth, while RAVE yields better alignments, though a little bit of underestimation. 

:



Figures and Tables:

INVESTIGATION OF THE APPROXIMATION ERROR IN STOCHASTICENVIRONMENTSTo thoroughly investigate the impact of aleatoric uncertainty on hybri-RL methods, we construct a demonstrative toy environment(fig.1(a)). The agent starts from s 0 = 0, chooses an action a t from A = [-1, +1] at each time step t. The transition of the environment is s t+1 = s t + at |at| + k • N (0, 1). We compare two different environments:k = 0 and k = 1, where k = 0 represents the deterministic transition, and k = 1 represents the stochastic transition. The episode terminates at (|s| > 5 ), where the agent acquires a final reward. The agent gets a constant penalty(-100) at each time step to encourage it to reach the terminal state as soon as possible. Note that the deterministic environment actually requires more steps in expectation to reach |s > 5| compared with the stochastic environment, thus the value function at the starting point of k = 1 (Ground truth = 380+) tends to be lower than that of k = 0(Ground truth = 430+).

Figure 1: Curves of estimated Q values in the toy environment at the starting point over the environment steps.Each experiment is run four times.

two improvements based on MVE and STEVE. Firstly, we apply an ensemble of probabilistic models (PE) to enable the environment model to capture the uncertainty over possible trajectories. Secondly, inspired by risk sensitive RL, we calculate the confidence lower bound of the target value(fig 1(c)).

Figure 2: A casual illustration of three Distributional Value Expansion(DVE) trajectories. Before introducing RAVE we start with the Distributional Value Expansion(DVE). Compared with MVE that uses a deterministic environment model and value function, DVE uses a probabilistic environment model, and we independently sample the reward and the next state using the probabilistic environment models(see equation 9 and fig. 2). st+2 ∼ N ( fs,ζs (s t+1 , ãt+1 = µ θ (s t+1 )), σ2 s,ζs (s t+1 , ãt+1 )) rt+1 ∼ N ( fr,ζr (s t+1 , ãt+1 , st+2 ), σ2 r,ζr (s t+1 , ãt+1 , st+2 )) d(s t+2 ) ∼ N ( fd,ζ d (s t+2 ), σ2 d,ζ d (s t+2 ))

Figure 3: Average return over environment frames in MuJoCo and Roboschool environments. Each experiment is run four times.

Figure 4: Confidence ellipses of the distribution of estimated values Q versus the ground truth in Hopper-v1. The points are extracted from environment steps of 1M, each includes statistics from 10,000 points. The x-axis and y-axis represent the statistical cumulative discounted returns(ground truth) and the predictive Q-values(both are normalized), respectively..

r t | fr,ζr (s t , a t , s t+1 ), σ2 r,ζr (s t , a t , s t+1 )) (4) Ensemble of deterministic(DE) models maintains an ensemble of parameters, which is typically trained with different training samples and different initial parameters. E.g, given the ensemble of parameters ζ r,1 , ζ r,2 , ..., ζ r,N , the expectation and the variance of the prediction is acquired from equation. 6

Table content: <table><tr><td>Êζr fr,ζr =</td><td>1 N</td><td>i</td><td>fr,ζr,i , Vζr fr,ζr =</td><td>1 N</td></tr></table>

Ensemble of probabilistic models(PE) keeps track of an collection of distributions {N ( fr,ζr,i , σ2 r,ζr,i )}, i ∈ [1, N ], which can further give the estimation of both uncertainties. A sampling form PE goes as follow Sample i unformly from[1, N ], then sample r ∼ N ( fr,ζr,i , σ2 on the sampled trajectory τζr,ζs,ζ d ,θ ,H (r t , s t+1 ), which is expressed in equation 7.

Table content: <table><tr><td>Qtarget (r</td></tr></table>

, s t+1 ) = Êζr,ζs,ζ d ,φ QDVE ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 ) -α Vζr,ζs,ζ d ,φ QDVE ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 )

Table content: <table/>

Finally, we define RAVE, which adopts the similar interpolation among different horizons as STEVE based on DVE and CLB, shown in equation. 13. Qtarget (r t , s t+1 ) ← QRAVE (r t , s t+1 ) =

Table content: <table><tr><td/><td>Hmax H=0 ω H</td><td colspan="2">Qα-CLB H Hmax H=0 ω H (r t , s t+1 )</td><td>,</td></tr><tr><td>with ω H = Vζr,ζs,ζ d ,φ</td><td colspan="2">ζr,ζs,ζ d ,θ ,φ ,H (r t , s t+1 ) QDVE</td><td>-1</td></tr></table>

Peer review (iclr2020):
Summary This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training). Review The core contribution of the paper is an extension to STEVE that uses an idea from risk-averse RL of biasing the underlying estimators away from high-variance predictions, and adds a dynamic weight to that bias. Strengths - The addition of a risk-aversion term to STEVE is a good contribution to the literature on safe RL. While it may be possible to criticize this contribution as somewhat trivial, I am disinclined to do so, as finding simple ideas that are effective is the hallmark of an important contribution, in my opinion. - Under the assumption that the baselines are fair, the empirical results show substantial improvements for a good selection of challenging tasks along three metrics: initial efficiency, asymptotic performance, and avoidance of negative outcomes. - The paper provides a careful and detailed section on the relevant preliminaries, giving precise notation that is expanded step-by-step from basic actor-critic approaches all the way through the description of RAVE. (Although see my comment below about possibly making the notation more concise.) - Overall, the paper’s presentation is clear and natural. (Although see my comment below about a strong need for proofreading.) Weaknesses - The paper opens with a claim that MBRL has worse performance than MFRL in environments with “noisy environments and long trajectories”. However, recent work [1] has shown that MBRL can outperform MFRL on many of the tasks considered in this paper using far fewer environment steps, even when training directly from pixels. - The experiments are all based on observations of the true state vectors rather than pixels, which dramatically simplifies working with imagination rollouts, as the dimensionality is so much lower. This hides the high computational cost and modeling challenges involved in using imagination-based MBRL or hybrid RL approaches in real-world environments, and (in my opinion) is a major shortcoming of this line of research (not only of this paper). Since the DDPG family of algorithms doesn’t have to do rollouts, they have a strong advantage over this type of approach on more realistic settings where the true state is unknown and must be inferred from high-dimensional observations. If experiments are not going to be done using pixels, the discussion should at least mention the trade-offs involved between the proposed algorithm and the baselines in that setting. - The core baseline of the paper is DDPG, which is unnecessarily weak. D4PG [2] came out a year and a half ago and has the same high-level properties as DDPG, but outperforms it on all of the Mujoco tasks considered in this paper. Additionally, it includes some of the ideas presented in this work, but in a model-free setting, including a distributional treatment and multi-step rollouts, so it is easy to imagine that some of the experimental gains presented here would be erased when using it as a baseline. - The use of ensembling is another confounder in the experiments. Ensembling models almost always yields an improvement, so any technique that relies on some form of ensembling needs to additionally demonstrate that the gains presented are not solely due to ensembling. In this case, the experiments with STEVE and RAVE should minimally be performed with different numbers of models in their ensembles. - (Minor) Similarly, the prediction horizon can have a large impact, and only STEVE and RAVE consider prediction horizons other than 1. Showing how the horizon affects performance would help make the comparisons more fair. - Figure 1 shows a bias problem in some current approaches in a clear toy problem, but does not show whether RAVE addresses that problem. Even though this section precedes the presentation of RAVE, it is necessary to show that the proposed solution actually helps on the toy problem. - There are a number of problems with the proposal for adaptively computing the alpha parameter. In general, such adaptive hyperparameters add a great deal of complexity to hyperparameter tuning, so such suggestions should be either strongly motivated by theory or by empirical results. Neither seems to be the case here. - Only one proposal for adaptive alpha (equation 13) is considered. Its justification is plausible, but it would be more convincing if other dynamic approaches were considered in the experiments. For example: - alpha(env_step) = alpha * env_step / max_env_step - alpha(s_t,a_t) = alpha * min{1.0, 1 / (Z * ||E[fhat(s_t,a_t)] - s_t+1||^2)} - The opposite of the proposed approach, where alpha starts high and gets lower over the course of training. - Indeed, a potentially useful quantity during evaluation, when s_t+1 is unknown, would be based on the variance of the predicted next state, rather than the difference of the expectation of the predicted next state and the true next state. E.g.: - alpha(s_t,a_t) = alpha / Var[fhat(s_t,a_t)] This formulation says that the precision of the prediction determines the confidence of the model, which is also intuitively reasonable (to me, anyway) and doesn’t rely on knowing the future. - (Minor) The adaptive variant appears to be only slightly better than alpha=0.5, but requires two hyperparameters that are unspecified in the main body of the paper — alpha and Z. The appendix lists those parameters, but no discussion is made on how much tuning was done to determine that pair of parameters. - (Minor) A plot of how the dynamic value of alpha changes during training would be useful, at least in the appendix. Recommendation In light of my comments above, I cannot currently recommend the acceptance of this paper at ICLR. However, I think that the core idea is likely to hold up under more careful experimental comparisons. If the authors submit a revised draft that addresses the substance of my concerns, I would be very likely to increase my rating. In particular, I would like to see much more careful experimental treatment of the idea, so that readers can have high confidence about the circumstances where RAVE is likely to be a good choice. [1] Hafner et al., “Learning Latent Dynics for Planning from Pixels”. ICML 2019. https://arxiv.org/abs/1811.04551 [2] Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients”. ICLR 2018. https://arxiv.org/abs/1804.08617 Other Comments and Questions - This paper needs a lot of proofreading. A few examples of errors that should be fixed before publication: - “equation. 2”: This should read “Equation 2”, “equation 2”, “Eq. 2” or “eq. 2”. The period in the last two options indicates that letters have been elided. The mistake of using a period where no elision has happened occurs throughout the paper. - “prophesy”: This is not the correct word. Just say “prediction”. - “image”: This is often used when the correct word would be either “imagine” or “imagination”. There are many other errors that could be fixed easily with the help of a native English speaker. - The mathematical notation in sections 3 through 5 is precise, but it is also a bit heavy. Consider whether there would be any ambiguity added if, for example, {hat Q}^DVE_{zeta_s,zeta_r,zeta_d,	heta’,phi’} were instead notated {hat Q}^DVE_{zeta,	heta’,phi’}. - Figure 2 adds nothing of value to the paper and should be removed. - Consider comparing on DMControl, which is the same set of tasks as Mujoco, but the scores are standardized such that each task has a maximum reward of 1000 per episode. - Why does DDPG get worse over time on the RoboSchool tasks? Without a clear explanation, it looks like a bug, and a bug like that calls into question the rest of the DDPG results as well.