Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
Authors: Vincent Mai
Kaustubh Mani
Liam Paull
Venue: ICLR 2022 Spotlight (2022)
Keywords: Deep reinforcement learning
uncertainty estimation
inverse-variance
heteroscedastic

Abstract: In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.

Published as a conference paper at ICLR 2022
SAMPLE EFFICIENT DEEP REINFORCEMENT LEARN- ING VIA UNCERTAINTY ESTIMATION
Vincent Mai, Kaustubh Mani and Liam Paull
Robotics and Embodied AI Lab
Mila Quebec Institute of Artificial Intelligence Université de Montréal, Quebec, Canada
{vincent mai, kaustubh, mani, 1 liam. e
ABSTRACT
In model-free deep reinforcement learning (RL) algorithms, using noisy value es- timates to supervise policy evaluation and optimization is detrimental to the sam- ple efficiency. As this noise heteroscedastic, its effects can be mitigated using uncertainty-base -base weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We pro- pose a method whereby two complementary uncertainty estimation methods ac- count for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.
INTRODUCTION
Deep reinforcement learning (DRL) methods have proven to be powerful at solving sequential decision-making tasks across domains (Silver et al.) 2016 OpenAl et al. 2019). Combining the flexibility of the reinforcement learning framework with the representational power of deep neural networks enables policy optimization in complex and high-dimensional environments with unknown dynamics models to maximize the expected cumulative reward (Sutton & Barto 2018).
An important limitation of DRL methods is their sample inefficiency: an enormous amount of data is necessary and makes training expensive. This makes applying DRL in the real world challenging, for example in robotics (Sünderhauf et al. [018 Dulac-Arrold et al., 2019). In tasks like manipulation, sample collection is a slow and costly process (Liu et al., 2021). is even more expensive in risk- averse applications like autonomous driving (Kothari et al. (02T).
Among the current -fate-of-the-arr approaches to improve learning efficiency, promising direction is to exploit the prevalence of uncertainty in the underlying DRL algorithm. By adopting a Bayesian framework, we can consider the sampled quantities in DRL as random variables and leverage infor- mation about their distributions to improve the learning process (Osband et al. 2018). In this paper, we consider the particular problem of unreliable supervision in the temporal difference update and the policy optimization process. In DRL, value predictions are used to supervise the training: in tem- poral difference-band labels; in actor-critic frameworks, the policy is trained to optimize them. That these value predic- tions are noisy slows the learning and brings instability (Kumar et al., 2019 2020). The amount of noise in the supervision depends on the uncertainty of the value prediction, which evolves during the training process and depends on the state (and action) evaluated. It is therefore heteroscedastic.
algorithms, they are included bootstrapped target values which are used as
While there is an extensive body of literature focused on using the uncertainty of the value prediction guide the exploration//xplottaiee
trade-off (Dearden et al. 1998 Sttress] 2001]] Osband et al.)
2016 Pathak et al.) 2017); Chen et al.] [017]] Osband et al. 2018 Fortunato et al.) 2019] Osband
ct al. 2019 2019| Flennerhag et | Clements et al., 2020 Jain et al. 2021 Aravindan & Lee 2021). l).
there are very few works focused on leveraging it to mitigate the impact of unreliable supervision.
"Canada CIFAR AT Chair

---

Published as a conference paper at ICLR 2022
Distributional RL (Bellemare et al. 2017) considers the value function a distribution to be learned as such. It is orthogonal to our proposition: we consider the uncertainty of the labels used to learn scalar value function. In the offline RL setting, where the dataset is limited, incertainty-weighee actor-critic (UWAC) (Wu et al.
weighting to discard out-of--istribue state-action pairs using Monte Carlo dropout (Gal & Ghahramani 2016) for uncertainty estimation.
(202)) uses
Closer to our work, Lee et al. 2021 propose SUNRISE, in which each sample of the Bellman backup in the TD update step weighted to lower the importance of the targets which have a high standard deviation. The weights w(s', a') are computed based on a sigmoid of the negative standard deviation asta aa') scaled by a temperature hyperparameter T, and then offset such that they are between 0.5 and I: w(s a) = (--Q((1))))) (s', a') T + 0.5. The uncertainty of the target is estimated by sampled ensembles. While SUNRISE proposes other contributions such as an exploration bonus, the heuristic weighting scheme and the limitations of sampled ensembles in capturing the predictive uncertainty leave space for improvement in the mitigation of the effects of unreliable supervision.
We propose unverse-variance reinforcement learning (IV-RL). IV-RL also uses weights to reduce the importance of uncertain targets training. It does so by addressing the problem from two view- points. First, we use variance networks (Kendall & Gal 2017). whose loss function for regression the negative log-likelihood instead of the L2 distance. For given state-action pair (s, a), the network learns the target's 's noise, due for example to the stochasticity of the environment or the up- date of the policy. It then naturally down- Second, we use variance ensembles (Lakshminarayanal et al. (017) to estimate the uncertainty of the target due to the prediction of Q(s'. .') during the cempora--differene
the highly noisy samples in the training process.
update. We merge the predicted variances of several variance networks through a mixture of Gaussians, which has been shown to be a reliable method to capture predictive uncertainty (Ovadia et al., 2019). We then use Batch Inverse-Variance (BIV) (Mai et al. 202I), which has been shown to sigmficcntly improve the performance of supervised learning with neural networks in the case of heteroscedastic regression. BIV is normalized, which makes it ideal to cope with different and time-varying scales of variance. We show analytically that these two different variance predictions for the target are complementary and their combination leads to consistent and significant improvements in the sample efficiency and overall performance of the learning process.
In summary, our contribution is threefold:
1. We present a systematic analysis of the sources of uncertainty in the supervision of model- free DRL algorithms. We show that the variance of the supervision noise can be estimated with two complementary methods: negative log-likelihood and variance ensembles.
2. We introduce IV-RL, a framework that accounts for the uncertainty of the supervisory sig- nal by weighting the samples in a mini-batch during the agent's training. IV-RL uses BIV, a weighting scheme that is robust to poorly calibrated variance estimation]]
3. Our experiments show that IV-RL can lead to significant improvements in sample efficiency when applied to Deep Q-Networks (DQN) (Mnih et al.] 2013) and Soft-Actor Critic (SAC) (Haarnoja et al. 2018).
In section 2 we introduce BIV as a weighting scheme for heteroscedastic regression, and variance ensembles as an uncertainty estimation method. We analyse the sources of uncertainty in the target in section 3] where we also introduce our IV-RL framework. We finally present our experimental results in section
2
BACKGROUND AND PRELIMINARIES
2.1 BATCH INVERSE- VARIANCE WEIGHTING
In supervised learning with deep neural networks, it is assumed that the training dataset consists of inputs na and labels yı- However, depending on the label generation process, the label may be noisy. In regression, we can model the noise as normal distribution around the true label: en = yk + 6% with 64 ~ (11 of) In some cases, the label generation process leads to different variances for the label noises. When these variances can be estimated, each sample is triplet (Ix, Dr, of).
'The code for IV-RI is available at
2

---

Published as a conference paper at ICLR 2022
Batch Inverse- Variance (BIV) weighting Mai et al. 2021) leverages the additional information of which is assumed to be provided, to learn faster and obtain better performance in the case of het- eroscedastic noise the labels. Applied to L2 loss, it optimizes the neural network parameters using the following loss function for a mini-batch of size
K Σ A==
(Jo(IE) yk k=0 of +
CBV (D,))
o +
This is normalized weighted sum with weights WA = 1 + E). Normalizing in the mini-batch enables control of the effective learning rate, especially in cases where the training data changes over time, such as in DRL. By focusing on the relative scale of the variances instead of their absolute value, it also provides robustness to poor scale-calibration of the variance estimates.
As explained in Mai et al. (2021). is hyperparameter that is important for the stability of the optimization process. A higher & limits the highest weights, thus preventing very small variance samples from dominating the loss function for a mini-batch. However, by controlling the discrimi- nation between the samples, is also key when the variance estimation is not completely trusted. provides control of the effective mini-batch size EBS, according to:
2
E
2
Σ
K
Wk
O +
E)
EBS- = ΣE w
Σx
2 (o + +2²
For example, imagine a mini-batch where most samples have very high variances, and only one has a very low variance. If E O, this one low-variance sample is effectively the only one count in the mini-batch, and EBS tends towards 1. Increasing & would give more relative importance to the other samples, thus increasing EBS. With very high compared to the variances, all weights equal, and EBS tends towards K; in this case, the BIV loss tends towards L2.
Tuning the & parameter The simplest way to set hyperparameter. However, the best value is difficult to evaluate priori and can change when the profile of variances changes during a task, as is the case in DRL.
choose a constant value as an additional
It is instead possible to numerically compute the value of & which ensures a minimal EBS for each mini-batch. This method allows automatically adapt the different scales of variance while ensuring a minimal amount of information from the dataset is accounted for by the algorithm. The minimal EBS is also a as it is simply fraction of the original batch size. As such, it can be set as a batch size ratio.
but it is easier set and to transfer among environments,
2.2 ESTIMATING THE UNCERTAINTY OF NEURAL NETWORK PREDICTION
The predictive uncertainty of a neural network can be considered as the combination of aleatoric and epistemic uncertainties (Kendall & Gal 2017). Alcatoric uncertainty is irreducible and characterizes the non-deterministic relationship between the input and the desired output. Epistemic uncertainty is instead related to the trained model: it depends the information available in the training data, the model's 's capacity to retain it, and the learning algorithm (Hüllermeier & Waegeman, (001). There is currently principled way quantify the amount of task-related information present in the input, the training data, or the model. The state of the art for predictive uncertainty estimation instead relies on different sorts of proxies. These sometimes capture other elements, such the noise of the labels, which we can use to our advantage. We focus here on the relevant methods to our work.
2.2.1 SAMPLED ENSEMBLES
Several networks independently train an ensemble of size that can be interpreted as a distribution over predictions. The expected behaviour is that different networks will only make similar pre- dictions if they were sufficiently trained for a given input. The sampled variance of the networks' outputs is thus interpreted as epistemic uncertainty. It is possible to include a random Bernoulli mask
We replaced in Mai et al. 2001 with to avoid confusion with the reinforcement learning conventions.
3

---

Published as a conference paper at ICLR 2022
of probability p to each training sample, ensure that each network undergoes different training. This method, used by CClements et al. (2020) and Lee et al. (202T), has the same principle as single network Monte- Carlo dropout (Gal & Ghahramani] 2016). As the variance is sampled, the standard deviation is usually on the same scale as the prediction.
When used at the very beginning of the training process, sampled ensembles present one particular challenge: as the networks are initialized, they all predict small values. The initial variance, instead of capturing the lack of knowledge, is then underestimated. To address this problem, Randomized Prior Functions (RPFs) enforce prior in the variance by pairing each network with a fixed, untrained network which adds its predictions to the output (Osband et al. 2019). RPFs ensure high variance at regions of the input space which are not well explored, and lower variance when the trained networks have learned to compensate for their respective prior and converge to the same output. The scale of the prior is a hyper seeer--rrmeterr
2.2.2 VARIANCE NETWORKS
With variance networks, the uncertainty is predicted using loss attenuation (Nix & Weigend 1994, Kendall & Gal 2017). A network outputs two values in its final layer given input the predicted mean (x) and variance (a)). Over a minibatch D of size K, the network parameters are optimized by minimizing the negative log- lillliiieiood of heteroscedastic Gaussian distribution:
K Σ (mO(IR)) - v(xx)) + In (TL))
(D,
(z(2))
Variance networks naturally down- weight the labels with high variance in the optimization process. The variance prediction is trained from the error between (1() and y(x)). Hence, noise on the labels or changes in the regression task over time will also be captured by loss attenuation. As the variance is predicted by neural network, it may not be well-calibrated and may be overestimated (Kuleshov) al. 2018 Levi et al. 2020 Bhatt et al.) 2021). This can (1) give wrong variance estimates but also (2) affect the learning process by ignoring a sample if the variance estimate is too high.
2.2.3 VARIANCE ENSEMBLES
Variance ensembles, or deep ensembles (Lakshminarayanan et al. 2017). are ensembles composed of variance networks. The predictive variance is given by Gaussian mixture over the variance predictions of each network in the ensemble. This method, when trained, is able to capture uncer- tainty more reliably than others (Ovadia et al., 2019), with = networks in the ensemble being sufficient. Similarly to sampled ensembles, variance ensembles suffer from underestimated early epistemic variance estimation. However, compared to variance networks, they seem empirically less prone calibration issues (1) in the final variance estimation because the mixture of Gaussians dampens single very high variances, and (2) in the learning process because even if one network does not learn correctly, the others will.
For better readability, we will use the terms var-network and var-ensemble in the rest of the paper.
2.3 UNCERTAINTY AND EXPLORATION IN DRL
While our work focuses on using uncertainty estimates mitigate the impact of unreliable super- vision, we can take advantage of the structure in place to better drive the exploration//xplootaiii this method,
trade-off. In particular, we used BootstrapDQN (Osband et al. (016) for exploration. a single network is sampled from an ensemble at the beginning of each episode to select the action. This method is improved with the previously described RPFs (Osband et al. 2018). In continuous settings, we instead followed et al. (2021) and added an Upper Confidence Bound (UCB) explo- ration bonus based on uncertainty prediction. As the variance in UCB is added to Q-values, must be calibrated: we evaluate it with sampled ensembles. Possible interactions between exploration strategies and IV-RL are discussed in Appendix H
UUsing RPFs in the context of DRL can destabilize the exploration due to significant initial bias added to the value of each state-action. It must be compensated for with uncertainty-awant exploration, such as Bootstrap (Osband et al. 2016), where it leads significant improvement in performance for exploration- hased tasks.
4

---

Published as a conference paper at ICLR 2022
3 INVERSE- VARIANCE REINFORCEMENT LEARNING
3.1 TARGET UNCERTAINTY IN REINFORCEMENT LEARNING
Many model-free free DRL algorithms use temporal difference updates. In methods such DQN (Mnih ct al, 2013). PPO (Schulman et al. 2017) and SAC (Haarnoja et al. 2013). neural network is trained to predict the QQ of given state-action pair Q2 (s, a by minimizing the error between
the target ((s a) and its prediction Qss, a). T(s, a) computed according Bellman's equation: T(s a)
(Q(''a'))
(4)
s' and r are sampled from the environment given a), and a' is sampled from the current policy given s'. Q(s',a') predicted by copy of the Q-network (called the target network) which is updated less frequently to ensure training stability. The neural network's parameters 0 are optimized using stochastic gradient descent minimize the following L2 loss function:
Lg [(s.a) Qo(s,a))
(5)
3.1.1 THE TARGET AS A RANDOM VARIABLE
The target a) is a noisy approximation of (* a) that is distributed ac- cording to its distribution (T( a). The generative model used produce samples of T(s, is shown in Figure [ and has the following components:
1. if the reward is stochastic, it is sampled from pn([rs, a)
2. if the environment dynamics are stochastic, the next state s' is sampled from ps (s'|s, a);
s, a
3. if the policy is stochastic a' is sampled from the policy T(a'|s");
s'
4. Q a prediction from a var-network PQ (Q) a'));
5. T is deterministically generated from r and Q by equation
a'
As the variance of the noise of Tss.a) ) is not constant, training the Q- network using Ly as in equation (5) is regression on heteroscedastic noisy labels.
3.1.2 VARIANCE OF THE TARGET
T
Figure Bayesian net- work representing the target sampling process
BIV can be used to reduce the impact of het- croscedastic noisy labels in regression, provided estimates of the label variances. We thus aim to evaluate o² (T|s, a) based on the sampling process described in section 3.1.1 and Q-value estimation are independent given and a, hence:
seen in section 2.1
of aT) = of (r|s, a) + +720 o A (Q]s,a) a)
(6)
where Ps Q is the compound probability distribution based components 2-4 in Figure !
Ps (o((s,,)) /ʃ po(@1s', ar d')s a'da'd
where ps (s' al)s, = (av(a'1s s')ps (5' (8|ss,a) given by:
a). Using the law of total variance, the variance of Q
5 Q (Q|s,a) = Es [08 (Q)s a²) |s, a
+ A (Eq [Q]s 18)
(8)
Plugging
8
into
gives:
2 (T
2
E To (Q)s',00 a') [s, + +2 (o (E [Q|s', a² (s)a)) + or
Predictive variance Q network
Policy and environment induced variance
'In this paper, we focus on Q-values, although the principle can also apply to state values. SStochastic reward can happen for example if the reward is corrupted or estimated (Romoff et al. 2018 Campbell al. 2015)
5

---

Published as a conference paper at ICLR 2022
We can identify two distinet components in equation 9 that contribute to the overall variance of the target. The first is the (expectation of the) variance that is due to the uncertainty in the neural network prediction of the value function, Eg ( (Qls', a') The second is the uncertainty due to the stochasticity of the environment and of the policy, o (r(s,, (s,)))) + 3 (Eq [Q]s', a']).
(
Uncertainty in neural network prediction For a given policy x and a given 8', a'. the agent may have seen enough samples have an accurate approximation of (" a'). This corresponds to an epistemic source of uncertainty that should be captured by sampled ensembles. However, as is updated, the regression target, (s', a'). is also changing. This can be interpreted variabil- ity in the underlying process which will be captured by var-networks. We can thus combine both sampling-basee and var-network methods and evaluate of (Q(s', with var-ensembles.
We assume that the estimate of og (21()) aa) given sampled (s',a)) () is unbiased and can therefore use to directly approximate the expectation A (Qls',a')): These values are used in BIV loss LBs (equation M across a mini-batch sampled from the replay buffer. this case, g is used control the trust in the variance estimation, explained in section 2.1
Stochastic environment and policy The other potential source of variance in the target is the result of the stochasticity of the environment encapsulated by pn (rls, a) and psr (s'|s, a) and of the policy represented by T (a' is"). Note that in model-free RI., we have no explicit representation of pn (rls, a. ps (s'(s, which necessary to estimate this source of uncertainty.
(", a) is defined as the expected value the return. As a result, even in the case where Q a') =Q"(s (s, a) in 44, there is still noise the value of the target that is being used the label due to the stochasticity of the environment and policy that generate r, a' and s'. Assuming a zero mean and normally distributed noise, this underlying stochasticity of the generating process is well-captured by a var-netw rer-network with a loss attenuation using the negative log-likeliinee likelihoodd formulation described in equation (3)
3.1.3 LOSS FUNCTION FOR IV-RL
Based on the motivation above, we propose our IV-RL loss over minibatch D of size K as linear combination of LBIV and LLA. balanced by hyperparameter A:
LIVRL 0) Leiv (D,0) + ALLA(D,
K
K
(120 (sk, ((sk, a1t)
200
Q +
7202, +
k=
(Mg) (sk, ax) T(Sk, aE))
À + K
In Q. (sk, ak)
(10)
0g. (sk,ar)
The var-network with parameters 0 predicts both PQ. (sk, ax) and oZ2 (sk, ax) for each element I of the mini-batch. 2 is instead provided by the target var-ensemble which predicts Q of)) when estimating the target T(sk, ax). In LBIV of equation [ of is replaced by 9²0g, according to equation 99. Empirically, the value of A is usually optimal around 5 or 10. Its impact is studied more in details in appendix [ High-varianes samples generated from the target estimation will be down- weighted the BIV loss, while high- high-variance samples due to the stochasticity of the underlying environment and policy will be down-weighted in the LA loss. In the remainder of the paper, show how this loss can be applied to different architectures and algorithms, and demonstrate that in many cases it significantly improves sample efficiency.
3.2
Q-VALUE UNCERTAINTY AND ACTOR-CRITUR STRUCTURES
we discussed how the target's uncertainty can be quantified and how the Bellman update then be interpreted heteroscedastic regression problem. This is applicable in most
In section 3.1

---

Published as a conference paper at ICLR 2022
model-free DRL algorithms, whether they based Q-learning policy optimization. In the special case of actor-critic algorithms, the state-values or Q-values predicted by the critic network also used to train the policy T 's parameters by gradient ascent optimization. An estimate of their variance can also be used to improve the learning process.
The objective is to maximize the expected Q-value:
Eg BawDDannaae (x) (s) a)]
(11)
where D is the state distribution over the probable agent trajectories. The expectation is computed by sampling a mini-batch of size K from a replay buffer containing state-action pairs (#k, ag). The Q-value is approximated by the critic (8k, 0x). The actor's parameters Φ are then trained to maximize the unweighted average:
K 1/K
Σ (sk: as)
(12)
A=
If instead consider the critic's estimation Q(5k: ax) as random variable sampled from P& (Q]ss, a), with mean Mq (sk, ax) and variance of (sk. ax), we can instead infer the expected value
in equation using Bayesian estimation (Murphy 2012):
Σ 1/0 (sk, as)
Σ
u as)
(13)
(sk, an)
R=O
The normalized inverse-variance weights are a direct with the BIV loss in equation I we therefore also use BIV to train the actor. As the target and the Q-networks have the same structure, o (s), ax) can be estimated using the same var-ensembles used in section 3.1.2
3.3 ALGORITHMM
We have adapted IV-RL to DQN and SAC to produce IV-DQN and IV-SAC. Comprehensive pseudo- code, as well as implementation details for each algorithm, can be found in appendix
RESULTS
We have tested IV-RL across a variety of different environments, using IV-DQN for discrete tasks and IV-SAC for continuous tasks. To determine the effects of IV-RL, we compare its performance to the original DQN and SAC, but also the ensemble-based baselines upon which IV-DON and IV-SAC were built: BootstrapDQN and EnsembleSAC. We also include SUNRISE (Lee et al. 2021).
All misemble-based methods result the average of runs over 25 seeds: 5 for the environment x 5 for network initialization. The hyperparameters are the result of a thorough fine-tuning process explained in appendix B As it uses ensembles, IV-RL is computationally expensive: computation time is discussed in Appendix D
ensemble size of N 5. Unless specified otherwise, each
4.1 IV-DQN
We tested IV-DQN on discrete control environments selected to present different characteristies. From OpenAl Gym (Brockman et al./ 20066), LunarLander is a sparse reward control environment and MountainCar is sparse reward exploration environment. From BSuite (Osband al. 2020), Cartpole-Noise is a dense reward control environment. The BootstrapDQN baseline follows Osband
al. (2018) as explained in section 2.3] The reported SUNRISE results are obtained by applying the
SUNRISE weights to Bootstrap DQN. The learning curves are shown in figure where the results
are averaged on 100 episode window.
IV-DON significantly outperforms BootstrapDDQ and SunriseDQN on control-based environments. However, in the exploration- based Mountain-Car where the challenge to find the state that pro- duces some reward, there is very little additional information in every other state for IV-DQN to improve over BootstrapDQN. All weighting schemes thus perform comparably.
"hhe results from BSuite are averaged over 20 runs with different environment and network seeds

---

Published as a conference paper at ICLR 2022
DON
DoS
SunriscDON
N-DON lours)
LunarLander
Cartgole Noise
MountainCar
-100
200
1000
-120
Natur
100
-140
400
-100
-200
150
Tp
Figure 2: Using IV-RL shows improved performance over baseline methods in Cartpole Noise and LunarLander, and does not impact MountainCar.
BeotstraDDON
VerEnsembicDDN
L2 VorfeetworDDGN Var NatworkDDON
BIV Bootal uppiC
BIV Vertnsemb (IADONI)
Lunartander
Cartoole
250
200
150
800
Iutum
100
400
-50
100
200
50
100
150
200
Epiuodes
Sprundes
Figure 3: Ablation study: depending on the environment, the BIV or the var-ensemble component is the most important factor of improvement.
Table shows the median number of episodes necessary to reach a given score for which the vironment is considered as solved. IV-DQN shows a clear improvement over baselines in sample efficiency in both control-based environments but fails improve the exploration in Mountain Car.
Table I: 25th 50th 75th percentiles of the number of episodes necessary for the return averaged with a 100-episode window to reach the solved score on different environments. IV-DQN shows significant improvements in sample efficiency when the environment is not exploration-based
LunarLander (200) Cartpole-Noise (750) MountainCar (-150)
DQN Boo SuN
296 316 349
171 193 max
304 333 403
305 317
174 196
134. 149 206
291 309 368
155 165 175
152 197 257
IV-DQN (ours)
220 227 239
105 112 117
142 163 200
In Figure 3 we perform an ablation study for LunarLander and Cartpole-Noise. In L2 VarEnsem- bleDQN, the loss function CRIV in equation [O]is replaced by the L2 loss. In BIV BootstrapDQN, sampled ensembles are used instead of var-ensembles, and trained only with LBIV. More details about these ablations are found in appendix A In both cases, the use of BIV or var-ensembles improves the performance, and their combination leads to the best sample efficiency. In Cartpole- Noise, var-ensembles carry most of the improvement, maybe because the low amount of states leads to low epistemic uncertainty. We also show that using one single var-network as opposed to var-ensembles is sub-optimal. This is likely due to the unstable nature of the single uncertainty es- timate, which affects the learning process. More details are shown in appendix and G The use of var-ensembles stabilizes the variance estimation, as explained in section 2.2.3
4.2 IV-SAC
IV-SAC was applied different continuous control environments from OpenAl Gym (Brockman et al. 2016) as implemented by MBBL. (Wang et al. 2019). EnsembleSAC is a baseline using ensembles with UCB exploration bonus (Lee al., 2021). ). Table shows the average return after 100k and 200k steps on 25 seeds. We note that the addition of an ensemble instead of a single network, along with the uncertainty-based exploration bonus, already allows the performance to increase compared to SAC. Except in Ant, the SUNRISE weights do not seem to lead to consistently better results. In comparison, IV-SAC leads significant improvements in performance.
This improvement in performance after a fixed amount of training steps can be explained by an improved sample efficiency. This can be seen in figure 4 Even when IV-SAC's return is not signifi-
8

---

Published as a conference paper at ICLR 2022
Table 2: Performance at 100K and 200K time steps (100 and 200 episodes) for several robotics
environments in OpenAI Gym. The results show the mean and standard error over 25 runs. Ant
Walker
HalfCheetah
Hopper
SlimHuman.
200k steps 100k steps
SAC
-392 + 187
3211 + 136
322 + 177
724 29
815 + 74
EnsembleSAC
-389 + 209
3938 112
1597 + 152
852 = 27
1043 + 31
SunriseSAC
4
3879 + 204
1618 + 195
834 + 130
1077 41
IV-SAC (ours)
857 + 231
4260 118
2237 + 116
948 46
1122 + 34
SAC
371 189
3978 = 148
1635 = 162
985 82
1020 0 52
EA
1337 + 278
4757 + 92
2652 + 91
981 65
1294 + 39
SunriseSAC
1423 + 295
4785 + 228
2572 + 119
1462 154
1383 56
IV-SAC (ours)
3009 + 193
5451 + 151
2889 + 50
1281 = 101
2023 - 114
SAC
InsembleS/C
SumiiseSCCC
IV-SAC Cours)
HalfChestah
Happer
SllmHumanold
5000
1500
2000
1403
2000
Razum
2000
0-
1000
D00
2000
3000
600
500
150
200
100
150
200
Stops (10000
Steas (x2000)
Stees bX1000)
Steps (x1000)
(X0000)
Figure 4: IV-SAC learns faster and leads to significantly better results than the baselines.
cantly better than the baselines 200k steps, such as Ant or Hopper, it clearly is learning faster, which is also reflected by the scores at 100k steps.
Similarly IV-DQN, we can separate the contribution from BIV and loss attenuation, as shown in the two first plots of figure 5 While both BIV and var-ensembles alone have significant impact in Walker, it's only their combination that brings the most improvement in HalfCheetah. Similarly to the discrete control case, simple var-network leads to a small improvement over SAC, which is discussed in appendix G We also show in figure the BIV weights provided with var-ensemble variance estimations than the SUNRISE weights, or even the inverse variance weights of UWAC (Wu et al. 202I). The normalization of BIV allows better cope with changes the variance predictions, as seem in appendices and
EnsemoleSAC
VertnseemteeSA
VertebsarttACC VarletworiSAC
UNAAE VerEneamtrrrae
Ensembresa
Bt VovEnsembleSAC INASACI
Survibe We-firsemtieSAc
Walleer
HallCheetah
Walker
HalfCheetah
0000
6000
3000
3000
5000
5000
Rotum
1000
4500
1000
4500
4000
4000
-1000
3500
1000
3500
-2000
3000
2000
3000
1.00
150
50
75
100
125
150
375
200
300
150
200
50
300
150
175
200
Staps (K3000))
Staps (x12000)
Staps 1x10000
Staps
(x1000)
Figure 5: Ablation Study: (first two figures) Impact of using different uncertainty estimation meth- ods (last two figures) Comparing different weighting schemes with var-ensembles.
5 CONCLUSION
We present Inverse Variance Reinforcement Learning (IV-RL.,, framework for model-free deep reinforcement learning which leverages uncertainty estimation to enhance sample efficiency and performance. A thorough analysis of the sources of noise that contribute errors in the target mo- tivates the use of a combination of Batch Inverse Variance (BIV) weighting and variance ensembles to estimate the variance of the target and down weight the uncertain samples in two complementary ways. Our results show that these two components are beneficial and that their combination signif- icantly improves the state of the art in terms of learning efficiency, in particular for control tasks. We applied IV-RL to DQN and SAC, but it can be adapted to any model-free algorithm. Model- based, active continuous learning could also benefit from the underlying ideas presented in this paper. We thus believe that IV-RI a significant step to enable the application of DRL. to real-world problems such robotics, where sample efficiency the main challenge.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
In this work, we present a method to enhance the sample efficiency of deep reinforcement learn- ing algorithms. As such, it is agnostic to the applications, and per se does not raise any particular ethical issue, We however strongly encourage the user our algorithm to ensure they have care- fully thought about the ethical issues related to the particular field of application, such medicine, robotics, communication, finance, etc.
As environmental sustainability can also be considered an ethical issue (Université de Montréal 2018), we publish the carbon footprint of our work.
Experiments were conducted using private infrastructure, which has carbon efficiency of 0.028 khCO2ee/kWh. cumulative of 12367 days, or 296808 hours, of computation was mainly per- formed on the hardware of type RTX 8000 (TDP of 260W). We assume full power usage of the GPUs, although this was not always the case.
Total emissions are estimated to be 2160.76 kgCOzeq of which 0 percents were directly offset. This equivalent 8730 km driven by an average car, or 1.08 metric tons of burned coal.
Estimations were conducted using the MachineLLcaring Impact calculator presented (2019).
preseed in Lacoste et al.
REPRODUCIBILITY STATEMENT
To allow reproducibility of our results, the code, well as the hyperparameters used to produce each result presented in this paper, are available at athttss://gtthhae com real://tt.t/ooooamcccccceaaamaaa www.reeerrbooicsseeooo iv
The environments and implementation we used (OpenAI Gym, BSuite, MBBL) are all publicly accessible, although a Mujoco license is needed to run some of them.
ACKNOWLEDGEMENTS
The authors wish to thank the DEEL project CRDPJ 537462-18 funded by the National Science and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation Aerospace in Québec (CRIAQ). They also acknowledge the support of NSERC as well as Canadian Institute for Advanced Research (CIFAR).
REFERENCES
Siddharth Aravindan and Wee Sun Lee. State-aware variational Thompson sampling for deep Q- networks. In Proceedings of the 20th International Conference on Autonomous Agents and Multi- Agent Systems, AAMAS '21, pp. 124-132. International Foundation for Autonomous Agents and Multiagent Systems, 2021.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 458-458. PMLR, 06-11 Aug 2017.
Dhaivat Bhatt, Kaustubh Mani, Dishank Bansal, Krishna Murthy, Hanju Lee, and Liam Paull. f-Cal: Calibrated aleatoric uncertainty estimation from neural networks for robot perception. arXiv:2109.13973 Sep 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv:1606.01550, Jun 2016.
Jeffrey S. Campbell, Sidney N. Givigi, and Howard M. Schwartz. Handling stochastic reward delays in machine reinforcement learning. In 2015 IEEE 28th Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 314-319, 2015. doi: 10. 00.100//CEECE.2055.72299
Richard Y. Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB exploration via Q- Nov 2017.
ensembles.
10

---

Published as a conference paper at ICLR 2022
William R. Clements, Bastien Van Delft, Benoit-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. TrXw::9055.09638, Sep 2020.
Richard Dearden, Nir Friedman, and Stuart J. Russel. Bayesian Q-learning. 1998.
URL http:
7/ai Staifotafoord. edw/ htllu//nirrPaaeeee
pdf
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXhv:1904.12011 Apr 2019.
Sebastian Flennerhag, Jane X. Wang, Pablo Sprechmann, Francesco Visin, Alexandre Galashov, Steven Kapturowski, Diana L.. Borsa, Nicolas Heess, Andre Barreto, and Razvan Pascanu. Tem- poral difference uncertainties as a signal for exploration. arXiv:2010.02255: Oct 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, and et al. Noisy networks for explo- ration. arXiv: 10295, Jul 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pp. 1050-1059. PMLR, 20-22 Jun 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning. 110(3):457-066, Mar 2021. doi:
Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. arXiv:2102.08501 Feb 2021.
Alex Kendall and Yarin Gal. What uncertainties we need bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30:5574-5584, 2017. 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv:2111.06882 Nov 2021.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pp. 2796-2804 PMLR, 2018.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32:11784-11794, 2019.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. DisCor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33, 2020.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv: 191.00970 2019.

---

Published as a conference paper at ICLR 2022
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.
Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and calibrating uncertainty prediction in regression tasks. arXiv: 1659, Feb 2020.
Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, and Birgitta Dresp-Langley. Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review.
Robotics, 10(1):22, Jan 2021. ISSN 2218-6581. 2102.04148.
doi: 10.390/robottess10000022 arXiv:
Vincent Mai, Waleed Khamies, and Liam Paull. Batch inverse-variance weighting: Deep het- eroscedastic regression. urXir:2107.04497, Jul 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. NIPS, pp. 9, 2013.
Kevin P. Murphy. Machine learning: probabilisticc perspective. Adaptive computation and ma- chine learning series. MIT Press, 2012. ISBN 978-0-262-01802-9,
J. A. Nelder and R. Mead. A Simplex Method for Function Minimization. The Computer Journal, 7(4):308-313, 01 1965. ISSN 0010-4620. doi: 10. 00.1093/comjn177...300
D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) pp. 55-60 vol.1. IEEE, 1994. doi: 10.
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, and et al. Dota 2 with large scale deep reinforcement learning. arXiv:1912.066
Dec 2019.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems, 29:4026-400 4034, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62 2019.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, and et al. Behaviour suite for rein- forcement learning. (a0:::908..0000 Feb 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32:13991-14000 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent François-Lavet, and Joelle Pineau. Re- ward estimation for variance reduction deep reinforcement learning. (CoRL):1-26, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. MoKiv:/707.066: Aug 2017.
12

---

Published as a conference paper at ICLR 2022
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, and et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484-4488 489, Jan 2016.
Malcolm Strens. A bayesian framework for reinforcement learning. Proceedings of the Seventeenth International Conference on Machine Learning, 02 2001.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive com- putation and machine learning series. The MIT Press, second edition edition, 2018. ISBN 978-0- 262-03924-6.
Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, and et al. The limits and potentials of deep learning for robotics. The International Journal of Robotics Research, 5744-5):405-0 420, Apr 2018. doi: 10.117702278664918777777
William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285, Dec 1933. ISSN 00063444. doi: 10.2307/2332286.
Université de Montréal. Montreal declaration for responsible development of artificial intelligence, 2018.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. 1007:00900.00057 Jul 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. Proceed- ings of the 38th International Conference on Machine Learning, volume 139, pp. 11319-11328 PMLR, 18-24 Jul 2021.
A ALGORITHMS IN DETAILS
In this section, we are detailing the different algorithms described the paper, and explaining the design choices that made.
A.1 BOOTSTRAPDQN
BootstrapDON (Osbnn et al. 2016). improved with Randomized Prior Function (RPF) (Osannd al. 2018), is ensemble-based modification DQN (Mnih et al. (013). Instead of Q- and a target network, we use a Q-ensemble and target ensemble. It is the base algorithm upon which IV-DON built. The detailed steps at training are presented in algorithm
Network architecture The Q-ensemble and the target ensemble are initialized identically with N 5 neural networks. We used multi-layer perceptrons with 2 ReL.U-cctivated hidden layers of 64 nodes.
TD update Following the implementation of Osband et al. (2016). the networks are paired one to one between the Q- and the target networks during the TD update. The system can be considered as several DQNs working in parallel. At training time, the best action a' for each Q-network is selected based on the corresponding target network.
The presentation of algorithms is optimized for readability. During the implementation, many for-loops replaced by computation or parallel computing.
13

---

Published as a conference paper at ICLR 2022
Exploration During exploration at training time, network of the ensemble is used to predict follows[Oaband the best action for whole episode. This follows (Osband et al. (2016), and is an approximation of Thompson sampling (Thompson, [933). We added random prior functions (Osband et al. 2018 to tackle the variance underestimation at early training.
At test time, the best action a' is instead selected by a vote from the networks in the ensembles. Ties are broken randomly. Using the mean value across the ensemble was also tested, with slightly lower performances.
Masked replay buffer To mask the replay buffer, each step saved in the replay buffer associated with a N-sized boolean vector which is used as mask during training. Each element of m indicates the corresponding network in the Qeenseblee should be trained using this sample. These elements of are independently generated when the sample is saved, using Bernoulli distribution with probability Pm- They are kept fixed during the training. The role of the mask is to ensure variability in the training of the different networks of the Q-ensemble and thus maintain significant variance prediction.
Algorithm BootstrapDQN Training
1: Input: RPF scale ôRPF, minibatch size K, mask probability Pm, ensemble of N Q-networks
2, ensemble N target-networks T, ensemble of prior-networks P
2: for each episode do
3: Pick a Q-network from Q using Uniform (1..N)
> Oshand et al. 2016)
4: for each time step do
5: a < arg max,, (Qi a)) + (urpPa(ss,
D Random prior (Osband et al. 2018) Collect next-state s' and reward r from the environment by taking action Sample bootstrap mask M = m, ~ {Bernoullipp..]l E N,...,N)
6:
7:
8:
Add (s, a, , 2r,M, M) in buffer B
9:
+ 8'
10:
Sample minibatch D from replay buffer B for j in [1...N] do do for each tuple (sk, ag, s'e, rk, M&) ) in minibatch D do
11:
D Each member of the ensemble
12:
Computing the targets
13:
QN + 25(ax, 8K) + SRPP (sk, ax))
Expected Q-value
14:
de < arg max, (8 a) + SORPpP ( a))
15:
Th P (T, (8 a4)) + ÓRPP (8),,))))
Target
16:
end for
17:
Mask minibatch using Mj
18:
Update 2,'s 's parameters 0, by minimizing L2 loss: (Q - T)² over D
19:
Soft update of the Tj's parameters 03 + (1 - -r)0 + =02 end for
20:
21:
end for
end for
A.2 IV-DQN
IV-DQN is built upon BootstrapDQN. The main differences are the use of var-networks in the en- semble, the computation of the variance in the TD update, and the use of LIVRL when training the networks" parameters commonly used to represent epistemic uncertainty in ML. The pseudo-code for IV-DQN is given in algorithm where the differences with BootstrapDQN are highlighted. At test time, action selection for IV-DON is same as BootstrapDQN.
Var-networks architecture Similar to BootstrapDQN, we used ensembles consisting of N networks. The var-networks were also multi-layer perceptrons with 2 ReLU-activated hidden layers of 64 nodes. The only difference is that, instead of single head for given (s, a), these networks now provide heads to predict the mean μ4 and the variance oe of the Q-value. To differentiate these outputs of network N given state-action pair (s, a), we write A/ a) and No* (s,aa)
14

---

Published as a conference paper at ICLR 2022
Computing the uncertainty of ((s',a') a') As justified in section 3 variance of (Q("'aa)) is estimated
using var-ensembles. For each (5 4/4) pair used to train single Q-network 2, in ensemble Q, the value of Q(s a'²) is the mean jay predicted by the corresponding target-network Tj. However, evaluate og every target-network Ti in the target ensemble predicts a mean Ma, and a variance oi. for (6k: d/4). Then, o a44) is computed as the variance of the mixture of Gaussians composed of all the uad and ".
N
0a k
2 + Pk - 1
(14)
where 1 is the average of all p ss
Using LIVEL Given o each network can now learn predict mean μ and variance o² during the TD update using LIVRL instead of L2. To select the value E, we specify a minimal effective batch size EBS. We use equation (2) to determine if, for the values of o2 and = 0, the effective batch size EBS is lower than MEBS If this the case, & increased until EBS 2 MEBS using a Nelder Mead optimization method (Nelder & Mead 1965>.
Algorithm 2 IV-DQN Training. In red, the differences with BootstrapDQN.
Input: RPF scale òRPF, minibatch size K, mask probability Pm. LA weight X, minimal effec- tive batch size M BSS, initial state var-ensemble of N Q-var-networks Q, var-ensemble of
target-var-nttworks 2, var-ensemble of N prior-var-networks
2: for each episode do
3: Pick a Q-network from Q using i Uniform (I..N)
Osband et al. 2016
4: for each time step do
5:
> Random prior OSband et al. 2018 Collect next-state s' and reward r from the environment by taking action Sample bootstrap mask M = my ~ [Bernoullippm))) E 1..,, N)
6 arg max, (2,ss, ox) + ÓRPP (x a))
6:
7:
8:
Add (s, 8 M) in buffer B
9:
s +
10:
Sample minibatch D from replay buffer B for j in [I...N] do for each tuple (sk, aks ske, Tk, Mx) in mini-batch D do M + 2" (ax; + ORPFP ax)) d < arg max, e (T a) + ŌuPP (sk,,,
11:
12:
13:
Expected Q-value
14:
15:
Th rk + (Tg ) òRPF PaorP,le (g)),
Targget
16:
for in [1...N] do
l and o² for each network of the ensemble
17:
Nkl1 + (s'g f ab) + ORPEPP (8),
18:
of. Ti²² (s'ss a())
19:
end for
20:
Mkkalll <- 1/N Σ 1=1 Megt
D Average the mean predictions Variance of mixture of Gaussians
21:
0 1/N -i=1
+ n2.1
22:
end for
23:
Estimate & based on all o2 so that EBS > MEBS using Eq.
24:
Mask minibatch using Mj
25:
Train 2, 's parameters 8, with LIVRL 10 using Mq. T, E and of on D
26:
Soft update of Tj's parameters 02 + (1 +)0 70;
27:
end for
28:
end for
29:
end for
A.3 ABLATIONS FOR IV-DQN
In the ablation study presented in section 4.1 4 new algorithms are presented. We detail here their differences with IV-DON and algorithm
15

---

Published as a conference paper at ICLR 2022
BIV + BootstrapDQN In BIV + BootstrapDQN, the networks used in the ensembles single- head networks, similar to BootstrapDQN. Thus, these are sampled ensembles. 2 is thus computed as the sampled variance of the networks' predictions. Additionally, instead of LIVRL, we simply use LBIV.
Compared to algorithm
2 lines to 17 are the same. Line 18 is removed. Line 21 changed by:
a Q
+ 1/N Σi= (PQ. MQ.ai). Finally, in line 25 LIvRL is changed to LBIV.
1.2 + VarEnsembleDON In I.2 + VarEnsembleiDON, we do apply BIV weights. The differ- ence with IV-DQN is that we simply change LBIv for the L2 loss in LIVRL- The networks are still var-networks trained with the LLA part of the LIVEL loss, and the rest of the algorithm is the same as algorithm In practice, the computation of o and & are now useless, and lines 16 to 21 as
well as line 23 can be removed.
BIV + VarNetworkDON In BIV + VarNetworkDON, we use a single var-network instead of var-ensemble. Concretely, = This means that line 2 is effectively the variance output k
by the single target network.
Additionally, masks and RPF do not make sense here, so they not applied, As the BootstrapDQN exploration cannot apply either, an e-greedy strategy is used instead.
L2 + VarNetworkDQN L2 + VarNetworkDQN is BIV + VarNetworkDQN, without BIV weights. Crv is replaced by the L2 loss in LIVRL-
A.4 IV-SAC
IV-SAC is built upon EnsembleSAC, which is inspired by SUNRISE (Lee et al. 2021). We describe here the main elements IV-SAC, and present the detailed process algorithm
Network architecture The variance networks used in IV-SAC are multilayer perceptrons with flattened inputs and 2 ReLU-activated hidden layers of 256 nodes. Our implementation is based on r
Critic SAC is complex DRL algorithm with several details which orthogonal IV-RL, for example, the accounting of policy entropy in the definition of the Q-value. On the critic side, the implementation of IV-RL is mostly similar IV-DQN's TD update.
ensemble for the actor-network when using ensembles for the critic. Each network in the actor is linked to pair of networks in the critic. When training, the Q-value and its variance for each state-action
Actor Following the example Lec et al. (2021).
computed similarly when training the critic. As justified in section 3.2, we also apply the BIV loss function when training the actor. However, instead of the L2 error shown in equation (I, we use the SAC loss function for the actor. For minibatch D of size K, we have:
K Σ
Lactor (D.
Σ (8 ax) +
(a (a M ax))
og (sk. ax) +
(15)
Action selection The of ensemble for the d allows for different methods to select the next action given a state. In this respect, we follow again the example of Lee et al. (2021).
During the training of the Q-network, we pair each Q-network with its corresponding actor. To encourage exploration, we additionally take advantage the ensemble to implement UCB. Each policy network outputs an action. The mean Q-value na (8, a) and variance ož 8, a) for each action
16

---

Published as a conference paper at ICLR 2022
computed by sampling over all networks of the critic. The chosen action has the highest UCB score: a arg max,|uQ(s. a) + AUCBoQ(S, a)], where AUcB is a hyperparameter.
To estimate the standard deviation for UCB, we use sampled ensembles: this is because, for UCB, the variance estimation must be calibrated, order to correctly scaled with the Q-values. Var- ensembles are therefore disqualified.
During testing, the agent takes the average action, as sampled from the output of the mean prediction by the policy networks.
Algorithm 3 IV-SAC Training
Input: LA weight A, UCB AucB, minimal effective batch size MEBS, policy networks m ensemble of N Q-var-networks Q, ensemble of N target-var-networks T
2: for each episode do
3: for each time step do
4:
Collect the action samples: A (a, (alas) N1.
5:
f arg Ealu Q(s, a.)) + AucBo a:))
> UCB exploration
6:
Collect next-state s' and reward r from the environment by taking action a
7:
Sample bootstrap mask M {mi Bcrnoulli(pm)] E [1,..,,
8:
Store tuple (s, a, s', r, M)) in buffer B s' end for
9:
10:
11:
for each optimization step do
12:
Sample minibatch D from the replay buffer B for j in [1 [1...N] .N do for each tuple (sk, dg, s6e, rk, M&) in minibatch D do Sample next-action: dk., (a((s()))
13:
Each member of the ensemble b Commuting the targets
14:
15:
Compute Expected Q-value: HQ.K + (ak Sk)
17:
Compute Target: Tj < Pк + *TT)"(s dns)
18:
o2 Q Predictive-Varincee Q, 8k, ak.))
Algorithm
19:
Q.
Predictive- Variiacee $ a/p.)
Algorithm
20:
end for Mask minibatch using Mj
21:
Estimate &a so that (BS(
② MEBS using Eq. (2
23:
Update FTys 's parameters dj with Lactor 15 using 1 Jj & and o2
24:
Estimate & that EBS(O 2 MEBS using Eq. 2
25:
Update 2j's parameters Og with LIVRL 110 using 14 T, & and og on D
Soft update of Tÿ's parameters 8, + (1 T 10 + 70,
27:
end for
28:
end for
29:
end for
Algorithm 4 Variance Estimation for IV-SAC
1: procedure PREDICTIVE-VARIANCCI 8,
2: Input: Var-networks ensemble: N, State: s, Action: a
3: for / in (1...N) do
4: 14 < N3 (s,
5:
of N2" (s,a)
6:
end for
7:
pN 1/N Σ
1=1 Mi
> Average of the mean predictions
8:
t - 1/N
N Ei oF 4 -
Variance of mixture of Gaussians
9:
return of
10: end procedure
17

---

Published as a conference paper at ICLR 2022
A.5 ABLATIONS FOR IV-SAC
We also do ablation studies with our IV-SAC model (presented in section 4.2 additional variations. We detail here their differences with IV-SAC with respect to algorithm R
where we show 6
EnsembleSAC EnsembleSAC. is the base on which IV-SAC is built. uses regular networks as critics instead of variance networks in the ensemble, and it does not consider uncertainty. Compared to algorithm 4, lines to 17, and 20, 21, 26 to 29 are identical. Lines 18, 19, 22 and 24 are removed. In line 23 and 25, the usual losses of SAC are used instead of Lactor and LBIV-
BIV EIV+eeeemlCC Similar to EnsembleSAC, this version uses regular networks as critic instead of variance networks in the ensemble. However, it is uncertainty aware: it estimates the variance of the prediction using sampled ensembles. is thus similar to SunriseSAC (Lee et al. (02T), but with BIV weights instead of SUNRISE weights. Qu and o²
Q.A being computed as the sampled variance of the networks' predictions, we change these lines compared to algorithm
line
18 to:
1/N EI (UO) 1/N Σ Σm= m= MQ m)) Q +
line
N I/N E/ I/N - m)
Q
Additionally, instead of LIVRL, use LBIV for the critic at line 25
L2 + VarEnsembleSACC Here, we do not apply BIV weights both actor and critic losses. The critic networks are still var-networks trained with the LLA part of the LIVEL loss, and the rest of the algorithm is the same as algorithm 3 In practice, the computation of o² and are now useless, and lines
22 and can be removed.
BIV + VarNetworKSAC We use a single var-network instead of var-ensemble as the critic. Concretely, N 1. This means that at lines 18 and 19, o2 and 0g effectively the variance output by the single-targer network.
Q.
L2 + VarNetworksAC VarNetwork is BIV + VarNetwork, without BIV weights in both actor and critic losses. LBIV is replaced by the L2 loss in LIVRL-
UWAC + VarEnsemble Here we change the weighing scheme in IV-SAC to the one proposed in Wu et al. (2021), keeping everything else the same.
Sunrise VarEnsemble For this version we modify the weighing scheme in IV-SAC the one proposed in Lee et al. (2021), keeping everything else the same.
B HYPERPARAMETER TUNING
Deep reinforcement learning algorithms are known to be particularly difficult to evaluate due to the high stochasticity of the learning process and the dependence hyperparameters. To thoroughly compare different algorithms, is important to carefully tune the hyperparameters, based on several environments and network initialization seeds (Henderson al. 2018
For every result presented in this paper, the hyperparameters for each algorithm were tuned using grid search. Each combination of hyperparameters initialization and environment. The best 3 or configurations were then selected to be run 25 times this time, the combinations of environment and 5 initialization seeds. The configuration selected to be shown in the paper is the best of these configurations based on the 25 runs
five times, with different seeds on both
Table 3 describes the type and sets of parameters that were optimized for the relevant algorithms. Note that the learning rate Ir, discount rate and target new update rate т are also relevant for SAC. Because there
many hyperparameters tune, kept for all SAC experiments the
18

---

Published as a conference paper at ICLR 2022
Table 3: List and set of hyperparameters that were tuned when testing the algorithm. Hyperparameter
Relevant algorithms
Set
{32, 64, 128, 256 } for DQN+
Effective minibatch size
All
{256, 512, 1024] for SAC+
Learning rate Ir
DQN+
{0.0005, 0.001, 0.005, 0.01)
Discount rate y
DQN+
{0.98, 0.99,0.9951
Target net update rate
DQN+
0.005, 0.01, 0.05)
DQN+ with
e-decay
{0.97, 0.98, 0.99, 0.991, 0.995)
e-greedy exploration
All with masked ensembles
Mask probability
{0.5, 0.7, 0.8, 0.9, 13
RPF prior scale
All with RPF
{0.1,
Loss attenuation weight À
All with loss attenuation
{0.01,
5,
[16/32, 24,224/32 28/32,30/32, 31/33,11/11/ 31/32] for DON+
Minimal EBS ratio
All with minimal EBS
(0.5, 0.9, 0.95, 0.99) for SAC+
Minimal variance e
All with constant e
(0.5, 1, 5, 100, 1000, 10000)
All SAC+ with UCB (0 means Bootstrap)
UCB weight AUGB
{0, 10] suggested in Lee et al. 2021 {10, 20, 50) as suggested in Lee al.
SUNRISE temperature
All with SUNRISE
UWAC 3
All with UWAC
(0.8, 2.6, suggested in Wu et al.
(202)
default values used in riki imppementation actorlr = 0.0003, criticlr = 0.0003, 7 = 0.99 and 0.005.
The hyperparameters used for each curve can be found in the configuration file the code submitted as additional material.
C A IN LIVRL
As expressed in equation (T).. the LIVRI. loss function we use train the Q-value estimation is combination of two components, LBIV and LI.,, balanced by hyperparameter X.
We use Adam (Kingma & Ba 2015) optimizer: any constant multiplied the loss function is absorbed and does not impact the gradient. Therefore, effectively controls the ratio between LBIV and LLA-
The sensitivity to the value of X depends on the scale of the return, the stochasticity of the environ- ment, and the learning dynamics. As can be seen in figure 6 changing the values of À between I and 10 lead to different results in LunarLander, HalfCheetah and Ant, while in SlimHumanoid, Walker and Hopper, the performance stays very similar.
In figure 7 we analyze how the performance of IV-RL methods change when À is set to extreme values. Very high values of À lead to sub-optimal results, first because LBIV is practically ignored, but also because using variance ensembles alone may lead to a slower or less stable learning curve. Very low values of X, on the other hand, also bring the performance down, first as LLA is ignored, but also as the variance prediction is not trained: the uncertainty estimates provided to LBIV are less reliable.
From figure 7 we can see that the effect of depends on the environment. For LunarLLnders smaller A leads to slower learning and sub- sub-optimal performance in comparison to À 10. On the other hand, very high À leads to poor performance and never reach the goal score of 200. For SlimHumanoid, both extremes show inferior performance to when À 10, but. although there are differences in the dynamics, they reach similar a score after 200k steps. In Walker, high À is also slower, however, the impact of low À is less important. This can be attributed to the fact that, for Walker, the BIV component contributes more to performance than the LA component of IV-RL loss, as seen in figure
19

---

Published as a conference paper at ICLR 2022
I-SAC
IVSAC -
M-SAC -
IV-SAC 10
LunarLander
SlimHumanoid
Walker
250
200
2000
2000
150
100
1500
1000
50
1000
0-
-50
500
-000
100
150 200
250
350
400
100
150
200
50
100
150
200
Episodes
Stops (x2000)
Steps (x1000)
Hopper
HalfCheetah
6000
3000
5500
1400
5000
2000
1200
4500
4000
1000
3500
800
3000
600
200
100
150
200
50
100
Steps (x1000)
Steps (x2000)
Steps (x1000)
Figure 6: IV-RI with À values around the range of to 10.
LunarLander
SlimHumanoid
Walker
250
200
2000
3000
150
100
1500
1000
50
1000
1000
-50
2000
Episodes
Steps (x1000)
Steps (x1000)
Figure 7: IV-RL with extreme values of À
D COMPUTATION TIME
As IV-RL uses ensembles, the computation time necessary for the training of the agents is signifi- cantly increased compared to single-network algorithms such as SAC DQN. This can be seen in tables and
The computation of for MBSS well additional forward passes to evaluate the variance add some computational load compared to other ensemble-based methods such as BootstrapDQN and SunriseDQN.
While this can limit the applicability of IV-RL, we consider the cases where computation time secondary to sample efficiency. This is the case when samples are the more expensive element, such as in the real world.
Table 4: Average training time for 1000 steps on walker2d environment.
SAC EnsembleSAC SunriseSAC IV-SAC (ours)
Runtime/100k steps (s) 8.34
62.16
62.1
74.56
20

---

Published as a conference paper at ICLR 2022
Table 5: Average training time per episode on LunarLander-v2 environment
DQN
BootstrapDQN
SunriseDQN
IV-DQN (ours)
Runtime/ episode (8)
0.51
1.5
1.53
1.87
BesturrapiiN
VartmembeeOO
BIV + SootstraphiO
VorEnsembeeONN IV-DOW
VarNetworkDDN
LunorLander
Cartpole-Nolse
101
Ween verence
10*
Mean varance
301-
207
102
100
200 Epteodes
350
100
150
Eoisodes
Figure 8: Mean target variance predictions for discrete environments
E VARIANCE ESTIMATION
The difference of performance between IV-RL with sampled ensembles and variance ensembles is sometimes significant. Looking at the variance prediction can allow us to better understand this result.
E. IV-DQN
In figure 8 we show the mean of the sample variance estimations per mini-batch, depending on the variance estimation method.
Different profiles can be seen in LunarLander and Cartpole-Noise. In the first environment, the scale of the variance predicted by sampled ensembles with BootstrapDQN and variance ensembles does not differ significantly. In Cartpole-Noise however, the variance ensembles quickly capture much more uncertainty. This explains the difference in dynamics that can be in figure
Although we do not have results for single variance networks Cartpole-Noise, we can see in Lunarl.ander why single variance networks less reliable than variance ensembles. The mean variance they predict is very high, but, as can be seen in figure the median is a lot lower. The mean thus driven by extremely high variance predictions. Instead, sampled and variance ensembles show similar profiles for mean and median variance. The low median variance confirms that single variance networks fail to capture epistemic uncertainty, but also that they have very high volatility,
E.2 IV-SAC
In the continuous control environments, the story is different, can be seen in figures 10 and 11
Besherapttgn
VarinssebbeOO
WerNetwordDON
SootstrapDON
VorEnsomiieDOO
VoNNatworkDDN
Lunarlander
Cortpole-Noise
10'
30a
103
Hedlen venence
Mediun vartonce
30*
207
10'
50
100
150
200 Eptsodes
250
300
350
400
50
100
150
200
250
300
Edtsodes
Figure 9: Median target variance predictions on discrete environments
21

---

Published as a conference paper at ICLR 2022
ErsembeeSCC
EnsembreSNC
12 VerEisembeeSC
BI + Vo EnsemblesAc (MSSAC
Walker
HalfCheetah
10
Mear varlance
100
-07-
10
100
Figure 10: Mean target variance predictions on continuous environments
VarEnsembeSSCC
EnsembleSCC
Walker
Half HaifCheetah
Nediun
-02-
30
100
100
175
(=1000)
(c1000)
Figure 11: Median target variance predictions continuous environments
The first thing notice is the volatility of the mean of the variance produced by variance ensembles. Although the scale is not completely unreasonable and is still close to the median, (on the opposite of variance networks DQN), it still suffers from some calibration issues. Second, we see that the epistemic variance in EnsembleSAC is higher than the one in BIV EnsembleSAC. One hypothesis explain this that using IV reduces the impact of very noisy supervision which may confuse the networks and increase the variance in the ensemble. Variance ensembles do not seem to suffer from this.
F WEIGHTING SCHEMES
The study on the variance allows a better understanding of the advantage of the BIV weights (Mai
et al. 2021) compared to the UWAC weights (Wu et al. 2021) and SUNRISE weights (Lee et al. 2021
In these works, the weight wy of a sample l in a minibatch of size K is given by:
K
1
WBIV
(16)
+
of
+
B
WUWAC,& 1/ min
1.5
(17)
o
WSUNRISE,K 1/ I/gimooo - T) + 0.5
(18)
Figure 12 shows the result of applying such weights to variance ensembles. Note that the first two plots are repeated from figure in section 4.2
It is clear that the SUNRISE weights, which are based on a heuristic, are not optimal compared to the more theoretically grounded UWAC and BIV weights.
However, we can note that UWAC weights perform much better than SUNRISE, but are still some- what inferior BIV weights. Our understanding is that this is due the changes in the scale of the variance estimation along the learning process, as seen in figures 8 and 10
In UWAC, this first leads continuous changes in the effective learning rate. For example, multi- plying the variances by 2 divides the weights accordingly by 2. While the multiplication the loss by constant is captured by optimizers such Adam (Kingma & Ba 2015) in the long term, it can
22

---

Published as a conference paper at ICLR 2022
Vartroentie
JNMC Vertnnembble
Suntoe Vermmemre
Walker
HelCChetah
LunerLander
200
2000
Return
100
4000
-200
2000
50
150
100
(c1000)
Steps (41000)
Speodes
Figure 12: Comparing BIV, UWAC and SUNRISE weights with var-ensemble
NetworkSAC tourst
Walker
HaifChootahh
Happer
Ant
5000
1200
4000
1000
1000
-1000
3000
-1500
500
600-
2000
2000
200
Stops (x1000)
Stops (x1200)
Steps IXL000)
(X1000)
Figure 13: Single variance networks combined with BIV lead to very slight improvement.
still cause short term disturbances in the learning process. In comparison, the normalization of the BIV weights ensures the stability of the loss function.
UWAC weights also hard-clipped by the value 1.5, after being scaled by a constant hyperparam- eter B. When the scale of the variances changes, this can disturb the control over the discriminatory power of the weights. For example, if all variances are smaller than B, all weights become identical at 1.5, and the weighting scheme does not discriminate between certain and un- certain samples. On the other hand, if most variances are very high compared to 3, the weights will be very low. A single low-variance sample may thus completely outweigh all other samples, which will get ignored in the gradient update. The discrimination is then too strong. BIV instead uses &, which is dynamically computed to maintain a minimal effective batch size M EBS as explained in section 2.1 The relative weights of the samples are thus preserved if the variances are multiplied by a constant. As such, the BIV weights maintain a stable discriminatory power in the face of changing variance scales along the learning process, leading better results.
clipped and
Note that the curves here present the combination of UWAC and SUNRISE with var-ensembles, which is different from the original works proposed by Wu et al. (202T) (using MC-Dropout) and Lee et al. 2021) (using sampled ensembles).
G VARIANCE NETWORKS
What if, instead of using predictive ensembles, we simply replaced the Q-network by variance network and trained it using the IV-RL loss?
Figure shows the performances on 4 continuous environments, compared to SAC. We can see slight improvement, but is not statistically significant. Based on the insights given section F we make the hypothesis that it is due to the volatility of the variance estimation with single variance network.
H IV-RL AND OPTIMISM IN THE FACE OF UNCERTAINTY
A question that arose during this research is how does IV-RL interact with the principle of Optimism in the Face of Uncertainty (OFU), which is often used drive exploration in reinforcement learning. for example with Upper Confidence Bounds (UCB).
The principle behind IV-RL is down-weight highly uncertain samples during the TD-update; the idea of OFU is to encourage the agent to choose actions with high uncertainty during exploration. Do these two approach go against each other?
23

---

Published as a conference paper at ICLR 2022
First, would like to distinguish the uncertainty of the value estimation state-action pair Q(. used by OFU, and the uncertainty of the value estimation of the targer [ss, a) used by IV-RL. They separate:
tell the agent to go where the Q(s, a) is uncertain, and then to sample target T(s, a) by projecting over the next time step with Q(s', d'). T(s, a) has its own uncertainty, which not that of Q((s, a). If the agent believes (s, a) is highly uncertain, it should not give it too much weight. Tss, a) may also have low uncertainty: the agent will then give it more importance.
However, two factors come and blur this distinction. First, in the context of reinforcement learning where the experience is built with trajectories, the information the agent has received at (s', a') is correlated to the information it received at (s, Second, the use of function approximation may cause the uncertainty
Q(s, a) and the uncertainty of Q(s', a') to be similar, especially if (s, a) is close to (s', a') on the state-action space. These correlations may induce interactions between IV-RL and OFU, which may not completely decoupled.
Another element needs to be considered: in both IV-DQN and IV-SAC, exploration is made fol- lowing the Q-networks, whereas the value of Q(s', a') to compute the target is estimated by target networks, which are delayed compared to --networks. This reduces the correlations between the uncertainties of Q(s, a) and (ss.a).
As empirical element this answer, we show in the paper that OFU and IV-RL work together: IV-SAC experiments include UCB bonus for exploration. strategies can conflict, IV-RL proposes the use of hyperparameter MBS which allows us to control the intensity of its discrimination. By increasing the M FBS, one can give more importance high- variance samples, and find an optimal balance.
to noted that, if these two
Nevertheless, we believe this question is important for all strategies based on weighting samples according to variance in DRL and could be an interesting line of future investigation.
24


Reviews:
[{"review_id": "b3-C-AfCQp", "reviewer": "Reviewer_9Z1v", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Clarity.**\n\nThe paper is well-written and fairly clear in its justification, background, and empirical results. However, the lack of clear math and pseudocode applying the method to the RL setting makes it unclear as to what exactly the method is and how to implement it. For the next iteration of the paper, this information is necessary. Including only equations for supervised learning (i.e., equations 1, 3, 10) for one of the main contributions of the paper is insufficient in an RL paper. Minor: The math in section 3.1.2 seems clear and correct, although should there be an $s,a$ in the RHS conditioning of equation 8?\n\n\n**Novelty and significance.**\n\nThe main components of the proposed method (i.e., down-weighting by variance and using probabilistic ensembles) are well-known to the community, as cited already by this paper (e.g., Lee et al, 2020, Wu et al, 2021, and Lakshminarayanan et al, 2017). The particular proposed method for down-weighting is taken directly from supervised learning and applied to RL as a method for the above combination. Thus the novelty is just the combination of these, which is insufficient in my view to merit acceptance. The analysis of the sources of uncertainty (sec 3.1.2) is perhaps novel (I did not do a thorough literature review given the other issues with the paper) but insufficient to merit acceptance. \n\nDespite the lack of novelty, if the theoretical or empirical results were strong, then acceptance could be warranted. However, the empirical evidence here is limited and thus does not tip the balance. The environments are all very similar and none are particularly challenging, so it’s unclear the results will hold for a broader range of environments (for example, only 1 of the 3 discrete environments actually improve due to BIV). Further, if the argument is that the normalization used in BIV is crucial relative to UWAC, then I’d like to see this explicitly. I’d similarly like to see a more detailed set of experiments comparing the weighting schemes of UWAC, SUNRISE, and BIV (with all else identical) to know how they compare. \n\n**Question:** how does variance down-weighting interact with the benefits of optimism in the face of uncertainty in RL? Down-weighting by high variance elements of the batch could be interpreted as a form of pessimism, which would run counter to this.\n", "summary_of_the_paper": "The paper presents inverse-variance RL, which combines an ensemble of probabilistic neural networks with batch inverse-variance weighting. The former provides a method for uncertainty estimation and the latter provides a method for incorporating that uncertainty, by multiplying the loss of each entry in each minibatch by the (normalized) inverse of its variance. Experiments on simple discrete and continuous control environments from the OpenAI Gym show that the resulting method, when added to DQN and SAC outperforms these baselines and SUNRISE.", "summary_of_the_review": "Overall, this paper lacks sufficient novelty for acceptance. It recombines a set of existing techniques in a slightly different way than has been done before and shows that this seems to improve performance on the small set of fairly similar domains presented. However, for such a minor change from what has come before, it lacks sufficiently strong theoretical or empirical evidence for acceptance.\n\n\n----------\n\nAfter rebuttal, the majority of my concerns have been addressed by a comprehensive update to the paper. To reflect this, I have updated my scores to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xytqj4nn3KW", "reviewer": "Reviewer_gusa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The proposed technique is theoretically sound, and I think it makes a lot of sense to use label uncertainty (aleatoric) in Reinforcement Learning in DQN/SAC in particular, and this improves sample complexity as more information is extracted from the training process.\n- I believe that it is very important to work on improving sample complexity of Reinforcement Learning, as it is the most important challenge in order to apply RL to real-world problems.\n-  There is a decent improvement in sample complexity when applying BIV to DQN and SAC with no drawbacks. In two of the tested discrete environments (LunarLander and CartPole noise) there is a clear improvement, while on MountainCar there is no improvement but also the method works with same performance as the baseline. For continuous environments, in three out of four there are clear improvements in sample complexity, while ant performance is kind of the same than the best baseline and slightly down. In all cases the IV-RL methods outperform the sample complexity of the original variations without IV-RL.\n- The proposed IV-RL method seems to be an improvement without drawbacks (except for additional hyper-parameters), and overall I think the idea of estimating aleatoric uncertainty in the Q-function being learned is a great idea, as this extracts additional information from the training process, the information seems to be there and it should be exploted as standard part of the Q-learning or SAC training process. This additional information is extracted using a heteroscedatic Gaussian negative log-likelihood loss which can estimate aleatoric uncertainty of the targets without supervision.\n- The authors perform ablations (Figures 3 and 5) that shows the effect of different components in their proposed network, namely if using ensembles, inverse variance weights, and aleatoric uncertainty quantification through the Gaussian NLL loss.\n\nWeaknesses\n\n- One important detail that I think was left out the paper is the effect of the multi-task loss parameter lambda (Eq 10), if I am not mistaken, there is no information on the paper to which value this parameter is set. The supplementary mentions that it is tuned using grid search, and only the range of this parameter is provided, not the actual value that was used to obtain the results in the paper or ablations to show the effect of this parameter on performance. This is important information for the reader and reproducibility that should be included in the paper or supplementary.\n- A second important issue is Eq 10, where the authors claim that the loss function they use is a standard linear combination of the BIV and LA losses, and the BIV loss is defined in Eq 1. This equation shows that the BIV loss contains another sub-loss term ( L (f(xk , θ), ỹk) ) where I assume that it is a squared error loss like the one in Eq 5, but then Eq 3 which shows the LA loss also contains a squared error weighted with the inverse variance (without the ξ term), so it would not make much sense to combine the BIV and LA losses like that directly, I think there is a missing detail here or the author's proposed loss has two squared error terms which can be inconsistent or strange. Please clarify.\n\nMinor Points\n\n- I think that Sections 2.2.1, 2.2.2 and 2.2.3, I think the authors narrative about deep ensembles being sampled ensembles with variance prediction is incorrect, as the original deep ensembles paper (Lakshminarayanan 2017) does not use Dropout or Masks in the ensemble network members, and there is the inconsistency that the papers the authors argue are about sampled ensembles are from 2020, while the deep ensembles paper is from 2017. Also I do not like the term \"probabilistic networks\" and \"probabilistic ensembles\", as it is not clear here what makes them probabilistic, I think better terms are for probabilistic networks are variance networks (as they output mean and variance of the targets), and plain ensembles instead of probabilistic ensembles. Also please note that deep ensembles for regression uses individual variance networks as ensemble members, so the only difference between an ensemble and a variance network is the number of members in the ensemble, where an ensemble of one network is a variance network. I think these terms will make the paper more consistent with the uncertainty quantification literature, and should also be reflected in the label for Figures 3 and 5.\n- I think the paper does not mention how and why the RL environments were selected I think it is better to evaluate on more environments or at least provide a good justification for the environments that were used. Papers like SUNRISE are evaluated on many more environments (ATARI, Gym, Deepmind Control Suite) which ensures that the results and conclusions are robust.\n- I think in the labels of Figure 3 and 5, it is better to relate the components of your loss function (Eq 10) and to using ensembles or single networks, instead of using the names of probabilistic networks/ensembles, this connects also with my previous comment about Sec 2.2.2 and 2.2.3.\n- I would like to point the authors to another paper submitted to ICLR 2022 ( https://openreview.net/forum?id=aPOpXlnV1T ) which deals with practical issues in aleatoric uncertainty estimation using the Gaussian NLL loss for regression, which could help improve their method. In that paper the authors also propose a weighting for each loss term based on their variance.\n\nThe issues pointed above have been addressed by the authors in their newest revision.", "summary_of_the_paper": "This paper is about uncertainty quantification in Q-Learning/Actor-Critic arquitectures. The authors deal with the problem of noise in the targets from where the Q function or value/critic function is learned, by estimating their uncertainty (aleatoric mostly) during the training process, and the authors propose to use this uncertainty with batch inverse variance weighting (BIV), in order to weight samples by inverse variance.\n\nThis idea is simple and yet quite powerful, as results show that using target uncertainty and BIV, DQN and SAC models are able to learn faster and improve their sample complexity by reaching the same average return with less number of episodes in a series of standard RL benchmarks for discrete and continuous environments.\n", "summary_of_the_review": "I believe that this is a good paper, after rebuttal the issues I mentioned have been resolved. I like the idea, it is an important problem in RL, it seems to be correctly evaluated, and it will be a good contribution to the RL community as aleatoric uncertainty is generally not modelled in RL formulations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QsimJ7puVS", "reviewer": "Reviewer_Ncaq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is very well written with clear motivations for the modeling choices used to develop the proposed IV-RL algorithms. The contributions listed at the end of Section 1 are each interesting and valid improvements to the current literature seeking to develop robust Deep RL paradigms. The paper covers a lot of supporting material used to ground the proposed approach and appropriately covers sufficient technical details to appropriately frame the contributions put forward. In particular, I enjoyed the connections drawn to frame TD learning as regression on heteroskedastic noisy labels. This insight drove home the necessity of using BIV to account for the uncertainty induced by approximating the Q function.\n\n*This however leads to the first major weakness in the paper.* It is unclear, as presented at the end of Section 3.1.2, what exact quantities or network structures (ensembles and etc.) are used in BIV and LA exactly. As good as the framing and motivation are, there is a major lack of specificity or explicit formulation of the objectives used to learn the parameters of the policies. The paper would be significantly improved if more careful and direct development of the IV-RL loss was presented. In particular, what forms the estimate for $\\sigma_{\\bar{Q}}^2$? Is it an ensemble of networks? How does that connect to the various terms in Equation 1? What is the Loss function in the BIV Loss? Is it the standard TD-error? In the end, it would be very helpful if the separate components which were introduced in context of the grounding literature were extended to more explicitly communicate how they fit into the DRL pipeline. Clarifying this component is one of two major weaknesses that, if sufficiently addressed, will motivate me to raise my score.\n\nA minor weakness connected to the lack of clarity just described is that the description of the various baselines are quite vague. It's not clear what is meant by a \"non-IV version of the improvements of IV-RL\" nor what the ablations in Figure 3 are really showing. The abbreviations/acronyms included in Figures 3 and 5 are not that informative. Again, being more specific here will make the following experimental analyses much easier to understand. \n\n*The second major weakness* that I want to point out is that there it is not abundantly clear what the major emphasis of IV-RL is. At the outset of the paper the Introduction positions the work as different from the DRL literature harnessing uncertainty to investigate exploration/exploitation tradeoffs, emphasizing that this work focuses instead on the quality of supervision while maximizing the expected reward. However, much of the discussion in Section 4 (Experimental Results) uses concepts of exploration or lack thereof as a justifying reason behind some of the presented performance. This inconsistency dilutes the major contributions and significance of the paper as outlined in Section 1. This lack of precision affects the major takeaways and is evident in the Conclusion as there are very few concrete points that motivate further use of IV-RL. I was hoping to read what the authors felt their proposed IV-RL approach was particularly well suited for, what types of problems they feel are easier to solve because of the advances put forward in the paper. In addition to more concrete conclusions, it would be really nice if the authors were able to express limitations to the proposed framework as currently implemented. Are there particular inefficiencies and computational overhead that need to be considered before implementing IV-RL? What other weaknesses are there? \n\nBeyond these specific points I do want to commend the authors for their extensive and rigorous empirical analysis when comparing to baseline and ablated algorithms. The presented results are convincing and there is some decent discussion (it could be sufficiently tightened a little bit as discussed above) to elaborate the findings. Overall, it appears that the proposed IV-RL algorithm works quite well and improves over prior approaches.", "summary_of_the_paper": "This paper incorporates inverse variance weighting and probabilistic ensembles to account for sources of uncertainty present in the standard TD and Actor Critic learning paradigms. The authors present a systematic analysis of these sources of uncertainty and show how TD learning contributes heteroskedastic behavior, motivating the use of IV weighting of the samples used in agent training. Utlimately, a combination of complementary approaches is used to account for the variance in returns encountered in both the environment and approximation of the Q-Network. The proposed IV-RL algorithms are evaluated in both simple continuous control as well as some MuJoCo tasks, with favorable performance improvements over related prior approaches.", "summary_of_the_review": "Fantastic framing and decomposition of the sources of uncertainty present in learning an RL policy. The paper introduces inverse variance weighting to RL and does so in an interesting and well motivated manner. Despite some weaknesses in firmly anchoring the foundational pieces of the proposed IV-RL approach, the empirical results are shown to improve over comparative algorithms across a variety of domains. As expressed in my main review, greater clarity about the contributions of each component of the IV-RL loss, framed in a more specific terms, as well as more consistent focus on the proposed contributions throughout the paper will help me raise my assessment from borderline acceptance to full acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Po9AxCalk", "reviewer": "Reviewer_HGRr", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is exceptionally well written and a pleasure to read. I literally found no weaknesses and strongly recommend acceptance.\n\nStrengths:\n\n1. It is an important research topic and of very high interest to the community.\n2. The approach is very well embedded in the current literature.\n3. The paper is very easy to follow. It is theoretically sound and for all critical aspects there are intuitions and explanations.\n4. The evaluation is done very thoroughly (results over 25(!) seeds, hyperparameters are provided).\n5. The benefits of IV-RL are shown in both the discrete and continuous setting.\n\nWeaknesses:\nNone\n\nRemark:\nI would be interested in a more extensive discussion of policy degeneration in the LunarLander environment.", "summary_of_the_paper": "In the paper \"Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation\", a group of algorithms called \"Inverse Variance Reinforcement Learning\" is proposed that leverages different estimations of uncertainty to guide the loss of value-functions (and actors in actor-critic settings). IV-RL is evaluated for IV-DQN and IV-SAC in several benchmarks.", "summary_of_the_review": "I literally found no weaknesses and strongly recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously appreciated the clarity of the work as well as the framing of the proposed method. Congratulations.
Decision: Accept (Spotlight)