Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

1 INTRODUCTION . Deep reinforcement learning ( DRL ) methods have proven to be powerful at solving sequential decision-making tasks across domains ( Silver et al. , 2016 ; OpenAI et al. , 2019 ) . Combining the flexibility of the reinforcement learning framework with the representational power of deep neural networks enables policy optimization in complex and high-dimensional environments with unknown dynamics models to maximize the expected cumulative reward ( Sutton & Barto , 2018 ) . An important limitation of DRL methods is their sample inefficiency : an enormous amount of data is necessary and makes training expensive . This makes applying DRL in the real world challenging , for example in robotics ( Sünderhauf et al. , 2018 ; Dulac-Arnold et al. , 2019 ) . In tasks like manipulation , sample collection is a slow and costly process ( Liu et al. , 2021 ) . It is even more expensive in riskaverse applications like autonomous driving ( Kothari et al. , 2021 ) . Among the current state-of-the-art approaches to improve learning efficiency , a promising direction is to exploit the prevalence of uncertainty in the underlying DRL algorithm . By adopting a Bayesian framework , we can consider the sampled quantities in DRL as random variables and leverage information about their distributions to improve the learning process ( Osband et al. , 2018 ) . In this paper , we consider the particular problem of unreliable supervision in the temporal difference update and the policy optimization process . In DRL , value predictions are used to supervise the training : in temporal difference-based algorithms , they are included in bootstrapped target values which are used as labels ; in actor-critic frameworks , the policy is trained to optimize them . That these value predictions are noisy slows the learning and brings instability ( Kumar et al. , 2019 ; 2020 ) . The amount of noise in the supervision depends on the uncertainty of the value prediction , which evolves during the training process and depends on the state ( and action ) evaluated . It is therefore heteroscedastic . While there is an extensive body of literature focused on using the uncertainty of the value prediction to guide the exploration/exploitation trade-off ( Dearden et al. , 1998 ; Strens , 2001 ; Osband et al. , 2016 ; Pathak et al. , 2017 ; Chen et al. , 2017 ; Osband et al. , 2018 ; Fortunato et al. , 2019 ; Osband et al. , 2019 ; Flennerhag et al. , 2020 ; Clements et al. , 2020 ; Jain et al. , 2021 ; Aravindan & Lee , 2021 ) , there are very few works focused in leveraging it to mitigate the impact of unreliable supervision . Distributional RL ( Bellemare et al. , 2017 ) considers the value function as a distribution to be learned as such . It is orthogonal to our proposition : we consider the uncertainty of the labels used to learn a scalar value function . In the offline RL setting , where the dataset is limited , uncertainty-weighted actor-critic ( UWAC ) ( Wu et al. , 2021 ) uses inverse-variance weighting to discard out-of-distribution state-action pairs using Monte Carlo dropout ( Gal & Ghahramani , 2016 ) for uncertainty estimation . Closer to our work , Lee et al . ( 2020 ) propose SUNRISE , in which each sample of the Bellman backup in the TD update step is weighted to lower the importance of the targets which have a high standard deviation . The weights w ( s′ , a′ ) are computed based on a sigmoı̈d of the negative standard deviation Q̂std ( s′ , a′ ) scaled by a temperature hyperparameter T , and then offset such that they are between 0.5 and 1 : w ( s , a ) = σ ( −Q̂std ( s′ , a′ ) ∗T ) + 0.5 . The uncertainty of the target is estimated using sampled ensembles . While SUNRISE proposes other contributions such as an exploration bonus , the heuristic weighting scheme and the limitations of sampled ensembles in capturing the predictive uncertainty leave space for improvement in the mitigation of the effects of unreliable supervision . We propose inverse-variance reinforcement learning ( IV-RL ) . IV-RL also uses weights to reduce the importance of uncertain targets in training . It does so by addressing the problem from two viewpoints . First , we use variance networks ( Kendall & Gal , 2017 ) , whose loss function for regression is the negative log-likelihood instead of the L2 distance . For a given state-action pair ( s , a ) , the network learns the target ’ s noise , due for example to the stochasticity of the environment or the update of the policy . It then naturally down-weights the highly noisy samples in the training process . Second , we use variance ensembles ( Lakshminarayanan et al. , 2017 ) to estimate the uncertainty of the target due to the prediction of Q ( s′ , a′ ) during the temporal-difference update . We merge the predicted variances of several variance networks through a mixture of Gaussians , which has been shown to be a very reliable method to capture predictive uncertainty ( Ovadia et al. , 2019 ) . We then use Batch Inverse-Variance ( BIV ) ( Mai et al. , 2021 ) , which has been shown to significantly improve the performance of supervised learning with neural networks in the case of heteroscedastic regression . BIV is normalized , which makes it ideal to cope with different and time-varying scales of variance . We show analytically that these two different variance predictions for the target are complementary and their combination leads to consistent and significant improvements in the sample efficiency and overall performance of the learning process . In summary , our contribution is threefold : 1 . We present a systematic analysis of the sources of uncertainty in the supervision of modelfree DRL algorithms . We show that the variance of the supervision noise can be estimated with two complementary methods : negative log-likelihood and variance ensembles . 2 . We introduce IV-RL , a framework that accounts for the uncertainty of the supervisory signal by weighting the samples in a mini-batch during the agent ’ s training . IV-RL uses BIV , a weighting scheme which is robust to poorly calibrated variance estimation.1 3 . Our experiments show that IV-RL can lead to significant improvements in sample efficiency when applied to Deep Q-Networks ( DQN ) ( Mnih et al. , 2013 ) and Soft-Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) . In section 2 , we introduce BIV as a weighting scheme for heteroscedastic regression , and variance ensembles as an uncertainty estimation method . We analyse the sources of uncertainty in the target in section 3 , where we also introduce our IV-RL framework . We finally present our experimental results in section 4 . 2 BACKGROUND AND PRELIMINARIES . 2.1 BATCH INVERSE-VARIANCE WEIGHTING . In supervised learning with deep neural networks , it is assumed that the training dataset consists of inputs xk and labels yk . However , depending on the label generation process , the label may be noisy . In regression , we can model the noise as a normal distribution around the true label : ỹk = yk + δk with δk ∼ N ( 0 , σ2k ) . In some cases , the label generation process leads to different variances for the label noises . When these variances can be estimated , each sample is a triplet ( xk , ỹk , σ 2 k ) . 1Our code is available as supplementary material . We plan to open-source our codebase in case of acceptance/publication of the paper . Batch Inverse-Variance ( BIV ) weighting ( Mai et al. , 2021 ) leverages the additional information σ2k , which is assumed to be provided , to learn faster and obtain better performance in the case of heteroscedastic noise on the labels . Applied to L2 loss , it optimizes the neural network parameters θ using the following loss function for a mini-batch D of size K 2 : LBIV ( D , θ ) = ( K∑ k=0 1 σ2k + ξ ) −1 K∑ k=0 ( fθ ( xk ) − ỹk ) 2 σ2k + ξ ( 1 ) This is a normalized weighted sum with weights wk = 1/ ( σ2k + ξ ) . Normalizing in the mini-batch enables control of the effective learning rate , especially in cases where the training data changes over time , such as in DRL . By focusing on the relative scale of the variances instead of their absolute value , it also provides robustness to poor scale-calibration of the variance estimates . As explained in Mai et al . ( 2021 ) , ξ is a hyperparameter which is important for the stability of the optimization process . A higher ξ limits the highest weights , thus preventing very small variance samples from dominating the loss function for a mini-batch . However , by controlling the discrimination between the samples , ξ is also key when the variance estimation is not completely trusted . It provides control of the effective mini-batch size EBS , according to : EBS = ( ∑K k wk ) 2 ∑K k w 2 k = ( ∑K k 1 ( σ2k + ξ ) ) 2 ∑K k 1 ( σ2k + ξ ) 2 ( 2 ) For example , imagine a mini-batch where most samples have very high variances , and only one has a very low variance . If ξ = 0 , this one low-variance sample is effectively the only one to count in the mini-batch , and EBS tends towards 1 . Increasing ξ would give more relative importance to the other samples , thus increasing EBS . With a very high ξ compared to the variances , all weights are equal , and EBS tends towards K ; in this case , the BIV loss tends towards L2 . Tuning the ξ parameter The simplest way to set ξ is to choose a constant value as an additional hyperparameter . However , the best value is difficult to evaluate a priori and can change when the profile of variances changes during a task , as is the case in DRL . It is instead possible to numerically compute the value of ξ which ensures a minimal EBS for each mini-batch . This method allows ξ to automatically adapt to the different scales of variance , while ensuring a minimal amount of information from the dataset to be accounted for by the algorithm . The minimal EBS is also a hyper-parameter , but it is easier to set and to transfer among environments , as it is simply a fraction of the original batch size . As such , it can be set as a batch size ratio . 2.2 ESTIMATING THE UNCERTAINTY OF A NEURAL NETWORK PREDICTION . The predictive uncertainty of a neural network can be considered as the combination of aleatoric and epistemic uncertainties Kendall & Gal ( 2017 ) . Aleatoric uncertainty is irreducible , and characterizes the non-deterministic relationship between the input and the desired output . Epistemic uncertainty is instead related to the trained model : it depends on the information available in the training data , the model ’ s capacity to retain it , and the learning algorithm ( Hüllermeier & Waegeman , 2021 ) . There is currently no principled way to quantify the amount of task-related information present in the input , the training data , or the model . The state of the art for predictive uncertainty estimation instead relies on different sorts of proxies . These sometimes capture other elements , such as the noise of the labels , which we can use to our advantage . We focus here on the relevant methods to our work .

Summary: This paper incorporates inverse variance weighting and probabilistic ensembles to account for sources of uncertainty present in the standard TD and Actor Critic learning paradigms. The authors present a systematic analysis of these sources of uncertainty and show how TD learning contributes heteroskedastic behavior, motivating the use of IV weighting of the samples used in agent training. Utlimately, a combination of complementary approaches is used to account for the variance in returns encountered in both the environment and approximation of the Q-Network. The proposed IV-RL algorithms are evaluated in both simple continuous control as well as some MuJoCo tasks, with favorable performance improvements over related prior approaches.