Risk-Averse Offline Reinforcement Learning

1 INTRODUCTION . In high-stakes applications , the deployment of highly-performing Reinforcement Learning ( RL ) agents is limited by prohibitively large costs at early exploration stages ( Dulac-Arnold et al. , 2019 ) . To address this issue , the offline ( or batch ) RL setting considers learning a policy from a limited batch of pre-collected data . However , high-stakes decision-making is typically also risk-averse : we assign more weight to adverse events than to positive ones ( Pratt , 1978 ) . Although several algorithms for risk-sensitive RL exist ( Howard & Matheson , 1972 ; Mihatsch & Neuneier , 2002 ) , none of them addresses the offline setting . On the other hand , existing offline RL algorithms consider the average performance criterion and are risk-neutral ( Ernst et al. , 2005 ; Lange et al. , 2012 ) . Main contributions We present the first approach towards learning a risk-averse RL policy for high-stakes applications using only offline data : the Offline Risk-Averse Actor-Critic ( O-RAAC ) . The algorithm has three components : a distributional critic that learns the full value distribution ( Section 3.1 ) , a risk-averse actor that optimizes a risk averse criteria ( Section 3.2 ) and an imitation learner implemented with a variational auto-encoder ( VAE ) that reduces the bootstrapping error due to the offline nature of the algorithm ( Section 3.3 ) . In Figure 1 , we show how these components interact with each other . Finally , in Section 4 we demonstrate the empirical performance of O-RAAC . Our implementation is freely available at Github : https : //github.com/nuria95/O-RAAC . 1.1 RELATED WORK . Risk-Averse RL The most common risk-averse measure in the literature is the Conditional Valueat-Risk ( CVaR ) ( Rockafellar & Uryasev , 2002 ) , which corresponds to the family of Coherent RiskMeasures ( Artzner et al. , 1999 ) , and we focus mainly on these risk-measures . Nevertheless , other risk criteria such as Cumulative Prospect Theory ( Tversky & Kahneman , 1992 ) or Exponential Utility ( Rabin , 2013 ) can also be used with the algorithm we propose . In the context of RL , Petrik & Subramanian ( 2012 ) ; Chow & Ghavamzadeh ( 2014 ) ; Chow et al . ( 2015 ) propose dynamic programming algorithms for solving the CVaR of the return distribution with known tabular Markov Decision Processes ( MDPs ) . For unknown models , Morimura et al . ( 2010 ) propose a SARSA algorithm for ( CVaR ) optimization but it is limited to the on-policy setting and small action spaces . To scale to larger systems , Tamar et al . ( 2012 ; 2015 ) propose on-policy Actor-Critic algorithms for Coherent Risk-Measures . However , they are extremely sample inefficient due to sample discarding to compute the risk-criteria and the high-variance of the gradient estimate . While Prashanth et al . ( 2016 ) address sample efficiency by considering Cumulative Prospect Theory instead of Coherent Risk-Measures , their algorithm is limited to tabular MDPs and is also on-policy . Instead , Tang et al . ( 2020 ) propose an off-policy algorithm that approximates the return distribution with a Gaussian distribution and learns its moments using the Bellman equation for the mean and the variance of the distribution . Instead , we learn the full return distribution without making the Gaussianity assumption ( Bellemare et al. , 2017 ) . Perhaps most closely related is the work of Singh et al . ( 2020 ) , who consider also a distributional critic but their algorithm is limited to the CVaR and they do not address the offline RL setting . Furthermore , they use a sample-based distributional critic , which makes the computation of the CVaR inefficient . Instead , we modify Implicit Quantile Networks ( Dabney et al. , 2018 ) in order to compute different risk criteria efficiently . Although ( Dabney et al. , 2018 ) already investigated risk-related criteria , their scope is limited to discrete action spaces ( e.g. , the Atari domain ) in an off-policy setting whereas we consider continuous actions in an offline setting . Offline RL The biggest challenge in offline RL is the Bootstrapping Error : a Q-function is evaluated at state-action pairs where there is little or no data and these get propagated through the Bellman equation ( Kumar et al. , 2019 ) . In turn , a policy optimized with offline data induces a state-action distribution that is shifted from the original data ( Ross et al. , 2011 ) . To address this , Fujimoto et al . ( 2019 ) propose to express the actor as the sum between an imitation learning component and a perturbation model to control the deviation of the behavior policy . Other approaches to control the difference between the data-collection policy and the optimized policy include regularizing the policies with the behavior policy using the MMD distance ( Kumar et al. , 2019 ) or f-divergences ( Wu et al. , 2020 ; Jaques et al. , 2019 ) , or using the behavior policy as a prior ( Siegel et al. , 2020 ) . An alternative strategy in offline RL is to be pessimistic with respect to the epistemic uncertainty that arises due to data scarcity . Yu et al . ( 2020 ) take a model-based approach and penalize the per-step rewards with the epistemic uncertainty of their dynamical model . Using a model-free approach Kumar et al . ( 2020 ) ; Buckman et al . ( 2020 ) propose to learn a lower bound of the Q-function using an estimate of the uncertainty as penalty in the target of the equation . Our work uses ideas from both strategies to address the offline risk-averse problem . First , we use an imitiation learner to control the bootstrapping error . However , by considering a risk-averse criterion , we are also optimizing over a pessimistic distribution compatible with the empirical distribution in the data set . The connections between risk-aversion and distributional robustness are well studied in supervised learning ( Shapiro et al. , 2014 ; Namkoong & Duchi , 2017 ; Curi et al. , 2020 ; Levy et al. , 2020 ) and in reinforcement learning ( Chow et al. , 2015 ; Pan et al. , 2019 ) . 2 PROBLEM STATEMENT . We consider a Markov Decision Process ( MDP ) with possibly continuous s ∈ S and possibly continuous actions a ∈ A , transition kernel P ( ·|s , a ) , reward kernel R ( ·|s , a ) and discount factor γ . We denote by π a stationary policy , i.e. , a mapping from states to distribution over actions . We have access to a fixed batch data set collected with an unknown behaviour policy πβ . We call dβ the joint state , action , reward , next-state distribution induced by the behaviour policy and ρβ the marginal state distribution . We access this distribution by sampling from the fixed data set . Similarly , for any policy π , we call dπ the joint state , action , reward , next-state distribution induced by π on the MDP . In risk-neutral RL , the goal is to find a policy that maximizes the expected discounted sum of returns Edπ [ ∑∞ t=1 γ t−1R ( ·|St , At ) ] , where the expectation is taken with respect to the stochasticity introduced by the reward kernel , the transition kernel , and the policy . We define as Zπ ( s , a ) =D ∑∞ t=1 γ t−1R ( ·|St , At ) as the return distribution conditioned on ( S1 = s , A1 = a ) and following π thereafter . Here =D denotes equality in distribution . The risk-neutral RL objective is the expectation of the distribution of Zπ . In risk-averse settings , we replace the expectation with a distortion operator D that is a mapping from the distribution over the returns to the reals . Thus , the goal is to find a policy π that maximizes max π D [ ∞∑ t=1 γt−1R ( ·|St , At ) ] . ( 1 ) With this framework , we address many different risk-averse distortions . For example , this includes probability weighting functions ( Gonzalez & Wu , 1999 ; Tversky & Kahneman , 1992 ) , the CVaR ( Rockafellar & Uryasev , 2002 ) , the mean-variance criteria ( Namkoong & Duchi , 2017 ) or the Wang criteria ( Wang , 1996 ; Rabin , 2013 ) . 3 OFFLINE RISK-AVERSE ACTOR-CRITIC ( O-RAAC ) . We now present our algorithm O-RAAC for offline-risk averse RL . One of the main technical challenges in going beyond expected rewards is to find an analogue of the Q-function for the particular distortion operator we want to optimize . Unfortunately , the Bellman target of most risk-analogues does not have a closed-form expression . Therefore , we instead learn the full distribution of returns as proposed by Bellemare et al . ( 2017 ) . In Section 3.1 , we describe the training procedure for the distributional critic . Next , in Section 3.2 we define the actor loss as the risk-distortion operator on the learned return distribution and optimize it using a gradient-based approach . Up to this point , the actor-critic template is enough to optimize a risk-averse criteria . However , as we focus on the offline setting , we need to control the bootstrapping error . To this end , we use a variational auto-encoder VAE to learn a generative model of the behavior policy in Section 3.3 . Finally , in Section 3.4 we bring all the pieces together and instantiate our algorithm for different risk distortions D . 3.1 DISTRIBUTIONAL CRITIC LEARNING . To learn the distributional critic , we exploit the distributional Bellman equation of returns Zπ ( s , a ) =D R ( s , a ) + γZ π ( S′ , A′ ) for policy evaluation . The random variables S′ , A′ are distributed according to s′ ∼ P ( ·|s , a ) and A′ ∼ π ( ·|s′ ) . In particular , we represent the return distribution implicitly through its quantile function as proposed by Dabney et al . ( 2018 ) . We use this representation because many risk distortion operators can be efficiently computed using the quantile function of the underlying random variable . We parameterize the quantile function through a neural network with learnable parameters w. We express such implicit quantile function as Zπw ( s , a ; τ ) , where τ ∈ [ 0 , 1 ] is the quantile level . Whereas the neural network architecture proposed by Dabney et al . ( 2018 ) is for discrete actions only , we extend it to continuous actions by considering all s , a , and τ as the inputs and only the quantile value as the output . To learn the parameters w , we use the distributional variant of fitted value-iteration ( Bellemare et al. , 2017 ; Munos & Szepesvári , 2008 ) using a quantile Huber-loss ( Huber , 1964 ) as a surrogate of the Wasserstein-distance as proposed by Dabney et al . ( 2018 ) . To this end , we use a target network with parameters w′ and compute the temporal difference ( TD ) error at a sample ( s , a , r , s′ ) as δτ , τ ′ = r + γZ π w′ ( s ′ , a′ ; τ ′ ) − Zπw ( s , a ; τ ) , ( 2 ) for τ , τ ′ independently sampled from the uniform distribution , i.e. , τ , τ ′ ∼ U ( 0 , 1 ) and a′ ∼ π ( ·|s′ ) . The τ -quantile Huber-loss is Lκ ( δ ; τ ) = ∣∣∣τ − 1 { δ < 0 } ∣∣∣︸ ︷︷ ︸ Quantile loss · { 1 2κδ 2 if |δ| ≤ κ , |δ| − 12κ otherwise.︸ ︷︷ ︸ Huber loss ( 3 ) We prefer the Huber loss over the L2 or L1 loss as it is better behaved due to smooth gradientclipping ( Mnih et al. , 2015 ) . Finally , we approximate the quantile loss for all levels τ by sampling N independent quantiles τ and N ′ independent target quantiles τ ′ . The critic loss is Lcritic ( w ) = E ( s , a , r , s′ ) ∼dβ ( · ) a′∼π ( ·|s′ ) [ 1 N ·N ′ N∑ i=1 N ′∑ j=1 Lκ ( δτi , τ ′j ; τi ) ] . ( 4 )

Summary: The authors propose an RL algorithm for learning risk-averse policies from offline data. Empirically, it is shown that it can outperform some existing risk-neutral approaches on a number of challenging robotic control tasks under risk-sensitive performance measures. Although the empirical results are encouraging, the theoretical properties of the proposed algorithm are unclear and therefore it is not clear how easy it can be implemented in other tasks.