Risk-Averse Offline Reinforcement Learning

1 INTRODUCTION . In high-stakes applications , the deployment of highly-performing Reinforcement Learning ( RL ) agents is limited by prohibitively large costs at early exploration stages ( Dulac-Arnold et al. , 2019 ) . To address this issue , the offline ( or batch ) RL setting considers learning a policy from a limited batch of pre-collected data . However , high-stakes decision-making is typically also risk-averse : we assign more weight to adverse events than to positive ones ( Pratt , 1978 ) . Although several algorithms for risk-sensitive RL exist ( Howard & Matheson , 1972 ; Mihatsch & Neuneier , 2002 ) , none of them addresses the offline setting . On the other hand , existing offline RL algorithms consider the average performance criterion and are risk-neutral ( Ernst et al. , 2005 ; Lange et al. , 2012 ) . Main contributions We present the first approach towards learning a risk-averse RL policy for high-stakes applications using only offline data : the Offline Risk-Averse Actor-Critic ( O-RAAC ) . The algorithm has three components : a distributional critic that learns the full value distribution ( Section 3.1 ) , a risk-averse actor that optimizes a risk averse criteria ( Section 3.2 ) and an imitation learner implemented with a variational auto-encoder ( VAE ) that reduces the bootstrapping error due to the offline nature of the algorithm ( Section 3.3 ) . In Figure 1 , we show how these components interact with each other . Finally , in Section 4 we demonstrate the empirical performance of O-RAAC . Our implementation is freely available at Github : https : //github.com/nuria95/O-RAAC . 1.1 RELATED WORK . Risk-Averse RL The most common risk-averse measure in the literature is the Conditional Valueat-Risk ( CVaR ) ( Rockafellar & Uryasev , 2002 ) , which corresponds to the family of Coherent RiskMeasures ( Artzner et al. , 1999 ) , and we focus mainly on these risk-measures . Nevertheless , other risk criteria such as Cumulative Prospect Theory ( Tversky & Kahneman , 1992 ) or Exponential Utility ( Rabin , 2013 ) can also be used with the algorithm we propose . In the context of RL , Petrik & Subramanian ( 2012 ) ; Chow & Ghavamzadeh ( 2014 ) ; Chow et al . ( 2015 ) propose dynamic programming algorithms for solving the CVaR of the return distribution with known tabular Markov Decision Processes ( MDPs ) . For unknown models , Morimura et al . ( 2010 ) propose a SARSA algorithm for ( CVaR ) optimization but it is limited to the on-policy setting and small action spaces . To scale to larger systems , Tamar et al . ( 2012 ; 2015 ) propose on-policy Actor-Critic algorithms for Coherent Risk-Measures . However , they are extremely sample inefficient due to sample discarding to compute the risk-criteria and the high-variance of the gradient estimate . While Prashanth et al . ( 2016 ) address sample efficiency by considering Cumulative Prospect Theory instead of Coherent Risk-Measures , their algorithm is limited to tabular MDPs and is also on-policy . Instead , Tang et al . ( 2020 ) propose an off-policy algorithm that approximates the return distribution with a Gaussian distribution and learns its moments using the Bellman equation for the mean and the variance of the distribution . Instead , we learn the full return distribution without making the Gaussianity assumption ( Bellemare et al. , 2017 ) . Perhaps most closely related is the work of Singh et al . ( 2020 ) , who consider also a distributional critic but their algorithm is limited to the CVaR and they do not address the offline RL setting . Furthermore , they use a sample-based distributional critic , which makes the computation of the CVaR inefficient . Instead , we modify Implicit Quantile Networks ( Dabney et al. , 2018 ) in order to compute different risk criteria efficiently . Although ( Dabney et al. , 2018 ) already investigated risk-related criteria , their scope is limited to discrete action spaces ( e.g. , the Atari domain ) in an off-policy setting whereas we consider continuous actions in an offline setting . Offline RL The biggest challenge in offline RL is the Bootstrapping Error : a Q-function is evaluated at state-action pairs where there is little or no data and these get propagated through the Bellman equation ( Kumar et al. , 2019 ) . In turn , a policy optimized with offline data induces a state-action distribution that is shifted from the original data ( Ross et al. , 2011 ) . To address this , Fujimoto et al . ( 2019 ) propose to express the actor as the sum between an imitation learning component and a perturbation model to control the deviation of the behavior policy . Other approaches to control the difference between the data-collection policy and the optimized policy include regularizing the policies with the behavior policy using the MMD distance ( Kumar et al. , 2019 ) or f-divergences ( Wu et al. , 2020 ; Jaques et al. , 2019 ) , or using the behavior policy as a prior ( Siegel et al. , 2020 ) . An alternative strategy in offline RL is to be pessimistic with respect to the epistemic uncertainty that arises due to data scarcity . Yu et al . ( 2020 ) take a model-based approach and penalize the per-step rewards with the epistemic uncertainty of their dynamical model . Using a model-free approach Kumar et al . ( 2020 ) ; Buckman et al . ( 2020 ) propose to learn a lower bound of the Q-function using an estimate of the uncertainty as penalty in the target of the equation . Our work uses ideas from both strategies to address the offline risk-averse problem . First , we use an imitiation learner to control the bootstrapping error . However , by considering a risk-averse criterion , we are also optimizing over a pessimistic distribution compatible with the empirical distribution in the data set . The connections between risk-aversion and distributional robustness are well studied in supervised learning ( Shapiro et al. , 2014 ; Namkoong & Duchi , 2017 ; Curi et al. , 2020 ; Levy et al. , 2020 ) and in reinforcement learning ( Chow et al. , 2015 ; Pan et al. , 2019 ) . 2 PROBLEM STATEMENT . We consider a Markov Decision Process ( MDP ) with possibly continuous s ∈ S and possibly continuous actions a ∈ A , transition kernel P ( ·|s , a ) , reward kernel R ( ·|s , a ) and discount factor γ . We denote by π a stationary policy , i.e. , a mapping from states to distribution over actions . We have access to a fixed batch data set collected with an unknown behaviour policy πβ . We call dβ the joint state , action , reward , next-state distribution induced by the behaviour policy and ρβ the marginal state distribution . We access this distribution by sampling from the fixed data set . Similarly , for any policy π , we call dπ the joint state , action , reward , next-state distribution induced by π on the MDP . In risk-neutral RL , the goal is to find a policy that maximizes the expected discounted sum of returns Edπ [ ∑∞ t=1 γ t−1R ( ·|St , At ) ] , where the expectation is taken with respect to the stochasticity introduced by the reward kernel , the transition kernel , and the policy . We define as Zπ ( s , a ) =D ∑∞ t=1 γ t−1R ( ·|St , At ) as the return distribution conditioned on ( S1 = s , A1 = a ) and following π thereafter . Here =D denotes equality in distribution . The risk-neutral RL objective is the expectation of the distribution of Zπ . In risk-averse settings , we replace the expectation with a distortion operator D that is a mapping from the distribution over the returns to the reals . Thus , the goal is to find a policy π that maximizes max π D [ ∞∑ t=1 γt−1R ( ·|St , At ) ] . ( 1 ) With this framework , we address many different risk-averse distortions . For example , this includes probability weighting functions ( Gonzalez & Wu , 1999 ; Tversky & Kahneman , 1992 ) , the CVaR ( Rockafellar & Uryasev , 2002 ) , the mean-variance criteria ( Namkoong & Duchi , 2017 ) or the Wang criteria ( Wang , 1996 ; Rabin , 2013 ) . 3 OFFLINE RISK-AVERSE ACTOR-CRITIC ( O-RAAC ) . We now present our algorithm O-RAAC for offline-risk averse RL . One of the main technical challenges in going beyond expected rewards is to find an analogue of the Q-function for the particular distortion operator we want to optimize . Unfortunately , the Bellman target of most risk-analogues does not have a closed-form expression . Therefore , we instead learn the full distribution of returns as proposed by Bellemare et al . ( 2017 ) . In Section 3.1 , we describe the training procedure for the distributional critic . Next , in Section 3.2 we define the actor loss as the risk-distortion operator on the learned return distribution and optimize it using a gradient-based approach . Up to this point , the actor-critic template is enough to optimize a risk-averse criteria . However , as we focus on the offline setting , we need to control the bootstrapping error . To this end , we use a variational auto-encoder VAE to learn a generative model of the behavior policy in Section 3.3 . Finally , in Section 3.4 we bring all the pieces together and instantiate our algorithm for different risk distortions D . 3.1 DISTRIBUTIONAL CRITIC LEARNING . To learn the distributional critic , we exploit the distributional Bellman equation of returns Zπ ( s , a ) =D R ( s , a ) + γZ π ( S′ , A′ ) for policy evaluation . The random variables S′ , A′ are distributed according to s′ ∼ P ( ·|s , a ) and A′ ∼ π ( ·|s′ ) . In particular , we represent the return distribution implicitly through its quantile function as proposed by Dabney et al . ( 2018 ) . We use this representation because many risk distortion operators can be efficiently computed using the quantile function of the underlying random variable . We parameterize the quantile function through a neural network with learnable parameters w. We express such implicit quantile function as Zπw ( s , a ; τ ) , where τ ∈ [ 0 , 1 ] is the quantile level . Whereas the neural network architecture proposed by Dabney et al . ( 2018 ) is for discrete actions only , we extend it to continuous actions by considering all s , a , and τ as the inputs and only the quantile value as the output . To learn the parameters w , we use the distributional variant of fitted value-iteration ( Bellemare et al. , 2017 ; Munos & Szepesvári , 2008 ) using a quantile Huber-loss ( Huber , 1964 ) as a surrogate of the Wasserstein-distance as proposed by Dabney et al . ( 2018 ) . To this end , we use a target network with parameters w′ and compute the temporal difference ( TD ) error at a sample ( s , a , r , s′ ) as δτ , τ ′ = r + γZ π w′ ( s ′ , a′ ; τ ′ ) − Zπw ( s , a ; τ ) , ( 2 ) for τ , τ ′ independently sampled from the uniform distribution , i.e. , τ , τ ′ ∼ U ( 0 , 1 ) and a′ ∼ π ( ·|s′ ) . The τ -quantile Huber-loss is Lκ ( δ ; τ ) = ∣∣∣τ − 1 { δ < 0 } ∣∣∣︸ ︷︷ ︸ Quantile loss · { 1 2κδ 2 if |δ| ≤ κ , |δ| − 12κ otherwise.︸ ︷︷ ︸ Huber loss ( 3 ) We prefer the Huber loss over the L2 or L1 loss as it is better behaved due to smooth gradientclipping ( Mnih et al. , 2015 ) . Finally , we approximate the quantile loss for all levels τ by sampling N independent quantiles τ and N ′ independent target quantiles τ ′ . The critic loss is Lcritic ( w ) = E ( s , a , r , s′ ) ∼dβ ( · ) a′∼π ( ·|s′ ) [ 1 N ·N ′ N∑ i=1 N ′∑ j=1 Lκ ( δτi , τ ′j ; τi ) ] . ( 4 )

Summary: The paper studies the problem of safe reinforcement learning, where we want to learn risk-averse policies in the offline setting. It proposes “Offline Risk Averse Actor Critic” (ORAAC) which performs competitively as risk-neural agent, and outperforms D4PG based baseline as a risk-averse agent on D4RL benchmark. The algorithm involves modifying the losses to learn risk-averse actor, distributional critic and a VAE-based imitative policy.