A Risk-Sensitive Policy Gradient Method

Standard deep reinforcement learning ( DRL ) agents aim to maximize expected reward , considering collected experiences equally in formulating a policy . This differs from human decision-making , where gains and losses are valued differently and outlying outcomes are given increased consideration . It also wastes an opportunity for the agent to modulate behavior based on distributional context . Several approaches to distributional DRL have been investigated , with one popular strategy being to evaluate the projected distribution of returns for possible actions . We propose a more direct approach , whereby the distribution of full-episode outcomes is optimized to maximize a chosen function of its cumulative distribution function ( CDF ) . This technique allows for outcomes to be weighed based on relative quality , does not require modification of the reward function to modulate agent behavior , and may be used for both continuous and discrete action spaces . We show how to achieve an asymptotically consistent estimate of the policy gradient for a broad class of CDF-based objectives via sampling , subsequently incorporating variance reduction measures to facilitate effective on-policy learning . We use the resulting algorithm to train agents with different “ risk profiles ” in penalty-based formulations of six OpenAI Safety Gym environments , observing that moderate emphasis on improvement in training scenarios where the agent performs poorly both increases the accumulation of positive rewards and decreases the frequency of incurred penalties . We found that , in all environments tested , the same risk profile can be used to produce both stronger overall performance than standard Proximal Policy Optimization ( PPO ) and higher levels of positive reward than PPO constrained by Lagrangians to maintain the same cost levels . 1 INTRODUCTION . While deep reinforcement learning ( DRL ) has been used to master an impressive array of simulated tasks in controlled settings , it has not yet been widely adopted for high-stakes , real-world applications . One reason for this gap is the lack of distributional perspective in standard artificial agents . Endowing agents with such perspective could make their decision-making more robust , potentially leading to increased safety , increased trust from humans , and more widespread real-world adoption . In reinforcement learning ( RL ) , risk arises due to uncertainty around the possible outcomes of an agent ’ s future actions . It is a result of randomness in the operating environment , mismatch between training and test conditions , and the inherent randomness of a stochastic policy . Risk-sensitive policies , or those that consider more than a mean over the distribution of possible outcomes , offer the potential for added robustness under uncertain and dynamic conditions . There is an evolving landscape of algorithmic paradigms for handling risk in RL , from constraint-based approaches adapted from optimal control ( Achiam et al. , 2017 ; Chow et al. , 2019 ; Ray et al. , 2019 ; Zhong et al. , 2020 ) to adversarial approaches emerging from AI Safety ( Garcı́a & Fernández , 2015 ; Amodei et al. , 2016 ) . Within this landscape , learning approaches that optimize distributional measures offer the ability to express design preferences over the full distribution of potential outcomes , through the specification of a risk-sensitivity criterion . Distributional RL has been studied for value-based methods , with a popular strategy being to use the distributional Bellman equation to estimate the distribution of Q-values for each member of a discrete set of potential actions ( Bellemare et al. , 2017 ; Dabney et al. , 2018a ; b ) . However , distributional RL has not been widely explored for policy gradient methods , which could permit direct optimization of risk-sensitive measures and naturally accommodate both discrete and continuous action spaces . In the following , we introduce a novel framework for risk-sensitive learning using policy gradients . Our approach allows agents to be trained with different risk profiles through design-time specification of both utility and weight functions , with the latter being defined over the estimated distribution of full-episode rewards . This framework enables agent-based learning that captures aspects of human decision-making , such as overemphasis of rare occurrences and diminishing marginal utility relative to a reference outcome ( Kahneman & Tversky , 1979 ) . It also allows implementation of another key strategy of human learning : emphasizing improvement on tasks where one is deficient . We demonstrate the ability of our algorithm to use this strategy to improve performance relative to both unconstrained and constrained methods in six OpenAI Safety Gym environments ( Ray et al. , 2019 ) . 2 RELATED WORK . Constrained RL offers a set of approaches to safe exploration ( Garcı́a & Fernández ( 2015 ) ) that aim to enforce explicit constraints throughout the learning process via methods including Lagrangian constraints ( Ray et al . ( 2019 ) ) and constraint coefficients ( Achiam et al . ( 2017 ) ) . Differing from the safe exploration scenario , we here consider problems with distinct training and test phases where agent performance is to be evaluated . Our experiments indicate that risk-sensitive learning can offer performance improvements over constrained learning in scenarios where safety constraints need not be enforced during training . Distributional approaches to risk-sensitive RL have primarily been explored in the value-based setting . Therein , the value distribution has been explicitly modeled through categorical techniques ( Bellemare et al. , 2017 ) or quantile regression ( Dabney et al. , 2018a ) and used to improve both value predictions and overall performance . Recent works utilize distributional modeling in the actor-critic setting to enable application to continuous action spaces , again demonstrating improved performance over baseline approaches ( Ma et al. , 2020 ; Zhang et al. , 2021 ; Duan et al. , 2021 ) . In value-based approaches , risk-sensitivity criteria are applied at run time as a nonlinear warping of the estimated value distribution . Policy gradient approaches offer additional promise for risk-sensitive RL , but require direct optimization of a parameterized policy with respect to a distributional objective . Some existing methods are limited to a specific class of learning objective , such as the set of concave risk measures that allow a globally-optimal solution ( Tamar et al. , 2015 ; Zhong et al. , 2020 ) . Others allow a broader class of measures but are more restrictive in the class of policies that can be represented ( Prashanth et al. , 2016 ) . We aim for a risk-sensitive policy gradient approach that both offers significant flexibility in the choice of learning objective and can learn policies parameterized by a deep neural network . Various measures have been considered in the context of risk-sensitive RL , including exponential utility ( Pratt , 1964 ) , percentile performance criteria ( Wu & Lin , 1999 ) , value-at-risk ( Leavens , 1945 ) , conditional value-at-risk ( Rockafellar & Uryasev , 2000 ) , and prospect theory ( Kahneman & Tversky , 1979 ) . In this work , we consider a class of risk-sensitivity measures motivated by Cumulative Prospect Theory ( CPT ) ( Tversky & Kahneman , 1992 ) . CPT uniquely models two key aspects of human decision-making : ( 1 ) a utility function u , computed relative to a reference point that induces more risk-averse behavior in the presence of gains than losses and ( 2 ) a weight function w that prioritizes outlying events . Specific forms of u and w are given in Tversky & Kahneman ( 1992 ) ( and in Appendix A.4 ) , but the general form of CPT admits a wide variety of risk-sensitive objectives . Here we show how to train agents to optimize this class of objectives through sampling-based estimation of their policy gradients and requisite variance reduction . The final algorithm resembles well-known on-policy approaches such as Proximal Policy Optimization ( Schulman et al. , 2017b ) and is similarly widely applicable . Although we do not explore it here , the incorporation of an appropriate risk-sensitivity criterion could additionally enable risk-aware exploration and adversarial training for increased robustness ( Pinto et al. , 2017 ; Parisi et al. , 2019 ; Zhang et al. , 2020 ) . 3 RISK-SENSITIVE POLICY OPTIMIZATION . In this section we formalize the class of distributional objectives to be considered , derive a samplingbased approximation of its policy gradient , enact variance reduction on this estimate , and use the result to produce a practical learning algorithm . 3.1 PRELIMINARIES : PROBLEM AND NOTATION . Standard deep reinforcement learning seeks to maximize the expected reward of an agent over encountered trajectories ; that is , it maximizes the objective J ( θ ) = Eτ∼pθ ( τ ) [ ∑ t r ( st , at ) ] . ( 1 ) Here pθ ( τ ) is the distribution over trajectories τ ≡ s1 , a1 , . . . , sT , aT induced by a policy parameterized by θ ; st , at , and r ( st , at ) denote the state , action , and reward at time t , respectively . To enable the incorporation of distributional context , we instead consider the objective J ( θ ) = ∫ +∞ −∞ u ( r ( τ ) ) d dr ( τ ) ( w ( Pθ ( r ( τ ) ) ) dr ( τ ) , ( 2 ) where u ( r ( τ ) ) is the utility associated with full-trajectory reward r ( τ ) ≡ ∑ t r ( st , at ) and w is a piecewise differentiable weighting function of the CDF of trajectory reward Pθ ( r ( τ ) ) =∫ r ( τ ) −∞ pθ ( r ′ ) dr′ . Equation 2 is inspired by CPT ( Tversky & Kahneman , 1992 ) , which includes a pair of integrals of this form . It was chosen for its generality ; by using different utility functions u and weight functions w one may represent all of the risk measures mentioned in Section 2 and all of the risk measures evaluated by Dabney et al . ( 2018a ) . The form ( 2 ) reduces to ( 1 ) when u and w are both the identity mapping . While designed for the episodic setting , the objective ( 2 ) may be considered for infinite horizons through the choice of appropriately long windows . 3.2 RISK-SENSITIVE POLICY GRADIENT . To optimize the objective ( 2 ) , we first derive an approximation to its gradient with respect to the policy parameters θ . Working toward a representation that can be sampled , we assert the independence of the reward on θ and use the chain rule to write ∇θJ ( θ ) = ∫ ∞ −∞ u ( r ( τ ) ) d dr ( τ ) ( w′ ( Pθ ( r ( τ ) ) ) ∇θPθ ( r ( τ ) ) ) dr ( τ ) , ( 3 ) where w′ is the derivative of w with respect to Pθ ( r ( τ ) ) . The gradient of the CDF may be written as follows : ∇θPθ ( r ( τ ) ) =∇θ ∫ r ( τ ) −∞ pθ ( r ′ ) dr′ = ∇θ ∫ τ ′ H ( r ( τ ) − r ( τ ′ ) ) pθ ( τ ′ ) dτ ′ = ∫ τ ′ H ( r ( τ ) −r ( τ ′ ) ) ∇θpθ ( τ ′ ) dτ ′ = ∫ τ ′ H ( r ( τ ) −r ( τ ′ ) ) pθ ( τ ′ ) ∇θ log pθ ( τ ′ ) dτ ′ . ( 4 ) Here the first equality is the integral representation of Pθ ( r ( τ ) ) , the second uses the Heaviside step function to select all trajectories with total reward ≤ r ( τ ) , the third follows from the independence of reward on θ , and the fourth follows from the expression for the derivative of the natural logarithm . In the following , we also use the complementary expression ∇θPθ ( r ( τ ) ) = ∇θ ( 1− ∫ ∞ r ( τ ) pθ ( r ′ ) dr′ ) = − ∫ τ ′ H ( r ( τ ′ ) − r ( τ ) ) pθ ( τ ′ ) ∇θ log pθ ( τ ′ ) dτ ′ . ( 5 ) Either form , or a combination of the two , may be substituted into ( 3 ) and the result sampled over N trajectories by first ordering trajectories i = 1 . . . N by increasing reward r ( τ ) . Then ∇θJ ( θ ) ≈ N∑ i=1 u ( r ( τi ) ) ( w′ ( i N ) ∇θPθ ( r ( τi ) ) − w′ ( i− 1 N ) ∇θPθ ( r ( τi−1 ) ) ) , ( 6 ) where the term w′ ( 0 ) ∇θPθ ( r ( τ0 ) ) ≡ 0 . This ordering scheme produces an asymptotically consistent estimate , as shown in the context of CPT value estimation by Prashanth et al . ( 2016 ) . ∇θPθ ( r ( τi ) ) may be sampled in one of two ways , based on either ( 4 ) or ( 5 ) : ∇θPθ ( r ( τi ) ) ≈ 1 N i∑ j=1 Tj∑ t=1 ∇θ log πθ ( aj , t|sj , t ) ≈ − 1 N N∑ j=i+1 Tj∑ t=1 ∇θ log πθ ( aj , t|sj , t ) . ( 7 ) The expression ( 6 ) may be used to train a policy that optimizes the distributional objective ( 2 ) in a manner similar to REINFORCE ( Williams , 1992 ) .

Summary: This paper considers a generalization of the policy gradient method to optimize for arbitrary utility functions with weightings that depend on the entire CDF (rather than the expected reward). This generalization has two aspects: (1) a utility function on top of the trajectory reward and (2) a weighting function for the CDF of the trajectory reward with respect to which the expectation is performed.  The paper derives an expression for the policy gradient and also generalizes the standard variance reduction baselines. Inspired by the PPO loss, the authors then propose a clipped version of the policy gradient calling it C3PO and evaluate this on some benchmarks from the OpenAI Safety Gym, where it is found that the conservative weightings can offer improvements over the standard formulation.