Policy Gradient with Expected Quadratic Utility Maximization: A New Mean-Variance Approach in Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) with Markov decision processes ( MDPs ) is one type of dynamic decision-making problem ( Puterman , 1994 ; Sutton & Barto , 1998 ) . While the typical objective is the expected cumulative reward maximization , risk-aware decision-making has attracted great attention in real-world applications , such as finance and robotics ( Geibel & Wysotzki , 2005 ; García & Fernández , 2015 ) . The notion of risk is related to the fact that even an optimal policy may perform poorly owing to the stochastic nature of the problem . To capture the risk , various criteria have been proposed , such as Value at Risk ( Luenberger , 1998 ; Chow & Ghavamzadeh , 2014 ; Chow et al. , 2017 ) and variance ( Markowitz , 1952 ; Markowitz et al. , 2000 ; Tamar et al. , 2012 ; Prashanth & Ghavamzadeh , 2013 ) . Among them , we focus on the mean-variance trade-off . Typical mean-variance RL ( MVRL ) methods attempt to maximize the expected cumulative reward while maintaining the variance threshold ( Tamar et al. , 2012 ; Prashanth & Ghavamzadeh , 2013 ; 2016 ; Xie et al. , 2018 ; Bisi et al. , 2020 ; Zhang et al. , 2020 ) . However , most existing MVRL methods suffer from high computational costs owing to the double sampling issue when approximating the gradient of the variance term ( Tamar et al. , 2012 ; Prashanth & Ghavamzadeh , 2013 ; 2016 ) . To avoid the double sampling issue , Xie et al . ( 2018 ) proposed a method based on the Legendre-Fenchel duality ( Boyd & Vandenberghe , 2004 ) . Although the method does not suffer from the double sampling issue , we can not apply a standard policy gradient method and must use a coordinate descent algorithm . In addition , the method can not control risk at a certain desirable level . From an economics perspective , the difference in the RL objectives arises from the forms of utility functions . When the objective of an agent is expected cumulative reward maximization , the utility function is risk-neutral ; when an agent attempts to control the risk based on an expected reward , the utility function is risk-averse ( Mas-Colell et al. , 1995 ) . In economics , there have been several risk-averse utility functions proposed . The quadratic utility function is one such functions and is frequently used in financial economics ( Luenberger , 1998 ) . Under the quadratic utility function , the mean-variance portfolio maximizes the utility of the investor . In addition , other various financial theories are also based on the quadratic utility maximization ( Markowitz , 1952 ; Sharpe , 1964 ; Lintner , 1965 ; Mossin , 1966 ) . For more details , see Appendix A . In this study , as one of the MVRL approaches , we consider the expected quadratic utility maximization ( EQUM ) based on the policy gradient method ( Williams , 1988 ; 1992 ; Sutton et al. , 1999 ; Baxter & Bartlett , 2001 ) . The EQUM has the following advantages : ( i ) low computational cost ; ( ii ) numerous interpretations , and ( iii ) direct connections to real-world applications . In this study , as interpretations of EQUM , we propose the minimization of the variance under the constraint of the expected cumulative reward , reward-targeting optimization , and regularization . Thus , this study contributes to the context of riskaverse RL and MVRL . In the following sections , we first formulate the problem setting in Section 2 and review the existing methods in Section 3 . Then , we propose the main algorithms in Section 4 . Finally , we investigate the empirical effectiveness of the EQUM in Section 5 . 2 PROBLEM SETTING . We consider the standard RL framework , where a learning agent interacts with an unfamiliar , dynamic , and stochastic environment modeled by a Markov decision process ( MDP ) in discrete time . We define the MDP as the tuple ( S , A , r , P , P0 ) , where S is a set of states , A is a set of actions , r : S × A → R is a reward function , P : S × S × A → [ 0 , 1 ] is the transition kernel , and P0 : S → [ 0 , 1 ] is an initial state distribution . The initial state S1 is sampled from P0 . Let πθ : A × S → [ 0 , 1 ] be a parameterized stochastic policy mapping states to actions , where θ is the tunable parameter . At time step t , an agent chooses an actionAt according to a policy πθ ( · | St ) . We assume that the policy πθ is a differentiable function with respect to θ ; that is , ∂πθ ( a , s ) ∂θ exists . There are several performance measures for a policy πθ . One popular measure is the expected cumulative reward from time step t to u defined as Eπθ [ Rt : u ] , where Rt : u = ∑u i=0 γ ir ( St+i , At+i ) , γ ∈ ( 0 , 1 ) is a discount factor and Eπθ denotes the expectation operator over a policy πθ , and S1 is generated from P0 . When γ = 1 , to ensure the cumulative reward well defined , it is usually assumed that all policies are proper ( Bertsekas & Tsitsiklis , 1996 ) ; that is , for any policy πθ , the agent goes to a recurrent state S∗ with probability 1 . After the agent passes the recurrent state S∗ at a time τ , the rewards are always 0 . Such a finite horizon case is called episodic MDPs ( Puterman , 1994 ) . For brevity , we denote Rt : u as R when the meaning is obvious . Under these criteria , the agent may attempt to obtain a higher cumulative reward while taking higher risks . In real-world applications , such as portfolio management in finance ( Markowitz , 1952 ; Markowitz et al. , 2000 ) , such risky decision-making is not always desired , and we , therefore , consider the trade-off between the expected cumulative reward and the variance . Thus , while the goal of the risk-neutral MDP problem is to find the parameter θ that maximizes the total reward , we consider the mean-variance trade-off between the cumulative expected reward and the variance of the cumulative reward R in the MVRL problem . Note that even when the reward r is deterministic , the cumulative reward is a random variable owing to the stochastic policy , and there exists the mean-variance trade-off . In addition , even if the optimal policy is deterministic , the proposed method empirically has the potential to improves the stability of the training from the observations in the experiments of Section 5.3 . 3 EXISTING MVRL METHODS . In this section , we introduce existing studies of MVRL . 3.1 CONSTRAINED TRAJECTORY-VARIANCE PROBLEM . Tamar et al . ( 2012 ) , Prashanth & Ghavamzadeh ( 2013 ) , and Xie et al . ( 2018 ) formulated MVRL by a constrained optimization problem defined as maxθ∈Θ Eπθ [ R ] s.t . Varπθ ( R ) ≤ η . In their formulation , the goal is to maximize the expected cumulative reward with controlling the trajectory-variance at a certain level . To solve this problem , Tamar et al . ( 2012 ) , Prashanth & Ghavamzadeh ( 2013 ) , and Xie et al . ( 2018 ) consider a penalized method defined as maxθ∈Θ Eπθ [ R ] − δg ( Varπθ ( R ) − η ) , where δ > 0 is a constant and g : R → R is a penalty function , such as g ( x ) = x or g ( x ) = x2 . 3.1.1 DOUBLE SAMPLING ISSUE . Tamar et al . ( 2012 ) , Prashanth & Ghavamzadeh ( 2013 ) , and Xie et al . ( 2018 ) report the double sampling issue in MVRL , which requires sampling from two different trajectories for estimating the policy gradient . For instance , in an episodic MDP with the discount factor γ = 1 and the stop- ping time τ = min { t | St = S∗ } , the gradients of Eπθ [ R ] , Eπθ [ R2 ] , and ( Eπθ [ R ] ) 2 are given as ∇θEπθ [ R ] = Eπθ [ R ∑τ t=1 ∇θ log πθ ( St , At ) ] , ∇θEπθ [ R2 ] = E [ R2 ∑τ t=1 ∇θ log πθ ( St , At ) ] , and ∇θ ( Eπθ [ R ] ) 2 = 2Eπθ [ R ] ∇θEπθ [ R ] ( Tamar et al. , 2012 ) . Besides , the gradient of the vari- ance is given as Eπθ [ R2 ∑τ t=1 ∇θ log πθ ( St , At ) ] − 2Eπθ [ R ] ∇θEπθ [ R ] . Because optimizing the policy πθ using the gradients directly is computationally intractable , we replace them with their unbiased estimators . Suppose that there is a simulator generating a trajectory k with { ( Skt , Akt , r ( Skt , Akt ) ) } τ k t=1 , where τ k is the stopping time of the trajectory . Then , we can naively construct unbiased estimators of Eπθ [ R ] and Eπθ [ R2 ] as ∇̂θEπθ [ R ] = R̂k ∑τk t=1 ∇θ log πθ ( Skt , Akt ) and ∇̂θEπθ [ R2 ] = ( R̂k ) 2 ∑τk t=1 ∇θ log πθ ( Skt , Akt ) , where R̂k is a sample approximation of Eπθ [ R ] at the episode k. However , obtaining an unbiased estimator of ∇θ ( Eπθ [ R ] ) 2 = 2Eπθ [ R ] ∇θEπθ [ R ] is difficult because it requires sampling from two different trajectories for approximating Eπθ [ R ] and ∇θEπθ [ R ] . This issue makes the optimization problem difficult . 3.1.2 EXISTING SOLUTIONS TO DOUBLE SAMPLING ISSUE . For the double sampling issue , Tamar et al . ( 2012 ) , Prashanth & Ghavamzadeh ( 2013 ) , and Xie et al . ( 2018 ) presented the following solutions . Multi-time-scale stochastic optimization : The proposed methods of Tamar et al . ( 2012 ) and Prashanth & Ghavamzadeh ( 2013 ) are based on stochastic approximation to find an equilibrium point of an ordinary differential equation and saddle point of the objective , respectively . Coordinate descent optimization : Xie et al . ( 2018 ) proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm . First , based on Lagrangian relaxation , Xie et al . ( 2018 ) set an objective function as maxθ∈Θ Eπθ [ R ] − δ ( Varπθ ( R ) − η ) . Then , Xie et al . ( 2018 ) transformed the objective function as maxθ∈Θ , y∈R 2y ( Eπθ [ R ] + 12δ ) −y2−Eπθ [ R2 ] and estimated a parameter by solving the optimization problem via a coordinate descent algorithm . Weakness of existing approaches : The multi-time-scale approaches by Tamar et al . ( 2012 ) and Prashanth & Ghavamzadeh ( 2013 ) are known to be sensitive to the choice of the step-size schedules , which are not easy to be controlled ( Xie et al. , 2018 ) . The method by Xie et al . ( 2018 ) does not reflect the constraint condition η as shown above ; that is , in the objective function of Xie et al . ( 2018 ) , there exits penalty coefficient δ , but does not exist the constraint condition η . Note that the problem of Xie et al . ( 2018 ) is owing to their objective function based on the penalty function g ( x ) = x : Eπθ [ R ] − δ ( Varπθ ( R ) − η ) , in which the first derivative does not include η . In addition , when using quadratic function as g ( x ) = x2 to consider η , we can not remove E [ R2 ] even with Legendre-Fenchel dual ; that is , the method also suffers the double sampling issue .

Summary: The paper proposes a policy gradient style RL algorithm that optimizes an expected quadratic utility, a commonly used objective of risk management in finance and economics. The key idea here is based on the observation that when using the quadratic utility function, the use of mean-variance RL methods can be shown to optimize the utility of the agent. To this effect, the paper considers the use of expected quadratic utility maximization in the policy gradient. The quadratic utility can be naturally modeled using mean and variance. The paper implements two variations -- policy gradient and actor-critic with EQUM framework.