Deep Reinforcement Learning for Equal Risk Option Pricing and Hedging under Dynamic Expectile Risk Measures

1 INTRODUCTION . This paper considers solving risk-averse dynamic decision making problems arising from applications where risk needs to be evaluated according to risk measures that are coherent . In particular , we draw our motivation from the financial application of equal-risk pricing ( ERP ) and hedging ( Guo & Zhu ( 2017 ) ) , where two dynamic hedging problems need to be solved , one for the buyer and one for the seller of a financial derivative ( a.k.a option ) , for determining a fair transaction price that would expose both parties to the same amount of hedging risk . The need to meaningfully model each party ’ s best hedging decision in a financial market , namely that no arbitrage is allowed , and to have a meaningful comparison between the two parties ’ risk exposures , namely that the risks should be measured in the same units , has led to the use of coherent risk measures for capturing both parties ’ hedging risks in this application ( see Marzban et al . ( 2020 ) ) . To this date , most solution methods proposed for solving risk-averse dynamic decision making problems under a coherent risk measure have either relied on traditional dynamic programming ( DP ) , which suffers from the curse of dimensionality and assumes the knowledge of a stochastic model that precisely captures the dynamics of the decision environment , or on the use of a static risk measure , i.e. , that disregards the temporal structure of the random variable ( e.g . Marzban et al . ( 2020 ) , Carbonneau & Godin ( 2020 ) , and Carbonneau & Godin ( 2021 ) in the case of the ERP application ) . The latter raises the serious issue that the resulting policy could be time inconsistent , i.e . that the actions prescribed by the policy may be considered significantly sub-optimal once the state is visited . In an application such as ERP , this issue implies that policies obtained based on static risk measures will not be implemented in practice , raising the need to consider dynamic risk measures . Focusing on deep reinforcement learning ( DRL ) methods , while there has been a large number of approaches proposed to address risk averse Markov decision processes ( MDPs ) using coherent risk measures , to the best of our knowledge , all of them , except for two exceptions , consider a static risk measure ( see Prashanth & Ghavamzadeh ( 2013 ) ; Chow & Ghavamzadeh ( 2014 ) ; Castro et al . ( 2019 ) ; Singh et al . ( 2020 ) ; Urpı́ et al . ( 2021 ) ) and therefore suffer from time-inconsistency . The two exceptions consist of Tamar et al . ( 2015 ) and Huang et al . ( 2021 ) who propose actor-critic reinforcement learning ( ACRL ) algorithms to deal with a general dynamic law-invariant coherent risk measures . Unfortunately , the two algorithms respectively either assume that it is possible to generate samples from a perturbed version of the dynamics , or rely on training three neural networks ( namely a state distribution reweighting network , a transition perturbation network , and a Lagrangean penalisation network ) concurrently with the actor and critic networks . Furthermore , only Huang et al . ( 2021 ) actually implemented their method . This was done on a toy tabular problem involving 12 states and 4 actions where it produced questionable performances1 . In this paper , we develop a new model-free ACRL algorithm for solving a time-consistent risk averse MDP under a dynamic expectile risk measure.2 Overall , we may summarize the contribution as follows : • Our ACRL algorithm is the first to naturally extend the popular model-free deep deterministic policy gradient algorithm ( DDPG ) ( see Lillicrap et al . ( 2015 ) ) to a risk averse setting where a time consistent coherent risk measure is used . Unlike the ACRL proposed in Huang et al . ( 2021 ) , which employs five neural networks , our algorithm will only require an actor and a critic network . While our policy network will be trained following a stochastic gradient procedure similar to Silver et al . ( 2014 ) , we are the first to leverage the elicitability property of expectile risk measures to propose a procedure for training the “ risk-to-go ” deep Q-network that is also based on stochastic gradient descent . • Our ACRL is the first model-free DRL-based algorithm capable of identifying optimal risk averse option hedging strategies that are time-consistent with respect to a dynamic coherent risk measure , and of computing their associated equal risk prices . A side benefit of timeconsistency will be that after training for an option with a given maturity , one obtains equal risk prices and hedging strategies for any other shorter maturities . While our algorithm certainly has a broader set of applications , we believe that ERP constitutes an original and fertile application in which to develop and test new risk averse DRL methods . • We evaluate the training efficiency and the quality of solution , in terms of quality of option hedging strategies and of estimated equal risk prices , obtained using our ACRL algorithm on a synthetic multi-asset geometric Brownian motion market model . These experiments constitute the first real application of a risk averse DRL algorithm that employs a dynamic coherent risk measure . The rest of this paper is organized as follows . Section 2 introduces equal risk pricing and its associated DP equations . Section 3 proposes the new ACRL algorithm for general finite horizon risk averse MDP with dynamic expectile measures . Finally , Section 4 presents and discusses our numerical experiments . We note that a reader only interested in the ACRL algorithm can skip right ahead to Section 3 . 2 APPLICATION : EQUAL RISK PRICING AND HEDGING UNDER DYNAMIC . EXPECTILE RISK MEASURES As described in Marzban et al . ( 2020 ) , the problem of ERP can be formalized as follows . Consider a frictionless market , i.e . no transaction cost , tax , etc , that contains m risky assets , and a risk-free bank account with zero interest rate . Let St : ⌦ ! Rm denote the values of the risky assets adapted to a filtered probability space ( ⌦ , F , F : = { Ft } Tt=0 , P ) , i.e . each St is Ft measurable . It is assumed that St is a locally bounded real-valued semi-martingale process and that the set of equivalent local martingale measures is non-empty ( i.e . no arbitrage opportunity ) . The set of all admissible selffinancing hedging strategies with the initial capital p0 2 R is shown by X ( p0 ) : X ( p0 ) = ( X : ⌦ ! RT 9 { ⇠t } T 1 t=0 , Xt = p0 + t 1X t0=0 ⇠ > t0 St0+1 , 8t = 1 , . . . , T ) , 1At the time of writing this paper , the risk averse implementation of this algorithm reported in Huang et al . ( 2021 ) is unable to recommend an optimal policy in a deterministic setting , while the risk neutral implementation produces policies that are outperformed by risk averse ones in a stochastic setting . 2Our ACRL algorithm exploits the elicitabilty property of expectile risk measures , which is the only elicitable coherent risk measure . where St+1 : = St+1 St , the hedging strategy ⇠t 2 Rm is a vector of random variables adapted to the filtration F and captures the number of shares of each risky asset held in the portfolio during the period [ t , t+ 1 ] , and Xt is the accumulated wealth . Let F ( { St } Tt=1 ) denote the payoff of a derivative . Throughout this paper , we assume F ( { St } Tt=1 ) admits the formulation of F ( ST , YT ) where Yt is an auxiliary fixed-dimensional stochastic process that is Ft-measurable . This class of payoff functions is common in the literature , ( see for example Bertsimas et al . ( 2001 ) and Marzban et al . ( 2020 ) ) . The problem of ERP is defined based on the following two hedging problems that seek to minimize the risk of hedging strategies , one is for the writer and the other is for the buyer of the derivative : ( Writer ) % w ( p0 ) = inf X2X ( p0 ) ⇢w ( F ( ST , YT ) XT ) ( 1 ) ( Buyer ) % b ( p0 ) = inf X2X ( p0 ) ⇢b ( F ( ST , YT ) XT ) , ( 2 ) where ⇢w and ⇢b are two risk measures that capture respectively the writer and the buyer ’ s risk aversion . In words , equation ( 1 ) describes a writer that is receiving p0 as the initial payment and implements an optimal hedging strategy for the liability F ( ST , YT ) . On the other hand , in ( 2 ) the buyer is assumed to borrow p0 in order to pay for the option and then to manage a portfolio that will minimize the risks associated to his final wealth . With equations ( 1 ) and ( 2 ) , ERP defines a fair price p⇤0 as the value of an initial capital that leads to the same risk exposure to both parties , i.e . % w ( p⇤0 ) = % b ( p⇤0 ) . In particular , based on Proposition 3.1 and the examples presented in section 3.3 of Marzban et al . ( 2020 ) , together with the fact that both ⇢w and ⇢b are dynamic recursive law invariant risk measures , a Markovian assumption allows us to conclude that the ERP can be calculated using two sets of dynamic programming equations . Assumption 2.1 . [ Markov property ] There exists a sufficient statistic process t adapted to F such that { ( St , Yt , t ) } Tt=0 is a Markov process relative to the filtration F. Namely , P ( ( St+s , Yt+s , t+s ) 2 A|Ft ) = P ( ( St+s , Yt+s , t+s ) 2 A|St , Yt , t ) for all t , for all s 0 , and all sets A . Specifically , on the writer side , we can define V wT ( ST , YT , T ) : = F ( ST , YT ) , and recursively V wt ( St , Yt , t ) : = inf ⇠t ⇢̄ ( ⇠ > t St+1 + V wt+1 ( St + St+1 , Yt + Yt+1 , t+1 ) |St , Yt , t ) , where ⇢̄ ( ·|St , Yt , t ) is a law invariant risk measure that uses P ( ·|St , Yt , t ) . This leads to considering % w ( 0 ) = V w0 ( S0 , Y0 , 0 ) . On the other hand , for the buyer we similarly define : V bT ( ST , YT , T ) : = F ( ST , YT ) and V bt ( St , Yt , t ) : = inf ⇠t ⇢̄ ( ⇠ > t St+1 + V bt+1 ( St + St+1 , Yt + Yt+1 , t+1 ) |St , Yt , t ) , with % b ( 0 ) = V b0 ( S0 , Y0 , 0 ) . The following lemma summarizes how DP can be used to compute the ERP . Lemma 2.1 ( Marzban et al . ( 2020 ) ) . Under Assumption 2.1 , the ERP that employs dynamic expectile risk measure can be computed as : p⇤0 = ( V w 0 ( S0 , Y0 , 0 ) V b0 ( S0 , Y0 , 0 ) ) /2 . In what follows , we will further assume that the risk measure is a dynamic expectile risk measure . Definition 2.1 . A dynamic expectile risk measure takes the form : ⇢ ( X ) : = ⇢̄0 ( ⇢̄1 ( . . . ⇢̄T 1 ( X ) ) ) where each ⇢̄ ( · ) is an expectile risk measure that employs the conditional distribution based on Ft. Namely , ⇢̄t ( Xt+1 ) : = argminq ⌧E ⇥ ( q Xt+1 ) 2+|Ft ⇤ + ( 1 ⌧ ) E ⇥ ( q Xt+1 ) 2 |Ft ⇤ where Xt+1 is a random liability measureable on Ft+1 . Like conditional value at risk , the expectile measure ( see Bellini & Bignozzi ( 2015 ) ) covers the range of risk attitudes from risk neutrality , when ⌧ = 1/2 , to worst-case risk , when ⌧ ! 1 .

Summary: The paper offers a solution to the problem of Equal Risk Pricing (ERP) by framing it as a risk-sensitive MDP formulation. In particular, the authors considers a dynamic risk-measures using expectiles measures as conditional risk-measure for each step. An extension of DDPG is derived by exploiting the properties of the chosen risk-measure and it is tested on two experiments. The first one involves a vanilla option, while the second one feature a more complex basket option.