Year: 2021

Abstract:
Motivated by the application of equal-risk pricing and hedging of a financial derivative, where two operationally meaningful hedging portfolio policies needs to be found that minimizes coherent risk measures, we propose in this paper a novel deep reinforcement learning algorithm for solving risk-averse dynamic decision making problems. Prior to our work, such hedging problems can either only be solved based on static risk measures, leading to time-inconsistent policies, or based on dynamic programming solution schemes that are impracticable in realistic settings. Our work extends for the first time the deep deterministic policy gradient algorithm, an off-policy actor-critic reinforcement learning (ACRL) algorithm, to solving dynamic problems formulated based on time-consistent dynamic expectile risk measure. Our numerical experiments confirm that the new ACRL algorithm produces high quality solutions to equal-risk pricing and hedging problems and that its hedging strategy outperforms the strategy produced using a static risk measure when the risk is evaluated at later points of time.

heading: 1 INTRODUCTION
text: This paper considers solving risk-averse dynamic decision making problems arising from applications where risk needs to be evaluated according to risk measures that are coherent. In particular, we draw our motivation from the financial application of equal-risk pricing (ERP) and hedging (Guo & Zhu (2017)), where two dynamic hedging problems need to be solved, one for the buyer and one for the seller of a financial derivative (a.k.a option), for determining a fair transaction price that would expose both parties to the same amount of hedging risk. The need to meaningfully model each party’s best hedging decision in a financial market, namely that no arbitrage is allowed, and to have a meaningful comparison between the two parties’ risk exposures, namely that the risks should be measured in the same units, has led to the use of coherent risk measures for capturing both parties’ hedging risks in this application (see Marzban et al. (2020)).
To this date, most solution methods proposed for solving risk-averse dynamic decision making problems under a coherent risk measure have either relied on traditional dynamic programming (DP), which suffers from the curse of dimensionality and assumes the knowledge of a stochastic model that precisely captures the dynamics of the decision environment, or on the use of a static risk measure, i.e., that disregards the temporal structure of the random variable (e.g. Marzban et al. (2020), Carbonneau & Godin (2020), and Carbonneau & Godin (2021) in the case of the ERP application). The latter raises the serious issue that the resulting policy could be time inconsistent, i.e. that the actions prescribed by the policy may be considered significantly sub-optimal once the state is visited. In an application such as ERP, this issue implies that policies obtained based on static risk measures will not be implemented in practice, raising the need to consider dynamic risk measures.
Focusing on deep reinforcement learning (DRL) methods, while there has been a large number of approaches proposed to address risk averse Markov decision processes (MDPs) using coherent risk measures, to the best of our knowledge, all of them, except for two exceptions, consider a static risk measure (see Prashanth & Ghavamzadeh (2013); Chow & Ghavamzadeh (2014); Castro et al. (2019); Singh et al. (2020); Urpı́ et al. (2021)) and therefore suffer from time-inconsistency. The
two exceptions consist of Tamar et al. (2015) and Huang et al. (2021) who propose actor-critic reinforcement learning (ACRL) algorithms to deal with a general dynamic law-invariant coherent risk measures. Unfortunately, the two algorithms respectively either assume that it is possible to generate samples from a perturbed version of the dynamics, or rely on training three neural networks (namely a state distribution reweighting network, a transition perturbation network, and a Lagrangean penalisation network) concurrently with the actor and critic networks. Furthermore, only Huang et al. (2021) actually implemented their method. This was done on a toy tabular problem involving 12 states and 4 actions where it produced questionable performances1.
In this paper, we develop a new model-free ACRL algorithm for solving a time-consistent risk averse MDP under a dynamic expectile risk measure.2 Overall, we may summarize the contribution as follows:
• Our ACRL algorithm is the first to naturally extend the popular model-free deep deterministic policy gradient algorithm (DDPG) (see Lillicrap et al. (2015)) to a risk averse setting where a time consistent coherent risk measure is used. Unlike the ACRL proposed in Huang et al. (2021), which employs five neural networks, our algorithm will only require an actor and a critic network. While our policy network will be trained following a stochastic gradient procedure similar to Silver et al. (2014), we are the first to leverage the elicitability property of expectile risk measures to propose a procedure for training the “risk-to-go” deep Q-network that is also based on stochastic gradient descent.
• Our ACRL is the first model-free DRL-based algorithm capable of identifying optimal risk averse option hedging strategies that are time-consistent with respect to a dynamic coherent risk measure, and of computing their associated equal risk prices. A side benefit of timeconsistency will be that after training for an option with a given maturity, one obtains equal risk prices and hedging strategies for any other shorter maturities. While our algorithm certainly has a broader set of applications, we believe that ERP constitutes an original and fertile application in which to develop and test new risk averse DRL methods.
• We evaluate the training efficiency and the quality of solution, in terms of quality of option hedging strategies and of estimated equal risk prices, obtained using our ACRL algorithm on a synthetic multi-asset geometric Brownian motion market model. These experiments constitute the first real application of a risk averse DRL algorithm that employs a dynamic coherent risk measure.
The rest of this paper is organized as follows. Section 2 introduces equal risk pricing and its associated DP equations. Section 3 proposes the new ACRL algorithm for general finite horizon risk averse MDP with dynamic expectile measures. Finally, Section 4 presents and discusses our numerical experiments. We note that a reader only interested in the ACRL algorithm can skip right ahead to Section 3.

heading: 2 APPLICATION: EQUAL RISK PRICING AND HEDGING UNDER DYNAMIC
text: EXPECTILE RISK MEASURES
As described in Marzban et al. (2020), the problem of ERP can be formalized as follows. Consider a frictionless market, i.e. no transaction cost, tax, etc, that contains m risky assets, and a risk-free bank account with zero interest rate. Let St : ⌦! Rm denote the values of the risky assets adapted to a filtered probability space (⌦,F ,F := {Ft}Tt=0,P), i.e. each St is Ft measurable. It is assumed that St is a locally bounded real-valued semi-martingale process and that the set of equivalent local martingale measures is non-empty (i.e. no arbitrage opportunity). The set of all admissible selffinancing hedging strategies with the initial capital p0 2 R is shown by X (p0):
X (p0) = ( X : ⌦! RT 9{⇠t} T 1 t=0 , Xt = p0 + t 1X
t0=0
⇠>t0 St0+1, 8t = 1, . . . , T ) ,
1At the time of writing this paper, the risk averse implementation of this algorithm reported in Huang et al. (2021) is unable to recommend an optimal policy in a deterministic setting, while the risk neutral implementation produces policies that are outperformed by risk averse ones in a stochastic setting.
2Our ACRL algorithm exploits the elicitabilty property of expectile risk measures, which is the only elicitable coherent risk measure.
where St+1 := St+1 St, the hedging strategy ⇠t 2 Rm is a vector of random variables adapted to the filtration F and captures the number of shares of each risky asset held in the portfolio during the period [t, t+ 1], and Xt is the accumulated wealth.
Let F ({St}Tt=1) denote the payoff of a derivative. Throughout this paper, we assume F ({St}Tt=1) admits the formulation of F (ST ,YT ) where Yt is an auxiliary fixed-dimensional stochastic process that is Ft-measurable. This class of payoff functions is common in the literature, (see for example Bertsimas et al. (2001) and Marzban et al. (2020)). The problem of ERP is defined based on the following two hedging problems that seek to minimize the risk of hedging strategies, one is for the writer and the other is for the buyer of the derivative:
(Writer) %w(p0) = inf X2X (p0) ⇢w(F (ST ,YT ) XT ) (1)
(Buyer) %b(p0) = inf X2X ( p0) ⇢b( F (ST ,YT ) XT ) , (2)
where ⇢w and ⇢b are two risk measures that capture respectively the writer and the buyer’s risk aversion. In words, equation (1) describes a writer that is receiving p0 as the initial payment and implements an optimal hedging strategy for the liability F (ST ,YT ). On the other hand, in (2) the buyer is assumed to borrow p0 in order to pay for the option and then to manage a portfolio that will minimize the risks associated to his final wealth. With equations (1) and (2), ERP defines a fair price p⇤0 as the value of an initial capital that leads to the same risk exposure to both parties, i.e. %w(p⇤0) = % b(p⇤0).
In particular, based on Proposition 3.1 and the examples presented in section 3.3 of Marzban et al. (2020), together with the fact that both ⇢w and ⇢b are dynamic recursive law invariant risk measures, a Markovian assumption allows us to conclude that the ERP can be calculated using two sets of dynamic programming equations. Assumption 2.1. [Markov property] There exists a sufficient statistic process t adapted to F such that {(St,Yt, t)}Tt=0 is a Markov process relative to the filtration F. Namely, P((St+s,Yt+s, t+s) 2 A|Ft) = P((St+s,Yt+s, t+s) 2 A|St,Yt, t) for all t, for all s 0, and all sets A.
Specifically, on the writer side, we can define V wT (ST ,YT , T ) := F (ST ,YT ), and recursively
V wt (St,Yt, t) := inf ⇠t ⇢̄( ⇠>t St+1 + V wt+1(St + St+1,Yt + Yt+1, t+1)|St,Yt, t) ,
where ⇢̄(·|St,Yt, t) is a law invariant risk measure that uses P(·|St,Yt, t). This leads to considering %w(0) = V w0 (S0,Y0, 0). On the other hand, for the buyer we similarly define: V bT (ST ,YT , T ) := F (ST ,YT ) and
V bt (St,Yt, t) := inf ⇠t ⇢̄( ⇠>t St+1 + V bt+1(St + St+1,Yt + Yt+1, t+1)|St,Yt, t) ,
with %b(0) = V b0 (S0,Y0, 0). The following lemma summarizes how DP can be used to compute the ERP. Lemma 2.1 (Marzban et al. (2020)). Under Assumption 2.1, the ERP that employs dynamic expectile risk measure can be computed as: p⇤0 = (V w 0 (S0,Y0, 0) V b0 (S0,Y0, 0))/2.
In what follows, we will further assume that the risk measure is a dynamic expectile risk measure. Definition 2.1. A dynamic expectile risk measure takes the form: ⇢(X) := ⇢̄0(⇢̄1(. . . ⇢̄T 1(X))) where each ⇢̄(·) is an expectile risk measure that employs the conditional distribution based on Ft. Namely, ⇢̄t(Xt+1) := argminq ⌧E ⇥ (q Xt+1)2+|Ft ⇤ + (1 ⌧)E ⇥ (q Xt+1)2 |Ft ⇤ where Xt+1 is a random liability measureable on Ft+1.
Like conditional value at risk, the expectile measure (see Bellini & Bignozzi (2015)) covers the range of risk attitudes from risk neutrality, when ⌧ = 1/2, to worst-case risk, when ⌧ ! 1.

heading: 3 A NOVEL ACTOR-CRITIC ALGORITHM FOR RISK AVERSE MDP UNDER A DYNAMIC EXPECTILE RISK MEASURE
text: With the dynamic programming equations in hand, it now becomes apparent that each option hedging problem in ERP can be formulated as a finite horizon Markov Decision Process (MDP) described
with (S,A, r, P ). In this regard, the agent (i.e. the writer or buyer) interacts with a stochastic environment by taking an action at ⌘ ⇠t 2 [ 1, 1]m after observing the state st 2 S , which includes St, Yt, and t. Note that to simplify exposition, in this section we drop the reference to the specific identity (i.e. w or b) of the agent in our notation. The action taken at each time t results in the immediate stochastic reward that takes the shape of the immediate hedging portfolio return, i.e. rt(st, at, st+1) := ⇠>t St+1 when t < T and otherwise of the option liability/payout rT (sT , aT , sT+1) := F (ST ,YT )(1 2 · 1{agent=writer}), which is insensitive to sT+1. Finally, the Markovian exogeneous dynamics described in Assumption 2.1 are modeled using P as P (st+1|st, at) = P(St+1,Yt+1, t+1|St,Yt, t). Overall, each of the two dynamic derivative hedging problems presented in Section 2 reduce to a version of the following general risk averse reinforcement learning problem:
%(0) = V0(S0,Y0, 0) = min ⇡
Q⇡0 (s̄0,⇡0(s̄0)) ,
where s̄0 := (S0,Y0, 0) is the initial state in which the option is priced while Q⇡t (st, at) := ⇢̄( rt(st, at, st+1) + Q⇡t+1(st+1,⇡t+1(st+1))|st), Q⇡T (sT , aT ) := rT (sT , aT , sT ), and where ⇢̄ is an expectile risk measure, i.e. ⇢̄(X) := argminq E ⇥ ⌧(q X)2+ + (1 ⌧)(q X)2 ⇤ . Equipped with these definitions, we can now motivate our proposed extension of the model-free off-policy deterministic ACRL algorithm to the general finite horizon risk-averse MDP setting. In doing so, we start with a proposition (see Appendix A.1 for a proof) that will provide the motivation for a stochastic gradient scheme to optimize the actor network, while the optimization of the critic network will follow from the elicitability property of the expectile risk measure. Proposition 3.1. Let ⇡̄ be an arbitrary reference policy and µ an arbitrary distribution over the initial state, such that there is a strictly positive probability of reaching all of S for all t 1.3 For any ⇡⇤ that satisfies
⇡⇤ 2 argmin ⇡ E t̃⇠{0,...,T},s0⇠µ st+1⇠P (·|st,⇡̄t(st)) [Q⇡t̃ (st̃,⇡t̃(st̃))] (3)
where t̃ is uniformly drawn, we necessarily have that ⇡⇤ 2 argminQ⇡0 (s̄0,⇡0(s̄0)) hence %(0) = Q⇡ ⇤
0 (s̄0,⇡ ⇤ 0(s̄0)).
In the context of a deep reinforcement learning approach, we can employ a procedure based on offpolicy deterministic policy gradient (Silver et al., 2014) to optimize (3). Specifically, given a policy network ⇡✓, we wish to optimize:
min ✓ E t̃⇠{0,...,T 1} st+1⇠P (·|st,⇡̄t(st))
[Q⇡ ✓ t̃ (st̃,⇡ ✓ t̃ (st̃))] ,
using a stochastic gradient algorithm. In doing so, we rely on the fact that:
r✓E t̃⇠{0,...,T 1} st+1⇠P (·|st,⇡̄t(st))
[Q⇡ ✓ t̃ (st̃,⇡ ✓(st̃))]
= E t̃⇠{0,...,T 1} st+1⇠P (·|st,⇡̄t(st))
 r✓Q⇡ ✓
t̃ (st̃, a) a=⇡✓
t̃ (st̃)
+ raQ⇡ ✓ t̃ (st̃, a)r✓⇡ ✓ t̃ (st̃)
a=⇡✓
t̃ (st̃)
⇡ E t̃⇠{0,...,T 1} st+1⇠P (·|st,⇡̄t(st))
 raQ⇡ ✓ t̃ (st̃, a)r✓⇡ ✓ t̃ (st̃)
a=⇡✓
t̃ (st̃)
.
Note that in the above equation, we have dropped the term that depends on r✓Q⇡ ✓
t̃ as is commonly
done in off-policy deterministic gradient methods and usually motivated by a result of Degris et al. (2012), who argue that this approximation preserves the set of local optima in a risk neutral setting, i.e. ⇢̄(·) := E[·]. While we do consider as an important subject of future research to extend this motivation to more general risk measures, our numerical experiments (see Section 4.3) will confirm empirically that the quality of this approximation permits the identification of nearly optimal hedging policies.
3In our option hedging problem, given that st is entirely exogenous, the distribution of st+1 is unaffected by ⇡̄, which can therefore be chosen arbitrarily. Moreover, µ can put all the mass on s̄0.
Given that we do not have access to an exact expression for Q⇡ ✓
t̃ (st̃, a), this operator needs to be
estimated directly from the training data. Exploiting the fact that ⇢ is a utility-based shortfall risk measure, we get that:
Q⇡t (st, at) 2 argminq Est+1⇠P (·|st,at)[`(q + rt(st, at, st+1) Q ⇡ t+1(st+1,⇡t+1(st+1)))]
where `(y) := (⌧1{y > 0} (1 ⌧)1{y  0})y2 is the score function associated to the ⌧ -expectile risk measure. As explained in Theorem 3.2 of Shen et al. (2014), in a tabular MDP environment one can apply the following stochastic gradient step:
Q̂t(st, at) Q̂t(st, at) ↵@`(Q̂t(st, at) + rt(st, at, st+1) Q̂t+1(st+1,⇡t+1(st+1)) ,
where @`(y) := 2(⌧ max(0, y) (1 ⌧)max(0, y)) is the derivative of `(y), within a properly designed Q-learning algorithm and have the guarantee that Q̂t(st, at) will almost surely converge to Q⇡t (st, at) for all t, st, and at.
In the non-tabular setting, we replace Q̂⇡t (st, at) with two estimators: i.e. the “main” network Q⇡t (st, at|✓Q) for the immediate conditional risk and the “target” network Q⇡t (st, at|✓Q 0 ) for the next period’s conditional risk. The procedure consists in iterating between a step that attempts to make the main network Q⇡t (st, at|✓Q) a good estimator of ⇢( r(st, at, st+1) + Q⇡t+1(st+1, at+1|✓Q 0 )) and a step that replaces the target network Q⇡t (st, at|✓Q 0 ) with a network more similar to the main one Q⇡t (st, at|✓Q). The former is achieved, similarly as with the policy network, by searching for the optimal ✓Q according to:
min ✓Q E t̃⇠{0,...,T 1} st+1⇠P (·|st,⇡̄t(st)) [`(Q⇡t̃ (st̃, ⇡̄t̃(st̃)|✓ Q)+rt(st̃, ⇡̄t̃(st̃), st̃+1) Q⇡t̃+1(st̃+1,⇡t̃+1(st̃+1)|✓
Q0))] ,
which suggests a stochastic gradient update of the form ✓Q ✓Q ↵ , where is
@`(Q⇡t̃ (st̃, ⇡̄t̃(st̃)|✓ Q)+rt(st̃, ⇡̄t̃(st̃), st̃+1) Q⇡t̃+1(st̃+1,⇡t̃+1(st̃+1)|✓ Q0))r✓QQ⇡t̃ (st̃, ⇡̄t̃(st̃)|✓ Q) .
These two types of updates are integrated in our proposed expectile-based actor-critic deep RL (a.k.a. ACRL) algorithm. A first version, Algorithm 1, is designed for a simulation-based environment. One may note that in each episode, the reference policy ⇡̄t is updated to be a perturbed version of the main policy network in order to focus the accuracy of the main critic network’s value and derivatives on actions that are more likely to be produced by the main policy network. We also choose to update the target networks using convex combinations operations as is done in Lillicrap et al. (2015) in order to improve stability of learning. A second more general version of ACRL, which mimics the original DDPG, by generating minibatches using a replay buffer can also be found in Appendix A.2.
We finally note that in our problem, P (st+1|st, at) = P (st+1|st, a0t) = P(St+1,Yt+1, t+1|St,Yt, t), meaning that the action is not affecting the distribution of state in the next period. This is a direct consequence of using a translation invariant risk measure, which eliminates the need to keep track of the accumulated wealth in the set of state variables as explained in Marzban et al. (2020) and allows the reward function to provide an immediate signal regarding the quality of implemented actions. In the context of our deep reinforcement learning approach, we observed that convergence speed is significantly improved in training due to this property (see Figure 4 in Appendix).

heading: 4 EXPERIMENTAL RESULTS
text: In this section we provide two different sets of experiments that are run over one vanilla and one basket option. We will compare both algorithmic efficiency and quality, in terms of pricing and hedging strategies, of the dynamic risk model (DRM), which employs a dynamic expectile risk measure and is solved using our new ACRL algorithm, and the static risk model (SRM), which employs a static expectile measure and is solved using an AORL algorithm similar to Carbonneau & Godin (2021). All experiments are done using simulated price processes of five risky assets: AAPL, AMZN, FB, JPM, and GOOGL. The price paths are simulated using correlated Brownian motions considering the empirical mean, variance, and the correlation matrix of five reference stocks (AAPL, AMZN, FB, KPM, and GOOGL) over the period that spans from January 2019 to January 2021. In
Algorithm 1: The actor-critic RL algorithm for the dynamic recursive expectile option hedging problem (ACRL) Randomly initialize the main actor and critic networks’ parameters ✓⇡ and ✓Q; Initialize the target actor, ✓⇡
0 ✓⇡ , and critic, ✓Q0 ✓Q, networks; for j = 1 : #Episodes do
Randomly select t 2 {0, 1, ..., T 1}; Sample a minibatch of N triplets {(sit, ait, sit+1)}Ni=1 from P (·|st, ⇡̄t(st)), where ⇡̄t(st) := ⇡t(st|✓⇡) +N (0, ); Set the realized losses yit := rt(sit, ait, sit+1) +Qt+1(sit+1,⇡t+1(sit+1|✓⇡ 0 )|✓Q0); Update the main critic network:
✓Q ✓Q ↵ 1
N
NX
i=1
@`(Qt(s i t, a i t|✓Q) yjt )r✓QQt(sit, ait|✓Q) ;
Update the main actor network:
✓⇡ ✓⇡ ↵ 1 N
NX
i=1
raQt(sit, a|✓Q)|a=⇡t(sit|✓⇡)r✓⇡⇡t(s i t|✓⇡) ;
Update the target networks:
✓Q 0 ↵✓Q + (1 ↵)✓Q 0 , ✓⇡ 0 ↵✓⇡ + (1 ↵)✓⇡ 0 ; (4)
end
both experiments, the maturity of the option will be one year and the hedging portfolios will be rebalanced on a monthly basis. Table 3 in the appendix provides the descriptive statistics of our underlying hidden stochastic process.
In what follows, we first explain the architectures of our ACRL model. Then, the training procedure of the networks under the dynamic risk measurement is elaborated. Finally, the main numerical results of the paper are presented for pricing and hedging a vanilla, where the precision of our approach will be empirically demonstrated, and a basket option. All codes are available at https://anonymous.4open.science/r/ERP-Dynamic-Expectile-RM-4BEA.

heading: 4.1 ACTOR AND CRITIC NETWORK ARCHITECTURE
text: Our implementation of the ACRL algorithm involves two simple networks presented in Figure 1. Since the underlying assets follow a Brownian motion, the actor and critic networks can define the input state as the logarithm of the cumulative returns of each asset and the time remaining to maturity (i.e. dimension = m + 1). The actor network is composed of three fully connected layers where the number of neurons are considered to be k = 32 in the first two layers and then maps back to the number of assets in the last layer to generate the investment policy accordingly for each asset. The activation functions in our networks are considered to be tanh functions. In the last layer, this implies that the actions will lie in [ 1, 1]m. The critic network only concatenates the m dimensional action information vector after its third layer. In the case of SRM, only the actor network is used.

heading: 4.2 TRAINING PROCEDURE AND LEARNING CURVES
text: Recall that in an SRM setting, overfitting of any DRL algorithm can be controlled by measuring the performance of the trained policy on a validation data set using an empirical estimate of the riskaverse objective as validation score. Unfortunately, this is no longer possible in the case of DRMs since the risk measure relies on conditional risk measurements of the trajectories produced by our policy. In theory, estimates of such conditional measurements could be obtained by training a new critic network using the validation set (while maintaining the policy fixed to the trained one). In practice, this is highly computationally demanding to perform in the training stage and raises a new
issue of how to control overfitting of the validation score estimate. Our solution for this problem is to rely on using static risk measures as validation score, namely a set of static expectiles at risk levels that are larger or equal to the risk level of the DRM. Figures 2 and 3 in the appendix show examples of learning curves for the validation performance of DRM and SRM approaches on vanilla and basket options at a risk level of ⌧ = 90%, with a maturity T = 12. SRM appeared to have a faster rate of convergence than DRM, due to its simpler architecture. Being a time-inconsistent model, SRM must however be retrained whenever the maturity of the option is modified. When comparing convergence rates between vanilla and basket options, we observed similar behavior, which indicates that the training time might not be very sensitive to the number of assets, thus suffering less from the curse of dimensionality. We finally note that both our training and validation sets included 1000 trajectories from the underlying geometric Brownian motion process, implying that the procedure can be applied in settings where only historical data is available.

heading: 4.3 VANILLA OPTION HEDGING AND PRICING
text: In our first set of experiments, we consider pricing and hedging an at-the-money vanilla call option on AAPL. In this setting, it is possible to obtain (approximately) optimal solutions by dynamic programming via discretization of the state space. The initial price of AAPL is set to 78.81 and options with time to maturity ranging from one month to one year are considered. Both DRM and SRM are trained using a one year maturity/horizon.
With the trained DRM and SRM policy networks, we can evaluate the writer and the buyer’s (outof-sample) risk exposure over a pre-specified time horizon so as to calculate the corresponding ERP. We consider the following three metrics for measuring the realized risk under different hedging policy and explain the methods used for calculating the metrics:
• Out-of-sample static expectile risk: Given a trained policy, use the test data to calculate the static expectile risk. This is the metric that should be minimized by the SRM.
• RL based out-of-sample dynamic expectile risk estimation: Given the trained policy, use the test data to only train a critic network using ACRL to produce an estimate of out-of-sample dynamic expectile risk. This is an estimate of the metric minimized by the DRM.
• DP based out-of-sample dynamic expectile risk estimation: Given a trained policy, evaluate the “true” dynamic expectile risk by solving the dynamic programming equations using a high precision discretization of the states, actions, and transitions.4 This serves as the true metric minimized by the DRM.
We note that our RL based estimate of out-of-sample dynamic risk is a novel approach, which tackles the important challenge of policy evaluation in RL with dynamic risk measures.
Table 1 summarizes the evaluations of out-of-sample dynamic risk for DRM policies trained for 1 year maturity at risk level ⌧ = 90% then applied to options of different maturities ranging from 12
4Note that this metric is available neither for the case of basket option nor in a data-driven environment.
months to 1 months. One can observe that the risk of the writer decreases monotonically for options of shorter maturities, whereas the risk of the buyer increases monotonically. This is consistent with the fact that there is less uncertainty for a shorter hedging horizon, which favors the writer’s risk exposure more than the buyer’s when considering an at-the-money option. This also provides the evidence that the DRM policies, albeit only trained based on the longest time to maturity, i.e. one year, can be well applied to hedge options with shorter time to maturity and be used to draw consistent conclusion. Another important observation one can make is that the RL based out-ofsample dynamic risk estimate is generally very close to the DP based estimate across all conditions.
Table 1 also reports the out-of-sample static risk for both SRM policies and DRM policies. The results are interesting and perhaps surprising. First, the DRM policies outperform SRM policies in terms of static risk exposure for short maturities, even though they were trained using a different risk measure. Second, unlike with DRM, we observed at other risk levels (see Figure 6(e) and (f) in Appendix) that the static risk of SRM policies for the seller (resp. buyer) can increase (resp. decrease) when hedging an option with shorter maturity. The possibility that a seller’s policy may actually increase risk when applied to an option with shorter maturity is clearly problematic here as it is inconsistent with the fact that there is less uncertainty (and lower expected value) regarding the payout of such options. Both observed phenomenon are consequences of the fact that SRM violates the time consistency property. We suspect that the possibility that SRM policies may not account properly for risk aversion at some future time point or for other range of option maturities should seriously hinder their use in practice.
Finally, Table 1 reports the equal risk prices calculated based on RL based out-of-sample dynamic risk estimate and based on the discretized DP (referred as True ERP).5 One first confirms that the RL based estimate is of high quality, with a maximum approximation error of 0.01 over all maturities. Moreover, we can see that the prices for the SRM polices are generally higher than the prices for the DRM polices, perhaps due to the fact that it is the writer that benefits most from the improved DRM policy than the buyer, as he is more exposed to tail risks in this transaction. We further refer the reader to section B.2 of the appendix for additional results regarding the performance of SRM and DRM in this vanilla option setting.

heading: 4.4 BASKET OPTION HEDGING AND PRICING
text: In our second set of experiments, we extend the application of ERP pricing framework to the case of basket options where traditionnal DP solution schemes are not computationally tractable. In
5Note that in a real data-driven setting, the ERP could either be estimated using the in-sample trained critic network, or by calculating our RL based estimate using some freshly reserved data to reduce statistical biases.
particular, we consider an at-the-money basket option with the strike price of 753$ on five underlying assets: AAPL, AMZN, FB, JPM, and GOOGL, where the option payoff is determined by the average price of the underlyings. In this section, dynamic risk is only estimated using the RL based estimator defined in Section 4.3, given that exact DP resolution has become intractable.
Table 2 presents the dynamic risk obtained from training the DRM policy for a one year maturity option and applying it on the test data for maturity ranging from 1 to 12 months. Similar to the vanilla option case, the dynamic risk of the writer is monotonically decreasing as we get closer to the maturity of the option, while for the writer the monotonic behavior seems to be slightly perturbed by estimation error. The table also compares the static risk under DRM and SRM. One can first recognize the same monotone convergence to zero of the two sides of the options. However, contrary to the case of the vanilla option, the difference between the static risk performance of DRM and SRM policies are rather similar for all maturity times. It therefore appears that in these experiments with a basket option, both SRM and DRM produce more similar polices. One possible reason could be that the range of “optimal” risk averse investment plans, whether using DRM or SRM, is more limited. Indeed, while for the vanilla option, we observed that the optimal policies generated investments in the range [0, 1] and [-1, 0] for the writer and the buyer respectively, for the basket option we observed wealth allocations that are more concentrated around 0.20 (i.e. the uniform portfolio known for its risk hedging properties) and -0.20 for each of the 5 assets asset respectively. Finally, Table 2 presents the equal risk prices computed based on our RL based out-ofsample dynamic risk estimator. Once again, the higher ERP price for the SRM policy are notable, which again can be attributed to the better performing (in terms of dynamic risk) hedging policy produced by ACRL for the DRM, compared to the policy produced by AORL for the SRM. Further details are presented in section B.3 of the Appendix.

heading: 5 CONCLUSION
text: Motivated by the application of ERP, in this paper we considered solving risk averse MDP problems formulated based on dynamic expectile risk measures, and proposed a novel ACRL algorithm that extends the model-free off-policy deterministic ACRL algorithm to a general finite horizon riskaverse MDP setting. In comparison to existing model-free deep RL methods for solving risk-averse MDP formulated based on dynamic risk measures, our method is more amenable to practical implementation, allowing for tackling real applications such as the ERP problem. Indeed, as a natural risk-averse extension of the popular model-free DDPG, our method can easily accommodate any finite horizon MDP applications solved by DDPG. More in-depth studies of these other applications are left for future work. The extension of our method to an infinite horizon MDP setting is also worth investigating further. Finally, the exploration of our method to accommodate other utilitybased shortfall risk measures should also be of great interest for future study.

Review summaries:
The paper offers a solution to the problem of Equal Risk Pricing (ERP) by framing it as a risk-sensitive MDP formulation. In particular, the authors considers a dynamic risk-measures using expectiles measures as conditional risk-measure for each step. An extension of DDPG is derived by exploiting the properties of the chosen risk-measure and it is tested on two experiments. The first one involves a vanilla option, while the second one feature a more complex basket option.