Deep Reinforcement Learning for Equal Risk Option Pricing and Hedging under Dynamic Expectile Risk Measures
Authors: Saeed Marzban
Erick Delage
Jonathan Li
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep reinforcement learning
risk averse Markov decision processes
expectile risk measures
derivative pricing

Abstract: Recently equal risk pricing, a framework for fair derivative pricing, was extended to consider coherent risk measures. However, all current implementations either employ a static risk measure or are based on traditional dynamic programming solution schemes that are impracticable in realistic settings: when the number of underlying assets is large  or only historical trajectories are available. This paper extends for the first time the deep deterministic policy gradient algorithm to the problem of solving a risk averse Markov decision process that models risk using a time consistent dynamic expectile risk measure. Our numerical experiments, which involve both a simple vanilla option and a more exotic basket option, confirm that the new ACRL algorithm can produce high quality hedging strategies that produce accurate prices in simple settings, and outperform the strategies produced using static risk measures when the risk is evaluated at later points of time.

Under review as a conference paper at ICLR 2022
DEEP REINFORCEMENT LEARNING FOR DYNAMIC EX- PECTILE RISK MEASURES: AN APPLICATION TO EQUAL RISK OPTION PRICING AND HEDGING
Anonymous authors Paper under double-blind review
ABSTRACT
Motivated by the application of equal-risk pricing and hedging of financial derivative, where two operationally meaningful hedging portfolio policies needs be found that minimizes coherent risk measures, we propose in this paper a novel deep reinforcement learning algorithm for solving risk-averse dynamic de- cision making problems. Prior to our work, such hedging problems can either only be solved based on static risk measures, leading to time-inconsistent policies, based on dynamic programming solution schemes that are impracticable in re- alistic settings. Our work extends for the first time the deep deterministic policy gradient algorithm, an off-policy actor-critic reinforcement learning (ACRL) algo- rithm, to solving dynamic problems formulated based on time-consistent dynamic expectile risk measure. Our numerical experiments confirm that the new ACRL algorithm produces high quality solutions equal-risk pricing and hedging prob- lems and that its hedging strategy outperforms the strategy produced using static risk measure when the risk is evaluated later points of time.
INTRODUCTION
This paper considers solving risk-averse dynamic decision making problems arising from applica- tions where risk needs to be evaluated according to risk measures that are coherent, In particular, we draw our motivation from the financial application of equal-risk pricing (ERP) and hedging (Guo & Zhu (017)) ), where two dynamic hedging problems need to be solved, one for the buyer and one for the seller financial derivative (a.k.a option), for determining a fair transaction price that would expose both parties to the same amount of hedging risk. The need to meaningfully model each party's best hedging decision in a financial market, namely that no arbitrage is allowed, and to have meaningful comparison between the two parties' risk exposures, namely that the risks should be measured in the same units, has led to the use of coherent risk measures for capturing both parties' hedging risks in this application (see Marzban al. (020)).
To this date, most solution methods proposed for solving risk-averse dynamic decision making prob- lems under coherent risk measure have either relied on traditional dynamic programming (DP), which suffers from the curse of dimensionality and assumes the knowledge of a stochastic model that precisely captures the dynamics of the decision environment, or on the use of static risk mea- sure, i... that disregards the temporal structure of the random variable (e.g. Marzban et al. 2020 Carbonneau & Godin (020). and Carbonneau Godin (202T) in the case the ERP application).
The latter raises the serious issue that the resulting policy could be time inconsistent, i.e. that the ac- tions prescribed by the policy may considered significantly sub-optimal once the state is visited. In application such as ERP, this issue implies that policies obtained based on static risk measures will not be implemented in practice, raising the need to consider dynamic risk measures.
Focusing on deep reinforcement learning (DRL) methods, while there has been large number of approaches proposed to address risk averse Markov decision processes (MDPs) using coherent risk measures, to the best of our knowledge, all of them, except for two exceptions, consider static risk measure (see Prashanth & Ghavamzadehi (2013); [hoo & Ghavamzadeh (2014); Castro et al.
2019:
[Singh al. (2020) Urpí et al. (202) and therefore suffer from time-inconsistency. The

---

Under review as a conference paper at ICLR 2022
two exceptions consist of Tamar al. 2015) and Huang al. 2021 inforcement learninggg (ACRL) algorithms to deal with general dynamic law-invariant coherent risk measures. Unfortunately, the two algorithms respectively cither assume that it is possible generate samples from a perturbed version of the dynamics, or rely on training three neural networks (namely state distribution reweighting network, transition perturbation network, and Lagrangean penal- isation network) concurrently with the actor and critic networks, Furthermore, only Huang et al. 202T actually implemented their method. This was done on toy tabular problem involving 12 states and 4 actions where it produced questionable performance
who propose actor-critic
In this paper, we develop new model-free ACRL algorithm for solving a time-consistent risk averse MDP under a dynamic expectile risk measure Overall, we may summarize the contribution as follows:
Our ACRL algorithm is the first to naturally extend the popular model-free deep deter- ministic policy gradient algorithm (DDPG) (see Lillicrap al. (2013)) to risk averse setting where a time consistent coherent risk measure is used. Unlike the ACRL proposed in [Huang al. (20TT). which employs five neural networks, our algorithm will only re- quire an actor and a critic network. While our policy network will be trained following a stochastic gradient procedure similar to Silver et al. (2014), we are the first to leverage the elicitability property of expectile risk measures to propose a procedure for training the "risk-to-go" deep Q-network that is also based on stochastic gradient descent.
Our ACRL is the first model-free DRL-based algorithm capable of identifying optimal risk averse option hedging strategies that are time-consistent with respect to a dynamic coherent risk measure, and of computing their associated equal risk prices. A side benefit of time- consistency will be that after training for an option with given maturity, one obtains equal risk prices and hedging strategies for any other shorter maturities. While our algorithm certainly has a broader set of applications, we believe that ERP constitutes an original and fertile application in which to develop and test new risk averse DRL methods.
We evaluate the training efficiency and the quality of solution, in terms of quality of option hedging strategies and of estimated equal risk prices, obtained using our ACRL algorithm a synthetic multi-asset geometric Brownian motion market model. These experiments constitute the first real application of a risk averse DRL algorithm that employs a dynamic coherent risk measure.
The rest of this paper is organized as follows. Section introduces equal risk pricing and its asso- ciated DP equations. Section R proposes the new ACRL, algorithm for general finite horizon risk averse MDP with dynamic expectile measures. Finally, Section presents and discusses our numer- ical experiments. We note that reader only interested in the ACRL algorithm can skip right ahead to Section k
APPLICATION: EQUAL RISK PRICING AND HEDGING UNDER DYNAMIC EXPECTILE RISK MEASURES
As described in Marzban et al. 2020), the problem of ERP can be formalized as follows. Consider frictionless market, i.e. transaction cost, tax, etc, that contains risky assets, and risk-free bank account with zero interest rate. Let S, Ω Rn denote the values of the risky assets adapted filtered probability space (2, F (F.) IP), i.e. each S, is F, measurable. It is assumed that S, is a locally bounded real-valued semi-martingale process and that the set of equivalent local martingale measures non-empty (i.e. no arbitrage opportunity). The set of all admissible self- financing hedging strategies with the initial capital Po E RR is shown by X(oo):
1-1
X'(po)
x : + 1 R" BEE X, = Po + Σ AT == T
#=
At the time of writing this paper, the risk averse implementation of this algorithm reported in Huang et al.
2021) is unable recommend an optimal policy deterministic setting, while the risk neutral implementa-
tion produces policies that are outperformed by risk averse ones in a stochastic setting.
?Our ACRL algorithm exploits the elicitabilty property of expectile risk measures, which is the only elic- itable coherent risk measure.
2

---

In particular, based on Proposition 3.1 and the examples presented in section 3.3 of Marzban et al. 2020 together with the fact that both p" and de are dynamic recursive law invariant risk measures, conclude that the ERP can be calculated using two sets of Under review as a conference paper at ICLR 2022
where AS == S St, the hedging strategy & IR' is vector of random variables adapted to the filtration and captures the number of shares of each risky asset held in the portfolio during the period [t, ++ 1), and X, is the accumulated wealth.
Let F((S))=1) denote the payoff of derivative. Throughout this paper, we assume F F({S))=))) admits the formulation of F(Sr, Yr) where Y, is an auxiliary dinensinnns stochastic process that is F.-meauurabl.. This class of payoff functions is common in the literature, (see for example Bertsimas al. (200T) and Marzban et al. (2020). The problem of ERP is defined based on the
following two hedging problems that seek to minimize the risk of hedging strategies, one is for the
writer and the other for the buyer of the derivative:
(Writer)
(po)
inf
(F Yr) -Xr)
(1)
XEX(P))
(Buyer)
e' (po))=
inf xEX((po)
(- Yr) XT),
(2)
where pd and d are two risk measures that capture respectively the writer and the buyer's risk aversion. In words, equation describes a writer that is receiving Pa as the initial payment and implements an optimal hedging strategy for the liability F(S,, Y)). On the other hand, in 2 the
buyer is assumed to borrow po in order to pay for the option and then to manage portfolio that
will minimize the risks associated to his final wealth. With equations ① and 2 ERP defines a
fair price ph as the value of an initial capital that leads to the same risk exposure to both parties, i.c.
e" (P6) = (pp)).
Markovian assumption allows
dynamic programming equations.
Assumption 2.1. [Markov property] There exists sufficient statistic process 1b, adapted to
such that {(St. Y, vt) a Markov process relative to the filtration F. Namely, P((Se+s, Y i-ss w/++)) E A(Fi) P(((stss Yots, thets) ASs,, Y,, 1h2) for all for all 0, and all sets A.
Specifically, on the writer side, we can define Vi (Sr, Yr, (T) = F(Sr, Yr), and recursively (S,, Y,, vh) == inf p( E AS V (S, + AS,+1, Y, + AY 11+1) Y,, ve)
E:
where p(-S,, Y,, v/r) is a law invariant risk measure that uses P(-|S,, Y,, th). This leads to considering eº (0)
V0"h (So. 4o). On the other hand, for the buyer we similarly define:
(+ Yr, UT) = - F F(T,, YT) and
(S, Y., v1) == inf p( - AS + Pb #1 (S, + ASH11 Y, AY 11t1)) Y,, Y,,))),
with e0(0)= Vi Yo, vo). The following lemma summarizes how DP can be used to compute the ERP.
Lemma 2.1 Marzban et al. (020)). ) Under Assumption 2.7 the ERP that employs dynamic expec- tile risk measure can be computed as: ph (Vo Yo. vo) (So, Yo, 22))
In what follows, we will further assume that the risk measure is a dynamic expectile risk measure. Definition 2.1. A dynamic expectile risk measure takes the form: p(X) := : (X))) where each p(-) is an expectile risk measure that employs the conditional distribution based on Ft. Namely, pe (X(+1)) == arg min, TE [[q xxx+1)2 + - (1- [(q [F. where X is a random liability measureable on F.+1-
Like conditional value risk, the expectile measure (see Bellini & Bignozzi 2015)) covers the range of risk attitudes from risk neutrality, when 7 = 1/2, to worst-case risk, when 7 - 1.
A NOVEL ACTOR-CRITIC ALGORITHM FOR RISK AVERSE MDP UNDER A DYNAMIC EXPECTILE RISK MEASURE
With the dynamic programming equations in hand, it now becomes apparent that each option hedg- ing problem in ERP can be formulated as finite horizon Markov Decision Process (MDP) described
3

---

Under review as a conference paper at ICLR 2022
with AS, P). In this regard, the agent (i.e. the writer or buyer) interacts with a stochastic vironment by taking an action O = & E [-1,1]m after observing the state 8t E S, which in- cludes S,, Y,, and 4/, Note that to simplify exposition, in this section drop the reference to the specifie identity (i.e. w or b) of the agent in our notation. The action taken at each time i re- sults in the immediate stochastic reward that takes the shape of the immediate hedging portfolio return, i.c Te( Ct, St+1) == & AS when t < I and otherwise of the option liability/payout rr (ST, aT, ST+1) == F'ST, YT) 2 Kagent-writer)), which insensitive to ST+1. Fi- nally, the Markovian exogeneous dynamics described in Assumption 2.1 are modeled using P as P(s441 (z) P(S Y441, 1t+1 S.,, Y. 1h)). Overall, each of the two dynamic derivative hedging problems presented in Section reduce to version of the following general risk averse reinforcement learning problem:
e(0) Vo Yo, vo) - min Q7 (5o, mo(()))
where 50 == (So: You vD) is the initial state in which the option is priced while Q7 (st, at) == p(-re(st, Gt, St+1) + Qu (8t+1, Te+1 (8++1)))))) (sT, (1))s,rr))))) == -rr(sr, ar, ST), and where an expectile risk measure, ie. p(X) == arg ming E [ - 1)} (1- - x) Equipped with these definitions, we can now motivate our proposed extension of the model-free off-policy deterministic ACRL algorithm the general finite horizon risk-averse MDP setting. In doing so, we start with a proposition (see Appendix A for a proof) that will provide the motivation for stochastic gradient scheme to optimize the actor network, while the optimization of the critic network will follow from the elicitability property of the expectile risk measure.
Proposition 3.1. Let H be an arbitrary reference policy and an arbitrary distribution over the initial state, such that there is strictly positive probability of reaching all of S for all Z For any πT" that satisfies
n² E arg min E E-((0...T)..o00 Tos (Qp [s(:))
(3)
9141~ ((le-Pe)),)e(ee))
where i is uniformly drawn, we necessarily have that xt arggin min Q8 (5a, ko(5))) hence e(0) Q5" (5o, (5a)).
In the context of a deep reinforcement learning approach, we can employ procedure based on off-
policy deterministic policy gradient (Silver et al. 2014) to optimize i Specifically, given a policy
network x², we wish to optimize:
min E i-
1Q (sis (s:)).
P(
using a stochastic gradient algorithm. In doing so, we rely on the fact that:
V&E E=(0....TTI) s11 (())
(Q (s (s;))]
E
in T T-13 P (((ee))
VoQF" (s),0) a=))(ss)
+ Vali (o (x))((0)
E -.. 8141 P (el.
VaQi (s7 a VoT (=))(())
Note that in the above equation, we have dropped the term that depends on QoQ;" as is commonly done in off- off-policy deterministic gradient methods and usually motivated by result of Degris et al. (2012). who argue that this approximation preserves the set of local optima in a risk neutral setting, L.e. p()) E[.]. While we do consider as an important subject of future research to extend this motivation to more general risk measures, our numerical experiments (see Section 4.3) will confirm empirically that the quality of this approximation permits the identification of nearly optimal hedging policies.
In our option hedging problem, given that 8, is entirely exogenous, the distribution of se+1 is unaffected by म. which can therefore chosen arhitrarily. Moreover, f can put all the 8p-
4

---

Under review as a conference paper at ICLR 2022
Given that we do not have access to exact expression for QF" (s;, a). this operator needs to be estimated directly from the training data. Exploiting the fact that is a utility-based shortfall risk measure, we get that:
(st: E arg min P(
Qi ($441 (st, de, ( (c+1())))))
where {(v) == (r 03 - (1 T) S 0})y' is the score function associated to the 7-expectile risk measure, As explained in Theorem 3.2 of Shen et al. (0114) in a tabular MDP environment one can apply the following stochastic gradient step:
Qe(se, at) -- (st,,, az) aQl(Qi as) + (s ar 8141) - ( (++1())11));
where Ol(y) 2(T max(0, y) (1 - 7) max(0,--)) the derivative of fly), within a properly designed Q-learning algorithm and have the guarantee that (se, 0z) will almost surely converge to QT (sr, at) for all St, and dr.
In the non-tabular setting, replace (s), with two estimators: i.c. the "main" net- work (s;; az (0) for the immediate conditional risk and the "target" network (Tt, ag 109") for the next period's conditional risk. The procedure consists in iterating between a step that attempts to make the main network Q7 (st, a,,,)) good estimator of p(-r(se; ae, St41) + QT+1 (5:+1, a++1 ()))) and a step that replaces the target network (T( ar109') with a network more similar to the main one Q (st, a, aj) The former is achieved, similarly as with the policy
network, by searching for the optimal 0 according to:
min @
& (-10...1-11 (((e,))))))
[e(2, (sp, (((:))09)+tr1(s)) (z(;;) (441)) -0t (8141 T(+1 Q4+1 (8141) )01
which suggests stochastic gradient update of the form 09 + 00 aA, where A is
(())))))))) +r) [7 -7 T (51+1)10 Vo (0))))
These two types of updates are integrated in our proposed expectile-based actor-critic deep RL (a.k.a first version, Algorithm | designed for simulation-basee environment. One may note that in each episode, the reference policy म, is updated to be perturbed version of the main policy network in order focus the accuracy of the main critic network's value and derivatives on actions that are more likely to be produced by the main policy network. We also choose to update the target networks using convex combinations operations as is done in Lillicrap al. (2015) in order to improve stability of learning. A second more general version of ACRL, which mimics the original DDPG, by generating minibatches using replay buffer can also be found Appendix A.2
ACRL) algorithm.
We finally note that in our problem, P(8141/81, ax)
P(81+119;, at)
P(S,41, Y.+1, |S,, Y,, V2). meaning that the action is not affecting the distribution of state in the next period. This is a direct consequence of using translation invariant risk measure, which eliminates the need to keep track of the accumulated wealth in the set of state variables as explained in Marzban et al. (2020) and allows the reward function to provide regarding the quality of implemented actions. In the context of our deep reinforcement learning approach, observed that convergence speed is significantly improved in training due to this property (see Figure 4 in Appendix).
immediate signal
4 EXPERIMENTAL RESULTS
In this section we provide two different sets of experiments that are run over one vanilla and one basket option. We will compare both algorithmic efficiency and quality, in terms of pricing and hedging strategies, of the dynamic risk model (DRM), which employs dynamic expectile risk measure and is solved using our new ACRL algorithm, and the static risk model (SRM), which employs a static expectile measure and is solved using an AORL algorithm similar to Carbonneau & Godin (202T). All experiments are done using simulated price processes of five risky assets: AAPL, AMZN, FB, JPM, and GOOGL. The price paths simulated using correlated Brownian motions considering the empirical mean, variance, and the correlation matrix of five reference stocks (AAPL, AMZN, FB, KPM, and GOOGL. over the period that spans from January 2019 January 2021. In
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: The actor-critic RI. algorithm for the dynamic recursive expectile option hedging problem (ACRL)
Randomly initialize the main actor and critic networks' parameters 6* and 82;
Initialize the target actor, 6r <- 8*, and critic, 0² + -2, networks;
for j = 1 # Episodes do
Randomly select E {0, 1,..,, 1);
Sample minibatch of N triplets {(6 aj ((41)) from P(-|a,, (s:))) where (x(5t) == ny (5 (O(
Set the realized losses vi : :- (s (sts ag 4/42) Q 1041 7x
met (0 100²;
Update the main critic network:
0 - 0
Σ a( oe Qe(),,
Update the main actor network:
1 - 0t
(aQ((s),((09) (ommtett))))) VorTe
Update the target networks:
0 - a09 + a1-a
Ør' + at +
(1- - a))""
(4)
end
both experiments, the maturity of the option will be one year and the hedging portfolios will be rebalanced on monthly basis. Table B the appendix provides the descriptive statistics of our underlying hidden stochastic process.
In what follows, we first explain the architectures of our ACRL model. Then, the training pro- cedure of the networks under the dynamic risk measurement is elaborated. Finally, the main nu- merical results of the paper are presented for pricing and hedging a vanilla, where the precision of our approach will be empirically demonstrated, and basket option. All codes are available at hotp::/anoonymous.opp...... wwtp://ccceeeeeeeeeeeaieaeeeeeee RM--4AAA
4.1 ACTOR AND CRITIC NETWORK ARCHITECTURE
Our implementation of the ACRL algorithm involves two simple networks presented in Figure Il Since the underlying assets follow a Brownian motion, the actor and critic networks can define the input state the logarithm of the cumulative returns of each asset and the time remaining to maturity (i.e. dimension = m + 1). The actor network is composed of three fully connected layers where the number of neurons are considered be k 32 in the first two layers and then maps back to the number of assets in the last layer to generate the investment policy accordingly for each asset. The activation functions in our networks are considered to be tanh functions. In the last layer, this implies that the actions will lie in [-1, 1 1]mm. The critic network only concatenates the m dimensional action information vector after its third layer. In the case of SRM, only the actor network is used.
4.2 TRAINING PROCEDURE AND LEARNING CURVES
Recall that in an SRM setting, overfitting of any DRL algorithm can be controlled by measuring the performance of the trained policy on validation data set using an empirical estimate of the risk- averse objective validation score. Unfortunately, this is no longer possible in the case of DRMs since the risk measure relies on conditional risk measurements the trajectories produced by our policy. In theory. estimates of such conditional measurements could be obtained by training critic network using the validation set (while maintaining the policy fixed to the trained one). In practice, this is highly computationally demanding to perform in the training stage and raises new

---

Under review as a conference paper at ICLR 2022
Actor network
Tuby cornecee lim-3ink
Fullyy cormecte goomt
mm
hully
Tanh activeion function
Critic network
mm)
Fuly
Tuhyy comnectdd
Fulyy comeendd Eimsoo
(uyy commrrtd gel
Spor (mm-))
Tully PePeesomeett Booi
Figure 1: The architecture of the actor and critic networks in ACRL algorithm.
estimate. Our solution for this problem is to rely on using static risk measures as validation score, namely set of static expectiles at risk levels that are larger equal the risk level of the DRM. Figures
issue of how control overfitting of the validation
2 and and33in the appendix show examples of learning curves for the validation performance of DRM and SRM approaches on vanilla and basket options at risk level of 90%, with a maturity T 12. SRM appeared to have faster rate of convergence than DRM, due to its simpler architecture. Being lime-inconsisten model, SRM must however be retrained whenever the maturity of the option is modified. When comparing convergence rates between vanilla and basket options, we observed similar behavior, which indicates that the training time might not be very sensitive the number of assets, thus suffering less from the curse of dimensionality. We finally note that both training and validation sets included 1000 trajectories from the underlying geometric Brownian motion process, implying that the procedure can be applied in settings where only historical data is available.
4.3 VANILLA OPTION HEDGING AND PRICING
In our first set of experiments, we consider pricing and hedging an e-the--mneyy vanilla call option on AAPL. In this setting, it is possible to obtain (approximately) optimal solutions by dynamic programming via discretization of the state space. The initial price of AAPL is set to 78.81 and options with time to maturity ranging from one month to one year are considered. Both DRM and SRM are trained using a one year maturity/horizon.
With the trained DRM and SRM policy networks, we can evaluate the writer and the buyer's (out- of-sample) risk exposure over a pre-specified time horizon so as to calculate the corresponding ERP. We consider the following three metrics for measuring the realized risk under different hedging policy and explain the methods used for calculating the metrics:
Out-of-sampp static expectile risk: Given a trained policy, use the test data to calculate the static expectile risk. This is the metric that should be minimized by the SRM.
RL based out-of-sane of--ple dynamic expectile risk estimation: Given the trained policy, the test data to only train a critic network using ACRL to produce an estimate of out-of-sample dynamic expectile risk. This is an estimate of the metric minimized by the DRM.
DP based the "true" dynamic expectile risk by solving the dynamic programming equations using high precision discretization of the states, actions, and transitions This serves as the true metric minimized by the DRM.
dynamic expectile risk estimation: Given trained policy, evaluate
We note that our RL based estimate of out-of-sample dynamic risk is novel approach, which tackles the important challenge of policy evaluation in RL with dynamic risk measures.
Table I summarizes the evaluations of -ft-ole dynamic risk for DRM policies trained for year maturity at risk level = 90% then applied to options of different maturities ranging from 12
Note that this metric is available neither for the of basket option nor data-driven environment.

---

Under review as a conference paper at ICLR 2022
Table 1: The out-of-sample dynamic and static -x%eecpe risk imposed to the two sides of vanilla at-the-money call options over AAPL.
Time to maturity
Policy
Est.
12
10
8
4
3
2
Dynamic 90%-expectic risk
RL
0.77
0.73
0.69
0.65
0.62
0.45
0.38
0.29
0.23
Writer's DRM
DP
0.75
0.71
0.68
0.65
0.61
0.43
0.38
0.31
0.23
RL
-0.22
-0.21
-0.20
0.19
-0.18
-0.11
-0.09
-0.07
-0.05
Buyer's DRM
DP
-0.23
-0.22
-0.21
-0.20
-0.18
-0.12
-0.11
-0.08
-0.06
Static 90%
expectile risk
Writer's SRM
ED
0.55
0.54
0.54
0.53
0.53
0.48
0.46
0.41
0.31
Writer's DRM
ED
0.56
0.54
0.52
0.50
0.47
0.36
0.33
0.29
0.24
Buyer's SRM
ED
-0.35
-0.33
-0.30
-0.27
-0.23
-0.09
-0.07
-0.07
-0.06
Buyer's SRM
ED
-0.36
-0.34
-0.32
-0.30
-0.28
-0.18
-0.14
-0.11
-0.06
Equal risk prices with DRM
True ERP
0.49
0.47
0.45
0,42
0.40
0.28
0.24
0.19
0.14
DRM
RL
0.50
0.47
0.45
0.42
0.40
0.28
0.24
0.18
0.14
SRM
RL
0.49
0.46
0.44
0.43
0.40
0.30
0.27
0.24
0.22
Estimation (Est.) is either made based reinforcement learning (RL), discretized dynamic programming (DP), or the empirical distribution (ED).
months to months. One can observe that the risk of the writer decreases monotonically for options of shorter maturities, whereas the risk of the buyer increases monotonically. This is consistent with the fact that there is less uncertainty for shorter hedging horizon, which favors the writer's risk exposure more than the buyer's when considering an at-the-money option. This also provides the evidence that the DRM policies, albeit only trained based on the longest time maturity, i.e. one year, can be well applied to hedge options with shorter time to maturity and be used to draw consistent conclusion. Another important observation one can make is that the RL based out-of- sample dynamic risk estimate is generally very close to the DP based estimate across all conditions.
Table also reports the out-of-sample static risk for both SRM policies and DRM policies. The results are interesting and perhaps surprising. First, the DRM policies outperform SRM policies in terms of static risk exposure for short maturities, even though they were trained using different risk measure. Second, unlike with DRM, we observed at other risk levels (see Figure (e) and (f) in Appendix) that the static risk of SRM policies for the seller (resp. buyer) can increase (resp. decrease) when hedging actually increase risk when applied to an option with shorter maturity is clearly problematic here as it is inconsistent with the fact that there is less uncertainty (and lower expected value) regarding the payout of such options. Both observed phenomenon are consequences of the fact that SRM violates the time consistency property. We suspect that the possibility that SRM policies may account properly for risk aversion at some future time point or for other range of option maturities should seriously hinder their use in practice.
option with shorter maturity. The possibility that seller's 's policy may
Finally, Table D reports the equal risk prices calculated based on RL based out-of-sample dynamic risk estimate and based on the discretized DP (referred as True ERP B One first confirms that the RL based estimate is of high quality, with maximum approximation error of 0.01 over all maturities. Moreover, we can see that the prices for the SRM polices are generally higher than the prices for the DRM polices, perhaps due to the fact that it is the writer that benefits most from the improved DRM policy than the buyer, as he is more exposed to tail risks this transaction. We further refer the reader to section B.2 of the appendix for additional results regarding the performance of SRM and DRM in this vamilla option setting.
4.4 BASKET OPTION HEDGING AND PRICING
In our second set of experiments, we extend the application of ERP pricing framework to the case of basket options where traditionnal DP solution schemes are not computationally tractable. In
NNoe that in real data-driven setting, the ERP could either be estimated using the in-sample trained critic network, by calculating RI. hased estimate using some freshly reserved data to reduce statistical hiases.
8

---

Under review as a conference paper at ICLR 2022
particular, we consider -oethe--moeyy basket option with the strike price of 7538 on five underlying assets: AAPL, AMZN, FB, JPM, and GOOGL, where the option payoff is determined by the average price of the underlyings, In this section, dynamic risk is only estimated using the RL based estimator defined in Section given that exact DP resolution has become intractable.
Table 2: The out-of-sample dynamic and static 90%- expectile risk imposed to the two sides of basket mone call options. Associated ERPs under the DRM are also compared.
Time maturity
Policy
Est.
12
11
10
9.
8
4
3
2
Dynamic 90% expectile risk
Writer's DRM
RL
3,92
3.62
3.38
3.15
2.95
2.00
1.70
1.39
1.10
Buyer's DRM
RL
-0.48
-0.49
-0.51
-0.52
-0.50
-0.47
-0.37
-0.33
-0.29
Static 90% expectile risk
Writer's s SRM
ED
2.43
2.36
2.28
2.16
2.08
1.61
1.45
1.26
0.94
Writer's DRM
ED
2.38
2.28
2.18
2.06
1.96
1.51
1.39
1.20
0.92
Buyer's SRM
ED
-1.31
-1.24
-1.15
-1.01
-0.94
-0.56
-0.48
-0.36
-0.22
Buyer's SRM
-1.39
-1.32
-1.24
-1.13
-1.07
-0.66
-0.56
-0.40
-0.23
Equal risk prices with DRM
DRM
RL
2,20
2.06
1.95
1.84
1.73
1.24
1.04
0.86
0.70
SRM
RL
2.23
2.10
2.01
1.91
1.79
1.21
1.03
0.92
0.82
Estimation (Est.) is either made based on reinforcement learning (RL), the empirical distribution (ED).
Table presents the dynamic risk obtained from training the DRM policy for one year maturity option and applying it the test data for maturity ranging from I to 12 months. Similar to the vanilla option case, the dynamic risk of the writer is monotonically decreasing as we get closer to the maturity of the option, while for the writer the monotonic behavior seems to be slightly perturbed by estimation error. The table also compares the static risk under DRM and SRM. One can first recognize the same monotone convergence to zero of the two sides of the options. However, contrary to the case of the vanilla option, the difference between the static risk performance of DRM and SRM policies are rather similar for all maturity times. It therefore appears that in these experiments with a basket option, both SRM and DRM produce more similar polices. One possible reason could be that the range of "optimal" risk averse investment plans, whether using DRM or SRM, is more limited. Indeed, while for the vanilla option, we observed that the optimal policies generated investments in the range [0, 1] and [-1, 0] for the writer and the buyer respectively, for the basket option we observed wealth allocations that are more concentrated around 0.20 (i.e. the uniform portfolio known for its risk hedging properties) and -0.20 for each of the assets asset respectively. Finally, pabsenpresents the equal risk prices computed based on our RL based out-of- sample dynamic risk estimator. Once again, the higher ERP price for the SRM policy are notable, which again can be attributed to the better performing (in terms of dynamic risk) hedging policy produced by ACRL for the DRM, compared to the policy produced by AORL for the SRM. Further details are presented in section B.3 of the Appendix.
5 CONCLUSION
Motivated by the application of ERP, in this paper we considered solving risk averse MDP problems formulated based on dynamic expectile risk measures, and proposed a novel ACRL algorithm that extends the model-free off- off-policy deterministic ACRL algorithm to a general finite horizon risk- averse MDP setting. In comparison to existing model-free deep RL methods for solving risk-averse MDP formulated based on dynamic risk measures, our method is more amenable to practical im- plementation, allowing for tackling real applications such as the ERP problem. Indeed, as a natural risk-averse extension of the popular model-free DDPG, our method can easily accommodate any finite horizon MDP applications solved by DDPG. More in-depth studies of these other applications are left for future work. The extension of our method worth investigating further. Finally, the exploration of our method to accommodate other utility- based shortfall risk measures should also be of great interest for future study.
infinite horizon MDP setting is also

---

Under review as a conference paper at ICLR 2022
REFERENCES
Fabio Bellini and Valeria Bignozzi. On elicitable risk measures. Quantitative Finance, 15(5):725- 733, 2015.
Dimitris Bertsimas, Leonid Kogan, and Andrew Lo. Hedging derivative securities and incomplete markets: an 6-arbitrage approach. Operations research, 49(3):372-397, 2001.
Alexandre Carbonneau and Frédéric Godin. Equal risk pricing of derivatives with deep hedging. Quantitative Finance, pp. 1-16, 2020.
Alexandre Carbonneau and Frédéric Godin. Deep equal risk pricing of financial derivatives with multiple hedging instruments. arXiv preprint arXiv:2102.12694 2021.
Dotan Di Castro, Oren, and Shie Mannor. Practical risk measures in reinforcement learning. ArXiv, abs/1908.08379) 2019.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in MDPs. Ad- vances neural infor- mation processing systems,
2014.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp- 179-186, Madison, WI, USA, 2012. Omnipress.
Ivan Guo and Song-Ping Zhu. Equal risk pricing under convex trading constraints. Journal of Economic Dynamics and Control, 76:136-151, 2017.
Audrey Huang, Liu Leqi, Zachary C. Lipton, and Kamyar Azizzadenesheli. On the convergence and optimality of policy gradient for markov coherent risk, 2021.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.000 02997,, 2015.
Saeed Marzban, Erick Delage, and Jonathan Yumeng Li. Equal risk pricing and hedging of financial derivatives with convex risk measures. arXiv preprint arXiv:2002.02876 2020.
L.A. Prashanth and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive MDPs. Advances in neural infor- mation processing systems, abs/ 106.3449:252-22222 2013.
Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic programming. Operations Research Letlers, 45(4):377-381, 2017..
Yun Shen, Michael Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural Computation, (6(7):1298-1322) 2014.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395. PMLR, 2014.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk distribu- tional reinforcement learning. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Proceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pp. 958-968, 2020.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for co- herent risk measures. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learn- ing. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
10


Reviews:
[{"review_id": "jSk9-YJoQlm", "reviewer": "Reviewer_wndf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The target problem is interesting and maybe potentially useful.\n2. The proposed algorithm seems to be rigorous, the theoretical guarantees are appreciated.\n\nWeaknesses:\n1. A first question I would hope the authors to clarify is that: from the organization and presentation of this paper, this paper is more like an application paper of DRL to pricing and hedging tasks. But for an AI conference, the readers are expecting to see some general form behind, so that this solution can also benefit related studies. If it focuses on one particular application, then an ML&finance conference or journal would be a better choice for this work.\n2. From Alg. 1, one key change from DDPG to the proposed ACRL is taking an average of N, which would effectively reduce variance.  Then, I believe it would be more valuable to discuss the variance.\n3. As for the experiment results in Section 4.3 and Section 4.4, the major issue is that the rationale of performance metrics are not clearly presented, so it is not easy to evaluate the results.\n\nAfter rebuttal:\n1. The first concern is addressed.\n2. The second one still remains, so there is a question about novelty.\n3. The third one still remained, which is still not very easy to evaluate its values to practioners.", "summary_of_the_paper": "This paper targets a fundamentally important issue: risk-aversion in pricing and hedging. The authors claims the first algorithm to identify optimal risk averse for option hedging strategies, which are time-consistent with respect to a dynamic convex riskmeasure, etc., those are impressive.", "summary_of_the_review": "Interesting application, but need the authors to justify some ambiguity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mb-aB5mEpx", "reviewer": "Reviewer_VoGN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is novel since, as highlighted by the authors, they provide the first DDPG-like algorithm optimizing a dynamic risk-measure. Differently from previous approaches, their work allow to produce time-consistent hedging policies.\nOne problem with this article consists with its accessibility to a wide audience: the financial introduction may result difficult to understand to a reader who is unfamiliar to option hedging. The paper has also some other passages which may be difficult to follow, for instance in the derivation of the DDPG-like algorithm and in the experimental analysis. \nIn particular I would like to ask the authors for two specific issues.\n1) A stochastic gradient step is used to improve expectile estimators taken from (Shen et al., 2014): can the authors elaborate more on why the suggested one is a good estimate?\n2) In section 4.3 you say that \"First, unlike with DRM, the static risk of SRM policies for the seller (resp. buyer) can increase (resp. decrease) when hedging an option with shorter maturity. The possibility that a seller’s policy may actually increase risk when applied to an option with shorter maturity is clearly problematic here\", however, it is not clear to me how this is apparent from the results of Table 1.\n\nMoreover, it is not clear to me wheter the algorithm is applicable only to option hedging or it always applicable in MDP where the objective is the dynamic risk-measure in exam.\nThe article also lacks a conclusion, which the author should surely add to their final version in order to summarize their work and to indicate some directions for future research.", "summary_of_the_paper": "The paper offers a solution to the problem of Equal Risk Pricing (ERP) by framing it as a risk-sensitive MDP formulation. In particular, the authors considers a dynamic risk-measures using expectiles measures as conditional risk-measure for each step. An extension of DDPG is derived by exploiting the properties of the chosen risk-measure and it is tested on two experiments. The first one involves a vanilla option, while the second one feature a more complex basket option.", "summary_of_the_review": "The paper offer a novel perspective on Equal Risk Pricing for option hedging using reinforcement learnin and dynamic risk-measures. The contribution is mainly an algorithmical and experimental one, but the proposed approach seems to be sound and effective. I would like to propose a borderline accept to this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AXOW7Oeyv3", "reviewer": "Reviewer_2QwB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written but certain parts, especially finance-related concepts could be further elaborated for the general AI/ML audience. \n\nThe paper addresses the important issue of risk-sensitive RL. However, the focus here is very narrow, on applications in option hedging and equal-risk pricing. Usually, when it comes to methods targeting a specific application, the paper must at least achieve one of the following two:\n1) Can the proposed approach be easily applied or generalized to other problems or application domains ? \nThis is not obvious. The dynamic risk measures have very good theoretical properties -- everything is Markov, and therefore very convenient for dynamic-programming-based solutions. The proposed DDPG extensions are therefore quite straightforward compared to the case of static risk measures. Furthermore, as pointed out by the authors, the state-transition dynamic depends only on external uncertainties thereby making the problem even easier than solving general MDPs.\n\n2) If the above condition is not satisfied, then the target application must itself be significant (e.g. solve chess).\nHere, the problem is option hedging and equal-risk pricing. I can accept these as significant but the evidence presented in the paper is insufficient. \n - Firstly, all experiments are only done in simulated environments based on the fitted Brownian-motion model (for both training and testing). This means that RL can be trained using unlimited data and there is no train-test mismatch. This does not test the RL ability to generalize beyond the training examples. We have many years of stock prices available, so why not test on real data?\n - The comparison between policies obtained via optimizing static/dynamic risks seems confusing. It seems that using methods that optimize the static risk directly, we would obtain a better test-set static risk. Similarly, using methods optimizing the dynamic risk directly, we would obtain better test-set dynamic risk. What is wrong here?\n\n", "summary_of_the_paper": "The authors propose a risk-sensitive reinforcement learning algorithm based on DDPG for the problem of risk-averse option hedging and equal risk pricing based on dynamic expectiles. The proposed algorithm is empirically tested on a simulated option hedging environment.", "summary_of_the_review": "Based on my concerns above, I cannot recommend acceptance at this stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a risk-sensitive actor critic reinforcement learning (RL) method that optimizes the policy with respect to a dynamic (iterated) expectile risk measure.  The expectile risk measure has the elicitability property and can be expressed as the minimizer of an expected scoring function, which is exploited in critic update.  The proposed approach is applied to option pricing and hedging.

A main point of discussion was the applicability and effectiveness of the proposed method beyond particular financial problems.  The original submission was indeed specialized to particular financial tasks.  The authors have rewritten the paper in a way that it claims to propose a risk-sensitive RL method for the general MDP with finite horizon.  This however leaves the question regarding the advantages of the proposed approach over existing methods of risk-sensitive RL, including those that work with non-coherent (dynamic) risk measures (since coherence is often not needed in domains outside finance).
Decision: Reject