Mean-Variance Efficient Reinforcement Learning by Expected Quadratic Utility Maximization
Authors: Masahiro Kato
Kei Nakagawa
Kenshi Abe
Tetsuro Morimura
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
Mean-variance tradeoff

Abstract: Risk management is critical in decision making, and mean-variance (MV) trade-off is one of the most common criteria. However, in reinforcement learning (RL) for sequential decision making under uncertainty, most of the existing methods for MV control suffer from computational difficulties owing to calculating the gradient of the variance term. In this paper, in contrast to strict MV control, we consider learning MV efficient policies that achieve Pareto efficiency regarding MV trade-off. To achieve this purpose, we train an agent to maximize the expected quadratic utility function, a common objective of risk management in finance and economics. We call our approach RL based on expected quadratic utility maximization (EQUMRL). The EQUMRL does not suffer from the computational difficulties because it does not include gradient estimation of the variance. We confirm that the maximizer of the objective in the EQUMRL directly corresponds to an MV efficient policy under a certain condition. We conduct experiments with benchmark settings to demonstrate the effectiveness of the EQUMRL.

Under review as a conference paper at ICLR 2022
MEAN- VARIANCE EFFICIENT REINFORCEMENT LEARNING BY EXPECTED QUADRATIC UTILITY MAXIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In reinforcement learning (RL) for sequential decision making under uncertainty, existing methods proposed for considering mean-variance (MV) trade-off suffer from computational difficulties in computation of the gradient of the variance term. this paper, we aim to obtain MV-efficient policies that achieve Pareto efficiency regarding MV trade-off. To achieve this purpose, we train an agent to maximize the expected quadratic utility function, in which the maximizer corresponds to the Pareto efficient policy. Our approach does not suffer from the computational difficulties because does not include gradient estimation of the variance. In experiments, confirm the effectiveness of our proposed methods.
INTRODUCTION
Reinforcement learning (RL) trains intelligent agents to solve sequential decision-making problems (Puterman] 1994. Sutton & Barto 1998). While typical objective is to maximize the expected
cumulative reward, risk-aware RL has recently attracted much attention in real-world applications, such as finance and robotics (Geibel & Wysotzki 2005 Garcia & Fernández|22111
Various criteria
have been proposed capture a risk, such as Value at Risk (Chow & Ghavamzadehi 2014| Chow et al. [201]] and variance (Markowitz, 1952; Luenberger et 1997] Tamar et al. [2012] Prashanth Ghavamzadeh] 2013). Among them, we consider the mean- variance RL (MVRL) methods that attempt train an agent while controlling the mean-variance (MV) trade-off (Tamar et al. 2012
Prashanth & Ghavamzadeh, 2013) 2016
Xie et al. 2018 Bisi al. 2020
Zhang et al., 2021b).
Xie et al.. 2018), typically maximize the expected cumulative reward while keeping the variance of the cumu- lative reward certain level equivalently, minimize the variance while keeping the expected cumulative reward at a certain level. These MVRL methods simultaneously estimate the expected reward variance while training an agent and solve the constrained optimization problem relaxed by penalized methods. These studies have reported that RL-based methods suffer from high compu- tational difficulty owing the double sampling issue (Section 3 when approximating the gradient of the variance term (Tamar et al. 20122; Prashanth & Ghavamzadehi 2013 2016). To avoid this, Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) proposed multi- time-scale stochastic optimization. Further, Xie et al. (2018) proposed method based on the Legendre-Fenche
Existing MVRL methods (Tamar et al., 2012 Prashanth & Ghavamzadeh 2013
2016;
duality (Boyd & Vandenberghe (2004). Although these methods avoid the double sampling issue, as we experimentally report in Figure of Section 6 there still remains the difficulty in training a policy.
To avoid these difficulties, this paper considers another approach for MVRL by focusing on obtaining policy that is located on the Pareto efficient frontier in the sense of MV trade-off; that is, we cannot increase the expected reward without increasing the variance and decrease the variance without decreasing the expected reward (Section 2.2). To achieve this purpose,
propose training RL agent by direct expected quadratic utility maximization (EQUMRL) with a policy gradient method (Williams 1988 1992. Sutton et al.) 2000 Baxter & Bartlett 2001). We show that the maximizer of the objective in EQUMRL is Pareto efficient in the sense of MV trade-off.
As important property, EQUMRL does not suffer from the double-sampling problem because it does not include the variance estimation, which is a cause of the problem. In conventional methods, we can also obtain MV-efficient policy when succeed in solving the constraint problem. However,

---

Under review as a conference paper at ICLR 2022
as discussed in the work and shown in our experiments, such Figure 2 those methods do not perform well due to the difficulty in computation. Compared with them, EQUMRL is computationally friendly and experimentally returns more MV-efficient policies. In addition. the EQUMRL is well suited to financial applications, such as portfolio management because in economic theory. using MV portfolio is justified maximizer of the expected quadratic utility function (Markowitz) T952, Luenberger et al., 1997). We introduce brief survey on expected quadratic utility maximization and finance in Appendix We list the following three advantages to the EQUMRL approach:
ii) our proposed EQUMRL is able to learn Pareto efficient policies and has plenty of interpretations from various perspectives (Section 4.3);
(ii) our proposed EQUMRL does not suffer from the double sampling issue by avoiding explicit upproximation of the variance (Section[4) 4)
(iii)
experimentally show that our proposed EQUMRL returns more Pareto efficient policies than existing methods (Section
Then, we propose In the following sections, we first formulate the problem setting in Sections 33
the main algorithms in Section 4 Finally, we investigate the empirical effectiveness in Section 6
2 PROBLEM SETTING
We consider a standard setting of RI,, where an agent interacts with an unfamiliar, dynamic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time. We define an MDP as a tuple (S, A, P. Po). where S is a set of states, A is set of actions, r S x A - IR is a stochastic reward function with finite mean and variance, P S x S x A [0, 1] is transition kernel, and Po 5 - [0, 1] is an initial state distribution. The initial state S₁ is sampled from Po- Let To A x S + [0, be a parameterized stochastic policy mapping states to actions, where a tunable parameter, and we denote the parameter space by O At time step I, an agent chooses an action A₁ following a policy Tt(- S.). We assume that the policy my is differentiable with respect to 0; that is, @ma(sla) exists.
Let us define the expected cumulative reward from time step to Ex, [Gruw, where Gru E'O 7'r(Se+i, Acts), 7 E (0, L] is discount factor and Ex,, denotes the expectation operator over policy na. and S₁ is generated from Pg. When well-defined, we usually assume that all policies are proper (Bertsekas & Tsitsiklis) 1995): that is, for any policy Te, agent goes to recurrent state S with probability 1, and obtains 0 reward after passing the recurrent state 5* at a stopping time r. This finite horizon setting is called episodic MDPs (Puterman 1994). For brevity, we denote Gt... G when there is ambiguity.
1, to ensure that the cumulative reward is
In this paper, we consider the trade-off between the mean and variance of the reward. Let us denote the variance of a random variable W under Te by V a. (W). Note that EE,,, [G] and Vx (G) are finite because r r(,,, Ar) has finite mean and variance.
2.1 TRAJECTORY VARIANCE PERSPECTIVE.
direction of MVRL to consider the trajectory mean E,, (Geu) and trajectory variance Wn (Gea):
E
trade-off
V (G).
A typical method for considering the MV trade-off is to train a policy under some constrains. Tamar al. (2012), Prashanth Ghavamzadehi (2013). and Xie et al. (2188) formulated MVRL
max Em, [G] s.t. Vxo (G) 17. D50
We call an algorithm based on this formulation trajectory MV-controlled RL. In the trajectory MV-controlled RL, the goal is to maximize the expected cumulative reward, while controlling the trajectory-variance at a certain level. To be more precise, their actual constraint condition is (G) S 17. However, if Vn, (G) = n is feasible, the optimizer satisfies the equality in applications, where we need to consider MV trade-off, such as financial portfolio management. Therefore, we only consider an equality constraint. To solve this problem, Tamar et al. (2012). Prashanth & Ghavamzadeh (2013), and Xie et al. (008)) considered a penalized method defined as
2

---

Under review as a conference paper at ICLR 2022
maxéce Bx,[G] (g(WV (G) 1)), where 5 > 0 is constant and g R - R is a penalty function, such g(x) x or g(z) x²? See Remark 1 for more details.
2.2 MV-EFFICIENT POLICY.
To consider the MV trade-off avoiding the computational difficulty, this paper aims to train a Pareto efficient policy in the sense of the MV trade-off, where cannot increase the mean (resp. decrease the variance) without increasing the variance (resp. decreasing the mean). Following the existing literature mainly in economics, finance, and operations research (Luenberger al. 1997). we define a trajectory MV-efficient policy as a policy To such that there is no other policy with 0' E θ, where Eua G - E [G] and |.] V [G], or V [G] 2 VTT, [G] and ET,, [G] < T [G]. Efficient frontier is defined as set of the MV-efficient policies. From the definition of the MV-controlled policies, the trained policies belong to the efficient frontier.
POLICY GRADIENT AND DOUBLE SAMPLING ISSUE
In this paper, consider training a policy by policy gradient methods. In MVRL. we usually require
the gradients of E. G and (G) = E [G E., [G. Tamar et al. (2012) ) and Prashanth &
Ghavamzadeh 2013) show that the gradients of E [G] and EN |G are given as
VolEs [G] Esp
G
Ve log (S As)
Vol.a |2
G
Ve log No (S Ad)
Because optimizing the policy Te directly using the gradients is computationally intractable, we replace them with their unbiased estimators. Suppose that there is simulator generating a trajectory k with ((S A r(S) } where 7 is the stopping time of the trajectory. Then, we can construct unbiased estimators of TE,,, [G) and ER,, [G]] as follows (Tamar et al. 2012):
V ET G
Vo log (S At) and V VOL,, [ (G+)
log To (S) At), (1)
where G is sample approximation of Ex, [G] the episode k. Besides, because Vo (Ex [G])
2. [G] VolE, [G]. the gradient of the variance given as VoVn (G) = V Exg [G2]
(E., (G) = Ena EG V log To Ar) - 2E G [G].
However, obtaining an unbiased estimator of V (E,, [G]] 2E,,, Vo [G] is difficult because requires sampling from two different trajectories for approximating E,, [G] and Vyl.,, [G). This issue is called double sampling issue and makes the optimization problem difficult when we include the variance V. ,. (G) into the objective function directly. Tamar al. (2012) and Prashanth & Ghavamzadeh (2013) reported this double sampling issue caused from the gradient estimation of the variance and proposed solutions based on multi-time-scale stochastic optimization. Recall that they consider the penalized objective function maxsee Er, [G] -ógg (V,, nu (G) 17) to obtain MV-controlled policies, where the double sampling issue caused by the gradient of ög (V.o (G) 17).
4 EQUMRL WITH TRAJECTORY VARIANCE PERSPECTIVE
In social sciences, we often assume that an agent maximizes the expected quadratic utility for considering MV trade-off. Based on this, we propose the EQUMRL for obtaining an MV-efficient policy. For a cumulative reward G, we define the quadratic utility function as u trajectoryy (G; a, 8) = oG : 3G2 where a and 0. EQUMRI, we train a policy by maximizing the expected value of the quadratic utility function,
1
Eng [u trajectory (G:a, = OIEE [G]
BE, [G
(2)
2
The EQUMRL agnostic to the learning method, that is, we can implement it with various existing algorithms such as REINFORCE and actor-crtic.
3

---

Under review as a conference paper at ICLR 2022
4.1 EQUMRL AND MV EFFICIENCY
The expected quadratic utility function Ea, [u (ujeeootooyy (G;a, 3)] is known to be Pareto efficient in the sense of mean and variance when its optimal value satisfies En [G] S 3. In order to confirm this, we can decompose the expected quadratic utility as
Bio [n u.aeeccorry (G: (,,
3)] = -- ET.
1
a
2
2 (V)
B
23
When a policy π E II is the maximizer of the expected quadratic utility, it is equivalent to an MV-efficient policy (Borch 1969 [aaoon 1977] [uenberger al. 1997). Following (Luenberger
et al. 1997 p.237-239), we explain this as follows:
Among policies me with a fixed mean [= = μ. the policy with the lowest variance
maximizes the expected quadratic utility function because E., [w [utrajectory (G; a, a.
op - --2²² 3% ne (G) a monotonous decreasing function on Va (G);
2. Among policies Th with a fixed variance VT (G) = o² and mean Ex, < 3 the
policy with the highest mean maximizes the expected quadratic utility function because
Ey (rajeeecy (G; a, B)] - [ET 3)
a² _o is monotonous increas- 23
ing function on E., [G] $ 3.
Based on the above property, propose maximizing the expected quadratic utility function in RL; that is, training an agent to directly maximize the expected quadratic utility function for MV control instead of solving constrained optimization. We call the framework that makes the RL objective function an expected quadratic utility EQUMRL.
Unlike the expected cumulative reward maximization in the standard RL setting, at time 4, it is desirable to include the past cumulative reward to the state S. because our objective function depends on it even given Sy. Let us consider the objective time with the infinite horizon setting:
C.TETT [Gu:oe - 3G So, Au, rSo, Ap), ... St-1, A r(St-1, 11
= =C + an ((1 2,3y² Gu: To (t Ss]] - By 1S:3.
where C' constant and recall that Goa-1 = Σ ('r(S,, ThuTh for better decision makking we include the past cumulative reward into the state space.
4.2 IMPLEMENTATION OF EQUMRL
In this section, we introduce how to train policy with the EQURL. We defined the objective function of the EQUMRL, and EQUMRL is an agnostic in learning method. As examples, we show an implementation based on REINFORCE (Williams) 1992; Brockman et al. 2016) and Actor-Critie (AC) methods (Williams & Peng 1991 Mnih et al. 2016). We use unbiased estimators of the gradients defined in (I).
For an episode with the length n, the proposed algorithm replaces Ex, [G] and Er, G2 with the sample approximations Σー=1r~->r( Ar) and
REINFORCF-based trajectory EQUMRI.
t== n 1 (S,,, A.) respectively Tamar et al. 2012;; that is, the unbiased gradients given as Therefore, for sample approximation Thh of E,o [G²] at the episode k, we optimize the policy with ascending the unbiased gradient
V
L trajectory (G;a,
-(act
B (3t) 2
a log (St At).
CWe summarize the algorithm as the pseudo-code in Algorithm
Here, we present three advantages of EQUMRL.The first advantage concerns computation. In EQUMRL, we do not suffer from the double sampling issue because the term (Es, [G)² which causes the problem, is absent, and we must only estimate the gradients of E., [G] and Er,, [
The second advantage is that provides a variety of interpretations, as listed in Section
4

---

Under review as a conference paper at ICLR 2022
The third advantage is the ease of theoretical analysis. For example, referring to the results of [Bertsekas & Tsitsiklis (1996), we derive the following result on the convergence of the gradient, which can be applied to simple pol- icy gradient algorithmss, not only our proposed REINFORCE-based algorithm.
Algorithm REINFORCE based EQUMRL
Initialize the policy parameter On:
for k 111,2... do
Generate {(S) A r(s, A on Te;
Update policy parameters 0k+1 t 02
[u (uraaeectory aG;a, 1/t V.
end for
Theorem 1. Consider an update rule such that
where the learning rates Th are non-negative and satisfy =k=0 17k: 00 and Lk=o The < oc. Suppose that (a) episode always finishes in finite
Ok41 < Og + 1/. Vna,, [u [u"rajectory
horizon n2 (b) the policy To has always bounded first and second partial derivatives. Then,
lim;- VOLETO [aG 1. almost surely.
It is expected that non non-asymptotic results can be derived by restricting the policy class and the optimization algorithm as Agarwal et al. (2020) and Zhang al. (2021a), but this is not the scope of this paper, which aims to provide a general framework.
Another implementation of the EQUMRL is to apply the AC algorithm (Williams & Peng. 1991 Mnih et al. (2016). For an episode k with the length n, following Prashanth & Ghavamzadeh (2013) 2016). we train the policy by a gradient defined as
AC-based trajectory EQUMRL.
Ve log No (5,
aG
(MM 26
(S,)
1 BM 2 (5
2
where Ekk Giitt-1
Gh Chetn-1 + 7"h TmM (Sttn), ai
Giitn-1
2
(Ghitn-1)
2nth Gk Ma) (St+n) + 72t M(2) (St and M( (S)) and M Gbenn-1 ab
(St) are models
a
a( and w For more details, see Prashanth & of Ioo and E G nool with parameters Ghavamzadeh 2013 2016).
Remark (Existing approaches). For the double sampling issue, Tamar et al. (012) and Prashanth & Ghavamzadeh 2003 2016) proposed multi-tin tet--tieeeccae scale stochastic optimization. Their approaches are known be sensitive to the choice of step-size schedules. which are not easy to control (Xie et al., 2018). Xie et al. (2018) proposed using the Regendre-Fencher dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, Xie et al. (2018) set an objective function as maxeçe Ex, [G] - 5 (V,,, (G) 17). Then, Xie et al. (2018) transformed the objective function as maxoge. 2y (E_,, [G] +
2² Er, G and trained an agent by solving the optimization problem via coordinate descent algorithm. However, this approach does not reflect the constraint n) because the constraint condition 7) vanishes from the objective function. This problem is caused by their objective function based on the penalty function ((x) = x: Ez [G] 8 (V nT (G) - 11), where the first derivative does not include 17. To avoid this problem, we need iterative algorithm to decide an optimal 8 or change g(x) from x but it is not obvious how to incorporate them into the approach.
Remark 2 (Difference from existing MV approaches). Readers may assert that EQUMRL simply omits (Ez,, [G)) from existing MV-controlled RI. methods, which usually includes the explicit variance term in the objective function, and is the essentially the same. However, there are significant differences; one of the main findings of this paper our formulation of simpler RL problem to obtain an M MV-efficient policy. Existing MV-controlled RL methods suffer from computational difficulties caused by the double sampling issue. However, we can obtain MV-efficient policy without going through the difficult problem. In addition, EQUMRL shows better performance in experiments even from the viewpoint of the constrained problem because is difficult to choose parameters to avoid the double sampling issue in existing approaches. Thus, the EQUMRL has advantage in avoiding solving more difficult constrained problems for considering MV trade-off.
4.3 INTERPRETATIONS OF EQUMRL WITH GRADEINT ESTIMATION
We can interpret EQUMRL as an approach for (i) targeting optimization problem to achieve an expected cumulative reward Ç. (ii) an expected cumulative reward maximization with regularization, and (iii) expected utility maximization via Taylor approximation.
5

---

Under review as a conference paper at ICLR 2022
First, can also interpret EQUMRL mean squared error (MSE) minimization between a cumula- tive reward R. and a target return Ç; that is,
arg min J(:; () arg min ETD G))
(4)
$50
DEE
We can decompose the MSE into the bias and variance as
EE [( (5-G)) (s Erg (C))
2E
(s
Erg
(B)) -GG)
G)
+Eio [Br,[G]- G)
Bias
Variance
= =(² - 2g G + (En, [G]
Em [G. G)
Thus, the MSE minimization (4) is equivalent to EQUMRL B), where Ç 3. The above equation provides an important implication for the setting of Ç. we know the reward is shifted by x. we only have to adjust to Ç + r. This is because Ç only affects the bias term in the above equation. The equation also provides another insight. If our assumption max,, Exp [G] S C violated, E,, G will not be maximized with fixed variance and will be biased towards Ç; that is, EQUMRL cannot find the V-efficieen policies. In applications, we can confirm whether the optimization works by checking whether average value of the empirically realized cumulative rewards is less than Ç.
Second, we can regard the quadratic utility function as an expected cumulative reward maximization with regularization term defined as E R²]] that is, minimization of the risk R(Ta):
R(0)~
E [G]
R [
Risk of expected cumulative reward maximization
Regularization term
where v > O is regulation parameter and 1/ 2a
X As 0 (5 oo). R(0) + -En, [G].
Third, the quadratic utility function is the quadratic Taylor approximation of smooth utility function u(C u(G) because for Go E IR, we can expand as u(G) u(Go) + (G - Go) + (G)
Go)² +...; that is, quadratic utility is approximation of various risk-averse utility functions. This property also supports the use of the quadratic utility function in practice (Kroll et al. 1984).
It also should be noted that the EQUMRL is closely related to the fields of economics and finance, where the ultimate goal is to maximize the utility of an agent, which is also referred to as an investor. The quadratic utility function is a standard risk-averse utility function often assumed in financial theory Luenberger et al. (1997) to justify an MV portfolio; that is, an MV portfolio maximizes an investor's utility function if the utility function is quadratic (see Appendix (A). Therefore, our approach can be interpreted as a method that directly achieves the original goal.
Remark 3 (Specification of utility function (hyper-parameter selection)). Next, we discuss how to decide the parameters a and 3, which are equivalent to Ç, E, and s. The meanings are equivalent constrained conditions of MVRL; that is, we predetermine these hyperparameters depending on our attitude toward risk. For instance, we propose the following three directions for the parameter 20 ve based on economic theory or market research (Ziemba et al. 1974, Kallberg et al. 1983) (Appendix Ap. [uenberger et al. (1997) proposed some questionnaires to as the targeted reward that investors aim to
choice. First, we can determine 9
investors for the specification. Second, we set Ç can optimize the regularization parameter v to maximize
achieve. Third, through vross-validation, some criteria, such as the Sharpe ratio (Sharpe 1966).. However, we note that in time-series related tasks, cannot standard cross -oas-vvaliatioon owing to dependency. Therefore, in our experiments with a real-world financial dataset, we show various results under different parameters.
Remark 4 (From MV-efficient RI. MV-controlled RI.. We can also apply the MV-efficient RL. method as the MV-controlled RL method. The parameters of the expected quadratic utility function correspond to the variance that we want achieve. For example, the larger ( is in (4). the larger the variance will be. Although we do not know the explicit correspondence between the parameter of the expected quadratic utility function and the variance, it possible to control the variance by choosing an appropriate policy from among those learned under several parameters. In Figure i Section
[ we show the mean and variance of several policies trained with several parameters. These results are measured with test data, but by outputting multiple candidates with the training data (ex. vooss-validttion,, we can choose a policy with the desired variance.

---

Under review as a conference paper at ICLR 2022
REIFFORCE
<-10
(od
Cid
Qe2
REFEFORCE
15
10-
(H2
-1
0
100
200
300
400
500
100
200
300
400
500
Episode
Episode
Figure 1: The CRs and Vars in the training process of the experiment using the synthetic dataset.
EQUMRL UNDER PER-STEP VARIANCE PERSPECTIVE
Another direction of MVRL consider the trade-off between the per-step mean (r A:)] and
V (r(S,, A.)). As well as the trajectory perspective, Bisi et al. (2020) and Zhang et al. (222bb) proposed training a policy to maximize the penalized objective function, m Σ/= x" (E (r( AV (r(S,, At))). We call an algorithm with this formulation per-step MV-controlled RL. See Appendix B more details.
per-step variance V s. (r Se, At)), Exg [r(S,, As)
trade-off
We define a per-step MV-efficient policy as a policy To such that there is no other policy with 8' E O where, for each t, E [r(S, Ar)] < [r(e, Ac)] and W [r(s,, A.)] > Va [r(S,, Ar). or Wna [r([,, ([]S,,A,)] A,]] 2 V sor (r(S, Ar] and E., (r(S, Ar]] < Eng. (r(S,,A)). For constants 3 > we define the quadratic utility function for per-step reward setting as aperstee(r(SS, At); ar = =ar(S,, At) - (S,
For an finite horizon case with y 1, we consider maximizing the expected cumulative quadratic utility function defined as E.. ET=, uper-step (r(S,, Ar); 3)] alE,,, Σt= r:S;, 3B.,, Ei=rr (S,, A.)
When applying REINFORCE-based algorithm to train an agent, the sample approximation of the gradient given as VE, Σ pper-ste((rSSt ab);
(ar(s, A/)
3 (r((S, A3)) V log (St A Appendix B contains the performance using synthetic dataset.
EXPERIMENTS
This section investigates the empirical performance of the proposed EQUMRL. in trajectory variance setting using synthetic and real-world financial datasets. We train the policy by GEINFORCE-basee algorithm. We conduct two experiments. In the first experiments, following Tamar et al (2012 2014), and Xie al. (0188). conduct portfolio management experiments with synthetic datasets. In the third experiment, we conduct a portfolio management experiment with dataset of Fama & French (1992), standard benchmark in finance. In Appendix C following Tamar et al. 2012|2014) and Xie et al. (2018), we also conduct American-sty. option experiments with synthetic datasets. We implemented algorithms following the Pytorch example For algorithms using neural networks, we use a three layer perceptron, where the numbers of the units in two hidden layers are the same that of the input node, and that of the output node 2. We note that in all results, naively maximizing the reward minimizing the variance do not ensure better algorithm; we evaluate algorithm based on how controls the MV trf We denote the hyperparameter of the EQUMRL by Ç. which has the same meaning as (a, B) and vb. For all experiments, we adopt episodic MDPs; that is, 7 = 1.
6.1 PORTFOLIO MANAGEMENT WITH A SYNTHETICC DATASET
Following Tamar al. (2012) and Xie et al. 2018). we consider portfolio composed of two asset types: a liquid asset with fixed interest rate ri. and non-liquid asset with a time-dependent interest
rate taking either ui or nnl high and the transition follows a switching probability Pswitch- An investor can sell the liquid asset at every time step f = 1, ..,.,.. but the non-liquid asset can only be sold after the maturity W periods. This means that when holding 1 liquid asset, we obtain per period;
.hihh when holding 1 non-liquid asset the 1-th period, obtain ,.lo or r.i the W-th period.
https:

---

Under review as a conference paper at ICLR 2022
Table I: The results of the portfolio management with synthetic data. The upper table shows the CRs and Vars over 100, 000 trials. The lower table shows the MSEs for Ç over 100, 000 trials.
REINFORCE
EQUM
Tamar
Xic
<< 0c)
C 10
<
6
< 4
V 80
V/ - 50
A= 100
A 10
CK
6.729
6.394
4.106
2.189
6.709
2,851
0.316
0.333
Var
32,551
31,210
24.424
18.518
32.586
21.573
15,883
15.992
REINFORCE
EQUM
Target Value
<< = oc)
<= 10
8=8
C= 6
C= 4
C=2
MSE from
10
$1.669
53.199
65.307
83.061
42.586
42.975
43.375
45,730
55.816
71.480
34.154
36.570
=44
48.420
46.127
42.345
30,577
25.324
27.455
63.337
59.703
53.830
35.000
22.078
17.442
Besides, the non-liquid asset has a risk of not being paid with a probability Priski that is, if the non-liquid asset defaulted during the W periods, we could not obtain any rewards by having the asset. An investor can change the portfolio by investing a fixed fraction w of the total capital M in the non-liquid asset at each time step. A typical investment strategy is to construct a portfolio using both liquid and non-liquid assets for decreasing the variance. Following Xie et al. (2018), we set
2, Pswitch = 0.1, Prisk = 0.05, W = 4, w = 0.2, and M = 1. As performance metric, we use the average cumulative reward (CR) and its variance (Var) when investing for 50 periods. We compare the EQUMRL with the REINFORCE, MV-controlled methods proposed Taamar et al. (2012) (Tamar), and Xie al. (2018) (Xie). We denote the variance constraint of Tamar et al. (2012) as Var and Lagrange multiplier of Xie et al. (2018) as A. For training Tamar, Xie, and EQUMRL,, we the Adam optimizer with learning rate 0.01 and weight decay parameter 0.1. For cach algorithm, we report performances under various hyperparameters as much as possible.
n = 1.001, nloo = 1.1, nl
First, we show CRs and Vars of the EQUMRL during the training process in Figure D where we conduct 100 trials on the test environment to compute CRs and Vars for each episode. Here, we also show the performance of REINFORCE, which corresponds to the EQUMRL with Ç = a/B co. As Figure shows, the EQUMRL trains MV-efficient policies well depending on the parameter Ç. Next, we compare the EQUMRL on the test environment with the REINFORCE. Tamar, and Xie. We conduct 100, 000 trials on the test environment to compute CRs and Vars. In Figure 2 we plot performances under several hyperparameters, where the hori- zontal axis denotes the Var, and the vertical axis denotes the CR. Trained agents with a higher CR and lower Var are Pareto effi- cient. As the result shows, the EQUMRL returns more efficient portfolios than the others in almost all cases. We conjecture that this is because while the EQUMRL is an end-to-end optimization for obtaining an efficient agent, the other methods consist of several steps for solving the constrained op- timization, where those multiple steps can be sources of the suboptimal result. We show CRs and Vars of some of their results in the upper table of Table [ The MSEs between ( and CR are also shown in the lower table of Table [ where we can confirm that the EQUMRI. succeeded in minimizing the MSEs.
8
REINFORCE
10
EQUM
var 90
6
Tamar
Mar - 80
C-6
vir 60
4
Pareto efficient
2
C-3
2-2
var 40
0
1=
1000
Var
30
100.
var
20
-2
10
15
20
25
30
35
40
45
Var
Figure 2: MV efficiency of the portfolio manage- ment experiment. Higher CRs and lower Vars methods are MV Pareto efficient.
6.2 PORTFOLIO MANAGEMENT WITH A REAL-WORLD DATASET
We use standard benchmarks called Fama & French (FF) dataset to ensure the reproducibility (Fama & French 1992). Among FF datasets, we use the FF25, FF48 and FF100 datasets, where the FF25 and FF 100 dataset includes 25 and 100 assets formed based on size and book-to-market ratio; the FF48 dataset contains 48 assets representing different industrial sectors We use all datasets covering
ht ///ba/////aaa/// tuck.
wtw.aaart.cccd..d........ dartmouth http:hhhhwppppggeeaaaa.tty///// http://aaattt//eee..reeeenc////a french/data librarrr___bbry...o
html
8

---

Under review as a conference paper at ICLR 2022
Table 2: The performance of each portfolio model during the out-of-sample period (from July 2000 to June 2020) for FF25 dataset (upper table). FF48 (middle table), and FF100 (lower table). For each dataset, the best performance is highlighted in bold.
Method
EW
MV
EGO
BLD
V
Tamar
V
À
Xie
100
EQUM
0.75
1.5
F1225
CRT
0.80
0.09
0.81
0.84
1.15
1.01
0.50
0.90
1.15
1.15
1.33
1.25
1.11
Vari
28.62
53.37
30.65
22.16
18.35
15.04
25.29
25.56
25.77
25.77
24:27
15.02
11.77
R/R1
0.52
0.04
0.51
0.62
0.93
0.90
0.55
0.61
0.79
0.79
1.07
1.12
1,13
MaxDDD
0.54
0.58
0.56
0.36
0.31
F9488
CRT
0.11
0.97
0.75
1,08
1.60
Varl
22.91
77.02
31.91
15.98
40.69
16.00
18.04
27.12
23.77
26.31
31.97
18.69
10.34
0.39
0.60
0.98
MaxDD]
0:30
0.48
0.31
0.25
0:32
0.20
0.21
0.27
0.25
0.26
0.29
0.21
0.16
FF100
CRT
0.81
0.11
0.81
0.85
1.13
0.79
0.67
0.37
0.58
1.24
0.95
0.95
1.43
Var.
29.36
57.97
32.35
21.83
25.50
20.34
16:50
87.92
28.78
41.06
15.50
14.36
33.09
R/R1
0.32
0.05
0.49
0.63
0.77
0.61
0.57
0.21
0.63
0.67
0.83
0.87
0.35
MaxDDI
0.33
0.46
0.34
0.27
0.25
0.23
0.25
0.51
0.32
0.31
0.19
0.26
0.31
monthly data from July 1980 to June 2020. Consider an episodic MDP. Let the action space be the set of m assets, and ya. and Wa,s be the return and the portfolio weight of asset at time t. The reward (portfolio return) at time 1 S t S T is defined as yt = >a=1 Wa, - A _ |aat Wa.e-1l where A 0.001 is the penalty of the portfolio weight turnover(change of the action). On the f-th period, the agent observes state S. = ((ya,=- 11 M0, Wa.t-1)acA Σt= Vs), and decides a portfolio weight (Wa,)aELA as Wa.t T (a, 8t)). Between periods / and ( [ 1, the agent has asset with the ratio See Section 4. for the reason why we include sum 203/ in the state.
We use the following portfolio models: an equally-weighted portfolio (EQ, DeMiguel et al. 2009); mean-variance portfolio (MV, Markowitz 1952): Kelly growth optimal portfolio Shen al. (2019): Portfolio blending via Thompson sampling (BLD.SShen & Wang (016).. We also compare our proposed method with the methods proposed by Tamar et al. (2012) and Xie al. (2018). Denote a method proposed by Tamar et al.] (2012) by Tamar, and choose the parameter var from (15,30, 60). Denote a method proposed by Xie et al. (2018) by Xie, and choose the parameter from (10, 100, 1000}. Denote the REINFORCE- choose the parameter from (0.5, 0.75, 1.5).
trajectory EQUMRL by EQUMRL, and
We apply the following standard measures in finance for evaluation (Brandt 2010). The cumulative reward (CR), annualized risk as the standard deviation of return (RISK) and risk-adjusted return (R/R) are defined as follows: CR 1/T E Yt. V = L/ - CR)2, and R/R. = /12 x OR/vVar. R/R is the most important measure for a portfolio strategy and is often referred to as the Sharpe Ratio (Sharpe 1966). We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure (Magdon IMsaal & Atiya 2004). In particular, MaxDD is the
largest drop from peak defined as MaxDD = min 0. max-eel.e W, - 1). where W. is the
cumulative return of the portfolio until time k; that is. W. II (1 yr).
Table reports the performances of the portfolios. almost all cases, the EQUMRL achieves the highest R/R and the lowest MaxDD. Therefore, we confirm that the EQUMRL has high R/R, and avoids a large drawdown. The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP, and EQUMRL is shown in Appendix D Except for FF48's MVP, the objective itself is smaller than that of EQUMRL. Since the values of the objective are proportional to R/R, we can empirically confirm that the better optimization, the better performance.
7 CONCLUSION
In this paper, we proposed EQUMRL for MV-efficient RL. Compared with the conventional MVRL methods, EQUMRL is computationally friendly. The proposed EQUMRL also includes various inter- pretations, such targeting optimization and regularization. which expands the scope of applications of the method. We investigated the effectiveness of the EQUMRL compared with the standard RL and existing MVRL methods through experiments using synthetic and well-world datasets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. Optimality and approximation with policy gradient methods in markov decision processes. In Abernethy, and Agarwal, (eds.). Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 64-66, 09-12 Jul 2020.
Baron, D. On the utility theoretic foundations of mean-variance analysis. The Journal of Finance. 32, 1977.
Baxter, J. and Bartlett P. L. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319-350, 2001.
Bertsekas, D. and Tsitsiklis, J. N. Neuro-dynamic programming: An overview. In Proceedings of 1995 34th IEEE Conference on Decision and Control, volume 1, pp. 560-564. IEEE, 1995.
Bertsekas, D. P. and Tsitsiklis, J. N. Neuro-dynamic programming. Athena Scientific, Belmont, MA, 1996.
Bisi, L.. Sabbioni, L., Vittori, E., Papini, M., and Restelli, M. Risk-averse trust region optimization for reward-volatiliy reduction. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020.
Bodnar, T., Parolya, N., and Schmid, W. On the exact solution of the multi-period portfolio choice problem for exponential utility under return predictability. European Journal of Operational Research, 246(2):528 542, 2015a.
Bodnar, T., Parolya, N., and Schmid, W. A closed-form solution of the multi-period portfolio choice problem for quadratic utility function. Annals of Operations Research, 229(1)) (121:1))111518 2015b.
Bodnar, T., Okhrin, Y., Vitlinskyy, V., and Zabolotskyy, T. Determination and estimation of risk aversion coefficients. Computational Management Xcience, 15(2):297-317, 2018.
Borch, K. A Note on Uncertainty and Indifference Curves. The Review of Economic Studies, 1969.
Boyd, S. and Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
Brandt, M. W. Portfolio choice problems. In Handbook of Financial Econometrics: Tools and Techniques, pp. 269-336. Elsevier, 2010.
Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.
Bulmuş. T. and Özekici, S. Portfolio selection with hyperexponential utility functions. OR spectrum, 36( (6):))733 2014.
Chow, Y. and Ghavamzadeh, M. Algorithms for cvar optimization in mdps. Advances in Neural Information Processing Systemx, 27:3509-3517, 2014.
Chow, Y., Ghavamzadeh, M., Janson, L.,, and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.
DeMiguel, V., Garlappi, L.., and Uppal, R. Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy? The Review of Financial Studies, 22(5):1915-195) 2009.
Fama, E. F. and French, K. R. The cross-section of expected stock returns. The Journal of Finance, 1992.
Fishburn, P. C. and Burr Porter, R. Optimal portfolios with one safe and one risky asset: Effects of changes in rate of return and risk. Management science, 22(10):0064-100 1976.
Garcia, J. and Fernández, F. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Geibel, and Wysotzki, F. Risk-sensitive reinforcement learning applied to control under constraints.
Journal of Artificial Intelligence Research, 24:81-108, 2005.
10

---

Under review as a conference paper at ICLR 2022
Kallberg, J., Ziemba, W., et al. Comparison of alternative utility functions in portfolio selection problems. Management Science, 29(11):2257-1276, 1983.
Kroll, Y., Levy, H., and Markowitz, H. M. Mean-variance versus direct utility maximization. The Journal of Finance, 39(1):47-8) 61, 1984.
Lintner, J. The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets. Review of Economics and Statistics, 47(1))13-37, 1965.
Luenberger, D. G. et al. Investment science. OUP Catalogue, 1997.
Magdon-Ismail, M. and Atiya, A. F. Maximum drawdown. Risk Magazine, 2004.
Markowitz, H. Portfolio selection. The Journal of Finance, 1952.
Markowitz, H. Portfolio selection: efficient diversification of investments. Yale university press, 1959.
Markowitz, H. M. and Todd, G. Mean- Variance Analysis in Portfolio Choice and Capital Markets, volume 66. John Wiley & Sons, 2000.
Mas-Colell, A., Whinston, M. D., Green, J. R., et al. Micrveconomic Theory, volume 1. Oxford university press New York, 1995.
Merton, R. Lifetime portfolio selection under uncertainty: The continuous-time case. The Review of Economics and Statistics, pp. 247-257, 1969.
Mnih, V., Badia, A. P. Mirza, M., Graves, A., Lillicrap. T., Harley, Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp.
2016.
Morgenstem, O. and Von Neumann, J. Theory of games and economic behavior. Princeton University Press, 1953.
Mossin, Equilibrium in a capital asset market. Econometrica: Journal of the econometric society, pp. 768-783, 1966.
Prashanth, L. and Ghavamzadeh, M. Actor-critic algorithms for risk-sensitive mdps. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume pp- 252-260, 2013.
Prashanth, L. and Ghavamzadeh, M. Variance-constrained awt. average reward mdps. Machine Learning, 105(3):367-417, 2016.
actor-critic algorithms for discounted and
Puterman, M. I.. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
Sharpe, W. F. Capital asset prices: theory of market equilibrium under conditions of risk. The Journal of Finance, 19(3):255 442, 1964.
Sharpe, W. F. Mutual fund performance. The Journal of Business, 1966.
Shen, W. and Wang, J. Portfolio blending via thompson sampling. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 1983-1989, 2016.
Shen, W., Wang, B., Pu, J., and Wang, J. The kelly growth optimal portfolio with ensemble learning. 2019.
In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, 1998.
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pp. 1057-1063, 2000.
11

---

Under review as a conference paper at ICLR 2022
Tamar, A., Di Castro, D., and Mannor, S. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning. pp. 1651-1658, 2012.
Tamar, A., Mannor, S., and Xu, H. Scaling up robust mdps using function approximatio.. In International Conference on Machine Learning, pp. 181-189, 2014.
Tobin, J. Liquidity preference as behavior towards risk. The Review of Economic Studies, 25(2): 65-86, 1958.
Williams, R. Toward a Theory of Reinforcement-larrinng Connectionist Systems. Northeastern University, 1988.
Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229 256,, 1992.
Williams, R. J. and Peng, J. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Xie, T., Liu, B., Xu, Y., Ghavamzadeh, M., Chow, Y., Lyu, D., and Yoon, D. A block coordinate ascent algorithm for mean- variance optimization. Advances in Neural Information Processing Systems, 31:1065-1075, 2018.
Zhang, J. Kim, J., O'Donoguue, B., and Boyd, S. Sample efficient reinforcement learning with reinforce. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):1088-110855 May 2021a.
Zhang, S., Liu, B., and Whiteson, S. Mean-variance policy iteration for risk-averse reinforcement learning. Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021b.
Ziemba, W., Parkan, C., and Brooks-Hin Hill, R. Calculation of investment portfolios with risk free borrowing and lending. Management Science, 21(2), 1974.
12

---

Under review as a conference paper at ICLR 2022
A PRELIMINARIES OF ECONOMIC AND FINANCIAL THEORY
A.1 UTILITY THEORY
Utility theory the foundation of the choice theory under uncertainty including economics and financial theory. A utility function ul-) measures agent's s relative preference for different levels of total wealth W. According Morgenstern & Von Neumann (1953), rational agent makes an investment decision to maximize the expected utillity of wealth among a set of competing feasible investment alternatives. For simplicity, the following assumptions are often made for the utility function used in economics and finance. First, the utility function is assumed to be at least twice continuous differentiable. The first derivative of the utility function (the marginal utility of wealth) is always positive, i.e., U' (W) > O, because of the assumption of non-satiation. The second assumption concerns risk attitude of the agents called "risk averse". When assume that agent is risk averse, the utility function is described as curve that increases monotonically and is concave. The most often used utility function of risk-averse agent is the quadratic utility function follows:
u uW;a,, aW - -
(5)
where 2 8 O. Taking the expected value of the quadratic utility function in yields:
Elu(W;a, [)] <.E[W] 3il[
(6)
Substituting [E[3]] = V[W] E[ into 5 gives
E [[u(W:a, 8)] aE[W] - B(W]3)
B[V[]]
Equation ( shows that expected quadratic utility can be described in terms of mean E[] and variance V v[W] of wealth. Therefore, the assumption of a quadratic utility function is crucial to the mean-variance analysis.
Remark (Approximation by quadratic utility function). Readers may be interested in how the quadratic utility function approximates other risk averse utility functions. Kroll et al. (1984) empiri- cally answered this question by comparing MV portfolio (maximizer of expected quadratic utility function) and maximizers of other utility functions. In their study, maximizers of other utility func- tions also almost located in MV Pareto efficient frontier; that expected quadratic utility function approximates other risk averse utility function well.
Remark (Non-vNM utility functions). Unlike MV trade-off, utility functions maximized by some recently proposed new risk criteria, such as VaR and Prospect theory, do not belong to traditional vNM utility function.
A.2 MARKOWITZ'S PORTFOLIO
Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al. (2012) and Xie et al. (2018) pointed out. The mean-variance trade-ofii is justified by assuming either quadratic utility function to the economic agent or multivariate normal distribution to the financial asset returns (Borch, 1969 Baron 1977, Luenberger et al., 1997). This means that if either the agent follows the quadratic utility function or asset return follows the normal distribution, the agent's 's expected utility function is maximized by maximizing the expected reward and minimizing the variance. Therefore, the goal of Markowitz's ''s portfolio is not only to construct the portfolio itself but also to maximize the expected utility function of the agent. See Figure
Markowitz 1952) proposed the following steps for constructing MV controlled portfolio (Also see Markowitz
1959
page 288, and Luenberger et al. (1997))
Constructing portfolios minimizing the variance under several reward constraint;
Among the portfolios constructed in the first step, the economic agent chooses portfolio maximizing the utility function.
13

---

Under review as a conference paper at ICLR 2022
MV efficient frontier
Relationship between extstingg MVRL
Expected refurn
Investor's Quadratic Utliiyy function
Efficient portfolios
EQUM
Constrained condition
EEtheer of the mean var varance that masimize the ullity fundtion.
Maximizer of an expected quadratic utiity function
Train an agent to maximize the expected quadratic utiltyy function
Train an agent to satiety the constraint condition
Estimate the mean variance and panalize the objective fundtion
Efficient frontier
Maximize the expected quadratic utility Point on Pareto efficient frontier
Variance
Figure 3: The concept of MV Preto efficient frontier and difference between EQUMRL and existing MVRL approaches.
Conventional financial methods often adopt this two-step approach because directly predicting the reward and variance to maximize the expected utility function is difficult; therefore, first gathering information based on analyses of economist, then construct the portfolio using the information and provide the set of the portfolios to an economic agent. However, owing to the recent development can directly represent the complicated economic dynamics using flexible
of machine learning, models, such as deep neural networks. In addition, as Tamar et al. (012) and Xie et al. (2018) reported, when constructing the mean-variance portfolio in RL. sufter from the double sampling issue. Therefore, this paper aims to achieve the original goal of the mean-variance approach; that is, the expected utility maximization. Note that this idea is not restricted financial applications but can be applied to applications where the agent utility can be represented only by the mean and variance.
A.3 MARKOWITZ'S PORTFOLIO AND CAPITAL ASSET PRICING MODEL
Markowitz's portfolio known as the mean-variance portfolio (Markowitz 1952: Markowitz & Todd 2000). Constructing the mean-variance portfolio is motivated by the agent's s expected utility maximization. When the utility function given as the quadratic utility function, or the financial asset returns follow the multivariate normal distribution, portfolio maximizing the agent's expected utility function is given as a portfolio with minimum variance under a certain standard expected reward.
The Capital Asset Pricing Model (CAPM) theory is a concept which is closely related to Markowitz's portfolio (Sharpe 1964 Lintner 1965 Mossin 1966). This theory theoretically explains the expected return of investors when the investor invests in a financial asset; that is, it derives the optimal price of the financial asset. To derive this theory, well Markowitz's portfolio, assume the quadratic utility function to the investors or the multivariate normal distribution to the financial assets.
Merton (1669) extended the static portfolio selection problem to a dynamic case. Fishburn & Burr Porter (1976) studied the sensitivity of the portfolio proportion when the safe and risky asset distributions change under the quadratic utility function. Thus, there are various studies investigating
relationship between the utility function and aver- optimization (Tobin 1958 Kroll et al. 1984
Bulmuy Özekici 2014 Bodnar et al. 2015a b).
A.4 MV PORTFOLIO AND MVRL
Traditional portfolio theory have attempted to maximize the expected quadratic utility function by providing MV portfolios. This is because MV portfolio easier interpret than EQUMRL, and we can obtain solution by quadratic programming. One of the main goals of MVRL methods is also to construct MV portfolio under a dynamic environment (Tamar et al. 2012). However, we conjecture that there are three significant differences between them. First, unlike static problem, MVRL suffers computational difficulties. Second, while static problem solves quadratic programming given the expected reward and variance, MVRL methods simultancously estimate these values and solve the constrained problem. Third, while static MV portfolios gives us an exact solution of the constrained problem, MVRL often relaxes the constrained problem by penalized method, which
14

---

Under review as a conference paper at ICLR 2022
causes approximation errors. In particular, for the second and third points, the difference in how to handle the estimators of expected reward and variance is essential.
A.5 EMPIRICAL STUDIES ON THE UTILITY FUNCTIONS
The standard financial theory is built on the assumption that the economic agent has the quadratic utility function. For supporting this theory, there are several empirical studies estimate the parameters of the quadratic utility function. Markowitz & Tood (2000) discussed how the quadartic utility function approximates the other risk-averse utilityy functions. ZZiebba et al. (1974) investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian financial dataset. Recently, Boonnrr et al. (2018) investigate the risk parameter (a and / in formulation of the quadratic utiliity function) using the markets indexes in the world. They found that the utility function parameter depends on the market data model.
A.6 CRITICISM
For the simple form of the quadratic utility function, the financial models based on the utility are widely accepted in practice. However, there is also criticism that the simple form cannot capture the real-world complicated utility function. For instance, Kallberg et al. (1983) criticized the use of the quadratic utility function and proposed using utility function, including higher moments. This study also provided empirical studies using U.S. financial dataset for investigating the properties of the alternative utility functions. However, to the best of our knowledge, financial practitioners still prefer financial models based on the quadratic utility function. We consider this is because the simple form gains the interpretability of the financial models.
A.7 ECONOMICS AND FINANCE
To mathematically describe an attitude toward risk, economics and finance developed expected utility theory, which assumes the Bernoulli utility function u(R) on an agent. In the expected utility theory, an agent acts to maximize the von Neumann-Morgensten (vNM) utility function U(F(r)) = (r))dF(r), where F(r) is distribution of R (Mas- MaleColell et al., 1995). We can relate the utility function form to agents with three different risk preferences: the utility function u( (t) is concave for risk averse agents; linear for risk neutral agents; and convex risk seeking agents. For instance, agent with u u(R)= R corresponds to a risk neutral agent attempting maximize their expected cumulative reward in a standard RL problem. For more detailed explanation, see Appendix or standard textbooks of economies and finance, such as Mas-Colell et al. (1995) and Luenberger et al. (1997).
To make the Bernoulli utility function more meaningful, we assume that it is increasing function with regard to R; that is, R S for all possible R. Even without the assumption, for a given pair of
(ox, B), optimal policy maximizing the expected utility does not change; that is, the assumption is only related to interpretation of the quadratic utility function and does not affect the optimization.
On the other hand, the constraint condition is determined by an investor to maximize its expected utility (Luenberger et al. 1997). Finally, in theories of economics and finance, investors can maximize their utility by choosing portfolio from MV portfolios. In addition, when R 3 for all possible value of R, policy maximizing the expected utility function is also located MV Pareto efficient frontier.
B PER-STEP VARIANCE PERSPECTIVE
Bisi et al. 2020) defines the per-step reward random variable R, a discrete random variable taking the values in the image of r, by defining its probability mass function as p(R x) E.a dno (s, a) [r a) z). where 1 is the indicator function and d,, is the normalized discounted state-action distribution, dr, = (1 7) Ee '' Prppp. = 8, A₁ = a). Here, it is known that for y < 1, Eng GG] + E.,a da ()r(s a). It follows that Em, [R]] = (1 - (ma)) Bisi et al.
2020 showed that the per-step variance V (R) < Vr, (r(S..A)) which implies that the minimiza-
(1-x)²
15

---

Under review as a conference paper at ICLR 2022
tion of the per-step variance Vx, (r(S,, At)) also minimizes trajectory- V (R). Therefore,
they train a policy тæ by maximizing [x( Exg (r(( =V (r(S, A.)), where A > is a
parameter of the penalty function. Bisi et al. 2020) reveals that (T) B. [R K/ R Ex [ which implies that optimizing (0) is equivalent to optimize the canonical risk-neutral objec- tive of new MDP with the same the original MDP except that the new reward function
(s,, a) = r(s. a) X(r(s, a) (1 ())() This reward function depends on the policy æ, making the reward function nonstationary and conventional RI. method unusable. This problem is issue. To solve this segiic--ependent-reee
called bolicy-dependent-rewvee
issue, the methods of Bisi al. (2020) and Zhang al. (2021b) are based on the trust region policy optimization (Schulman duality (Xie et al., 2018), respectively.
et al. 2015) and coordinate descent with Legendre-
For constants a, 3 >
define the quadratic utility function for per-step reward setting as ner-steep (r(Se. At); a, () ar(S, A.) - (S,,
For an infinite horizon case with y 1, we consider maximizing the following expected cumulative quadratic utility function:
DC
p
Em
per-step (r(S,, Ar): =
arlE
Σ
r(a.. At)
Σ r 2 Ar)
Lt=1
For finite horizon case with 7 = 1, we consider maximizing the following expected cumulative quadratic utility function:
T
T Σ r(S A1) Ar) - BB
E Emp
w perr-ster (r(s,,A)); B)
Σ (S A.)
When applying REINFORCE-bain based algorithm to train an agent by maximizing the objective function, the sample approximation of the gradient, for instance, is given as
E
Σ
w per-step (r)
T
T
r(s A log (S At) Σ (()) log t=1
To log
Att
= a
1=1
We investigate the empirical performance using the same setting as the portfolio management experiment of Section 6.1 Let ( be a/3 in the trajectory EQUMRL, and p be a/3 in the per- step EQUMRL. Under both the trajectory and per-step EQUMRLs, we train policies with the REINFORCE- basse algorithm. The other settings are identical to that in Section 6.1 As Figure shows, under both the trajectory and per-step EQUMRLs, the REINFORCE- al algorithms train MV-efficient policies well for the parameters Ç and p.
C AMERICAN-STYLE OPTION WITH A SYNTHETIC DATASET
An American-style option refers to a contract that we can execute an option right at any time before the maturity time т; that is, a buyer who bought a call option has right to buy the asset with the strike price Keall at any time; a buyer who bought put option has a right to sell the with the strike price Kput at any time.
In the setting of Tamar et al. (2014) and Xie et al. (2018), the buyer simultaneously buy call and put options, which have the strike price Keall 1.5 and Kput 1., respectively. The maturity time is set
20. If the buyer executes the option time t. the buyer obtains reward Tt = ma K Kputt xx) + max(0, It Wean), where Ie is an asset price. We set 20 = and define the stochastic process follows: 2, It-1fa with probability 0.45 and It-1fa with probability 0.55, where fu and fa. These parameters follows Xie et al. (2018).
As well as Section 6. FORCE, Tamar, and Xie. The other settings are also identical to Section We show performances under several hyperparameter in Figure and CRs and Vars of some of their results in the upper table
compare the EQUMRL with policy gradient (EQUM) with the REIN-
16

---

Under review as a conference paper at ICLR 2022
8
trajectory EQUM
1.9
1.8
per-step EQUM
6
p 0.8
4
p= 0.7
AR
2
p 0.3
0.4
0.5
( = 2.5
3
0 0.2
0.1
-=2
0
{ 1.5
-2
0 (=1
0
10
20
30
40
Var
Figure 4: MV efficiency of the portfolio management experiment with the trajectory and per-step EQUMRLs. Higher CRs and lower Vars methods are MV Pareto efficient.
0.355
REINFORCE
1.8 =
1.6
2.2
var
0.350
EQUM
10
0.345
Tamar
{= 1.2
var
REINFORCE
Xie
0,01
{= 0.8
var 0.001
0.340
var 0.0001
Pareto efficient
100.
À 10.
CR
0.335
? 0.6
==
0.330
= 0.1
0.325
<= 0.4
À 0.01
0.320
{ 0.2
0.315
0.080
0.085 0.090 0.095 0.100
0.105
Var
Figure 5:
MV efficiency of the American-style option experiment. Higher CRs and lower Vars methods are MV Pareto efficient.
of Table We also denote MSE between and achieved CR of the lower table of Table 3 From the table,
find that while EQUMRL minimize the MSE for lower the MSE of REINFORCE is smaller for higher Ç. We consider this is because owing to the difficulty of MV control under this setting, naively maximizing the CR minimizes the MSE more than considering the MV trade-off for higher Ç.
D DETAILS OF EXPERIMENTS OF PORTFOLIO MANAGEMENT WITH A LEAL-WORLD DATASET
We use the following portfolio models. An equally- portfolio (EW) weights the financial assets equally (DeMiguel et al. 2009 mean- arean-variance portfolio (MV) computes the optimal variance under mean constraint (Markowitz) 1952). To compute the mean vector and covariance matrix, we use the latest 10 years (120 months) data. A Kelly growth optimal portfolio with ensemble learning (EGO) is proposed by Shen al. 2019). We set the number of resamples as
17

---

Under review as a conference paper at ICLR 2022
Table 3: The results of American-style option. The upper table shows the CRs and Vars over 100 000 trials. The lower table shows The lower table shows the MSEs for Ç over 100, 000 trials.
REINFORCE
EQUM
Tamar
Xie
(< = 20)
- 1,6
(= 0.8
<= 0.4
V =
V 0.1
À = 100
X = 0.1
CR Var
0.353
0.352
0.341
0.322
0.352
0.349
0.335
0.333
0.099
0.098
0.092
0.083
0.102
0.096
0.089
0.088
RHINFORCE
HQUM
Target Value
(C = oc)
<= 2.2
1.8
= 1.4
1.0
= 0.6
MSE from (= 2.2
3.512
3.512
3.517
3.523
3.532
3.572
< < 1.8
2.194
2.195
2.197
2.203
2.209
2.239
< = 1.4
1.197
1.197
1.198
1.202
1.206
1.226
Ç = 1.0
0,520
0.520
0.519
0.522
0.523
0.532
< 0.6
0.162
0.163
0.160
0.161
0.160
0.159
my = 50, the size of each resample m2 = 5r, the number of periods of return data T = 60, the number of resampled subsets mg 50, and the size of each subset 14 n2.77 where m is number of assets; that is, m = 25 in FF25, m 488 in FF48 and m = 100 in FF100. Portfolio blending via Thompson sampling (BLD) proposed by [Shen Wang (016). We use the latest 10 years (120 months) data to compute for the sample covariance matrix and blending parameters. Denote a method proposed by Tamar et al. (2012) by Tamar, and choose the parameter var from (15, 30, 60}. Denote a method proposed by Xie et al. 2018) by Xie, and choose the parameter from (10, 100, 1000]. Denote the REINFORCE trajectory EQUMRI. by EQUMRI, and choose the parameter Ç from (0.5, 0.75, 1.5]. To train the policies of MVRL methods, we assume the stationarity on time-series. Given historical datasets (to.:) (t....... for each trajectory, we simulate portfolio management and generate ((St, r(S,, A[)) Recall that state is given as St = ((Va,t-1,... to 12, 1a.t-1)oEA E Vs), and the portfolio weight as given as Wa,: = (a, 3z) (see Section 6.2p. Then, using the trajectory observations, we train the policies. We
show the pseudo-code of the modified REINFORCE-bass trajectory EQMRL in Algorithm ?
Algorithm 2 REINFORCE- EQUMRL in Section
Historical dataset (Ba,))t((L... (1.....)),,. (L.......
Initialize the policy parameter tu:
for 1: = 1,2,... do Generate {(S) Al, r(S,, Af)) on To by simulating the portfolio management using the historical dataset (Va,))EE (I..... (e(()))ee...))... m).
Update policy parameters 0x+1 + 0% + TT VESO, [u trajecttoyy B)].
end for
Table shows the performance of each portfolio model without the penalty of the turnover A 0. We report both results with and without the penalty of the turnover in the following. The average of real objective (minimizing variance with a penalty
return targeting) for Tamar, MVP and EQUMRL from July 2000 to June 2020 is shown in Table 5 We also divide the performance period into two for robustness checks. Table 6 and 7 shows the first-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020. In almost all cases, the EQUMRL achieves the highest R/R. We plot results of Tamar, Xie, and EQUMRI. with various parameters in Figure 6 We choose the parameter of Tamar (V) from {10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65 Xie (A) from {10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000}. and EQUMRL (C) from (0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1, 1.15,, 1.2,,,,.,,,13,1 1.375, 1.5). We only annotate the points of V = 10, 23, 40, 55, X = 10, 200, 500, 500, and ( = 0.125, 0.5. 0.875. 1.25. Unlike Section 6. is not easy to control the mean and variance owing to the difficulty predicting the real-world financial markets. However, the EQUMRL tends to return more efficient results than the other methods.
18

---

Under review as a conference paper at ICLR 2022
Table 4: The performance of each portfolio model without the penalty of the turnover A 0 during the out-of-sample period (from July 2000 to June 2020) for FF25 dataset (upper table), FF48 (middle table), and FF100 (lower table). For each dataset, the best performance is highlighted in bold.
Method
EW
MV
EGO
BLD
Tamar
30
00
A
X
100
1001
0.5
EQUM
0.75
FF23
CRT
0.80
0.11
0.77
0.56
1.16
100
0.96
0.00
1.12
Vanu
28.02
53.58
30.83
12.01
18.41
15.10
25.22
24.94
14.98
18.37
11.37
21.29
16.25
R/RT
0.89
0.67
0.87
MaxDD1
0.54
0.75
0.57
0.37
0:35
0.35
0.55
046
0.34
0.49
0.33
0.30
1148
CRT
0.15
1.04
0.52
0.63
1.06
0.52
0.34
0.35
1.49
1.01
Var.
22.91
9.85
40:13
30.96
RRRT
0.30
0.06
0.86
0.58
0349
0.35
0386
0.3
0.46
047
0.82
0.30
0.87
MaxDD]
0.30
0.48
0.31
0.18
0.32
0.19
0.21
0.20
0.17
0.17
0.20
0.27
0.28
CRT
0.81
0.14
0.86
0.53
1.00
1.86
1.36
1.33
1.05
1.07
1.35
1.30
1.58
Vari
29.56
57.90
32,40
11.79
31.61
85.74
20.05
43.65
2034
87.24
20.62
16.56
27.50
030
MaxiDD1
0.35
0.46
0.34
0.22
0.29
0.57
0.22
0.36
0.23
0.51
0.23
0.20
0.28
Table 5: The average of real objective (minimizing variance with penalty on return targeting) for Tamar, MVP and EQUMRL for FF25 dataset (upper panel). FF48 (middle panel) and FF100 (lower panel) from July 2020 to June 2020.
Tamar
MVP
EQUM
V
15
V
30
V
60
À
= 10
T00
1000
< 0.5
0.75
L.5
With Turnover
Penalty A
0.001
FF25
-5,466
-7,504
-7,646
-6,227
-6,398
-7,769
-7,853
-8,758
-9,027
FF48
-5,395
-7,399
-7,614
-6,647
-7,763
-7,677
-9,000
-8,436
-7,743
FF100
-6,711
-7,491
-6,233
-6,362
-5,354
-6,983
-8,681
-7,366
-8,362
Without Turnover Penalty
FF25
-5,312
-7,486
-1,329
66,130
-6.390
-7,340
-7,664
-7,660
-8,91
FF48
-5,464
-7,053
-7,505
-6,489
-9,763
-7,506
-8,729
-8,456
-7,421
FF100
-6,749
-7,287
-5,915
-6,047
-5,446
-7,008
-9,484
-7,306
-8,280
Table The performance each portfolio without turnover penalty (Larbbda = 0) during first half out-of-sample period (from July 2000 to June 2010) and second half out-of-sample period (from July 2010 to June 2020) for FF25 dataset (upper panel). FF48 (middle panel) and FF100 (lower panel). Among the comparisons of the various portfolios, the best performance within each dataset is highlighted in bold.
PF25
EW
MV
BGO
BLD
15
Jamar
30
60
10
X
100
1000
<
0.5
EQUM
0.75
1.5
Firs-Half Pertod
July
June
2010)
CRT
0.58
-042
0.64
0.4
1.51
1.18
0.96
1.15
Vara
3121
69.36
33.29
12.50
11.77
11.31
11.8
21.38
10.99
11.50
9.96
22.79
17.10
RRT
0.36
4.17
0.38
0.41
1.23
1.35
1.1
0.72
1.38
1.18
1,09
0.97
1,02
MaxDD]
0.16
Second-Ham Half Perind
(fro July
2000 Jane 2010)
CRT
100
0.6
1,83
0..00
0158
1841
0336
0.70
066
0.69
0.00
0.81
RORT
0.59
0.36
0.57
0.72
0.72
0.55
0.24
0.62
0.56
0.45
0.60
0.70
0.72
MaxDD]
0.51
0.50
0.51
0.22
0:35
0.35
0:55
0.46
0.34
0.49
0.27
0.32
0.30
Tamar
V
1000
0.55
0.75
First-Half Pertod férom July
2000 June 2010)
CRT
0.60
0.41
0.75
0.34
01.8
1.35
1.03
40.25
40.26
1.25
1.72
1.09
Viri
25.85
10.97
41.06
14.46
13,85
7/07
0,02
11:15
27.16
RIRT
0.1
0.19
0.41
0.35
0.35
(1.2
1.26
1.34
-6.46
4.30
1.28
1.15
0.88
MaxDDI
0.27
0.33
0.31
0.18
0.30
0.17
0.17
0.17
0.00
0.00
0.16
0.27
0.18
Second
June
2010)
CR1
1.02
2.12
1.33
0.69
0.23
0.59
1527
0.0
0.94
0.7
0.61
1.27
0.92
Viri
19.88
96.14
24.24
8.27
38.29
17.16
22.13
1600
12.72
12.63
18.87
34.66
36.24
0.99
0.53
MaxlDD]
0.27
0.48
0.27
0.18
0.28
0.19
0.17
0.17
0.17
0.20
0.26
0.28
FF100
EW
MV
EGO
BLD
Tamar
MVP
RQUM
Fins--Hll
Periell (from July
2000 June 2010)
CRT
0.1
20.77
0.73
(121
1.18
1.88
1.67
1.42
1.20
1073
1.4
1.30
1.03
Vari
32.02
79.35
35.11
12,43
19.67
109.44
11.33
10.30
13.16
92.30
17.16
12.80
21.45
0.62
0.77
MaxiDD
0.28
0.46
0.30
0.17
0.22
0.52
0.18
0.30
0.15
0.51
0.20
0.17
0.23
Second-Haal Hall Period (from July
2000 June 2010)
0.86
0.53
Vari
26.61
35.92
29.64
11.12
43.56
62.05
23.58
46.48
28.61
81445
24.07
20.25
$2.94
R/R1
0.68
0.38
0.63
0.68
0.36
0881
0.73
0.63
0552
0.54
0.90
0.84
0.50
0222
0.27
19

---

Under review as a conference paper at ICLR 2022
Table 7: The performance of each portfolio with turnover penalty (A = 0.001) during first half out-of-sample period (from July 2000 to June 2010) and second half out-of-sample period (from July 2010 to June 2020) for FF25 dataset (upper panel) FF48 (middle panel) and FF100 (lower panel). Among the comparisons of the various portfolios, the best performance within each dataset is highlighted in bold.
FP25
EW
MV
BGO
BLD
V=
2.5
Jamar
80
Xie
100
1000
0.5
EOUM
0.75
1.
Pertodtrrom
July
2010)
CRT
0.58
4043
0.64
0.56
1.32
1.31
1.19
0.93
1.17
1.17
1.5W
1.33
1.54
Vari
31.21
09.33
13.29
25788
14.66
11.8
1131
11.38
11.93
1135
16.47
11.18
9.96
1.20
1.17
1.35
MaxDD1
0.54
0.75
0.58
0.52
0.21
0.15
0.19
0.20
0.17
0.17
0.26
0.13
0.10
Second-Ha Halt Perindtfrom July 1000 June 2010)
0.87
Vari
25.92
$7.28
27.95
19009
22.54
18.72
38.46
39.97
39.61
39.51
32,08
18.67
13.31
R/RT
0.09
0.34
0.05
0.89
0.71
0.56
0.23
0.48
0.62
0.62
0.90
0.76
0.00
MaxDD1
FR488
EW
MV
B00
BLD
V=
2.5
Tamar
30
A= D0
100
A= 1030
= 0.5
C 0.75
EQUM
First Half
Pertodifomm July
2000
2010)
1.36
0.38
0.82
1.35
Varl
25.85
57.26
39.39
18.09
41:27
14.49
13.70
18.51
17.61
19.28
29.97
20.19
6.04
RTR
0141
0.16
0.38
0.30
1373
0.37
174
0.47
1.00
0.54
0.35
0.51
1.92
MaxDD1
0.19
0.25
Second Hall
Periodifrom
Tuly
2000 June 2010)
CRT
1.02
4..1
1.25
1.02
0000
0.36
0.52
0.45
0.94
1.50
1.86
0.92
0.45
24.28
14.22
0.79
0.05
0.88
0.98
0.29
0.35
0.25
0.59
0.72
1.11
0.27
0.41
Mox
0.27
0.18
0.28
0.22
0.28
0.20
0.21
0.27
0.25
0.26
0.27
0.20
0.16
800
BLD
Tamar
MVP
EQUM
Pirst
Pertodtrrm July 2000 to June 2010)
CRT
0161
414
0.67
0150
117
038
1.23
0.24
1.26
1.30
1.19
32.02
35.06
21255
13.2
K7KT
0.37
0.16
0.39
0.4
0.99
0.91
122
0.09
1.09
0.90
1.12
1.34
1.01
MaxDi)
0.28
0.46
0.30
0.25
0.17
0.18
0.18
0.51
0.21
0.21
0.17
0.15
0.22
Second-Hall
Perood July
CRT
1.01
0.63
0.95
1.12
1.13
0.50
0.1
0.90
0.70
1.09
0.70
0.72
1.64
Varl
26.61
35.96
29.61
19227
36.80
26.69
20.99
82.15
11.40
53.59
17.46
19.26
48.15
R/R
0.06
0.34
0:38
0.52
0.58
Maxt
0.23
1.6
T2-120
1.6
COUN
1.6
POJM
Pasts
1.4
1.4
3301
1.2
+2-20
1.0
Pavee atfave
15
0.3
0.6
Pareoo Pertoettieer officee
0.8
0.4
0.8
0.2
0.6
20
40
60
80
100
Var
Figure MV efficiency of the portfolio management experiment. Higher CRs and lower Vars methods are MV Pareto efficient. Left graph: FF25. Center graph: FF48. Right graph: FF100.
1.6
Tamar
1.6
Tamer
1.4
Xxc
X
Paneto efficient
Parato efficient
1.2
Pareto efficient
1.4
5 1.0
ei-e3
5
1.0
cens
0.8
0.4
0.8
0.2
0.6
40
50
30
40
50
80
100
Var
Var
Var
Figure 7: MV efficiency of the portfolio management experiment. Higher CRs and lower Vars methods are MV Pareto efficient. Left graph: FF25. Center graph: FF48, Right graph: FF100.
20


Reviews:
[{"review_id": "vAvN2m5SB-s", "reviewer": "Reviewer_FXvX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe method in this paper is intuitive, and it builds a connection between utility optimization, a popular topic in finance, and reinforcement learning. The proposed method can mitigate the double sampling issue that appeared in prior work, and can be combined with any existing policy-gradient-based algorithms. The experimental results show that the algorithm has better performance compared to baselines.\n\nWeakness:\nI think there are some weakness in the experimental section. In Section 6.2, the portfolio management experiment using real-world data does not seem to be an RL problem. The dataset is static, which contains the returns of each asset at each time. This return does not change when the agent's action (weight w) changes. The policy network is essentially a \"classifier\" that maps the input feature vector (returns of assets) to a probability vector. Thus, the problem is more like an online learning (or supervised laerning) problem rather than an RL problem.\n\nI agree that for this problem, it is still useful to better optimize the mean-variance trade-off, but I think the authors should clarify this better to avoid confusion. Moreover, since the authors propose this algorithm as an RL algorithm, it would be interesting to test the performance of the algorithm on a more realistic RL benchmark.\n\nMinor:\nSection 2.1: “A typical method for consider the trade-off is train a policy under some constrains”: this sentence has grammar mistake and typo.\nSection 6.2: “The policy determines e the weight.”: typo.\n", "summary_of_the_paper": "This paper proposes a method for mean-variance trade-off optimization in RL. The method, named EQUMRL, tries to optimize the Pareto efficiency by maximizing the quadratic utility function. The proposed method mitigates the double sampling issue appeared in prior work thus simplifies the optimization procedure. The authors also provided experiments to demonstrate the benefits of the algorithm using both synthetic datasets and real world data.", "summary_of_the_review": "The proposed algorithm is interesting and has solid contribution. The experiment section is not satisfactory enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qduzJbSXzWK", "reviewer": "Reviewer_MBfd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow in general. The survey in the appendix is helpful for understanding concepts and motivations from a finance and economics perspective. The authors provide detailed discussion of the related literature and comparison of the present work with existing ones. They also present ample experimental results under various settings (related to investment) to justify the efficacy of the algorithms.\n\nQuestions:\n- pp4, line 7, E[G] + \\lambda (...) should be E[G] - \\lambda (...)?\n- pp5, line 4, \\hat{G}^k should approximate E[G]?\n- pp5, the displayed equation seems to imply that (\\hat{G}^k)^2 is an unbiased estimateor of E[G^2]. Can the authors provide intuition of why this is the case?\n- I am curious of how the actor-critic implementation would fare against the REINFORCE-based method in experiments.\n\n\n---- Post rebuttal ----\n\nI appreciate the authors' response. I have read through other reviewers' comments and authors' rebuttal. I would like to maintain my score.", "summary_of_the_paper": "The paper studies the reinforcement learning problem with mean-variance tradeoff. It investigates a quadratic formulation that aims to optimize for the mean-variance efficiency, in a sense similar to the Pareto efficiency. It establishes connections between the quadratic formulation with the standard mean-variance formulation studied in the literature, and provides various interpretations of the mean-variance efficiency. The paper also proposes novel algorithmic implementations of the quadratic formulation based on REINFORCE and actor-critic methods. Three investment-related experiments are performed for the REINFORCE-based algorithm and demonstrate its good performance compared with existing algorithms.", "summary_of_the_review": "The paper considers an important problem and its proposed algorithm is justified by extensive experiment results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ipPw1u0duTu", "reviewer": "Reviewer_fgBH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is nicely written with a clear motivation, i.e., to resolve the computational/sampling issues for traditional methods that aim to optimize the mean-variance objective.\n- The proposed expected quadratic utility objective is easy to work with. \n- The interpretation of the proposed objective is well-explained. \n\nWeaknesses/Questions:\n- Is it possible to provide some quantifiable gap between the optimal policy under the proposed objective and the optimal policy under the original mean-variance objective?\n- In the experiments, it seems that only the mean performances are shown. What are the standard deviations for the performances of the methods? \n- side points: (1) I am not sure how the traditional MVRL objectives can be rephrased as the constraint optimization problem with *equality* constraint. Could the authors provide more details on that? (2) The \\nabla_{REINFORCE} notation seems very unconventional.\n\n", "summary_of_the_paper": "The paper proposed an alternative objective (expected quadratic utility) to the mean-variance objective in an episodic RL setting. The new objective does not involve the squared expectation term, thus resolving the double sampling issue for obtaining the gradient of the mean-variance objective. In addition, the new objective has a couple of nice interpretations including the Pareto-efficiency one.  ", "summary_of_the_review": "The paper was nicely written and provided a simple alternative (the expected quadratic utility function) to the mean-variance objective, which has nice interpretations and avoids the double sampling issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "6udkGIiBRyY", "reviewer": "Reviewer_TnBm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: Authors use well known results from mathematical finance to improve state-of-the-art for portfolio optimization using reinforcement learning. The paper is also well written and the flow is clear.\n\nWeakness: I have concerns on results presented for both the examples illustrated in the paper. In addition, authors need to improve the writing and update literature review to provide better context of the contribution of this work. Please see below for detailed comments:\n\n*Contribution/Results:*\n1. It is not clear why the proposed algorithm works better than the Tamar et. al. and Xie et. al. Author’s state: “We conjecture that this is because while the EQUMRL is an end-to-end optimization for obtaining an efficient agent, the other methods consist of several steps for solving the constrained optimization, where those multiple steps can be sources of the suboptimal result.”. This provides no concrete explanation or evidence on the difference.\n2. Results in Figure 2 are somewhat concerning. It is not clear why target variance is much higher than the observed variance (e.g., blue square with “var=50” has an observed variance of 33). Since returns increase with increasing variance, I would expect the constraint to be binding.\n3. For the real world dataset, it is not clear how the authors have trained the model. Since there is only one historical trajectory of each stock in the portfolio, it is not obvious how the training works. It will be helpful to provide some details on model training in this context.\n\n*Writing:*\n1. It will be helpful if authors could clearly set up the portfolio optimization problem in Section 2. The current description is very generic. As an example, they do not describe what is the action space (e.g., portfolio weights or actual dollar investments), the state space (returns or stock prices) or the rewards for the problem considered in the paper. It only starts to become somewhat clear when the reader is in Section 6. There are a lot of different ways one can set up the portfolio optimization problem, so providing clarity on it is important.\n2. Authors do not describe the sequence of steps required for implementing the algorithm. The practitioners will find the paper far more useful if at least it could provide the complete algorithm at one place.\n3. The discussion on per-step variance perspective has no corresponding results. Authors should probably consider moving it as a discussion topic at the end or provide additional results to support the setup.\n\n*Literature:*\nThere is a rich literature (over 20 years of work) on portfolio optimization in mathematical finance using stochastic control which also solves for dynamic decision problems and is closely related to the work done by the authors. Providing some context of how this work relates to that domain would be helpful.\n\n*Typos:*\n1. Section 6, first line “variacen” → “variance”\n2. Page 1, 4th last line - should “Pareto efficient policy” be “pareto efficient frontier”?\n", "summary_of_the_paper": "In this paper, authors propose a reinforcement learning approach to solve mean-variance portfolio optimization problem. The main idea of the paper relies on transforming mean variance optimization to maximizing expected quadratic utility. This allows the authors to avoid inefficient approximation of the gradient of the variance and instead use the same sample paths to approximate the gradient of the first and second moments of the reward. They implement two algorithms, first based on REINFORCE and another based on Actor-Critic method. Authors also illustrate the performance of the method on two problems, first based on synthetic dataset and another based on Fama & French dataset.", "summary_of_the_review": "My recommendation is to reject because the results require more justification on the performance of the method relative to state-of-the-art. The work does not provide any novel perspective over what is already known to the financial mathematics or reinforcement learning community. Authors need to provide more clarity in writing, In particular, in formulating the optimization problem and how the method is implemented.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1lVVzEa_qZi", "reviewer": "Reviewer_wFpo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. The studied mean-variance problem is very interesting and well-motivated.\n2. This paper conducts extensive experimental results and demonstrate the empirical performance superiority of the proposed EQUMRL.\n3. The authors provide code, which is helpful for readers to reproduce their results.\n\nWeakness:\n\nThis paper lacks theoretical analysis. In particular, what regret/mean-covariance/Pareto efficiency guarantee can the proposed EQUMRL achieve? I expect to see formal theorems and rigorous proofs.\n\n\n----After Rebuttal----\n\nI read the authors' response. In general, I think that the weakness of this paper falls on theoretical analysis. Although the authors explained that the regret analysis in RL is an open problem, to my best knowledge, there are a number of papers that study regret bounds for (online) RL under tabular or linear MDP settings in recent years. \n\nFor example,\n\nAzar, M. G., Osband, I. and Munos, R. (2017). Minimax regret bounds for reinforcement\nlearning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70.\nJMLR. org.\n\nJin, C., Allen-Zhu, Z., Bubeck, S. and Jordan, M. I. (2018). Is Q-learning provably efficient?\nIn Advances in Neural Information Processing Systems.\n\nJin, C., Yang, Z., Wang, Z. and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory.\n\nI more or less understand that it is hard to analyze a policy gradient based RL algorithm with sophisticated parametric models like neural networks. But the authors' response did not show me sufficient contributions of this paper.\n\nIn my opinion, this paper lacks solid theoretical analysis, and its experiment part looks standard (does not provide empirical evaluations under sufficiently comprehensive and challenging experimental benchmarks/setups). I did not find enough reasons to recommend accept.\nSo I plan to stick to my score 5.", "summary_of_the_paper": "This paper learns MV-efficient policies that achieve Pareto efficiency in terms of MV trade-off. The authors propose an approach called expected quadratic utility maximization (EQUMRL), which trains an agent to maximize the expected quadratic utility function. The proposed EQUMRL does not suffer from the computational difficulties. Experimental results are provided to demonstrate the effectiveness of the proposed EQUMRL.", "summary_of_the_review": "Overall, I think that the studied mean-covariance problem in this paper is very interesting. The idea behind the proposed approach looks non-trivial. Extensive experimental results are provided to show the superior empirical performance. However, this paper lacks theoretical analysis, and thus I could not judge whether the proposed approach achieves good theoretical results as well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper with some reviewers voted for acceptance and some think it is not still ready. What is clear is more efforts by the authors is needed to make the paper appealing to reviewers with different interests. Changes such as better writing, more in depth literature review, more convincing experiments can definitely improve the quality of the paper. I personally do not think regret analysis is needed for this work, but it was mentioned by a reviewer. I would suggest the authors to use the reviewers' comments, revise their work, and prepare it for future conferences.
Decision: Reject