Policy Gradient with Expected Quadratic Utility Maximization: A New Mean-Variance Approach in Reinforcement Learning
Authors: Masahiro Kato
Kei Nakagawa
Venue:  (2021)
Keywords: mean-variance reinforcement learning
finance

Abstract: In real-world decision-making problems, risk management is critical. Among various risk management approaches, the mean-variance criterion is one of the most widely used in practice. In this paper, we suggest expected quadratic utility maximization (EQUM) as a new framework for policy gradient style reinforcement learning (RL) algorithms with mean-variance control. The quadratic utility function is a common objective of risk management in finance and economics. The proposed EQUM framework has several interpretations, such as reward-constrained variance minimization and regularization, as well as agent utility maximization. In addition, the computation of the EQUM framework is easier than that of existing mean-variance RL methods, which require double sampling. In experiments, we demonstrate the effectiveness of the proposed framework in benchmark setting of RL and financial data.

Under review as a conference paper at ICLR 2021
POLICY GRADIENT WITH EXPECTED QUADRATIC UTILITY MAXIMIZATION: A NEW APPROACH IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In real-world decision decision-making problems, risk management is critical. Among var- ious risk management approaches, the vaaiinae criterion is one of the most widely used practice. In this paper, we suggest expected quadratic utility max- imization (EQUM) as new framework for policy gradient style reinforcement learning (RL) algorithms with mcan-variance control. The quadratic utility func- tion is a common objective of risk management in finance and economics. The proposed EQUM has several interpretations, such as reward-constran conttrrnnd variance minimization and regularization, as well as agent utility maximization. In addi- tion, the computation of the EQUM is easier than that of existing mean-variance RL methods, which require double sampling. In experiments, we demonstrate the effectiveness of the EQUM in benchmark setting of RL and financial data.
INTRODUCTION
Reinforcement learning (RL) with Markov decision processes (MDPs) is type of dynamic decision-making problem (Puterman, 1994; Sutton & Barto, 1998). While the typical objec- tive is the expected cumulative reward maximization, risk-aware decision-making has attracted great attention in real-world applications, such as finance and robotics (Geibel & Wysotzki. 2005; Gareía & Fernández, 2015). The notion of risk is related the fact that even an optimal pol- icy may perform poorly owing to the stochastic nature of the problem. To capture the risk, vari- ous criteria have been proposed, such as Value at Risk (Luenberger, 1998; Chow & Ghavamzadeh, 2014; Chow et al., 2017) and variance (Markowitz, 1952; Markowitz et al., 2000; Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013). Among them, we focus on the mean-variance trade-off.
Typical mean-variance RL (MVRL) methods attempt to maximize the expected cumulative reward while maintaining the variance threshold (Tamar et al., 2012; Prashanth Ghavamzadeh, 2013; 2016; Xie et al., 2018; Bisi et al., 2020; Zhang et al., 2020). However, most existing MVRL meth- ods suffer from high computational costs owing to the double sampling issue when approximating the gradient of the variance term (Tamar et al., 2012: Prashanth & Ghavamzadeh, 2013; 2016). To avoid the double sampling issue, Xie et al. (2018) proposed a method based on the Segendre-Fenchte duality (Boyd& & Vandenberghe, 2004). Although the method does not suffer from the double sam- pling issue, we cannot apply a standard policy gradient method and must use a coordinate descent algorithm. In addition, the method cannot control risk certain desirable level.
From an economics perspective, the difference in the RL objectives arises from the forms of utility functions. When the objective of an agent is expected cumulative reward maximization, the utility function is risk-neutral; when an agent attempts to control the risk based on an expected reward, the utility function is risk-averse (Mas-Colell et al., 1995). In economics, there have been several risk-averse utility functions proposed. The quadratic utility function is one such functions and is frequently used in financial economics (Luenberger, 1998). Under the quadratic utility function, the mean-variance portfolio maximizes the utility of the investor. In addition, other various finan- cial theories are also based on the quadratic utility maximization (Markowitz, 1952; Sharpe, 1964; Lintner, 1965; Mossin, 1966). For more details, Appendix A. In this study, one of the MVRL approaches, we consider the expected quadratic utility maximization (EQUM) based on the policy gradient method (Williams, 1988; 1992; Sutton al., 1999; Baxter & Bartlett, 2001).

---

Under review as a conference paper at ICLR 2021
The EQUM has the following advantages: (i) low computational cost; (ii) numerous interpretations, and (iii) direct connections to real-world applications. In this study, as interpretations of EQUM, we propose the minimization of the variance under the constraint of the expected cumulative reward, reward-targeting optimization, and regularization. Thus, this study contributes to the context risk- averse RI and MVRL. In the following sections, we first formulate the problem setting in Section 2 and review the existing methods in Section 3. Then, we propose the main algorithms in Section 4. Finally, we investigate the empirical effectiveness of the EQUM in Section 5.
2 PROBLEM SETTING
We consider the standard RL framework, where a learning agent interacts with an unfamiliar, dy- namic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time. We define the MDP as the tuple (S,A, r, P. Po), where S is set of states, A is set of actions, r 8 x A + R is a reward function, P x S x A + [0, 1] is the transition kernel, and Po 8 + [0, is an initial state distribution. The initial state Si is sampled from Po- Let To A x S - [0, 1] be a parameterized stochastic policy mapping states to actions, where is the tunable parameter. At time step t. an agent chooses action A, according to a policy TT(: (-) St).
We assume that the policy mh is a differentiable function with respect to 0; that is, @ma(a.s) exists. 20
There are several performance measures for a policy Tg. One popular measure is the expected cu- mulative reward from time step l to u defined as [Re]) where Res. = - 7'r(Si4s; Ar=:), y (0, 1) is a discount factor and E.,, denotes the expectation operator over policy Ty. and S₁ is generated from Po- When y = 1, to ensure the cumulative reward well defined, it is usually assumed that all policies are proper (Bertsekas & Tsitsiklis, 1996); that is, for any policy no. the agent goes to a recurrent state S with probability 1. After the agent passes the recurrent state S at time т, the rewards are always O. Such finite horizon case is called episodic MDPs (Puterman, 1994). For brevity, we denote Res. as R when the meaning is obvious. Under these criteria, the agent may attempt to obtain higher cumulative reward while taking higher risks. In real-world applications, such as portfolio management in finance (Markowitz, 1952; Markowitz et al., 2000), such risky decision-makin the expected cumulative reward and the variance. Thus, while the goal of the risk-neutral MDP problem is to find the parameter 0 that maximizes the total reward, we consider the mean-variane trade-off between the cumulative expected reward and the variance of the cumulative reward R in the MVRL problem. Note that even when the reward r is deterministic, the cumulative reward is a random variable owing to the stochastic policy, and there exists the mean- variance trade-off. In ad- dition, even if the optimal policy is deterministic, the proposed method empirically has the potential to improves the stability of the training from the observations in the experiments of Section 5.3.
is not always desired, and we, therefore, consider the trade-off between
EXISTING MVRL METHODS
In this section, we introduce existing studies of MVRI..
3.1 CONSTRAINED TRAJECTORY- TRAJECTORY-VARIANCE VARIANCE PROBLEM
Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) formulated MVRI. by a constrained optimization problem defined as maxage [R]] s.t. Var,, (R) < n In their formulation, the goal is to maximize the expected cumulative reward with control- ling the trajectory-variane Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) consider penalized method defined as maxage Ex, [R] - 5g Var=. (R) - v)), where 5 > 0 is a constant and IR + R is a penalty function, such as g(x) = x or g = x2
at a certain level. To solve this problem, Tamar et al. (2012),
3.1.1 DOUBLE SAMPLING ISSUE
Tamar al. (2012), Prashanth & Ghavamzadeh (2013), and Xie al. (2018) report the double sam- pling issue in MVRL, which requires sampling from two different trajectories for estimating the policy gradient. For instance, in an episodic MDP with the discount factor 7 1 and the stop-
2

---

Under review as a conference paper at ICLR 2021
ping time т min(l ( S. S. the gradients of Brg RJ. ET [R- and (En, (R)) given
as Vo R T. R Ei= Vo log (St. At)), Vo [ = E R2 Ei= Vo log To A:)]. and (E, [R))
n [R] Vellso (Tamar et al., 2012). Besides, the gradient of the vari- Because optimizing the policy ve using the gradients directly is computationally intractable, we replace them with their unbiased estimators. Suppose that there is a simulator generating a trajectory k with (IS%AA4 r(S A))) where 7K is the stopping time of the trajectory. Then, we can naively AA)
ance is given as ETg, [n² Σو= Ve log ( At)] - 2 | VolEort
construct unbiased estimators of E,, [R] and ETo [²] as VO = Rd Ei= Volog To (o(
and VO [R_] = (T*) ΣE= Vo log T A/), where No a sample approximation
of B [R] at the episode k. However, obtaining an unbiased estimator of Vo [1))² 21E x, [ v VoLE, . [R] is difficult because it requires sampling from two different trajectories for ap- proximating E [R] and V VgETT [R). This issue makes the optimization problem difficult.
3.1.2 EXISTING SOLUTIONS TO DOUBLE SAMPLING ISSUE
For the double sampling issue, Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) presented the following solutions.
Multi-time-scale stochastic optimization: The proposed methods of Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are based on stochastic approximation to find an equilibrium point of an ordinary differential equation and saddle point of the objective, respectively.
Coordinate descent optimization: Xie et al. (2018) proposed using the Legendre-Fencher dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, Xie et al. (2018) set objective function maxsee Ex, [R] (Var x, (R) 7). Then, Xie et al. (2018) transformed the objective function as ma 2y [E., R + 26 y² --- Er [ and estimated a parameter by solving the optimization problem via coordinate descent algorithm.
Weakness of existing approaches: The multi-time-scale approaches by Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are known to be sensitive to the choice of the step-size schedules. which are easy to be controlled (Xie al., 2018). The method by Xie al. (2018) does not reflect the constraint condition 7) shown above; that is, in the objective function of Xie et al. (2018), there exits penalty coefficient 6, but does not exist the constraint condition 1). Note that the problem of Xie et al. (2018) is owing to their objective function based on the penalty function g(x) = x:: E,,, Rt) (Var,, , (R) - n)). which the first derivative does not include r7. In addition,
when using quadratic function as g(x) x² to consider 7), we cannot remove [B[A²)] even with Legendre-FFenche dual; that is, the method also suffers the double sampling issue.
3.2 CONSTRAINED PER-STEP VARIANCE PROBLEM
Bisi et al. (2020) and Zhang et al. (2020) proposed solving a constrained per-step variance prob- lem for MVRL. Bisi al. (2020) showed that the per-step variance Var (R) < Var,, (r(S,.A)))
(1-7)²
which implies that the minimization of the per-step variance Var(r(S,, At)) also minimizes trajectory-variance Var,, (R). Therefore, they train a policy na by maximizing E,, [r(S,, A,) x varr ((S,A))), where > 0 is parameter of the penalty function. The methods of Bisi et al. (2020) and Zhang et al. (2020) are based on the trust region policy optimization (Schulman al., 2015) and coordinate descent with Legendre- Fenchel duality (Xie et al., 2018), respectively.
3.3 CONSTRAINED CUMULATIVE EXPECTED REWARD PROBLEM
While existing MVRL studies mainly focus constrained rujectory-varianee problem, strained cumulative expected reward optimization is also frequently used in practical situations, such finance (Markowitz, 1952; Markowitz et al., 2000). In the constrained cumulative expected
3

---

Under review as a conference paper at ICLR 2021
reward problem, solve the following problem:
min Varz, (R) s.t. E,,, [2] =
6EB
Our proposed EQUM framework is based on this motivation; that is, variance minimization. As shown in the following section, from computational perspective, there is critical difference be- tween the constrained cumulative expected reward and trajectory-variance problems.
4 EQUM FRAMEWORK
In this paper, as novel approach for MVRL, we propose a Expected Quadratic Utility maximization RL (EQUM). In economic model, by using two parameters function for the cumulative return R is defined as (R; a, 3) = aR 3R for a > 0, 2 (Luenberger, 1998). The quadratic utility function captures the preference of risk-averse agent over the cumulative return R. Let us consider the expected quadratic utility function defined as
0 and B 0, the quadratic utility
Evo IU a, B)] alE [8
B RRa
alBaa [R]
3 (En, [R)
- (E.. R [R)=
R)-].
In the EQUM framework, we train policy by maximizing the expected quadratic utility function.
4.1 INTERPRETATIONS
Here, we introduce four interpretations EQUM. We can interpret the EQUM as an approach for (i) expected utility maximization, (ii) a targeting optimization problem to achieve an expected cu- mulative reward ç, (iii) a constrained trajectory- problem with a quadratic penalized function, and (iv) expected cumulative reward maximization with regularization.
First, we discuss the connection with respect to training an agent to achieve a predefined return (Berger, 1985). Let Ç be target return that the algorithm aims achieve. Then, consider the mean squared error (MSE) minimization between the expected deviation of the return and ç:
arg min ((: arg min E, (s- R)² lee
(3)
dE@
We can decompose the MSE into the bias and variance as follows:
(5 R]
Exa
(C BTT [R2)) + 28 (5 R,, [R]) (E,,, [R] Rt)
+ E
(E., [R] R2)
Bias
Variance
= 2 2(E [ + EE [ + Eno (F - RR)
Thus, the minimization problem (3) trains the policy Tp consider the trade-off between the bias
(S - Ex,, [ and variance E. (B.,, [R] R) (Section 9.5 of Luenberger (1998)). Moreover,
we find that the EQUM is equivalent to the reward-targeting optimization when Ç : that is,
arg min J ( 0 8ee
arg min
a
2 Eng [R]
(E,, (R)) + Enp (R [R] R?)*
REB 3
arg max E.g ((R:a,))) #
Second, the bias-variance trade-off heuristically provides a solution to a constraint optimization problem (1) with the constraint& { = _ by solving the following penalized problem:
min Var (R) + (ETo [R] - -))²
(4)
d2

---

Under review as a conference paper at ICLR 2021
Third, regard the quadratic utility function expected cumulative reward maximization with a regularization term defined as EE [ that is, minimization the risk (TT)):
R(0)
ET 118
a B [R2)
(5)
Risk of expected cumulative roward maximization
Regularization term
where á is a regulation parameter and eé 2a
As 0o, R(TO) -ET [R]
26
4.2 MERITS OF THE EQUM FRAMEWORK
In this section, we present two advantages of the EQUM framework. The first advantage com- putation. The EQUM framework is MVRL method. However, compared with existing MVRL methods, which involve the double sampling issue, the computation of the EQUM framework is much simpler because we transform the MVRL problem (4) into the optimization problem without the term (EEr, [ (see (2) and (5)). The second advantage that it provides variety of interpre- tations. Because the EQUM framework can interpret economic theory, it is applicable for modeling economic dynamics. In addition, as one of the MVRL methods, it suitable for various real-world applications, such as finance (Deng et al., 2016) and playing games. By regarding the proposed EQUM as a regularization framework, we can combine with existing RL methods.
4.3 IMPLEMENTATIONS OF THE EQUM FRAMEWORK
Here, we discuss the implementations of the EQUM framework.
Simplest policy gradient with EQUM With the EQUM framework, we introduce a main algo- rithm with simplest policy gradient (SPG) algorithm (Brockman al., 2016), which is also called the REINFORCE algorithm (Williams, 1992). For an episode l with length n, the method replaces the expectations Era [R] and Ex,,, [ with the sample approximations E 1 Ar) and A.)) respectively (Brockman et al., 2016). Then, the unbiased gradients are R
PoET. [
Ve log (S) At) and VoEn. [R²]]
12t
Vo log no (S At).
Therefore, for a sample approximation Reth of lE [1 at the episode k, we optimize the policy with
ascending the following gradient: (A - 1 Rd 2
En= Pologro(S; At).
For another combination with the EQUM framework, apply actor-critic (AC) based algorithms, which is also refereed to as the advantage actor-critic (A2C) algorithm (Williams & Peng, 1991; Mnih et al., 2016). Extending the AC algorithm, for episode with the length n, we train the policy by a gradient defined as
Actor-critic with EQUM:
)((aM)))))
(M ai (S 3M (2) (5t) (st)) 2(22)
1
V log To (S,
a
where nre - Pek
b +2 M() (Stm)) and k Rat+n11
A R + M() (S
2(00 8tn
and (1) (00) (8)) and M22)
(,) (SP) are value functions approximating B B[[e+I:00] and E. R2 with
parameters (1) and at respectively.
(2)
For other RL algorithms, we can heuristically extend our proposed framework EQUM to accept the other RI. algorithms by adding E ERR² as regularization term.
Determining & and 3: Next, we discuss the parameter tuning of a and 3, which are equiva- lent to C, and τ)). As explained, the parameter v has several meanings based the inter- pretations of the EQUM, such as the quadratic utility function, targeting optimization, and con- strained optimization. In addition, from the regularization perspective, can adjust the parameter
5

---

Under review as a conference paper at ICLR 2021
Stmbre Devanton 6 G 66
Averog Roward
100
200
100
200
300
400
500
Epicode
Eplodd
Figure 1: The ARs and SDs in the training process of the experiment using the synthetic dataset.
to maximize the expected cumulative reward in the validation data. Thus, we propose the follow- ing four directions for the parameter tuning. First, in economic applications, such as finance, we choose 2a
A = & based on the thcoretical economic assumptions and economic empirical studies
(Ziemba et al., 1974; Kallberg& & Ziemba, 1983) (Appendix A). For instance, Capital Asset Pricing Model (CAPM), which is one of the most popular economic models, is also base on the quadratic utility function (Sharpe, 1964; Lintner, 1965; Mossin, 1966). Second, we set Ç 2 as the targeted reward to achieve. Third, we regard the parameter v as the constrained problem (1). Fourth, through cross-validation, optimize the regularization parameter vb.
EXPERIMENTS
This section investigates the empirical performance of the proposed EQUM with synthetic and real- world financial datasets. The goal of these experiments is to construct a portfolio with a well- controlled mean and variance through algorithms. In addition, we also show experimental results using CartPole and Atari games. Note that the per-step rewards of portfolio selection are stochastic variables and those of CartPole and Atari games are deterministic variables. However, even when the reward is deterministic, the cumulative reward has randomness owing to the stochastic policy.
5.1 EXPERIMENTS WITH SYNTHETIC PORTFOLIO SELECTION DATASETS
Following Tamar et al. (2012) and Xie et al. (2018), we artificially generate a portfolio dataset. Let us consider a portfolio composed of two types of assets: liquid asset, which has a fixed interest rate r = 1.001, and a non-liquid asset, which has a time-dependent interest rate (4) E (1.1, 2). While we can sell the liquid asset at every time step 2, T, we can sell the non-liquid asset only after a maturity period of N steps: that is, when holding liquid asset, we obtain 1.001 per period; when holding non-liquid asset at the 1-th period, we obtain 1.1 or 2 at the ++ N-th periods. In addition, the non-liquid asset has some risk of not being paid with a probability Prisk; that is, if the non-liquid asset defaulted during the N periods, we cannot obtain any profits by having the asset. In this setup, typical investment strategy is to construct portfolio using both liquid and non-liquid assets to control the mean and variance. In our model, the investor may change his portfolio by investing fixed fraction a = 0.2 to the non-liquid asset at each time step. As performance metric of the portfolio, we focus on the mean and variance of the cumulative reward when having the cash 1 at the first period and investing the cash for 50 periods following algorithm.
In particular, we aim to investigate the sensitivity of the EQUM against the parameter vo 2/C5 and compare the
EQUM Tamar
5
EQUM with the standard SPG and AC algorithms. We ap- ply the SPG algorithm with the EQUM (Section 4.3) to the synthetic datasets. For Ç, we use ( 1, 3, 4, 5, 6. Note that from the regularization perspective with a parameter vi. EQUM with v = 0.1 is equal to minimize the MSE between the cumulative reward and Ç 5 1/(2 x 0.1). We train the model with 500 episodes. In Figure 1, we calculate the average reward and standard deviation at each episode of the training process over 1, 000 trials. In Table 1, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials. For each trial, we compute the AR and SD over 100 trial and compute the average of each AR and SD over 1, 000 trials. From Figure I and Table 1, we can clearly confirm that the EQUM framework
4
3
Paneto afficiant
2
1
[-2
30
0
C-15
ve
-1
(-ca
20
10
20
30
Var
Figure 2: Higher AR and lower Var
methods are Pareto efficient.

---

Under review as a conference paper at ICLR 2021
Table 1:
The experimental results of the synthetic dataset with the ARs and SDs over 1, 000 trials.
Standar SPG
SPG
6 6-12 5,07
XR
757
6251
SD
5.34
1977
3971
4,832
5,089
5,000
Table 2: The experimental results of the synthetic dataset with the MSES for Ç over 1000 trials.
Studard Begst Value
5PG
EQUM
22:551
30.119
10,014
6,5M8
16,404
26.975
28.345
21.984
16.362
7,454
11.846
17.593
19,456
20.921
15,450
25.111
10.399
9.289
12,699
13.950
15,096
10:155
19,690
15.35 35.
8.732
0.785
10.456
11:272
[-6
8.381
48.208
22.258
10.174
8,852
8.948
9.448
realize the control of the mean-variance trade-off. As shown in the results, we can reduce the vari- The MSE from target ( is shown in Table 2. In Figure 2, for the EQUM with various Ç and the method of Tamar et al. (2012) (Tamar) with various variance constraints (var), we plot the AR and variance (Var) computed on the train environment over 1, 000 trials. We show the Var instead of the SD because the Tamar controls the variance. Compared with Tamar, the EQUM returns Pareto efficient portfolios; that is, higher AR and lower Var. We consider unlike the di- rect optimization of the EQUM, the Tamar suffers from the complicated optimization mechanism. Figure in Appendix B.1 corresponds to the test environment version of Figure 2.
ance by increasing
In Appendix B.2, we also show experimental results using another synthetic dataset of American- style option following Tamar et al. (2014) and Xie et al. (2018).
5.2 EXPERIMENTS WITH WWALD DATASET
We use well-known benchmarks called Fama & French (1992) (FF) datasets to ensure the repro- ducibility of the experiment. We use the FF25, FF48 and FF100 datasets. The FF25 and FF 100 dataset includes 25 and 100 portfolios formed based size and book-to-market ratio; the FF48 dataset contains 48 portfolios representing different industrial sectors. We formulate the problem by an episodic MDP. We use all datasets covering monthly data from July 1980 to June 2020. The state is past 12 months returns of each asset and the action is defined as portfolio weight; that is, the number of actions is equal that of assets. The reward is obtained as the portfolio return. Here, the portfolio return at time 1 S 1 < T is defined as Y/ = Σ Wj. where Uj.t is the return of j asset at time f, Wj,t-1 is the weight of j asset in the portfolio at time - 1, and m is the number of assets. The length of the episode is 1 years (12 months). For the stochastic policy, a three-layer feed-forward neural network with the ReLU activation function where the number of units in each respective layer is equal the number of assets, 100, 50, and the number of actions. We use the softmax function for the output layer.
adopt
portfolio (EW) weights the financial assets equally (DeMiguel et al., 2007). A mean- variane portfolio (MV) com- putes the optimal variance under a mean constraint (Markowitz, 1952). For computing the mean vector and covariance matrix, we use the latest 10 years (120 months) data. An Kelly growth opti- mal portfolio with ensemble learning (EGO) is proposed by Shen et al. (2019). We set the number of resamples as m 50, the size of each resample m2 57, the number of periods of return data r = 60, the number of resampled subsets m3 = 50, and the size of each subset m4 = 0.7 where m 100 in FF100. A portfolio blending via Thompson sampling (BLD) is proposed by Shen & Wang (2016). We use the latest 10 years (120 months) data to compute for the sample covariance matrix and blending parameters. A policy gradients with variance related risk criteria (Tamar) is proposed by Tamar et al. (2012). We set the target variance terms 17 as 350,500,1000, A block coordinate ascent algorithm proposed by Xie et al. (2018), which is referred to as Mean- Variance Portfolio (MVP). We set the regularization parameters as 10, 100, 1000. Then, let us denote the SPG with the EQUM framework as EQUM. The parameter v is chosen from 1/3, 2/3, and 1. For optimizing Tamar, MVP and EQUM, we set the Adam optimizer with learning rate 0.01 and weight decay parameter 0.1. We train the neural networks for 10 episodes. Each portfolio is updated by sliding mnn-month-ahead.
Portfolio models: We use the following portfolio models. An
number of assets; that is, m 25 in FF25, m 48 in FF48 and
Performance metrics: The following measures widely used in finance to evaluate portfolio strate- gies (Brandt, 2010) are chosen. The cumulative return (CR), annualized risk as the standard de-
7

---

Under review as a conference paper at ICLR 2021
Table 3: The performance of each portfolio models during out-of-sample period (from July 2000 to June 2020) for FF25 dataset (upper table). FF48 (middle table) and FF100 (lower table). For each dataset, the best performance is highlighted in bold.
Method
EW
MV
100
1010
CRT
191.35
25.0
209.66
134,21
231.02
256.34
256,89
21308
215.85
24535
300.15
194.2
24335
RISKL
18.35
25.36
19:17
12.8
24.22
20:21
20,20
19:25
19.29
1339
13:22
12.19
14.95
KRT
0.52
1008
1835
1.56
0.38
0.71
00.11
1.35
0.55
0337
1.12
0.50
0.32
Ma<DD1
0.54
0.75
0.57
0.37
0.64
0.58
0.58
0.57
0.57
0.30
0.31
0.30
0.36
FR48
CKF
194.5
34.81
248.14
12625
120.6
158.00
19065
344.22
335.66
20095
226339
246.6
256:45
RISKL
16.58
30.23
19.56
18.76
13.0
14.6
14.75
2538
21.87
2022
10:0
15:35
11.75
RR1
0:39
0.8
0.38
0:38
0331
072
0.76
0.3
0.37
0.30
131
MicD0:
0.33
0.84
0.3
033
0.5
0.55
052
CR
103.01
35.35
29007
viation of return (RISK) and risk-adjusted return (R/R) are defined follows: CR E Ut.
RISK / V C - CR/T)2, and R/R 1 x CR/RISK. R/R the most important mea- sure for a portfolio strategy. We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure Magdon- Ismail & Atiya (2004) for the portfolio strategy. In particular,
MaxDD is the largest drop from peak defined MaxDD minee(1,T] (0, max-en W.
W, 1).
where Wi is the cumulative return of the portfolio until time k; that is, W. = -II yye).
Table 3 reports the performances of the portfolios. In almost all cases, the EQUM portfolio achieves the highest R/R and the lowest MaxDD. Therefore, we can confirm that the EQUM portfolio has a high R/R, and avoids large drawdown. The real objective (minimizing variance with penalty on return targeting) for Tamar, MVP, and EQUM is shown in Appendix B.3. Except for FF48's MVP, the objective itself is smaller than EQUM's. Since the values of the objective is the same the RR, we can empirically confirm that the better optimization, the better performance.
5.3 EXPERIMENTS WITH CARTPOLE AND ATARI GAMES
We also conduct experiments using CartPole and Atari games, where the reward is given as deter- ministic, and the randomness of the cumulative reward depends only on the stochastic policy. We investigate the sensitivity of &b = 1/(2() with CartPole and the compared the performance of the EQUM framework with that of Tamar et al. (2012) and Xie et al. (2018). The results are shown in Appendix B.4. In many experimental results, contrary to our expectations, we observed that the EQUM also improves the expected cumulative reward, not only the variance. We hypothesize that this is because there is often a limit on the cumulative rewards achieved by the standard expected cumulative reward maximizing algorithms. For instance, when the reward at each period is and the discount factor is 0.99, the infinite sum is 100. In such a case, instead of naive reward maxi- mization, MSE minimization against the target reward 100 may result in a more stable performance empirically. We also hypothesize that even if an optimal policy is deterministic, the EQUM can improve the stability of the training process by reducing the variance. This observation is an open problem related to exploration and exploitation trade-off. Therefore, unless we could increase the cumulative reward infinitely, the proposed EQUM framework can stabilize the performance. From this aspect, we can confirm that the EQUM provides regularization effect.
6 CONCLUSION
In this paper, we proposed an EQUM framework as variant of MVRL. Compared with existing MVRL methods, the EQUM framework is computationally friendly. The proposed EQUM frame- work also includes various interpretations, such as targeting optimization and regularization and is suitable for many real-world applications, such as finance and playing games. We investigated the effectiveness of the EQUM framework compared with the standard RL and existing MVRL methods through the experiments. In the results, the proposed method successfully controls the mean-vant veraanvariance trade-off. As open problem, also observed that even when an optimal policy is deterministic, the proposed method improves performance. We hypothesize that the proposed method affects the stabilization of the training process.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jonathan Baxter and Peter L. Bartlett. Infinite-horizon cial Intelligence Research, 2001.
estimation. Journal of Artifi-
J.O. Berger. Statistical Decision Theory and Bayesian Analysis. 1985.
Dimitri Bertsekas and John Tsitsiklis. Neuro-Dynamic Programming. 1996.
Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatiliyy reduction. IJCAI, 2020.
Taras Bodnar, Nestor Parolya, and Wolfgang Schmid. On the Exact Solution of the Multi-Period Portfolio Choice Problem for Exponential Utility under Return Predictability. European Jour- nal of Operational Research, 2015.
Taras Bodnar, Yarema Okhrin, Valdemar Vitlinskyy, and Taras Zabolotskyy. Determination and estimation of risk aversion coefficients. Computational Management Science, 2018.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Michael Brandt. Portfolio choice problems. In Handbook of financial econometrics: Tools and
techniques. 2010.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
T. Bulmus and S. Özekici. Portfolio selection with hyperexponential utility functions. OR Spectrum, Jan 2014.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for evar optimization in mdps. In NeurlPS. 2014.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. Journal of Machine Learning Research, 2017.
Victor DeMiguel, Lorenzo Garlappi, and Raman Uppal. Optimal versus naive diversification: How inefficient the 1/n portfolio strategy? The review of Financial studies, 2007.
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforcement learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems, 2016.
Eugene F Fama and Kenneth R French. The cross-section of expected stock returns. the Journal of Finance, 1992.
Peter C. Fishburn and R. Burr Porter. Optimal portfolios with one safe and one risky asset: Effects of changes rate of return and risk. Management Science, 1976.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 2015.
Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied control under constraints. Journal of Artificial Intelligence Research, 2005.
J. G. Kallberg and W. T. Ziemba. Comparison of alternative utility functions in portfolio selection problems. 1983.
Yoram Kroll, Haim Levy, and Harry Markowitz. Mcan-variance versus direct utility maximization. Journal of Finance, 1984.
John Lintner. The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets. The Review of Economics and Statistics, (1):13-37, 1965.
D.G. Luenberger. Investment Science. 1998.

---

Under review as a conference paper at ICLR 2021
Malik Magdon-Ismail Ismaal and Amir F Atiya. Maximum drawdown. Risk Magazine, 2004.
H. Markowitz. Portfolio Selection: Efficient Diversification of Investments. 1959.
Harry Markowitz. Portfolio selection. The journal of finance, 1952.
H.M. Markowitz, G.P. Todd, and W.F. Sharpe. Mean-Variance Variance Analysis in Portfolio Choice and Capital Markets. 2000.
Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomic Theory. 1995.
Robert Merton. Lifetime portfolio selection under uncertainty: The continuous time case. The
Review of Economics and Statistics, 1969.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
Jan Mossin. Equilibrium capital asset market. Econometrica, 34, 1966.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In NeurlPS, pp. 8024-8035. Curran Associates, Inc., 2019.
L. A. Prashanth and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. In NeurIPS, 2013.
L. A. Prashanth and Mohammad Ghavamzadeh. Varitnce- actor-critic algorithms for discounted and average reward mdps. Machine Learning, 2016.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley Sons, Inc., 1994.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
William F. Sharpe. Capital asset prices: theory of market equilibrium under conditions of risk*. The Journal of Finance, 1964.
Weiwei Shen and Jun Wang. Portfolio blending via thompson sampling. In IJCAI, 2016.
Weiwei Shen, Bin Wang, Jian Pu, and Jun Wang. The kelly growth optimal portfolio with ensemble learning. In AAAI, 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 1998.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NewrIPS, 1999.
Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In ICML, 2012.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In ICML, 2014.
J. Tobin. Liquidity Preference as Behavior Towards Risk. Review of Economic Studies, 1958.
R.J. Williams. Toward a Theory of Reinforcement-termiing Connectionist Systems. 1988.
Ronald Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 1991.
Ronald Williams. Simple statistical grolient- algorithms for connectionist reinforcement learning. Machine Learning, 1992.
10

---

Under review as a conference paper at ICLR 2021
Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu. and Daesub Yoon. A block coordinate ascent algorithm for mean-variance optimization. In NeurIPS. 2018.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning, 2020.
W. T. Ziemba, C. Parkan, and R. Brooks-Hill. Calculation of investment portfolios with risk free borrowing and lending. Management Science, 1974.

---

Under review as a conference paper at ICLR 2021
A ECONOMIC THEORY AND QUADRATIC UTILITY FUNCTION
Considering the mean- vean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al. (2012) and Xie al. (2018) pointed out. The mean-vane variinai trade-off is justified by assuming either quadratic utility function to the economic agent or multivariate normal distribution to the financial assets. By assuming the quadratic utility function or the normal distri- bution, we assume that the expected utility function of the agent is maximized by maximizing the expected reward and minimizing the variance. Based on this observation, Markowitz (1952) proposed the following steps for providing a portfolio to an economic agent (Also Markowitz (1959), page 288, and Luenberger (1998)):
Constructing portfolios minimizing the variance under several reward constraint;
Among the portfolios constructed in the first step, the economic agent chooses a portfolio maximizing the utility function.
Therefore, the goal of Markowitz's 's portfolio is not only construct the portfolio itself but also to maximize the expected utility function of the agent. In traditional economics, this two-step is adopted because directly predicting the reward and variance maximize the expected utility func- tion is difficult; therefore, first gathering information based on analyses of an economist, then we construct the portfolio using the information and provide the set of the portfolios an economic agent. However, owing to the recent development of machine learning, we can directly represent the complicated economic dynamics using flexible models, such as deep neural networks. In addition, as Tamar et al. (2012) and Xie et al. (2018) reported, when constructing the mean- vvrainn portfolio RL, we suffer the double sampling issue. Therefore, in this paper, we aim to achieve the original goal of the mean-variance approach; that is, the expected utility maximization. Note that this idea is not restricted to financial applications but can be applied to applications where the agent utility can be represented only by the mean and variance. In the following subsection, we review the existing studies of finance and quadratic utility function.
A.1 MARKOWITZ'S PORTFOLIO AND CAPITAL ASSET PRICING MODEL
Markowitz's ' portfolio is known as the mean-variance portfolio (Markowitz, 1952; Markowitz et al., 2000). Constructing the mean-variance portfolio motivated by the agent's expected utility maxi- mization. When the utility function is given as the quadratic utility function, or the financial asset follows the multivariate normal distribution, a portfolio maximizing the agent's expected utility function is given as a portfolio with minimum variance under a certain standard expected reward.
The Capital Asset Pricing Model (CAPM) theory is concept which closely related to Markowitz's s portfolio (Sharpe, 1964; Mossin, 1966; Lintner, 1965). This theory theoretically ex- plains the expected return of investors when the investor invests in a financial asset; that is, it derives the optimal price of the financial asset. To derive this theory, as well as Markowitz's 's portfolio, we assume the quadratic utility function to the investors or the multivariate normal distribution to the financial assets.
Merton (1969) extended the static portfolio selection problem to a dynamic case. Fishburn & Porter (1976) studied the sensitivity of the portfolio proportion when the safe and risky asset distribu- tions change under the quadratic utility function. Thus, there are various studies investigating rela- tionship between the utility function and risk-averse optimization (Tobin, 1958; Kroll et al., 1984; Bulmuy& & Özekici, 2014; Bodnar et al., 2015).
A.2 EMPIRICAL STUDIES ON THE UTILITY FUNCTIONS
The standard financial theory is built on the assumption that the economic agent has the quadratic utility function. For supporting this theory, there are several empirical studies to estimate the param- eters of the quadratic utility function. Ziemba et al. (1974) investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian financial dataset. Recently, Bodnar et al. (2018) investigate the risk parameter (a and / in our formulation of the quadratic utility function) using the markets indexes in the world. They found that the utility function parameter depends the market data model.
12

---

Under review as a conference paper at ICLR 2021
EQUM Tamar
Pareto efficient
var 65
AR
55
1.5
50
0
Ver
0.5
V355
-2
var
v20
0
10
20
30
40
Var
Figure 3: The AR and Var on the test environment. Higher AR and lower Var methods are Pareto efficient.
A.3 CRITICISM
For the simple form of the quadratic utility function, the financial models based the utility widely accepted in practice. However, there also criticism that the simple form cannot capture the real-world complicated utility function. For instance, Kallberg & Ziemba (1983) critieized the use of the quadratic utility function and proposed using utility function, including higher moments. This study also provided empirical studies using U.S. financial dataset for investigating the properties of the alternative utility functions. However, to the best of our knowledge, financial practitioners still use financial models based on the quadratic utility function. We consider this is because the simple form gains the interpretability of the financial models.
B DETAILS OF EXPERIMENTS
In this section, we show the additional experiments and describe the details of the experiments. To implement SPG and AC algorithms, mainly follow the Pytorch example'.
B. PARETO EFFICIENCY ON THE TEST ENVIRONMENT OF THE SYNTHETIC PORTFOLIO SELECTION DATASET
For investigating the Pareto efficiency, show the AR and Var of the train environment in Figure of Section 5.1. Here, we also show the AR and Var of the test environment in Figure 3.
B.2 EXPERIMENTS USING THE SYNTHETIC AMERICAN-STYLLE OPTION DATASET
Among various options finance, an American-style option refers to contract that we can execute an option right at any time before the maturity time T; that is, buyer who bought a call option has right to buy the asset with the call option strike price W Weall any time; a buyer who bought put option has right to sell the with the call option strike price W put at any time.
In the setting of Tamar al. (2014) and Xie et al. (2018), the buyer simultaneously buy call and put options, which have the strike price W.all = and W Wput =, respectively. The maturity time is set as =. If the buyer executes the option at time f, the buyer obtains reward r, = max(0, Wput- Jt) max(0,
1 and define the stochastic process as follows: 2, = xt-1ff with probability 1 and H2 = xt-1fa with probability 1, where fu and fa- These parameters follows Xie al. (2018). Under this setting, we investigate the performance of our EQUM. For the EQUM, we use ( = 0.1, 0.5. 0.7, 1, 1.3. For the other settings, we follow the previous experiment for portfolio management.
W..t), where is an asset price. We set
'https://gtt http::///ggtubb.ooo...o http:://hhhtccooooo/////e/eeeeee
13

---

Under review as a conference paper at ICLR 2021
0.42
0.38
RERFORCE
0.40
0.38
0.34
0.34
RERFORTE (-00)
0.32
C-10
Average
0.32
0.30
0.28
C-055
[-0.1
C-0.7
0e0.3
0.28
0.30
0.26
(=10
0.26
100
200
300
400
500
Episode
Episode
Figure 4: The ARs and SDs in the training process of the experiment using the synthetic dataset.
Table 4: The experimental results of the synthetic dataset with the ARs and SDs over 1000 trials.
Standard (= 0
EQUM
C - 0.1
<= 0.3
< = 0.5
= 0.7
C 1.00
[ < 1.3
AR
0.367
0.301
0.321
0,356
0.372
0.377
0.377
SD
0.307
0.275
0.285
0.306
0.317
0.318
0.320
Table The experimental results of the synthetic dataset with the MSEs for Ç over 1000 trials.
Standard { ==0
EQUM
Target Value
( 0.1
= 0.3
Ç 0.5
0.7
{ <=10
1.0
(= 1.3
MSE from = 0.1
0.072
0.041
0.050
0.067
0.076
0.078
0.078
0.006
0.005
0.5
0.019
0.040
0.033
0.022
0.018
0.017
0.017
SPG
0.7
0.112
0.160
0.145
0.120
0.109
0.106
0.106
0.402
0.489
0.462
0.416
0.395
0,390
1.3
0.872
0.999
0.960
0.893
0.862
0.854
0.853
In Figure 4, we calculate the average reward and standard deviation at each episode of the training process by conducting 1, 000 trials. In Table 4, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials. We show the MSE between the realized reward and the target reward Ç in Table 5, where the lowest MSE method is highlighted in bold. As well as the experimental results with the synthetic portfolio dataset, confirm that the EQUM with CartPole and Atari games, we can observe that the EQUM also increases the expected reward contrary our expectation.
control the risk well. In addition, well the experiments
B.3 DETAILS OF EXPERIMENTS OF PORTFOLIO OPTIMIZATION
The real objective (minimizing variance with penalty on return targeting) for Tamar, MVP and EQUM is shown in Table 6. Except for FF48's s MVP, the objective itself is smaller than EQUM's. Since the values of objective is the same as the RR, we can empirically confirm that the better optimization, the better performance.
We also divide the performance period into two for robustness checks. Table 7 shows the first-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020. In almost all cases, the EQUM portfolio achieves the highest R/R.
Table The real objective (minimizing variance with penalty on return targeting) for Tamar, MVP and EQUM for FF25 dataset (upper panel). FF48 (middle panel) and FF100 (lower panel). Among the comparisons the various portfolios, the best performance within each dataset highlighted in bold.
Tamar
MVP
EQUM 2/3
250
300
1000
10
100
1000
PF25
-3.359
-7,346
-7.444
-6,206
-6.206
-7,49
-8,83
-7,521
-7.620
FF48
-3.265
-7,252
-7335
-6384
99,616
-7,369
-7,514
-8,343
-8.772
FF100
-6,600
-7.158
-6,042
-6.148
-3.27
-6.13
-3.69
-7,270
-9,650
14

---

Under review as a conference paper at ICLR 2021
Table 7: The performance of each portfolio during first half out-of-sample period (from July 2000 to June 2010) and second half out-of-sample period (from July 2010 to June 2020) for FF25 dataset (upper panel), FF48 (middle panel) and FF100 (lower panel). Among the comparisons of the various portfolios, the best performance within each dataset is highlighted in bold.
FF25
EW
MV
EGO
BLD
Tamar
MVP
EQUM
1000
TOD
1000
173
2/3
First- Period (from July 2000
June 2010)
CRT
69.36
-50.09
86.53
49.76
79.06
159.61
139.56
132.79
132.79
146.55
186.21
141.17
158.2
RISK]
19.35
28.85
19.99
12.25
21.32
15.95
15.95
13.91
13.91
14.33
11.58
11.61
11.69
RRT
0.36
-0.17
0.43
0.41
0.37
1.00
1.00
0.95
0.95
1.02
1.61
1.22
1.35
MaxDD.
0.54
0.75
0.57
0.37
0.33
0.31
0.31
0.22
0.22
0.21
0.13
0.14
0.18
Second
Hall Period (from July 2000
to June 2010)
CRT
121.99
75.71
123.25
84.45
151.96
126.73
127.33
83.16
83.16
97.43
113.97
52,96
85.70
RISK]
17.64
21.14
18.30
11.73
26.76
23.71
23.70
23.43
23.43
13.56
14.97
12.62
17.56
RRT
0.69
0.36
0.67
0.72
0.57
0.53
0.54
0.35
0.35
0.72
0.76
0.42
0.49
Ma.
0.31
0.50
0.31
0.22
0.64
0.58
0.58
0.57
0.57
0.30
0.31
0.30
0.36
Tamar
MVP
EQUM
FF48
EW
MV
EGO
BLD
250
500
1000
10
100
1000
1/3
2/3
First-Har Half Pemiod (from July 2000
to June 2010)
CRT
72.02
49.11
89.90
41,26
119.64
64.99
78.38
209.60
185.42
195.13
118.40
127.62
163.23
RISK]
17.61
26.27
21.72
11.48
8.48
9.03
6.61
19.73
17.21
18.11
21.49
8.83
8.96
RRT
0.41
0.19
0.41
0.36
1.41
0.72
1.19
1.06
1.08
1.08
0.55
1.45
1.82
MaxDD.
0.53
0.58
0.53
0.38
0.05
0.15
0.04
0.25
0.24
0.21
0.45
0.15
0.05
Socond Half Period (from July 2000
to June 2010)
CR
122.71
-14.30
159.25
83.19
1.32
125.11
72.27
135.12
148,24
15.86
107.99
118.99
73.20
RISKI
15.45
33.97
17.06
9.96
16.46
19.10
19.75
27.53
25.70
21.88
17.97
19.88
13.87
RRT
0.79
-0.04
0,93
0.84
0.01
0.37
0.449
0.58
0.07
0.60
0.60
0.53
MaxDD_
0.26
0.84
0.28
0.17
0.58
0.54
0.45
0.56
0.52
0.65
0.32
0.45
0.29
FF100
EW
MV
EGO
BLD
Tamar
MVP
EQUM
250
500
1000
100
1000
173
2/3
First-Hi Half Period (from July 2000
to June 2010)
CR
73.25
.4.84
87.78
48.85
143.26
146.59
166.74
218.62
55.87
170.36
215.59
142.66
199.92
RISK]
19.60
30.86
20.53
12.21
16.39
11.21
18.45
20.52
31.72
22.13
13.03
12.69
13.11
RRT
0.37
40.15
0.43
0.40
0.84
1.31
0.90
1.07
0.18
0.77
1.65
1.12
132
MaxDD_
0.55
0.72
0.58
0.36
0.39
0.18
0.27
0.46
0.46
0.31
0.11
0.15
0.16
Second- Half Period (from July 2000 to June 2010)
CRT
120.66
79.10
119.59
79.07
17T.50
91.69
105.37
75.45
172.66
148.55
46.95
84.39
127.25
RISK]
17.87
20.76
18.86
11.55
22.25
16.88
21.56
24.87
30.91
23.62
13.18
14.47
17.52
RR1
0.68
0.38
0.63
0.68
0.77
0.54
0.49
0.30
0.56
0.63
0.36
0.58
0.73
MaxDD_
0.32
0.35
0,31
0.22
0.38
0.49
0.38
0.50
0.65
0.46
0.24
0.31
0.39
B.4 DETAILS OF EXPERIMENTS WITH CARPOLE AND ATARI GAMES
In this section, we report the experimental performances of the proposed EQUM framework with well-known benchmarks. We investigate how the behaviors of existing RL methods are changed by adding the additional E[R²) term. We use a simple two-layer perceptron for modeling the policy following the Pytorch example (Paszke et al., 2019). For the SPG-based algorithms, we define the cumulative reward finite sum with γ 1 following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with y = 0.99 following (Prashanth & Ghavamzadeh, 2013). For the SPG-based algorithms, ward as finite sum with = following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with 7 0.99 following (Prashanth & Ghavamzadeh. 2013).
define the cumulative
B.4.1 SENSITIVITY ANALYSIS ON +b
First, we investigate the sensitivity of the EQUM framework to the parameter ve using the Cart- Pole. We apply the SPG and AC methods (Section 4.3) with the EQUM framework to the Cartpole problem, which is classic control problem. We use e 0.001, 0.002, 0.003, 0.005, 0.01, 0.1 and compare the results with the standard SPG and AC methods. For instance, from the targeting opti- mization perspective. EQUM with s 0.001 is equal to minimize the MSE between the cumulative reward and < = 500 = 1/(2 * 0.001). We train the model by 300 episodes. We calculate the av- erage reward and standard deviation at each period by conducting 300 trials. The results are shown
15

---

Under review as a conference paper at ICLR 2021
(4-0)
240
(Q-0)
220
220
200
200
180
180
Averao
Standard
140
120
120
100
100
50
100
150
200
250
300
30
100
150
200
250
300
240
240
A2C
w00.001
Averege Rewerd
9-0.02
200
9-0 033
180
A2C Ny-0)
180
v-0.025 0.0055
160
0-0 002
0-0 003
160
W-0.03
9-0.1
140
0-0 003
140
0-0.00 005
U-0
v-0.1
30
250
300
50
100
250
300
Episode
Episode
Figure 5: Sensitivity analysis regarding v.. The upper two graphs the results using the SPG-based method and the lower two graphs are the results using the AC-based algorithms. The average reward of the SPG result with v 0.1 lower than 80, and we do not show the result here.
in Figure 5. In the SPG results, we confirm the mean-variance trade-off. In contrast, in the SPG with vi 0.001 and all AC results, the EQUM framework improves the expected cumulative reward. These results contrary the expected cumulative return to decrease the variance. We discuss this topic in more detail in Section 5.3.
expectations because MVRL methods typically decrease
B.4.2 EXPERIMENTS OF ATARI GAMES
When playing games, often consider risk-control while maintaining certain level of reward. Un- der this motivation, we benchmark the proposed EQUM framework on four Atari game tasks from the OpenAl gym (Brockman al., 2016). Among the games, choose BeamRider, Seaquest, Qbert, and Spacelnvaders in which SPG and AC methods work well. We use simplified environ- ments in which the observations We compare our methods with the standard SPG, and MVLR methods of Tamar et al. (2012) and Xie et al. (2018). We calculate the average reward (AR) at the last episode over trials and the stan- dard deviation (SD). In Table 8, we show the results of the SPG algorithm with the standard setting and EQUM framework, where v chosen from 0.001, 0.003, 0.005, 0.01, 0.03, and 0.3. Note that when using vo = 0.005, it is equal to minimize the MSE between the expected cumulative reward and the target 100. almost all cases, the EQUM framework shows the better AR than the standard methods. As well as the previous sensitivity experiment, this result is contrary our expectation. Among the games, we choose simplified BeamRider, Seaquest, Qbert, and SpaceInvaders which the SPG algorithm work well, and the observations are the RAM of the Atari machine, consisting of only 128 bytes. We compare our methods with the standard SPG algorithms, and MVRL meth- ods of Tamar et al. (2012) and Xie et al. (2018). We denote the SPG-based methods proposed by Tamar et al. (2012) and Xie et al. (2018) as Tamar and MVP, respectively. We calculate the average reward (AR) the last episode over trials and the standard deviation (SD). We choose the pa- rameter of Tamar from 200 and 2000, which is denoted as b in Tamar et al. (2012). We choose the parameter of MVP from and 10, which is denoted as À in Tamar al. (2012). The parameters are denoted as param in Table 8.
the RAM of the Atari machine, consisting of only 128 bytes.
16

---

Under review as a conference paper at ICLR 2021
Table The results on Atari games with SPG-based methods.. The highest ARs and lowest SDs highlighted in bold.
Al1
217
Al1
201
AR
A3
117
AIL
30
AHI
201
A36
107
AlL
30
AVE
Seamutcarr
1.03
135.
4004
21.
Qben
40.1
64.8
18.0
47.4
25.8
11.3
40g
5436
27.5
58.3
13.0
42.4
4001
543
13
47.4
50.0
06.5
84.5
17


Reviews:
[{"review_id": "r_XAzWqt2m", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Brief Summary\nThe paper proposes an Expected Quadratic Utility Maximization (EQUM) framework for policy grandient Mean-Variance control. The authors claim that current state-of-the-art methods suffer either suffer from computational issues or cannot control risk at desiderable level, hence they propose their approach as a possible solution. They provide different interpretation for the EQUM: standard objective with a regularization, variance minimization with a constraint on expected return and a return targeting optimization. A policy gradient algorithm for EQUM is proposed together with its Actor-Critic extension.\nIn the experimental session, three settings are provided: a sensitivity analyisis on the \\phi parameter on the CartPole environment, and two risk-averse experiments on Atari games and a Portfolio optimization environment.\n\n## Strong Points\n- The multiple interpretations of the EQUM framework makes it an interesting objective to be studied\n- The state-of-the-art is complete and citations are correctly reported.\n- The paper is readable and easy to follow in all of its parts.\n- The experiments are executed in a proper way, and the proposed baselines are correct. Details are reported to reproduce the results.\n\n## Weak Points\n\n- In the \"Existing MVRL Methods\" section the authors claim that Xie et al. (2018) approach does not allow to control risk at a desirable level, and they use this point to prove the need of an alternative framework. However, this is not true, since different penalty parameter values allow to control risk in different ways. \n\n- It is not clear why one should prefer the EQUM framework over the classic Mean-Variance one. In particular, the gradient they consider is equivalent to ignoring the gradient of the squared expected return term in the classic mean-variance policy gradient, hence it is clearly not equivalent. The EQUM formulation has the same problem of the Mean-Variance one, namely, it penalizes positive deviation as well as negative ones. Moreover, the interpretation as a variance minimization problem with a squared penalization over deviation from target return shows that EQUM is pejorative from this point of view, since positive deviation are penalized again in the soft constraint. Finally, it is not clear in which setting is is convenient to set a target return instead of trying to maximize it.\n\n- In section 4.3 the authors implement the EQUM Policy Gradient, directly extending the REINFORCE estimator. However, the actor-critic formulation they provide is not a correct one, but it is simply a value function baseline. An actor-critic framework should instead substitutes the actual return with a bootstrapped estimate in the gradient formula. Moreover, the authors claim that the extension to TRPO and other state-of-the-art algorithms is straightforward. This is not true, since recent safe policy search algorithms relies on theoretical basis, that are not guaranteed to be valid also in this new framework. Therefore, the authors should provide some analogue safe guarantees also for the framework they propose, otherwise the application of those techniques is not justyfied.\n\n- The first two experiments on CartPole and Atari games seem not to be very well suited for a risk-averse approach: they represent, indeed, two deterministic environments, in which there is not any clear trade-off between risk and expected return.\n\n- The Average Reward measure is used also to evaluate the actor-critic experiments, however, in those cases, there is a discound factor, hence, the expected return would have been the correct performance to report.\n\n- In the financial experiments, performance are not measured w.r.t. the real objective (minimizing variance with a penalty on return targeting). One would expect to see it among the reported measures, otherwise, it is not clear if the method is effective in optimizing its real objective. \n\n- Tamar baseline is tested with only one risk-aversion parameter, hence, the comparison is not fair. Moreover, it would have been more interested to see among the results the approximated Pareto Frontier, obtained by the proposed approach and the baseline with a sensible set of risk-aversion coefficients.\n\n- Some minor fixes:\n    + \"Alternative Objective functions of AC\" in the appendix: it should be specified that the second proposed approach is a n-step Temporal Difference.\n    + The value function notation is difficult to read: it would be better, for example, to use \"V\" for the classical value function, and \"M\" for the second moment one.\n    + Computational issues are different from sampling issues: they become \"computational\" issue only when we need a simulator.\n    + \"Tamer\" should be \"Tamar\".\n    + Plots lower parts in Figure 1 are hard to read.\n\n## Recommendation\nMy reccomendation is to reject this work: there are incorrect statements and derivation, which have been pointed out before, and the overall contribution is small. The proposed framework is interesting for the multiple interpretations that offers, however, it is unclear when it could be useful in a practical case. One of the proposed techiques is flawed and need to be revised, while the other one is a trivial extension of the standard REINFORCE algorithm. Experiments are also not convincing in showing either the effectiveness of the technique, or its superiority w.r.t. the state-of-the-art.\n\n## Questions for the authors\n- Why should one prefer this framework over the mean-variance one, a part from sampling issues?\n- Can the authors provide more examples in which it is convenient to minimize variance targeting a specific return?\n\n## Additional Feedback\n- Since the second moment gradient is equal to the Mean-Variance one, ignoring the gradient of the squared expected return, the authors could use the work by (Tamar, 2012) to correct their Actor-Critic part and include an analysis on compatible features too.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PvGh73wfAD3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a policy gradient style RL algorithm that optimizes an expected quadratic utility, a commonly used objective of risk management in finance and economics. The key idea here is based on the observation that when using the quadratic utility function, the use of mean-variance RL methods can be shown to optimize the utility of the agent. To this effect, the paper considers the use of expected quadratic utility maximization in the policy gradient. The quadratic utility can be naturally modeled using mean and variance. The paper implements two variations -- policy gradient and actor-critic with EQUM framework. \n\nThe main contributions of the paper listed as follow:\n1 The paper shows that maximizing the expected utility by balancing the mean and variance term is equal to reward targeting optimization or constrained cumulative expected reward problem, and could also be interpreted as maximizing the expected reward with the variance as regularizer.\n2 It introduces two algorithms, standard policy gradient and actor-critic, with the EQUM framework.\n\n\nI quite like the paper. The problem is well motivated. I like the idea of using EQUM motivated by economics as the objective function. The paper addresses all relevant works and is written well. The theoretical justification is quite good. The algorithms appear to be well designed and as far as I can see, they are technically correct.  The proposed framework is easier to compute and could be extended to many policy gradient methods, so there is a lot of potential for future works and applications.\n\nWhile I like the paper, there is clearly some room for improvement.\n\nThe justification of \\psi as \\beta/2\\alpha is not well justified. I find the explanation that this is based on economic players and economic empirical studies quite vague. I would have preferred to see the citations and proof. This is crucial to make sure that the algorithm does converge to the correct values. The values are quite important and not being precise will make it hard to reproduce the results. \n\nThe choice of \\psi through cross-validation also appears vague. I understand the nice properties that you have in your framework (particularly avoiding double sampling) but replacing a costly sampling with a cross-validation search for a regularization parameter is not quite convincing. As shown in Figure 1, it is clear that the method is quite sensitive to the choice of \\psi and so this choice needs to be well defined. \n\nWhile the empirical evaluation appears nice and comprehensive, I was a bit disappointed at the choice of atari games for evaluation. I would have liked to see more domains from finance and economics that would provide a better idea of the efficacy of the proposed methods. \n\nIt is true that EQUM methods perform best in R/R, but there is no consistent story here. Different psi can have arbitrarily different performance and I cannot tease out any performance discussion clearly. What really is missing is a good analysis of the empirical performance.\n\nI think the paper has a lot of merit and can be of high impact. But more work is needed in the analysis and discussion part of the paper. With this, it can be a really good submission.\n\nTypos:\n\nPage 2 section 3.1: The formula of maximizing expected reward with regularization has a pair of redundant parenthesis.\n\nPage 3: “Our proposed EQUM framework more focus on this problem.” Reads awkward\n\nPage 4: “Here, we introduce three interpretations of EQUM.” And you present a list of 4.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Z0fKkYp941", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper the author proposed a new mean-variance algorithm whose policy gradient algorithm is more simpler than other SOTA methods and it has an unbiased gradient. Instead of formulating the problem as an traditional mean variance constrained problem, the authors utilized quadratic utility theory and formulate the problem as variance minimization problem with a mean reward equality constraint. Then by reformulating the problem with the penalized problem and opening up the variance formulation, they showed that this mean-variance formulation does indeed have an unbiased policy gradient, that does not require advanced techniques such as double sampling or frenchel duality. To demonstrate the effectiveness of this method on balancing risk and return, they also evaluate their methods on several risk-sensitive RL benchmarks (such as portfolio optimization) and compared with a wide range of risk-sensitive RL methods.\n\nIn general, I find this paper well-written with ample discussions of state-of-the art mean-variance RL algorithms. I also like the flow of this paper which first enumerates the existing issues of mean-variance RL approaches (that requires double sampling or frenchel duality trick to get an unbiased gradient for actor critic), and then propose an alternative algorithm with unbiased gradient that is much simpler yet circumvents the aforementioned complexity. They also demonstrate the performance of this new method on sufficient number of  experiments, ranging from discrete action atari domains, and the portfolio optimization problem, including comparisons with most known mean-variance RL algorithms, and showed that the proposed method achieves some of the best results.\n\nHowever i do have several questions about this paper:\n1) How does the per-step variance discussion in Section 3.2 relate to the proposed method?\n\n2) Can the authors provide more motivations for the problem formulation (1)? It's different from the standard Markowitz mean-variance formulation. How does one set \\psi instead of treating it as a tunable parameter. Even if  one is convinced that (1) is the \"right\" RSRL formulation to solve, why is the penalty formulation in (3) equivalent to the original formulation of (1)? Is there any formal proof?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
In this paper, the authors proposed the expected quadratic utility maximization (EQUM) to implement the risk-aware decision-making for mean-variance RL. The EQUM framework directly optimizes the weighted sum of first and second order moments, while ignores the square of first moments, and thus, largely reduces the difficulty in optimization. The authors tested the proposed policy gradient based algorithm empirically. 

However, the connection to classic mean-variance is not clearly by simply ignoring the square of first moments in the objective theoretically (R2, R3, R4). Meanwhile, the effect of tunable weights (\psi) is not clear and consistent empirically (R2, R3, R4).  

As all reviewers agree this paper is interesting and promising, I encourage the authors to address these issues and consider to submit to next venue.  
Decision: Reject