Question: Why does the proposed algorithm work better than the Tamar et al. and Xie et al.?
Evidence: In reinforcement learning (RL) for sequential decision making under uncertainty, existing methods proposed for considering mean-variance (MV) trade-off suffer from computational difficulties in computation of the gradient of the variance term. In this paper, we aim to obtain MV-efficient policies that achieve Pareto efficiency regarding MV trade-off. To achieve this purpose, we train an agent to maximize the expected quadratic utility function, in which the maximizer corresponds to the Pareto efficient policy. Our approach does not suffer from the computational difficulties because it does not include gradient estimation of the variance. In experiments, we confirm the effectiveness of our proposed methods.
Following and, we consider a portfolio composed of two asset types: a liquid asset with a fixed interest rate r l , and a non-liquid asset with a time-dependent interest rate taking either r low nl or r high nl , and the transition follows a switching probability p switch . An investor can sell the liquid asset at every time step t = 1, 2, . . . , T but the non-liquid asset can only be sold after the maturity W periods. This means that when holding 1 liquid asset, we obtain r l per period; when holding 1 non-liquid asset at the t-th period, we obtain r low nl or r high nl at the t + W -th period. Besides, the non-liquid asset has a risk of not being paid with a probability p risk ; that is, if the non-liquid asset defaulted during the W periods, we could not obtain any rewards by having the asset. An investor can change the portfolio by investing a fixed fraction w of the total capital M in the non-liquid asset at each time step. A typical investment strategy is to construct a portfolio using both liquid and non-liquid assets for decreasing the variance.  In Figure, we plot performances under several hyperparameters, where the horizontal axis denotes the Var, and the vertical axis denotes the CR. Trained agents with a higher CR and lower Var are Pareto efficient. As the result shows, the EQUMRL returns more efficient portfolios than the others in almost all cases. We conjecture that this is because while the EQUMRL is an end-to-end optimization for obtaining an efficient agent, the other methods consist of several steps for solving the constrained optimization, where those multiple steps can be sources of the suboptimal result. We show CRs and Vars of some of their results in the upper, where we can confirm that the EQUMRL succeeded in minimizing the MSEs.
In this section, we introduce how to train a policy with the EQURL. We defined the objective function of the EQUMRL, and EQUMRL is an agnostic in learning method. As examples, we show an implementation based on REINFORCE and Actor-Critic (AC) methods. We use unbiased estimators of the gradients defined in (1).
For an episode k with the length n, the proposed algorithm replaces E π θ G and E π θ G 2 with the sample approximations n t=1 γ t−1 r(S t , A t ) and n t=1 γ t−1 r(S t , A t ) 2 , respectively; that is, the unbiased gradients are given as
(1). Therefore, for a sample approximation G k of E π θ G 2 at the episode k, we optimize the policy with ascending the unbiased gradient
CWe summarize the algorithm as the pseudo-code in Algorithm 1.
Here, we present three advantages of EQUMRL.The first advantage concerns computation. In EQUMRL, we do not suffer from the double sampling issue because the term
The third advantage is the ease of theoretical analysis. For example, referring to the results of, we derive the following result on the convergence of the gradient, which can be applied to simple policy gradient algorithms, not only our proposed REINFORCE-based algorithm. Theorem 1. Consider an update rule such that θ k+1 ← θ k + η i ∇E π θ k u trajectory (G; α, β) , where the learning rates η k are non-negative and satisfy
Suppose that (a) episode always finishes in finite horizon n; (b) the policy π θ has always bounded first and second partial derivatives. Then,
It is expected that non-asymptotic results can be derived by restricting the policy class and the optimization algorithm as and, but this is not the scope of this paper, which aims to provide a general framework.
AC-based trajectory EQUMRL. Another implementation of the EQUMRL is to apply the AC algorithm. For an episode k with the length n, following, we train the policy by a gradient defined as
ω()
ω()
ω()
For more details, see. Remark 1 (Existing approaches). For the double sampling issue, and proposed multi-time-scale stochastic optimization. Their approaches are known to be sensitive to the choice of step-size schedules, which are not easy to control. proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, set an objective function as transformed the objective function as
and trained an agent by solving the optimization problem via a coordinate descent algorithm. However, this approach does not reflect the constraint η because the constraint condition η vanishes from the objective function. This problem is caused by their objective function based on the penalty function g
, where the first derivative does not include η. To avoid this problem, we need an iterative algorithm to decide an optimal δ or change g(x) from x but it is not obvious how to incorporate them into the approach. Remark 2 (Difference from existing MV approaches). Readers may assert that EQUMRL simply omits
2 from existing MV-controlled RL methods, which usually includes the explicit variance term in the objective function, and is the essentially the same. However, there are significant differences; one of the main findings of this paper is our formulation of a simpler RL problem to obtain an MV-efficient policy. Existing MV-controlled RL methods suffer from computational difficulties caused by the double sampling issue. However, we can obtain MV-efficient policy without going through the difficult problem. In addition, EQUMRL shows better performance in experiments even from the viewpoint of the constrained problem because it is difficult to choose parameters to avoid the double sampling issue in existing approaches. Thus, the EQUMRL has advantage in avoiding solving more difficult constrained problems for considering MV trade-off.
In this section, we introduce how to train a policy with the EQURL. We defined the objective function of the EQUMRL, and EQUMRL is an agnostic in learning method. As examples, we show an implementation based on REINFORCE and Actor-Critic (AC) methods. We use unbiased estimators of the gradients defined in (1).
For an episode k with the length n, the proposed algorithm replaces E π θ G and E π θ G 2 with the sample approximations n t=1 γ t−1 r(S t , A t ) and n t=1 γ t−1 r(S t , A t ) 2 , respectively; that is, the unbiased gradients are given as
(1). Therefore, for a sample approximation G k of E π θ G 2 at the episode k, we optimize the policy with ascending the unbiased gradient
CWe summarize the algorithm as the pseudo-code in Algorithm 1.
Here, we present three advantages of EQUMRL.The first advantage concerns computation. In EQUMRL, we do not suffer from the double sampling issue because the term
The third advantage is the ease of theoretical analysis. For example, referring to the results of, we derive the following result on the convergence of the gradient, which can be applied to simple policy gradient algorithms, not only our proposed REINFORCE-based algorithm. Theorem 1. Consider an update rule such that θ k+1 ← θ k + η i ∇E π θ k u trajectory (G; α, β) , where the learning rates η k are non-negative and satisfy
Suppose that (a) episode always finishes in finite horizon n; (b) the policy π θ has always bounded first and second partial derivatives. Then,
It is expected that non-asymptotic results can be derived by restricting the policy class and the optimization algorithm as and, but this is not the scope of this paper, which aims to provide a general framework.
AC-based trajectory EQUMRL. Another implementation of the EQUMRL is to apply the AC algorithm. For an episode k with the length n, following, we train the policy by a gradient defined as
ω()
ω()
ω()
For more details, see. Remark 1 (Existing approaches). For the double sampling issue, and proposed multi-time-scale stochastic optimization. Their approaches are known to be sensitive to the choice of step-size schedules, which are not easy to control. proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, set an objective function as transformed the objective function as
and trained an agent by solving the optimization problem via a coordinate descent algorithm. However, this approach does not reflect the constraint η because the constraint condition η vanishes from the objective function. This problem is caused by their objective function based on the penalty function g
, where the first derivative does not include η. To avoid this problem, we need an iterative algorithm to decide an optimal δ or change g(x) from x but it is not obvious how to incorporate them into the approach. Remark 2 (Difference from existing MV approaches). Readers may assert that EQUMRL simply omits
2 from existing MV-controlled RL methods, which usually includes the explicit variance term in the objective function, and is the essentially the same. However, there are significant differences; one of the main findings of this paper is our formulation of a simpler RL problem to obtain an MV-efficient policy. Existing MV-controlled RL methods suffer from computational difficulties caused by the double sampling issue. However, we can obtain MV-efficient policy without going through the difficult problem. In addition, EQUMRL shows better performance in experiments even from the viewpoint of the constrained problem because it is difficult to choose parameters to avoid the double sampling issue in existing approaches. Thus, the EQUMRL has advantage in avoiding solving more difficult constrained problems for considering MV trade-off.
In this paper, we aim to obtain MV-efficient policies that achieve Pareto efficiency regarding MV trade-off. To achieve this purpose, we train an agent to maximize the expected quadratic utility function, in which the maximizer corresponds to the Pareto efficient policy.****We conjecture that this is because while the EQUMRL is an end-to-end optimization for obtaining an efficient agent, the other methods consist of several steps for solving the constrained optimization, where those multiple steps can be sources of the suboptimal result.****For the double sampling issue, Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013; 2016) proposed multi-time-scale stochastic optimization. Their approaches are known to be sensitive to the choice of step-size schedules, which are not easy to control (Xie et al., 2018).****MV-controlled RL methods suffer from computational difficulties caused by the double sampling issue. However, we can obtain MV-efficient policy without going through the difficult problem
Answer: Xie et al. formulates the MVRL with a constrained optimization problem, but the method cannot consider the constraint because the constraint is not used in optimization. As a result, the objective is the same as ours, the expected quadratic utility function. However, their method includes complicated steps, which may not be necessary, as we pointed out. Tamar et al. applies computationally difficult steps in optimization, multi-time-scale stochastic optimization, which requires control of learning parameters. This difficulty of this optimization is well-known in RL, as pointed out by Xie et al. While we point out the computational difficulties of the method of Tamar et al., which is also mentioned in the other existing studies, we avoid explicitly mentioning the problem in Xie et al., as that would be a discussion of the correctness of that paper.