# An Improvement of Stochastic Gradient Descent Approach for Mean-Variance Portfolio Optimization Problem

Authors: Stephanie S. W. Su; Sie Long Kek
Journal: Journal of Mathematics
 (2021)

---

## Abstract

In this paper, the current variant technique of the stochastic gradient descent (SGD) approach, namely, the adaptive moment estimation (Adam) approach, is improved by adding the standard error in the updating rule. The aim is to fasten the convergence rate of the Adam algorithm. This improvement is termed as Adam with standard error (AdamSE) algorithm. On the other hand, the mean-variance portfolio optimization model is formulated from the historical data of the rate of return of the S&P 500 stock, 10-year Treasury bond, and money market. The application of SGD, Adam, adaptive moment estimation with maximum (AdaMax), Nesterov-accelerated adaptive moment estimation (Nadam), AMSGrad, and AdamSE algorithms to solve the mean-variance portfolio optimization problem is further investigated. During the calculation procedure, the iterative solution converges to the optimal portfolio solution. It is noticed that the AdamSE algorithm has the smallest iteration number. The results show that the rate of convergence of the Adam algorithm is significantly enhanced by using the AdamSE algorithm. In conclusion, the efficiency of the improved Adam algorithm using the standard error has been expressed. Furthermore, the applicability of SGD, Adam, AdaMax, Nadam, AMSGrad, and AdamSE algorithms in solving the mean-variance portfolio optimization problem is validated.


---

## Body

## 1. Introduction

Recently, the application of the stochastic gradient descent (SGD) approach to machine learning and deep learning is actively explored. Moreover, due to the ability of the SGD approach in handling the stochastic optimization problems [1] and for solving optimization problems under the uncertainty environment [2, 3], the SGD approach and its variants have been developed rapidly. By virtue of this, the mean-variance portfolio optimization problem [4], which deals with risk and return, has attracted the attention of the investment community. The optimal decision on the portfolio selection is necessarily needed, where the scientific approach is employed in maximizing the return with the minimum risk [5]. However, this optimal decision is difficult to be made in advance.In this paper, the disadvantage of the SGD approach, which is the slow convergence [6, 7], is noticed. To improve this weakness, the standard error from the sampling theory is added to the updating rule of the adaptive moment estimation (Adam) algorithm [8], which is the current variant of the SGD approach. On this basis, the convergence rate of the Adam algorithm is improved significantly. This improved version is then known as Adam with standard error (AdamSE) algorithm. On the other hand, the application of SGD methods, including Adam, adaptive moment estimation with maximum (AdaMax), Nesterov-accelerated adaptive moment estimation (Nadam), AMSGrad, and AdamSE approaches, for solving the mean-variance portfolio optimization problem is further studied. For this purpose, the historical data of the rate of return for the S&P 500 stock, 10-year Treasury bond, and money market are employed. Then, the mean-variance portfolio optimization model is formulated. During the calculation procedure, the iterative solution converges to the optimal portfolio solution, and the performance of these algorithms is presented.The rest of the paper is organized as follows. In Section2, the mean-variance portfolio optimization problem is described, where the expected return and the covariance matrix are expressed. In Section 3, the enhancement of the convergence rate of the Adam algorithm by using the standard error from the sampling theory is further discussed. The calculation procedure for the SGD, Adam, AdaMax, Nadam, AMSGrad, and AdamSE algorithms is summarized. In Section 4, a mean-variance portfolio optimization model is formulated using the historical data of S&P 500 stock, 10-year Treasury bond, and money market. Then, the model is solved by using the algorithms discussed, and the results are presented. Finally, some concluding remarks are made.


## 2. Problem Description

Consider a general mean-variance portfolio optimization problem forn risky assets given by(1)minimizefw=12wTΣw,subject towTμ=R,wT1=1,w≥0,where w=w1,…,wnT is the vector of portfolio weights of the assets and Σ represents the covariance for the assets. Here, wTΣw gives the variance of the portfolio, μ=μ1,…,μnT is the vector of portfolio return mean, and 1=1,…,1T is the vector with 1s elements. Note that the targeted expected return R depends on the risk tolerance of investors.Furthermore, by using the geometric mean, the portfolio return mean is computed from(2)μi=∏t=1T1+ri,t1/T−1,where ri,t is the rate of return of asset i at time t=1,…,T and μi is the mean of the rate of return for the asset i, whereas the covariance matrix of assets is defined as(3)Σ=1T∑i=1Tri,t−μirj,t−μj,for i,j=1,…,n. The assumption of the mean-variance portfolio optimization defined in (1) is to minimize the risk of an investment that is represented by the variance, at the same time, to satisfy the targeted return of the portfolio [9]. Since the risk is always related to randomness and uncertainty [10], the stochastic optimization approach will be used in solving the optimization problem defined in (1).


## 3. Stochastic Optimization Method

Now, let us define the Lagrange function as follows:(4)Lw,λ=12wTΣw+λ1R−wTμ+λ21−wT1+λ3Tw,where λ=λ1,λ2,λ3T is the vector of the Lagrange multiplier. Then, the following first-order necessary conditions are derived:(5)∂L∂w=Σw−λ1μ−λ21+λ3=0,(6)∂L∂λ1=R−wTμ=0,(7)∂L∂λ2=1−wT1=0,(8)λ3Tw=0,λ3≥0.


### 3.1. Analytical Optimal Solution

From (5), the optimal weighted value of the portfolio is calculated from(9)w∗=λ1Σ−1μ+λ2Σ−11.Refer to (6), the targeted expected return is provided by(10)R=μTΣ−1Σw∗.Then, substitute (9) into (10) to have the targeted expected return in terms of the Lagrange multipliers, that is,(11)R=λ1μTΣ−1μ+λ2μTΣ−11.Rewrite (7) to be(12)1=1TΣ−1Σw∗and substitute (9) into (12):(13)1=λ11TΣ−1μ+λ21TΣ−11.From (11) and (13), after doing some algebraic manipulations, the Lagrange multipliers are computed from(14)λ1=μTΣ−11−R1TΣ−111TΣ−1μμTΣ−11−μTΣ−1μ1TΣ−11,(15)λ2=R1TΣ−1μ−μTΣ−1μμTΣ−111TΣ−1μ−1TΣ−11μTΣ−1μ,and λ3=0. Therefore, from the discussion above, the analytical solution of the mean-variance portfolio optimization problem defined in (1), which is given by (9), (14), and (15), is resulted. However, this analytical solution is assumed to be not available due to the uncertainty and randomness of the variables.


### 3.2. Stochastic Gradient Descent Algorithm

Referring to the mean-variance portfolio optimization problem defined in (1), let us introduce an augmented objective function faw as(16)faw=12wTΣw+λ1R−wTμ+λ21−wT1+λ3Tw,with w=w1,…,wnT. Since the existence of the uncertainty, the augmented objective function faw defined in (16) can be rewritten as the expected objective function, given by(17)Efaw=1n∑i=1nfa,iw,where fa,iw,i=1,…,n, is the element of the augmented objective function faw that is uniformly sampled at random and E⋅ is the expectation operator. By virtue of this, the sampled gradient ∇fa,iw is denoted as an unbiased estimator to be(18)E∇faw=1n∑i=1n∇fa,iw.Notice that the first-order necessary condition for (16) is equivalent to the first-order necessary condition (5). That is,(19)∂faw∂w=∂Lw,λ∂w.For convenience, define the stochastic gradient(20)gw=∂L∂w,which can be calculated from (5). The updating rule of the SGD approach is given by(21)wk+1=wk−α⋅gjwk,with the step size α, which is also known as the learning rate, k is the number of iterations, and the random index j is the gradient referred.Hence, the calculation procedure of the SGD algorithm is summarized as in Algorithm1.Algorithm 1:SGD algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: update the vectorwk from (21). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The tolerance isε=10−6, and the learning rate is α = 0.001.


### 3.3. Adaptive Moment Estimation Algorithm

In the Adam approach [11], the exponential decaying averages of past gradients mk and past squared gradients vk are considered as follows:(22)mk=β1mk−1+1−β1gk,(23)vk=β2vk−1+1−β2gk2,where gk is the gradient, β1 and β2 are the decay rates, which are close to 1. Notice that mk and vk are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients, respectively. These biases are counteracted by using the bias-corrected first- and second-moment estimates, given by(24)m^k=mk1−β1k,(25)v^k=vk1−β2k.Thus, Adam updating rule has been presented as follows:(26)wk+1=wk−α⋅m^kv^k+δ,where δ is the smoothing term used to avoid division by zero.The calculation procedure of the Adam algorithm is summarized as in Algorithm2.Algorithm 2:Adam algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: update the vectorwk from (26). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


### 3.4. Adaptive Moment Estimation with Maximum

AdaMax, which is the adaptive moment estimation with maximum [11], is a variant of the Adam optimizer that uses the infinity (ℓ∞) norm, while the Adam optimizer itself uses the ℓ2-norm for optimization. When generalizing the Adam algorithm to the ℓ∞-norm, and hence in AdaMax, the gradient update is the maximum between the past gradients and current gradient, which is shown as(27)vk=β2∞vk−1+1−β2∞gk∞=maxβ2⋅vk−1,gk.Then, the updating rule of AdaMax is(28)wk+1=wk−αvk⋅m^k.The calculation procedure of the AdaMax algorithm is summarized as in Algorithm3.Algorithm 3:AdaMax algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (27).Step 5: calculate the bias-corrected first-moment estimate from (24).Step 6: update the vectorwk from (28). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, the tolerance is ε=10−6, and the learning rate is α = 0.001.


### 3.5. Nesterov-Accelerated Adaptive Moment Estimation

Nadam, which is the Nesterov-accelerated adaptive moment estimation, combines Adam and NAG, which is the Nesterov acceleration gradient [12]. The Nadam algorithm is employed for noisy gradients or gradients with high curvatures. The NAG algorithm allows performing a more accurate step in the gradient direction by updating the parameters with the momentum step before computing the gradient. The learning process is accelerated by summing up the exponential decay of the moving averages for the previous and current gradients. It is resulting in a little faster training time than the Adam algorithm. Its updating rule is shown by(29)wk+1=wk−αv^k+δβ1m^k−1+1−β11−β1k.gk.The calculation procedure of the Nadam algorithm is summarized as in Algorithm4.Algorithm 4:Nadam algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: update the vectorwk from (29). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


### 3.6. AMSGrad

In setting, where the Adam algorithm converges to a suboptimal solution, it has been observed that some minibatches provide large and informative gradients, but as these minibatches only occur rarely, exponential averaging diminishes their influence, which leads to poor convergence. To fix this behaviour, a new algorithm, which is known as the AMSGrad algorithm [13], that uses the maximum of past squared gradients rather than the exponential average to update the parameters is created:(30)v^k=maxv^k−1,vk.Therefore, the updating rule of AMSGrad is(31)wk+1=wk−αv^k+δ⋅mk.The calculation procedure of the AMSGrad algorithm is summarized as Algorithm5.Algorithm 5:AMSGrad algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected moment estimate based on (30).Step 6: update the vectorwk from (31). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


### 3.7. Improved Adaptive Moment Estimation Algorithm

Consider the standard error (SE) from the sampling theory [14],(32)SE=σn,where σ is the population standard deviation and n is the sample size for the sampling. Thus, for improving the updating rule of the Adam algorithm, assume that the standard error of the bias-corrected first-moment estimate m^k is defined by(33)s^k=v^k+δk,where v^k represents the sample standard deviation of the gradient gk and k is the number of iterations. From (26), the updating rule of the Adam algorithm is modified to be(34)wk+1=wk−α⋅m^ks^k.For the modification made, this improved Adam algorithm is also known as Adam with standard error (AdamSE) algorithm [11, 14].The calculation procedure for the AdamSE algorithm is summarized as Algorithm6.Algorithm 6:AdamSE algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: calculate the standard error of the bias-corrected first-moment estimate from (33).Step 7: update the vectorwk from (34). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001 as the same as in the Adam algorithm.


## 3.1. Analytical Optimal Solution

From (5), the optimal weighted value of the portfolio is calculated from(9)w∗=λ1Σ−1μ+λ2Σ−11.Refer to (6), the targeted expected return is provided by(10)R=μTΣ−1Σw∗.Then, substitute (9) into (10) to have the targeted expected return in terms of the Lagrange multipliers, that is,(11)R=λ1μTΣ−1μ+λ2μTΣ−11.Rewrite (7) to be(12)1=1TΣ−1Σw∗and substitute (9) into (12):(13)1=λ11TΣ−1μ+λ21TΣ−11.From (11) and (13), after doing some algebraic manipulations, the Lagrange multipliers are computed from(14)λ1=μTΣ−11−R1TΣ−111TΣ−1μμTΣ−11−μTΣ−1μ1TΣ−11,(15)λ2=R1TΣ−1μ−μTΣ−1μμTΣ−111TΣ−1μ−1TΣ−11μTΣ−1μ,and λ3=0. Therefore, from the discussion above, the analytical solution of the mean-variance portfolio optimization problem defined in (1), which is given by (9), (14), and (15), is resulted. However, this analytical solution is assumed to be not available due to the uncertainty and randomness of the variables.


## 3.2. Stochastic Gradient Descent Algorithm

Referring to the mean-variance portfolio optimization problem defined in (1), let us introduce an augmented objective function faw as(16)faw=12wTΣw+λ1R−wTμ+λ21−wT1+λ3Tw,with w=w1,…,wnT. Since the existence of the uncertainty, the augmented objective function faw defined in (16) can be rewritten as the expected objective function, given by(17)Efaw=1n∑i=1nfa,iw,where fa,iw,i=1,…,n, is the element of the augmented objective function faw that is uniformly sampled at random and E⋅ is the expectation operator. By virtue of this, the sampled gradient ∇fa,iw is denoted as an unbiased estimator to be(18)E∇faw=1n∑i=1n∇fa,iw.Notice that the first-order necessary condition for (16) is equivalent to the first-order necessary condition (5). That is,(19)∂faw∂w=∂Lw,λ∂w.For convenience, define the stochastic gradient(20)gw=∂L∂w,which can be calculated from (5). The updating rule of the SGD approach is given by(21)wk+1=wk−α⋅gjwk,with the step size α, which is also known as the learning rate, k is the number of iterations, and the random index j is the gradient referred.Hence, the calculation procedure of the SGD algorithm is summarized as in Algorithm1.Algorithm 1:SGD algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: update the vectorwk from (21). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The tolerance isε=10−6, and the learning rate is α = 0.001.


## 3.3. Adaptive Moment Estimation Algorithm

In the Adam approach [11], the exponential decaying averages of past gradients mk and past squared gradients vk are considered as follows:(22)mk=β1mk−1+1−β1gk,(23)vk=β2vk−1+1−β2gk2,where gk is the gradient, β1 and β2 are the decay rates, which are close to 1. Notice that mk and vk are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients, respectively. These biases are counteracted by using the bias-corrected first- and second-moment estimates, given by(24)m^k=mk1−β1k,(25)v^k=vk1−β2k.Thus, Adam updating rule has been presented as follows:(26)wk+1=wk−α⋅m^kv^k+δ,where δ is the smoothing term used to avoid division by zero.The calculation procedure of the Adam algorithm is summarized as in Algorithm2.Algorithm 2:Adam algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: update the vectorwk from (26). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


## 3.4. Adaptive Moment Estimation with Maximum

AdaMax, which is the adaptive moment estimation with maximum [11], is a variant of the Adam optimizer that uses the infinity (ℓ∞) norm, while the Adam optimizer itself uses the ℓ2-norm for optimization. When generalizing the Adam algorithm to the ℓ∞-norm, and hence in AdaMax, the gradient update is the maximum between the past gradients and current gradient, which is shown as(27)vk=β2∞vk−1+1−β2∞gk∞=maxβ2⋅vk−1,gk.Then, the updating rule of AdaMax is(28)wk+1=wk−αvk⋅m^k.The calculation procedure of the AdaMax algorithm is summarized as in Algorithm3.Algorithm 3:AdaMax algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (27).Step 5: calculate the bias-corrected first-moment estimate from (24).Step 6: update the vectorwk from (28). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, the tolerance is ε=10−6, and the learning rate is α = 0.001.


## 3.5. Nesterov-Accelerated Adaptive Moment Estimation

Nadam, which is the Nesterov-accelerated adaptive moment estimation, combines Adam and NAG, which is the Nesterov acceleration gradient [12]. The Nadam algorithm is employed for noisy gradients or gradients with high curvatures. The NAG algorithm allows performing a more accurate step in the gradient direction by updating the parameters with the momentum step before computing the gradient. The learning process is accelerated by summing up the exponential decay of the moving averages for the previous and current gradients. It is resulting in a little faster training time than the Adam algorithm. Its updating rule is shown by(29)wk+1=wk−αv^k+δβ1m^k−1+1−β11−β1k.gk.The calculation procedure of the Nadam algorithm is summarized as in Algorithm4.Algorithm 4:Nadam algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: update the vectorwk from (29). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


## 3.6. AMSGrad

In setting, where the Adam algorithm converges to a suboptimal solution, it has been observed that some minibatches provide large and informative gradients, but as these minibatches only occur rarely, exponential averaging diminishes their influence, which leads to poor convergence. To fix this behaviour, a new algorithm, which is known as the AMSGrad algorithm [13], that uses the maximum of past squared gradients rather than the exponential average to update the parameters is created:(30)v^k=maxv^k−1,vk.Therefore, the updating rule of AMSGrad is(31)wk+1=wk−αv^k+δ⋅mk.The calculation procedure of the AMSGrad algorithm is summarized as Algorithm5.Algorithm 5:AMSGrad algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected moment estimate based on (30).Step 6: update the vectorwk from (31). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001.


## 3.7. Improved Adaptive Moment Estimation Algorithm

Consider the standard error (SE) from the sampling theory [14],(32)SE=σn,where σ is the population standard deviation and n is the sample size for the sampling. Thus, for improving the updating rule of the Adam algorithm, assume that the standard error of the bias-corrected first-moment estimate m^k is defined by(33)s^k=v^k+δk,where v^k represents the sample standard deviation of the gradient gk and k is the number of iterations. From (26), the updating rule of the Adam algorithm is modified to be(34)wk+1=wk−α⋅m^ks^k.For the modification made, this improved Adam algorithm is also known as Adam with standard error (AdamSE) algorithm [11, 14].The calculation procedure for the AdamSE algorithm is summarized as Algorithm6.Algorithm 6:AdamSE algorithm.
Data: given the initial valuew0=w0, the number of samples n, the step size α, and the tolerance ε. Set k=0.Step 1: evaluate the augmented objective function from (16).Step 2: compute the stochastic gradient from (20).Step 3: set the random indexj.Step 4: compute the decaying averages of past and past squared gradients from (22) and (23).Step 5: calculate the bias-corrected first- and second-moment estimates from (24) and (25).Step 6: calculate the standard error of the bias-corrected first-moment estimate from (33).Step 7: update the vectorwk from (34). If wk+1−wk<ε, then stop the iteration. Otherwise, set k=k+1, and repeat from Step 1.Remark:The default values for the decay rates areβ1 = 0.9 and β2 = 0.999, and the smoothing term is δ=10−8, while the tolerance is ε=10−6, and the learning rate is α = 0.001 as the same as in the Adam algorithm.


## 4. Illustrative Example

Consider a portfolio optimization problem [15], where the portfolio selection is based on three securities, namely, S&P 500 stock, 10-year Treasury bond, and money market (MM). The corresponding historical data of the annual rate of return for these securities, which are dated from 1961 to 2003, are shown in Table 1.Table 1
Annual rate of return for the portfolio.
YearStockBondMM196126.812.202.331962−8.785.722.93196322.691.793.38196416.363.713.85196512.360.934.321966−10.105.125.40196723.94−2.864.50196811.002.256.031969−8.47−5.638.9619703.9418.924.90197114.3011.244.14197218.992.395.331973−14.693.309.951974−26.474.008.53197537.235.525.20197623.9315.564.651977−7.160.386.5619786.57−1.2610.03197918.611.2613.78198032.50−2.4818.901981−4.924.0412.37198221.5544.288.95198322.561.299.4719846.2715.298.38198531.7332.278.27198618.6722.396.9119875.25−3.036.77198816.616.848.76198931.6918.548.451990−3.107.747.31199130.4619.364.4319927.627.342.92199310.0813.062.9619941.32−7.325.45199537.5825.945.60199622.960.135.29199733.3612.025.50199828.5814.454.68199921.04−7.515.302000−9.1017.226.402001−11.895.511.822002−22.1015.151.24200328.680.540.98By using (2) and (3), the mean of the return and the related covariance of this portfolio selection are calculated, and their values are, respectively, given by(35)μ=0.10730.07370.0627andΣ=0.027780.003870.000210.003870.01112−0.000200.00021−0.000200.00115.Consequently, the mean-variance portfolio optimization model is formulated as follows:(36)minimizefw1,w2,w3=120.02778w12+20.00387w1w2+20.00021w1w3+0.01112w22+2−0.00020w2w3+0.00115w32,subject to0.1073w1+0.0737w2+0.0627w3=R,w1+w2+w3=1,w1,w2,w3≥0.with the initial weight w1=0.3, w2=0.3, and w3=0.4 and the targeted expected return. R=0.065.As a result, the optimal portfolio in percentage is shown in Table2, where the final iterative solutions obtained from the SGD, Adam, AdaMax, Nadam, AMSGrad, and AdamSE algorithms are compared. The optimal solution from [15] is enclosed as the benchmark solution. It could be noticed that these algorithms are able to provide the optimal weight for the portfolio selection, which is given by (2.63, 10.24, 87.13). The Lagrange multipliers are λ1=7.2×10−3 and λ2=5.3268×10−4.Table 2
Optimal portfolio percentage.
SolutionStockBondMMBenchmark2.63709710.22601687.136887SGD2.63039810.24402587.125477Adam2.63040010.24402387.125478AdaMax2.63064610.24393987.125515Nadam2.63039910.24402387.125479AMSGrad2.63040010.24402287.125479AdamSE2.63036110.24402487.125707The performance of the respective methods in solving the mean-variance portfolio optimization problem is shown in Table3, where the numbers of iterations are presented.Table 3
Performance of algorithms.
AlgorithmSGDAdamAdaMaxNadamAMSGradAdamSENumber of iterations22652242720822432273303Apparently, the AdamSE algorithm has the smallest number of iterations, which is an 86 percent reduction from the iteration numbers of the Adam algorithm, while the Adam algorithm leads the iterative solution to the convergence with 1 percent faster than the SGD algorithm. At the same time, some variants of Adam algorithm, namely, AdaMax, Nadam, and AMSGrad, show more iteration numbers than the Adam algorithm. Also, the convergence behaviour for each algorithm, which is represented by the norm of the stochastic gradient, is shown in Figures1–6, respectively. For better visualization, the iterative results for the first 300 iteration numbers of the SGD and AdaMax algorithms are presented as shown in Figures 1 and 3, respectively. Therefore, the modification of the Adam algorithm by equipping the standard error significantly enhances the rate of the convergence of the Adam algorithm and the efficiency of the AdamSE algorithm is definitely proven.Figure 1
Convergence behavior of SGD algorithm for the first 300 of 2265 iteration numbers.Figure 2
Convergence behavior of Adam algorithm.Figure 3
Convergence behavior of AdaMax algorithm for the first 300 of 7208 iteration numbers.Figure 4
Convergence behavior of Nadam algorithm.Figure 5
Convergence behavior of AMSGrad algorithm.Figure 6
Convergence behavior of AdamSE algorithm.In addition to this, the objective function has a minimum risk of5.0182×10−4, where the changes in the variance presented by each algorithm during the iteration procedure are shown in Figures 7–12, respectively. From these figures, it is noticed that the variance is dramatically increasing before meeting a peak point for the different algorithms, and then the variance is gradually decreasing and heading to the minimum variance of 5.0182 × 10−4. This behaviour indicates that the divergence issue has been counted when reaching a peak point, and after satisfying the constraints, the optimal solution is successfully determined to give the optimal weight of the portfolio selection.Figure 7
Changes in the variance of SGD algorithm.Figure 8
Changes in the variance of Adam algorithm.Figure 9
Changes in the variance of AdaMax algorithm.Figure 10
Changes in the variance of Nadam algorithm.Figure 11
Changes in the variance of AMSGrad algorithm.Figure 12
Changes in the variance of AdamSE algorithm.


## 5. Concluding remarks

The enhancement of the convergence rate of the Adam algorithm, which is addressed by using the standard error, was discussed in this paper. This improved version of the algorithm is known as the AdamSE algorithm. In addition, the application of the algorithms of SGD, Adam, AdaMax, Nadam, AMSGrad, and AdamSE in solving the mean-variance portfolio optimization problem was also studied. The result obtained showed that the AdamSE algorithm is an efficient approach, especially for solving the mean-variance portfolio optimization problem. In conclusion, the practicality of the SGD algorithm and its current variants, which are the Adam, AdaMax, Nadam, AMSGrad, and AdamSE algorithms, is particularly validated for the mean-variance portfolio optimization problem.



---

*Source: 8892636-2021-03-25.xml*