# Gradient-Sensitive Optimization for Convolutional Neural Networks

Authors: Zhipeng Liu; Rui Feng; Xiuhan Li; Wei Wang; Xiaoling Wu
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Convolutional neural networks (CNNs) are effective models for image classification and recognition. Gradient descent optimization (GD) is the basic algorithm for CNN model optimization. Since GD appeared, a series of improved algorithms have been derived. Among these algorithms, adaptive moment estimation (Adam) has been widely recognized. However, local changes are ignored in Adam to some extent. In this paper, we introduce an adaptive learning rate factor based on current and recent gradients. According to this factor, we can dynamically adjust the learning rate of each independent parameter to adaptively adjust the global convergence process. We use the factor to adjust the learning rate for each parameter. The convergence of the proposed algorithm is proven by using the regret bound approach of the online learning framework. In the experimental section, comparisons are conducted between the proposed algorithm and other existing algorithms, such as AdaGrad, RMSprop, Adam, diffGrad, and AdaHMG, on test functions and the MNIST dataset. The results show that Adam and RMSprop combined with our algorithm can not only find the global minimum faster in the experiment using the test function but also have a better convergence curve and higher test set accuracy in experiments using datasets. Our algorithm is a supplement to the existing gradient descent algorithms, which can be combined with many other existing gradient descent algorithms to improve the efficiency of iteration, speed up the convergence of the cost function, and improve the final recognition rate.


---

## Body

## 1. Introduction

As a basic technology of deep learning, convolutional neural networks (CNNs) have been widely used in many fields since their inception. CNNs play an important role in fields such as the classification of images [1, 2], the analysis of microexpressions [3], the recognition of faces [4], and the detection of objects [5]. Various improved CNN models have been applied to various images—basic CNN is a supervised learning algorithm. However, CNN’s outstanding image feature extraction ability has related problems [6–8]. This also prompted many scholars to conduct in-depth research. They proposed a method that combines a CNN with an algorithm that provides a classification basis to form an unsupervised learning algorithm such as CSFL, which has broader application prospects [9, 10].During the process of training the CNN model, we must identify the minimum value of the cost function. To this end, gradient descent optimization (GD) is the basic algorithm used [11–13]. Then, a series of improved algorithms are derived. Generally, these improved algorithms can be divided into two groups.The first group is represented by the momentum method [14, 15], in which the introduction of the momentum factor effectively alleviates the problem of large noise in each iterative calculation. This factor flattens the convergence curve and improves the convergence speed to a certain extent. Then, Nesterov’s acceleration [16, 17] further smooths and stabilizes the convergence curve by using the prediction of future updates in the iteration process.The other group includes the adaptive gradient algorithm (AdaGrad) [18] and root mean square prop algorithm (RMSprop) [19], which consider the cumulative changes of each parameter in their respective iterative optimization processes. Taking the cumulative changes into consideration, the step of the next iteration is adjusted. A smaller step is performed on those with more cumulative changes, while a larger step is performed on those with less cumulative changes. This kind of adaptive method makes iterative parameter optimization more targeted. AdaDelta is another adaptive algorithm that is executed without the use of the global learning rate. Its window restriction on the accumulation of historical gradients enabled AdaDelta to avoid the learning rate decrease inherent in the AdaGrad [20].Diederik P. Kingma designed a ground-breaking combination of these two ideas with his adaptive moment estimation algorithm (Adam) [21]. The learning rate of each parameter is adaptively adjusted by using the first-order moment estimation and second-order moment estimation of the gradient so that the parameter optimization process is stabilized and the convergence rate of the cost function is improved. The excellent performance of Adam has made it widely recognized. Since then, algorithms for CNNs have been improved mostly based on Adam. For example, Jun Hu improved the second-moment estimation of Adam based on hybrid power [22]. Dozat T. used Nesterov’s acceleration to improve the first-moment estimation of Adam [23]. H. Iiduka designed an iterative algorithm that combines the existing adaptive learning rate optimization algorithm with the conjugate gradient method. This paper considers the nonconvex optimization problem of deep neural networks and can be applied to the nonsmooth and convex optimization of deep neural networks [24]. However, these algorithms focus too much on the cumulative change in global gradients and ignore the importance of current gradients and recent gradients in the model optimization process. Sadaqat ur Rehman et al. proposed a modified resilient backpropagation (MRPROP) algorithm [25]. In the algorithm, an increase or decrease factor is given to the weight update values based on the sign of the current and recent gradients. The algorithm also solves the problem of CNN overtraining by introducing a tolerant band and a global best concept. They did not ignore the importance of current gradients and recent gradients, but they lacked attention to the cumulative change in global gradients. Shiv Ram Dubey et al. also improved Adam for this problem. These researchers introduced a new adaptive learning rate factor to adjust the step size according to the size of the gradient change and proposed the diffGrad algorithm [26]. In this approach, the adaptive learning rate factor provided by Adam regulates the global gradient, and the adaptive learning rate factor provided by the diffGrad algorithm adjusts the local gradient. The combination of the two adaptive learning rate factors has achieved good results.In this paper, we also pay attention to local gradients and introduce a new adaptive learning rate factor. According to the positive and negative signs of the current and recent gradients, we can judge whether the parameter is in the convergence stage of the cost function or if it is near the value that can achieve the minimum of the cost function. While the parameters are in the convergence stage of the cost function, a large learning rate is given to the parameter iteration so that the cost function converges quickly. While the parameter is near the value that can achieve the minimum of the cost function, a small learning rate is given to the parameter iteration.Due to the above idea, this paper introduces an adaptive learning rate factor related to the current and recent gradients. The main work of this paper can be summarized as follows:(1)
Based on Adam, we introduce an adaptive learning rate factor related to the current and recent gradients to optimize the CNN training process.(2)
We use an online learning framework to analyze the convergence of the proposed algorithm.(3)
Various test functions, such as Booth’s function, the Beale function, and the Styblinski–Tang function, are used to evaluate the convergence speed of the algorithm proposed in this paper. A comparison is made with Adam and diffGrad.(4)
We conduct a case study for the algorithm proposed in this paper. We compare the convergence speed and accuracy of the algorithm proposed in this paper with those of Adam and diffGrad on MNIST datasets.The structure of this paper is as follows: The second section introduces the preliminary work of optimizing the algorithm. The third section introduces the adaptive learning rate factor related to the current and recent gradients and introduces the method proposed in this paper. In the fourth section, the convergence of the algorithm is analyzed. The fifth section uses test functions to evaluate the performance of the proposed algorithm. The sixth section uses public datasets for empirical testing and gives the experimental results, comparison, and analysis. The seventh section is the discussion of the paper.


## 2. Preliminaries

The basic algorithm for CNN model optimization is GD. In GD, parameters are iterated as follows:(1)θt+1=θt−η∗gt,where θt+1 is the updated parameter value. θt is the parameter value before the update. η is the learning rate. gt is the gradient of the cost function of the parameter, which is defined as follows:(2)gt=∂Ek∂θt,where Ek is the cost function. When the output contains multiple parameters y1,y2,...,yk, it is usually expressed by the mean squared error of the output, which is defined as follows:(3)Ek=12∑i=1ky^i−yi2,In each iterative calculation, the gradient contains a relatively large amount of noise. Referring to the idea of momentum in physics, the momentum method introduces a variablev. By adding up all gradients after varying degrees of attenuation, gradients at all times are involved in the iterative operation of the parameters, which effectively alleviates the noise problem. In the momentum algorithm, the variable v and the parameters are iterated by the following rules:(4)vt=αvt−1−ηgt,(5)θt+1=θt+vt,where α is the attenuation coefficient and η is the learning rate.In addition to the noise of the gradient, GD uses the same global learning rate for all parameters in the CNN training process. The loss sight of difference between each parameter is clearly not perfect. Therefore, an adaptive learning rate algorithm arises.The AdaGrad algorithm sets a global learning rate. The actual learning rate corresponding to each parameter is inversely proportional to the accumulation of the square of the respective gradient. The specific iterative rules are as follows:(6)rt=rt−1+gt2,(7)θt+1=θt−ηδ+rtgt,where gt2 indicates the elementwise square gt and δ is a small value included to avoid division by zero.The accumulation of gradient squares causes the denominator in the middle and later stages to become too large, which leads to the premature end of the iterative updating procedure. To solve this problem, the RMSprop algorithm was developed, which no longer caused a simple summation of gradient squares. By introducing an attenuation coefficientρ, r is attenuated by a certain proportion during each iteration. The specific iterative rules are as follows:(8)rt=ρrt−1+1−ρgt2,(9)θt+1=θt−ηδ+rtgt,where gt2 indicates the elementwise square gt and δ is a small value included to avoid division by zero.Adam is a good combination of momentum and the RMSprop algorithm. The first- and second-moment estimators of the gradient are used to assign the corresponding adaptive learning rate to each parameter. At the same time, a bias correction is introduced creatively, which improves the training speed and makes the parameters more stable. The specific iterative method is as follows:(10)mt=β1mt−1+1−β1gt,(11)vt=β2vt−1+1−β2gt2,(12)mt^=mt1−β1t,(13)vt^=vt1−β2t,(14)θt=θt−1−η×mt^δ+vt^,where mt and vt are the first-order and second-order moment estimates of the gradient, respectively. gt2 indicates the elementwise square. mt^ and vt^ are the first-order and second-order moment estimates of the gradient after the bias correction, respectively. β1 and β2 are the attenuation coefficients. η is the learning rate. δ is a small value included to avoid division by zero.Then, according to the gradient difference, a new learning rate factor is introduced to improve Adam. Shiv Ram Dubey proposed the diffGrad method. According to their algorithm, the new learning rate factor and parameter iteration rules are as follows:(15)ξt=11+e−gt−1−gt,(16)θt+1=θt−η×ξt×m^tδ+v^t,where gt−1−gt is the difference between two successive gradients, and mt^ and vt^ are the first-order and second-order moment estimates of the gradient after the bias correction, respectively. η is the learning rate. δ is a small value included to avoid division by zero.All algorithms above contain the learning rateη. η∈10−4,10−2 is a good choice in most cases. Attenuation coefficients β1=0.9 and β2=0.999 are recommended in Diederik P. Kingma’s paper [21].


## 3. Proposed Optimization

It can be seen from the second section that the learning rate plays an important role in the iteration of the parameters in GD. A larger learning rate yields a larger step size for parameter iteration, as well as a larger risk of missing the optimal solution. A smaller learning rate can achieve more refined learning. However, a smaller learning rate results in a slower convergence speed [27]. Therefore, the main developmental trend of GD is the adaptive learning rate. By giving each parameter dimension an adaptive learning rate, each parameter can find the optimal solution more quickly and realize the convergence of the cost function.Beginning with AdaGrad, numerous adaptive algorithms are increasingly being developed. Almost all of these algorithms produce the adaptive learning rate factor by global estimation of the gradient, but they ignore the local gradient to some extent. In this paper, we introduce an adaptive learning rate factor to further improve the attention given to the current and recent gradients. At the same time, the basic algorithm Adam ensures the control of the global gradient.The essence of GD used in CNN model training is its idea of multiobjective optimization. The sign of the gradient implies much information. As shown in Figure1, during parameter iteration, while the learning rate is too small, the gradients of the objective function are continuous with the same sign, and the parameter is located at a falling edge or rising edge. Therefore, we can use a larger learning rate for parameter iteration. Similarly, while the learning rate is too large, positive and negative alternations occur in the gradients, which means that the objective function may hover around a minimum value. Therefore, we can use a smaller learning rate for parameter iteration.Figure 1
(a) Parameter iteration process with a small learning rate. (b) Parameter iteration process with a large learning rate.
(a)(b)Based on the above idea, we introduce a new adaptive learning rate factor, which can be defined as follows:(17)λt=siggt⊙gt−13,where gt and gt−1 are two successive gradients.The sig function is similar to the sigmoid function. We can increase the learning rate when two gradients possess the same sign and decrease the learning rate when two gradients possess different signs by using an adaptive learning rate factor. Through the introduction of the operator, we can effectively reduce the serious oscillation of parameters in the iterative process due to the large step size. When the step size is small, it can also effectively accelerate the convergence speed. Finally, the degree of step size adjustment is also affected by the specific gradient value of the image, which has stronger adaptability. The specific definition of the function is as follows:(18)sigx=21+p−x.The characteristic image of the sig function is shown in Figure2. We can find that the sig function is a monotonically increasing function. The rising edge of the sig function becomes steeper as the base p increases. When the value of x is negative, the value range of the sig function is0,1; when the value of x is positive, the value range of the sig function is 1,2; and when the value of x is zero, the value of the sig function is 1.Figure 2
Characteristic graphs of sig functions with different basesp.We use the above adaptive factorλt to adjust the learning rate for the parameter. Then, we can obtain the effect shown in Figure 3.Figure 3
(a) Comparison of the parameter iteration process with and without the adaptive factor while the learning rate is small. (b) Comparison of the parameter iteration process with and without the adaptive factor while the learning rate is large.
(a)(b)In this paper, the proposed method can be combined with Adam. We still use the adaptive learning rate factor constructed by the offset-corrected first-order moment and the offset-corrected second-order moment to control the global gradient. However, to control the local gradient, a new adaptive learning rate factor is introduced. Therefore, in the proposed method, the final iteration of the parameters is realized according to the following rule:(19)θt=θt−1−η×λt×mt^δ+vt^,where θt is the parameter value of the tth iteration. θt−1 is the parameter value of the t−1th iteration. η is the learning rate. λt is the new adaptive learning rate factor introduced in this paper. δ is a small value included to avoid division by zero. mt^ and vt^ are the first-order and second-order moment estimates of the gradient after the bias correction, respectively. When λt is equal to 1, the iterative rule of the algorithm proposed in this article is the same as that of Adam.


## 4. Convergence Analysis

For the establishment of our method, we suppose that the cost function possesses a convex property [28]. Then, we use an online learning framework to analyze the convergence of the proposed method in our article. Since an unknown and arbitrary sequence of convex cost function fθ is given, ftθt is the cost function at the tth iteration, and θt is the parameter at the tth iteration. We attempt to predict θt and evaluate the corresponding cost function ftθt at each time t. To solve the above, we use a regret function. We can assume that a globally optimal parameter θ∗ exists. Then, for a feasible set χ, we can use the globally optimal parameter θ∗ and predicted parameter θt to calculate the difference between the sum of ftθ∗ and ftθt for each of the iterations. Thus, the regret function can be defined as(20)Regretθ∗,ft=∑t=1Tftθt−f1:Tθ∗,where θ∗=argminθ∈χ∑t=1Tftθ. Then, we can find that our algorithm possesses a regret bound of οTT⟶∞. The proof is shown in the appendix. Normally, we analyze multivariate functions fθ,1,θ,2,...,θ,i instead of the univariate function fθ. To simplify the process of our proof, we use the following definition. We define gt,i≜∇ftθt,i, g1:t,i=g1,i,g2,i,...,gt,i, and γ≜β12/β2. Our theorem below holds when β1,t decays with the attenuation coefficient δ.Theorem 1.
In a smooth convex functionft, for all θ∈Rd, we make the following assumptions:(1)
The gradients of the functionft are bounded (∇ftθ2≤G, ∇ftθ∞≤G∞)(2)
The distances between anyθt calculated by the proposed algorithm are bounded (θn−θm2≤D2, θn−θm∞≤D∞ for any m,n∈1,2,...,T)(3)
γ≜β12/β2 and β12/β2<1 for β1,β2∈0,1(4)
β1,t=β1δt−1,δ∈0,1
Then, the algorithm achieves the following guarantee for allT≥1.(21)Regretθ∗,ft≤D22α1−β1∑i=1d1+p−gT,igT−1,i3Tv^T,i2+2+β1αG∞1−β11−β21−γ2∑i=1dg1:T,i2+D∞2G∞1−β22α∑i=1d11−β11−δ2.Since the data feature of dimension (d) is sparse and bounded, we can obtain∑i=1dg1:T,i2≪dG∞T and ∑i=1d1+p−gT,igT−1,i3Tv^T,i≪dG∞T. Since we use a decay β1,t=β1δt−1,δ∈0,1, we can obtain the third term of the above formula, which is irrelevant to the parameter T. Finally, we can achieve(22)Regretθ∗,ftT≪D2/2α1−β1dG∞T+2+β1αd1−β11−β21−γ2T+dD∞2G∞1−β22α1−β11−δ2T.All parameters exceptT are given constants, so we can obtain limT⟶∞Regretθ∗,ft/T=0.


## 5. Test Functions

The test function is used to test the algorithm’s ability to find the global minimum, which is consistent with the CNN model training process. In this section, we use several different forms of test functions to test the performance of the algorithm proposed in this paper. These algorithms share the same parametersβ1 and β2 fixed at 0.9 and 0.999, respectively. The global learning rate η is fixed at 0.01. The base p of the sig function is the natural constant e. The initial point of the test is set as the origin.


### 5.1. Booth’s Function

Booth’s function is a test function whose shape is similar to that of a plate. The specific expression of the function is as follows:(23)fx,y=x+2y−72+2x+y−5.The test interval of the function isx,y∈−10,10. The function possesses a global minimum f1,3=0. Figure 4 shows Booth’s function.Figure 4
Booth’s function.
(a)(b)The parameters are iterated 1,000 times. The results are shown in Figure5 and Table 1.Figure 5
Parametric iteration trajectories of each algorithm in Booth’s function test.
(a)(b)Table 1
Parameter change in each algorithm during the iteration process in Booth’s function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0, 0][0, 0][0, 0][0, 0][0, 0][0, 0]100[1.4058, 1.4764][0.2942, 0.263][0.1499, 0.134][1.2047, 3.2149][1.6262, 1.7724][1.0039, 2.9961]200[1.676, 1.8651][0.5888, 0.5271][0.2998, 0.2681][1.0501, 3.0501][1.7108, 2.1871][1, 3]300[1.7353, 2.0636][0.8743, 0.7857][0.4492, 0.4019][1.05, 3.05][1.6243, 2.3547][1, 3]400[1.717, 2.1872][1.1423, 1.0342][0.5977, 0.5352][1.05, 3.05][1.4945, 2.4908][1, 3]500[1.6712, 2.2771][1.3817, 1.2665][0.7445, 0.6677][1.05, 3.05][1.3777, 2.6111][1, 3]600[1.6179, 2.3495][1.5798, 1.4768][0.8889, 0.7989][1.05, 3.05][1.283, 2.7085][1, 3]700[1.5647, 2.4116][1.7253, 1.6603][1.0297, 0.9283][1.05, 3.05][1.2104, 2.7833][1, 3]800[1.5144, 2.4665][1.8123, 1.8155][1.1656, 1.0552][1.05, 3.05][1.1564, 2.8389][1, 3]900[1.468, 2.5156][1.8428, 1.945][1.2948, 1.1788][1.05, 3.05][1.1168, 2.8797][1, 3]1000[1.4255, 2.56][1.8267, 2.0551][1.4155, 1.2982][1.05, 3.05][1.0879, 2.9095][1, 3]Although AdaGrad, Adam, and diffGrad did not find the global minimum after 1,000 iterations, the trajectory of the parameter iteration is similar to the algorithm that has already found the global minimum, and more iterations help them complete the job. Compared to Adam, Adam improved by the algorithm proposed in this article performs better. After 1,000 iterations, the result is close to the global minimum. In test functions, there is little noise. RMSprop perform better than the other existing algorithms. As a result, RMSprop improved by the algorithm proposed in this article has the best performance.


### 5.2. Beale’s Function

The Beale function is another test function. The specific expression of the function is as follows:(24)fx,y=1.5−x+xy2+2.25−x+xy22+2.625−x+xy32.The test interval of the function isx,y∈−4.5,4.5. The function possesses global minimumf3,0.5=0. Figure 6 shows the Beale function.Figure 6
Beale’s function. The parameters are iterated 1,000 times. The results are shown in Figure7 and Table 2.
(a)(b)Figure 7
Parametric iteration trajectories of each algorithm in the Beale function test.
(a)(b)Table 2
Parameter change in each algorithm during the iteration process in the Beale function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0, 0][0, 0][0, 0][0, 0][0, 0][0, 0]100[0.183, 0.2357][0.8557, 0.7214][0.893, 0.6537][1.0131, 0.4524][1.9231, 0.1244][2.1743, 0.2187]200[0.2632, 0.3384][1.4836, 0.0146][1.5212, 0.0012][1.9186, 0.1407][2.2412, 0.2508][2.6504, 0.399]300[0.3236, 0.42][1.906, 0.1342][1.9331, 0.144][2.6845, 0.4158][2.3906, 0.308][2.8522, 0.4608]400[0.3737, 0.4935][2.1682, 0.2294][2.1876, 0.2366][2.9801, 0.5012][2.4847, 0.3426][2.9463, 0.4864]500[0.417, 0.5629][2.3421, 0.2933][2.3568, 0.2987][2.9918, 0.5041][2.5519, 0.3663][2.9824, 0.4956]600[0.4556, 0.6287][2.4663, 0.338][2.4781, 0.3421][2.9921, 0.5042][2.6035, 0.384][2.9935, 0.4984]700[0.4902, 0.6904][2.5602, 0.3704][2.57, 0.3738][2.9922, 0.5042][2.6449, 0.3977][2.9986, 0.4996]800[0.5217, 0.7464][2.6341, 0.395][2.6426, 0.3978][2.9922, 0.5042][2.6792, 0.4089][2.9999, 0.4999]900[0.5504, 0.7955][2.6941, 0.4141][2.7015, 0.4165][2.9922, 0.5042][2.7082, 0.4181][2.9998, 0.5]1000[0.5766, 0.8364][2.7435, 0.4294][2.7501, 0.4314][2.9922, 0.5042][2.7333, 0.4259][2.9999, 0.5]Similar to the test on Booth’s function, Adam and RMSprop improved by the algorithm proposed in this article have better performance. In the comparison of RMSprop and GS-RMSprop, as we expected, we can see that the parameter iteration using GS-RMSprop is more stable.


### 5.3. Styblinski–Tang Function

The Styblinski–Tang function can be used for multidimensional testing. Its two-dimensional expression is as follows:(25)fx,y=x4−16x2+5x+y4−16y2+5y2.The test interval of the function isx,y∈−5,5. The function possesses a global minimum f1,3=0. Figure 8 shows the Styblinski–Tang function.Figure 8
Styblinski–Tang function.
(a)(b)The parameters are iterated 1,000 times. The results are shown in Figure9 and Table 3.Figure 9
Parametric iteration trajectories of each algorithm in the Styblinski–Tang function test.
(a)(b)Table 3
Parameter change in each algorithm during the iteration process in the Styblinski–Tang function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0,0][0,0][0,0][0,0][0,0][0,0]100[−1.2098, 1.2098][−0.896, 0.896][−0.5919, 0.5919][−0.5941, 0.5941][−2.6916, 2.6916][−2.7698, 2.7698]200[−1.7251, 1.7251][−1.3934, 1.3934][−0.9353, 0.9353][−1.114, 1.114][−2.8835, 2.8835][−2.9034, 2.9034]300[−2.0724, 2.0724][−1.7744, 1.7744][−1.1952, 1.1952][−1.6207, 1.6207][−2.8977, 2.8977][−2.9035, 2.9035]400[−2.3214, 2.3214][−2.092, 2.092][−1.4119, 1.4119][−2.116, 2.116][−2.9011, 2.9011][−2.9035, 2.9035]500[−2.5013, 2.5013][−2.3606, 2.3606][−1.6018, 1.6018][−2.5901, 2.5901][−2.9023, 2.9023][−2.9035, 2.9035]600[−2.6297, 2.6297][−2.5784, 2.5784][−1.773, 1.773][−2.9033, 2.9033][−2.9029, 2.9029][−2.9035, 2.9035]700[−2.7197, 2.7197][−2.7372, 2.7372][−1.9306, 1.9306][−2.9035, 2.9035][−2.9031, 2.9031][−2.9035, 2.9035]800[−2.7815, 2.7815][−2.8339, 2.8339][−2.0769, 2.0769][−2.9035, 2.9035][−2.9033, 2.9033][−2.9035, 2.9035]900[−2.8233, 2.8233][−2.8805, 2.8805][−2.2129, 2.2129][−2.9035, 2.9035][−2.9034, 2.9034][−2.9035, 2.9035]1000[−2.8511, 2.8511][−2.8977, 2.8977][−2.339, 2.339][−2.9035, 2.9035][−2.9034, 2.9034][−2.9035, 2.9035]In the test of the Styblinski–Tang function, the parametric iteration trajectory is almost identical, but it has different speeds. Among them, Adam and RMSprop improved by the algorithm proposed in this paper still performed best.


## 5.1. Booth’s Function

Booth’s function is a test function whose shape is similar to that of a plate. The specific expression of the function is as follows:(23)fx,y=x+2y−72+2x+y−5.The test interval of the function isx,y∈−10,10. The function possesses a global minimum f1,3=0. Figure 4 shows Booth’s function.Figure 4
Booth’s function.
(a)(b)The parameters are iterated 1,000 times. The results are shown in Figure5 and Table 1.Figure 5
Parametric iteration trajectories of each algorithm in Booth’s function test.
(a)(b)Table 1
Parameter change in each algorithm during the iteration process in Booth’s function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0, 0][0, 0][0, 0][0, 0][0, 0][0, 0]100[1.4058, 1.4764][0.2942, 0.263][0.1499, 0.134][1.2047, 3.2149][1.6262, 1.7724][1.0039, 2.9961]200[1.676, 1.8651][0.5888, 0.5271][0.2998, 0.2681][1.0501, 3.0501][1.7108, 2.1871][1, 3]300[1.7353, 2.0636][0.8743, 0.7857][0.4492, 0.4019][1.05, 3.05][1.6243, 2.3547][1, 3]400[1.717, 2.1872][1.1423, 1.0342][0.5977, 0.5352][1.05, 3.05][1.4945, 2.4908][1, 3]500[1.6712, 2.2771][1.3817, 1.2665][0.7445, 0.6677][1.05, 3.05][1.3777, 2.6111][1, 3]600[1.6179, 2.3495][1.5798, 1.4768][0.8889, 0.7989][1.05, 3.05][1.283, 2.7085][1, 3]700[1.5647, 2.4116][1.7253, 1.6603][1.0297, 0.9283][1.05, 3.05][1.2104, 2.7833][1, 3]800[1.5144, 2.4665][1.8123, 1.8155][1.1656, 1.0552][1.05, 3.05][1.1564, 2.8389][1, 3]900[1.468, 2.5156][1.8428, 1.945][1.2948, 1.1788][1.05, 3.05][1.1168, 2.8797][1, 3]1000[1.4255, 2.56][1.8267, 2.0551][1.4155, 1.2982][1.05, 3.05][1.0879, 2.9095][1, 3]Although AdaGrad, Adam, and diffGrad did not find the global minimum after 1,000 iterations, the trajectory of the parameter iteration is similar to the algorithm that has already found the global minimum, and more iterations help them complete the job. Compared to Adam, Adam improved by the algorithm proposed in this article performs better. After 1,000 iterations, the result is close to the global minimum. In test functions, there is little noise. RMSprop perform better than the other existing algorithms. As a result, RMSprop improved by the algorithm proposed in this article has the best performance.


## 5.2. Beale’s Function

The Beale function is another test function. The specific expression of the function is as follows:(24)fx,y=1.5−x+xy2+2.25−x+xy22+2.625−x+xy32.The test interval of the function isx,y∈−4.5,4.5. The function possesses global minimumf3,0.5=0. Figure 6 shows the Beale function.Figure 6
Beale’s function. The parameters are iterated 1,000 times. The results are shown in Figure7 and Table 2.
(a)(b)Figure 7
Parametric iteration trajectories of each algorithm in the Beale function test.
(a)(b)Table 2
Parameter change in each algorithm during the iteration process in the Beale function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0, 0][0, 0][0, 0][0, 0][0, 0][0, 0]100[0.183, 0.2357][0.8557, 0.7214][0.893, 0.6537][1.0131, 0.4524][1.9231, 0.1244][2.1743, 0.2187]200[0.2632, 0.3384][1.4836, 0.0146][1.5212, 0.0012][1.9186, 0.1407][2.2412, 0.2508][2.6504, 0.399]300[0.3236, 0.42][1.906, 0.1342][1.9331, 0.144][2.6845, 0.4158][2.3906, 0.308][2.8522, 0.4608]400[0.3737, 0.4935][2.1682, 0.2294][2.1876, 0.2366][2.9801, 0.5012][2.4847, 0.3426][2.9463, 0.4864]500[0.417, 0.5629][2.3421, 0.2933][2.3568, 0.2987][2.9918, 0.5041][2.5519, 0.3663][2.9824, 0.4956]600[0.4556, 0.6287][2.4663, 0.338][2.4781, 0.3421][2.9921, 0.5042][2.6035, 0.384][2.9935, 0.4984]700[0.4902, 0.6904][2.5602, 0.3704][2.57, 0.3738][2.9922, 0.5042][2.6449, 0.3977][2.9986, 0.4996]800[0.5217, 0.7464][2.6341, 0.395][2.6426, 0.3978][2.9922, 0.5042][2.6792, 0.4089][2.9999, 0.4999]900[0.5504, 0.7955][2.6941, 0.4141][2.7015, 0.4165][2.9922, 0.5042][2.7082, 0.4181][2.9998, 0.5]1000[0.5766, 0.8364][2.7435, 0.4294][2.7501, 0.4314][2.9922, 0.5042][2.7333, 0.4259][2.9999, 0.5]Similar to the test on Booth’s function, Adam and RMSprop improved by the algorithm proposed in this article have better performance. In the comparison of RMSprop and GS-RMSprop, as we expected, we can see that the parameter iteration using GS-RMSprop is more stable.


## 5.3. Styblinski–Tang Function

The Styblinski–Tang function can be used for multidimensional testing. Its two-dimensional expression is as follows:(25)fx,y=x4−16x2+5x+y4−16y2+5y2.The test interval of the function isx,y∈−5,5. The function possesses a global minimum f1,3=0. Figure 8 shows the Styblinski–Tang function.Figure 8
Styblinski–Tang function.
(a)(b)The parameters are iterated 1,000 times. The results are shown in Figure9 and Table 3.Figure 9
Parametric iteration trajectories of each algorithm in the Styblinski–Tang function test.
(a)(b)Table 3
Parameter change in each algorithm during the iteration process in the Styblinski–Tang function test.
Iterations/methodsAdaGradAdamdiffGradRMSpropGS-AdamGS-RMSprop0[0,0][0,0][0,0][0,0][0,0][0,0]100[−1.2098, 1.2098][−0.896, 0.896][−0.5919, 0.5919][−0.5941, 0.5941][−2.6916, 2.6916][−2.7698, 2.7698]200[−1.7251, 1.7251][−1.3934, 1.3934][−0.9353, 0.9353][−1.114, 1.114][−2.8835, 2.8835][−2.9034, 2.9034]300[−2.0724, 2.0724][−1.7744, 1.7744][−1.1952, 1.1952][−1.6207, 1.6207][−2.8977, 2.8977][−2.9035, 2.9035]400[−2.3214, 2.3214][−2.092, 2.092][−1.4119, 1.4119][−2.116, 2.116][−2.9011, 2.9011][−2.9035, 2.9035]500[−2.5013, 2.5013][−2.3606, 2.3606][−1.6018, 1.6018][−2.5901, 2.5901][−2.9023, 2.9023][−2.9035, 2.9035]600[−2.6297, 2.6297][−2.5784, 2.5784][−1.773, 1.773][−2.9033, 2.9033][−2.9029, 2.9029][−2.9035, 2.9035]700[−2.7197, 2.7197][−2.7372, 2.7372][−1.9306, 1.9306][−2.9035, 2.9035][−2.9031, 2.9031][−2.9035, 2.9035]800[−2.7815, 2.7815][−2.8339, 2.8339][−2.0769, 2.0769][−2.9035, 2.9035][−2.9033, 2.9033][−2.9035, 2.9035]900[−2.8233, 2.8233][−2.8805, 2.8805][−2.2129, 2.2129][−2.9035, 2.9035][−2.9034, 2.9034][−2.9035, 2.9035]1000[−2.8511, 2.8511][−2.8977, 2.8977][−2.339, 2.339][−2.9035, 2.9035][−2.9034, 2.9034][−2.9035, 2.9035]In the test of the Styblinski–Tang function, the parametric iteration trajectory is almost identical, but it has different speeds. Among them, Adam and RMSprop improved by the algorithm proposed in this paper still performed best.


## 6. Testing Dataset

In the test on the dataset, we use the simplest but most classical CNN architecture, LeNet-5 [29]. The network consists of two convolution layers, two pooling layers, and one fully connected layer. The convolution layer uses 5∗5 convolution kernels, and the pooling layer uses the 2∗2 max pooling method. The MNIST dataset is composed of 60,000 training samples and 10,000 testing samples, all of which are 28∗28-pixel handwritten digits [30]. The excellent classification ability of CNNs on the MNIST dataset has drawn wide attention from the public. We still use the MNIST dataset for testing. To speed up the training process, the test is based on a minibatch gradient descent [31, 32]. We test AdaGrad, RMSprop, Adam, diffGrad, AdaHMG and improved Adam, and RMSprop using the algorithm proposed in this paper.Unlike test functions, the MNIST dataset is more realistic and complex. Different basesp may have different effects on the results. Therefore, we choose 5, 10, and 15 as the bases p to test the effect of different sizes of p on the algorithm. The results are shown in Figure 10.Figure 10
Convergence curves of the cost function in the CNN model training process using Adam with the adaptive learning rate factor proposed in this article with different basesp of the sig function.We can see in the figure that the convergence effect increases with increasing basep. However, when p increases from 10 to 15, the improvement of the convergence effect becomes less obvious. Therefore, when compared with other algorithms, we choose 15 as the base p.The convergence curves of the cost function under these algorithms and the classification error rates of the models on the verification set after training are compared. Since there are many random numbers in the initialization of the CNN, five groups of tests are performed for each algorithm in this paper.The convergence curves of the cost function in the CNN model training process are shown in Figure11. In these tests, we use the following parameters: the global learning rate η is 0.01, the batch size is 50, and the number of epochs is 3. Every 100 iterations, we average the value of the cost function.Figure 11
Convergence curves of the cost function in the CNN model training using seven algorithms. (a) The test of group 1. (b) The test of group 2. (c) The test of group 3. (d) The test of group 4. (e) The test of group 5.
(a)(b)(c)(d)(e)Since the core of our algorithm introduces a new adaptive learning rate factor, the algorithm proposed in this article can be combined with other gradient descent algorithms to improve its performance. Combined with the algorithm proposed in this paper, the performance of Adam and RMSprop is improved. In the training process of CNN, the convergence curves of the improved algorithms are below those of the original algorithm, indicating that these algorithms have a faster convergence speed. Among them, RMSprop improved by our algorithm has the best performance.Table4 and Figure 12 show the error rates of sample classification for each model on the testing data after training.Table 4
Error rates of sample classification on the testing data.
Group/algorithm12345AdaGrad0.03840.03960.03510.04760.0424Adam0.01700.01730.0158001880.0187RMSprop0.02230.02170.02760.01980.0191diffGrad0.02130.02560.02350.02240.0269AdaHMG0.01860.01810.02140.02190.0191GS-Adam0.01500.01700.01690.01610.0168GS-RMSprop0.01510.01570.01560.01640.0165The lowest error rate in each test is indicated in bold.Figure 12
Error rates of sample classification on the testing data.The classification error rate of the CNN model trained by the improved Adam and improved RMSprop is lower than that of the model trained by the original algorithm. Adam and RMSprop improved by the algorithm proposed in this article have the lowest error rate in terms of test set classification, which is superior to other existing algorithms.


## 7. Discussion and Conclusions

In this paper, we propose a new optimization method. The architecture of CNN we used is LeNet-5. Our method is not an improvement of the CNN based on its architecture. Therefore, the complexity of our algorithm is the same as that of LeNet-5. We also compared the computation time. Due to the introduction of a new adaptive learning rate, several parameter operations were added. The time consumption of each iteration was approximately 15% higher than that of Adam and RMSprop. By introducing a new adaptive learning rate factor based on the current and recent gradients, we not only control global gradient changes well but also address local gradient changes. When the sign of the gradient remains the same, it indicates that the parameter is located at the falling or rising edge of the objective function. A larger step size helps to find the minimum value of the objective function more quickly. In this case, the current and recent gradient values provide an adaptive learning rate adjustment factor greater than 1 through our algorithm to increase the learning rate of the corresponding parameters. When the sign of the gradient value alternates between positive and negative, it indicates that the parameter oscillates near the minimum value of the objective function. A smaller learning step helps the objective function converge to the minimum value. At this time, the current and recent gradient values provide an adaptive learning rate adjustment factor greater than 0 and less than 1 through our algorithm to decrease the learning rate of the corresponding parameters. Our algorithm is a supplement to the existing algorithms, which can be combined with many other existing algorithms. The results show that Adam and RMSprop combined with our algorithm have better performance. Our method can not only improve the convergence speed of the original algorithm but also achieve higher accuracy in the classification of test sets. Although our algorithm performs well in the experiment, it has a flaw. In the introduction of our algorithm, we know that while the gradient exhibits the phenomenon of positive and negative alternation, we use a smaller learning rate for training. This may increase the risk of falling into a local minimum. For further research, we can use current and recent gradients to construct other adaptive learning rate factor calculation functions so that the local gradients can play their proper role in the CNN training process.



---

*Source: 6671830-2021-03-23.xml*