# AdaCN: An Adaptive Cubic Newton Method for Nonconvex Stochastic Optimization

Authors: Yan Liu; Maojun Zhang; Zhiwei Zhong; Xiangrong Zeng
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

In this work, we introduce AdaCN, a novel adaptive cubic Newton method for nonconvex stochastic optimization. AdaCN dynamically captures the curvature of the loss landscape by diagonally approximated Hessian plus the norm of difference between previous two estimates. It only requires at most first order gradients and updates with linear complexity for both time and memory. In order to reduce the variance introduced by the stochastic nature of the problem, AdaCN hires the first and second moment to implement and exponential moving average on iteratively updated stochastic gradients and approximated stochastic Hessians, respectively. We validate AdaCN in extensive experiments, showing that it outperforms other stochastic first order methods (including SGD, Adam, and AdaBound) and stochastic quasi-Newton method (i.e., Apollo), in terms of both convergence speed and generalization performance.


---

## Body

## 1. Introduction

Stochastic gradient descent (SGD) [1] is the workhorse method for nonconvex stochastic optimization in machine learning, particularly for training deep neural networks (DNNs). During the last decades, many accelerated first order variants of SGD are widely used due to their simplicity and versatility, including the accelerated SGD (ASGD) methods using Nesterov scheme [2], momentum [3] and heavy-ball method [4], and the adaptive methods such as AdaGrad [5], AdaDelta [6], RMSProp [7], and Adam [8]. Recently, Adam has become the default optimization method for many deep learning applications because of its rapid convergence speed and relatively insensitive choices of hyperparameters [9], and it also has engendered an ever-growing list of modifications, such as AdamW [10], NosAdam [11], AMSGrad [12], AdaBound [13], Radam [14], Sadam [15], and Adax [16], to name a few. The main difference between the ASGD methods and the adaptive methods is the former scales the gradients in different directions uniformly while the latter uses adaptively element-wisely scaled learning rates, which usually causes that the latter is able to converge faster and less sensitive to the learning rate than the former. However, it has been observed that the adaptive methods may converge to bad/suspicious local optima, leading to worse generalization ability than the ASGD methods [17], or fail to converge because of unstable and extreme learning rates [13].The abovementioned methods, belonging to the stochastic first order method family, only use gradient information and do not consider the curvature of the loss landscape, thereby leading to their suboptimal behavior in algorithmic iterations. Instead, existing stochastic second order methods can capture and exploit the curvature properties of the loss landscape by incorporating both gradient and Hessian information. For example, stochastic Newton methods are typical ones that adopt exact stochastic Hessians. However, computing the full Hessian for training large-scale DNNs is prohibitively expensive, and thus it is necessary to approximate it or avoid directly computing it in algorithmic iterations. According to the way of approximating the stochastic Hessian matrix, stochastic second order methods for training large-scale DNNs can be broadly categorized into two branches: the stochastic quasi-Newton (SQN) methods approximate the Hessian as a series sum of first order information from prior iterations, such as AdaQN [18], SdLBFGS [19], and Apollo [20]; the stochastic second order Hessian-free methods compute the Hessian-vector product exactly through an efficient procedure proposed in [21], such as AdaHessian [22] which approximates the Hessian diagonal using Hutchinson’s method based on the Hessian-vector product, which is significantly more costly than SQN methods.Recently, based on a classic method in the nonstochastic setting, the cubic regularized Newton method [23], the stochastic adaptive regularization methods using cubics (SARC) [24–26] are proposed to address relatively small-scale nonconvex stochastic optimization problems, and they find the minimizer of a local second order Taylor approximation with a cubic regularization term at each iteration. It is observed that the SARC methods are able to escape saddle points more efficiently, leading to better generalization performance than most of the abovementioned stochastic first order and second order methods [23, 27] in relatively small-scale machine learning problems. More recently, a variant of SARC combining with negative curvature (SANC) is proposed in [28] with even better generalizability since a direction of negative curvature also benefits escaping strict saddle points [29, 30]. Unlike previous SARC methods, the SANC uses independent sets of data points of consistent size over all iterations to attain stochastic gradient and Hessian estimators, making it more practical than SARC. However, SARC and SANC use a Krylov subspace method to iteratively solve a cubic regularized Newton subproblem and use a trust region-like scheme to determine if an iteration is successful or not and update only when it is successful, which hinders their applications in large-scale nonconvex stochastic optimization (in the sense of large datasets and/or high-dimensional parameter spaces). Therefore, these existing cubic regularized Newton methods are not suitable for training large-scale DNNs.In this work, we develop a novel adaptive cubic Newton method, AdaCN, for large-scale nonconvex stochastic optimization, and it can inherit the superiority of SARC and SANC methods (i.e., great generalizability), but tackle their aforementioned challenges (i.e., unsuitable for training large-scale DNNs). AdaCN is designed for nonconvex stochastic optimization through dynamically capturing the curvature of the loss function by diagonally approximated Hessian and the norm of difference between previous two estimates. It only requires at most first order gradients and updates with linear complexity for both time and memory, thus it is quite suitable for large-scale nonconvex stochastic optimization problems. In order to reduce the variance introduced by the stochastic nature of the problem, AdaCN hires the first and second moment to implement an exponential moving average on iteratively calculated gradients and approximated Hessians, respectively. Therefore, these moments are able to not only accelerate convergence speed but also smooth the noisy curvature information and get an approximation to the global curvature information, avoiding misleading local gradient and Hessian information which can be catastrophic. The superiority of AdaCN can be illustrated with two simple 2D functions (one is convex and the other is nonconvex, and these two functions give hints to local behavior of optimizers in deep learning), as shown in Figure1, where we show the trajectories of different optimizers. As can be seen, AdaCN can converge much faster than stochastic first order methods such as SGD and Adam and stochastic quasi-Newton method Apollo. Furthermore, we will experimentally show the superiority of AdaCN through image classification tasks: LeNet [31] on Mnist and VGG11 [32], ResNet34 [33], and DenseNet121 [34] on CIFAR10 and CIFAR100 [35] dataset; and language modeling task: LSTM [36] on Penn Treebank [37] dataset.Figure 1
Trajectories of SGD, Adam, Apollo, and AdaCN on a convex function (a) and a nonconvex function (b), respectively. Parameters of two experiments are set as follows: the learning rates of AdaCN, Apollo, Adam, and SGD are set to2×10−3, 2×10−3, 3×10−2, and 5×10−4, respectively. For SGD and Apollo, β=0.9, whereas for AdaCN and Adam, β1=0.9 and β2=0.9. The model is trained for 2.5×103 epochs. (a) Loss function is fx,y=x+y2+x−y2/10. (b)fx,y=x−12+100y−x22.
(a)(b)Notation: we use italics lettersα, β to denote scalars, bold lowercase letters x, y to denote vectors, and bold uppercase letters A, B to denote matrices. For vectors, we use ⋅ to denote the l2-norm.


## 2. Formulation of Adaptive Cubic Newton Method

### 2.1. Problem Statement

In this paper, we consider the following stochastic optimization problem:(1)minx∈ℝdfx=Eξ∼Dfx;ξ,where f is a continuously differentiable function and possibly nonconvex, x∈ℝd is the parameter to be optimized, and E denotes the expectation with respect to x, a random variable with the distribution D. A special case of (1) that arises frequently in supervised machine learning is the empirical risk minimization (ERM) problem [38]:(2)minx∈ℝdfx=1n∑i=1nfix;ξi,where fi:ℝd⟶ℝ is the loss function corresponding to the i-th data instance, and n is the number of data samples that is assumed to be extremely large.


### 2.2. Newton Update from Cubically Regularized Model

We begin with a stochastic Newton (SN) method. At a high level, we sample two independent mini-batchesSg and SB at each iteration, and the stochastic gradient and Hessian estimators, say, gk and Bk, can be defined as(3)gk=1Sg∑i∈Sg∇fix;ξi,Bk=1SB∑i∈SB∇2fix;ξi.In each iteration of the SN method, gradient descent finds the minimizer of a local second order Taylor expansion(4)xk+1=argminxfxk+gkTx−xk+12x−xkTBkx−xk,and its corresponding Newton update is shown as(5)xk+1=xk−Bk−1gk.For large-scale stochastic optimization problems, the stochastic HessianBk is properly approximated. For example, AdaQN [18], SdLBFGS [19], and Apollo [20] (belonging to stochastic quasi-Newton methods) approximate Bk as a series sum of first order information from prior iterations, while AdaHessian [22] (one of the stochastic second order Hessian-free methods) approximates the Hessian diagonal using Hutchinson’s method [39] based on the Hessian-vector product.A new principled variant of the SN method that could enjoy global convergence guarantees is proposed in [23], and it finds the minimizer of the following cubically regularized second order approximation of f with gk, Bk and a sufficient large ρ>0:(6)xk+1=argminxfxk+gkTx−xk+12x−xkTBkx−xk+ρ6x−xk3,where xk is the value at k-th iteration. By first order optimality conditions, we set the derivative of the objective to zero, which immediately yields(7)gk+Bkx−xk+ρ2x−xkx−xk=0,which is a nonlinear system and can be approximated by a linear one as follows:(8)gk+Bkx−xk+ρ2xk−xk−1x−xk=0,yielding a novel Newton update:(9)xk+1=xk−Bk+ρ2xk−xk−1⋅I−1gk,where I represents the identity matrix with the same size as Bk. Comparing (5) with (9) additionally makes use of the norm of difference between previous two estimates, leading to better performance since it captures more curvature information.


### 2.3. UpdatingBk

The stochastic HessianBk can be updated based on the weak secant equation [40, 41]:(10)Bk=argminBB−Bk−1,s.t.skTBsk=skTykweak secant equation,where sk=xk−xk−1 and yk=gk−gk−1. The solution of the above problem with Frobenius matrix based on the variational technique in [42] is given by(11)Bk=Bk−1+skTyk−skTBk−1sksk44+εDiagsk2,where Diagv is the diagonal matrix with diagonal elements from vector v. Further, with rectifying operation which guarantees the positive-definiteness, the updated cubic Newton becomes(12)xk+1=xk−Dk−1gk,where(13)Dk=maxabsBk+ρ2xk−xk−1⋅I,θ⋅I,where θ is a positive parameter, and the cost of computing D is marginal since Bk is diagonal, absV takes absolute values of all the elements of the matrix V. It is worth mentioning that, according to equations (5) and (12), max⋅,⋅ operation enables Dk to prevent the step size from becoming arbitrary large since there exists zero value in Bk.


### 2.4. Moments forgk and Dk

In this paper, we adopt the moments forgk and Dk, given by(14)mk⟵β1mk−1+1−β1gk,Vk⟵β2Vk−1+1−β2Dk⊙Dk,where ⊙ denotes the elementwise multiplication, and β1,β2∈0,1 are the first and second moment hyperparameters that are also used in Adam [8] and its many variants. The moments are further bias corrected as(15)m^k⟵mk1−β1k,V^k⟵Vk1−β2k.Using the first and second moment amounts to carrying out an exponential moving average on iteratively updated stochastic gradients and approximated stochastic Hessians plus the norm of difference between previous two estimates, which can smooth the noisy curvature information and get an approximation to the global curvature information, avoiding misleading local gradient and Hessian information which can be catastrophic.To summarize, the complete algorithm of AdaCN is given in Algorithm1, in which at most first order gradients are required, and Bk, Dk, Vk, and V^k are all diagonal. Therefore, AdaCN updates with linear complexity for both time and memory.Algorithm 1: AdaCN.
Require:ng //Mini-batch sizeRequire:η //StepsizeRequire:β1,β2∈0,1//Parameters of exponential moving averageRequire:ε,ρ,θ//Positive parametersRequire:x0,g0,B0,m0,V0//Initialize variables as zero vectors or zero matricesRequire:k⟵0//Initialize timestep(1)
Whilexk not converged do(2)
k⟵k+1(3)
sampleSg⟵ξii=1ng(4)
gk⟵1/Sg∑i∈Sg∇fix,ξi//Stochastic gradient at timestep k(5)
sk⟵xk−xk−1(6)
yk⟵gk−gk−1(7)
Bk⟵Bk−1+skTyk−skTBksk/sk44+εDiagsk2//Update diagonal Hessian(8)
Dk⟵maxBk+ρ/2xk−xk−1⋅I,θ(9)
mk⟵β1mk−1+1−β1gk(10)
Vk⟵β2Vk−1+1−β2Dk⊙Dk(11)
m^k⟵mk/1−β1k(12)
V^k⟵Vk/1−β2k(13)
xk+1⟵xk−ηV^k−1/2m^k(14)
end while(15)
returnxk+1


## 2.1. Problem Statement

In this paper, we consider the following stochastic optimization problem:(1)minx∈ℝdfx=Eξ∼Dfx;ξ,where f is a continuously differentiable function and possibly nonconvex, x∈ℝd is the parameter to be optimized, and E denotes the expectation with respect to x, a random variable with the distribution D. A special case of (1) that arises frequently in supervised machine learning is the empirical risk minimization (ERM) problem [38]:(2)minx∈ℝdfx=1n∑i=1nfix;ξi,where fi:ℝd⟶ℝ is the loss function corresponding to the i-th data instance, and n is the number of data samples that is assumed to be extremely large.


## 2.2. Newton Update from Cubically Regularized Model

We begin with a stochastic Newton (SN) method. At a high level, we sample two independent mini-batchesSg and SB at each iteration, and the stochastic gradient and Hessian estimators, say, gk and Bk, can be defined as(3)gk=1Sg∑i∈Sg∇fix;ξi,Bk=1SB∑i∈SB∇2fix;ξi.In each iteration of the SN method, gradient descent finds the minimizer of a local second order Taylor expansion(4)xk+1=argminxfxk+gkTx−xk+12x−xkTBkx−xk,and its corresponding Newton update is shown as(5)xk+1=xk−Bk−1gk.For large-scale stochastic optimization problems, the stochastic HessianBk is properly approximated. For example, AdaQN [18], SdLBFGS [19], and Apollo [20] (belonging to stochastic quasi-Newton methods) approximate Bk as a series sum of first order information from prior iterations, while AdaHessian [22] (one of the stochastic second order Hessian-free methods) approximates the Hessian diagonal using Hutchinson’s method [39] based on the Hessian-vector product.A new principled variant of the SN method that could enjoy global convergence guarantees is proposed in [23], and it finds the minimizer of the following cubically regularized second order approximation of f with gk, Bk and a sufficient large ρ>0:(6)xk+1=argminxfxk+gkTx−xk+12x−xkTBkx−xk+ρ6x−xk3,where xk is the value at k-th iteration. By first order optimality conditions, we set the derivative of the objective to zero, which immediately yields(7)gk+Bkx−xk+ρ2x−xkx−xk=0,which is a nonlinear system and can be approximated by a linear one as follows:(8)gk+Bkx−xk+ρ2xk−xk−1x−xk=0,yielding a novel Newton update:(9)xk+1=xk−Bk+ρ2xk−xk−1⋅I−1gk,where I represents the identity matrix with the same size as Bk. Comparing (5) with (9) additionally makes use of the norm of difference between previous two estimates, leading to better performance since it captures more curvature information.


## 2.3. UpdatingBk

The stochastic HessianBk can be updated based on the weak secant equation [40, 41]:(10)Bk=argminBB−Bk−1,s.t.skTBsk=skTykweak secant equation,where sk=xk−xk−1 and yk=gk−gk−1. The solution of the above problem with Frobenius matrix based on the variational technique in [42] is given by(11)Bk=Bk−1+skTyk−skTBk−1sksk44+εDiagsk2,where Diagv is the diagonal matrix with diagonal elements from vector v. Further, with rectifying operation which guarantees the positive-definiteness, the updated cubic Newton becomes(12)xk+1=xk−Dk−1gk,where(13)Dk=maxabsBk+ρ2xk−xk−1⋅I,θ⋅I,where θ is a positive parameter, and the cost of computing D is marginal since Bk is diagonal, absV takes absolute values of all the elements of the matrix V. It is worth mentioning that, according to equations (5) and (12), max⋅,⋅ operation enables Dk to prevent the step size from becoming arbitrary large since there exists zero value in Bk.


## 2.4. Moments forgk and Dk

In this paper, we adopt the moments forgk and Dk, given by(14)mk⟵β1mk−1+1−β1gk,Vk⟵β2Vk−1+1−β2Dk⊙Dk,where ⊙ denotes the elementwise multiplication, and β1,β2∈0,1 are the first and second moment hyperparameters that are also used in Adam [8] and its many variants. The moments are further bias corrected as(15)m^k⟵mk1−β1k,V^k⟵Vk1−β2k.Using the first and second moment amounts to carrying out an exponential moving average on iteratively updated stochastic gradients and approximated stochastic Hessians plus the norm of difference between previous two estimates, which can smooth the noisy curvature information and get an approximation to the global curvature information, avoiding misleading local gradient and Hessian information which can be catastrophic.To summarize, the complete algorithm of AdaCN is given in Algorithm1, in which at most first order gradients are required, and Bk, Dk, Vk, and V^k are all diagonal. Therefore, AdaCN updates with linear complexity for both time and memory.Algorithm 1: AdaCN.
Require:ng //Mini-batch sizeRequire:η //StepsizeRequire:β1,β2∈0,1//Parameters of exponential moving averageRequire:ε,ρ,θ//Positive parametersRequire:x0,g0,B0,m0,V0//Initialize variables as zero vectors or zero matricesRequire:k⟵0//Initialize timestep(1)
Whilexk not converged do(2)
k⟵k+1(3)
sampleSg⟵ξii=1ng(4)
gk⟵1/Sg∑i∈Sg∇fix,ξi//Stochastic gradient at timestep k(5)
sk⟵xk−xk−1(6)
yk⟵gk−gk−1(7)
Bk⟵Bk−1+skTyk−skTBksk/sk44+εDiagsk2//Update diagonal Hessian(8)
Dk⟵maxBk+ρ/2xk−xk−1⋅I,θ(9)
mk⟵β1mk−1+1−β1gk(10)
Vk⟵β2Vk−1+1−β2Dk⊙Dk(11)
m^k⟵mk/1−β1k(12)
V^k⟵Vk/1−β2k(13)
xk+1⟵xk−ηV^k−1/2m^k(14)
end while(15)
returnxk+1


## 3. Experiments

In this section, we access the performance of AdaCN on learning tasks: image classification and language modeling, comparing with stochastic first order optimizers such as Adam [8], SGD [1], AdaBound [13], and stochastic second order optimizer Apollo [20], listed in Table 1. For image classification, we investigate models by LeNet [31] on Mnist and VGG11 [32], ResNet34, and DenseNet121 [34] on CIFAR10 and CIFAR100 [35], while for language modeling, LSTM [36] on Penn Treebank [37] is tested. Moreover, the robustness to hyperparameters is tested, through comparing AdaCN and Apollo with different values of ε and learning rate. The results are shown in the following subsections.Table 1
Summaries of the settings used in experiments.
TaskDatasetNetwork typeArchitectureOptimizerImage classificationMnistConvolutional neural networkLeNetAdaCN, Apollo, Adam, SGD, AdaBoundCIFAR10CIFAR100VGG11, ResNet34, DenseNet121Language modelingPenn TreebankRecurrent1, 2, 3-layer LSTM


### 3.1. Experiments Setup

We perform a careful hyperparameter tuning in experiments as follows. AdaCN: we setθ=1, β1=0.9, β2=0.999, ε=10−8, and ρ=5×0.9200−k at k-th iteration. The learning rate η=0.4 for Mnist dataset, 0.2 for CIFAR datasets, and 30 for Penn Treebank dataset. SGD: the momentum is set to 0.9, while the learning rate is searched among a×10b where a∈1,2,3,4,5,6,7,8,9 and b∈−3,−2,−1,0,1. Adam, AdaBound, and Apollo: the learning rate is searched as SGD, and other parameters are set as their own default values in the literature.


### 3.2. Image Classification

At first, we evaluate the convergence and generalization of AdaCN on image classification. We use LeNet on Mnist and VGG11, ResNet34, and DenseNet121 on CIFAR10 and CIFAR100 dataset.Mnist. Results on Mnist are shown as Figure 2: the curves of train and test accuracy on Mnist and Table 2, from which we can see that AdaCN achieves the best convergence speed and classification accuracy.Figure 2
The curves of train and test accuracy on Mnist. (a) Train Accuracy. (b) Test Accuracy.
(a)(b)Table 2
Test accuracy of LeNet on Mnist.
ModelAdamSGDAdaBoundApolloAdaCNLeNet0.9840.9810.9870.9880.989The best result is shown in bold.CIFAR10. We report the results on CIFAR10 in Figure 3 and Table 3. For all three network architectures, AdaCN obviously outperforms other optimizers with comparable convergence speed and best classification accuracy.Figure 3
The curves of train and test accuracy on CIFAR10. (a) Train accuracy of VGG11 on CIFAR10. (b) Test accuracy of VGG11 on CIFAR10. (c) Train accuracy of ResNet34 on CIFAR10. (d) Test accuracy of ResNet34 on CIFAR10. (e) Train accuracy of DenseNet121 on CIFAR10. (f) Test accuracy of DenseNet121 on CIFAR10.
(a)(b)(c)(d)(e)(f)Table 3
Test accuracy of VGG11, ResNet34, and DenseNet121 on CIFAR10.
ModelAdamSGDAdaBoundApolloAdaCNVGG1188.4089.7290.6088.9091.34ResNet3493.0293.6294.7594.7995.15DenseNet12192.7993.0794.5894.8595.52The best results are shown in bold.CIFAR100. Results on CIFAR100 are shown in Figure 4 and Table 4. For VGG11, AdaCN is better than Adam and Apollo, but worse than SGD and AdaBound in terms of convergence speed and classification accuracy. For ResNet34 and DenseNet121, AdaCN achieves the best classification accuracy.Figure 4
The curves of train and test accuracy on CIFAR100. (a) Train accuracy of VGG11 on CIFAR100. (b) Test accuracy of VGG11 on CIFAR100. (c) Train accuracy of ResNet34 on CIFAR100. (d) Test accuracy of ResNet34 on CIFAR100. (e) Train accuracy of DenseNet121 on CIFAR100. (f) Test accuracy of DenseNet121 on CIFAR100.
(a)(b)(c)(d)(e)(f)Table 4
Test accuracy of VGG11, ResNet34, and DenseNet121 on CIFAR100.
ModelAdamSGDAdaBoundApolloAdaCNVGG1158.9666.0165.9258.5065.51ResNet3471.3976.9876.8876.4277.11DenseNet12171.8579.3579.0079.0179.50The best results are shown in bold.


### 3.3. Language Modeling

On language modeling, we experiment with 1, 2, 3-layer LSTM model on Penn Treebank dataset. As Figure5 and Table 5 show, AdaCN can also keep fastest convergence speed and achieve the lowest perplexity among the optimizers for 1, 2, 3-layer LSTM.Figure 5
The curves of test perplexity on Penn Treebank for 1, 2, 3-layer LSTM. Lower is better. (a) Test perplexity for 1-layer LSTM. (b) Test perplexity for 2-layer LSTM. (c) Test perplexity for 3-layer LSTM.
(a)(b)(c)Table 5
Test accuracy of 1, 2, 3-layer LSTM on Penn Treebank dataset.
ModelAdamSGDAdaBoundApolloAdaCN1-layer LSTM86.8686.1485.6884.9083.402-layer LSTM68.8168.5468.5869.7168.173-layer LSTM64.9164.5065.1866.3763.26The best results are shown in bold.Finally, we explore the effects of the hyperparameters includingε and learning rates on the performance of AdaCN, respectively. The results on CIFAR10 dataset are reported in Figures 6 and 7, where the values of ε are ranging from 10−3 to 10−7 in a log-scale grid on ResNet34 and learning rate ranging from 5×10−3 to 2×10−1 on VGG11, respectively. As can be seen, the test accuracies of AdaCN are above 95% for all values of ε, while Apollo achieves an accuracy consistently below 95%. Moreover, the results validate the robustness of AdaCN to ε and learning rate.Figure 6
The curves of train and test accuracy of ResNet34 on CIFAR10 with respect to different values ofε. (a) Train accuracy of AdaCN on CIFAR10. (b) Train accuracy of Apollo on CIFAR10. (c) Test accuracy of AdaCN on CIFAR10. (d) Test accuracy of Apollo on CIFAR10.
(a)(b)(c)(d)Figure 7
The curves of train and test accuracy of VGG11 on CIFAR10 with respect to different values of learning rate. As can be seen, AdaCN is more robust to learning rate. (a) Train accuracy of AdaCN on CIFAR10. (b) Train accuracy of Apollo on CIFAR10. (c) Test accuracy of AdaCN on CIFAR10. (d) Test accuracy of Apollo on CIFAR10.
(a)(b)(c)(d)


## 3.1. Experiments Setup

We perform a careful hyperparameter tuning in experiments as follows. AdaCN: we setθ=1, β1=0.9, β2=0.999, ε=10−8, and ρ=5×0.9200−k at k-th iteration. The learning rate η=0.4 for Mnist dataset, 0.2 for CIFAR datasets, and 30 for Penn Treebank dataset. SGD: the momentum is set to 0.9, while the learning rate is searched among a×10b where a∈1,2,3,4,5,6,7,8,9 and b∈−3,−2,−1,0,1. Adam, AdaBound, and Apollo: the learning rate is searched as SGD, and other parameters are set as their own default values in the literature.


## 3.2. Image Classification

At first, we evaluate the convergence and generalization of AdaCN on image classification. We use LeNet on Mnist and VGG11, ResNet34, and DenseNet121 on CIFAR10 and CIFAR100 dataset.Mnist. Results on Mnist are shown as Figure 2: the curves of train and test accuracy on Mnist and Table 2, from which we can see that AdaCN achieves the best convergence speed and classification accuracy.Figure 2
The curves of train and test accuracy on Mnist. (a) Train Accuracy. (b) Test Accuracy.
(a)(b)Table 2
Test accuracy of LeNet on Mnist.
ModelAdamSGDAdaBoundApolloAdaCNLeNet0.9840.9810.9870.9880.989The best result is shown in bold.CIFAR10. We report the results on CIFAR10 in Figure 3 and Table 3. For all three network architectures, AdaCN obviously outperforms other optimizers with comparable convergence speed and best classification accuracy.Figure 3
The curves of train and test accuracy on CIFAR10. (a) Train accuracy of VGG11 on CIFAR10. (b) Test accuracy of VGG11 on CIFAR10. (c) Train accuracy of ResNet34 on CIFAR10. (d) Test accuracy of ResNet34 on CIFAR10. (e) Train accuracy of DenseNet121 on CIFAR10. (f) Test accuracy of DenseNet121 on CIFAR10.
(a)(b)(c)(d)(e)(f)Table 3
Test accuracy of VGG11, ResNet34, and DenseNet121 on CIFAR10.
ModelAdamSGDAdaBoundApolloAdaCNVGG1188.4089.7290.6088.9091.34ResNet3493.0293.6294.7594.7995.15DenseNet12192.7993.0794.5894.8595.52The best results are shown in bold.CIFAR100. Results on CIFAR100 are shown in Figure 4 and Table 4. For VGG11, AdaCN is better than Adam and Apollo, but worse than SGD and AdaBound in terms of convergence speed and classification accuracy. For ResNet34 and DenseNet121, AdaCN achieves the best classification accuracy.Figure 4
The curves of train and test accuracy on CIFAR100. (a) Train accuracy of VGG11 on CIFAR100. (b) Test accuracy of VGG11 on CIFAR100. (c) Train accuracy of ResNet34 on CIFAR100. (d) Test accuracy of ResNet34 on CIFAR100. (e) Train accuracy of DenseNet121 on CIFAR100. (f) Test accuracy of DenseNet121 on CIFAR100.
(a)(b)(c)(d)(e)(f)Table 4
Test accuracy of VGG11, ResNet34, and DenseNet121 on CIFAR100.
ModelAdamSGDAdaBoundApolloAdaCNVGG1158.9666.0165.9258.5065.51ResNet3471.3976.9876.8876.4277.11DenseNet12171.8579.3579.0079.0179.50The best results are shown in bold.


## 3.3. Language Modeling

On language modeling, we experiment with 1, 2, 3-layer LSTM model on Penn Treebank dataset. As Figure5 and Table 5 show, AdaCN can also keep fastest convergence speed and achieve the lowest perplexity among the optimizers for 1, 2, 3-layer LSTM.Figure 5
The curves of test perplexity on Penn Treebank for 1, 2, 3-layer LSTM. Lower is better. (a) Test perplexity for 1-layer LSTM. (b) Test perplexity for 2-layer LSTM. (c) Test perplexity for 3-layer LSTM.
(a)(b)(c)Table 5
Test accuracy of 1, 2, 3-layer LSTM on Penn Treebank dataset.
ModelAdamSGDAdaBoundApolloAdaCN1-layer LSTM86.8686.1485.6884.9083.402-layer LSTM68.8168.5468.5869.7168.173-layer LSTM64.9164.5065.1866.3763.26The best results are shown in bold.Finally, we explore the effects of the hyperparameters includingε and learning rates on the performance of AdaCN, respectively. The results on CIFAR10 dataset are reported in Figures 6 and 7, where the values of ε are ranging from 10−3 to 10−7 in a log-scale grid on ResNet34 and learning rate ranging from 5×10−3 to 2×10−1 on VGG11, respectively. As can be seen, the test accuracies of AdaCN are above 95% for all values of ε, while Apollo achieves an accuracy consistently below 95%. Moreover, the results validate the robustness of AdaCN to ε and learning rate.Figure 6
The curves of train and test accuracy of ResNet34 on CIFAR10 with respect to different values ofε. (a) Train accuracy of AdaCN on CIFAR10. (b) Train accuracy of Apollo on CIFAR10. (c) Test accuracy of AdaCN on CIFAR10. (d) Test accuracy of Apollo on CIFAR10.
(a)(b)(c)(d)Figure 7
The curves of train and test accuracy of VGG11 on CIFAR10 with respect to different values of learning rate. As can be seen, AdaCN is more robust to learning rate. (a) Train accuracy of AdaCN on CIFAR10. (b) Train accuracy of Apollo on CIFAR10. (c) Test accuracy of AdaCN on CIFAR10. (d) Test accuracy of Apollo on CIFAR10.
(a)(b)(c)(d)


## 4. Conclusion

We have proposed AdaCN, a novel, efficient, and effective adaptive cubic Newton method for nonconvex stochastic optimization. This method is designed for large-scale nonconvex stochastic optimization problems which are the core of state-of-the-art deep learning literature. Experimental results on image classification tasks and language modeling task demonstrate the superiority of AdaCN, in terms of convergence speed and generalization performance.



---

*Source: 5790608-2021-11-10.xml*