L-SR1 Adaptive Regularization by Cubics for Deep Learning
Authors: Aditya Ranganath
Mukesh Singhal
Roummel Marcia
Venue: ICLR 2022 Submitted (2022)
Keywords: Non-convex optimization
deep learning
quasi-Newton methods
adaptive cubic regularization

Abstract: Stochastic gradient descent and other first-order variants, such as Adam and AdaGrad, are commonly used in the field of deep learning due to their computational efficiency and low-storage memory requirements. However, these methods do not exploit curvature information. Consequently, iterates can converge to saddle points and poor local minima. To avoid these points, directions of negative curvature can be utilized, which requires computing the second-derivative matrix. In Deep Neural Networks (DNNs), the number of variables ($n$) can be of the order of tens of millions, making the Hessian impractical to store ($\mathcal{O}(n^2)$) and to invert ($\mathcal{O}(n^3)$). Alternatively, quasi-Newton methods compute Hessian approximations that do not have the same computational requirements. Quasi-Newton methods re-use previously computed iterates and gradients to compute a low-rank structured update. The most widely used quasi-Newton update is the L-BFGS, which guarantees a positive semi-definite Hessian approximation, making it suitable in a line search setting. However, the loss function in DNNs are non-convex, where the Hessian is potentially non-positive definite. In this paper, we propose using a Limited-Memory Symmetric Rank-1 quasi-Newton approach which allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified Adaptive Regularized Cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions. We investigate the performance of our proposed method on autoencoders and feed-forward neural network models and compare our approach to state-of-the-art first-order adaptive stochastic methods as well as L-BFGS.

Under review as a conference paper at ICLR 2022
ADAPTIVE REGULARIZED CUBICS NEWTON APPROXIMATIONS
USING
QUASI-
Anonymous authors Paper under double-blind review
ABSTRACT
Stochastic gradient descent and other first-order variants, such as Adam and Ada- Grad, are commonly used in the field of deep learning due to their computational efficiency and low-storage memory requirements. However, these methods do not exploit curvature information. Consequently, iterates converge to sad- dle points and poor local minima. To avoid this, directions of negative curvature be utilized, which requires computing the second-derivative matrix. In Deep Neural Networks (DNNs), the number of variables (n) can be of the order tens of millions, making the Hessian impractical store (O(n²)) and to invert (O)))). Alternatively, Newsin methods compute Hessian approximations that do not have the same computational requirements. Quwsi-Newton methods re-use previously computed iterates and gradients to compute a low-rank struc- tured update. The most widely used quasi-Newton update is the L-BFGS, which guarantees positíve semi-definite Hessian approximation, making it suitable in line search setting. However, the loss function in DNNs are non-convex, where the Hessian is potentially non-positive definite. In this paper, propose using a Limited-Menn Memorr Symmetric Rank-1 quasi-Newton approach which allows for indefinite Hessian approximations, enabling directions of negative curvature be exploited. Furthermore, we use a modified Adaptive Regularized Cubics ap- proach, which generates sequence of cubic subproblems that have closed-form solutions. We investigate the performance of our proposed method on autoen- coders and feed-forward neural network models and compare our approach state-of sifie-of-ther ff-hh--art art first-order adaptive stochastic methods as well as L-BFGS.
INTRODUCTION
Most deep learning problems involve minimization of the empirical risk of estimation
min f(r; (),
(1)
where O E R" is the set of weights and is some scalar-valued loss function. To solve T.. various optimization approaches have been implemented, which describe below. Throughout this paper, we write f() and f(z; +) interchangeably.
Gradient and adaptive gradient methods are widely used for training deep neural networks (DNN) for their computational efficiency. The most common approach is Stochastic Gradient Descent (SGD) which, despite its simplicity, performs well over wide range of applications. However, in sparse training data setting, SGD performs poorly due to limited training speed (Luo et al. (019 To address this problem, adaptive methods such as AdaGrad (Duchi al. (201 AdaDelta ZZeiler 22022). ) RMSProp (Hinton et al. (2012) and Adam (Kingma & Ba (2014)) have been proposed. These methods take the root mean square of the past gradients to influence the current step. Amongst all of these adaptive methods, Adam is arguably the most widely used in deep learning setting due
to it rapid training speed.
Newton's method has the potential to exploit curvature information from the second-order deriva- tive (Hessian) matrix (see e.g.. Gould et al.] (2000).. Generally, the iterates are defined by where ag > 0 is steplength defined by linesearch
O++1 O a f(x)) v((O)), DNN setting. we know that the number of parameters (n) of the network can be of the order of millions. Thus storing the Hessian which takes O(n))
criterion (Nocedal& Wright (006)).

---

Under review as a conference paper at ICLR 2022
memory, becomes impractical. In addition, the inversion of the Hessian matrix, which takes ( operations, also impractical. Even though Newton's 's method achieves convergence in fewer steps, the method becomes computationally intractable to use on large-scale DNNs.
Quasi-Newton methods are alternatives to Newton methods. They compute Hessian approxima- tions, BB+11, that satisfy the secant condition given by Yı- = B5+1 where Są O O and Ys -f( - v) The most commonly used quasi-Newtor method, including in the
realm of deep learning, is the limited-memory BFGS update, or L-BFGS (see e.g., Liu & Nocedal
1989 ) where the Hessian approximation is given by
y&yE Y 5k
BA
B B++1 Bg
(2)
BeSk
The generic L-BFGS quasi- update scheme is described in Algorithm [ and numerous vari- ants of L-BFGS exist (see Goldfarb al. (2020); Moritz et al. (2016); Gower al.) (2016). One advantage of using an I-BFGS update is that the Hessian approximation can be guaranteed to be definite, which highly suitable in line-search settings because the update sæ is guaranteed to descent direction, meaning there is some step length along this direction that results in decrease in the objective function (see Nocedal L-BFGS update is positive definite, it does not readily detect directions of negative curvature for avoiding saddle points. In contrast, the Symmetric- Rank One (SR1) quasi-Newton update is not guarateed to be positive definite and can result in ascent directions for line-search methods. How- ever, trust-region settings where indefinite Hessian approximations are an advantage because they capture directions of negative curvature, the limited- perform L-BFGS DNNs for classification (see Erway et al. (2020)). We discuss this in more detail but in the context of Adaptive Regularization using Cubics.
Wright (2006), Algorithm 6.1). However, because the
SR1 (L-SRI) has been shown to out-
in Section 2
Algorithm L-BFGS Quasi-Newton Method with Line Search
Require: Initial weights Og. batch size d, learning rate a, dataset D, loss function f()). for k 1, do
Sample mini-batch of size d: Dx C Perform the forward backward pass over the current mini-batch
Compute the limited memory approximation B₃ using 2 Compute step S aB, (e)((xx) where is the linc-search step length
end for
L-SR1 ADAPTIVE REGULARIZATION USING CUBICS METHOD
We begin by discussing the L-SRI update and the adaptive regularizion using cubics methods for large-scale optimization.
Unlike the BFGS update which is a rank-two update, the SR1 update rank-one update, which is given by
1
Ba+1 = Bg +
5 (yx B (yx - BB8K)(8 BBSK)
(3)
(see Khalfan et al. (1993)). As previously mentioned, B in (3) is not guaranteed to be definite. However, it can be shown that the SRI matrices can converge to the true Hessian (see Conn et al. 1991) for details). We note that the pair (s). is accepted only when |s] (y BB >> e|ly B for some constant e > 0 (see Nocedal & Wrightt (006)) Sec. 6.2, for details). The SR1 update can be defined recursively as
B = Bo +
1= S [y, Bys;) (y, - Bys)) B.
(4)
In limited-memos SRI (L-SRI) settings, only the last m & n pairs of (s). are stored and used. Sk and Y1+1 yoo Y1 Yk ]. then B admits compact
If S
So
2

---

Under review as a conference paper at ICLR 2022
representation of the form
MM+1][
k+1
B&+1 By + Px+1
(5)
where
V MR+1
Yk+1 1141-Ba B and M (D&+1 + + Bo
(6)
where LK+1 is the strictly lower triangular part, Uk+1 is the strictly upper triangular part, and D Dx+1
is the diagonal part of S k+1 Yxt1 L841 + Da-1 + Uk+1 (see Byrd et al. (1994) for further details).
Because of the compact representation of BR+1, its partial eigendecomposition can be computed (see Burdakov et al. (2017). In particular, we compute the QR decomposition of k4+1. then we can write Bk+1 B = U U where Uj [nx(k+1) has orthonormal columns and À E ((++))((++)) is diagonal matrix. If Bu = 6g1 (see e.g., Lemma 2.4 in Erway et al. (200 ) where 0 < 8% < 8max is some scalar and I is the identity matrix, then we obtain the eigendecom- UR+1As+1 Uk+1. where Uk+1
position Bx+1
[U] U.]. with U1 [n= (k+((+1)) and
U R+1 U k+1 = I. Here, (As+1); = 6% + E for i & + 1, where A is the ith diagonal in A and +
(A); = 6% for
Since the SR1 Hessian approximation can be indefinite, some safeguard must be implemented to ensure that the resulting search direction 85 a descent direction. One such safeguard is to use "regularization" term.
The Adaptive Regularization using Cubics (ARCs) method (Griewank (1981); Cartis et al. (0IIT) can be viewed as alternative to line-search and trust-region methods. At each iteration, an approximate global minimizer a local (cubic) model,
1 min my (s) = g% s + B + 3 ($))3 SER"
2
is regularization parameter, and 4 is func- tion (norm) that regularizes s. Typically, the Euclidean norm is used. In this work, we propose an alternative "shape-changing" norm that allows to solve each subproblem (7) exactly. This shape-changing norm was proposed in Burdakov et al. 2017). and it is based on the partial eigende- composition of B, Specifically, if B A₂ U. is the eigendecomposition of B,, then can
determined, where ge vf(02), M4
define the norm |s| U. we can redefine the cubic subproblem as
def | Applying change of basis with U and Bx U gk.
min (s) g SER
A
|s||
(8)
2
3
With this change of basis, can find closed-form solution of 8 easily, The proposed Adaptive Regularization using Cubics with L-SRI (ARCSLSRI) algorithm is given in Algorithm 2
2.1
CONTRIBUTIONS
The main contributions of this paper are follows: L-SRI quasi-Newton methods: The most commonly used quasi-Newton approach is the L-BFGS method. In this work, we use the L- SRI update better model potentially indefinite Hessians of the non-convex loss function. 2. Adaptive Regularization using Cubics (ARCs): Given that the quasi-Newton approximation is al- lowed to be indefinite, we use an Adaptive Regularized using Cubics approach to safeguard each search direction. 3. Shape-changing regularizer: We use shape-changing norm to define the cubic regularization term, which allows us to compute the closed form solution to the cubic subproblem 77. 4. Computational complexity: Let be the number of previous iterates and gradients stored in memory. The proposed LSR1 ARC approach is comparable to L-BFGS in terms of storage and compute complexity (see Table
3

---

Under review as a conference paper at ICLR 2022
Algorithm 2 LLmited- Symmetric Rank-1 Adaptive Regularization using Cubics
Given: θ, ?2 2 71, 1/2 m 0, and do
2: for == 0,,1....... do
3: Obtain S = Su S Y yo Yk
4: Solve the generalized eigeaproblem S Yuu = ÄS Sau and ant 6x = min(}+}
5:
Compute 3 = Y2 S.
6:
Perform QR decomposition v = QR
7:
Compute the eigendecomposition RMR = PAP
8:
Assign U| QP and U =
9:
..... Cm) Define CJ diag ................
where C; Aet +
2
and gi| U g
10:
Compute a*
2
where g =g
U |8)
8 + 55 + +
Compute step
g
U (o'I. Im -0---- C
12:
Compute (s*) and = (f(@)) - [(Ox+1))/mm(²)
Set
0.5pu
if Ph: > 1/2,
O + Sk, if Pk: 2 M
0++1
and M4+1
0.5 + + 0.5 + +2)
if mm S Pk S 12. otherwise
kk
otherwise
end for
Table Storage and compute complexity the methods used in our experiments.
Algorithms
Storage complexity
Compute complexity
SGD/Adaptive methods
(n)
(n)
L-BFGS
Onn + mn)
(mm)
ARCs-LSRI
Onn + mn)
O(m + 2mm)
2.2 IMPLEMENTATION
Because full gradient computation is very expensive to perform, impement stochastic version of the proposed ARCs-LSRI method. In particular, we use the batch gradient approximation
Σ Vf((O))
g+
B
BE
In defining the SRI matrix, we use the quasi-Newton pairs (8k. FRh, where ÿo. gg++1 (see e.g.. Erway et al. 22200).
CONVERGENCE ANALYSIS
In this section, we prove convergence properties of the proposed method (ARCs-LSRI in Algorithm
2). The following theoretical guarantees follow the ideas from Cartis et al. (001] and Benson & Shanno (2018).
First, we make the following mild assumptions:
A1. The loss function f(O) is continuously differentiable, i.e., f e C'l (R*).
A2. The loss function f() is bounded below.
Next, we prove that the matrix B; in (4 is bounded.
Lemma The SRI matrix B in satsifies
B BBt Ip S KB for all k 2
for some KB 0.
4

---

Under review as a conference paper at ICLR 2022
Proof: Using the limited- limited-memory
memory SRI update with memory parameter in (4 have
I - B,s,)(y, - B,s,)
Batille & Bolle +
{8] (y, Bys;)]
m-k-++1
Using a property of the Frobenius norm, namely, for real matrices A, ||A = trace )) we have that || Ry, B |F [|y; - Bysyl Since the pair (s,. Yi) is accepted only when 1s (y, B,s,)l elly, Bys, for some constant O, and Bo 6g for some 0 < 8R < Ómax, we have
B Smax KB.
Given the bound on [BR+illF, we obtain the following result, which is similar to Theorem 2.5 in Cartis et al. (2011).
Theorem Under Assumptions Al and A2, if Lemma holds, then
lim inf |[: 0.
k
Finally, we consider the following assumption, which can be satisfied when the gradient, g(0), is Lipschitz continuous on O.
A3. } and {Or, are subsequences of {O&). then lgg. -r, + 0 whenever ||0;,- - - -,,00-0 as co. If we further make Assumption A3, we have the following stronger result (which is based on Corollary 2.6 in Cartis et al. (20IT)):
Corollary Under Assumptions AI, A2, and A3, if Lemma
holds, then
lim gg = 0.
4 EXPERIMENTS
To empirically compare the efficiency of the method against popular optimization methods like SGD, ADAGRAD, ADAM, RMSProp and L-BFGS, we focus two broad deep learning problems: image classification and image reconstruction. We choose these tasks due to their broad importance and availability of reproducible model architectures. We each experiments times with a random initialization each experiment. The number of parameters, convolutional layers and fully connected layers are mentioned in Table
average of
Dataset: We measure the classification performance of each optimization method on 4 image
datasets: MNIST (LeCun al. (2010)), FashionMNIST (Xiao et al. (2017)) IRIS (Dua & Graff (2017)) and CIFARIO (Krizhevsky et al.). ments in Table
We have provided comprehensive view of the experi-
Dataset
Network
Type
IRIS MNIST
MLP
Classification
MLP
Classification
FMNIST
Convolutional
Classification
CIFAR10
Convolutional
Classification
FaShionMNIST
Convolutional
Reconstruction
MNIST
Convolutional
Reconstruction
Table 2: List of experiments
Hyperparameter tuning: We empirically fine-tune the hyperparameters and select the best for each update scheme. We have made comprehensive list of all the learning rates for the gradient and adaptive gradient based algorithms Table in the Appendix. The additional parameters are defined as follows:
5

---

Under review as a conference paper at ICLR 2022
We apply an perturbation of 1.0 10-9. Bu and $1 chosen be 0.9 and 0.999, respectively.
ADAM:
ADAGRAD:
The initial accumulator value is set to 0. The perturbation e is set to 1.0 x 10-10, We use momentum of 0.9.
SGD:
RMSPROP: We set 0.99. The perturbation is set 1.0 X 10
L-BFGS: The table in Appendix A presents the initial learning rate for the stochastic step in L-BFGS. We set the default learning rate to 1.0. We choose a history size m of 10 and max iterations 10. The tolerance on function value/parameter change is set to 1.0 10 and the first-order optimality condition for termination is defined as 1.0 x 10-9.
ARC-LSRI:
We choose the same parameters L-BFGS.
Network architecture: For each problem, define the model architecture in Table 3 in the ap- pendix. We define the process the forward and backward pass of a DNN in Algorithm in the appendix.
Dataset
Network
Convolution layers
Fully connected layers
Parameters
IRIS MNIST
Classifier
2953
Classifier
3
397510
CIFAR10
Classifier
3
62006
MNIST
Autoencoder
6
4
53415
FashionMNIST
Autoencoder
6
4
53415
Table 3: List of experiments
Testbed and software: All experiments were conducted using open-source software PyTorch Paszke et al. (2019)), SciPy (Virtanen et al.] (2020) and NumPy (Harris et al. (2020)). We use an Intel Core 17-8700 CPU with clock rate of 3.20 GHz and an NVIDIA RTX 2080 Ti graphics card.
5 RESULTS
We have divided the sections into two categories: classification and image reconstruction. We present both the training results and the testing results for all methods.
5.1 CLASSIFICATION RESULTS
For each classification problem, we define the network architecture, the corresponding hyperparam- eters (other than the learning rate) for each optimization scheme.
Since this dataset is relatively small, we assume a small network for our deep-learning model. The model is described in 3 We set the history size for the proposed approach and L-BFGS to 10 and the number of iterations to 10. Figure I shows the comparative performance of all the methods. Note that our proposed method (ARCLSRI) achieves the highest classification accuracy in the fewest number of epochs.
IRIS:
We trained the network for 20 epochs with a batch size of 256 images each. We keep the same history size and number of iterations as the IRIS dataset for L-BFGS and the proposed ARCLSRI approach. For training, can seen Figure @ that nearly all methods achieve opti- mal training accuracy. However, closely inspecting the testing curve, we notice that the proposed approach achieves higher accuracy than all the existing methods.
MNIST:
We train the network for 20 epochs with batch size of 256 images. We keep the history size the same as the IRIS and MNIST experiments for the proposed approach and L-BFGS. For this method, the proposed ARCLSRI approach is comparable to L-BFGS but outperforms the adaptive methods (see Figure 3).
FMNIST:
CIFAR10: We use the parameters presented in Table in the previous section for the adaptive methods. For ARCLSRI and L-BFGS, we have a history size of 100 with a maximum number of iterations of 100 and batch size of 1024. Figure 4 represents the training loss (cross-entropy

---

Under review as a conference paper at ICLR 2022
10
1.
Bgp
80
<5om
ARCLSR]
0.5
Adag
0.8
RMSShop Adagred
50
0.0
0
15
20
25
30
10
epochs
epochs
(b)
Figure 1: The classification results the IRIS dataset. (a) Training loss of the network. The y-axis represents the negative log-likelihood loss and the r-axis represents the number of epochs. (b) The classification accuracy for each method, i... the percentage of testing samples conrectly predicted in the testing dataset.
100
lbtgs
<temm
ARCLSR
SCD
04
5G0
RArSProp
Addggrad
opachs
(a)
(b)
Figure 2: The classification results on MNIST. The y-axis represents the classification accuracy on the MNIST dataset, and the æ-axis represents the number of epochs. (a) Training response. (b) Testing response.
ASCUSRT
15
a1
(b)
Figure 3: The plots above show the classification results on the Fashion MNIST dataset. We run this experiment for 20 epochs. In this experiment, the proposed method is comparable to L-BFGS but outperforms the adaptive methods.
loss). Figure b) represents the testing accuracy, i.e.. number of sample correctly predicted in the testing set. To demonstrate the efficacy of the proposed method on larger networks, additional experimentation the ResNet50 architecture be found in the appendix (Figure

---

Under review as a conference paper at ICLR 2022
Bdge odam AFC QZD RM45Pro
25
o0am
ARCLSRT
Apagro
50D
15
Adagrad
20
80
0
40
60
90
100
epochs
epochs
(b)
Figure 4: The classification results on CIFARIO. (a) The y-axis represents the training cross-entropy loss, and the x-axis represents the number of batches. (b) The y-axis represents the testing response accuracy and the a-axis represents the number of epochs.
5.2 IMAGE RECONSTRUCTION RESULTS
The image reconstruction problem involves feeding feedforward convolutional autoencoder model (with randomly initialized weights) a batch of the dataset. It follows the same deep learning con- vention mentioned in Algorithm in Appendix A The loss function is defined between the reconstructed image and the original image.
An image E Rd is fed to the network, compressed into latent space E R where 1 < n, and reconstructed back to its original image size ह E R We compute the mean-squared loss error between the reconstruction and the true image. The weights are initialized randomly, Each experiment has been conducted times and we considered a batch size of 256 images each with 50 epochs. The results for the image reconstruction can be seen in Figure
MNIST:
0.08
0.07
0.06
550
0.00
0.05
0.01
#porhs
epochs
(a)
(b)
Figure 5: This graph represents the training accuracy on the training samples. The r-axis shows the number of epochs and the y-axis represents the accuracy (Mean-Squared error). (a) shows the initial training error and (b) shows the final training error.
One can notice that the initial descent provided by the proposed approach provides a significant decrease the objective function. To understand better, we provide the details of the results during the initial epoch (Figure I()) and the final epoch (Figure 9b)). We notice that the ARCLSRI method has minimized efficintlyy in the first half of the first epoch. This is empirical evidence that the method converges to the minimizer in fewer steps in comparison to the adaptive methods. In Figure (b), we notice that all the adaptive methods eventually converge to the same point. For training response results on the F-MNIST dataset, Section B in the appendix.
8

---

Under review as a conference paper at ICLR 2022
0.37
0.00
300m
200m
ARCI
0.44
0.33
0.08
epochs
epochs
(b)
Figure 6: The testing accuracy on MNIST dataset. y-axis represents the Mean-Sou -quar-S Error loss on the testing set and z-axis represents the number of epochs. (a) shows the initial testing response. (b) shows the final testing response.
5.3 TIME COMPLEXITY ANALYSIS
We understand that the proposed approach performs competitively against existing methods. We now analyze the time-constraints of each method. We choose to clock the computationally demanding algorithm here CIFARIO classification. history size of 100 for L-BFGS and the ARCs LSRI, with batch size of 1024 images. Figure 7 plots the time required by each of the methods to reach non-overtee overtraardd minima with a batch size of 1024 images. As can be seen, the proposed approach is able to reach the desired minima in much less time than the rest of the algorithms. L-BFGS finds hard to converge due to a very noisy loss function and small batch size, thus causing the algorithm break. Ozyildirim & Kiran (2020) argue that a large batch size is required for quasi- methods to perform well. However, the ARCLSRI method performs well with small batch size as well.
chose maximum iterations 100 with
60
50
40
lotg5
30
ARCLSRI
20
RMSProp
Adegrad
500
1000
1500
2000
2500
3000
3500
4000
4500
time (seconds)
Figure 7: Timing analysis for CIFARIO. (a) Evolution of model accuracy with respect to time (x- axis time seconds and y-axis is accuracy of prediction in percentage). (b) Computational cost for each epoch (the z-axis corresponds to epochs and the y-axis is time).
CONCLUSION
In this paper, we proposed a novel quasi-Newton approach in a modified adaptive regularized cubic setting. We able to empirically and theoretically show how L-SRI quasi-Newton approxi- mation in an ARCs setting was able to perform either better or comparably to most of the state of the art optimization schemes. Even though the approach has yielded exceptional results, we need to test the method's is efficacy when the network size and dataset size is large and when availability of data sparse.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hande Y. Benson and David F. Shanno. Cubic regularization in symmetric rank-1 quasi- newton methods. Mathematical Programming Computation, 10(4))45-486, Dec 2018. ISSN 1867-2957. doi: 512532-012 018--136-1
URL
Oleg Burdakov, Lujin Gong, Spartak Zikrin, and Ya-xiang Yuan. On efficiently combining limited- memory and trust-region techniques. Mathematical Programming Computation, 91):101-134, 2017.
Richard. H. Byrd, Jorge. Nocedal, and Robert. B. Schnabel. Representations of quasi-Newton ma- trices and their use in limited-memory methods. Math. Program., 63:129 1994.
Coralia Cartis, Nicholas IM Gould, and Philippe L Toint. Adaptive cubic regularisation methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mathematical Programming, 127(2):245-29) 295, 2011.
Andrew R. Conn, Nicholas. M. Gould, and Philippe. L. Toint. Convergence of quasi-newton matrices generated by the symmetric rank one update. Mathematical Programming, 50(1):177- 195, Mar 1991. ISSN 1436-4646. doi: 10. (0.1007/BF015949544 UtL.httss:/ddo..ooo 0007/BF01594900
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URLhhtt://arcchvee
ics. uci. .duumml
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Jennifer B. Erway, Joshua D. Griffin, Roummel F. Marcia, and Riadh Omheni. Trust-region algo-
rithms for training responses: machine learning methods using indefinite hessian approximations. Optimization Methods and Software, 35:460 487, 2020.
Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practical quasi-newton methods for training deep neural networks. arXiv preprint arXiv:2006.08877 2020.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/www. www.eeeepeex.nnppooooooo org
Nicholas I. M. Gould, Stefano Lucidi, Massimo Roma, and Philippe L. Toint. Exploiting negative
curvature directions in linesearch methods for unconstrained optimization. Optimization Methods doi: 10. 00.1000015557788000007777777
and Software, 14(1-2):75-95 2000.
Robert Gower, Donald Goldfarb, and Peter Richtarik. Stochastic block bfgs: Squeezing more cur- vature out of data. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1869-1878, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https::/proceediine
mlr wwwpppeessss...eee http:///////wwwwe/e/////m html
Andreas Griewank. The modification of Newton?s method for unconstrained optimization by bound- ing cubic terms. Technical report, Technical report NA/12, 1981.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten Н. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard- Maraddat, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Ar- ray programming with NumPy. Nature, 585(7825)3357-322 September 2020. doi: 10.1038/ 211586-020-244--2 URL httpt:://///.. ttttss://do... org 1038/ 4 58 020-0049-22222
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
10

---

Under review as a conference paper at ICLR 2022
H Fayez Khalfan, Richard H Byrd, and Robert B Schnabel. A theoretical and experimental study of the symmetric rank-one update. SIAM Journal on Optimization, 3(1):1-24, 1993.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv::412.0 69800 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). URL http://www. Cs toronto. edu/ wtt.rataaaaa.mmmm.m.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [On- line Available: http://yann. lecun. com/exdb/mmist, 2,18, 2010.
Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale opti- mization. Mathematical Programming, 45(1))503-528, Aug 1989. ISSN 1436-4646. doi:
10.1007/BF0158916 l.tppss:::///o 1t
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate, 2019.
Philipp Moritz, Robert Nishihara, and Michael Jordan. linearly-covvergent stochastic L-BFGS algorithm. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th In- ternational Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. ml r.rr.presss /tt/////t/t//1/////////// 16.1166666 1ttmrtttt6.h.lll
Cadiz, Spain, 09-11 May 2016. PMLR. URL
Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, New York, NY, USA, second edition, 2006.
Buse Melis Ozyildirim and Mariam Kiran. Do optimization methods in deep learning applications matter?, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http http::://ppppp....u.seeaee htwp:erss...rri...c..c..oaoa 9015 att----oorraa-a---eeeaeeeee-eeee
pdf
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ithan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 1038/03/4411 1e..11////112220111111 592-019--066--22
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. 2017.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.570) 2012.


Reviews:
[{"review_id": "vDfjtFPt0_p", "reviewer": "Reviewer_YQSL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ingredients in the algorithm are not new, though the shape-changing regularizer used in this paper was originally described for a trust-region method rather than ARC (and the author of the shape-changing regularizer paper focused on L-BFGS as his main algorithm, while at the same time mentioning L-SR1 and other quasi-Newton methods as options).  However, the particular combination is new as far as I know, and the empirical results seem quite promising, though all involve relatively small networks.\n\nTo the extent that the combination is new, I would have liked to see a little more rationale for the details of the regularizer adjustment and step acceptance criteria.  These seem to be adapted from the paper of Burdakov to the ARC setting, but there is no discussion of convergence results.  If the algorithm is the main contribution of the paper, a statement of these results would be nice to have.  It would also be nice to have the (very short) appendix A folded into the main paper text so that the paper could be implemented by a reader without referring to the supplementary material.\n\nThe text is missing some critical details that I think make the algorithm much more interesting (and make me interested in seeing a more developed paper in the future!).\n\nFirst, it seems that the L-SR1 and L-BFGS algorithms are both being used with stochastic gradient estimates in lieu of actual gradients.  This is not entirely clear in the earlier examples where the authors refer to mini-batch sizes (those could have just been for the SGD variants), but in the final paragraph of Section 4, just before the conclusion, the authors refer to comparing L-SR1 and L-BFGS with different batch sizes.  Though the authors mention prior work arguing that L-BFGS works poorly with stochastic gradient estimates (at least without very large batch sizes), there is no hint as to why stochastic gradients should work so much better with L-SR1.  This seems like a real mystery: is it the L-SR1 Hessian estimate that works better, or the use of ARC rather than line search, or something else entirely?  There are also several aspects of the algorithm that are under-explained in the presence of stochastic estimation.  For example, the step acceptance and regularization adaption are based on an improvement ratio that superficially requires evaluating the full empirical loss at two points; are these replaced by means over a mini-batch in the implementation?  If so, is the same mini-batch used to evaluate both terms?  And similarly with the gradient treatment: is the gradient difference in the L-SR1 algorithm based on stochastic gradient estimates using the same mini-batch, or independent draws?  Also, in the conclusion, the authors mention that they would like to \"explore a stochastic version of computing the Hessian approximation\", and this confused me in the context of the apparently-stochastic gradients feeding into the L-SR1 Hessian approximation.\n\nSecond, the authors never specify the activation function they are using in their experiments.  But if one was to use a ReLU activation (which seems likely), the true empirical risk becomes non-smooth.  There has been interesting work done on using BFGS for non-smooth functions, and it seems to work surprisingly well in that setting; however, to my knowledge, the theory for why is still lacking.  I don't know of any work on SR1 (or L-SR1) for non-smooth objectives.\n\nThe paper might still be ready for publication despite missing theoretical details if there were a strong enough empirical justification.  However, while the experiments are interesting, they all involve relatively small networks (only one over 100K parameters) for relatively small data sets, though this might be based on the hardware used (one node with a GPU).\n\nI also have several more minor comments:\n\n- The citation of Luo et al on the first page should be parenthesized.\n\n- The quoted theorem 1 is for convex optimization problem; the start of the next sentence says \"Newton's method avoid (sic) these saddle points,\" but convexity means that saddle points cannot be the problem.\n\n- \"Thus storing the Hessian O(n^2) becomes impractical\" should be something like \"Thus storing the Hessian, which takes O(n^2) memory, becomes impractical.\"  Similarly \"the inversion of the Hessian matrix can make it an additional computational burden O(n^3)\" should be something like \"the inversion of the Hessian matrix takes O(n^3), which is also impractical\" (or just \"the inversion of the Hessian matrix takes O(n^3)\" -- the reader will likely get the point).\n\n- In Algorithm 3 line 3, the matrix Y_k should have columns y_0 through y_k (vs s_0 through s_k).  Also, I was confused my the update for mu_{k+1} -- it seems like it is being assigned an interval rather than a specific value?\n\n- In the results, please do say more about the architecture (activation functions in particular).  This could go into supplementary details, but the experiments could not be reproduced with what is given here.\n\n- On page 7, \"Figure 4(b) tepresents\" -> \"Figure 4(b) represents\"", "summary_of_the_paper": "The paper describes a limited memory quasi-Newton method based on SR1 updating using a variant on the adaptive regularized cubic (ARC) approach to globalization.  The algorithm is applied to training deep neural networks for image classification and autoencoding, and compared to L-BFGS and various SGD variants.  The authors claim the main contributions to be the different optimizer ingredients (L-SR1, ARC safeguarding, and the particular shape-changing regularizer) as well as computational complexity similar to L-BFGS.", "summary_of_the_review": "The ingredients of the algorithm are known for smooth, non-noisy functions, though the specific combination seems new.  However, the details of the algorithm are not fully justified even in this setting; and the actual setting of interest seems to involve stochastic function and gradient estimates and possibly non-smooth objectives (depending on the activation).  The experiments that are in the current version are intriguing because they suggest that the algorithm works well (significantly better than L-BFGS) in this setting.  But to accept, I would have liked to see more exploration of these aspects of the problem, either theoretically or empirically.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uhArid_ksj_", "reviewer": "Reviewer_cfGs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper highlights the potential of L-SR1 in the highly non-convex deep neural network setting, where neglecting negative curvature information might seriously degrade the accuracy of the Hessian approximation.\n- The idea of using the specific type of shape-changing norm that varies in every iteration in order to simplify the local cubic sub-problem is interesting.\n\nCons:\n- The novelty of this paper is very limited. The only novel contribution compared to Cartis et al.* and Erway et al*. seems to be the particular form of shape-changing regularization. This has the potential to be a valuable contribution, but the exposition of the technique is very confusing. Authors point out that that closed-form solution can be found easily, but provide no further explanation, simply presenting the full algorithm as is. The proposed algorithm, which is the main contribution of this paper, is not discussed in much detail and several steps are left unexplained.\n- The significance of the results is questionable. Since the paper does not provide any theoretical insights into the introduced algorithm (e.g. convergence guarantees), it has to be thoroughly verified through relevant experiments. The proposed algorithm is not compared to its closest competitor, Erway et al, and the L-BFGS algorithm used in the comparison is a naive implementation of mini-batch L-BFGS that is known to be unstable on larger models/datasets. There are several improved versions of L-BFGS tailored to the stochastic setting (Moritz et al.*, Gower et al.* and more). Furthermore, the experiments are performed on very small and simple models and datasets, casting doubt on the practical relevance of the method. Even in this limited setting the gains over other techniques are not too significant.\n- Authors posit that Newton's method avoid saddle points by exploiting curvature information. To the best of my knowledge, the opposite is true and Newton's method is prone to converging to saddle points. The motivation that SR1 avoids saddle points by using negative curvature information is similarly unsupported. It seems to me that the benefit of SR1 over BFGS would lie in the more accurate Hessian representation and not in the avoidance of saddles.\n- The structure and organization of the paper is confusing and has unnecessary parts. For instance I don't see why the specific forms of adaptive methods are discussed in such detail (including table) when it is not used later in the paper. Including forward and backward pass of a DNN as an algorithm is also unnecessary at a conference like ICLR. Detailed description of the well-known datasets, and the very detailed discussion of minor hyperparameters could be moved to the appendix. The paper has many typos, grammatical errors, unfinished sentences.  \n- The experimental results are confusing and inconsistent. In some plots we don't know exactly what quantity is plotted (Fig. 3). In some plots, epochs are not integers, which is confusing and in this case iterations would be more clear. Figure 7 shows 'computational cost' but it is not disclosed how authors calculated this quantity. In the same plot CIFAR-10 training on a small model takes 80k seconds (approximately 22 hours), which seems highly unrealistic. In 'Conclusions' authors also mention that the proposed algorithm's performance is fairly independent of batch size, which is not supported at all by the experiments. Overall, the experiments have brought me more questions than answers.\n\n*Citations are equivalent to those in the paper.", "summary_of_the_paper": "This paper investigates the application of a certain Quasi-Newton algorithm, the Limited-Memory Symmetric Rank-1 (L-SR1) algorithm, in deep learning problems. The benefit of this technique over similar more widely investigated methods that use a positive definite approximation of the Hessian, such as stochastic L-BFGS, is the fact that the L-SR1 approximation is not guaranteed to be definite, and thus has the potential to have a more accurate approximation of the true Hessian. Since in this case line-search methods may return ascent directions, authors propose a specific form of Adaptive Regularization using Cubics (ARC) as an alternative. Numerical simulations are provided comparing the performance of the proposed algorithm to SGD, adaptive methods such as Adam and a naive L-BFGS implementation.", "summary_of_the_review": "Even though the paper has a neat idea to transform the ARC subproblem into an easily solvable form, the paper has many shortcomings when it comes to novelty and significance to the ML community detailed above. At this point the paper reads more as a draft, and would require very significant modifications in my opinion to improve its quality. Therefore I recommend rejection. I am however open to update my score if the authors or other reviewers address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RWMYDUhKnQ", "reviewer": "Reviewer_oGs7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My major concerns with this paper are : \n-Except thm1 from Reddi et al. paper, I dont see any theoretical result neither in the paper nor in the  appendix. Am I missing some thing or no theory is given to support the claims in the paper?\nFor instance, after Thm1 statement, you claim that Newton's method avoid saddle points...but I did not see any theoretical results supporting this...\n-The novelty of the contributions stated at page 4: the methods SR1 to approximate the Hessian & ARC  exist already in the literature. \n\nYou claim at the end of page 2 \"... an L-BFGS update is that the Hessian approximation can be guaranteed to be definite, which is highly suitable in line-search settings because the update s_k is guaranteed to be a descent direction...\" why? give a reference or show it?\n\nLine 4 in Algo 1: what is the operator G?\n\nwhat is f_t?\n\nThm1: what do you mean by online convex optimization exactly? \n\nSeveral typos in the paper. \n...", "summary_of_the_paper": "The paper proposes to use a Symmetric Rank-1 (SR1) quasi-Newton approach to approximate the Hessian and  to use an Adaptive Regularized Cubics (ARC) with an adaptive norm framework. To assess the performance of the new method, numerical experiments using autoencoders and feed-forward neural network models are supplied.", "summary_of_the_review": "See above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "27PXFiPBBpe", "reviewer": "Reviewer_qdob", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although I agree that introducing quasi-newton methods is an interesting direction, this paper seems to be underdeveloped and needs quite a lot of work to be ready for publishing. First of all, there are some errors/missing parts in the paper that seriously need to be corrected.\n\n1. In Algorithm 1, line 4, what is \\mathcal{G}? Is it an operator? I believe the authors are trying to use the framework by Reddi et al, 2018, but the notations are not clear enough.\n\n2. The authors claim that the counterexample by Reddi et al. 2018 can be extended to all adaptive optimization algorithms, but that is not true. The counterexample is designed particularly for Adam because its second moment (the denominator) could decrease when optimization progresses, and the telescopic summation in the proof of Adam is thus wrong. I encourage the authors to read the paper of Reddi et al. 2018 more carefully to get a better idea of why Adam is poorly-designed. The other proposed adaptive algorithm, such as AMSGrad, AdaBound (Luo et al, 2019), and NosAdam (Huang et al, 2019) are well-designed and they would not have similar issues.\n\n3. Where is the step size in the definition of newton and quasi-newton methods?\n\n4. I am expecting a theoretical analysis of the convergence behavior of the proposed algorithm, maybe even in the online convex setting to show the advantage of the proposed algorithm. However, I do not see such theorems or propositions.\n\n5. Some of the experimental results are not convincing enough. First of all, I would encourage the authors to run experiments on some larger datasets, such as ImageNet, or on some different tasks apart from image tasks, such as language modeling on PennTreeBank. The current tasks are only image classification/generation, and only on small datasets such as MNIST and CIFAR10, which seem very limited. Moreover, I don't understand why the experiments on CIFAR10 is terminated at the 15-th epoch, while the other algorithms run for as many as 4k epochs. From my experience, 100-150 epochs are pretty sufficient for any algorithm and any NN model to train on CIFAR10. I would like the authors to explain why.\n\nSashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond, ICLR 2018.\nLiangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate, ICLR 2019.\nHaiwen Huang, Chang Wang, Bin Dong Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate, IJCAI 2019", "summary_of_the_paper": "This paper proposes a quasi-newton method for neural network optimization. The authors have conducted some experiments to verify the effectiveness of their algorithm.", "summary_of_the_review": "I believe this paper still needs a lot of work to be ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an adaptive gradient method for neural net training inspired by L-BFGS. All of the reviewers recommend rejection. They raise concerns about the amount of novelty, the clarity of the writing, and the experimental comparisons. I encourage the authors to take the reviewers' comments into account and improve the submission for the next cycle.
Decision: Reject