Title: L-SR1: A Second Order Optimization Method for Deep Learning
Authors: Vivek Ramamurthy
Nigel Duffy
Venue: ICLR 2017

Abstract: We describe L-SR1, a new second order method to train deep neural networks. Second order methods hold great promise for distributed training of deep networks. Unfortunately, they have not proven practical. Two significant barriers to their success are inappropriate handling of saddle points, and poor conditioning of the Hessian. L-SR1 is a practical second order method that addresses these concerns. We provide experimental results showing that L-SR1 performs at least as well as Nesterov's Accelerated Gradient Descent, on the MNIST and CIFAR10 datasets. For the CIFAR10 dataset, we see competitive performance on shallow networks like LeNet5, as well as on deeper networks like residual networks. Furthermore, we perform an experimental analysis of L-SR1 with respect to its hyper-parameters to gain greater intuition. Finally, we outline the potential usefulness of L-SR1 in distributed training of deep neural networks.

TL;DR: We describe L-SR1, a new second order method to train deep neural networks.

Review discussion:
Dear Authors,
1) How does the experimental results look in terms of the wallclock time?
2) The generalized trust region method that overcomes the saddle point problem is presented in the work "Identifying and attacking the saddle point problem in
high-dimensional non-convex optimization", where they avoid expensive Hessian computations by approach similar to Krylov subspace descent. This work has to be contrast in more details with your approach. Could you do that?
1. Our current L-SR1 implementation is not optimized and we will add wallclock time shortly.
But essentially it requires one more function calculation (about 30% of the time for a gradient calculation) each iteration.
So we expect an optimized L-SR1 implementation to take 30% more time than SGD like methods.
2. We will contrast our approach with the approach presented in "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization", in our next revision.
We have contrasted our approach with the approach presented in "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization", in the section where we review recent related work.
Please describe the time and space complexity of your approach
and remake some plot where x-axis shows time and not epochs.
Worst case time and space complexity is described, and a plot has been remade where x-axis is time and not epochs.
It is an interesting idea to go after saddle points in the optimization with an SR1 update and a good start in experiments, but missing important comparisons to recent 2nd order optimizations such as Adam, other Hessian free methods (Martens 2012), Pearlmutter fast exact multiplication by the Hessian. From the mnist/cifar curves it is not really showing an advantage to AdaDelta/Nag (although this is stated), and much more experimentation is needed to make a claim about mini-batch insensitivity to performance, can you show error rates on a larger scale task?
We have experimentally compared our approach to Adam. We have also discussed the Hessian free method of Martens and Pearlmutter fast exact multiplication by the Hessian in our literature review. We have added new experimentation that studies mini-batch insensitivity.
L-SR1 seems to have O(mn) time complexity. I miss this information in your paper.
Your experimental results suggest that L-SR1 does not outperform Adadelta (I suppose the same for Adam).
Given the time complexity of L-SR1, the x-axis showing time would suggest that L-SR1 is much (say, m times) slower.
"The memory size of 2 had the lowest minimum test loss over 90" suggests that the main driven force of L-SR1
was its momentum, i.e., the second-order information was rather useless.
We have addressed the time complexity of our approach. Our new experimental results suggest that our approach is competitive with Adam. O(mn) is the worst case time complexity, and in practice, L-SR1 is not m times slower than SGD. With an optimized implementation, we expect L-SR1 to be about 30% slower.
Regarding your comment on second order information being useless when memory size is 2, it may be analytically shown that the Hessian approximation in this case is given by the secant equation, i.e. that it is equal to the ratio of the difference of gradients to the difference of iterates. This is second order information, that is used in the updates. It is not merely a linear combination of past gradients.
The paper proposes a new second-order method L-SR1 to train deep neural networks. It is claimed that the method addresses two important optimization problems in this setting: poor conditioning of the Hessian and proliferation of saddle points. The method can be viewed as a concatenation of SR1 algorithm of Nocedal & Wright (2006) and limited-memory representations Byrd et al. (1994).
First of all, I am missing a more formal, theoretical argument in this work (in general providing more intuition would be helpful too), which instead is provided in the works of Dauphin (2014) or Martens. The experimental section in not very convincing considering that the performance in terms of the wall-clock time is not reported and the advantage over some competitor methods is not very strong even in terms of epochs. I understand that the authors are optimizing their implementation still, but the question is: considering the experiments are not convincing, why would anybody bother to implement L-SR1 to train their deep models? The work is not ready to be published.
We think we have provided a sufficient theoretical argument for our approach, and it is not clear to us how this may be improved, given page limit constraints. In our new experiments, we have found the default version of L-SR1 to be slightly better than the default version of Adam, and competitive with the optimized SGD with momentum, for training a deep residual network.
While we agree that there is potential for more experimentation, we also believe there is compelling evidence to suggest the usefulness of our approach.

Meta-review:
role: PC
data: comment: The paper proposes an interesting approach, in that (unlike many second-order methods) SR1 updates can potentially take advantage of negative curvature in the Hessian. However, all reviewers had some significant concerns about the utility of the method. In particular, reviewers were concerned that the method does not show a significant gain over the Adam algorithm (which is simpler/cheaper and easier to implement). The public reviewer also points out that there are many existing quasi-Newton methods designed for DL, so it is up to the authors to compared to at least one of these. For these reasons I'm recommending rejection at this time.

Decision: Reject