Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: L-SR1 seems to have O(mn) time complexity. I miss this information in your paper. 
Your experimental results suggest that L-SR1 does not outperform Adadelta (I suppose the same for Adam). 
Given the time complexity of L-SR1, the x-axis showing time would suggest that L-SR1 is much (say, m times) slower. 
"The memory size of 2 had the lowest minimum test loss over 90" suggests that the main driven force of L-SR1 
was its momentum, i.e., the second-order information was rather useless.
review_title: O(mn)?

final_score: rating: 5: Marginally below acceptance threshold
review_content: It is an interesting idea to go after saddle points in the optimization with an SR1 update and a good start in experiments, but missing important comparisons to recent 2nd order optimizations such as Adam, other Hessian free methods (Martens 2012), Pearlmutter fast exact multiplication by the Hessian. From the mnist/cifar curves it is not really showing an advantage to AdaDelta/Nag (although this is stated), and much more experimentation is needed to make a claim about mini-batch insensitivity to performance, can you show error rates on a larger scale task?
review_title: Address better optimization at saddle points with symmetric rank-one method which does not guarantee pos. def. update matrix, vs. BFGS approach. Investigating this optimization with limited memory version or SR1

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a new second-order method L-SR1 to train deep neural networks. It is claimed that the method addresses two important optimization problems in this setting: poor conditioning of the Hessian and proliferation of saddle points. The method can be viewed as a concatenation of SR1 algorithm of Nocedal & Wright (2006) and limited-memory representations Byrd et al. (1994). First of all, I am missing a more formal, theoretical argument in this work (in general providing more intuition would be helpful too), which instead is provided in the works of Dauphin (2014) or Martens. The experimental section in not very convincing considering that the performance in terms of the wall-clock time is not reported and the advantage over some competitor methods is not very strong even in terms of epochs. I understand that the authors are optimizing their implementation still, but the question is: considering the experiments are not convincing, why would anybody bother to implement L-SR1 to train their deep models? The work is not ready to be published.
review_title: Interesting work, but not ready to be published

final_score: rating: null
review_content: Dear Authors,

1) How does the experimental results look in terms of the wallclock time?
2) The generalized trust region method that overcomes the saddle point problem is presented in the work "Identifying and attacking the saddle point problem in
high-dimensional non-convex optimization", where they avoid expensive Hessian computations by approach similar to Krylov subspace descent. This work has to be contrast in more details with your approach. Could you do that?
review_title: Clarifications

final_score: rating: null
review_content: Please describe the time and space complexity of your approach 
and remake some plot where x-axis shows time and not epochs.
review_title: complexity

Meta-review: The paper proposes an interesting approach, in that (unlike many second-order methods) SR1 updates can potentially take advantage of negative curvature in the Hessian. However, all reviewers had some significant concerns about the utility of the method. In particular, reviewers were concerned that the method does not show a significant gain over the Adam algorithm (which is simpler/cheaper and easier to implement). The public reviewer also points out that there are many existing quasi-Newton methods designed for DL, so it is up to the authors to compared to at least one of these. For these reasons I'm recommending rejection at this time.
Decision: Reject