Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content:  
This paper presents a new 2nd-order algorithm that implicitly uses curvature information, and it shows the intuition behind the approximation schemes in the algorithms and also validates the heuristics in various experiments.  The method involves using Neumann Series and Richardson iteration to avoid Hessian-vector product in second order method for NN.  In the actual performance, the paper presents both practical efficiency and better generalization error in different deep neural networks for image classification tasks, and the authors also show differences according to different settings, e.g., Batch Size, Regularization.  The numerical examples are relatively clear and easy to figure out details.

1. While the paper presents the algorithm as an optimization algorithm, although it gets better learning performance, it would be interesting to see how well it is as an optimizer.  For example, one simple experiment would be showing how it works for convex problems, e.g., logistic regression.  Realistic DNN systems are very complex, and evaluating the method in a simple setting would help a lot in determining what if anything is novel about the method.

2. Also, for deep learning problems, it would be more convincing to see how different initialization can affect the performances. 

3. Although the authors present their algorithm as a second order method at beginning, the final algorithm is kind of like a complex momentum SGD with limited memory.  Rather than simply throwing out a new method with a new name, it would be helpful to understand what the steps of this method are implicitly doing.  Please explain more about this.

4. It said that the algorithm is hyperparameter free except for learning rate.  However, it is hard to see why there is no need to tune other hyperparameters, e.g., Cubic Regularizer, Repulsive Regularizer.  The effect/sensitivity of hyperparameters for second order methods are quite different than hyperparameters for first order methods, and it is of interest to know how hyperparameters for implicit second order methods perform.

5. For Section 4.2, the well know benefit by using large batch size to train models is that it could reduce training time and epochs.  However, from Table 3, there is no such phenomenon.  Please explain.


review_title: See below.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The paper proposes Neumman optimizer, which makes some adjustments to the idealized Neumman algorithm to improve performance and stability in training. The paper also provides the effectiveness of the algorithm by training ImageNet models (Inception-V3, Resnet-50, Resnet-101, and Inception-Resnet-V2). 
 
Comments:
I really appreciate the author(s) by providing experiments using real models on the ImageNet dataset. The algorithm seems to be easily used in practice. 

I do not have many comments for this paper since it focuses only in practical view without theory guarantee rigorously. 

As you mention in the paper that the algorithm uses the same amount of computation and memory as Adam optimizer, but could you please provide the reason why you only compare Neumann Optimizer with Baseline RMSProp but not with Adam? As we know, Adam is currently very well-known algorithm to train DNN. Do you think it would be interesting if you could compare the efficiency of Neumann optimizer with Adam? I understand that you are trying to improve the existing results with their optimizer, but this paper also introduces new algorithm.  

The question is that, with the given architectures and dataset, what algorithm should people consider to use between Neumann optimizer and Adam? Why should people use Neumann optimizer but not Adam, which is already very well-known? If Neumann optimizer can surpass Adam on ImageNet, I think your algorithm will be widely used after being published.  
 
Minor comments:
Page 3, in eq. (3): missing “-“ sign
Page 3, in eq. (6): missing “transpose” on \nabla \hat{f}
Page 4, first equation: O(|| \eta*mu_t ||^2)
Page 5, in eq. (9): m_{k-1}

review_title: Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new algorithm, where they claim to use Hessian implicitly and are using a motivation from power-series. In general, I like the paper.

To me, Algorithm 1 looks like some kind of proximal-point type algorithm. Algorithm 2 is more heuristic approach, with a couple of parameters to tune it.  Given the fact that there is convergence analysis or similar theoretical results, I would expect to have much more numerical experiments. E.g. there is no results of Algorithm 1. I know it serves as a motivation, but it would be nice to see how it works.

Otherwise, the paper is clearly written.
The topic is important, but I am a bit afraid of significance. One thing what I do not understand is, that why they did not compare with Adam? (they mention Adam algorithm soo many times, that it should be compared to).

I am also not sure, how sensitive the results are for different datasets? Algorithm 2 really needs so many parameters (not just learning rate). How \alpha, \beta, \gamma, \mu, \eta, K influence the speed? how sensitive is the algorithm for different choices of those parameters?


 
review_title: [UPDATED] Interesting idea, nice experiments, good motivation but lacks theoretical understanding

Meta-review: Pros:
+ Clearly written paper.
+ Easily implemented algorithm that appears to have excellent scaling properties and can even improve on validation error in some cases.
+ Thorough evaluation against the state of the art.

Cons:
- No theoretical guarantees for the algorithm.

This paper belongs in ICLR if there is enough space.

Decision: Accept (Poster)