Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a block-diagonal hessian-free method for training deep networks. 

- The block-diagonal approximation has been used in [1]. Although [1] is using Gauss-Newton matrix, the idea of "block-diagonal" approximation is similar. 

- Is the computational time (per iteration) of the proposed method similar to SGD/Adam? All the figures are showing the comparison in terms of number of updates, but it is not clear whether this speedup can be reflected in the training time. 

- Comparing block-diagonal approximation vs original HF method: 
It is not clear to me what's the benefit using block-diagonal approximation. Is the time cost per iteration similar or faster? 
Or the main benefit is to reduce #CG iterations? (but it seems #CG iterations are fixed for both methods in the experiments). 
Also, the paper mentioned that "the HF method requires many hundreds of CG iterations for one update". Is this true?
 Usually we can set a stopping condition for solving the Newton system.

- It seems the benefit of block-diagonal approximation is marginal in CNN. 

[1] Practical Gauss-Newton Optimisation for Deep Learning. ICML 2017. 
review_title: The paper proposes a block-diagonal second order method for training deep networks. The algorithm is not  novel. Experimental results are good in training auto-encoder and LSTM (in terms of number of updates). 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The paper considers second-order optimization methods for training of neural networks.
In particular, the contribution of the paper is a Hessian-free method that works on blocks of parameters (this is a user defined splitting of the parameters in blocks, e.g., parameters of each layer is one block, or parameters in several layers could constitute a block). 
This results into a block-diagonal approximation to the curvature matrix, in order to improve Hessian-free convergence properties: in the latter, a single step might require many CG steps, so the benefit from using second-order information is not apparent.
This is mainly an experimental work, where the authors show the merits of their approach on deep autoencoders, convolutional networks and LSTMs: results show favourable performance compared to the original Hessian-free approach and the Adam method.

Originality: 
The paper is based on the works of Collobert (2004) and Le Roux et al. (2008), as well as the work of Martens: the twist is that each layer of the neural network is considered a parameter block, so that gradient interactions among weights in a single layer are more useful than those between weights in different layers. This increases the separability of the problem and reduces the complexity. 

Importance: 
Understanding the difference between first- and second-order methods for NN training is an important topic. Using second-order methods could be considered at its infancy, compared to the wide variety of first-order methods. Having new results on second-order methods with interesting results would definitely attract some attention at the conference. 

Presentation/Clarity: 
The paper is well structured and well written. The authors clearly place their work w.r.t. state of the art and previous works, so that it is clear what is new and what is known.

Comments:
1. It is not clear why the deficiency of first-order methods on training NNs with big batches motivates us to turn into second-order methods. Is there a reasoning for this statement? Or is it just because second-order methods are kind-of the only other alternative we have?

2. Assuming we can perform a second-order method, like Newton's method, on a deep NN. Since originally Newton's method was designed to find solutions that have gradient equal to zero, and since NNs have saddle points (probably many more than local minima), even if we could perfectly perform second-order Newton motions, there is no guarantee whether we converge to a local minimum or a saddle point. However, since we perform Newton's method approximately in practice, this might help escaping saddle points. Any comment on this aspect (I'm not aware whether this is already commented in Schraudolph 2002, where the Gauss-Newton matrix was proposed instead of the Hessian)?

review_title: There is nothing particularly wrong with the paper - it is a nice work, that is based a lot on previous attempts, like those in Martens. Coming from a more theoretical background, I would like to see more theory. Nevetheless this does not lessens the value of the paper. For the moment, weak accept until I also read the other reviews. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, authors discuss the use of block-diagonal hessian when computing the updates. The block-diagonal hessian makes it easier to solve the "newton" directions, as the CG can be run only on smaller blocks (and hence less CG iterations are needed).

The paper is nicely written and all was clear to me. In general, I agree that having larger batch-size is the way to go, for very large datasets and a pure SGD type of methods are having problems to efficiently utilize large clusters.

The only negative thing I find in the paper is the lack of more numerical results. Indeed, the paper is clearly not a theoretical paper, is proposing a new algorithm, hence there should be evidence that it works. For example, I would like to see how the choice of hyper-parameters influences the speed of the algorithm. Was "CG" cost included in the "x"-axis? i.e. if we put "passes" over the data as x-axis, then 1 update \approx 30 CG + some more == 32 batch evaluation.
So please try to make the "x"-axis more fair.

 
review_title: Nice paper, however, I would be happier if more experiments on larger datasets are presented

Meta-review: Pros:
+ Clearly written paper.

Cons:
- Limited empirical evaluation: paper should compare to first-order methods with well-tuned hyperparameters, since the block Hessian-free hyperparameters likely were well tuned, and plots of convergence as a function of time need to be included.
- Somewhat limited novelty in that block-diagonal curvature approximations have been used before, though the application to Hessian-free optimization is new.

The reviewers liked the clear description of the proposed algorithm and well-structured paper, but after discussion were not prepared to accept it primarily because (1) they wanted to see algorithmic comparisons in terms of convergence vs. time in addition to the convergence vs. updates that were provided; (2) they wanted more assurance that the baseline first-order optimizers had been carefully tuned; and (3) they wanted results on larger scale tasks.

Decision: Reject