Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Compared to the other reviewer I found the approach interesting. While I'm not so keen on exact time complexity, algorithmically the approach seems scalable. I agree that the experimental section is a bit disappointing, and that there might be real concerns on how this particular approximation of the curvature works in practice. But given that it is a workshop submission, I find the proposal very simple and elegant, and I wager that with a bit of care and dedication it could work surprisingly well in practice. 

My score however rests heavily on the fact that this is just a workshop submission, I think a lot of work still needs to be done to convert this work in a proper paper. 
review_title: Interesting approach

final_score: rating: 3: Clear rejection
review_content: I am not quite sure about the time complexity of O(m^3 + n). 
" The algorithm does require computation of the eigenvalues and eigenvectors of the m × m matrix G^H × G". Should not G^H \times G first be computed? Given than G is in R^{m x n}, I would expect m x n somewhere in the complexity formula. To compute g as the average of gradients you would need m x n, right?

The experimental results are disappointing:
a) SGD as the only baseline and no comparison with second-order methods and their approximates/alternatives
b) little networks of 16k parameters which raises the question of scalability  
c) weird hyperparameter selection "we keep most of our hyper-parameters constant, including learning rates (0.0003 and 0.01)" given that "several experiments diverged when using too large a learning rate, whereas this was beneficial to the convergence rate of SGD" suggesting that the selection of the learning rate was in favor of the proposed approach.
review_title: Time complexity, baselines, hyperparameter selection

Decision: Accept