Revisiting Natural Gradient for Deep Networks
Authors: Razvan Pascanu, Yoshua Bengio
ICLR 2014

Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model.


Peer review:
[Review]:
Thank you for the review. All your points are well taken. The only thing I can say in my defense is that I was worried of the paper being already too long, but you are right. It is important to make the paper clear about what it tries to say. Needless to say I will fix all the typos pointed out (thank you for spotting them). 

I will also try to add  more on the effect of the minibatch size and number of iterations for inverting G. What I can say for now, based on sporadic observations that I've made is that the number of iterations is not that important as long as it is larger than some threshold. The threshold does depend on both the data and model that you use (basically on the error surface). 


Regarding the minibatch size, my claim is that as long as you have a learning rate in front of your gradient, and you are willing to damp your metric (some of these things might result in slower convergence) you can go to much smaller minibatches. For the NNCG algorithm, going to smaller minibatches seems to hurt. 

My understanding of this behaviour is as follows. NNCG (or even NGD with a line search) tries to move at each step as much as it can to minimize the cost on that minibatch. If the minibatch is not sufficiently large, what happens is that you end up overfitting the current minibatch, resulting in overall worse training error and validation error. Doing the linesearch on a different minibatch than the one used to compute the gradient helps to some degree (makes this overfitting to happen less, as the gradient might not point exactly in the directions required to overfit the minibatch), it is important to have a sufficiently large minibatch to have the same statistics as the whole training set (if we only care about optimization). Some of these ideas are also presented in section 7, but I agree that they are in a very condense manner. I will try to expand on this idea, also providing more evidence (extra experiments). 

I will write back once I've updated the paper with a detail list of changes. It might take me a few days though (especially with the ICML deadline in the background).