Revisiting Natural Gradient for Deep Networks
Authors: Razvan Pascanu, Yoshua Bengio
ICLR 2014

Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model.


Peer review:
[Review]:
This paper explores the link between natural gradient and techniques such as Hessian-free or Krylov subspace descent. It then proposes a novel way of computing the new search direction and experimentally shows its value.

I appreciate getting additional interpretations of these learning techniques and this is a definite value of the paper. The experiments, on the other hand, are weak and bring little value.

Additional comments:
- To my knowledge, Krylov subspace descent is much older than 2012. It is for instance mentioned in Numerical Optimization (Nocedal) or 'Iterative scaled trust-region learning in Krylov subspaces via Peralmutter's implicit sparse Hessian-vector multiply' (Mizutani and Demmel). As it is, the literature looks very deep learning centric.

- I appreciate the comment about using different examples for the computation of the Fisher information matrix and the gradient. However, it would be interesting to explore why using the same examples also hurts the training error. This also goes against the fact that using other examples from the training set yields a lower training error than using examples from the unlabeled set.

- Figure 1: why do you use a minibatch of size 256 in the first case and a mainibatch of size 384 in the second? This introduces an unnecessary discrepancy.
- Figure 1: 'top' and 'bottom' should be 'left' and 'right'.

- Figure 2 is not very convincing as the red curve seems to drop really low, much lower than the blue one. Thus, you prove that NGD introduces less variance than MSGD, but you do not prove that the ratio of the importance of each example is different in both cases. Could you use a log scale instead?

- Finally, the experimental section is weak. The only 'deep' part, which appears prominently in the title, consists of an experiment ran on one deep model. This is a bit short to provide a convincing argument about the proposed method.

Pros: nice interpretation of natural gradient and connections with existing algorithm

Cons:
Weak experimental section
The paper could be better written
It feels a bit too 'deep' centric to me, whether in the references or the title despite its little influence on the actual paper.