Revisiting Natural Gradient for Deep Networks
Authors: Razvan Pascanu, Yoshua Bengio
ICLR 2014

Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model.


Peer review:
[Review]:
The paper contains some interesting analysis showing how certain previous methods
can be viewed as implementations of Natural Gradient Descent.

The paper is novel and is on an important topic, I believe. There are parts where
I believe the paper could be clearer (see detailed comments below)- parts could
be expanded upon.  I believe the ICLR length guidelines are not strict.

There are also some interesting experiments that compare NG (and its variants)
to SGD.  (it would be nice to have some more details on where the time is taken
in NG, and what happens when you change the size of the subset used to compute
the metric, and the number of iterations used in approximate inversion of G).

Detailed comments follow.


-----------
density probability function -> probability density function

not sure how p_{	heta}(z) relates to DNNs-- what is z?  The input features?  The class-label output?  Both?

Definition of Fisher information matrix?

In Sec. 2.1 when you talk about 'the probabilistic interpreation p_{	heta}(t | x)', this is very
unclear.  I figured out what you meant only after looking at the Appendix.  In fact, the notation
is inherently quite unclear because for me, the canonical example of deep learning is with softmax
output, and here you view t as a multinomial distribution over a single categorical variable,
which would be a scalar not a vector, so the use of bold-face t is misleading, because it's more
like p_{	heta}(y | x) where y is the discrete label.  I think if
you want people to understand this paper it would make sense to devote a paragraph or two  here to
explaining what you mean.

computed by model -> computed by the model


Around Eq. (15), where you talk about the Levenburg-Marquardt heuristic, it was
very unclear to me.  I think you need to take a step back and explain what it is
and how you are applying it.  Otherwise (and this is a risk with other parts of
the paper too) there is a danger that the only people who can understand it are
those who already understand the material so deeply that the paper's conclusions
would already be obvious.

betweent -> between

Eucladian -> Euclidean

batch model -> batch mode

Riebiere -> Ribiere (twice)

in Sec. 10-- I don't think you address how many iterations you use in the solver for
 approximately mutiplying by G^{-1}, and how much of the total time this takes.



--
Appendix:

be obtain -> be obtained

should sum_i^o be instead sum_{i=1}^o ?  (this appears more than once)

In the last line of Eq. 27, the notation doesn't really work, you are treating the vector y like a scalar,
e.g. dividing by it.  This same notation appears in the main text.
It might be necessary to put, say, diag({f p}), and explain that p_i = 1/(y_i (1-y_i)).
Or maybe you can point out somewhere in the text that these equations should be interpreted
elementwise.
In eqs 28 and 29 you should have non-bold in y_i, and also there is an odd inconsistency of notation with
eq. 27-- you drop the J [something] J^T notation which I think is still applicable here.

certain extend -> certain extent

The text 'Following the functional manifold interpretation..' with eqs. 30 and 31 in the appendix,
is quite unclear.  I think you are maybe assuming a deep familiarity with Martens' Hessian-Free
paper and/or the Levenburg-Marquardt algorithm.