EXACT ANALYSIS OF CURVATURE CORRECTED LEARNING DYNAMICS IN DEEP LINEAR NETWORKS
Authors: Dongsung Huh
Venue:  (2020)

Abstract: Deep neural networks exhibit complex learning dynamics due to the highly non-convex loss landscape, which causes slow convergence and vanishing gradient problems. Second order approaches, such as natural gradient descent, mitigate such problems by neutralizing the effect of potentially ill-conditioned curvature on the gradient-based updates, yet precise theoretical understanding on how such curvature correction affects the learning dynamics of deep networks has been lack- ing. Here, we analyze the dynamics of training deep neural networks under a generalized family of natural gradient methods that applies curvature corrections, and derive precise analytical solutions. Our analysis reveals that curvature corrected update rules preserve many features of gradient descent, such that the learning trajectory of each singular mode in natural gradient descent follows precisely the same path as gradient descent, while only accelerating the temporal dynamics along the path. We also show that layer-restricted approximations of natural gradient, which are widely used in most second order methods (e.g. K-FAC), can significantly distort the learning trajectory into highly diverging dynamics that significantly differs from true natural gradient, which may lead to undesirable net- work properties. We also introduce fractional natural gradient that applies partial curvature correction, and show that it provides most of the benefit of full curvature correction in terms of convergence speed, with additional benefit of superior numerical stability and neutralizing vanishing/exploding gradient problems, which holds true also in layer-restricted approximations.

Under review as a conference paper at ICLR 2020
CURVATURE-CORRECTE LEARNING DYNAMICS IN DEEP LINEAR NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep neural networks exhibit complex learning dynamics due to highly non- convex loss landscape. Second order approaches, such as natural gradient descent, mitigate such problems by neutralizing the effect of potentially ill-conditioned curvature, yet largely unknown how the current theory of deep learning gen- eralizes beyond gradient descent to these higher order learning rules. To answer these questions, we derive exact solutions to learning dynamics of deep linear networks under spectrum of curvature-corrected learning rules. Our analysis reveals that curvature corrected learning preserves a core feature of gradient de- scent, conservation law, such that the learning trajectory follows precisely the same path in the underlying manifold as gradient descent, only accelerating the temporal dynamics along the path. We also show that layer-restricted approxi- mations of natural gradient, which are widely used in most second order methods (e.g. K-FAC), can significantly distort the learning trajectory into highly diverging dynamics that significantly differs from true natural gradient, which may lead to undesirable network properties. We also introduce fractional natural gradient that applies partial curvature correction, and show that it provides most of the benefit of full curvature correction in terms of convergence speed, with additional ben- efit of superior numerical stability and neutralizing vanishing/exploding gradient problems, which holds true also in layer-restricted approximations.
INTRODUCTION
Difficulty in training deep neural networks arises from the fact that the network's input-output map fo(.) is nonlinearly related to its parameters 0. This causes non-convex loss landscape with prolifera- tion saddle-points and poorly-conditioned curvature where gradient-bases first order optimization methods perform poorly (Martens 2010 Dauphin et al. 2014). Second order methods, such as nat- ural gradient descent (Amari 1998), compensate for the effect of curvature by using the distance metric intrinsic to the space of input-output maps define the update steps (Pascanu & Bengio, 2013 Martens] 2014; [Bernacchia et al. 2018), rather than the parameter space. Recent advance- ments led to approximate implementations of these methods that prove efficient for practical scale applications (Ba et al. 2016 Grosse & Martens, 2016 Martens et al. 2018 Osawa et al. 2019).
Despite their practical effectiveness, however, the exact nature of such curvature-correcte learning process remains largely unknown. Do curvature-correate learning methods simply accelerate con- vergences towards the same minimum solutions as gradient descent, or do they impose implicit bias toward qualitatively different solutions?
As first step toward establishing theoretical understanding of these questions, we analyze the exact learning dynamics of deep linear networks under a spectrum of curvature-correete update rules. Deep linear networks provide an excellent mathematical framework for developing insightful theo- retical understanding of the complex inner workings of deep nonlinear networks (Goodfelllow et al.) 2016). Despite their simplicity, deep linear networks capture the essential nonlinear relationship be- tween network's 's input-output maps and their parameters, and exhibit comparable learning behavior to their nonlinear counterparts that can be exactly solved for rigorous analysis. Indeed, many recent works analyzed the learning trajectories of deep linear networks under gradient descent to compute the convergence rate under various initial conditions (Arora et al.) 20188]] b) Bartlett et al.) 2019 Du & Hu 2019), revealed decoupled modes of convergence dynamics explain the origin of multiple

---

Under review as a conference paper at ICLR 2020
stage-like loss profiles (Saxe al. 2013), and showed the implicit bias for regularization (Du al.
2018 Arora et al. 2019) and resistance to overfitting (Advani & Saxe 2017 Lampinen & Ganguli
2018 Poggio al. 20188) Yet, it is uncertain whether these convergence properties generally apply
for update rules beyond gradient descent.
Our contribution The main results are summarized as follows.
We derive generalized conservation law that describes the optimization paths of network parameters under gradient descent as well as servature-correcte update rules. Conse- quently, curvature correction only affects the speed of convergence without affecting other qualitative properties of parameter update process.
2. There is trade-off between map dynamics and parameter dynamics. The full curvature correction effect of natural gradient descent (NGD) completely linearizes the map learn- ing dynamics of deep networks, equivalent to that of shallow networks. Such complete linearization, however, sacrifices stability of parameter update dynamics to explode when gradient vanishes and vice versa.
3. We introduce regularized version of NGD that partially corrects for the effect of curva- ture, called vNGD, which facilitates the parameter update dynamics by eliminating the vanishing/exploding update problems. This makes the map dynamics slightly nonlinear, but more so than that of single hidden layer networks under gradient descent.
4. NGD makes the learning process prone to overfitting by simultaneously learning both the signal and the noise dimensions data, whereas v vNGD partially retains gradient descent's resistance overfitting by separating the time-scales between the signal and the noise dimensions.
5. The widely-used block-diagone approximation of NGD breaches the aforementioned con- servation law, resulting in highly divergent parameter update dynamics, which breaks the weight balance across layers. In contrast, block biokkdiiagonalization of vNGD preserves sta- bility of parameter update dynamics, yiclding efficient and stable learning algorithms.
SETUP AND NOTATIONS
Consider depth d network that consists of an input layer, d - hidden layers, output layer, and weight matrices w = [w; {wi] that connect the adjacent layers. The network's input-output map is
v = I Wis = Wd .. W1, such that fw (x) = uæ. The network learns the input-output statistics i by minimizing the squared-error loss:
of dataset D = {x* yd )
l(w) } ED|l war v/13] Tr [ (0- w. ).) w.)T const,
where ED is the expectation the dataset D, E, ED[xxT] is the input correlations, and w. = ED[yxT] Σ-1 Neglecting the constant term, the loss function is expressed as
L(w) = Tr AE,TT
A = W w.)
(1)
where denotes the displacement between and w..
Shallow networks (d 1, t0 wx) exhibit linear learning dynamics under gradient descent, whose convergence rates scale with eigenvalues of Σ,, In this case, curvature correction has the well-understood effect of normalizing the convergence rates, which is also achievable by simple pre-whitening of input correlation. Instead, we are interested the less- understddsto effect of how curvature correction facilitates the complex nonlinear dynamics of deep networks (d 2). There- fore, we consider pre-whitened input distribution Σ, = to isolate the nonlinear effect of curvature correction, but this condition not critical for the analysis.
2

---

Under review as a conference paper at ICLR 2020
Gradient and Hessian, We use bold symbols to collectively represent network parameters and
derivatives in array form. For example, to
W1 and g w2
8wm
AL
A W represent the
w A
aL
awz
continuous-time weight update and the gradient of depth d 2 network. Hessian is fully charac- terized by its operation on weight update, which, by definition, produces gradient update:
[w] Á + AJ p + Au
H و
(À = w2 + (22
However, true Hessian-bads based methods (e.g. Newton- Newton-Raphson method) can converge to any extrema types. To guarantee convergence to (local) minimum solutions, natural gradient methods use positive semi-definite (PSD) approximations of Hessian (e.g. Fisher matrix (Amari 1998; Heskes, 2000)
Martens & Grossel
2015) Bernacchia al. 2018), Siteraaized-GaussNNeee
matrix (Martens
2014
Botev et al. 2017 Roux et al. 2008
which correspond to
w A w
H
(3)
This operation is indeed PSD, since in H Tr(u w Á + d wi TH(ÁÂT] 2 0, where the dot-product denotes a == >d 1m Tr(abb]]. We refer to this operation as Hessian,.
Null-space and Conservation laws input-output map w is invariant under transformations that multiply arbitrary matrix m to one layer
Deep linear networks exhibit inherent symmetries that their
[w]] W2 W2m
mwi
mwa
and its inverse to the next layer, i.e.
+
Vm. ummll =
are the equivalent
W22th
continuous-time transformations that yield the invariance Á = Wotmw W2rnew 0, Vm.
These transformations form the null-space of H,, since u H. W Tr|Áåt] = 0, which is orthogonal to gradient, since Wmall g Tr[AÂT] = 0. Also orthogonal to the null-space is natural gradient, since :uulll Hig g H. H ritull O, where H denotes Moore-Penrose pseudo-inverse.
These continuous symmetries imply the following, self-explanatory theorem (Noether's theorem):
Theorem All update rules in that are orthogonal to the null-space, i.e.
wil Σ Tr([wai] w we+ )m,] = 0, Vm,
exhibit the following conservation law
d/dt (wam] w mx+1) = 0, Vi
(4)
This result was previously only known for gradient descent dynamics (Arora et al. 2018b Du et al. 2018), which is generalized here.
3 LEARNING DYNAMICS
In this section, analyze the learning dynamics of the network parameters (Section 3.1) and the update dynamics of the input-output map w (Section 3.2) under a spectrum of curvature-corrected update rules. We then analyze how block-diagonal approximation modifies the curvature-comrected dynamics (Section 3.3).
3.1
PARAMETER DYNAMICS
We follow the singular value decomposition (SVD)-based analysis of Saxe al. (2013); Advaní & Lampinen & Ganguli 2018), by considering network weights that are initialized to
Saxe (2017;
Fisher matrix and Snteraliee-GGausee Gewtt--Newton matrix equivalent in many cases, including the least squares problem considered here (Pascanu Bengio 2013 Martens 2014..

---

Under review as a conference paper at ICLR 2020
have their map's singular vectors aligned with those of w. Under such initialization, the update dynamics of weight matrices simplifies to their singular value dynamics, with their singular vec- tors remain unchanging. This simplified case admits exact analytic solutions, which provide good approximation to general learning dynamics. Moreover, this aligned singular vector condition is au- tomatically satisfied for networks initialized with small random weights [axe al. (2013): Advani & Saxe (2017).
Steepest gradient descent (SGD) Under SGD update, deep networks' weight parameters exhibit coupled nonlinear dynamics: (d example, 1y: learning rate)
wi + 1 w A
i + 1/
Ut 7) A A = O.
(5)
The SVD analysis decomposes eq (5) singular mode is described by See S.I.)
individual singular mode dynamies. The dynamics of one
B
d
6, + - =0
(os = 5 - J,, 5 ==
II 0.)
(6)
==1
where d; G., c, dg are the singular values of Wi, W., w. A, and j; = 05/00; =0/0; denotes the coupling between the input-output map and parameters, i.e. Jacobian. Note that this singular mode dynamics follows the hyperbolic paths
of o constant,
Vi,k
which is the direct consequence of the conservation law to the displacement |aal and the coupling strength Il.ill
(4
The update speed ||]|| is proportional
| x | ||اااا,
(10=1 = 1b:1 == of)
(8)
which vanishes for networks with small coupling strength and explodes for large coupling strength.
Natural gradient descent (NGD) NGD finds the minimum-norm update solution (min v 1b)
subject to the constraint (i.e. Moore- pseudo-inverse solution)
w ( + )
H. + ng =
(á + + = 0,
(9)
which can be solved using Lagrange multipliers to yield (See S.I.)
i + 17 wi A
0,
(10)
جh + 17 A
where A satisfies
= S(A)) = 0.
(S(A) = (1) A + A(w[wi)= - A)
(11)
Remarkably, the only change from SGD update (5) replacing A with A as the main drive of
dynamics eq (10). which preserves orthogonality to null null-space and hence the conservation law 4
The singular mode dynamics of NGD update eq (T)) i
à: A
0,
(12)
||
where da of SGD (6 is replaced by の = / / /|/l| the singular values of A (See S.I.) NGD dynamics eq (12 follows the same hyperbolic paths SGD eq (7). but with modified update speed
||
(13)
which inversely scales with ||:|]. Therefore, NGD's update speed explodes for small coupling strength, reciprocal to SGD's vanishing speed phenomenon.
2Given SVD of weight matrices w. L.D.R and w, L.D.RI, where D are diagonal singu- lar value matrices and L/R are the left/right singular vector matrices, the aligned singular vector condition assumes Rı = Ro, La = L,, and R++1 = T. for all layers 1 < i 1.
The dynamics eq 6 (2) apply to all active singular modes. Inactive modes that have The number active modes is determined by the bottleneck size, fee. the narrowest width network.
stay frozen.
'The Moore-Po Penroee pscudo-inverse solution is guaranteed to be orthogonal to the null-space, since non- zero null-space components only increase the solution's norm without affecting the constraint eq (9)
4

---

Under review as a conference paper at ICLR 2020
SGD
B
NGD
C
VNGD
0
D
NGD-d
E
vNGD
02
2/0,
3,6,
0
J6.
01
01
Figure 1: Learning dynamics of singular mode of single hidden layer network (d 2). The contour lines visualize the manifolds of constant displacement levels dA = 0102 ō.. The op- timal solution aa of Hessian,. The vector field visualizes the displacement-normaalie amplitude is the normalized update speed: ||+l//[al x ||i||=22/, (A,B,C) SGD, NGD, vNGID share the same update directions defined by the hyperbolic paths that conserve of of (red lines), orthogonal to the null-space. But they exhibit different update speed: SGD exhibits vanishing speed problem for small weights, while NGD has the opposite problem. In contrast, vNGD exhibits con- stant normalized speed. (D) NGD-d exhibits radially diverging vector field that conserves a 10/0-- The learning trajectories of NGD and NGD-d traverse the contour lines with synchronized timing (red dots), (E) vNGD-d exhibits vector field of constant direction and amplitude that conserves |ai| - [02)
shown in black. Tangent space of the manifolds defines the null-space
update [ō1, b1.02//(oa// whose
Fractional Natural Gradient Descent (/GG) Above results can be generalized a spectrum
of update rules that apply partial curvature corrections, described by HH + ng = 0, where /H, is fractional power of Hessian, (q 2 1). The singular mode dynamics of VNGD is
o, + 7
0,
(14)
1//|2/4
which interpolates between NGD (q = 1) and SGD (q - xi). Eq (14) follows the same hyperbolic
paths of SGD eq 77) but with modified update speed
|| x |aa |||i 1-2//
(15)
Note that for q termed vNGD, the update speed becomes independent of the coupling strength
||²||
|as|,
(16)
thus eliminating the vanishing/exploding update speed problems of SGD/NGD (See Fig 1C).
Relation to Regularized NGD
Alternative interpolation solves (H,u + +gg) cI(w ng)
(c 2 0), which yields the regularized (or damped) inverse
i = ( + 1) + H.)
(17)
similar to Govenberg-Maqquane damping (less the (c + 1) term), whose singular mode dynamics is
ji Diil
a +
o.
||lll O,
(a = =a= lblii
(18)
+
6 This expression reduces to SGD in the limit + oc, which differs from the usual regularized inverse -nee H.) g. which reduces to O.
5

---

Under review as a conference paper at ICLR 2020
A
B
p 1/2
p
p
2
10
20
500
1000
1500
Loss
0.01
0.01
0.01
0.01
2 time
10
20
500
1000
1800
time
Figure 2: Learning curves of ouputt map and loss profile for various stiffness p levels: p corresponds to NGD update, 1/2 < p < corresponds to vNGD update, and S p < corresponds to SGD update for network depth ranging between 2 s d
Top: Learning curves of map singular modes (c) from eq (21). Dashed lines show the mode-strengte of dataset o.. Note that large p increases the stiffness of dynamics, i.e. extreme changes of time-scale: between extreme slow and extreme fast. Half-max points (black circles) are shown to visualize the overall time-scale of learning dynamics, which decreases with mode strength as 0. Bottom: Corresponding loss profiles. Initial conditions: (()) 0 for p < and (a) 5./100 for perpee 1
where the ratio a = e/|ljill describes the effective degree of interpolation between NGD (a - 0) and SGD (a + oc). Note that should be large enough provide sufficient damping, but too large to nullify the effect of curvature correction, which is difficult to simultaneously satisfy across all singular modes with fkxed e. vNGD can be considered as providing ideally and adaptively tuned regularization
for all singular modes, where the regularization most effective.
3.2 MAP DYNAMICS
The parameter update ultimately drives the learning dynamics of the input-output map, via Jacobian
d
ō J
(19)
which yields the following map learning dynamics under & NGD update
14
â = - o.)11i
(20)
In general, eq 20 does not admit closed-form solutions due the coupling strength term, with the exception of NGD (q 1). As shown by the vector field in Figure 1, however, the coupling strength changes in streotypical manner along the learning trajectories. Therefore, the general charac- teristics of map dynamics can be appreciated from the representative case of balanced weights: o; = 1/d Vi, or in terms of the conserved quantities, w.w! wT,, w:+1 = O. Note that this bal- anced weight condition automatically approximately satisfied if the networks are initialized with small random weights.
Under the balanced weight condition, eq 20 simplifies to
(p = 2(d - 11 - 1)
8= ( - 5.) T
(21)
dg
where 7 = 17 dd'-1/6 the depth--alibratee learning rate, and p represents the combined effect of depth and curvature correction that determines the stiffness, degree of nonlinearity, of map

---

Under review as a conference paper at ICLR 2020
dynamics. Figure 2 shows the following notable closed-form solutions, as well as p = 2 case:
(t) = o. -c e -
(p - 0)
(c) o. tanli ((Vo.112)
(p 0.5)
0+
(1) 1+ (ō./0 1)e
(p (p=11 1)
n0.t nj.t
where zero initial condition (0) 0 is assumed for p < cases.
NGD update (q = 1, 0) Under NGD update, the map dynamics exhibits fully linearized convergence dynamics with constant time-scale 7) for all depth d and data mode-strength o... Its learning curves exhibit finite growth rate near zero (()) = -jō,t, which entails exploding parameter update speed as the coupling strength approaches zero. Therefore, the full curvature correction of NGD sacrifices stability of parameter dynamics in order to perfectly cancel out all nonlinearities of map dynamics.
vNGD update (q 2, p= 1- 1/d) For vNGD update, the stiffness ranges from p 0.5 for single hidden layer networks to p + 1 in infinite depth limit. Its learning curves exhibit polynomial growth near zero, ((t) x t1/(1-)), which takes finite time to escape from zero initial condition. even though the initial growth rate vanishes with the coupling strength. The overall time-scale of learning decreases with mode strength as T_?, such that stronger singular modes (large ō.) learn faster than weaker modes.
SGD update (q - co, p p= 2/d) Under SGD update, the stiffness ranges from p for single hidden layer networks to p + 2 in infinite depth limit. Its learning curves exhibit sigmoidal shape that take infinite time to escape from the saddle point at zero initial condition: the escape time diverges as (-- (o)) for p
and (0) for p 1. Also, the increased p causes greater separation of time-scales (No²) -1 across singular modes, which results in stage-like transitions over the course training, with each singular mode making sudden transition from slow learning to rapid convergence (Saxe al. 2013).
Effective Depth Network depth d and curvature correction q interact in a symmetric manner, which can be intuitively understood by representing stiffness in terms of the corresponding network depth under SGD update, called the effective depth:
dg
dan d d+q-1'
(22)
which approaches the actual depth der + d in the SGD limit (g - 20), and similarly, approaches der + q in the limit of infinite depth (d + oo). Therefore, VNGD reduces the network's 's effective depth to be strictly less than q. For vNGD, this upper-limit is 2, í.e. single hidden layer network.
To summarize, curvature correction lowers the nonlinearty//siffees of map dynamics of deep net- works by reducing their effective depth. The full curvature correction effect of NGD perfectly cancels out all nonlincarities of map dynamics to exhibit linear convergence, equivalent to shallow network learning, but it sacrifices stability of parameter dynamics to explode at the saddle point. In contrast, partial curvature correction of vNGD directly facilitates the parameter update dynam- ies, which eliminates the vanishing/exploding update problem, and it makes the map dynamics only slightly nonlinear, but no more so than that of single hidden layer networks under gradient descent.
3.3 EFFECT OF LAYER-RESTRICHE RESTRICTED APPROXIMATION
Block-diagonal NGD (NGD-d) In most practical deep learning applications, numerically esti- mating and inverting Hessian, becomes prohibitively expensive. Instead, most second-order meth- ods approximate NGD by applying layer-restricted curvature corrections, ignoring the off-block-
diagonal Hessian, terms across different layers (Martens & Grosse 2015) Ba et al. 2016 Grosse & example)
Martens 2016; Martens et al.) 2018 Bernacchia al. 2018): (d
Wi + 7191] H2 + 1292
wh W2 un 77 1 A
0.
(23)
W [ + 77 Aw]
7

---

Under review as a conference paper at ICLR 2020
which nevertheless satisfies the NGD constraint (9 if
Σ
?i 7). H, denotes the block-diagona
Hessian, term of layer i. Singular mode dynamics of eq
(23
(with 17, = n7/d)
77 Ō, +
O,
(24)
dj
where the layer-restricted factor of substitutes the full curvature correction factor |lill of NGD (2)
This olock-diagonalizatti significantly modifies the parameter update dynamics by adding non- zero null space components. Instead of the hyperbolic paths (7). eq (24) follows radially diverging paths that conserve 0i/OR constants of motion. Consequently, NGD-d update exhibits larger parameter update speed than NGD and converges to less efficient, large norm solutions that are highly sensitive to initial conditions and perturbations (Fig ID, red line). Despite the vastly dif- ferent parameter dynamics, however, NGD-d exhibits identical map learning dynamics as NGD I -n (o σ.) (Fig IBD, red dots), because the input-output map is invariant under null-space transformations.
Block-diagonal
NGD (vNGD-d) More generally, block-diagonalized fractional NGD
H: w; + -99//d = 0 yields
702 , 1-2 =0, ///9 Ji
à;
(25)
which conserves of singular mode dynamies
2(1-1/4) 1/4) as constants of motion. For q 2, called vNGD-d, the
-Ok
nos sign (j:) = 0,
+
(26)
vd
follows non-diverging, parallel paths that conserve |o,1-lox|, with identical parameter update speed VNGD's || 7 | (Fig 1E). Therefore, VNGD-d yields neutrally-stable update dynamics that neutralizes the vanishing/exploding update speed problems.
4 IMPLICIT BIAS FOR REGULARIZATION
Recent works have shown that learning dynamics of deep networks under SGD update exhibits implicit bias towards keeping the network well regularized. Here, we consider two such properties, and analyze how they generalize under curvature-corrected update rules.
Weight balance Deep neural networks often exhibit redundant parameterizations, such that con- figurations of parameters implement the same input-outpur map. One such redundancy, or symmetry, that concerns both deep linear networks and deep ReLU networks is homogeneity, such that multi- plying a layer by a positive scalar e and divide another layer by c, does not change the input-output map. The problem is that valid solutions. Such unboundedness poses major theoretical difficulty for convergence analysis of gradient-based local optimization methods (Lee et al.) 2016: Shamir 2018).
be arbitrarily large small, yielding potentially unbounded, yet
Fortunately, SGD update exhibits implicit bias toward automatically balancing the norm of different layer's weight matrices. The proof directly follows from the following conserved quantity of scalar multiplication symmetry |[w I || which is a relaxed version of the aforementioned conservation law eq Thus, if the weights are initially small, this differences between squared norm will remain small through out learning process, thus establishing balancedness across layers Du et al. (2018).
As shown in section curvature-corrected updates (e.g. NGD and vNGD) retain orthogonality to the null-space of symmetry, and thus comply with the same conservation laws as SGD. We show numerical confirmation of this prediction in S.I. The conservation of squared difference of norms for homogeneous ReLU networks still requires similar numerically confirmation.
In contrast, block- diok-diiggn methods do not follow the same conservation law. NGD-d conserves the ratio between singular values across layers 0./0k. which does not guarantce balancedness even
7 || 2 2 ||&D can be shown using Jensen's 's incquality: 1 Di
8

---

Under review as a conference paper at ICLR 2020
with small initialization. VNGD-d, however, conserves the absolute difference of singular values across layers |oy| |o%) which guarantees balancedness, at least under the condition of aligned singular vectors: That is, the ratio between the singular values would approach close to 1 if they grow from small initial values, while maintaining the small absolute difference. Although this does not constitute a formal proof for general case, vNGD-d confirms to maintain balancedness across layers in numerical simulations (See S.I.).
Low rank approximation / Generalization dynamics The learning dynamics of the input-output map under SGD update separates the time-scales of learning across singular modes (7ja²)_1 such that the singular modes with stronger data correlation preferentially are learned faster (Saxe et al.) 2013). This property yields an implicit regularization property for deep networks to efficiently extract low-rank structure of the dataset, such as for finding matrix f'actorizations with minimum nuclear norm (Gunasekar et al.) 2017 Arora al., 2019). It also allows deep networks to avoid overfitting via early stopping by first learning the signal dimensions of noisy dataset, before the overfitting of the noise dimensions occurs, as long as signal signal-to-noiss to-noise ratio is sufficiently large [Ad-] vani & Saxe (2017): [Lmppien & Ganguli (2018) which yields good generalization performance to unseen data. However, this approach requires the network to trained from small random weight initialization, where SGD suffers from vanishing gradient problem.
In curvature-correcte cases, the learning speed of map dynamics eq 21 scales o. Under NGD, the map dynamics is perfectly linearized (p = 0), which also removes its ability to separate out the time-scales. This makes NGD update prone to large generalization error due learning the noise dimension simultaneously with the signal. In contrast, vNGD partially retains time-scale separation in the learning dynamics, while also accelerating the parameter update dynamics near zero weights.
We the generalization property curvature corrected learning rules with student-teacher task from Lampinen & Ganguli (018)), in which the training and test dataset generated by teacher net- work y = W,.' + 2tt where x" E R-V is the input data, yd e R is the output, W..x is the signal and =d R.V is the noise. Teacher's input-output map W. N is assumed to have low-rank structure (rank 3), and the student is a depth d = 4 network of constant width N = 16, whose weight matrices are initialized have maximum singular value of 0.05. The number of training dataset {x", y" "p= is set to be equal to the effective number of parameters P = N, which makes the learning process most susceptible overfitting.
For numerical calculation of NGD and NGD, the Hessian block between layer and l: are com- puted as described in [Bernacchia et al. (2018) (eq 42), which are then concatenated to the full Hessian,, matrix and numerically inverted (or sqrt-inverted) via eigen-decomposition. Levenberg- Marquardt damping of = 10- and update clipping are used for numerical stability of NGD. VNGD does not require such clipping or damping terms.
Figure shows the result of training. SGD exhibits stage-like transitions, which first learns the three signal modes, well separated from the onset of overfitting of the noise modes begins, which allows effective early stopping scheme. However, suffers from the long plateaus due to vanishing gradient problem.
NGD (and NGD-d) updates learn all singular modes simultancously including the noise modes (See Fig 3 D), which leads to high generalization error. Note that NGD's loss profile deviates from exponential decay due to the clipping. In contrast, vNGD (and vNGD-d) allows fast learning while separating the signal dimensions from the noise dimensions, achieving comparable test loss as SGD update, but also with fast early-stopping time comparable to NGD update. Note that all three update rules achieve the same test loss after overfitting is complete. This is due to the shared learning path for each singular mode across the methods.
5 CONCLUSION
To summarize our contribution, we derived a generalized conservation law that describes the opti- mization paths of network parameters under gradient descent well curvature-correcte update rules. Consequently, curvature correction only affects the speed of convergence without affecting other qualitative properties of parameter update process.

---

Under review as a conference paper at ICLR 2020
Strength
B
10
10-1
NGD-4
10"2
10-2
10"1
10"1
0
Singular
10
15
122
10
122
Ieration
30"
1.2
307
12'
101
10'
Itoatton
10"
141
10"
<RGD
VSiD -d
20-
101
30a
107
122
300
1.2
10
10'
107
122
122
1.7
307
307
10'
121
120
Iteratio
Figure 3: Curvature correction effect generalization dynamics: (A) Singular mode strength of input-output correlation of training dataset. Dataset is generated from a rank-3 teacher network with added noise (SNR 10). (B, C) Training and testing loss profiles of 3-hidden-layer student network (See text). Note that all methods eventually converge to the identically overfitted solution (horizontal dashed-line).
(D) Time-separated learning dynamics of across singular modes. To obtain the individual mode components, we computed the network's 's input-output correlation matrix, and projected it to the singular vector basis of the data correlation matrix.
We revealed trade-off' between map dynamics and parameter dynamics: The full curvature cor- rection effect of natural gradient descent (NGD) completely linearizes the map learning dynamics of deep networks, equivalent to that of shallow networks. Such complete lincarization, however, sacrifices stability of parameter update dynamics to explode when gradient vanishes and vice versa. Moreover, we introduced VNGD that partially corrects for the effect of curvature, which facili- tates the parameter update dynamics by eliminating the vanishing/exploding update problems. This makes the map dynamics slightly nonlinear, but no more so than that of single hidden layer net- works under gradient descent. Moreover, NGD makes the learning process prone overfitting by simultaneously learning both the signal and the noise dimensions of data, whereas vNGD partially retains gradient descent's resistance to overfitting by separating the time-scales between the signal and the noise dimensions. We also showed that the widely-used block-diagonal approximation of NGD breaches the aforementioned conservation law, resulting in highly divergent parameter up- date dynamics, which breaks the weight balance across layers. In contrast, block-diagonalization of VNGD preserves stability of parameter update dynamics, yielding efficient and stable learning algorithms.
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXh::1710.03667 2017.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. arXiv preprint arXiv:1810.02281,, 2018a.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizatton.. arXiv preprint arXiv: 1802.0680000 2018b.
Sanjeev Arora, Naday Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. arXiv preprint arXiv: /905. 13655, 2019.
10

---

Under review as a conference paper at ICLR 2020
Jimmy Ba, Roger Grosse, and James Martens. Distributed second-order optimization using conneckr-ffcctrre approximations. 2016.
Peter L Bartlett, David P Helmbold, and Philip M Long. Gradient descent with identity initializa- tion efficiently learns positive-definite linear transformations by deep residual networks. Neural computation, 31 31(3):477-502 2019.
Alberto Bernacchia, Mate Lengyel, and Guillaume Hennequin. Exact natural gradient in deep linear networks and its application to the nonlinear case. In Advances Neural Information Processing Systems, pp. 5941-5950, 2018.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In Proceedings of the 34th International Conference on Machine Yearning-Volumee 70, pp. 557-565, JMLR. org, 2017.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. In Advances in neural information processing systems, pp. 2933-2941, 2014.
Simon S Du and Wei Hu. Width provably matters in optimization for deep linear neural networks. arXiv preprint arXiv: 190110572. 2019.
Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in Neural Information Processing Sys-
tems, pp. 384-395, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Roger Grosse and James Martens. A kronecker- factored approximate fisher matrix for convolution layers. International Conference on Machine Learning, pp. 573-582, 2016.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pp. 6151-61555 2017.
Tom Heskes. On "natural" learning and pruning in multilayered perceptrons. Neural Computation, 129,) 2000.
Andrew Lampinen and Surya Ganguli. An analytic theory generalization dynamics and transfer learning in deep linear networks. arXiv preprint 13Xiv:8809.10374 2018.
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only
converges to minimizers. In Conference on learning theory, pp. 1246-1257, 2016.
James Martens. Deep learning via hessian-free optimization. In ICML, volume 27, pp. 735-742, 2010.
James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193, 2014.
James Martens and Roger Grosse. Optimizing neural networks with kronecker- approximate curvature. In International conference on machine learning, pp. 2408-2417, 2015.
James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. 2018
Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka. Large-scale distributed second-order optimization using kronecker-factee approximate curva- ture for deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 22359-12367 2019.

---

Under review as a conference paper at ICLR 2020
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.
Tomaso Poggio, Qianli Liao, Brando Miranda, Andrzej Banburski, Xavier Boix, and Jack Hidary. Theory iib: Generalization in deep networks. arXiv preprint arXiv: 2018.
Nicolas L Roux, Pierre-Antoing Manzagol, and Yoshua Bengio. Topmoumoute online natural gra- dient algorithm. In Advances in neural information processing systems, pp. 849-856, 2008.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.61220 2013.
Ohad Shamir. Are resnets provably better than linear predictors? In Advances in neural information processing systems, pp. 507-516, 2018.
Supplemental Materials
S.1.1 MOORE-PENROSE INVERSE SOLUTION: EQ (15,16,11 IN SECTION 3.2
In section 4, find the Moore-Penrose inverse solution that minimizes the update norm v - Σ, tr] while satisfying the natural gradient constraint: (d = 2 example)
W ( + + (Á + >
H ú 7/
0.
(S.I.)
This constrained optimization problem is described by the following Lagrangian:
L(wi, vi2 A2, A2) = (1i1 ui + 1hs /22 + A - wo + 7)) + A2 (Á + n) Jue.
Wythi + wiy and dot notation denotes inner-product: b Tr(a Optimality condition on vù; yields
where Á =
ac/0ú; = u + w]w2Ai + w. A2wi 0 OC/Owz = W - A2 w + Ar = 0
(S.1.2)
(S.1.)
which, via change of variables (w2A1 A2w1 1/1, reduces to
Φ + mwo. A = 0 dh. + nmwr
(S.1.)
(S.1.5)
which can be plugged into the optimality condition on
ac/0A1 w/( (Á nイ) 0C/0A. = (À + n W =0
(S.1.6)
(S.1.7)
to produce linear equation for A4:
S(A) = (A)) = where S(A) = (W2wJ)A + A(w w)
(S.L8)
(S.I.9)
Note that if W2, w1 are invertible, it is easy to that eq converging dynamics À + A à + 77w wt) = O. with the solution of S(A) 0 driving the parameter update eq (S.144 This result also holds true for the over-complete cases, where the hidden layer width is larger than the minimum of input layer output layer size. For the under- complete cases, i.e. with bottleneck hidden layers, the exponential convergence applics only to the subspace dimensions permitted by the bottleneck, with the other dimensions remain frozen.
reduce to exponentially

---

Under review as a conference paper at ICLR 2020
d
d
d 3
d
5
560
-0234
NG2
1111
MG0-4
304
10'
10.
10
10
302
10
10
10 301 10
1.5
10
10
30 107
10
307 10
Figure S.I.1: Scaling relationships for normalized loss update L/L and weight/parameter update speed |/tll//L. shown across different depth and curvature correction factor q. a the over- all scale factor for the weights. Dotted-lines show predictions for the case of balanced weights, eq
The numerical results show the result from networks of random weights sam- pled from Gaussian distribution, (Xavier normalization (Glorot & Bengio 2010 They exhibit the same power-law exponent as the dotted-lines (i.e. same slope), but shifted toward right. Block- diagonal approximation does not affect the power-law exponents, but it affects the coefficient such that NGD-d exhibits much larger parameter update size than NGD, while exhibiting identical zero power-law for loss update. In contrast, sNGD-d exhibits identical match with sNGD for both pa- rameter update and loss update. See S.I. for detailed setup. Error bars show standard deviation from 10 simulations with random weights.
S.I.2 SINGULAR MODE ANALYSIS BQ (18) IN SECTION 3.2
We follow the SVD-base analysis under the aligned singular vector condition (Saxe et al. 2013). We introduce og, a TA, which represent the singular values of w,, A, A. S(A) of one particular singular mode, and j: Ba.
da
II σy In this representation, eq S.1.4 S.I.5 reduce to
à; ajj
(S.L.00
whereas, eq
S.L.8 S.1.9 reduce to
0,05 =
(S.L.11)
d o
7
(S.1.12)
1=1
An active singular mode should have at least one non-zero og, which according to eq (S.I.1)
implies 0s O. Therefore, from eq (S.1.12).
σ = A
(S.1.13)
which plugs into S.I.10 produce the result in the main text
b: 7
(S.L.4))
EP13
S.I.3 SCALING LAWS OF UPDATE DYNAMICS
The parameter update (14 and map learning dynamics (20) of singular mode exhibit the following scaling relationships with respect to the coupling strength Till:
|| 11/11
å/0s -111122-21
(S.L.5)
2

---

Under review as a conference paper at ICLR 2020
defined for individual singular modes under the well-aligned singular vector condition.
For the balanced weight case where the coupling strength term is constant across all singular modes, eq (S.1.15) reduces to more generally applicable scaling relationships:
| x a(d-1)(12/01)
(S.1.16)
[ x (dd-1)(2-)
(S.L.17)
where a the overall scale factor for the weight matrices. Instead of the detailed description of individual singular mode dynamics, eq (6. encapsulate the overall scaling law between weights and update speed that can be readily measured without requiring the aligned singular vector condition.
Under SGD (q + oo), weight update scales as d-1 and double the power for the loss update, exhibiting the vanishing update problem for small a. Under sNGD (g 1/2), weight update is constant (zero power-law) with respect to and loss update scales as dd Under NGD (g 1), loss update constant (zero power-law) with respect a, but the weight update inversely scales as (d-11)) exhibiting the exploding update problem for small a.
Numerical experiments indeed confirm these scaling laws (Figure S1): For the case balanced weights the predictions hold exactly, and approximately for random weight matrices. Note that the zero-power law predictions for weight update under sNGD and for loss update under NGD are exact, because they indeed satisfy the balanced condition: the coupling strength term with zero exponent [|5l1 is indeed constant.
4 NUMERICAL CONFIRMATION OF CONSERVATION LAW
See Figure S2. Figure S2 plots the learning trajectory of 3 layer network, and shows the elements of the weight matrices evolving over time (W), W2, Wz). also shows the conserved quantities wrwr wi uz, a2ws w us, which indeed remain constant for SGD, NGD and vNGD, while blows up for NGD-d. NGD also violates the conservation law, but the weights remain balanced over time.

---

Under review as a conference paper at ICLR 2020
535
NGD
1800
N0D
Loss
100
100
10
W3
W22
141
15
18
100
150
mm 0/2
12210
011
121
Iteration #
Figure 2: Training of a linear 3-layer network (D = 2).


Reviews:
[{"review_id": "Bkgq6wO6KS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the training dynamics of natural gradient descent with linear DNNs. \nSpecifically, they showed that the curvature corrected natural gradient descents preserve the learning trajectory of plain gradient descent and only affect the temporal dynamics. A fractional natural gradient \ndescent method that only applies partial curvature correction is proposed to address the numerical stability \nissue of vanilla natural gradient descent.\n\nThe paper is well presented and the derivations of analytical solutions are clear. Although the analysis is \nlimited to the linear case, it does give some insight into the behaviors of curvature corrected gradient descents.\nHowever, it would be interesting to see how these algorithms would perform under the no-linear case. Also, \nwill these results hold for other neural network structures? For example, the Hessian of each layer of\nResNet will be close to orthogonal and how will that affect the NGD?\n\nOverall I think this is an interesting paper with strong results and vote for accepting.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxrxSCgqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors analyse curvature corrected optimization methods in the context of deep learning. They build their analysis on Saxe et.al.s work. They show that curvature corrected methods preserve properties of SGD. They also show the disadvantages of layer restricted approximations. They show the importance of time scales in optimization. The paper looks to deep learning from a dynamical systems perspective and hence their experiments are fitting to this framework.\nI have checked the analysis and it seems solid. However, my only concerns it the use case. I would like to see such methods show difference in real world datasets. I believe this is a big missing part of the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx3CVotqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, the authors analyze the dynamics of training deep linear neural networks under a generalized family of natural gradient methods that apply curvature corrections. They first show that the learning trajectory (direction of singular mode dynamics) in natural gradient descent follows the same path as gradient descent, while only accelerating the temporal dynamics along the path. Moreover, the authors show that the learning trajectory in layer-restricted approximations of natural gradient descent can significantly differ from the true natural gradient. Also, the authors proposed a fractional natural gradient that applies partial curvature correction which in addition to faster convergence, neutralizes vanishing/exploding gradient problems. \n\nI vote to reject this paper due to the following major concerns.\n1.\tThe results of the paper are limited to specific deep linear neural networks which are not practical. More specifically, in addition to only covering the deep linear case, the results hold under the following unexplained assumptions: (a) the correlation matrix of the input data is assumed to be whitened; (b) the singular vectors of adjacent weight matrices are well aligned; (c) Section 4, the singular values of all the weight matrices are the same.\n\n2.\tThe paper lacks a cohesive introduction that includes a literature review on this very rich area of research.\n\n3.\tThe paper is not clearly written and is missing many details.\n\nI discuss my comments on the manuscript in details next.\n\n•\tWhile the assumption on the input correlation matrix being whitened can be dropped, other assumptions in the paper are unrealistic and are stated without any intuition or explanation (the singular vectors of adjacent weight matrices are well aligned; (c) Section 4, the singular values of all the weight matrices are the same.)\n  \n•\tThe analysis performed in the paper cover the deep linear case and no intuition is provided on how these results can help in understanding the more general non-linear case. By citing [Saxe et al. 2013], the authors claim that the effect of curvature of loss landscape on deep learning dynamics can be well captured by deep linear networks. However, in [Saxe et al. 2013] this was only shown empirically on a specific example. In recent years, we have encountered many results that hold in deep linear networks but fail to hold when including simple non-linearities. For instance, when the target matrix Y is full rank, every local minimum of the studied problem is global in the deep linear case. However, this result does not hold even with the simplest non-linearities.\n\n•\tThe paper is unpolished and the objectives and contributions are not clearly stated. Moreover, the results are not rigorously stated and proved. Moreover, the algorithms stated and proposed in the paper are directly used without any introduction or explanation (how they work? What update rules do they use?  …)\n\n•\tThe introduction in the paper is very short and does not include a literature review on this rich area of research. Hence, the problem is not motivated by a gap in the literature and not linked to other results that study the behavior of algorithms used for training deep neural networks.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to study the effect of curvature correction techniques on training dynamics. The focus is on understanding how natural gradient based methods affect training dynamics of deep linear networks. The main conclusion of the analysis is that it does not fundamentally affect the path of convergence but rather accelerates convergence. They also show that layer correction techniques alone do not suffice. In the discussion the reviewers raised concerns about extrapolating too much based on linear networks and also lack of a cohesive literature review. One reviewer also mentioned that there is not enough technical detail. These issues were partially addressed in the response. I think the topic of the paper is interesting and timely. However, I concur with Reviewer #2 that there are still lots of missing detail and the connection with the nonlinear case is not clear (however the latter is not strictly necessary in my opinion if the rest of the paper is better written). As a result I think the paper in its current form is not ready for publication. 
Decision: Reject