Information Geometry of Orthogonal Initializations and Training
Authors: Piotr Aleksander Sokół
Il Memming Park
Venue:  (2020)
Keywords: Fisher
mean-field
deep learning

Abstract:     Recently mean field theory has been successfully used to analyze properties
    of wide, random neural networks. It gave rise to a prescriptive theory for
    initializing feed-forward neural networks with orthogonal weights, which
    ensures that both the forward propagated activations and the backpropagated
    gradients are near \(\ell_2\) isometries and as a consequence training is
    orders of magnitude faster. Despite strong empirical performance, the
    mechanisms by which critical initializations confer an advantage in the
    optimization of deep neural networks are poorly understood. Here we show a
    novel connection between the maximum curvature of the optimization landscape
    (gradient smoothness) as measured by the Fisher information matrix (FIM) and
    the spectral radius of the input-output Jacobian, which partially explains
    why more isometric networks can train much faster. Furthermore, given that
    orthogonal weights are necessary to ensure that gradient norms are
    approximately preserved at initialization, we experimentally investigate the
    benefits of maintaining orthogonality throughout training, and we conclude
    that manifold optimization of weights performs well regardless of the
    smoothness of the gradients. Moreover, we observe a surprising yet robust
    behavior of highly isometric initializations --- even though such networks
    have a lower FIM condition number \emph{at initialization}, and therefore by
    analogy to convex functions should be easier to optimize, experimentally
    they prove to be much harder to train with stochastic gradient descent. We
    conjecture the FIM condition number plays a non-trivial role in the optimization.

Published as a conference paper at ICLR 2020
INFORMATION GEOMETRY OF ORTHOGONAL INITIALIZATIONS AND TRAINING
Piotr Aleksander Sokół and II Memming Park
Department of Neurobiology and Behavior Departments of Applied Mathematics and Statistics, and Electrical and Computer Engineering Institutes for Advanced Computing Science and AI-driven Discovery and Innovation Stony Brook University, Stony Brook, NY 11733 [memming, meemming.park, piotr. sokol 0 @esttnsbrooo.eeoo
ABSTRACT
Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near &2 isome- tries and as consequence training orders of magnitude faster. Despite strong empirical performance, the mechanisms by which critical initializations confer advantage in the optimization deep neural networks are poorly understood. Here show novel connection between the maximum curvature of the optimization landscape (gradient smoothness) as measured by the Fisher information matrix (FIM) and the spectral radius of the input-output Jacobian, which partially explains why more isometric networks can train much faster. Furthermore, given that or- thogonal weights are necessary to ensure that gradient norms are approximately preserved at initialization, we experimentally investigate the benefits of maintaining orthogonality throughout training, and we conclude that manifold optimization of weights performs well regardless of the smoothness of the gradients. Moreover, we observe surprising yet robust behavior of highly isometric initializations even though such networks have lower FIM condition number initialization, and therefore by analogy to convex functions should be easier to optimize, experimen- tally they prove to be much harder train with stochastic gradient descent. We conjecture the FIM condition number plays a non trivial role in the optimization.
INTRODUCTION
Deep neural networks (DNN) have shown tremendous success in computer vision problems, speech recognition, amortized probabilistic inference, and the modelling of neural data. Despite their performance, DNNs face obstacles in their practical application, which stem from both the excessive computational cost of rumning gradient descent for a large number of epochs, as well as the inherent brittleness of gradient descent applied to very deep models. A number of heuristic approaches such as batch normalization, weight normalization and residual connections (He et al. 20166] loffe & Szegedy 2015 Salimans Kingmal 2016) have emerged an attempt address these trainability issues. Recently mean field theory has been successful in developing a more principled analysis of gradients of neural networks, and has become the basis for a new random initialization principle. The mean field approach postulates that in the limit of infinitely wide random weight matrices, the distribution of pre-activations converges weakly
Gaussian. Using this approach, series of works proposed to initialize the networks in such way that for each layer the inpuf-output Jacobian has mean singular values of (Schoenholz et al. (207). This requirement was further strengthened to suggest that the spectrum of singular values of the input-output Jacobian should concentrate 1, and it was shown that this can only be achieved with random orthogonal weight matrices.
Under these conditions the backpropagated gradients are bounded in l2 norm (Pennington al. 2017) irrespective depth, i.... they neither vanish nor explode. It was shown experimentally in (Pennington et al.) 2017 Xiao et al. 20188) Chen et al.) 2018) that networks with these critical initial conditions train orders of magnitude faster than networks with arbitrary initializations. The empirical success invites questions from optimization perspective how the spectrum of the hidden layer

---

Published as a conference paper at ICLR 2020
input-output Jacobian relates to notions of curvature of the parameters space, and consequentially to convergence rate. The largest effective (initial) step size d/ for stochastic gradient descent is inversely proportional to the local gradient smoothness M (Bottou et al. 2018 Boyd & Vandenberghe, 2004). Intuitively, the gradient step can be at most as large as the fastest change in the parameter landscape. Recent attempts have been made analyze the mean field geometry of the optimization using the Fisher information matrix (FIM) (Amari et al. 2019 Karakida et al. 2019). The theoretical and practical appeal of measuring curvature with the FIM due to among other reasons the fact that the FIM is necessarily positive relationship with the Hessian matrix. Karakida al. (2019) derived an upper bound on the maximum eigenvalue, however this bound is not satisfactory since it is agnostic of the entire spectrum of singular values and therefore cannot differentiate between Gaussian and orthogonal weight initalizations.
Semiiidefinite even for non-convex objectives, and due to it its intimate
In this paper, we develop new bound on the parameter curvature M given the maximum eigenvalue of the Fisher information A which holds for random neural networks with both Gaussian and orthogonal weights. We derive this quantity inspect the relation between the singular value distribution of the input-output Jacobian and locally maximal curvature of the parameter space. We use this result to probe different orthogonal, nearly isometric initializations, and observe that, broadly speaking, networks with a smaller initial curvature train faster and generalize better, as expected. However, consistent with previous report (Pennington al.] (018),, we also observe highly isometric networks perform worse despite having a slowly varying loss landscape (i.e. small initial Amax (max(G)) (G)). We conjecture that the long term optimization behavior is depends on trivially on the smallest eigenvalue m and therefore surprisingly there is a sweetspot with the condition number being m M
We then investigate whether constraining the spectrum of the Jacobian matrix of each layer affects optimization rate. We do so by training networks using Riemannian optimization constrain their weights to be orthogonal, or nearly orthogonal and we find that manifold constrained networks are insensitive to the maximal curvature at the beginning of training unlike the unconstrained gradient descent (hereafter "Fuclidean"). In particular, we observe that the advantage conferred by optimizing over manifolds cannot be explained by the improvement of the gradient smoothness as measured by Amax ( G). Finally, we observe that contrary to Bansal et al. 2018)'s results Euclidean networks with a carefully designed initialization reduce the test misclassification error at approximately the same rate their manifold constrained counterparts, and overall attain higher accuracy.
BACKGROUND
2.1 FORMAL DESCRIPTION OF THE NETWORK
Following (Pennington al. 2017 2018 connected neural network with L hidden layers. Each layer E [i.... the form
Schoenholz et al. 2017). consider a fordaforward, fully ........) L} is given as a recursion of
oh'' n'= w'x' +b'
(1)
where x are the activations, h' are the pre-activations, W/ V'''' are the weight matrices, b' are the bias vectors, and (b()) is the activation function. The input is denoted as x". The output layer of the network computes & = (h9) where g is the link function of some generalized linear model (GLM) and ha W + ba.
The hidden layer input-output Jacobian matrix Jx x0
0xL Ox
J
D''' 1=1
where D' is a diagonal matrix with entries D 2017, Schoenholz et al. 22017), the conditioning of the Jacobian matrix affects the conditroning of the mack-propagater gradients for all layers.
d'(h)) As pointed out in (Pennington et al.
2.2 CRITICAL INITIALIZATIONS
Extending the classic result on the Gaussian process limit for wide layer width obtained by (Neal 1996). recent work (Matthews et al., 2018: Lee et al., (0188) has shown that for deep untrained 2 networks with elements of their weight matrices Wi, drawn from Gaussian distribution AO0 N7) )
2

---

Published as a conference paper at ICLR 2020
the empirical distribution of the pre-activations h' converges weakly to Gaussian distribution NO I) for each layer in the limit of the width N + cc. Similarly, it has been postulated that random orthogonal matrices scaled by ow give rise to the same limit. Under this mean-field condition, the variance of the pre-activation distribution q' is recursively given by,
q' = f o @ dp(h) + of
where u(k) denotes the standard Gaussian measure f dh /2 of the Gaussian distributed biases (Schoenholz et al.) 2017). The variance of the first layer pre- R
exp (=+²) and of denotes the variance
activations q¹ depends on Eg norm squared of inputs q² defined in equation 3 has a fixed point
[p(x) of The recursion NY
q' = = (V du(h) + of
(4)
which can be satisfied for all layers by appropriately choosing ow, ob and scaling the input x". To permit the mean field analysis of backpropagated signals, the authors (Schoenholz et al., 2017; Pennington et al., 2017; 2018 Karakida et al. (019) further assume the propagated activations and back propagated gradients to be independent. Specifically,
Assumption 1. /Mean field assumptions]
(i) -o h 4 __ 9-)
(ii) lim Cov [ h' J [khhhh]- [hhh==] =0 for all +
Under this assumption, the authors (Schoenholz et al., 2017; Pennington et al. 2017) analyze distributions of singular values of Jacobian matrices between different layers terms of small number of parameters, with the calculations of the backpropagated signals proceeding in selfsame fashion as calculations for the forward propagation of activations. The corollaries of Assumption [ and condition in equation 4 is that d' (h') for S 1 < I. are i.i.d. In order to ensure that J is well conditioned, (Pennington et al. (017) require that in addition to the variance of pre-activation being constant for all layers, two additional constraints be met. Firstly, they require that the mean square singular value of DW for each layer has certain value in expectation.
1 [DD =ow / [WWTTIJ dp(h)
x =
(5)
Given that the mean squared singular value the Jacobian matrix J is (x)², setting X = corresponds to critical initialization where the gradients asymptotically stable L + co. Secondly, they require that the maximal squared singular value siax of the Jacobian Jx be bounded. Pennington et al. (2017) showed that for weights with Gaussian distributed elements, the maximal singular value increases linearly in depth even if the network is initialized with x = 1. Fortunately, for orthogonal weights, the maximal singular value 8max is bounded even as T. + xc (Pennington jtt al. 2018).
3
THEORETICAL RESULTS: RELATING THE SPECTRA OF JACOBIAN AND FISHER INFORMATION MATRICES
To better understand the geometry of the optimization landscape, we wish to put Lipschitz bound on the gradient, which turn gives an upper bound on the largest step size of any first order optimization algorithm. For a general objective function f. the condition is equivalent to
[=)) v fz'))l2 M| 2' for all x. CSCR
The Lipschitz constant ensures that the gradient doesn't it change arbitrarily fast with respect to I, x', and therefore f defines a descent direction for the objective over a distance M. In general estimating the Lipschitz constant is NP-hard (Kunstner al.) 2019), therefore we seek to find local measures of curvature along the optimization trajectory. As we will show below the approximate gradient smoothness is tractable for randomly initialized neural networks. The analytical study of Hessians of random neural networks started with (Pennington & Bahri (017), but was limited to shallow architectures. Subsequent work by Amari et al. (2019) and Karakida et al. (2019)
3

---

Published as a conference paper at ICLR 2020
second order geometry of random networks shares much of the spirit of the current work, in that it proposes to replace the possibly indefinite Hessian with the related Fisher information matrix as a measure of curvature. The Fisher information matrix plays fundamental role in the geometry of probabilistic models under the Kullhack-Leible divergence loss - defines a (local) Riemannian metric, which in turn defines distances the manifold of probability distributions generated by the model. Notably, the FIM does not define a unique metric on this statistical manifold, and alternative notions of intrinsic curvature can be derived by replacing the Kullback-Leibler divergence with the 2- Wasserstein distance (Li & Montufar 2018). Moreover, since the Fisher information matrix bears special relation the Hessian can also seen as defining an approximate curvature matrix for second order optimization. Recall that the FIM is defined as
Definition. Fisher Information Matrix
G Epo(y())) (Ep())) Woloopoly V Vu logpo xx) ']]
(6)
-ppoyyle)) Bp(xe)) J J V
Epo()(x))
Ep(xo)
H
Ex Vihh
(7)
where L denotes the loss and h is the output layer. The relation between the Hessian and Fisher Information matrices is apparent from equation [ showing that the Hessian H is quadratic form of the Jacobian matrices plus the possibly indefinite matrix of second derivatives with respect to parameters.
Our goal is to express the gradient smoothness using the results of the previous section. Given equa- tion we can derive an analytical approximation to the Lipschitz bound using the results from the previous section; i.e. we will express the expected maximum eigenvalue of the random Fisher information matrix in terms of the expected maximum singular value of the Jacobian Jh To do so, let us consider the output of a multilayer perceptron as defining a conditional probability distribution pe (y) (y))x),, where = {vec( (vec(W'). (W").... vec(W")). ..... b²} is the set of all hidden layer parameters, and is the column vector containing the concatenation of all the parameters in O, As observed by Martens & Grosse (2013) the Fisher of a multilayer network naturally has block structure, with each corresponding to the weights and biases of each layer. These blocks with respect to parameter vectors a, E O can further be expressed as
Go. Jhh H
where the final layer Hessian H, defined as Vh log (y(()x)). We can re-express the outer product of the score function Vhe log po (y|xx)) as the second derivative of the log-likelihood (see equation 6 provided it satisfies certain technical conditions. What is important for us is that all canonical link function for generalized linear models, like the softmax function and the identity function allow this re-writing, and that this re-writing allows us drop the conditional expectation with respect to pe (y) fy)x)). The Jacobians in equation 8 can be computed iteratively. Importantly the Jacobian from the output layer to the a-th parameter block is just the product of diagonal activations and weight matrices multiplied by the Jacobian from the a-th layer to the a-th parameter. We define these matrices of partial derivatives of the a-th layer pre-activations with respect to the layer specific parameters separately for Wa and o as:
Jh" - 81 Jh
for vec vec(we))) (Wa) for a = ba
(9)
(10)
Under the infinitesmally weak correlation assumption (see Assumption we can further simplify the expression for the blocks of the Fisher information matrix equation g Lemma 1. The expected blocks with respect to weight matrices for all layers a, 3 + are
Gve " veet = x 1 8-1
H.G he
(11)
Lemma 2. The expected blocks with respect to a weight matrix Wa and a bias vector bd are
G Gvec
[xx--xx a-1
h ha
H g hd
(12)
4

---

Published as a conference paper at ICLR 2020
10
CIFAR-I0 performance
0.0015
0.000000
56
Training cross entropy
00020
0.001755
0.000.0
Test: aww.aaccuuyy
0.00125
Fudidean
Fucldean
0.00000
Obique
o"
0.00075
Obliue g"
Snirld
of 1/66
0.00550
30
o" P0
10C Epochs
Epochs
SVHN performance
0.0220
00055
Training entropy
00033
Test accoracy [N]
0.0015
Fooes
Eudidaan o" 1/64
0.010
of se
Obiiuue
Oblque of le x.x 10m4
Stark g"
Stield
10
30 100
125
1501
175
530
Epochs
Figure 1: Network with manifold constrained weights are relatively insensitive to the choice of initial weight scaling q": We compare training loss and test accuracy for Euclidean, Stiefel and Oblique networks with two different values of q². The manifold constrained networks minimize the training loss at approximately the same rate, being faster than both Euclidean networks. Despite this, there is little difference between the test accuracy of the Stiefel and Oblique networks and the Euclidean networks initialized with q = 1/64. Notably, the latter attains a marginally higher test set accuracy towards the end of training.
The crucial observation here is that in the mean-field limit the expectation of the product of activations x²-1 8-1 either zero or rank for activations in different layers. The case when both activations are in the same layer is trivially taken care of by our mean-field assumptions the term equal to the 2nd non-central moment, iec the covariance plus potentially a rank one mean term.
Now, leveraging Lemmas and 2 we derive block diagonal approximation which in turn allows us to bound the maximum eigenvalue Amax (G). In doing so we will use a corollary of the block Gershgorin theorem.
Proposition 1 ((iformal) Block Gershgorin theorem). The maximum eigenvalue Amax (G) is contained a union of disks centered around the maximal eigenvalue of each diagonal block with radia equal to the sum of the singular values of the off-diagonal terms.
For a rigorous statement of the theorem see Appendix A. It is noteworthy that block-diagonal approximations have been crucial the application of Fisher Information matrices preconditioners in stochastic second order methods (Botev et al.) 2017 Martens & Grosse 2015). These methods were motivated by practical performance their choice of number of diagonal blocks used for preconditioning. Under the mean-field assumptions we are able to show computable bounds on the error in approximating the spectrum the Fisher information matrix. The proposition [ suggest simple, easily computable way to bound the expected maximal eigenvalue of the Fisher information matrix-choos. the block with the largest eigenvalue and expected spectral radia for the corresponding off diagonal terms. We do so by making auxiliary assumption:
Assumption 2. The maximum singular value of Jh monotonically increases as a J 1.
We motivate this assumption a twofold fashion: firstly the work done by Pennington et al. 2017 2018) shows that the spectral edge, i.e. the maximal, neg-negative singular value in the support of of the spectral distribution increases with depth, secondly it has been commonly observed numerical experiments that very deep neural networks have ill conditioned gradients.
5

---

Under this assumption it is sufficient to study the maximal singular value of blocks of the Fisher information matrix with respect to vec(W) ). b' and the spectral norms of its corresponding off- diagonal blocks. We define functions max of each block as upper bounds on the spectral bounds of Published as a conference paper at ICLR 2020
the respective block. The specific values are given in the following Lemma:
Lemma 3. The maximum expected singular values of the off- aff-diagonal blocks v3 + are bounded
by Emax(:) (-):
for weight-to-weight blocks
[ (G (eee(("),vec(": S Emax (Greo( (e ).ec))
(13)
VN [o(h]] E [xx] 112 max [Jh max (Hg)) B
T max
(Jhr hi'
(14)
for weight-lo-bias blocks
(Goee(Wa).b)) S Emax (W1))))) [omax
(15)
[E[)) E max Jhh
E [m (Hg)) E mmax (J
(16)
and for bias-to-bias blocks
[omax (Gut S 2max (Gb1,80) &
mmax
(Jh² }
max (H₂)) m hh
(17)
For proof see Appendix A.2
Note that the expectations for layers > is over random networks realizations and averaged over data x"; i.e. they are taken with respect to the Gaussian measure, whereas the expectation for first layer weights is taken with respect to the empirical distribution of x" (see equation 4. Depending on the choice of q" and therefore implicitly both the rescaling of x" and the values of [[o(h) the singular values of the weight blocks might dominate those associated with biases dominate compare equation 14 and equation 17
Theorem (Bound on the Fisher Information Eigenvalues). If LE [x"] aled with by will dominate, giving upper bound Amax (G)
< then eigenvalue associ-
E Amax
GG) S EE |maxx (Gbt,84)) }²) + Emax (GBt
+
>E
Σ Emax (Gyt,o)) + Emax (Gvec(b'), vec(())))
B>1
otherwise the maximal eigenvalue of the FIM bounded by
E [Amaa (G)] [a veoo)))) + Emax
Σ Emax (Grec(W'))bo)
8>1
+ Emax (.e
Moreover, it is interesting note two things. Firstly, E [[mmxx (Jh') factor appear in all the above summands. Secondly, we can bound Tmax for the diagonal blocks with E | (Hg)) Jomax (Jh:))² These fact reveal that the FIM maximum eigenvalue is up- per upperbounded by a quadratic function of the spectral radius of the input-output Jacobian; i.e. Amax (G) is O maa (Jh²
For proof see Appendix A.4
The functional form of the bound essentially quadratic in [omax (Jh') since the term appears in the summand as with powers at most two. This result shows that the strong smoothness, given by maximum eigenvalue of the FIM, is proportional the squared maximum singular value of the input-output Jacobian (see Fig. 2b. Moreover, the bound essentially depends q via the expectation E[o(h)]. through J and implicitly through H,, For regression problems this dependence is monotonically increasing in q" (Pennington al. 2018 2017 since H., is just the identity. However, this does not hold for all generalized linear models since Amax H )) is not necessarily a monotonically increasing function of the pre-activation variance layer h We demonstrate this in the case of softmax regression in the Appendix A.. Finally, to obtain a specific bound on Amax (G) we might consider bounding each E [o (Jhp" appearing in Theorem 3 in terms of its Frobenius norm. The corresponding result is the eigenvalue bound derived by Karakıda et al. 2019).

---

Published as a conference paper at ICLR 2020
Distribution of (G) vs (oan (2 as function of
Distribution of Avwr(0) vs (195)) as function of for MSE loss
1C*
104
Nemooris exprmnents Menorrs eat repeiiertt
162
Figure 2: Left: at initialization the maximum curvature of the loss landscape (measured by the Amax of the Fisher correlates highly (p 0.65) with the maximum squared singular value of the Jacobian J The choice of choice of the preactivation variance, q" affects not only the conditioning of the gradients but also the gradient Lipschitz constant. Right: we verify this separately for mean-squared error, for which a strictly monotonic relationship between qº and Amax and J is predicted, given that the Hessian of the output layer is the identity. This is corroborated by correlation coefficient of 0.81.
3.1 NUMERICAL EXPERIMENTS
We first verify the bound derived in section for networks with random orthogonal weights. We then numerically investigate the behavior of the maximum FIM eigenvalue during training with particular attention being paid to the possible benefits of maintaining orthogonality near orthogonality during optimization in relation to unconstrained networks. Following Pennington et al. (2017) we trained a 200 layer tanh network on CIFAR-10 and SVHNT and set the width of each layer be N 400 and chose the ow, ob in such way to ensure that mean singular value of the input-output Jacobian concentrates on 1 but smax varies a function of q" (see Fig. 2). We considered four different critical initializations with q² = |10-, 1 as well as the resulting training speed and final test accuracy as reported by (Pennington et al.. 2017). In the main text we focus on the smaller values since those networks should be closer to being isometric and therefore, by our theory, ought to train better. The remaining two networks with 1" 8] are presented in the Appendix A.5 To test how enforcing strict orthogonality or near orthogonality affects convergence speed and the maximum eigenvalues of the Fisher information matrix, we trained Stiefel and Oblique constrained networks and compared them to the unconstrained "Euclidean" network described in (Pennington et al., 2017). We used a Riemannian version of ADAM (Kingma& & Ba 2015). When performing gradient descent on non-Euclian Euclidean manifolds, split the variables into three groups: (1) Euclidean variables (e.g. the weights of the classifier layer, biases), (2) non-negative scaling ow both optimized using the regular version of ADAM, and (3) manifold variables optimized using Riemannian ADAM. The initial learning rates for all the groups, as well as the oonoorthoyynllity penalty (seef43) (4) for Oblique networks were chosen via Bayesian optimization, maximizing validation set accuracy after 50 epochs. All networks were trained with a minibatch size of 1000. We trained networks of each kind, and collected eigenvalue and singular value statistics every 5 epochs, from the first to the fiftieth, and then after the hundredth and two hundredth epochs.
64 2, 8). which differ both in spread of the singular values
Based the bound on the maximum eigenvalue of the Fisher information matrix derived in Section 3 we predicted that at initialization Amax G) should covary with o max (Jh²) (h)) We tested our prediction using the empirical Fisher information matrix (Kunstner et al. 2019) and we find a significant correlation between the two (Pearson coefficient = 0.64). We additionally test the relation for mean-squared error loss, and observe that our theoretical predictions hold better in this case the discrepancy can be attributed to the effect of the Hessian of the output layer, and its non-monotonic relation between preactivation variance and maximum eigenvalue of the Hessian of the GLM layer. The numerical values are presented in Fig. 2 Additionally we see that both the maximum singular value and maximum eigenvalue increase monotonically as a function of q² Motivated by the previous work by (aaxe et al. 2014) showing depth independent learning dynamics in linear orthogonal networks, we included instantiations of this model in the comparison. The input the linear
Code available http:///gggtttoooooooooooo//oo//

---

Published as a conference paper at ICLR 2020
network was normalized the same way the critical, non-linear networks with q" 1/64. The deep linear networks had a substantially larger Amax(G) than its non-linear counterparts initialized with identically scaled input (Fig. 2 Having established connection between q" the maximum singular value of the hidden layer input-output Jacobian and the maximum eigenvalue of the Fisher information, we investigate the effects of initialization subsequent optimization. As reported by (Pennington et al. (2017), the learning speed and generalization peak at intermediate values of q" & 10-0.5 This result is counter intuitive given that the maximum eigenvalue of the Fisher information matrix, much like that of the Hessian in convex optimization, upper bounds the maximal learning rate (Boyd & Vandenberghe 2004] Bottou et al., 2018). To gain insight into the effects of the choice of q" on the convergence rate, we trained the Enclidean networks and estimated the local values of Amax during optimization. At the same time we asked whether we can effectively control the two aforesaid quantities by constraining the weights of each layer to be orthogonal or near orthogonal. To this end we trained Stiefel and Oblique networks and recorded the same statistics. We present training results in Fig. [ where it can be seen that Euclidean networks with 9" ~ 9 x 10 perform worse with respect to training loss and test accuracy than those initialized with q" = 1/64. On the other hand, manifold constrained networks are insensitive to the choice of g" Moreover, Stiefel and Oblique networks perform marginally worse on the test set compared to the Euclidean network with g² = 1/64, despite attaining lower training loss. We observe that reduced performance of Euclidean networks initialized with g* > 9 x 10 may partially be explained by their rapid increase in Amax (G) within the initial 5 epochs of optimization (see Fig. in the Appendix). While all networks undergo this rapid increase, it is most pronounced for Euclidean networks with q' R 10-1 The increase Amax (G) correlates with the inflection point in the training loss curve Interestingly, the manifold constrained networks optimize efficiently despite differences in Amaz G) (Khller et al. 2019).
that can be seen in the inset of Fig. [
Gradient smoothness A
1041
1033
10ab
1030
Obliqee
x- 1/64
10a.
Cbliqu
10
1023
Fuclidwwn
21-1100 1/664
Euclidean
4 10
CIFAR-10
Stiafel
8" 1/64
10F4
SVHN
Stiwle
10
10:
2015
25
30
75
100
195
150
175
200
100
Epochs
Figure 3: For manifold constrained networks, gradient smoothness is not predictive of optimization rate. Euclidean networks with a low initial Amax (G) rapidly become less smooth, whereas Euclidean networks with a larger Amaz (G) remain relatively smoother. Notably, the Euclidean network with q" 1/64 has almost order of magnitude smaller Amax (G) than the Stiefel and Oblique networks, but reduces training loss at slower rate.
4 DISCUSSION
Critical orthogonal initializations have proven tremendously successful in rapidly training very deep neural networks (Pennington et al. 2017; Chen et al., 2018 Pennington et al.) 2018 Xiao al. 2018). Despite their elegant derivation drawing on methods from free probability and mean field theory, they did not offer clear optimization perspective on the mechanisms driving their success. With this work we complement the understanding of critical orthogonal initializations by showing that the maximum eigenvalue of the Fisher information matrix, and consequentially the local gradient smoothness is proportional to the maximum singular value of the input-output Jacobian. This gives an information geometric account of why the step size and training speed depend on g"
via its effect on E 8 h² We observed in numerical experiments that the paradoxical results reported in (Pennington et al.) 2017) whereby training speed and generalization attains a maximum for 9 10 10. can potentially be explained by rapid increase of the maximum eigenvalue of the FIM during training for the networks initialized with Jacobians closer to being isometric (i.c., smaller q²). This increase effectively limits the learning rate during the early phase of optimization
8

---

Published as a conference paper at ICLR 2020
and highlights the need to analyze the trajectories of training rather than just initializations. Notably these effects persist under extensive search for the optimal learning rate, indicating the increased difficulty in training highly isometric neural networks reflects qualitative change of the optimization problem. We conjecture that this phenomenon relates to the stability of descent direction in response small parameter perturbations. An analysis of the cubic term (Riemmanian connection) would give insight into the initial change in curvature at initialization, however that currently seems infeasible both numerically and analytically, In lieu of that, let us consider the stability of neural networks linearized about their initial parameters. A number of works on overparametrized networks have derived stability conditions for the linearization both in infinitesmal regimes and also over potentially infinitely many gradient updates (Chizat et al. 2019; Lee et al. 2019). The crucial property governing this stability is ensuring that the relative change in parameters small proportion to the norm of the Gram matrix of the gradients of the networks. The latter curiously has spectrum that coincides with that of the Fisher information matrix under mean-squared loss (sce AA.7). Other arguments about function of both the largest and smallest eigenvalues of this Gram matrix (Lee et al. 2019). It is therefore tempting to understand the behavior of the lowest eigenvalue of this curvature matrix. Lower bounds on the smallest eigenvalue are typically much harder to obtain, however Karakida et al. (2019) showed that the smallest eigenvalue of the Fisher scales roughly reciprocally with the maximal one. This
the stability of the lincarization have proposed bounds that depend non non-trivially
[Go) may be undesirable, and a degree of anisotropy is necessary for the Fisher information matrix at initialization to be predictive of training performance. We observe that orthogonal linear networks experience similar rapid increase in the maximum eigenvalue of their FIM, while having an FIM condition number that is (()), up to applying similarity transform to the input data (see Appendix|A.3
might imply that a low condition number Amox(Go) (Go)
Finally, we compared manifold constrained networks with the Euclidean network, each evaluated with two initial values of q" From these experiments we draw the conclusion that manifold constrained networks are less sensitive to the initial strong smoothness, unlike their Euclidean counterparts. Furthermore, we observe that the rate at which Stiefel and Oblique networks decrease training loss is not dependent on their gradient smoothness, result which is consistent with the recent analysis of (Kohler et al. 2019).
REFERENCES
P.-A. Absil, R. Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, N.J.; Woodstock, December 2007. ISBN 978-0-691-13298 3.
Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher Information and Natural Gradi- ent Learning in Random Deep Networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 694-702, April 2019.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can We Gain More from Orthogonality Regularizations in Training Deep Networks? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 4261-4271. Curran Associates, Inc., 2018
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton Optimisation for Deep Learning. In International Conference on Machine Learning, pp. 557-565, July 2017.
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2):223-311, 2018. doi: 10.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, UK; New York, I edition edition, March 2004. ISBN 978-0-521-83378-3
Minmin Chen, Jeffrey Pennington, and Samuel S. Schoenholz. Dynamical Isometry and Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, ICML 2018 Stockholmsmässaan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 872-881. PMLR, 2018

---

Published as a conference paper at ICLR 2020
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ditextquotesingle Alché- Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 2937-2947. Curran Associates, Inc., 2019.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5225 5235- 2017.
Alan Edelman, Tomás Arias, and Steven T. Smith. The Geometry of Algorithms with Orthogonality Constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303-33,, April 1999. ISSN 0895-4798. doi: 10.1 10.1137/5099547885599999
Mehrtash Harandi and Basura Fernando. Generalized BackPropagation, Érude De Cas: Orthogonality. CoRR, abs/1611 .0595797 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109/CVPR.2206..99
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org, 2015.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1032 1041, April 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7, 2015, Conference Track Proceedings, 2015.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 806-815, April 2019.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approxi- mation for natural gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alchh-BBBs Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-1 December 2019, Vancouver, BC, Canada, pp. 4158-4169. 2019.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohk-Dicksteen. Deep Neural Networks as Gaussian Processes. In International Conference on Learning Representations, February 2018
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradient Descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Allché-Bue, Emily B. Fox, and Roman Garnett (eds.). Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 8570-8581 2019.
Wuchen Li and Guido Montúfar. Natural gradient via optimal transport. Information Geometry, 1(2): 181 184, December 2018. ISSN 2511 2-1492, doi: 10.10077/41884-018-00111
James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-facteen Approxi- mate Curvature. In International Conference on Machine Learning, pp. 2408-244-4 -8017,247, June 2015.
10

---

Published as a conference paper at ICLR 2020
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian Process Behaviour in Wide Deep Neural Networks. In International Conference on Learning Representations, February 2018.
Radford M. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer-
Verlag, New York, 1996. ISBN 978-0-387-947224-. doi: 00.1077/9781-4461207744
Jeffrey Pennington and Yasaman Bahri. Geometry of Neural Network Loss Surfaces via Random Matrix Theory. In International Conference on Machine Learning, pp. 209-22298800 July 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 4785 4795. Curran Associates, Inc., 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 1924-1932, March 2018.
Tim Salimans and Durk Kingma. Weight Normalization: Simple Reparameterization to Acceler- ate Training of Deep Neural Networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 901-909, Curran Associates, Inc., 2016.
Andrew M. Saxe, James L.. McClelland, and Surya Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Christiane Tretter. Spectral Theory of Block Operator Matrices and Applications. Imperial College Pr, London Singapore: Hackensack, NJ, October 2008. ISBN 978-1-86094-768-1.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference Machine Learning. pp. 3570-3578, July 2017.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-Capacity Unitary Recurrent Neural Networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4880-4888. Curran Associates, Inc., 2016.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, and Jeffrey Pennington. Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 5389 5398. PMLR, 2018.

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 BLOCK GERSHGORIN THEOREM
In Section we considered block diagonal approximation to the Fisher information matrix and derived an upper bound on the spectral norm for all the blocks. Using the properties of the off- aff-diagonal blocks, we can get more accurate estimate of the maximal eigenvalue of the Fisher information might be. First, let consider arbitrarily partitioned matrix A E RNXN, with spectrum A(A) The partitioning is done with respect to the set
π = {P}}
(18)
with the elements of the satisfying 0 P1 P2 ... < PL N. Then each block of the matrix Aig is potentially rectangular matrix in 1 We assume that Ai, is self-adjoint for all i.
Let us define disk
C(c, r) {}: |||| A rr.
-
(19)
The theorem as presented in (Tretter 2008 shows that the eigenvalues of X(A) are contained in union of Gershgorin disks defined as follows
L X(A) C CUUK
( (R(AA)
8max (Aia) }
U k=1
C
Σ
(20)
j=1,jæi
where the inner union is over a set disks for each eigenvalue of the block diagonal A, while the outer union is the L blocks in A. The radius of the disk is constant for every eigenvalue in the 7th diagonal block Ai and is given by the sum of singular values of the off diagonal blocks. Therefore, the largest eigenvalue of A lies in
L Uc 1=1
Amaax (A) c
Xmaax (Ai),
Σ
S Smax Ai,))
(21)
jim Light
Now suppose A positive semidefinite. This suggests strategy for upper-bounding the maximal eigenvalue of A- picking the disk containing the largest (non-negative) value will upper-bound the maximal eigenvalue
Amax (A) S max Xmmax(A +
Σ
Smax (Ai.j)
(22)
A.2 DERIVATION OF THE EXPECTED SINGULAR VALUES
We present derivation of the bounds presented in Lemma weight-to-weight blocks is
The singular value bound for the
Tmax (eeeeeW) = [amax (ø(h)1 x XT]] & E Tmax Jhb H
he hd
(23)
N [b()]]] E [x0] [|,
E max (Jh"
H hd
(24)
where ((h)1 1 is vector of ones times ((h)
< N E [oh)]]
max (JhP ] IE [amax (H,)] E mmax (hh²
(25)
The singular value bound for the toight-to-bias blocks is
Tnax (Gecc ).66.)
< E max
Tx
I)]
E mmax (Jh² h T ".
(26)
= | xx" 112 EE
maxx Jhp H hh
(27)
< ||
E xx" |||
Fmax
(Jhe 3) E Jo (H₂)) Fmax (Jhi
(28)
12

---

Published as a conference paper at ICLR 2020
max (G (w)))
E max (x2-11
I)]
(29)
N8 [o(()]]
max (Jhi H H )
(30)
Na E (b(h)
E
max
Jh
R mmax (H2)) max Jhh
(31)
The singular value bound for the weight-to-bias blocks is
Omax (G&z (0))))) =
mmax
(Jh"T
(32)
[J [mmax (H₂)) E max (JH:
(33)
max
A.3 MONTECARLO ESTIMATE OF MAXIMUM EIGEN OF THE HESSIAN OF THE OUTPUT LAYER FOR 10 WAY SOFTMAX CLASSIFICATION
0.4
Distribution of Aonaa 11
0.3
0.2
0.1
1.62E-03 5.256-03 1.70E-02 5 5.526-02
1.79E-01
5 80E-01 1.88E+00 6.09E-00 1.97++01 6.40E+01
Variance of preactivation q"
Figure 4: Distribution of Amax (Hg) as a function of q²: In general, increasing the variance of the distribution of h does not result in monotonic increase the spectral radius of the Hessian of the GLM layer. We plot the distribution of the maximum eigenvalues as function of the variance of the softmax layer obtained from factorizing 10,000 random matrices.
A.4 PROOF OF THEOREM
Using the results from Section A.2 we observe that all the terms off diagonal for the diagonal block G Veec(W"), are of the form
E [o(h)]
E
maxx [Jh
E [Tmax
(H2)) E Tmax Jhh
(34)
and
E
[x"]
E maax (Jh""
] mmax (H₂)) maxx (Jh:")
(35)
while the diagonal term is
Amax
E [x x
Fmax
Jhh"
max (Hg) E mmax (Jhi)
(36)
then by Gershgorin theorem the eigenvalues for the G veel wee bounded by
Amax v.eecW))))) <
E
Tmax
(Jh""
E [mma (H
X
Amax (E 1x
Tmax
Jh h
Σ (1 [[((h)]]
|| [xx]
E
max
)
8
(37)
13

---

Published as a conference paper at ICLR 2020
Similarly, for the G ba block, we observe that
Amax (G) v
E max
(Jh:
E [maax (H₂)l) x
(38)
Σ
(1+ VL+VN
[x]
Jhh max (
x
3
A.5 ADDITIONAL FIGURES
CIFAR-10 performance
60001
S3mple crozs entrogy
10003
00003
081
Lididean of
Fudiiian of 111011
60065
Dilape y-
Oblipe
00000
60005
00003
25
00
301
120
194
175
25
53
2.01
153
175
20
Epochs
Epocts
Figure 5: Training performance for networks with q² and q" 8. The behavior of the training loss as well as the validation accuracy is consistent with the observations that for large range of parameters q" the manifold constrained networks are insensitive to initialization and gradient smoothness.
A.6 MANIFOLD OPTIMIZATION
The potentially non-convex constraint set constitutes a Riemannian manifold, when it is locally isomorphic to R", differentiable and endowed with suitable (Riemannian) metric, which allows us to measure distances in the tangent space and consequentially also define distances on the manifold. There considerable freedom in choosing a Riemannian metric; here we consider the metric inherited from the Euclidean embedding space which defined as (W, W' & Tr (W.. . To optimize cost function with respect to parameters lying in non-Eut -on-Euclidean manifold we must define descent direction. This done by defining a manifold equivalent of the directional derivative. An intuitive approach replaces the movement along a vector t with movement along geodesic curve 7(), which lies in the manifold and connects two points W. W E M such that (0)) W, 7(1) W'. The derivative of an arbitrary smooth function f (t(()) with respect to t then defines tangent vector for each t.
Tangent vector
Ew tangent vector at W if Ew satisfies 7(0) W and Ew A df()(())
(0)
(39)
dt 1=
The of all tangents to M W is referred to the tangent space to M W and is denoted by M. The geodesic importantly is then specified by constant velocity curve (²) = 0 with initial velocity fw. To perform gradient step, we must then move along E while respecting the manifold constraint. This is achieved by applying the exponential map defined Expw (&w) 2 (1), which moves W to another point W" along the geodesic. While certain manifolds, such as the Oblique manifold, have efficient closed-form exponential maps, for general Riemannian manifolds, the computation of the exponential map involves numerical solution to a non-linear ordinary differential equation (Absil et al., 2007). An efficient alternative to numerical integration is given by an orthogonal projection onto the manifold. This projection is formally referred to retraction Rtw T + M.
Finally, gradient methods using Polyak (heavy ball) momentum (e.g. ADAM (Kingma & Ba 2015) require the iterative updating of terms which naturally lie in the tangent space. The parallel translation T<(E) TM ④ TM + TM generalizes vector composition from Euclidean to Eunleucli manifolds, by moving the tangent & along the geodesic with initial velocity Ç E T and endpoint W'. and then projecting the resulting vector onto the tangent space T M. As with the exponential map.
14

---

Published as a conference paper at ICLR 2020
parallel transport T may require the solution of non-linear ordinary differential equation. To alleviate the computational burden, we consider vector transport as an effective, projection like solution to the parallel translation problem. We overload the notation and also denote it T. highlighting the similar role that the two mappings share. Technically, the geodesics and consequentially the exponential map, retraction well transport 7 depend the choice of the Riemannian metric. Putting the equations together the updating scheme for Riemannian stochastic gradient descent on the manifold is
W41 = Ilw. (- grad
(40)
where is either the exponential map Exp the retraction Rt and grad is the gradient of the function f(W) lying in the tangent space T.
A.6.1 OPTIMIZING OVER THE OBLIQUE MANIFOLD
(Cho & Lee 2017) proposed an updating scheme for optimizing neural networks where the weights of each layer are constrained lie in the oblique manifold Ob(p; n). Using the fact that the manifold itself is product of p unit-norm spherical manifolds, they derived efficient, closed- form Riemannian gradient descent updating scheme. In particular the optimization simplifies to the optimization over Ob(1, n) for each column (1.......pp) of W.
Oblique gradient The gradient grad/ f of the cost function f with respect the weights lying in Ob(1, n) is given as a projection of the Euclidean gradient Grad the tangent at w
grad = Gradf - (w Gradf)w
(41)
Oblique exponential map initial velocity Ew
The exponential map Expw moving w to w' along geodesic with
Expw = Ew cos(||w|l)
+ w |w|
sin ([[]]
(42)
Oblique parallel translation geodesic with initial velocity Sw
The parallel translation T moves the tangent vector &w along the
Sw [[ ((1 cos([IGwl))) + sin((ISw/)) ())
Sw
Tew (Ew) = Ew
Tsw
[[swl
Orthogonal penalty
which enforces the weight matrices to be near orthogonal.
p(A, w)
w w I
I|
(43)
A.6.2 OPTIMIZING OVER THE STIEFEL MANIFOLD
Optimization over Stiefel manifolds in the context of neural networks has been studied by (Harandi &
Fernando 2016 Wisdom et al. 2016 Vorontsov et al. (017). Unlike (Wisdom al. 2016)] Vorontsov et al. 2017 7) we propose the parametrization using the Fuclidean metric, which results in a different definition of vector transport.
Stiefel gradient The gradient grad of the cost function with respect to the weights lying in
St(p, n) is given as a projection of the Euclidean gradient Grad fonto the tangent at W (Edelman et al. 1999 Absil et al. 2007)
Gradf grad = (I WW WWT)Gradf + W WWGGradf Gradf WW)
Stiefel retraction
The retraction Rtw (Ew) for the Stiefel manifold given by the Q factor the Absil et al. 2007).
QR decomposition
Riw(Ew) qf(W Ew)
(44)
15

---

Published as a conference paper at ICLR 2020
Stiefel vector transport The vector transport T moves the tangent vector Ew along the geodesic with initial velocity Sw for W E St(p, n) endowed with the Euclidean metric.
Tew (Ew) (I- Ew + Y (Y' Ew Ew 'Y)
(45)
where Y A Riw (Sw). It is easy to see that the transport T consists of a retraction of tangent Św followed by the orthogonal projection of 7/w at Rtw (Cw).
A.6.3 OPTIMIZING OVER NON-COMPACT MANIFOLDS
The critical weight initialization yielding singular spectrum of the Jacobian tightly concentrating on implies that a substantial fraction of the pre-activations lie in expectation in the linear regime of the squashing nonlinearity and as a consequence the network acts quasi-linearly, To relax this constraint during training we allow the scales of the manifold constrained weights to vary. We chose to represent the weights a product of a scaling diagonal matrix and matrix belonging to the manifold. Then the optimization of each layer consists in the optimization of the two variables in the product. In this work we only consider isotropic scalings, but the method generalizes easily the use of any invertible square matrix.
is interesting to contrast our approach with projected gradient descent with a spectral prox function, or penalizing the Jacobian norm through back propagation. PGD with projections onto compact spectral balls requires the user to pre-specify the desired spectral radius. On the other hand, both approach well penalizing the Jacobian norm be thought of penalizing decreases in function smoothness in an adaptive way. Finally, our approach is naturally amenable to incorporating data-dependent penalty this would allow to smoothly vary the spectral radia of the weight matrices.
A.7 FIM AND NTK HAVE THE SAME SPECTRUM
The empirical Neural Tangent Kernel (NTK) is defined as
J
(46)
which gives a | by | kernel matrix. By comparison the empirical Fisher information matrix with a Gaussian likelihood is
Σ EE|D
J TT h"
(47)
To that the spectra of these two coincide consider the third order tensor underlying both Jii for i E 1...|D1 [D], additionally consider and unfolding A with dimensions [0] by No|D]; i.e. we construct matrix with dimension of number of parameters by number of outputs times number of data points. Then
= A Ô AAT
(48)
(49)
(50)
and their spectra trivially coincide.
16

---

Published as a conference paper at ICLR 2020
A.8
LINEAR NEURAL NETWORKS WITH ORTHOGONAL INITIALIZATIONS
Performance a linear orthogonal network
100755
cros2 crozsrerting entiogy
Test acpuracy [S]]
lent ampothness
Unae 1294
2.00150
Per sample
2.00000
1.0005
Fpochs
101-1
130
155
170
30
Gpochs
Epocis
Epochs
Figure 6: Linear neural networks with orthogonal initializations increase rapidly in their maxi- mum eigenvalue of the empirical Fisher information, similar to their non-linear, nearly isometric counterparts.
17


Reviews:
[{"review_id": "ryxt7P3nYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the connection between the spectrum of the layer-to-layer or input-output Jacobian matrices and the spectrum of the Fisher information matrix / Neural Tangent Kernel. By bounding the maximum eigenvalue of the Fisher in terms of the maximum squared singular value of the input-output Jacobian, this paper provides a partial explanation for the successful initialization procedures obeying \"dynamical isometry\". By additionally investigating optimization on the orthogonal weight manifold, this paper sheds light on the important of maintaining spectral uniformity throughout training.  These two analyses help fill in important gaps in the understanding of initialization, dynamical isometry, and the training of deep neural networks. For these reasons I recommend this paper for acceptance.\n\nThere are two aspects of the paper that could nevertheless use some clarification and improvement. First, unless I missed something, this paper does not provide any bounds on the condition number or the minimum eigenvalue of the Fisher or the NTK. It seems like the main arguments only depend on the maximum eigenvalues. Generally, I think the insights into the maximum eigenvalues are useful and important on their own, but perhaps some additional discussion up front clarifying which results were derived theoretically and which were observed empirically could be useful.\n\nSecond, it should be noted that the the networks trained in the experiments are likely in a regime that is well outside the NTK regime, in two important ways: the dataset is large compared to the width and the optimal learning rates may be large as well.\n\nOverall, I think this is a good paper that adds important insights into the study of initialization, local geometry, and their effects on training speed.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rylq7Ph2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates a connection between the Fisher information matrix (FIM) and the spectral radius of the input-output Jacobian in neural networks. This results derive the eigenvalues' bound to theoretically study the convergence of several networks.  Here the upper bound further improves the upper bound of FIM derived in (Karakida et al., 2018). \nThis is a very interesting and useful direction of applying information matrices to study the initialization of deep networks. \n\nI suggest the weak acceptance of the paper. After addressing the following remarks, I can adjust my reviews. \n\n1. There are some typos, such as see[?] in page 7, the main theorem on page 6 should be written mathematically with a remark. \n\n2. What is the major technical difference between this paper and Karakida et al., 2018? \n\n3. Here the model is given by conditional probability is defined by a neural network. \nThe author may also be interested in implicit models, such as normalization flows and generative networks. \nIn this case, the Wasserstein information matrix, (Hessian of Wasserstein-2 loss), may be very suitable to be studied. \nSee:\n\n\"A. Lin, W.Li, S.Osher, G. Montufar, Wasserstein proximal of GANs, 2018.\"\n\n\"W.Li, G. Montufar, Natural gradient via optimal transport, 2018.\"\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SygON0qAYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the training behavior of wide networks and argues orthogonal initialization helps the training. They suggest projections to the manifold of orthogonal weights during training and provide analysis. Their main result seems to be a bound on the eigen-values of the Fisher information matrix for wide networks (Theorem on pg 6). In their experiments they train Stiefel and Oblique networks as examples of manifold constrained networks and claim they converge faster than unconstrained networks.\n\nCons:\n- Page 6, the main theorem of the paper, Theorem (bound on the fisher) doesn’t have a proof.\n- Fig 1. What’s the overhead wall-clock time of manifold constraint?, on cifar10 the two manifold don’t have the same rate. Euclidean on cifar10 has higher test accuracy. Test accuracy after 200 epochs is below 90 and below 60.\n- There are claims in the paper for providing explanations by making connections to Neural Tangent Kernel but it is mentioned only in the discussion section and they reiterate previously known results.\n- Fig 3: is the training plot for cifar10 in this figure the one in figure1? Where is the training curves for svhn? Where should we see the rate of reduction in training loss for these methods?\n- Section B.4:  To show that FIM and NTK have the same spectrum you need \\nabla^2 L to be identity which is only true for L2 loss function. This does not apply to other loss functions such as cross-entropy.\n\nAfter rebuttal:\nI raise my rating to weak accept. The writing has improved a lot and most of my concerns are addressed. It would be nice if authors could incorporate the timing plots in the appendix.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I've gone over this paper carefully and think it's above the bar for ICLR.

The paper proves a relationship between the eigenvalues of the Fisher information matrix and the singular values of the network Jacobian. The main step is bounding the eigenvalues of the full Fisher matrix in terms of the eigenvalues and singular values of individual blocks using Gersgorin disks. The analysis seems correct and (to the best of my knowledge) novel, and relationships between the Jacobian and FIM are interesting insofar as they give different ways of looking at linearized approximations. The Gersgorin disk analysis seems like it may give loose bounds, but the analysis still matches up well with the experiments.

The paper is not quite as strong when it comes to relating the anslysis to optimization. The maximum eigenvalue of the FIM by itself doesn't tell us much about the difficulty of optimization. E.g., if the top FIM eigenvalue is increased, but the distance the weights need to travel is proportionately decreased (as seems plausible when the Jacobian scale is changed), then one could make just as fast progress with a smaller learning rate. So in this light, it's not too surprising that the analysis fails to capture the optimization dynamics once the learning rates are tuned. But despite this limitation, the contribution still seems worthwhile.

The writing can still be improved.

The claim about stability of the linearization explaining the training dynamics appears fairly speculative, and not closely related to the analysis and experiments. I recommend removing it, or at least removing it from the abstract.

Decision: Accept (Poster)