Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper analyses the training behavior of wide networks and argues orthogonal initialization helps the training. They suggest projections to the manifold of orthogonal weights during training and provide analysis. Their main result seems to be a bound on the eigen-values of the Fisher information matrix for wide networks (Theorem on pg 6). In their experiments they train Stiefel and Oblique networks as examples of manifold constrained networks and claim they converge faster than unconstrained networks.

Cons:
- Page 6, the main theorem of the paper, Theorem (bound on the fisher) doesn’t have a proof.
- Fig 1. What’s the overhead wall-clock time of manifold constraint?, on cifar10 the two manifold don’t have the same rate. Euclidean on cifar10 has higher test accuracy. Test accuracy after 200 epochs is below 90 and below 60.
- There are claims in the paper for providing explanations by making connections to Neural Tangent Kernel but it is mentioned only in the discussion section and they reiterate previously known results.
- Fig 3: is the training plot for cifar10 in this figure the one in figure1? Where is the training curves for svhn? Where should we see the rate of reduction in training loss for these methods?
- Section B.4:  To show that FIM and NTK have the same spectrum you need \nabla^2 L to be identity which is only true for L2 loss function. This does not apply to other loss functions such as cross-entropy.

After rebuttal:
I raise my rating to weak accept. The writing has improved a lot and most of my concerns are addressed. It would be nice if authors could incorporate the timing plots in the appendix.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper formulates a connection between the Fisher information matrix (FIM) and the spectral radius of the input-output Jacobian in neural networks. This results derive the eigenvalues' bound to theoretically study the convergence of several networks.  Here the upper bound further improves the upper bound of FIM derived in (Karakida et al., 2018). 
This is a very interesting and useful direction of applying information matrices to study the initialization of deep networks. 

I suggest the weak acceptance of the paper. After addressing the following remarks, I can adjust my reviews. 

1. There are some typos, such as see[?] in page 7, the main theorem on page 6 should be written mathematically with a remark. 

2. What is the major technical difference between this paper and Karakida et al., 2018? 

3. Here the model is given by conditional probability is defined by a neural network. 
The author may also be interested in implicit models, such as normalization flows and generative networks. 
In this case, the Wasserstein information matrix, (Hessian of Wasserstein-2 loss), may be very suitable to be studied. 
See:

"A. Lin, W.Li, S.Osher, G. Montufar, Wasserstein proximal of GANs, 2018."

"W.Li, G. Montufar, Natural gradient via optimal transport, 2018."


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper analyzes the connection between the spectrum of the layer-to-layer or input-output Jacobian matrices and the spectrum of the Fisher information matrix / Neural Tangent Kernel. By bounding the maximum eigenvalue of the Fisher in terms of the maximum squared singular value of the input-output Jacobian, this paper provides a partial explanation for the successful initialization procedures obeying "dynamical isometry". By additionally investigating optimization on the orthogonal weight manifold, this paper sheds light on the important of maintaining spectral uniformity throughout training.  These two analyses help fill in important gaps in the understanding of initialization, dynamical isometry, and the training of deep neural networks. For these reasons I recommend this paper for acceptance.

There are two aspects of the paper that could nevertheless use some clarification and improvement. First, unless I missed something, this paper does not provide any bounds on the condition number or the minimum eigenvalue of the Fisher or the NTK. It seems like the main arguments only depend on the maximum eigenvalues. Generally, I think the insights into the maximum eigenvalues are useful and important on their own, but perhaps some additional discussion up front clarifying which results were derived theoretically and which were observed empirically could be useful.

Second, it should be noted that the the networks trained in the experiments are likely in a regime that is well outside the NTK regime, in two important ways: the dataset is large compared to the width and the optimal learning rates may be large as well.

Overall, I think this is a good paper that adds important insights into the study of initialization, local geometry, and their effects on training speed.
review_title: Official Blind Review #3

Meta-review: I've gone over this paper carefully and think it's above the bar for ICLR.

The paper proves a relationship between the eigenvalues of the Fisher information matrix and the singular values of the network Jacobian. The main step is bounding the eigenvalues of the full Fisher matrix in terms of the eigenvalues and singular values of individual blocks using Gersgorin disks. The analysis seems correct and (to the best of my knowledge) novel, and relationships between the Jacobian and FIM are interesting insofar as they give different ways of looking at linearized approximations. The Gersgorin disk analysis seems like it may give loose bounds, but the analysis still matches up well with the experiments.

The paper is not quite as strong when it comes to relating the anslysis to optimization. The maximum eigenvalue of the FIM by itself doesn't tell us much about the difficulty of optimization. E.g., if the top FIM eigenvalue is increased, but the distance the weights need to travel is proportionately decreased (as seems plausible when the Jacobian scale is changed), then one could make just as fast progress with a smaller learning rate. So in this light, it's not too surprising that the analysis fails to capture the optimization dynamics once the learning rates are tuned. But despite this limitation, the contribution still seems worthwhile.

The writing can still be improved.

The claim about stability of the linearization explaining the training dynamics appears fairly speculative, and not closely related to the analysis and experiments. I recommend removing it, or at least removing it from the abstract.

Decision: Accept (Poster)