Scaling Laws for the Principled Design, Initialization, and Preconditioning of ReLU Networks
Authors: Aaron Defazio
Leon Bottou
Venue:  (2020)
Keywords: initialization
mlp
relu

Abstract: Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training. We show how our measure of conditioning of a block relates to another natural measure of conditioning, the ratio of weight gradients to the weights. We prove that for a ReLU-based deep multilayer perceptron, a simple initialization scheme using the geometric mean of the fan-in and fan-out satisfies our scaling rule. For more sophisticated architectures, we show how our scaling principle can be used to guide design choices to produce well-conditioned neural networks, reducing guess-work.

Under review as a conference paper at ICLR 2020
SCALING LAWS FOR THE PRINCIPLED DESIGN, INI- TIALIZATION, AND PRECONDITIONING OF RELU NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we describe set of rules for the design and initialization of well- conditioned neural networks, guided by the goal naturally balancing the diagonal blocks of the Hessian at the start of training. We show how measure of conditioning of a block relates to another natural measure of conditioning, the ratio of weight gradients to the weights. We prove that for ReI.U-based deep multilayer perceptron, a simple initialization scheme using the geometric mean of the fan-in and fan-out satisfies show how our scaling principle can be used to guide design choices to produce well-conditioned neural networks, reducing guess-work.
scaling rule. For more sophisticated architectures,
INTRODUCTION
The design of neural networks is often considered a black-art, driven by trial and error rather than foundational principles. This is exemplified by the success of recent architecture aadom-seeach techniques (Zoph and Le 2016 Li and Talwalkar (019), which take the extreme of applying no human guidance all. Although and generalization in neural networks, this does not mean that we should proceed blindly.
field we far from fully understanding the nature of learning
In this work we define a scaling quantity "u for each layer that approximates the average squared singular value of the corresponding diagonal block of the Hessian for layer 1. This quantity is easy to compute from the (non-central) second moments of the prooad-propaggtedd values and the (non-central) second moments of the backward- pword-proppggtedd gradients. We argue that networks that have constant "t are better conditioned than those that do not, and we analyze how common layer types affect this quantity. We call networks that obey this rule preconditioned neural networks, in analogy to preconditioning of linear systems.
As an example of some of the possible applications of our theory, we:
Propose a principled weight initialization scheme that can often provide an improvement over existing schemes;
Show which common layer types automatically result in well-conditioned networks;
Show how to improve the conditioning of common structures such as bottlenecked residual blocks by the addition of fixed scaling constants to the network (Detailed in Appendix
NOTATION
We will use the multilayer perceptron (i.e. a classical feed-forward deep neural network) as a rumning example as it is the simplest non-trivial deep neural network structure. We use ReLU activation functions, and use the following notation for layer of L (following He et al. 2015):
y1 W bi.
F1+1 = Ru))
where Wj is nit nft matrix of weights, by is the bias vector, y the preactivation vector and xn the input activation vector for the layer. The quantities ne! and nj" called the fan-out and

---

Under review as a conference paper at ICLR 2020
(a) Fan-in
(b) Fan-out
(c) Arithmetic mean
Preconditioned
Figure I: Average singular value heat maps for the strided LeNet model, where each square represents a block the Hessian. The preconditioned network maintains an approximately constant block- diagonal weight. The scale goes from Yellow (larger) through green to blue (smaller).
fan-in of the layer respectively. We also denote the gradient of quantity with respect to the loss (i.e. the pack-propagated gradient) with the prefix A. We initially focus on the least-squares loss. Additionally, we assume that each bias vector is initialized with zeros unless otherwise stated.
3 CONDITIONING BY BALANCING THE HESSIAN
Our proposed approach focuses on the singular values of the diagonal blocks of the Hessian. In the case multilayer perceptron network (MLP) network, each diagonal block corresponds to the weights from single weight matrix Wi or bias vector by. This block structure is used by existing approaches such as K-FAC and variants (Martens and Grosse 2015 Grosse and Martens 2016 Ba al.) 2017, George et al. 2018), which correct the gradient step using estimates of second- order information. In contrast, our approach modifies the network to improve the Hessian without modifying the step.
Estimates of the magnitude of the singular values a; (Gi) of the diagonal blocks GI.... G of the Hessian G provide information about the singular values of the full matrix.
Proposition Let Gt be the lth diagonal block of real symmetric matrix G Then for all i 1.
Uiin (G) S (Gi))
Fmax (G) 2 (Gi))
We can use this simple bound to provide some insight into the conditioning of the full matrix: Corollary Let {sI....>} be the union of the sets of singular values of the diago- nal blocks G1, Gt of a real symmetric matrix G x Then the condition number x(G) max (G) (G G) is bounded as:
x(G) 2 max 51/ min 8t-
S.E3 6.ES
In particular, Hessian matrix with very large difference between the singular values of each block must be ill-conditioned. This provides strong motivation for balancing the magnitude of the singular values of each diagonal block, the goal of this work. Although idcally, would like state the converse, that matrix with balanced blocks is well conditioned, we can not make such statement without strong assumptions on the off-diagonal behavior of the matrix.
We use the average squared singular value of each block proxy for the full spectrum, as it is particularly easy to estimate expectation. Although the minimum and maximum for each block would seem like more natural quantity to work with, we found that any such bounds tend to be too pessimistic to reflect the behavior of the actual singular values.
When using the ReLU activation function, as we consider in this work, a neural network is no longer smooth function of inputs, and the Hessian becomes ill-defined some points in the parameter space. Fortunately, the spectrum is still well-defined at any twice-differentiable point, and this gives local measure of the curvature. ReLU networks are typically twice-differentiabl almost everywhere. We assume this throughout the remainder of this work.
2

---

Under review as a conference paper at ICLR 2020
100
102
10-1
conv1
conv2
fcl
fo2
Figure 2: Distributions of the ratio of theoretical scaling to actual for a strided LeNet network
3.1
GR SCALING: A MEASURE OF HESSIAN AVERAGE CONDITIONING
Our analysis will proceed with batch-size and a network with outputs. We consider the network initialization, where weights centered, symmetric and i..d random variables and biases are set to zero.
ReLU networks have particularly simple structure for the Hessian with respect any set of activations, as the network's 's output is a piecewise-linear function fed into a final layer consisting of loss. This structure results in greatly simplified expressions for diagonal blocks of the Hessian with respect to the weights.
We will consider the output of the network as a composition the remainder of the network h. We write this as function of the weights, i.c. f(W) h(g(W))). The dependence on the imput to the network implicit in this notation, and the network below layer I does not need to be considered.
functions, the current layer g. and
Let R₁ 2 h(y) be the Hessian of h, the remainder of the network after application of layer I (recall yt Wyrt). Let Ji be the Jacobian of with respect to Wi. The Jacobian has shape JI mi x (ng n("). Given these quantities, the diagonal block of the Hessian corresponding to Wi equal to:
21
G = Jf R.J.-
The Ith diagonal block of the (Generalized) Gauss-Newton matrix G (Martens) 2014). We discuss this decomposition further in Appendix A.1 We use the notation E[X*] "X for any matrix vector X denote the expectation of the element wise
wise non-cem non-central second moment.
Proposition 3. (The GR sculing) Under the assumptions outlined Section 3.2 the averuge squared singular value of Gt is equal to the following quantity, which call the GR scaling for MLP layers:
ElAy?
(GR scaling) n = n xP
We define "balanced" or "preconditioned" network as one in which n is equal for all (full derivation in Appendix
Balancing this theoretically derived GR scaling quantity network will produce an initial opti- mization problem for which the blocks of the Hessian are expected to be approximately balanced with respect to their average squared singular value.
Due to the large number of approximations needed for this derivation, which discuss further in the next section, we don't t claim that this theoretical approximation is accurate, or that the blocks will be closely matched practice. Rather, make the lesser claim that network with very disproportionate values of between layers is likely to have convergence difficulties during the early stages of optimization due to Cor.
To check the quality of our approximation, we computed the ratio of the convolutional version of the GR scaling equation (Fquation 1 to the actual (G product for a strided (rather than max-pooled, see Table LeNet model, where we use random input data and a random loss (i.e. for outputs y y Ry for an i.i.d normal matrix R), with batch-size 1024, and 32 x 32 input images. The results are shown in Figure 2 for 100 sampled setups; there generally good agreement with the theoretical expectation.
3

---

Under review as a conference paper at ICLR 2020
3.2 ASSUMPTIONS
The following strong assumptions are used in the derivation of the GR scaling:
(A1) The input and target values are drawn element-wise i.i.d from centered symmetric distribution with known variance.
(A2) The Hessian of the remainder of the network above each block, with respect the output, has Frobenius norm much larger than 1. More concretely, we assume that all but the highest order terms that are polynomial in this norm may be dropped.
(A3) All activations, pre-activations and gradients are independently distributed element-wise. In practice due to the mixing effect of multiplication by random weight matrices, only the magnitudes of these quantities are correlated, and the effect is small for wide networks due to the law of large numbers. Independence assumptions of this kind are common when approximating second-order methods; the block-diagonal variant of K-FAC (Martens and Grosse 2015) makes similar assumptions for instance.
Assumption A2 is the most problematic of these assumptions, and we make no claim that it holds in practice. However, we are primarily interested the properties of blocks and their scaling with respect to each other, not their absolute scaling. Assumption A2 results in very simple expressions for the scaling of the blocks without requiring a more complicated analysis of the top of the network. Similar theory can be derived for other assumptions the output structure, such as the assumption that the target values are much smaller than the outputs the network.
PRECONDITIONING ALSO BALANCES WEIGHT-TO-GRADIR RATIOS
We provide further motivation for the utility of preconditioning by comparing it to another simple quantity of interest. Consider at network initialization, the ratio of the (element-wise non-central) second moments of each weight-matrix gradient to the weight matrix itself:
F[AW?] E
DI
This ratio approximately captures the relative change that single SGD step with unit step-size Wi will produce. We call this quantity the faight-to-gradere ratio. When E[AW?] is very small compared to F. the weights will stay close their initial values for longer than when F. F(AW? large. In contrast, if E [[W?] is very large compared to F[W?]. then learning can be expected to be unstable, as the sign of the elements of W may change rapidly between optimization steps. A network with constant in is also well-behaved under weight-decay, as the ratio of weight-decay second moments to gradient second moments will stay constant throughout the network, keeping the push-pull of gradients and decay constant across the network. Remarkably, the weight-to- gradient ratio Di turns out to be equivalent to the GR scaling for MLP networks:
weight-to-gradient
Proposition 4. (Appendix () is equal the GR scaling " for i.i.d mean-zero randomly-initialle multilayer perceptron layers under the independence assumptions of Appendix 3.2
5 CONVOLUTIONAL NETWORKS AND CONDITIONING MULTIPLIERS
The concept of GR scaling may be extended to scaled convolutional layers y arConew; (m1) + by with scaling factor on, kernel width ky, batch-size b, and output resolution pt x pt A straight-forward derivation gives expressions for the convolution weight and biases of:
E E(A ElP
ElAy = E
=
at bm
in ki k
(1)
This requires an assumption of independence the values of activations within a channel that is not true in practice, n tends to be further away from empirical estimates for convolutional layers than for non-convolutional layers, although is still a useful guide. The effect of padding is also ignored here. Sequences of convolutions remains the same. The scaling of layers involving differing kernel sizes can be corrected using the alpha parameter (Appendix H and more generally any imbalance between the conditioning of layers
well-scaled against each other along the kernel size
4

---

Under review as a conference paper at ICLR 2020
can be fixed by modifying ay while the same time changing the initialization of Wi that the forward variance remains the same as the unmodified version. This adjusts "n while leaving all other y the same.
PRECONDITIONING OF NEURAL NETWORKS VIA INITIALIZATION
For ReLU networks with classical multilayer-perceprre (i.e. non-convolational, non-residual) structure, we show in this section that initialization using i.i.d mean-zero random variables with (non-central) second moment inversely proportional to the geometric mean of the fans:
E
for fixed constant results in constant GR scaling throughout the network.
Proposition 5. Let Wo x n and W1 x m be weight matrices satisfying the geometric initial- ization criteria of Equation 2 and let bu, be pero-initialized bias parameters. Then consider the following sequence of two layers where vn and Ay are iid, mean O, uncorrelated and symmetrically distributed::
Yo = Wo + bo:
ReLUUU)), y1 Wi b1
Then vy = 111 and so 7o 71-
Proof. Note that the ReLU operation halves both the forward and backward (non-central) second moments, due to our assumptions on the distributions of xp and Ayi. So:
E E
E'vll
E[ = E[aai]-
(3)
Consider the first weight-gradient ratio, using F(AW?] = E|
B
1
E|
E A V Tum.
E
Under assumptions, hack-propagation Ar₁ results in F[Anq] pF(W]]][AuR]
E[AッA]
E[Anf]
1 1
PEIWilE/AnT 2 /mp E[Ayi),
So:
BAW?
V E
2 /npLl<dl E[AR).
(4)
3
2c
mmp
Now consider the second
ratio:
E|Aw?1 1
pm [
E WW
Under our assumptions, applying forward propagation gives Elvål = E| and so from Equation we have:
1
El= nEW
2
Vnm
E AAW?
pm
Fl
Ay
E
2c
Vnm
VapEla
x E
which matches Equation 10 V/.
Remark This relation also holds for sequences of (potentially) strided convolutions, but only if the same kernel size is used everywhere and circular padding used. The initialization should be modified to include the kernel size, changing the expression c/ (kg in
5

---

Under review as a conference paper at ICLR 2020
6.1 TRADITIONAL INITIALIZATION SCHEMES
The most common approaches are the Kaiming (He et al. 2015) and Xavier (Glorot and Bengio
2010) initializations. The Kaiming technique for ReLU networks is actually one of two approaches:
(fan-in)
[ar[Wi]
72 or (ffn-outt
Var[W]]
(5)
12(
For the feed-forward network above, assuming random activations, the forward- variance will remain constant in expectation throughout the network fan-in initialization of weights (LeCun) et al. 2012) is used, whereas the fan-out variant maintains constant variance of the back proo-prop signal. The constant factor 2 in the above expressions corrects for the variance-reducing effect the ReLU activation. Although popularized by He et al. (2015), similar scaling was in use in early neural network models that used tanh activation functions (Bottou 1988).
These two principles are clearly in conflict; unless nı" nit" either the forward variance backward variance will become non-constant, or as more commonly expressed, either explode or vanish. No prima facie reason for preferring one initialization over the other is provided. Unfortunately, there some confusion in the literature as many works reference using Kaiming initialization without specifying if the fan-in or fan-out variant is used.
The Xavier initialization (Glorot and Bengio, 2010) is the closest to our proposed approach. They balance these conflicting objectives using the arithmetic mean:
Var[W]]
(6)
n2 + mg
to "... approximately satisfy our objectives of maintaining activation variances and back-propagated gradients variance as one moves up or down the network". This approach to balancing is essentially heuristic, in contrast to the geometric mean approach that our theory directly guides to.
6.2 GEOMETRIC INITIALIZATION BALANCES BIASES
We can use the same proof technique to compute the GR scaling for the bias parameters network. Our update equations change to include the bias term: y/ War + br, with by assumed to be initialized at zero. We show Appendix D that:
E Elui
It is easy to show using the techniques of Section 6 that the biases of consecutive layers have equal GR scaling as long as geometric initialization is used. However, unlike in the case of weights, we have less flexibility in the choice of the numerator, Instead of allowing all weights to be scaled by for any positive e, we require that e = 2, so that:
2
EW?1
/nim
Proposition 7. (Appendix D Consider the setup of Proposition As long as the weights are initialized following Equation with and the biases are initialized to O, we have that 7/ 78.
6.3 NETWORK INPUT SCALING BALANCES WEIGHTS AGAINST BIASES
traditional to normalize a dataset before applying a neural network so that the input vector has mean 0 and variance I in expectation. This principle is rarely quested in modern neural networks, even though there is no longer a good justification for its use in modern ReLU based networks. In contrast, theory provides direct guidance for the choice of input scaling. We show that the (non-central) second moment of the input affects the GR scaling of bias and weight parameters differently and that they be balanced by careful choice of the initialization.
Consider the GR scaling values for the bias and weight parameters in the first layer of a ReLU-based multilayer perceptron network, considered in previous sections. We assume the data is already

---

Under review as a conference paper at ICLR 2020
Table I: Scaling of common layers
Method
Maintains Scaling
Notes
Linear layer
Will not be well-conditioned against other layers unless geometric initialization is used
(Strided) convolution
X As above, but only if all kemel sizes the same Operations in residual blocks will be scaled correctly against each other, not against non non-residual operations
Skip connections
Average pooling
Max pooling
Dropout
ReLLU/LeakyRLLU
Any positively-homoogeouus function with degree
Sigmoid
Tanhh
Maintains scaling if entirely within the linear regime
centered. Then the scaling factors for the weight and bias layers are:
F[Avdl
F A
ng
[x]]]
7/o = Pā
E Elvöl
E Elydl
We can cancel terms to find the value of E
1 that makes these two quantities equal:
E
Kg
In common computer vision architectures, the input planes are the 3 color channels, and the kernel size is h 3, giving E [> xs 0.2. Using the traditional variance-one normalization will result in the effective learning rate for the bias terms being lower than that of the weight terms. This will result in potentially slower learning of the bias terms than for the input scaling we propose.
6.4 OUTPUT (NON-CENTRAL) SECOND MOMENTS
A neural network's behavior is also very sensitive the (non-central) second moment of the outputs. For a convolutional network without pooling layers (but potentially with strided dimensionality reduction), if geometric-mean initialization is used the activation (non-central) second moments given by:
E E[W?]E(a
ni
D E
m put
The application of sequence of these layers gives a telescoping product:
El0i+1]
II 1=0
n
out
no
We potentially have independent control over this (non-central) second moment at initialization, as we can insert a fixed scalar multiplication factor at the end the network that modifics it. This may be necessary when adapting network architecture that was designed and tested under different initialization scheme, as the success of the architecture may be partially due to the output scaling that happens to be produced by that original initialization. We are not guiding the choice of output variance at initialization for the case log- log-softmax losses, where has non-trivial effect on the back-propagated signals, although output variances of 0.01 to 0.1 appear work well. The output variance should always be checked and potentially corrected when switching initialization schemes.
of any existing theory
7 DESIGNING WELL-CONDITIONED NEURAL NETWORKS
Consider network where " is constant throughout. We may add a layer between any two existing layers without affecting this conditioning, as long as the new layer maintains the activation activation-gradient (non-central) secoddmomat product:
F F
7

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison on 26 LIBSVM repository datasets
Method
Average Normalized loss (#0.01)
Worst in #
Best in #
Arithmetic mean
0.90
14
3
Fan-in
0.84
3
5
Fan-out
0,88
9
12
Geometric mean
0.81
0
6
and dimensionality; this follows from Equation T For instance, adding a simple scaling layer of the form I1+1 2x; doubles the (non-central second moment during the forward pass and doubles the backward (non-central) second moment during back pack-propagation, which maintains this product:
1 E[A=]] 2E(z]]
When spatial dimensionality changes between layers we maintained just by balancing this product, as y depends directly on the square the spatial dimension. Instead, a pooling operation that changes the forward and backward signals in way that counteracts the change in spatial dimension is needed. The use of stride-2 convolutions, as well as average pooling, results in the correct scaling, but other common types of spatial reduction generally do not. Table Iiitts operations that preserve scaling when inserted into an existing preconditioned network. Operations such linear layers preserve the scaling of existing layers but are only themselves well- scaled if they are initialized correctly. For an architecture such as ResNet-50 that uses operations that break scaling, ay values can be introduced to correct scaling. ResNet-50, residual connections, max-pon poooloonnng, and varying kernel sizes need to be corrected for (we describe this procedure in Appendix
that the GR scaling is longer
It is particularly interesting to note that the evolution in state-of-the-an architectures corresponds closely to a move from poorly scaled building blocks to well-scaled ones. Early shallow architectures like LeNet-5 used tanh nonlinearities, which were replaced by the (well-scaled) ReLU, used for instance in the seminal AlexNet architecture (Krizhevsky et al.) 2012). AlexNet and the latter VGG architectures made heavy use of max- max-pooling and reshaping before the final layers, both operations which have been replaced in modern dully-convolutional architectures with (well-scaled) striding and wverage-poolingg respectively. The use of large kernel sizes is also in decline. The AlexNet architecture used kernel sizes of 11, and 3, whereas modern ResNet (He et al.) (016) architectures only use 7, and 1. Furthermore, recent research has shown that replacing the single 7x7 convolution used with sequence of three 3x3 convolutions improves performance (He et al. 2018).
EXPERIMENTAL RESULTS
We considered selection of dense and moderate moderate-sparsity multi-class classification datasets from the LibSVM repository, 26 in total. The same model was used for all datasets, a non-convolutional ReLUU network with 3 weight layers total. The inner layer widths were fixed at 384 and 64 nodes respectively. These numbers were chosen to result in a larger gap between the optimization methods, less difference could be expected if more typical 2x gap was used. Our results are otherwise generally robust to the choice of layer widths.
For every dataset, learning rate and initialization combination we ran 10 seeds and picked the median loss after epochs as the focus of our study (The largest differences can be expected early in training). Learning rates in the range 2 2- (in powers of 2) were checked for each dataset and initialization combination, with the best learning rate chosen in each case based off of the median of the 10 seeds. Training loss was used as the basis our comparison as we care primarily about convergence rate, and are comparing identical network architectures. Some additional details concerning the experimental setup and which datasets were used available in the Appendix.
Table I shows that geometric initialization is the most consistent of the initialization approaches considered. It has the lowest loss, after normalizing each dataset, and is never the worst of the 4 methods any dataset. Interestingly. the fan-out method is most often the best method, but consideration of the per-dataset plots (Appendix F) shows that it often completely fails to learn for some problems, which pulls up its average loss and results in it being the worst for 9 datasets.
8

---

Under review as a conference paper at ICLR 2020
10
10
Loss
10-2
Preconditioned (LR 0.05)
Arith. (LR 0.05)
10-3
Fan-in (LR 0.1)
Fan-out (LR 0.05)
0
1000
2000
3000
4000
5000
Step
Figure 3: CIFAR-I0 training loss for strided AlexNet architecture. An average of 10 seeds is shown for each initialization, where for each seed sliding window of minibatch training loss over 400 steps was used.
8.1 CONVOLUTIONAL EXPERIMENTS
Testing an initialization method on modern computer vision problems is problematic due to the heavy architecture search, both automated and manual, that is behind the current best methods. This search will fit the architecture to the initialization method, in sense, so any other initialization is disadvantage compared the one used during architecture search. This is further complicated by the prevalence of BatchNorm which is not handled in our theory. Instead, to provide clear comparison we use an older network with a large variability kernel sizes, the AlexNet architecture. This architecture has a large variety of filter sizes (11, 5, 3, linear), which according to our theory will affect the conditioning adversely, and which should highlight the differences between the methods. We found that network with consistent kernel sizes through-out showed only negligible differences between the initialization methods. The network was modified to replace max- max-pooling with striding max-pooling is not well-scaled by theory (further details in Appendix F
Following Section 6.4 we normalize the output of the network at initialization by running single batch through the network and adding a fixed scaling factor to the network to produce output standard deviation 0.05. For our preconditioned variant, we added alpha correction factors following Section
5 in conjunction with geometric initialization, and compared this against other common initialization methods. We tested CIFAR- 10 following the standard practice closely possible, as detailed in Appendix(F] We performed a geometric learning rate sweep over a power-of-two grid. Results are shown in Figure [3 for average of 10 seeds for each initialization. Preconditioning improves training loss over all other initialization schemes tested, although only by a small margin.
CONCLUSION
Although a panacea, by using the scaling principle have introduced, neural networks can be designed with reasonable expectation that they will be optimizable by stochastic gradient methods, minimizing the amount of guess-and-chect neural network design. As a consequence of our scaling principle, we have derived an initialization scheme that automatically preconditions common network architectures. Most developments neural network theory attempt to explain the success of existing techniques post-hoc. Instead, we show the power of the scaling law approach by deriving new initialization technique from theory directly.
REFERENCES
Jimmy Ba, Roger Grosse, and James Martens. Distributed second-orde order optimization using kronecker- factored approximations. International Conference On Learning Representations (ICIRR0177),

---

Under review as a conference paper at ICLR 2020
2017.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. International conference on machine learning (ICML), 2018.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar, signSGD: Compressed optimisation for non-convex problems. International conference machine learning (ICML), 2018.
Léon Bottou.
Reconnaissance de la parole par reseaux connexionnistes. In Proceedings of Neuro Nimes 88, pages 197-218, Nimes, France, 1988. http://:://een http://ewoo..tttoo.o.ooooo bottou.org
papers/bottou-Bel 88b.
Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Neural Information Processing Systems (NIPS), 2018.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker-factored eigenbasis, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010.
Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. International Conference on Machine Learning (ICML.2016). 2016.
Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? In Advances in Neural Information Processing Systems 31, pages 582-591. Curran Associates, Inc., 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. Technical report, Amazon Web Services, 2018.
C. G. J. Jacobi. Ueber eine neue auflösungsart der bei der methode der kleinsten quadrate vorkom- menden lineären gleichungen. Astron. Nachrichten, 1845.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference for Learning Representations (ICLR2015), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, pages 1097-1105, 2012.
Jean Lafond, Nicolas Vasilache, and Léon Bottou. Diagonal rescaling for neural networks. ArXiv e-prints, 2017.
Rafal Latala. Some estimates norms of random matrices. Proc. Amer. Math. Soc.,, 2005.
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Neural Networks: Tricks of the Trade, chapter Efficient BackProp. Springer, 2012.
Liam Li and Ameet S. Talwalkar. Random search and reproducibility for neural architecture search. CoRR, abs/1 900.07668, 2019.
James Martens. New insights and perspectives on the natural gradient method. In ArXiv e-prints, 2014.
10

---

Under review as a conference paper at ICLR 2020
James Martens and Roger Grosse. Optimizing neural networks with stonecker-factoree approximate curvature. International conference on machine learning (ICML), 2015.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In ArXiv e-prints, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015. Springer, 2015.
Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values. Proceedings of the International Congress of Mathematicians, 2010.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 13):211-252, 2015. doi: 10.1007/s 1263-015-0816-y.
Andrew M. Saxe, James I.. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ArXiv e-prints, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. international conference on learning representations (ICLR2015), 2015.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum deep learning. In International conference on machine learning (ICML), pages 1139-1147, 2013.
Terrance Tao. Topics in Random Matrix Theory. American Mathematical Soc.., 2012.
T. Tieleman and G. Hinton. Lecture 6.5-RmsProp: Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. Compressed sensing, 2012.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. ICML,, 2018
Jure Zbontar, Florian Knoll, Anuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Mare Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal, Adriana Romero, Michael Rabbat, Pascal Vincent, James Pinkerton, Duo Wang. Nafissa Yakubova, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: An open dataset and benchmarks for accelerated MRI. In ArXiv e-prints, 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations, 2019. URL
ew nwt net:
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. CoRR, abs/1611.01178, 2016. URL http: /tt.::///ax.v..og/abb/////// 01578

---

Under review as a conference paper at ICLR 2020
A
GR SCALING DERIVATION
Our quantity of interest is the average squared singular value of G1, which is simply equal to the (element-wise) non-central second moment of the product of with a i.d normal random vector r:
E[(Grr) =E(() ReJir 1.
Recall that our notation E[X²] refers to the element-wise non-central second moment of the vector. Through-out the remainder of this work we use non-central moments unless otherwise stated. To compute the second moment of the elements of Gyr, we can calculate the second moment of matrix- random- vector products against Jı, R₂ and J separately since R uncorrelated with Jı, and the back mack-propagated gradient Ay is uncorrelated with yı (Assumption A3).
JACOBIAN PRODUCTS JI AND Jf
Note that each of has ni" non-zero elements, each containing value from æ7. This structure can be written as a block matrix,
F1 0 0 0 31 0
J1
0
Where each zı is a 1 rn"" row vector. This can also be written a Kronecker product with an identity matrix as Ln- @ z7. The value 2 i.i.d random the bottom layer of the network (Assumption A1). For layers further up, the multiplication by random weight matrix from the previous layer ensures that the entries of z; are identically distributed (Assumption A3). So we have:
E -"
n" E"""""" = nii E[aT].
Note that didn't assume that the input øι mean zero, so V Var[z]] * E[=]]. This needed as often the input to a layer is the output from ReLU operation, which will not be mean zero.
For the transposed case, we have a single entry per column, so when multiplying by an i.i.d random vector u we have:
E
(J
El E[
(10)
UPPER HESSIAN R2 PRODUCT
Instead of using Rru, for a random u, we will instead compute it for w- u- wr/Effff it will have the same expectation since both Jir and yi are uncorrelated with R₁ (Assumption A3). The piecewise linear structure of the network above yt with respect to the y makes the structure of Rd particularly simple. It is a least-squares problem g(3)) = 1 1| - t||² for some & that is the linearization of the remainder of the network. The gradient is Ay ΦT (Ty 1) and the Hessian is simply R = क +. So we have that
E Avi
E
DT
(0y
t)l?
72
|| ( by
+
(Uncorr. Al)
|| | by
= [Riy))?
(Assumption A2)
nour
Applying this gives:
E. (Riu)²
Fl F/ [ 1/F((()
(11)
12

---

Under review as a conference paper at ICLR 2020
COMBINING
To compute B'(Gr) E[( ReJir) we then combine the simplifications from Equations
10
and 11 to give:
Al
=
E g
A.1 THE GAUSS-NEWTON MATRIX
Standard ReLU classification and regression networks have particularly simple structure for the Hessian with respect to the input, as the network's 's output is a piecewise-linear function feed into final layer consisting of convex log-softmax operation, in the Hessian with respect to the input being equivalent to its Gauss-Newton approximation. The Gauss-Newtor matrix can be written this work. We emphasize that this is just used as a convenience when working with diagonal blocks, the GN representation not an approximation in this case.
least-squares loss. This structure results
factored form, which used in the analysis we perform in
The (Generalized) Causs-Newton matrix G is positive semi- deninief approximation of the Hessian of a non-convex function f. given by factoring / into the composition of two functions f(x) = (x)()))
where h is convex, and g is approximated by its Jacobian matrix J at for the purpose of computing G:
G = J" (VPhg(=))) J.
The GN matrix also has close ties the Fisher information matrix (Martens 2014 providing another justification for its use.
Surprisingly, the Gauss-Newtor decomposition be used to compute diagonal blocks of the Hessian with respect to the weights Wi as well as the inputs (Martens) (014). To see this, note that for any
activation yn, the layers above may be treated in combined fashion the h in a f(Wi) = (Wi) ) decomposition of the network structure, as they are the composition of (locally) linear function and convex function and thus convex. this decomposition g(i) =Wiæ + by function of Wi with xạ fixed, and as this is linear in W,, the Gauss- approximation to the block is thus not an approximation.
B FORWARD AND BACKWARD SECOND MOMENTS
We make heavy use of the equations for forward propagation and backward propagation of second moments, under the assumption that the weights are uncorrelated to the activations gradients. For a convolution
= Cw (x:)
with input channels nh output channels n out and square k k kernels, these formulas are (recall notation for the second moments is element-wise for vectors and matrices):
E|Ar? - " B[
C THE WEIGHT GRADIENT RATIO IS EQUAL TO GR SCALING FOR MLP MODELS
Proposition 8. The seeight-gradien ratio Vi is equal to the GR scaling for i.id mean-zero cumdomly-inttaalicn multilayer perceptron layers under the independence assumptions of Appendix
Proof. To see the equivalence, note that under the zero-bias initialization, we have from yr Wi that:
Flyil ni F(w?)
(12)
13

---

Under review as a conference paper at ICLR 2020
and so:
Elyf E[W?]
ni[ E [x
The gradient of the weights is given by AW, Aycay and its second moment is:
A 7 = E Elui)
(13)
Combining these quantities gives:
B|AW E
El
nt [aî]²
11
Elyi
D BIAS SCALING
We consider the case of a convolutional neural network with spatial resolution X for greater generality. Consider the Jacobian y with respect the bias. has shape J (ngaa pi) (ngat). Each row corresponds to Yr output, and each column bias weight. As before, we will approximate the product of G with random i.i.d unit variance vector r:
JpT
The structure of J is that each block of p² rows has the same set of Is in the same column. Only single per row. It follows that:
[J}
The calculation of the product of R₁ with Jfr approximated in the same way in the weight scaling calculation. For the J&T product, note that there an additional p² each column has p² non-zero entries, each equal to 1. Combining these three quantities gives:
ElAvi p²
11
E Ely
Proposition Consider the setup of Proposition
5
with the addition of biases:
3/0 Woxo + bo:
ReLU ReLU(vo),
y1 W1 + b2.
As long as the weights are initialized following Equation 7 and the biases are initialized to 0, we have that
70 71
We will include e = as variable as it clarifies it's relation other quantities. We reuse some calculations from Proposition 5 Namely that:
Elvâl = ev
E El
1 El
E[Anf]
29
2
Plugging these into the definition of nd:
ElAyil
A
70
Elg
E
1L
2
14

---

Under review as a conference paper at ICLR 2020
For 71, require the additional quantity:
E'yil
mE(uiE [Wi]
m 2 n E[ai]
(vrpp)
c
m
c²
E|a0l-
2
p
Again plugging this in:
ELAvi
E'Wit
EIANi
Vp A
So comparing these expressions for 7 and 7 that 70 7 if and only if c 2.
E CONDITIONING OF RESNETS WITHOUT NORMALIZATION LAYERS
There has been significant recent interest in training residual networks without the use of batch- normalization or other normalization layers (Zhang et al. 2019). this section, we explore the modifications that necessary network for this to be possible and show how apply preconditioning principle to these networks.
The building block of ResNet model is the residual block:
21+1 ReL.U (F(=)) + 21).
where F is a composition of layers. Unlike classical feedforward architectures, the pass-through connection results an exponential increase in the variance of the activations in the network as the depth increases. A side effect of this is the output of the network becomes exponentially more sensitive to the input of the network as depth increases, a property characterized by the Lipschitz constant of the network (Hanin) 2018).
This exponential dependence can reduced by the introduction of scaling constants i to each block:
2++ ReLU (s/F(z)) + =1)
The introduction of these constants requires modification of the block structure to ensure constant conditioning between blocks. standard bottleneck block, as used the ResNet-50 architecture, has the following form:
Co(xo),
Ø1 ReLU(vo).
Ci (x1),
D2 ReLU(11),
= C2(22),
Ø3 ReLU(y2 xx).
In this notation, Co is a 1 x convolution that reduces the number of channels 4 fold, C₁ is a x convolution with equal input and output channels, and Cz is a x 1 convolution at increases the number of channels back up 4 fold to the original input count.
we introduce scaling factor s; to each block I, then we must also add conditioning multipliers 3 to each convolution change their GR scaling, as described in Section 5 The correct scaling
15

---

Under review as a conference paper at ICLR 2020
constant depends on the scaling constant of the previous block. A simple caleulation gives the equation:
1 1 8
B= 3
1+ 8/-1
The initial Bo and so may be chosen arbitrarily. a flat st = s used for all 1, then we may use 3t 1. The block structure including the Br factors is:
3o 3 Co(zo),
21 ReL.U(UD),
V1 = V33 Ci(x1),
22 = ReI.U(y1)
1/2 3 C2(22),
F3 ReL.U (sU2 + mo)
The weights each convolution must then be initialized with the standard deviation modified such that the combined convolution-scallee operation gives the same output variance would be given the geometric-mean initialization scheme is used without extra scaling constants. For instance, the initialization of the Co convolution must have standard deviation scaled down by dividing by so that the multiplication by 3 during the forward pass results in the correct forward variance. The 1/v3 factor is an a correction that corrects for change in kernel shape for the middle convolution.
The variance at initialization must be scaled to correct for the factor also.
E. CORRECTION FOR MIXED RESIDUAL AND NON-RESIDUAL BLOCKS
Since the initial convolution in a ResNet-50 model is also not within residual block, it's GR scaling different from the convolutions within residual blocks. Consider the composition of a non-residual followed by residual block, without max-pooling or ReLUs for simplicity of exposition:
yo = aCo(oo),
I1 = yo:
91 = si
21 1/ +
Without loss of generality, we assume that E [ 1, and assume a single channel input and output.
Our goal is to find a constant a so that initialize Co so that the variance of yo is independent the choice of a. Our scaling factor will also depend on the kernel sizes used in the two convolutions, so we must include those in the calculations.
71. Recall that when using a scaling factors we must
From Equation| D the GR scaling for Co is
E
70 az in12
21
E.
a =a*3
Note that ElAyi] (1 + st) EIA2 o so:
%o = (1+ + 2 RE
For the residual convolution, need to use modification of the standard GR equation due to the residual branch. The derivation of y for non-rusidual convolutions assumes that the remainder of the network above the convolution responds linearly (locally) with the scaling of the convolution, but here due to the residual connection, this is no longer the case. For instance, if the weight were scaled to zero, the output of the network would not also become zero (recall assumption of
16

---

Under review as a conference paper at ICLR 2020
cero-initializzaton for bias terms). This can be avoided by noting that the ratio E/ Elyil in the GR scaling may be computed further up the network, as long as any scaling in between corrected for. In particular, may compute this ratio at the point after the residual addition, as long include the factor 8f to account for this. So we in fact have:
E E
M1 = sfn/ARF
[
E.
siM?
1
We now equate To = =11
E7 A
k
(1+ (0 ((a2),
1 + 1
st
a
kg
(1+ stt
Therefore to ensure that 71 we need:
k1
st
a
ko (1 + st)
FINAL LAYER
A similar calculation applies when the residual block is before the non-residual convolution, as in the last layer linear the ResNet network, giving a scaling factor for the linear layer (effective kernel size 1) of:
$7 (1
a²
F FULL EXPERIMENTAL RESULTS
DETAILS OF LIBSVM DATASET INPUT/OUTPUT SCALING
To prevent the results from being skewed by the number of classes and the number of inputs affecting the output variance, the logit output of the network scaled to have standard deviation 0.05 after the first minibatch evaluation for every method, with the scaling constant fixed thereafter. LayerNorm was used on the input to whiten the data. Weight decay of 0.00001 was used for every dataset. To aggregate the losses across datasets we divided by the worst loss across the initializations before averaging.
LIBSVM PLOTS
Figure 4 shows the interquartile range (25%, 50% and 75% quantiles) of the best learning rate for each case.
CIFAR10
Following standard practice, training used random augmentations consisting of horizontal flips and random crops 32x32, as well normalization to the interval [-1,+1]. We used SGD with momentum 0.9, a learning rate schedule of decreases at 150 and 225 epochs, and no weight decay.
The network architecture is the following sequence, with circular "equal" padding used and ReLU nonlinearities after each convolution:
1. 11x11 stride-1 convolution with 3 input and 64 output channels,
2. 5x5 stride-2 convolution with 64 input and 192 output channels,
3. 3x3 stride-2 convolution with 192 input and 384 output channels,
17

---

Under review as a conference paper at ICLR 2020
4. 3x3 convolution with 384 input and 256 output channels,
5. 3x3 convolution with 256 input and 256 output channels,
6. Average pooling to Ixl followed by reshaping to non-spatial,
7. Linear layer with 256 input and 4096 output features,
8. Linear layer with 4096 input and 4096 output features,
9. Linear layer with 4096 input and 10 output features,
18

---

Under review as a conference paper at ICLR 2020
AmallNOR2-20 12002
0.14
2338
3.0
1.35
0.13
1.30
0:22
0.10
0.00
6.15
3.54
642
0.04
0.22
8.5
8118
112
011
3.4
0.22
0.00
6.6
Opirt
Ipoch
Epoch
Epect
Ipach
pendigia
shartio
zeve20
10
6370
0.2
6930
2.6
0:33
6.33
1.81-
9:14
0000
012
012
833
Qpick
1pock
Epoch
Epect
Spac)
Jetter
rempudde2
eadmaage
0.55
m93
0.-
0.30
3.
0.312
0.50
3
0,02
0.90
0.5
0.222
Spork
1pac
1poch
Epect
kpac)
ccntect-4
Alci ccol
>abille
Senserine
0.43
0.90
6.155
6450
0.3
632
3.-
0.3
1943
Ryon
spact
Accuetic coale
0.51
1.36
1.55
0.5
0.50
1.0
0.5
0.5
6:1
dlar10
1.7
1.8
Figure 4: Full results for the 26 datasets
19


Reviews:
[{"review_id": "r1lkMDOaKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new initialization scheme for training neural networks. The initialization considers fan-in and fan-out, to regularize the range of singular values of the Hessian matrix, under several assumptions.\n\nThe proposed approach gives important insights for the problem of weight initialization in neural networks. Overall, the method makes sense. However, I have several concerns:\n\n- The authors do not consider more recent neural network designs such as normalization layers, skip connections, etc. It would be great if the authors could discuss how these layers would change the derivation of the initialization method. Also, preliminary experimental results using these layers are needed. Additionally, to me, normalization layers [Huang et al. Decorrelated Batch Normalization. CVPR 2018] implicitly precondition the Hessian matrix as well. It would be great if the authors also compare their approach to [Huang et al. 2018].\n\n- The authors compared to other initialization schemes such as [He et al., 2015] and [ Glorot and Bengio 2010]. But as the authors mentioned, there are approaches that scales backpropagation gradients also [Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2017; George et al., 2018]. Since these methods are highly related to the proposed method, it would be great if the authors could show time complexities and performance differences of these methods as well.\n\n- Experiments on the CIFAR-10 dataset with AlexNet seem not exciting: the proposed Preconditioned approach only outperforms the Fan-out approach marginally. I would say that training a [He et al. 2015]-initialized neural network for 500 more iterations than a preconditioned neural network, yields a similar or better loss.\n\nOverall I think the work is very important and interesting. However, it lacks comprehensive comparison and consideration of more recent neural network layers.\n\nPost Rebuttal Comments\nI have read all reviewer comments and the author feedback. I appreciate that the authors addressed the skip connections in Appendix.\n\n1. The authors agree that batch norm requires different initialization schemes that are not included in this paper.\n2. I agree with the authors that their approach is complementary to the baseline optimization methods; and both approaches can be applied together. However, I still believe that it is informative to compare the two approaches because: (a). Both approaches address the same problem. Since the optimization based approach adds complexity and computational overhead to implementation, it would be great to show if using the proposed approach eliminates the need for the optimization based approach. (b). Is it necessary to use both approaches, or one of them is good enough?\n3. I understand that strong experimental evidence is not always required. However, I believe that the new technical insights of the paper alone is not significant enough (part of the reasons in point 1). Thus I was expecting stronger experimental evidences.\n\nOverall I agree with reviewer 1 that the topic is interesting, but in the paper’s current form, it is not ready. I keep my initial rating of weak reject.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkl5cQtQ9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the topic of the paper is interesting, though I think in its current form the paper is not ready. \n\nFirst of all I find the empirical section quite weak. While the authors attempt to formalize their intuition, as they mention in the work itself, such works are somewhat outside mathematical proof. This is due to the many approximations needed, and assumptions that can not hold in practice. As such the main experimental results in running AlexNet on Cifar-10 and LIBSVM datasets. I think more experimental evidence is needed, e.g. more architectures, more dataset (maybe different data modalities). Is hard to tell from this one main experiment (Cifar 10) to what extend one can trust this initialization. \n\n\nI think is worth noting that KFAC and (all?) cited methods actually use the Fisher Information matrix (hence being forms of natural gradient) and not the Hessian. The extended Gauss-Newton approximation is indeed the Fisher matrix, I think as discussed in Martens' work (which is heavily cited) though in other works as well. Just as an additional note, the extended Gauss-Newton was introduced in  Schraudolph, N. N. (2002). \"Fast curvature matrix-vector products for second-order gradient descent\" where it was presented as an approx to the Hessian, this was used by Martens and later the community (and he himself) observed that actually rather than an approx to Hessian this can be thought of as the Fisher Information Matrix.\n\nRelying on the expected squared singular value should be motivated better. The reasoning sounds fine (i.e. minimum and maximum would be too pessimistic) but no data is given. Some statistics over multiple runs. Overall this is a repeating theme in the work. While everything makes sense intuitively, I would have hoped more rigor. More empirical evidence for any such choice. The weight-to-gradient ratio is not a commonly used measure. Maybe show how this ratio progresses over time when training a model. Having multiple runs showing the correlation between things. Table 2 is not referenced in the text. While an average over 10 seeds is provided for Cifar, no error bars. \n\nOverall I think the direction of the work is interesting. And definitely we have not heard the last word on initialization. It plays a crucial role (and indeed bad initialization can easily induce bad local minima (https://arxiv.org/abs/1611.06310)). But I think the paper needs to be written more carefully, with a more thorough empirical exploration, showing different architectures, different datasets. Maybe trying to break the usual initialization, and showing you can do considerably better with newer initialization. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e3e6yTKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a set of rules for the design and initialization of well-conditioned neural networks by naturally balancing the diagonal blocks of the Hessian at the start of training. Overall, the paper is well written and clear in comparison and explanation. However, the reviewer is concerned with the following questions:\nAssumption A2 does not make sense in the context. In particular, it is not praised to assume it only for the convenience of computation without giving any example when the assumption would hold. Also the assumptions are vague and hard to understand, it is better to have concrete mathematical formulation after text description.\nAre the experiment results sensitive to the choice of different models with different width and layers or different batch sizes? Does it have a strong improvement than random initialization? It’s less clear the necessity of guaranteeing well-conditioned at initialization since during the training procedure, the condition number is harder to control.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new design space for initialization of neural networks motivated by balancing the singular values of the Hessian. Reviewers found the problem well motivated and agreed that the proposed method has merit, however more rigorous experiments are required to demonstrate that the ideas in this work are significant progress over current known techniques. As noted by Reviewer 2, there has been substantial prior work on initialization and conditioning that needs to be discussed as they relate to the proposed method. The AC notes two additional, closely related initialization schemes that should be discussed [1,2]. Comparing with stronger baselines on more recent modern architectures would improve this work significantly.

[1]: https://nips.cc/Conferences/2019/Schedule?showEvent=14216
[2]: https://arxiv.org/abs/1901.09321.
Decision: Reject