ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
Authors: Jiawei Zhao
Florian Tobias Schaefer
Anima Anandkumar
Venue: Accepted by TMLR ()

Abstract: Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, selecting the appropriate variance becomes challenging especially as the number of layers grows. In this work, we replace random weight initialization with a fully deterministic initialization scheme, viz., ZerO, which initializes the weights of networks with only zeros and ones (up to a normalization factor), based on identity and Hadamard transforms. Through both theoretical and empirical studies, we demonstrate that ZerO is able to train networks without damaging their expressivity. Applying ZerO on ResNet achieves state-of-the-art performance on various datasets, including ImageNet, which suggests random weights may be unnecessary for network initialization. In addition, ZerO has many benefits, such as training ultra deep networks (without batch-normalization), exhibiting low-rank learning trajectories that result in low-rank and sparse solutions, and improving training reproducibility.

Published Transactions on Machine Learning Rescarch (10/2022)
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
Jiawei Zhao Californio Institute of Technology
jaweiiCaltechh edu
Florian Schäfer
sorrinnschaeeerBec.co gatecch edu
Georgia Institute of Technology
Anima Anandkumar
animaccaltech. edu
California Institute of Technology NVIDIA
Reviewed OpenReview: https: // //tp://oppeee openreview. net/ formm? id= 1AmQpKmi Tc
Abstract
Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, selecting the appropriate variance becomes challenging especially the number of layers grows. In this work, replace random weight initialization with fully deterministic initialization scheme, viz., ZerO, which initializes the weights of networks with only zeros and ones (up a normalization factor), based on identity and Hadamard transforms. Through both theoretical and empirical studies, we demonstrate that ZerO is able to train networks without damaging their expressivity. Applying ZerO on ResNet achieves state-of the-art performance on various datasets, including ImageNet, which suggests random weights may unnecessary for network initialization. In addition, ZerO has many benefits, such as training ultra deep networks (without natch-normalization) exhibiting low-rank learning trajectorics that result in low-rank and sparse solutions, and improving training reproducibility!.
1 Introduction
An important question in training deep neural networks is how to initialize the weights. Currently, random weight initialization is the de-facto practice across all architectures and tasks. However, choosing the variance of the initial weight distribution is a delicate balance when training deep neural networks. the variance is too large, it lead training, resulting in eaploding gradients. On the other hand, if the weights are initialized too small, the activations may not propagate at all, resulting in vanishing gradients. These issues become more challenging as the number of layers in the network grows.
excessive amplification of the activations propagating through the network during
The above challenges can be avoided if identity initialization is used instead. It initializes each layer in the network as an identity mapping, such that the input data can identically propagated to the network output. In this case, there is no need to introduce any randomness or consider its variance. Identity initialization is well studied theoretically from optimization perspective. Hardt & Ma (2017) prove the existence of global minimum close to the identity parameterization in a deep residual network. Bartlett et al. (2019) further prove the rate of convergence of gradient-based optimization under identity initialization.
Code repository: https://githul http:/twww....eaeeeeeaa/aaaaaaaa

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Comparing ZerO initialization with randomly perturbed identity initialization. transformation' of an arbitrary layer 1,
F represents
Settings
Techniques
Related Works
2141 = aq + F(xi)
is randomly initialized with small variance
Hardt & Ma (2017)
2141 = 2; + Cono(F(ax))
is randomly initialized, kernel in Conv is zero
Zhang et al. (2019)
2141 = #t + (Nom(((xx))
randomly initialized, scale in Norm is zero
Goyal et al. (2017)
3141 = xt + (<(((z)))
randomly initialized, scalar a zero
Bachlechner et al. (2020)
All settings above
F is a Hadamard/identity transform²
ZerO (ours)
However, prior theoretical works on identity initialization assume that all layers had the same dimensionality, which docs not hold for practical networks. Typically, practical networks have varying dimensionality across layers, such as the variations of spatial and channel dimensions in ResNet architectures (Ile et al., 2016). Directly applying identity initialization to these networks leads to a problem of training degeneracy, as our theoretical study will demonstrate later.
To avoid the training degeneracy, previous works (summarized in Table 1) employ identity initialization with random perturbations to facilitate escapes from saddle point or to break feature symmetry (Blumenfeld et al., 2020). Broadly, these approaches satisfy the property of dynamical isometry, to preserve the signal propagation and ensure well-behaved gradients at initialization (Saxe et al., 2014). Despite the efficacy of random perturbations, they inevitably introduce additional tuning of variances, which can result in gradient explosion in deep networks without careful tuning.
Our work: we propose ZerO initialization that removes all randouness the weight initialization. illustrated in Figure 1, ZerO initializes networks with Hadamard and identity transforms, which assigns all the weights to only zeros and ones.
(hi ho: fa.....
Qulput
< (Bick, 6x.
(his dgg Rgy...
Indu
(21. (x.)
Figure 1: Illustrating ZerO in 3-layer network with input dimension Na, hidden dimension Na. and output dimension Ny, where N₂ > Nr, Ny. H and I are N₂ x N,, Hadamard and identity matrix, respectively. The iutension-increusiigg layer is initialized by columns of the Hadamard matrix. The rest layers initialized by identity matrix or rows of it.
ZerO is not affected by the problem of training degeneracy accuracy loss. Compared to random initializa- tion, ZerO provides state-of-the-art performance over various image classification tasks, including ImageNet. We further discover many unique properties and benefits of ZerO:
Stable training without batch normalization. ZerO ensures well-behaved signal propagation, which provides stable training without batch normalization. Testing ResNet with over 500 layers, we find ZerO converges faster than carefully designed random methods, such as Fixup and ReZero (Zhang et al., 2019; Bachlechner et al., 2020).
Low-rank learning trajectory. We find that ZerO exhibits a low-rank learning trajectory, where the rank of each matrix gradually increases during training. We believe this is the first time that the greedy low-rank
The transformation may contain linear, nonlinear convolutional operations. Both identity and Iladamard transforms deterministic parameterizations.

---

Published in Transactions on Machine Learning Research (10/2022)
learning (GLRL) trajectory, theoretical characterization of gradient descent, has been observed in large- scale deep learning applications. GLRLL is a consequence of implicit. rank regularization by gradient descent under infinitesimal initialization (Li et al., 2021; Razin al., 2021). It can be viewed as performing a rank- constrained optimization and greedily relaxing the rank restriction by one whenever it fails to reach global minimizer. GLRL has been used to explain the excellent generalization in gradient-based deep learning, as it converges to a global (or local) minima with the minimum rank. However, the GLRI trajectory has never been observed in practice due to its impractical requirement of infinitesimal initialization.
Sparse and low-rank solutions. We observe that ZerO-initialized networks converge sparse and low-rank solutions. Compared to randomly initialized networks, the sub-networks obtained in trained ZerO- initialized networks achieve 30% lower (matrix or tensor) rank or 25% higher spansity without sacrificing accuracy.
Better training reproducibility. Since ZerO does not require any random perturbations, it is a fully deterministic initialization scheme. Unlike determinism random initialization, which needs fixing pseu- dorandom number generators in hardware, the weights initialized by ZerO are fixed regardless of how the random seed varies or which hardware is used. ZerO significantly reduces the training variation and thus achieves better training reproducibility (the remaining randomness is only due batch selection). Com- pared to random initialization, ZerO produces 20%-40% lower standard deviation of the final accuracy over repeated experiments with different random seeds.
Theoretical analysis of ZerO. Theoretically, we demonstrate that ZerO breaks training degeneracy that arises when applying identity initialization to networks with varying dimensionality across layers. We prove that the training degeneracy necessarily occurs in standard identity initialization because the rank of any N₂ N₂ matrix in the network is upper bounded by the input and output dimensions N₂ and N₂ throughout the entire training, no matter how large the size of N,, is. This limits the expressivity of each matrix, resulting in the degeneracy training.
Our contributions summarized follows:
1. We design ZerO initialization, the first fully deterministic initialization that achicves statc-of-tho-art performance in practice.
2. ZerO is backed with theoretical understanding. As shown in Theorem 1, we prove how ZerO breaks the training degeneracy by applying Hadamard transforms.
3. ZerO has many benefits, such as training ultra deep networks (without batch-normalization), exhibit- ing low-rank learning trajectory, converging to sparse and low-rank solutions, and improving training reproducibility.
2 Is Randomness Necessary in Identity Initialization?
2.1 Background
We wish to train function F(æ) to learn a particular input-output map given set of P training samples (x0 (z") E RNN,N,, where f I...., P. Training is accomplished by minimizing the squared error L = Ei Ily" - 122 using gradient descent with step size 7,
We model F(x) to be a multilayer perceptron with L > 2 hidden layers, such that:
Wiz
2 p(x1))
with / 1, L. Let z x and F(x) 2t. fp an element- wise nonlinearity. We assume that F has uniform hidden dimension No. with Wi E N.N for E L-1, W, E N.X.. and W, E N,N.N

---

Published in Transactions on Machine Learning Rescarch (10/2022)
The input-output Jacobian is well-studied proxy for estimating the stability of signal propagation at initialization (Saxe et al., 2014). It is defined as:
dzr.
Jio Ozo
Proposed by (Saxc et al., 2014), dynamical isometry a condition where all singular values of the Jacobian Ji concentrate near 1. If Jo is well-conditioned, the backpropagation error da well-conditioned as well. This ensures stable signal propagation and well-behaved gradients at initialization.
8c at any layer I will be
Consider the case of N₂ = N2 = Nn- Identity initialization defined as initializing each matrix to be an identity matrix: W = I. In this case, the dynamical isometry property for a linear F can be easily verified as J = I. It also holds for certain nonlinearities when applying the identity initialization residual networks: 2 = p(xx) + x1-1 where =
= 0, such that no signal is passed through the residual branches at initialization. Table lists few related examples.
From optimization perspective, Hardt & Ma (2017) suggest that F has a global minimum very close to identity initialization, such that maxigigt. [illl S O(1/L) for large L, where W W Bartlett et al. (2019) also proves that under the identity initialization, gradient descent learns an e-approximation of FF within iterations polynomial in (1g(1/c).
2.2 Extending to Large Hidden Dimension
far, we have discussed identity initialization the special case of fixed dimensionality. Now extend discussion to more practical sctting where F is equipped with large hidden dimension, such that N >> Ny. We also focus on the specific case where o is a Relu nonlinearity.
Inpuc
T I*
I
+
A straightforward approach to identity initialization in this case to initialize Wi such that projects the input into a Ny-dimensional subspace of the Na-dimensional hidden space. This can be achieved by initializing Wi with a partial identity matric:
3-layer network F (Nr > N. N₂) where Wi, W, 1' and W2 = at ini- tialization.
Figure 2:
Definition (Partial Identity Matrix). Let 1 E Rixx, the partial identity matrix I' defined as follows:
(I,0)
where I Idd and 0 R if
r= = I where I E Rd.
if i
(I,O)T
where I RMr and 0 Rnd-r otherwise.
For vector E R", if r, then (a(a) clips the last few dimension such that (a)a) = (a1, a2,..... a)', If >> r, then I' padss / - additional dimensions with zero, such that (a1, (2,,a,,.... Cr, 0...0) 1 This is also known as a zero-padding operator, such as used channel-expanding layers in ResNet (He et al., 2016). the network F, I' only needs to be applied in the dimension-vare varying matrices W, and Wz, while leaving the remaining N₁, N₁, matrices to be identity matrix I.
We visualize this process in Figure This may seem like a natural extension of identity initialization to a large width setting, but will show in Section 2.3 it suffers from a problem call "training degeneracy". To avoid the problem, we use the Hadamard matrix H to initialize the dimension iiriooonnnnreeiingg matrices, such that W = HI' A Hadamard matrix is defined as follows:
Definition (Hadamard matrix). For any Hadamard matrix H Hm 22""22" where m positive integer, define H, by the identity, and the matrix with large is defined recursively:
Hm-1 Hm-1 Hm-1)
H =
R x2
Hm-1 -Hm-1

---

Published in Transactions on Machine Learning Research (10/2022)
The linear transformation described by the Hadamard matrix, called the Hadamard transform, rotates the coordinate axes to be equally weakly aligned with the standard basis. For example, in a two-dimensional plane, the Hadamard transform rotates the standard basis by an exact angle of degroe. This turns out to be important property for breaking the training degeneracy.
2.3 Identity Initialization limits Network Expressivity
We now present our main result diffcrentiating the training behavior of different initialization methods and describing the problem of training degeneracy.
Theorem 1. Let FF be neural network with L matrices, where Wi NaXNN for 1 L, W1 E R,,,,, and Wt E No, F has uaiform hidden dimension Na, input dimension Na, and output dimension Ny: where N₁ 2 Na, Ny- Defone residual component W = Wi- Let z((x) to be the activation the 1-th layer under the imput æ. Then we have the following results for different initialiaation::
(i) Consider a random perturbation p E S where each element is sampled from a Gaussian dis- tribution: pig ((00,²²). It is well-known that the randomly perturbed matrices Wi = =I lr (for + 1, L,) are full-rank almost surcly:
Prob(rank( (Wi) = Nx)=1 =1 for 2,,,..,L
(1)
(ii) When initializing W1, WL = 1' and the remaining matrices as Wi = (for L + L,,, for any x e N=,, the linear dimension of the set of all possible activations is bounded throughout training as
dim (span (1 J)) s No for E L....
(2)
As result, the ranks of the weight matrices remain bounded throughout training
rank (W;) s N₂ for lorle2 1-
(3)
(ii) When ireitializing = = HI,, W, I' and the remaining matrices as Wi = I (for 1 1, L, it is possible for the activations at an intermediate layer to attatn
dim (span (f=((x)) E {R) No.
(4)
breaking the construint the linear dimension described in Equation 2.
The constroints linear dimensions and matric ranks in Equation and suggest that matter how large hidden dimension Nn is, the network F is only optimized within low-dimensional sub- space (depending on input dimension N&) of the full parameter space. This restricts the marimum network expressivity of F, and thus the training may only converge an underfitting regime, leading training degeneracy.
Remark 1.
Remark 2. Under the assumptions of (iii), the breaking of training degeneracy described in Equation appears to be the generic case. As verified empirically in Figure 8, applying the Hadamard transform in (ii) also breaks the rank constraint Equation 3.
Remark (i) and (iii) avoid the training degeneracy from different directions. Unlike (i), the proposed (iii) doesn't introduce any randomness with the help of the Hadamord transform.
Almost all existing works use random weight initialization, which largely affects the rank of each matrix as shown in (i). (i) can be proved by showing any column (or row) in a random matrix linearly independent to the other columns (or rows), almost surely. A detailed proof can be found in the appendix.
Consider identity initialization without any randomness. In (ii), W1 identically maps the input into a subspace of the hidden layer 21, such that 2; = (x ...... 0) T. Thus, the linear dimension on z (i.e., the linear dimension on activations =/,..., x equivalent the linear dimension on æ, which is upper bounded by Nz. This result is held for every layer (where + 1, L).

---

Published in Transactions on Machine Learning Research (10/2022)
191
Standard Gi-Init
2730
Hadamard based Gi-nit
Rank
Rank
.003
Ns 512
231
NA
100
350
Ns 40266
7900
1000
7000
1000
13500
10000
17500
3000 5301
1560 10000
17500
10000 17000
Iterations
Iterations
Figure 3: Verify Theorem pracfice. We train network J. with L 3 on MNIST, and visualize rank(W2) over the training. The red dash line denotes the rank constraint on W2 predicted by the theorem, which is N, - 784. Left: verify (ii) in the theorem by varying Nr. No matter how large Ns. is, rank(Ws) follows the rank constraint through the entire training. Right: we verify (fii) where applying Haadmmrr transform breaks the rank constraint introduced in (ii) given No = 2048. We denote the initializations in (ii) and (iii) as standard and Hadamord based Gi-Init, respectively. As predicted in (i), random initialization achieves its mozimumn rank immediatell after the initialization.
To show the rank constraint in Equation track single derivative of the residual component at layer t:
ac
ac dzy
(5)
Ow
where @ denotes the outer product. We use the following well-known fact:
Lemma 1. Consider matrir M to be sum of vector outer products:M = Ei a &b" where a" RN.
and 6 E R. for μ 1,,,, Q. Let Q Na, Nb. If linear dimensions dim(span(a), ,.., aQ)) U and dim(span(b)) b.), S V, where U S Na and V S No, then:
mnk(W)) I mi)(U.V)
By Lemma 1, at initialization, the upper bound N₂ the linear dimension 21-1 results in rank Sc. has a zero-valued constraint on ran)(W?). The rank constraint holds during the entire training as SW7
Ng (NA N₂) sub-matrix at every iteration (as shown in the appendix). Since W/ = -T W - - after T weight updates (by gradient descent with a step size n), rank(W7)) bounded by N₂ no matter what T is. This results in the training degeneracy described in Remark 1. We also verify it empirically in Figure 3.
To avoid the training degeneracy, we need to overcome limitations on the linear dimension of the set of possible activations. This is indeed possible when using the Hadamard matrix Wi HI,, as will illustrated by means of example.
Lemma 2. Assume N₁, 4 and N₂ 3. For any vector æ e span(ez, es) where eg and es are coordinate vectors (0, 1) and (0, T, it holds that:
span ((=1 E No }) where HaluHHI's) Relu(-HI's Relu(HI' ez), and Relu(- -II--II" are linearly independent. indicates that:
Relu( Houu(-HI"ee e2), (elu(HH'e Relue))) ea), Relu HI ea)),
(6) This
dim (span (12) (z) E N))))) = No. N 3.
When using Hadamard matrix as Wi = HI*, the breaking of the training degeneracy described Lemma. appears to be the generic case, As verified empirically in Figure 3, this also breaks the rank constraint in Equation 3.

---

Published in Transactions on Machine Learning Research (10/2022)
We point out that the increase of the linear dimension of the set of possible is only possible due to the nonlinearity. F is a linear network, the linear dimension on every 21 is at most Nz, no matter how the weights are initializod.
Nevertheless, the nonlinearity can not increase the linear dimensionality if we initialize the network with a partial identity matrix. This is because when W1 = I', span ((z((z))< [B*))) is aligned with the standard basis, Le., each vector the span least has M,, - N₂ zero coefficients when expressed in the standard basis. Thus, an element-wis nonlinearity can not increase the linear dimension of its input beyond No.
To break the aligmment span (<=(<>))< E }R) with the standard basis, we use the Hadamard transform. This because transforms the subspace such that the new basis is equally weakly aligned with the standard basis. We note that other linear transforms may also detach the subspace from the standard basis, but the Hadamard transform is the most natural choice.
3 ZerO Initialization
The initialization analyzed in (iii) of Theorem 1 is based on network condition in which all hidden spaces 21..... #1-1 have the same dimension N,. Motivated by our theoretical understanding, we propose ZerO initialization, which initializes the weights any network with arbitrary hidden dimensions. As described in Algorithm 1, ZerO only initializes dimensional-inercasnng layers with Hadamard matrices to avoid the training degeneracy. Other layers are simply initialized by (partial) identity matrices. We also rescale Hadamard matrices by a normalization factor 2-(m-1)/2 resulting in an orthonormal Hadamard transform.
Algorithm 1 ZerO Initialization.
Input: a neural network FF with L matrices Wi IR P for 1 in l,..., I. I' is partial identity matrix defined in Definition 1. Hm is the Hadamard matrix defined in Definition 2.
For in 1.....I La
If P: = Qi: W I- If < Q:: Wi I' If PRS QF: W1 t- eIIH' I", where m = (log((P)) and == 2 2-
Identity mapping
> Propagate the first P1 dimensions D Apply Hadamard matrix
We also apply ZerO to the well-developed ResNet architectures in He et al. (2016). As shown in Algorithm 2, we apply ZerO convolution in similar way by considering the variation in channel dimensions. When K is 1xl convolution, K also can be viewed Cout Con matrix, which matches the initialization in Algorithm 1. We note that Algorithm 2 can be applied to every convolution in ResNet, including the first 3x3 convolution, 1x1 convolutions in and bottleneck block.
skip connections, and convolutions in basic block
To achieve dynamical isometry at initialization, we apply common technique that initializes the last convolution each residual block as zero. This helps suppress the signals from residual branches to stabilize signal propagations, studied in Zhang et al. (2019); Bachlechner et al. (2020).
Algorithm 2 ZerO Initialization on Convolution.
Input: number of input channels Cin, number of output channels Couts odd kernel size k. Return: Cout X Cin le x k convolutional kernel K.
Let n + [k/2]
If Coul = Cin: K [,, n, n + I
If Cont < Can: K
i, n.
If Cout >> Cin: K
1,,, n.
c C.'HH where m [Po] and c (mm-1)
We also apply Zer networks with or without batch normalization. For ResNet with batch normalization, follow the standard practice to initialize the scale and bias in batch normalization and zero,

---

Published in Transactions on Machine Learning Research (10/2022)
Dataset
Model
Initialization
Test Error (mean std)
CIFAR-10 ResNet-18
ZerO Init
5.13 + 0.08
Kaiming Init
5.15 + 0.13
Xavier Init
5.23 + 0.16
ImageNet
ResNet-50
ZerO Init
23.43 = 0.04
Kaiming Init
23.46 + 0.07
Xavier Init
23.65 + 0.11
Table 2: Benchmarking ZerO on CIFAR-10 and ImageNet. random seeds.
We repeat each run 10 times with different
respectively. For training without batch normalization, we adopt a technique proposed by Zhang et al. (2019), where the batch normalization replaced by learnable scalar multipliers and biases.
4 Experiments
In this section, we empirically benchmark ZerO on CIFAR-I0 and ImageNet datasets, where we evaluate ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet (Krizhevsky, 2009; Deng al., 2009). Both ResNet structures follow the design from He et al. (2016), which includes batch normalization by default.
Hyperparameter settings. We find that ZerO can fully utilize the default hyperparameters, which include learning rate 0.1, a momentum of 0.9, and a weight decay 0.0001. addition, we observe the learning rate warmup essential for ZerO to achieve large maximal learning rate, most of the weights start from the exact zero. We warm up the learning rate with 5 and 10 epochs for ImageNet and CIFAR-10, respectively.
We present our main results that compare different initialization schemes. For each dataset, all experiments use the same hyperparameter settings by default. Each experiment is repeated for ten runs with different random seeds. As shown in Table 2, ZerO achioves state-of-the-art accuracy on both datasets compared to other random methods.
In addition, compare ZerO with broad range of related works on CIFAR-10 using NesNet-18 and ResNet- 50, including ReZerO (Bachlechner et al., 2020), Fixup (Zhang et al., 2019), Skiplnit (De & Smith, 2020) and ConstNet (Blumenfeld al., 2020). As shown Table ZerO consistently achieves top performance compared to other methods.
We note that the ConstNet proposed by Blumenfeld et al. (2020) is also a deterministic initialization. However, unlike ZerO which preserves feature diversity, ConstNet is designed to eliminate the diversity by averaging the features through layers. The feature symmetric problem in ConstNet causes significant degradation, and additional random noise (e.g,, non-deterministic GPU operation and dropout) is needed to break the symmetry.
Method
ZerO
ReZero
Fixup
SkipInit
ConstNet
ConstNet
ResNet-18
5.13
5.20
5.17
5.26
72.39
5.41
ResNet-50
4.53
4.72
4.51
4.63
71.58
4.88
Table 3: Compare ZerO with other initialization methods on CIFAR-10. ConstNet* denotes ConstNet with non-deterministie GPU operations discussed in Blumenfeld et al. (2020). Top-1 test error is reported.

---

Published in Transactions on Machine Learning Research (10/2022)
Training ultra deep network without batch normalization Although there are methods attempting to train networks without batch normalization (by achieving dynamical isometry), they inevitably introduce random perturbations at initialization, affecting the training stability when the network sufficiently deep (Zhang et al., 2019; De & Smith, 2020). We benchmark ZerO with state-of-the-art methods training without batch normalization. As shown in Figure 4 (left), compared to other methods, ZerO achieves the best training stability for networks with even around 500 layers. also matches the baseline where batch normalization enabled.
Improved reproducibility. In addition, as shown in Table 2, ZerO achieves the lowest standard devi- ation the repeated runs. On ImageNet, the gap between ZerO and other methods is more than 40%. Thus, removing the randomeess in the weight initialization improves reproducibility, with possible implications for topics such as trustworthy machine learning and network interpretation.
Test Accuracy over 15 Epochs (%)
90
80
ZerO Init
70
Fixup Init
Rezero Init
Kaiming Init
Kaiming Init (BN)
50
40
100
200
300
400
500
Network Depth
Figure 4: Training extreme deep ResNet on CIFAR-10 over 15 epochs.
4.1 ZerO on Transformer
We also apply ZerO to Transformer and evaluate it on WikiText-2 dataset (Vaswani et al., 2017). In each Transformer layer, we use ZerO to initialize both multi-head attention and feed-forward layers. Because the embedding size is fixed in the multi-head attention, identity and the projection matrices of keys and values WK. Wy at zero. For the feod-forward layers, we initialize the connection matrices according to their hidden dimensions using Algorithm 1.
initialize the projection matrix of queries Wq
We train the Transformer models for 20 epochs with single learning rate decay epoch 10 2 We also vary the number of layers in the model from 2 to 20. As shown in Table 4, ZerO achieves similar performance compared to the standard initialization. In addition, it has better training stability over deeper Transformer models, which is consistent with our previous results on ResNet.
Number of layers
2
6
8
10
20
Standard
200.44
168.69
154.67
146.43
diverged
diverged
ZerO
192.34
169.73
151.91
149.27
145.62
141.81
Table 4: Evaluate Transformer on WikiText-2. We vary the number of layers in Transformer, where each layer consists multi-head attention and a feed-forward layer. Test perplexity is reported (lower is better).
We transformer architecture (provided by the link here) that smaller than the transformers typically used for this task, explaining the general degradation of the results.

---

Published in Transactions on Machine Learning Rescarch (10/2022)
300
ZerO Init
Kaiming Init (Random)
Stable Rank
30
Xavier Init (Random)
200
20
100
10
20
0
50
100
150
200
0
50
100
150
200
50
100
150
200
125
125
Stable Rank
50
100
100
40
75
75
50
50
25
25
10
0
0
20
40
60
80
0
20
40
60
80
20
40
60
80
Epochs
Epochs
Epochs
Figure 5: Low-rank training trajectories in ResNet-18 on CIPAR-10 (top row) and ResNet-50 on ImageNet (bottom row). We visualize trajectories the first convolutions in second, third, and fourth groups of residual blocks in ResNet.
5 Low-Rank Learning Trajectory
Although ZerO and random initialization achieve similar test accuracies, their training trajectories differ significantly. In contrast to random initialization, which begins optimization from a complex network (i.c., full-rank weight matrices, as shown in Figure 3), ZerO starts the training from a "simple" network and gradually increases its complexity.
To show the difference in practice, we track the ranks of convolutional kernels in ResNets during training, where the rank cach kernel can reflect its complexity. We measure the stable rank, which is defined as
/
Σ of (W) 1=1
for any matrix W with k singular values o,, The stable rank is soft version of the operator rank, and unlike the operator rank, it is insensitive to small singular values. We compare the stable ranks of various kernels between ZerO and random initialization during training. As shown Figure 5, contrast to random methods that begin with extremely high stable ranks, ZerO starts with low stable ranks and gradually increases them during training.
We believe ZerO's learning trajectory the first demonstration groedy low-rank learning (GLRL) large- scale deep learning applications. GLRL is theoretical characterization of gradient descent, such that: when matrices are initialized with infinitesimal values, gradient descent performs rank- mank-constrained optimization and greedily relaxes the rank restriction by one whenever it fails to reach a minimizer (Li et al., 2021; Razin et al., 2021).
For example, when matrix is initialized sufficiently small (where its rank is approximately zero), gradient descent first searches the solution over all rank-one matrices. it fails to find a minimizer, it will relax the rank constraint by one and search again over all rank-two matrices. The search is stopped rank-n if it finds a minimizer among all rank-n matrices.
10

---

Published in Transactions on Machine Learning Research (10/2022)
GLRL suggests that gradient descent implicitly biases the model towards simple solutions by searching through the solution space an incremental order of the matrix rank. This helps to explain the excellent gencralization gradient-based deep learning, as it converges global (or local) minima with the minimum rank.
Although the previous works have proved the existence of GLRL trajectory, it has never been observed in practice due to its impractical requirement of infinitesimal initialization. ZerO's learning trajectory we observed suggests that GLRL not only exists under infinitesimal initialization, but also under initialization around the identity. If matrix W is initialized as I, the low-rank structure is actually inside its residual component: W = Wi- W I. To be noted, every convolutional kernel conv(r) we measured Figure can be viewed the residual component of conv(x)
where the skip connection is included.
Figure also suggests that the kernels never reach their maximal stable ranks during training under ZerO initialization. This implies that the searching over the space of full-rank weight matrices may be unnecessary, suggesting avenues towards improved computational efficiency. We hope to explore this direction in future work.
We also observe that ZerO-based networks converge to low-complexity solutions. As shown both Figure and (left), the convolutional kernels trained by ZerO usually have lower ranks than the kernels trained by random initialization. We further measure model complexity through both network pruning and low-rank approximation.
For network pruning, lowest magnitudes in each layer (Frankle & Carbin, 2019). For low-rank approximation, we apply Tucker-2 decomposition over channel dimensions in convolutions to select the most significant components (Kim et al., 2016).
standard magnitude-based pruning that prunes portion of weights with the
As shown in Figure 6, compared to randomly initialized networks, the sub-networks obtained in trained ZerO-initialized networks achieve 25% higher sparsity or 30% lower (matrix or tensor) rank without sacri- ficing accuracy. This suggests ZerO encourages the networks to converge to low-complexity solutions, which improves the computational efficiency for inference.
Rank Trained ResNet-18
Network Pruning
Low-Rank Approximation
ZerO
Kalming
80
Kavier Ini (Random)
80
Kernel Rank
40
Test Accuracy
70
60
60
30
50
40
20
40
10
20
30
20
5
10
15
90%
60%
50%
60
50
40
30
20
10
Order of Conv Layers
Percent of Weights Remaining
Rank
Figure 6: Left: comparing kernel rank in ResNet-18 18 trained by ZerO and Kaiming methods. Middle: magnitude- based network pruning ResNet-18. Right: Tucker-2 decomposition for particular convolution with 512 channels in ResNet-18.
6 Related Works
To ensure stable training with random weight initialization, previous works such as Glorot & Bengio (2010); He et al. (2015) study the propagation of variance in the forward and backward pass under different acti- vations. Several studies provide more detailed characterization of the signal propagation with dynamical isometry (Saxe et al., 2014; Pennington et al., 2017; Xiao et al., 2018)
Inspired by the dynamical isometry property, various initialization methods are proposed to increase the convergence speed and stabilize the signal propagation, including Saxe et al. (2014); Bachlechner et al.

---

Published in Transactions on Machine Learning Research (10/2022)
(2020); Gehring et al. (2017); Balduzzi et al. (2017). De & Smith (2020); Hoffer et al. (2018) study the reason behind the success of batch normalization (loffe & Szegody, 2015) and Zhang et al. (2019); De & Smith (2020) propose initialization methods to train residual networks without batch normalization.
As summarized in Table 1, many of these methods can be categorized as identity initialization with random perturbations. However, ZerO eliminates all the randomness using Hadamard and identity transforms. In another related work, Blumenfeld et al. (2020) discusses whether random initialization needed from the perspective of feature diversity. They propose networks with identical features at initialization, which still require random perturbations to avoid the symmetric problem and improve performance.
Gradient descent biasing models towards low-rank solutions has been well studied in matrix factorization (Arora et al., 2019). Recent works also demonstrate the existence of greedy low-rank learning trajectory induced by gradient descent (Li et al., 2021; Razin et al., 2021; Jacot et al., 2021). However, no prior work demonstrates the greedy low-rank learning trajectory in large-scale applications of deep learning, as most only consider the problems of matrix factorization or applications of shallow neural networks.
7 Conclusion
In this work, we propose a simple and fully deterministic initialization called ZerO. Extensive experiments demonstrate that ZerO achieves state-of-the-art performance, suggesting that random weight initialization may not be necessary for initializing deep neural networks. ZerO has many benefits, such as training ultra deep networks (without notealioomallzation)), exhibiting low-rank learning trajectories that result in low- rank and sparse solutions, and improving training reproducibility.
We believe that ZerO opens many new possibilities given its various benefits. It can be applied to networks and tasks sensitive to the variances in weight initialization. Its low-rank learning trajectories enable the development of rank-conste conttrinst training methods that improve computational efficiency. Finally, the improved training reproducibility can aid model interpretability. We hope results will inspire other researchers to consider deterministic initialization schemes and to rethink the role of weight initialization in training deep neural networks.
Acknowledgments
We are grateful to the anonymous reviewers for their helpful comments and NVIDIA for the computational support. Dr. Anandkumar is supported by the Bren chair professorship at Caltech.
References
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factor- ization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Voncou- ver, BC, Canada, pp. 7411-7422 2019. URL https: http:::/ppoceeddngs.neeeee www.inngg..nuussseeeeee pourips..ccpappr//00000 D0CT880Erc0rr000E
Abstract.htal html.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. ReZero is All You Need: Fast Convergence at Large Depth. ArXiv preprint, abs/2003.0088$ 2020. URL https: / arxiv. org/ hbsp//bbs/000.... 04887.
David Balduzzi, Marcus Frean, Lennox Leary, J. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 342 350. PMLR, 2017. URL http://proceedingg. mlr, press//70/balduzze
html.
Peter L. Bartlett, David P. Helmbold, and Philip M. Long, Gradient Descent with Identity Initializa- tion Efficiently Learns Positive-Defintee Linear Transformations by Deep Residual Networks. Neural
12

---

Published in Transactions on Machine Learning Research (10/2022)
Computation, (3) 502, 2019. ISSN 0899-7667, 1530-888X. doi: 10. / a 01164. URL
https://direct.mit mit. www..//........etttttttii.cceeee http://eeee/rrrrrree////////////
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: Is feature diversity necessary in deep neural network initialization? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 960-969. PMLR, 2020. URL http://proceedding mlr. press/veesvieee htt.v//1l/llleeleeeeecc html.
Soham De and Samuel L. Smith. Batch normalization biases rosidual blocks towards the iden- tity function in deep networks. In Hugo Larochelle, Mare' Aurclio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:://proceedingg. courips.cc/paper/// Aeadd-bbbttattttt.e html.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248 255. IEEE Computer Socicty, 2009. doi: 10.1109 2CO.CCVV..00090006000
URL
1V09/CVPR.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nnt, 2019. URL https: /ttp://///eeeeeeeeee///mee http:///pppp.eeeeee.eeet/rttttta
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1243 1252. PMLR, 2017. URL http: attp://pproceedings. mlr. http://///ggeerng/gggg
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Tch and Mike Titterington (ods.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 249-
256, Chia Laguna Resort, Sardinia, Italy, 2010. PMLR. URL https: https://procceddings .ll.. www.mml......p. glorot10a 0a html.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in Hour. Arxiv preprint, abs/ 0te//706.026677 2017. URL petts://arxtts /tt///aarvv..oaaaeeaaaa orgg/bs////aa 0267777777..7..
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.neg net, 2017. URL https://openreevev www..//ooooiddddyyooott.to
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenct classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034 IEEE Computer Socicty, 2015. doi: 10. N0.1109/ICCC.2001 22...123. URL https://doi. orgooo/////// 10. 1109/ICCV 2015 123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778 IEEE Computer Society, 2016. doi: /0.100//CVP.2206.000 URL attps://dox org./0.gg..... httg//..1199///FFF.. 10..1119//VPR.2100 2016.99.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference Newral Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018,
13

---

Published in Transactions on Machine Learning Research (10/2022)
Montréal, Canada, pp. 10067199001777077559997777666-61 Abstrccc..attttt.t.......aa html.
2018. URL.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, Prance, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 148-456. JMLR.org, 2015. URL http://proceedingg iereeeddiinsmmm.......mmmmm.emme mlr wwt.mmlrrppea... press/v37/ ioffe15.hta
Arthur Jacot, Frangois Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-Saddle Dy- namics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity. ArXiv preprint, abs/2116..150 15933, 2021. URL https://arxiv wwwp//oog///.o ab/////////0.../ 15933.
Yong-Deok Kim, Funhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In Yoshua Bengio and Yann LeCun (ods.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arre //axiv org/abs///1111 http//////1111115/131 06530.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreviee
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4785-4795, 2017. URL htttps//procceednng mew.eedddnnn..uuuss...... Abstract.
cc/ppp///2017 /hwsh//hhhh/a (0F.0000000000000111440000000000
Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Rescarch, pp. 8913 8924. PMLR, 2021. URL http: http://prooceedings mlr, press/v139/razinntee html.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 7, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: p/prcee/pso heg.oeeddnngs...urs...... neurips. cc/paper/ 2017 http//hhhhhhhatea /ttaa//ff.eeeeeeeeeeeeeee11aaaa. Absttraaaaaaaaaaat..a.aaaaaaaa.. .tml..
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schocnholz, and Jeffrey Pennington. Dy- namical isometry and mean field theory of cmns: How to train 10, 000-layer vanilla convolutional neural networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference on Mochine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. $389-5398, PMLR, 2018. URL http:/ http::///ooceengs....e. www.innnnm.......mmmmmm....ee wtt....reesv////// press//800xxao18/ xiao18a..a.1...1 html.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without nor- malization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9. 9, 2019. OpenReview.nee net, 2019. URL
14


Reviews:
[{"review_id": "Iqep27Juq5", "reviewer": "Reviewer_qqPa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+ Initialization is an important topic for neural networks training. \n+ A deterministic method can benefit reproducibility. \n+ The performance looks good on CIFAR-10 over 15 epochs for ResNet with more than 150 layers.\n+ The paper is generally well written and easy to follow.\n\nWeakness\n- The writing and experiments also make it not so easy to compare the proposed method ZerO with existing methods Rezero [Bachlechner 2020] and ConstNet [Blumenfeld 2020]. If I understand correctly, there are two changes: (1) use Hadamard transformation instead of random initialization for some layers (2) the additional residual connection (AugSkip). \n- The AugSkip architecture change looks ad-hoc. Could the authors provide more theoretical support or insights? Does ZerO work for common ResNet structures?\n- The experiments are only on ResNet for CIFAR-10 and ImageNet, and can be improved for clarity with more ablation study. \n- Some related works are not discussed. See below for details\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a deterministic initialization method for ResNet where weights are initialized with zero, identity or a Hadamard matrix depending on the type/location of the layer. The proposed ZerO initialization also used a variant of ResNet with augmented skip connections. Experiments performed on CIFAR-10 and ImageNet suggest ZerO init can be comparable with Kaiming and Xavier initialization while being more stable as a deterministic initialization. ZerO init also significantly outperforms Fixup, Rezero and Kaiming init when evaluated on CIFAR-10 over 15 epochs for ResNet with more than 150 layers. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Is it possible to provide more ablation study/baseline methods in table 2? ZerO is only compared to Kaiming and Xavier init now. If I understand correctly, ZerO shares a lot of common parts such as the zero and identity initialization with Rezero and ConstNet, so it can be beneficial to compare these methods to show the effect of the proposed change: use Hadamard transformation for initialization. \n\nCould the authors discuss the motivation of the AugSkip architecture changes in Figure 4, and whether ZerO works for common ResNet structure?\n\nI have some concerns about Figure 6 for the evaluation on CIFAR-10 over 15 epochs for ResNet with various depths. How is the number of “epoch 15” chosen? Fixup [Zhang 2019] shows a figure for epoch 1 and the accuracy drop is very small up to depth 1000. Both the original ResNet [He 2015] and GradInit [Zhu 2021] train ResNet-1202 and achieve comparable or even better performance than shallower ResNets. \n\nI would encourage the authors to discuss [Dauphin 2019 MetaInit: Initializing learning by learning to initialize] and [Zhu 2021 GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training]. Additionally and optionally, [Gilmer 2021 A Loss Curvature Perspective on Training Instability in Deep Learning] suggests successful initialization (Fixup, MetaInit, GradInit, Warmup) can change the loss landscape. It might be good to study the effect of the proposed ZerO initialization. \n", "broader_impact_concerns": "No concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eyKOQwEStk", "reviewer": "Reviewer_ZBoj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although there are a large amount of study on the initialization of DNNs, I do think this work provide an interesting option to approach the problem. However, I have several concerns on the novelty and significance of the work.\nFirstly, I think there's lack of motivation behind having a fully deterministic initialization method as opposed to a random initial point. The final model is still not deterministic with the stochastic mini-batch training.\n\nAlso, it is not clear why the authors have chosen the Hadamard matrix. I might be missing something but after reading the paper I still don't get how Hadamard matrix is different from any other orthogonal or simply full-rank matrix in this case?\nIn addition, the singular value of Hadamard matrix is linear to $\\sqrt{n}$, wouldn't this lead to exploding gradients at the beginning of training?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors proposed an initialization scheme for deep neural networks called ZerO. The proposed method use the product of identity and Hadamard matrix as initial weight for training. Theoretical analysis show that compared with identity initialization, the model trained with ZerO initialization does not have rank constrained by input dimension. Experiments with ResNet on image classification tasks show the ZerO initialization achieves similar results as other initialization methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I'd like to see more discussion on the motivation behind the method design, e.g. what are the considerations and what are the gains.\n\nHow does the ZerO perform on vanilla ResNet (without AugSkip)? It seems to be missing from Table 2.\n\nThere's only empirical study on ResNet for image classification tasks, more empirical study on more types of DNNs and problems (e.x. Transformers on NLP tasks) is needed to fully understand the effect of ZerO initialization.\n", "broader_impact_concerns": "To my knowledge this work does not pose any negative societal impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AZiRUJFJFg", "reviewer": "Reviewer_8dKV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is easy to follow and the proposed method is novel. \n2. The empirical results are interesting, where one can achieve comparable results when we have all initial values in ones and zeros. \n3. The empirical findings that ZerO initialization leads to low-rank and sparse solutions are interesting.\n\nWeakness:\n1. I am concerned about the title of the paper since they only show that their method can work on a modified ResNet-18, which is not a general initialization method for other neural networks. Even for residual networks, they need to add one more skip connection. But the title looks like the proposed method works for all types of networks as an initialization method. Can we apply the proposed method to other models? Moreover, can we apply it to other tasks, especially for the language model? \n\n2. Some statements are overclaimed: 1) The proposed can help train neural networks without BatchNorm. This is not a unique property since other methods can help as well.  2) They claim their method achieves SOTA results. However, they only compared with some basic initialization methods such as Xavier Init and Kaiming Init. However, there're more initialization methods such as FixUp[1], MetaInit[2], GradInit[3], and so on. How is the performance compared to these methods?\n\n3. Why deterministic? I agree that deterministic initialization makes the standard error smaller, but is this property that important? In addition, is this network structure and optimization algorithm dependent? Does this property still hold with other optimizers such as adam or sam? \n\n4. Is there a chance that we can apply the proposed method with other pruning techniques to achieve better empirical results for sparse networks?\n\n6. How to initialize convolutional layers is not clear to me. Do we treat each kernel as a matrix or the whole conv block as a huge matrix? I notice that some of the convolutional layers are initialized to $HI^{\\ast}$ and $I$, and some of them are to zero. Is that heuristic? Can we have some ablation study to initialize more convolutional layers to $HI^{\\ast}$? I am just curious what will happen if we do not have skip aug or the network does not have residual blocks at all.\n\n[1] Zhang H, Dauphin Y N, Ma T. Fixup initialization: Residual learning without normalization[J]. arXiv preprint arXiv:1901.09321, 2019.\n[2] Dauphin Y N, Schoenholz S. MetaInit: Initializing learning by learning to initialize[J]. Advances in Neural Information Processing Systems, 2019, 32.\n[3] Zhu C, Ni R, Xu Z, et al. GradInit: Learning to initialize neural networks for stable and efficient training[J]. Advances in Neural Information Processing Systems, 2021, 34: 16410-16422.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new deterministic initialization method for residual networks, where all the initial weight values are ones and zeros. They theoretically show that, different from naive identity mapping, their initialization methods can avoid training degeneracy when the network dimension increases. In addition, they empirically show that they can achieve better performance than random initializations on image classification tasks, such as CIFAR-10 and ImageNet. They also show some nice properties of the model trained by their initialization methods, such as low-rank and sparse solutions.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The main issue to me is that the proposed method is not an initialization method. It can only be applied to a specific network. And it is unknown whether it will work for other networks.\n\n2. We need more experiments to support some of the claims. For example, compared to more initialization methods and with different optimizers or network structures.\n\n3. Captions for Figures 5, and 7 are not informative. What is 1*1*3*5? The kernel size is 1? I cannot tell which figure is for which one in Figure 7.\n\n4. I think making the proposed method work for other types of networks is essential.", "broader_impact_concerns": "I don't have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel initialization scheme for deep residual networks called ZerO. The proposed method relies on the use of Hadamard and zero matrices and they show theoretically that this scheme avoids training degeneracy. The reviewers raised issues with the lack of baselines, which were addressed by the authors in the rebuttal phase with Table 3. There were also concerns about the generality of this method since it applied only to modified resnets. This concern was also addressed in revisions by the authors. Experiments on Imagenet and WikiText-2 show that this method can be effective to train very deep networks.
Decision: Accept (poster)