ZerO Initialization: Initializing Residual Networks with only Zeros and Ones
Authors: Jiawei Zhao
Florian Tobias Schaefer
Anima Anandkumar
Venue: ICLR 2022 Submitted (2022)
Keywords: weight initialization
deep residual network
deterministic initialization
optimization

Abstract: Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, there is no consensus on how to select the variance, and this becomes challenging especially as the number of layers grows. In this work, we replace the widely used random weight initialization with a fully deterministic initialization scheme ZerO, which initializes residual networks with only zeros and ones. By augmenting the standard ResNet architectures with a few extra skip connections and Hadamard transforms, ZerO allows us to start the training from zeros and ones entirely. This has many benefits such as improving reproducibility (by reducing the variance over different experimental runs) and allowing network training without batch normalization. Surprisingly, we find that ZerO achieves state-of-the-art performance over various image classification datasets, including ImageNet, which suggests random weights may be unnecessary for modern network initialization.

Under review as a conference paper at ICLR 2022
ZERO INITIALIZATION:
INITIALIZING RESIDUAL NETWORKS WITH ONLY ZEROS AND ONES
Anonymous authors Paper under double-biee blind review
ABSTRACT
Deep neural networks are usually initialized with random weights, with ade- quately selected initial variance to ensure stable signal propagation during train- ing. However, there no consensus on how to select the variance, and this be- comes challenging especially as the number of layers grows. In this work, we replace the widely used random weight initialization with a fully deterministic ini- tialization scheme ZerO, which initializes residual networks with only zeros and ones. By augmenting the standard ResNet architectures with a few extra skip connections and Hadamard transforms, ZerO allows to start the training from zeros and ones entirely. This has many benefits such as improving reproducibility (by reducing the variance over different experimental runs) and allowing network training without batch normalization. Surprisingly, we find that ZerO achieves sfte-offthhe-r performance over various image classification datasets, including ImageNet, which suggests random weights may be unnecessary for modern net- work initialization
INTRODUCTION
An important question in training deep neural networks is how to initialize the weights. When Rumelhart et al.] first introduced backward propagation, they initialized the weights randomly to break symmetries among different parameters and activations. Currently, random weight initializa- tion is the de-facto practice across all architectures and tasks.
However, choosing the variance of the initial distribution or equivalently the scale of the initial weights a delicate balance when training deep neural networks. Too large a scale can lead to excessive amplification of the activations propagating through the network, resulting in exploding gradients during training. On the other hand, if the weights are initialized too small the activations may not propagate all, resulting in vanishing gradients, which is especially an issue the depth of the network grows.
He al. Glorot and Bengio study the propagation of variance in the forward and backward passes at initialization, using it as a proxy for the magnitudes of the gradients. By choosing seale parameters that depend the width of the neural networks, upper bound the variance, they design initialization schemes that avoid exploding gradients. However, this still does not fully address the issue of vanishing gradients because it is not easy to estimate the lower bounds on the variance.
In order to design networks robust to vanishing gradients at initialization, He et al. (b) introduced skip connections that take the activations of a layer, and add them the next layer. Given enough skip connections, the activations can, in principle, always be propagated through the entire network. Thus, skip connections allow us to mitigate the vanishing gradient problem and possibly avoid completely.
Zhang al. (a): Yang and Schoenholz observe that initializing the residual branches (i.e., the branch that is nor the skip connection) at (or close to) zero benefits signal propagation and optimization. To achieve this, Zhang et al. aa) initialize the last layer of each residual branch zero, which
We share our code via anonymous link here

---

Under review as a conference paper at ICLR 2022
also shown stabilize training without batch normalization. However, previous studies rely random weights to propagate signals to layers without skip connections, which still requires the consideration of the weight variance at initialization.
Our approach: In this work, we show that all randomness in the weight initialization can be removed, resulting in fully deterministic initialization scheme ZerO, which initializes residual networks with only zeros and ones. As illustrated in Figure [ ZerO achieves by adding extra skip connections to enable non-zero signal propagation throughout the network. Additionally, applies the Hadamard transform to avoid the training degeneracies in the latent space when there is width expansion across layers. Moreover, our approach does add any new learnable parameters the standard architectures and does not involve any significant computational overhead. We find that ZerO achieves state-of-the-ar performance over various image classification tasks.
Our contributions are summarized follows:
We carefully identify the training difficulties that occur when initializing residual networks to zero, including 1) the dead neuron layer, and 2) the degeneracies due to width expansion-addees transform.
by employing skip connections at each
by applying the Hadamard
2. We propose ZerO initialization for ResNet as an example to show how skip connections enable deterministic initialization. Our experiments show that ResNet with ZerO achieves state-of-the- art results over various image classification tasks
We find ZerO works well without batch normalization during training. With proper regular- ization, ZerO beats random initialization methods such Kaiming initialization while almost matching the batch normalization baseline.
We observe that compared random initialization methods, the deterministic ZerO initialization achieves 20%-40% lower standard deviation over repeated experiments, and thus ZerO enjoys the advantage of high reproducibility.
conn 3xd
3x3
Hadamard
conV 3x3
Hadkemand
cunvitd 3xd
comvixt
1x1
RodNet
layer
Basic black
Bottleneck bluck
original connections our siip
initialize partial-ddentty deterministic
Figure I: ZerO initialization for ResNet. We add extra skip connections and modify their locations in the standard ResNet. We also apply Hadamard transforms (defined at Definition when there is an expansion in the channel dimension. We omit Relu and batch normalization here, see Figure for detailed design. partial-identity 1x1 convolutional kernel defined Equation
WHEN DOES A ZERO-INITIALIZED NEURAL NETWORK WORK?
In this section, we discuss when zero-initialized neural network works by identifying the training difficulties that occur when initializing the weights to We also propose adequate methodologies addressing these problems in order to achieve zero initialization without damaging the performance.
We use the fully connected neural network
proxy to illustrate the difficulties and our idea. Consider a problem of learning a function ş = F(x) with parameters W by minimizing the
squared error L b y ú g- (æ. NNN, represent the sample data, and the learning
2

---

Under review as a conference paper at ICLR 2022
is accomplished via gradient descent. F(x) is a fully connected neural network with L layers: F(x)) = op (WZZ) ) where is composition operator, W E PR for and to is nonlinearity. We ignore bias terms, and denote x' p( x'ー1) as the activations of the /th layer. We also define W and W to be ith row and jth column of the matrix W
I E 1, 1,... L,
2.1 SKIP CONNECTIONS SOLVE DEAD NEURON PROBLEM
For simplicity, we first assume that F(x) is linear network with fixed dimension, such that N₂ = N₂ = pi = Q' for any I, and ip is an identity mapping. In this case, if we initialize the weights zero, every layer will output zero constantly no matter what the imputs are. On the other hand, every W receives zero derivatives in the backward pass, and thus there is no weight that can move out of the zero during the entire training. This extreme case of the vanishing gradient problem, well-known under the name dead neuron problem, is caused by zero-initialized F(æ) only generating zero signals at initialization.
Skip connections help propagate signals through the network and thus mitigate the vanishing gra- dient problem. A natural approach towards practical zero initialization is therefore to equip every layers of the network with skip connections. This ensures that the signal is propagated even with zero weights, with each layer applying a simple identity function to the activations. For example, equipping each layer of F(x) with a skip connection, we obtain
F(x) (W (W. + I)x.
The derivatives with respect to weight matrices are then given as dc
aw (W + I)... (W2 I)) (y - 0) xT,
OC
OW! = (WWL, + I)... (W + +1) (y- - ((W + +I)... (W + x)) forl E 2,... L -1,
3c
OW.. (y 9) ((WL-1+1) I). (w' + x))²
As zero-initialized weights initialize F(x) as an identity mapping, the loss C at initialization the error when directly using the inputs as the predicted outputs á x. Therefore, the derivatives at initialization uniformly become 2WT
ac = (y x) x for every W This is because the skip connections are applied to every layer, and thus the error derivatives flow back to every matrix equivalently. With the help of the skip connections, each weight matrix W receives non-zero derivatives, which break the dead neuron problem for the entire network.
This also works for residual networks with nonlincarity, as long as the derivative of the nonlinearity exists zero. Although there exists nonlinearity that is non-differentiable at such as Relu (Xu et al.),, this can be solved easily by utilizing its subderivatives at zero. For example, we define the derivative of Relu be its subderivative one at zero, which allows the signals be identically prop- agated through the Relu. We will apply this special modification to Relu in our later experiments.
We note that in our example above, the first iteration, all weight matrices receive the same deriva- tives. However, as training progresses, the gradient that weight matrix receives depends on its position in the network, and thus the full capacity of the network utilized.
2.2 WIDTH EXPANSION LEADS TO TRAINING DEGENERACY
For the deep residual networks discussed above, the dimensions of all hidden layers are equal to the input and output dimensions. However, for modern residual networks, the hidden dimensions are usually varying and larger than the input dimension, such as the large channel dimension in intermediate layers of ResNet. We find that for these networks, directly initializing weights to zero leads to another kind of dead neuron problem, which can not be avoided even with skip connections. To distinguish it from the previous dead neuron problem, we will refer to this problem as training degeneracy.
To simplify the presentation, we explain this phenomenon in the setting where the nonlinearities applied to the outputs of skip connections from previous layers. However, equivalent phe- nomenon is present in the case where skip connections skip the nonlinearity as well, which is the variant we using in Section
3

---

Under review as a conference paper at ICLR 2022
signals
2
Wil:
W
210
Wf, 20
1/
Fx Relu(fa))
H Relu(H r'x)
Figure 2: Left: the forward dynamic of the zero-initialized example network at initialization. Solid and dashed blue lines represent non-zero signals. Right: we represent the signals in the first layer at initialization on standard 2-dimensional basis. Dashed lines represent the signals before the Relu and solid lines represent the signals after the Relu, which are activations x!.
To understand the cause of this degeneracy, we consider simple 2-layer residual network: F(æ)) Relu(W2 + I?) o Relu(W' + I') ) where W I'' E IX2 and W _2 E R2x2 N₂ and N2 changed and 2 accordingly, and we use Relu the activation. Because the dimension is increased in the network, we apply zero padding to the skip connection I to match the dimension, where %0.0 and 11,0 0. This duplicates the previous activations while padding additional dimensions using zero, which is a standard parameter-free method applied in ResNet (He et al. c).
For such network, there exists a training degeneracy when directly initializing all the weights to zero. As illustrated in Figure 2] at initialization, the second dimension of the first layer activations zi receives zero signal padded by I', and it propagates the zero signal ğ as well. During backpropagation, aw;, and Oc 8W;,1 are zero because i Also, aW; zero as receives the
ac
ac
zero derivative from ūz]
ac Because x and ac keep receiving the zero signal, W W6, and W?, dul bound to be zero for the entirety of training, restricting the expressivity of the network.
2.3 HADAMARD TRANSFORM AVOIDS TRAINING DEGENERACY
The reason for the degeneracy is that the skip connection I with zero padding is only able to map the input æ into low-dimensional subspace of at. As illustrated in Figure 2 when initializing all the weights to zero, I' maps the single dimension to horizontal line (dashed line) in the two- dimensional space. Because this subspace aligned with the standard basis, its linear dimension invariant under application of the Relu, any component-wise nonlinearity. As the activations x. stay in a I-dimensional subspace span, its associated derivatives stay in such a subspace as well. This eventually causes the zero- zero-initialin inrtiiiiilllzed weights W to be updated into low-dimensional space, as they solely depend on the low-dimensional derivatives.
To ensure that the first hidden space receives high-dimensional activations æ we propose to apply a Hadamard transform H after F1 spread the information into a new space.
The Hadamard transform is an example of the generalized family of Fourier transforms that performs an orthogonal linear operation on 2nd real numbers (Pratt et al.). consists of a Hadamard matrix and normalization factor. A Hadamard matrix is defined as follows:
Definition 1 (Hadamard matrix). For any Hadamard matrix Hm E x22""x2" where m is a positive integer, we define Ho = 1 by the identity, and the matrix with large m is defined recursively:
Hm
Hm-1 Hm-1 Hm-1 - -H
1
R x2
We rescale the Hadamard matrix by (m-1)/ resulting in the orthonormal Hadamard transform. The Hadamard transform rotates the standard basis into new basis, such that each element of the new basis is equally weakly aligned with every element of the standard basis. For example, in wwo-dimensional space, the Hadamard transform rotates the standard basis by an angle of 45 degree, as illustrated in Figure 2
We a cquivalent using a standard Hadamard matrix with order-reversed I' without affecting our conclusions.
Hadamard matrix in the example to simplify the presentation. This is
4

---

Under review as a conference paper at ICLR 2022
ZerO Init
12
ZerO Init
Kaver Int
(no transfar))
Kaiming Init
10
Kaiming Irit
Eachorard
48
-loy(C)
pla)
0.0
-0.2
-0.1
0.0
0.1
0.2
0
2500
5000
7500
10000
12500
Iterations
Figure 3: Left: final weight distributions. Right: weight correlations during training. Each setting achieves 98% test accuracy on MNIST dataset except for ZerO Init (no transform).
When applying component-wise nonlinearities to the spaces spanned by the elements this basis, their linear dimension increases. In our example, after applying the Hadamard matrix H I'x forms new space that is not aligned with the standard basis. Under the application of the Relu, the new space (i.e., H x) is spread into a 2-dimensional span in x', which ensures that neither x nor i stay at zero forever. Therefore, with the Hadamard transform, the activation æ can propagate the non-zero signals to the next layer. This ensures the derivatives of WP, and W be non-zero, and thus breaks the training degeneracy.
2.4 MEASURING THE WEIGHT DISTRIBUTIONS AND TRAINING DEGENERACY
To verify the phenomenon explained above empirically, we train a fully connected residual network on the MNIST dataset to check the existence of the degeneracy and measure the weight distribution (Lecun et al.). Our network has three layers with hidden dimensions larger than the input dimension. The details of our setting introduced in the appendix.
We use empirical weight correlations as proxy to verify the existence the degeneracy, which is similar the measurement conducted by Blumenfeld et al. High weight correlations indicate that degeneracy may exist. For each weight matrix, we measure the correlations between row vectors and column vectors, respectively. For example, given M x N weight matrix W. we define forward correlations (between rows) Cj and backward correlations (between columns) C as follows:
M
M
W W
C M(M - 1)
Wa,lle l|Wlll22'
N
N
C. N(N - 1)
Σ
Σ
W W.
W. W. W W
where A - B denotes a dot product of vectors. We measure the weight correlations for the zero- initialized network with without the Hadamard transform, and compare them with popular initial- ization methods proposed by Glorot and Bengio: He et al. (a). As shown in Figure 3 the Hadamard transform largely decouples the weight correlations compared to the network without the transform, which suggests that the Hadamard transform avoids the problem of training degeneracy.
In addition, we measure the final weight distributions over various initialization methods. As shown in Figure [3] the weight variance generated by our initialization is significantly lower than the vari- ances generated by randomized methods. We believe that the fact that most of our weights are close to zero may help to train sparse neural networks with techniques such weight gruning(Han al.)
3 ZERO INITIALIZATION ON CONVOLUTIONAL RESIDUAL NETWORKS
In the last section, we found that training zero-initialized fully connected neural network is possible when the network structure satisfies two design principles: 1) the skip connection associated to every fully connected layer and the Hadamard transform is applied to every increasing dimennsion skip connection. We now propose ZerO by applying these design principles to the well-developed ResNet architectures.
5

---

Under review as a conference paper at ICLR 2022
It is natural direction to apply our proposed design principles modern convolutional neural net- works such as ResNet, as they usually have skip connections between residual blocks to allow the signals easily propagate through the network. However, modern convolutional networks require more consideration than simple plain networks because they have more complex designs and op- erations, such dimension-variee batch normalization (He et al. Ø Ioffe and Szegedy). To initialize ResNet deterministically, we propose ZerO that fully addresses the difficulties above by applying the following four steps.
residual block (e.g., bottleneck block), pooling operations, and
ZerO Initialization for ResNet
1. Add additional residual connections.
2. Apply Hadamard transform to dimension-increasing, residual connections.
3. Initialize residual blocks to zero and one.
4. Initialize both the first convolutional layer and last classification layer zero.
We now explain these steps in more detail.
We first focus on the design of the residual blocks, which is the core component of ResNet archi- tectures. We adopt the pre-activation block design proposed by He et al. (b). where the identity mappings (without nonlinearity) are applied as the skip connections. We first consider modifying the basic block by using the skip connections to avoid the dead neuron problem. As the channel dimension is fixed within each basic block, we can directly apply identity mappings to every convo- lutional block, which contains a convolutional layer, Relu activation, and batch normalization. This is illustrated in Figure
Double-residual bottleneck block. The bottleneck block proposed by He et al. ((c) used to re- duce the computational complexity when training deep models on large-scale datasets. The block consists of two Ixl and one 3x3 convolutions where Ixl convolutions are used to reduce and in- varied
crease the input and output dimensions of the 3x3 convolution. Because the dimensions within each bottleneck block, applying skip connection to every convolution leads to information loss. This is because the skip connection only can propagate part of activations when applying to layer that reduces the channel dimension.
To address this problem, we only apply the skip connections between layers that share the same dimension. As illustrated in Figure [ we keep the original skip connection between the first and the last Ixl convolutions and only add another skip connection the 3x3 one. This design can be viewed as building a residual layer inside another residual layer, which does not cause the infor- mation loss as we mentioned above. However, directly initializing all convolutions to zero is not applicable, as there is no skip connection between the first Ixl convolution and the 3x3 convolution.
Partial-identity initialization. To address this, we initialize each 1x1 convolution as a partial identity matrix, as it can be viewed as linear projection over the channel dimension. For I x Cin X Cout convolutional kernel K, where Cin is the number of input channels and Cout is the number of output channels. Initializing K equivalent initializing a Gn X Cout matrix K. As Can and Cout may be different, we initialize the largest square matrix in K as identity matrix and initialize the rest components to zero. This is illustrated in Figure 4 Specifically, let Cmin be the minimum of cin and Conts the matrix λ is initialized follows:
K,s for < S Cmin
1,
(I)
R1 = for 0 < E Gin j Cout - 1, and *
When Cin < Couts this essentially initializes the linear projection as an identity mapping with zero padding. This allows signals to be propagated through the layer while initializing the weights deter- ministically. Therefore, by initializing the first 1x1 convolution as a partial-identity matrix, the rest convolutions can be initialized to without damaging the signal propagation.
Hadamard transform for Clmeesioon-iccresiinn skip connections. Because the channel dimen- sion is increased gradually in ResNet, there various types of dimenssinninnceassing skip connec-

---

Under review as a conference paper at ICLR 2022
multipler
initialize
BN
3x3 corv
Initializn one
Relu
bies
0
0
0
Relu
0
with
without
1x1x385 partial-identity convolutional kernel
batch normalization
batch normalization
Figure 4: Left: associated operations for each convolutional layer. Right: an illustration of 1xlx3x5 partial-ideniity convolutional kernel.
tions developed, including simple identity mapping with zero padding and linear projection using 1x1 convolution. Although directly initializing 1xl convolution to zero leads to zero signals, we can follow our previous design to initialize it as a partial-identigg matrix, which also forms it as an identity mapping with zero padding. However, as discussed in Section there a training degen- eracy when applying the zero padding operation during the width expansion. Therefore, we apply Hadamard transform for every dimension-increasing skip connection to solve the degeneracy, as illustrated in Figure
In addition, depending on the channel dimension, we may apply a dimension-increasing skip con- nection and Hadamard transform for the first convolution of the ResNet. For the last fully con- nected layer, we can safely initialize to zero without any modification, as receives both non-zero inputs and derivatives initialization. For batch normalization layers, initialize the scale one and the bias as zero. For other operations ResNet, such as average or max pooling, we keep them by default they do not require any learnable parameters.
We note that applying Hadamard transforms does not introduce a significant overhead as the trans- form is computationally efficient. As defined at Definition [ Hadamard matrix only consists of -1 and 1. Therefore, except for the scaling operation, it does not require any multiplication as sign flips are all takes. It can be implemented efficiently with only (nnoog(n)) complexity (Fino and Algazi).
Training without batch normalization. As motivated the introduction, ZerO may help train neural networks without batch normalization, because both ZerO and batch normalization bias the residual network as an identity mapping at initialization. Therefore, we propose removing the batch normalization using learnable scalar multipliers and biases, which follows the design in (De and Smith Zhang et al. a). This illustrated Figure
EXPERIMENTS
In this section, empirically evaluate ZerO CIFAR-10 and ImageNet datasets, and also benchmark under normalization- settings. We evaluate ResNet-18 18 on CIFAR-10 and ResNet- 50 ImageNet (Krizhevsky) Deng al.). Both ResNet structures follow the design from He al. (c), which includes batch normalization by default.
Hyperparameeer settings. We find that ZerO can fully utilize the default hyperparameterss which include learning rate of 0.1, a momentum of 0.9, and a weight decay of 0.0001. In addition, we observe the learning rate warmup essential for ZerO to achieve large maximal learning rate, as most of the weights start from the exact zero. We warm up the learning rate with 5 and 10 epochs for ImageNet and CIFAR-10, respectively.
Fast convergence speed. We observe that ZerO achieves faster convergence speed than standard random initialization methods. As shown in Figure 5 at the initial training stage, ZerO converges faster than Xavier and Kaiming initialization when training ResNet-50 ImageNet. This is because

---

Under review as a conference paper at ICLR 2022
ZerO forms ResNet an identity mapping initialization, which is an important property to achieve optimal signal propagation (Yang and Schoenholz).
We present main results that compare architectures and initialization schemes on CIFAR- 10 and ImageNet. For each dataset, all experiments use the same hyperpa- rameter settings by default. Each experiment is repeated for five runs with different random seeds. We denote our ResNet with more skip connections as ResNet (AugSkip). As shown in Table ZerO achieves sfute-of-the-ard accu- racy on both datasets. Although we observe that ZerO is slightly worse than standard ResNet with Kaiming initial- ization by 0.1 0.,% we note that the degradation is induced by the differences in architectures instead of ZerO initial- ization itself. When comparing initialization schemes un- der the same ResNet (AugSkip). ZerO matches or even beats other random initialization methods.
ZerO Init
Kaiming Init
5.0
Xavier
Training Error
4.5
1.0
3.5
3.0
2.5
3
Epoch
Figure 5: Training error over the first 5 epochs on ImageNet.
Dataset
Model
Initialization
Test Error (mean + std)
CIFAR-10
R8sNet-118 (AugSkip)
ZerO Init (fully-deterministe)
5.26 + 0.10
ResNet-18
Kaiming Init
5.14 + 0.14
ResNet-18 (AugSkip)
Kaiming Init
5.27 + 0.13
ResNet-18
Xavier Init
5.20 + 0.14
ResNet-18 (AugSkip)
Xavier Init
5.28 + 0.17
ImageNet
ResNet-50 (AugSkip)
ZerO Init (fully-deterministe)
23.63 + 0.04
ResNet-50
Kaiming Init
23.46 + 0.07
ResNet-50 (AugSkip)
Kaiming Init
23.65 + 0.09
ResNet-50
Xavier Init
23.65 + 0.11
ResNet-50 (AugSkip)
Xavier Init
23.72 + 0.08
Table 1: Benchmarking ZerO on CIFAR-10 and ImageNet. ResNet-50AAugSkip) is our proposed network. We repeat each run 10 times with different random seeds.
Improved reproducibility. In addition, ZerO achieves the lowest standard deviation over the repeated runs. On ImageNet, the gap between ZerO and other methods is even more than 40%. Thus, removing the randomness in the weight initialization improves reproducibility, with possible implications for top- ies such as trustworthy machine learning.
4.1 TRAINING WITHOUT NORMALIZATION
We also evaluate the ability of ZerO to remove batch normalization during training. We use the residual blocks that replace batch normalization, as illustrated in Figure [[ We compare various initialization methods using ResNet-50 on ImageNet, including a recent normalization-fiee method called Fixup (Zhang et al., a). As the training without batch normalization usually suffers from overfitting, we apply stronger regularization technique Mixup Zhang et al. (b), which is the same as the training setting in (Zhang et al. a).. The Mixup coefficient is tuned for each setting, and we train ResNet-50 for 90 epochs. We find that training with ZerO is stable with default hyperparameters such as learning rate of 0.1. This is not always the case for training without normalization, as the training with Kaiming initialization fails when using 0.1 as the learning rate.
As shown in Table 2 without batch normalization, ZerO achieves significantly better results than standard random initialization such as Xavier and Kaiming. We observe that ZerO is slightly worse
8

---

Under review as a conference paper at ICLR 2022
than Fixup, which is likely due to the difference in the speed of convergence. When we train both models for 180 epochs, the gap largely reduced from 0.7% to 0.2%.
Method
Batch Normalization
Large Learning Rate
Test Error (mean + std)
Kaiming Init
23.46 + 0.07 23.16 + 0.09
Kaiming Init + Mixup
(Mixup enabled)
ZerO Init
X
24.52 + 0.06
Fixup Init
X
23.85 + 0.10
Kaiming Init
X
x
29.91 + 0.12
Xavier Init
X
25.89 + 0.11
Table 2: Benchmarking ResNet-50 without batch normalization on ImageNet
RELATED WORKS
Theoretical analysis deep networks ensure stable training with random weight initializa- tion. previous works such as Glorot and Bengio; He al. (a) study the propagation of variance in the forward and backward pass under different activations. For residual networks, by analyzing the optimization landscape of linear residual networks, HHartt and Ma suggests that all critical points in a neighborhood around zero are proved to be global minima, suggesting zero initialization should be a better choice from the optimization perspective.
Initialization for ResNet Various studies are proposed to address the initialization problem in
ResNet (Bachlechner et al.; /eehing et al., Balduzzi et al.). As the success of batch normalization (Toffe and Szegedy), De and Smith Hoffer et al. studied the effect of it, and Zhang et al. (a); De and Smithhsstdiie how to train residual networks without batch normalization. Also, Hardt and Ma Srivastava et al. Goyal et al. Zhang et al. (a): Bachlechner et al.) found that initializing the residual branches at (or close to) zero benefits the optimization but they still require random initialization. We take this idea to the extreme by initializing all the weights to zero and one.
In another related work. BBlumenfeld et al. discusses whether random initialization is needed from the perspective of feature diversity (Rahimi and Recht). They propose networks with identical fea- tures at initialization but they still require random noise to improve the performance.
CONCLUSION
In this work, we propose fully deterministic initialization ZerO by using skip connections and Hadamard transforms to modify the standard ResNet architecture. Extensive experiments demon- strate that ZerO achieves state-of-the-art performance, suggesting that random weight initialization may not be necessary for modern network initialization.
Our deterministic ZerO initialization opens up many possibilities. Theorists may be interested in why an over-parann parameetmetri neural network with deterministic initial weights achieves such good generalization. Practitioners may apply ZerO to train networks without normalization train sparse neural networks with pruning techniques. We hope that our results will inspire other researchers consider deterministic initialization schemes and rethink the role of weight initialization in training deep neural networks.
REFERENCES
David E Rumelhart, Geoffrey E Hintont, and Ronald Williams. Learning representations by back- propagating errors. page 4.

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In 2015 JEEE International Conference on Computer Vision (ICCV), pages 1026-1034. IEEE, a. ISBN 978 178-1-4673-8391-2 doi: 10.1109/ ICCV.2015.12 URL
leee or wttp:r////ooocc.o////eeao
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. page 8.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. b. URL
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. a. URL URLnttps://ppeeeeiiiee http:r/veeeeinneeffooouood/doood
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. URL https://papers. .niperss.ipps...... cc//paperr/211//aas/ 8
html
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network.
org wwt.:///./aabbb15555
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. c. URL UBLL:ttpp//axvvvorr///aa 03355
W.K. Pratt, J. Kane, and H.C. Andrews. Hadamard transform image coding. 57(1):58-68 68. ISSN 1558-2256. doi: 109/000C,
Conference Name: Proceedings of the IEEE.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- baadie learning applied to document recog- nition. 86(11) (1:22))-22322 ISSN 1558-2256. doi: 10.10095.72671. Conference Name: Pro- ceedings of the IEEE
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: Is feature diver- sity necessary in deep neural network initialization? International Conference on Machine Learning, pages 960-969. PMLR. URL Ittpss//proceeding: www.oceedingg...... mll press/v119/
bl blumenfe 1200a html
ISSN: 2640-3498.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. URL http::/arxiv.oom org. abs/
1510.00144
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. URL UTL.attpe//aarr....tt/a//////2/.
Fino and Algazi. Unified matrix treatment of the fast walsh-hadamard transform. 1-25(11):1142-
1146. ISSN 1557-9956. doi: 10.1 19.1100/...196..666 167666666999.. Conference Name: IEEE Transactions on Computers.
Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function deep networks.
wttp:///ggg.aas//.022... 10444
Alex Krizhevsky. Learning multiple layers of features from tiny images.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages
doi: 10.1109/CVPR.200055220 52068488 ISSN: 1063-6919.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. b. URL http:::tt//aaiiii
abs /ow0bb///7110191111
Moritz Hardt and Tengyu Ma. Identity matters in deep learning.
URL https:: /ttp://///xiiv..// orrg/
abs/ 611 04231v3
10

---

Under review as a conference paper at ICLR 2022
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. ReZero is all you need: Fast convergence at large depth. URL http:
//axxiv.org/ http://aaa/00....///
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine
Learning. pages 1243-1252. PMLR. URL httpas///ppoeeeennggssss..essses wtwp:einnnsm......e......
gehring17a. html
ISSN: 2640-3498.
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what the ques- tion? In Proceedings of the 34th International Conference Machine Learning, pages 342-350. PMLR. URL hattps:/pproeeedingg wwl.ggs............ http:pppceeeccll....pppeeeeeaaaa 2640-3498.
html ISSN:
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. U ///r//i..rr//bb//555..0000
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. URL /tRp:
www. abtg/abs/1700 C2677
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel ma- chines. In Advances Neural Information Processing Systems, volume 20, Cur- ran Associates, Inc.
URL httos:// .ips.. cc/papper/200/haaa/

---

Under review as a conference paper at ICLR 2022
A
ADDITIONAL DETAILS OF SECTION 2
A.1 DETAILS OF THE EXPERIMENTS ON MNIST
We construct simple network based the second layer to match the dimension of the labels. The network is: F(x) W a Relu ( + 12) Relu( W I't a where E 784 and y 12 which matches the input and label dimensions of the MNIST dataset. Both the first and second layer has 2048 and 2048 hidden spaces, where W I' 22048x 784 and W I2 IR X W IR 20x2088 is a linear mapping, I' is dimension increasing skip connection based on zero padding.
Figure where we add linear mapping after the
We train the network for 14 epochs using SGD with 0.1 learning rate. We compare with Kaiming Init and Xavier Init by applying their initialization with the standard setting the example network. All settings achieve the test accuracy above 98% after the training. Our weight distribution plot in the main text is also gathered at the end of the training.
A.1.1 WEIGHT DISTRIBUTION AT DIFFERENT TRAINING ITERATIONS
Wright
Weight
0.8
Xavier Init Kaiming Init
Xavier Init
Kaiming Init
p(a))
0.4
0.2
0.4
-0.2
0.0
0.2
0.4
00.2
-0.1
0.0
0.1
0.2
Wright distribution heration 1038
Weight distnbution iteration 5050
ZerO Init
ZerO Init
0.6
0.8
Kaiming Init
Kaiming Init
p(x)
0.4
p(x))
0.4
0.2
0.2
0.0
0.0
0.2
0.1
0.0
0.1
0.2
0.2
00.1
0.0
0.1
0.2
2
Wright distibution ltration 10030
Weight distribution Reration 13294
ZorO Irit
7 Init
Xavier Init
Xavier Init
0.6
Kaiming Init
0.6
Kaiming Init
0.4
0.4
0.2
0.2
0.0
0.0
0.1
0.1
0.1
0.1
12

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL DETAILS OF SECTION 3
B.1 PARTIAL-IDENTITY INITIALIZATION
Algorithm Partial-Identin Initializer for x convolutional kerne..
Input: a X Cin Cout convolutional kernel K, Cin number of input channels, Cout number of output channels.
Step 1. For ..... Gin and j 0, ... Cout 1, initialize K00,0,,, 0
Step 2. Cmin f- min Cout)
Step 3. For O, ... Cminn 1, initialize K[0, i,
Return K
C NETWORK PRUNING
As shown in Figure the network trained by ZerO has weight variance significantly smaller than the networks trained by random initialization methods. Because the weights that are close to zero usually have smaller effects the network, they can be pruned increase the network sparsity while preserving the prediction performance. For ZerO initialization, as most of the weights are extremely close zero even after training. this motivates us to discover whether ZerO helps to generate sparse networks through pruning.
To verify our hypothesis, we apply standard magnitude-based pruning for the trained networks initialized with different initializers. We then evaluate the test accuracy of the pruned networks. The magnitude-based pruning method prunes a portion of weights with the lowest magnitudes in each layer. We use the network described in Appendix A for MNIST and ResNet-18 18 for CIFAR- 10. Standard ResNet-18 adopted for Kaiming and Xavier initializers for better accuracy, and ResNet- (AugSkip) is applied for ZerO initializer.
As shown in Figure 6 compared to Kaiming and Xavier initializers, the networks trained with ZerO initializer can be pruned more aggressively while preserving the test accuracy. Because the state-of- the-art pruning methods adopt standard random initializers by default, we believe ZerO would be a powerful replacement that improves the pruning performance.
Pruned Fully Connected Network MNIST
Pruned ResNet-18 on CIFAR-10
100
80
Test Accuracy
Test Accuracy
60
ZerO
Xavier Init
20
Xavier
Kaiming Init
Kaiming Init
50%
30%
10%
5%%
1%
90%
80%
70%
60%
50%
Percent of Weights Remaining
Percent Weights Remaining
Figure 6: Test accuracy of the pruned networks trained with various initialization methods.
13

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL ABLATION STUDIES
D.1 BENCHMARKING HADAMARD TRANSFORM
In this ablation study, we evaluate the effects of our modifications the original ResNet, including 1) adding additional skip connections and 2) applying Hadamard transform. As shown in Table B we evaluate different configurations using ResNet-18 on CIFAR-IO. The results suggest that both additional skip connections and Hadamard transforms needed for ZerO initialization. This verifies that the problems we discussed in Section] 2 are crucial for initializing the weights to zero.
In addition, the results indicate that the slight accuracy degradation comes from the Hadamard trans- form instead of the additional skip connections. This motivates us to discover a better way to break the training degeneracy while preserving the performance in the future.
Additional skips
Hadamard transform
ZerO Init
Kaiming Init
Xavier Init
89.24
5.14
5.20
87.52
5.24
5.26
47.26
5.14
5.20
5.26
5.27
5.28
Table 3: The effects of additional skip connections and Hadamard transform ResNet-18 on CIFAR-10. ResNet-18 without both additional skips and Hadamard transform is the standard ResNet-18. Test errors (%) are reported.
D.2 LEARNING RATE WARMUP
We observe that learning rate warmup essential for ZerO to achieve a large maximal learning rate. This is because large initial learning rate leads to gradient explosion, as most of the weights start from the exact zero at the beginning. To demonstrate it, we measure the gradient norms during training for the settings with or without learning rate warmup.
As shown in Figure 7 we present the gradient norms wrt. the weights of the first convolutional layer in ResNet-18, which is trained CIFAR-10 for the first 400 iterations. The results indicate that learning warmup is needed for ZerO because the smaller initial learning rate prevents the gradients from explosion at the beginning of training.
Interestingly, we observe that the warmup has much larger effect on ZerO than other random ini- tialization methods. For Kaiming and Xavier initialization, gradient norms nearly unchanged after applying warmup. However, for ZerO initialization, the gradient norms become significantly smaller (even smaller than the baselines) after applying the warmup.
With Warmup (LR starts with 0.01)
Without Warmup (LR starts with 0.1)
ZerO
ZerO Init
Kaiming Init
Kaiming Init
Xavier Init
Xavier Init
100
200
300
400
0
100
200
300
400
Iterations
Iterations
Figure 7: Measuring gradient norms of ResNet-18 for the first 400 iterations.
For random initialization methods, the small changes of gradient norms are because the gradient norms largely dominated by the initial weight variance, which controlled by the initializers.
14

---

Under review as a conference paper at ICLR 2022
Choosing inappropriate initial weight variance usually leads to gradient explosion. To avoid the explosion, previous studies on random initialization propose various principles to control the initial weight variances, given different architectures and types of nonlinearities. However, as ZerO initial- ization trains the weights from zeros and ones without pre-defined weight variances, the learning rate becomes the dominant factor that controls the norm of the initial gradients. Hence, choosing ap- propriate initial learning rate is sufficient for avoiding exploding gradients under ZerO initialization, without the need of considering the variances of the initial weights.
15


Reviews:
[{"review_id": "teP5BPTOJjx", "reviewer": "Reviewer_RE44", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. It is interesting to show that ResNets can be trained to get competitive performance with zero-and-one initialized weights. The conventional view is that the weights should be initialized with random values for symmetry-breaking.\n2. The problems (e.g. dead neuron) discussed in the paper are well-illustrated. In Section 2.1, the authors use simple and clear examples to demonstrate how zero-initialized weights can cause training to get stuck and why short connections can rescue. In Section 2.2-2.3, the authors explain in a similar manner why Hadamard transform can deal with degeneracy problem caused by the dimensionality expansion.\n3. Experimental results confirm that randomness can be removed from weight initialization. The authors compare several initialization schemes on the standard ResNet models on CIFAR-10 and ImageNet and show theirs achieves competitive results in terms of convergence speed and test accuracy.\n\nWeaknesses:\n1. The motivation of the zero-and-one initialization is unclear. What are the theoretical advantages of removing randomness in initialization? It seems to me that keeping randomness in initialization has the advantage of potentially avoiding bad local minima and saddle point. Empirically, despite the authors have shown some comparison with He and Xavier methods, the zero-and-one initialization does not show *significant* advantages.\n2. The zero-and-one initialization is of a heuristic nature and lacking of theoretical background. There is no unique way to avoid the dead neuron problem. For a trivial example, one can simply add a shortcut connection from the dead neuron to a output neuron and one from a input neuron to the dead neuron. It would be desirable to motivate the proposed initialization scheme with some theoretical optimality.\n3. Additional costs of the zero-and-one initialization. The extra shortcut connections and Hadamard transform result in additional computational cost and model complexity. As they are cheap operations, I consider this issue a minor one.\n4. In Section 1, the authors state \"In order to design networks robust to vanishing gradients at initialization, He et al. introduced\nskip connections\". However, it is contradictory to the claim in the original paper of He et al, which states \"We argue that this optimization difficulty is unlikely to be caused by vanishing gradients.\"", "summary_of_the_paper": "This paper proposes a new initialization scheme for ResNet. The key feature is the initialization values are set to all zeros and ones, rather than random. The development of this initialization scheme is motivated by avoiding the \"dead neuron\" problem and \"training degeneracy\" problem for networks with zero-initialized weights. Some toy examples are discussed to illustrate the motivation. Last, experiments on standard benchmarks are conducted to verify the initialization scheme.", "summary_of_the_review": "The paper is technically correct but lack of strong motivation, theoretical advances and empirical performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qHTpzQxzZcL", "reviewer": "Reviewer_PQDc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* (Significance) The paper explores common-sense modifications that are important for understanding the operation of residual networks.\n* (Significance) I found the results of the paper to be interesting.\n* (Clarity) The majority of the paper is clearly written.\n* (Reproducibility) The paper provides code, helping reproducibility.\n* (Quality) The experiments include multiple seeds for every configuration, with a clear description of the used hyperparameters. The tasks used for experiments are highly relevant (e.g: Resnet50 for Imagenet). As such, I found the results to be highly reliable.\n\nWeaknesses:\n* (Significance/ novelty) Getting rid of batch normalization was already achieved by fixup, in a very similar method to what the paper presented. (Which did not require initialization with only ones and zeros). The experiments do not show a benefit for using the new modified resnet over fixup. Other works, like [SkipInit](https://arxiv.org/abs/2002.10444) and [ReZero](https://arxiv.org/abs/2003.04887) also offer similar methods to improve convergence speed and get rid of batch normalization. \n\n* (Significance) The paper pride its initialization on being deterministic, but this is poorly defined. I could argue that my initialization would be just as deterministic by using constant seed or importing weights saved on storage. \n\n* (Quality- Empirical) I was not convinced that the 4th contribution (lower standard deviation) is correct. Comparing standard deviation over 5 seeds does not give you a statistically significant result, this could easily be the result of random chance. I was also not convinced this is important: If my goal was to reproduce my run, I wouldn't have changed the seed.\n\n* (Novelty) Initializing residual networks to only zeros and constant was already done in [this work](https://arxiv.org/abs/2007.01038). This paper cites this work as requiring \"random noise to improve the performance\". However, the prior work also concluded that \"standard GPU operations, which are non-deterministic, can serve as a sufficient source of symmetry breaking to enable training\" (quote from the abstract). Given that this paper also uses GPUs and did not mention using them on deterministic mode (It's stochastic by default), the specified distinction between this work and the prior work is not valid.\n\n* (Clarity) I was confused by section 2.2. As far as I can tell, the degeneracy problem there is a direct result of the choice of padding $\\mathbb{I}_{1,2}$ with zeros when the network is wide. Since this is not a trivial solution to the expanding width problem, the title \"Width expansion leads to training degeneracy\" is misleading, and the addressed problem in this section is poorly defined. I am also confused by why this problem is different from the first dead neuron problem (Skip connections obviously can't solve dead-neuron problems for neurons it is not applied on). \n\n* (Clarity) Section 2.4:  my understanding was that training degeneracy refers to a problem where padded zero values cause the gradient to remain zero  (section 2.2). I do not understand why $C_f$ and $C_b$ are good measures for that kind of degeneracy. Also, $C_f$ and $C_b$ aren't properly defined well for zero-initialized weights.\n\n* (Quality) The paper concludes with results I consider to be negative. I thought this was interesting, as some of the changes made in this paper seem trivial and I did not expect them to harm performance. However, there was no proper ablation study (Figure 3 helped, but wasn't sufficient), so it remains unclear what element exactly caused the small degradation in performance. For example, I wonder if the single-convolution-skip architecture is responsible, but there is no way to disentangle the results from the entire zero-initialized-hadamard network. ", "summary_of_the_paper": "The paper suggests an adjustment to the resnet architecture, which enables successful training despite having deterministically initialized weights.", "summary_of_the_review": "The paper suggests some interesting modifications to the resnet architecture. However, the changes aren't very novel, and the paper did not convince me that these new adjustments are beneficial. The results in the paper appear to be (slightly) negative, but there is no proper ablation study for them to be informative", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iXvd7W7-4mT", "reviewer": "Reviewer_9bJj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe idea of ZerO is novel and has various benefits such as improving the reproducibility of models and training networks without batch normalization. \n\nWeakness:\n1. One drawback of ZerO is that it can only be applied to residual networks.\n2. Can the authors also provide a comparison with the baselines for space-time complexity? I think that the number of skip connections added is significant which may add to the complexity.\n3. How about applying ZerO for other core CV applications such as segmentation and object detection. Does the behavior of ZerO remain the same for such applications?", "summary_of_the_paper": "The paper proposes a deterministic weight initialization scheme called ZerO for residual networks.  ZerO works by first augmenting existing ResNet architectures with extra skip connections and Hadamard transforms and then initializes network weights with only zeros and\nones (instead of random weights). The authors show that the proposed scheme has various benefits such as improving reproducibility, training networks without batch normalization, and improved performance.", "summary_of_the_review": "Overall the paper presents some interesting ideas. However, there are some concerns such as the space-time complexity of models and limited applicability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dZaTx6T-gfc", "reviewer": "Reviewer_8Znp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper is easy to follow.\n- The proposed method is interesting and somewhat novel.\n\t\nCons:\n- This paper seems to propose a new structure (containing extra skip connections and Hadamard transforms) that contradicts the topic ( an initialization). The proposed method ZerO overly relies on auxiliary structures, compared with FixUp [1] that only uses a multiplier and bias. Therefore, the proposed method is not a pure initialization for ResNet.\n- In Table 2, does ZerO only use a structure with AugSkip? Wether other initializations (i.e., Fixup, Kaiming, and Xavier) choose vanilla ResNet? There is no description to mention it. In addition, ZerO performs worse than Fixup. Although it can reduce the gap by running 180 epochs, it costs too much on time.\n- It is suggested to change notation like $W^2$ to $W^{(2)}$.\n\n\nAdditional Questions:\n- As a fully-deterministic structure, the initial weights are the same with different seeds. Why is the std not equal to 0? What else makes an effect on the training?\n- Will the auxiliary structures influence time consumption?\n- Kaiming initialization is proposed for relu activation function and ResNet adopts relu. Therefore, is it weird that Kaiming is worse than Xavier in Table 2?\n\n[1] Hongyi Zhang, Yann N. Dauphin, Tengyu Ma. Fixup Initialization: Residual Learning Without Normalization. ICLR 2019.", "summary_of_the_paper": "To initialize a ResNet with only zeros and ones, this paper analyses the problem of all-zero initialization and then proposes a  method named ZerO to achieve the goal by augmenting the standard ResNet architectures with a few extra skip connections and Hadamard transforms. The proposed idea is interesting and somewhat novel. However, it seems to introduce a new structure rather than only initialization, and empirical results are not really good.", "summary_of_the_review": "Although the idea is somewhat novel, for the unremarkable empirical results, incongruent method contradicting the topic initialization, I regret to reject it weakly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JGVx3D_e2q", "reviewer": "Reviewer_P5yd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength:\n\n#### 1. The motivation of the paper is clear and sound.\n\n#### 2. The idea of combining Hadamard transform with identity mappings to initialize ResNet-style architectures is not previously seen in the literature.\n\n#### 3. Deterministic initialization reduces source of randomness, as is shown by the standard errors of trial results of different methods.\n\n#### 4. The paper is easy to comprehend.\n\n#### 5. Code is shared anonymously for reproducibility.\n\n\n### Weakness:\n\n#### 1. While the approach as a whole is new to my best knowledge, most ideas have been investigated in published work. For example, identity initialization of convolutional kernels was presented in Xiao et. al. (2018) under the name of Delta-Orthogonal initialization; additional residual connections appear in related work such as Hardt and Ma (2017); Hadamard transform layer is used in binary neural networks, among other neural network applications.\n\n#### 2. The experimental results do not show performance advantage of the proposed method over previous ones such as the original ResNet and the Fixup initialization (in the no normalization setting).\n\n#### 3. The mathematical / technical contributions are limited; though this may have the virtue of improving the overall readability.\n\n*Reference:*\n\n*Xiao et. al. (2018). Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks.*\n\n*Hardt and Ma (2017). Identity Matters in Deep Learning.* \n\n### Other comments and concerns:\n\n#### 1. I am interested in the final weight distributions shown in Figure 3 (Left). I wonder whether this figure implies the proposed method can obtain solutions \"closer\" to identity mapping.\n\n##### 1.1. Do the plots correspond to the same network architecture, only different initializations?\n\n##### 1.2 What are the initial weight variances of different initializations (as it could be that they start from very different distributions)?\n\n##### 1.3. Since the network block contains a scalar multiplier, there can be identical blocks (in terms of the input-output mapping) with very different weight distributions. Could that be the case here? \n\n##### 1.4. Could you provide more justification as to the connection of close-to-zero weight distribution to weight pruning? In my understanding, a more direct argument is to analyze the network's sensitivity to weight perturbation; due to the presence of scalar multipliers and/or normalization layers, small weight scale does not necessarily imply small sensitivity.\n\n\n#### 2. In \"Experiments - Hyperparameter settings\", you mention \"learning rate warmup is essential for ZerO to achieve a large maximal learning rate\". This seems to suggest that the proposed initialization still suffers from training instability resulted from gradient explosion.\n##### 2.1. To address this concern, could you provide some comparison (e.g. gradient norm plots) of your method vs. the baselines in the first hundreds of steps with and without warmup schedule?", "summary_of_the_paper": "In this paper, the authors propose some modifications to the classic ResNet architecture and initialization scheme, so that the network can be trained with fully deterministic initialization, even without any normalization layers. The authors motivate their design choices by identifying and analyzing the problems with the forward and backward dynamics of some naive ideas. Then the authors present their approach and explain why each choice they make can solve the specific problems they have presented. Then they evaluate their proposed method along with some baselines on two popular image classification datasets, showing that their deterministic initialization method is on par with the ones with random weight initialization. Finally, they present related work and conclusion.", "summary_of_the_review": "The paper introduces a novel method to initialize a ResNet-like architecture deterministically with binary weights (except for the Hadamard transform where additional rescaling is required). The paper contains some novelty in its model design, has good clarity, but does not show convincing practical significance compared with previous work. Some claims are hypothetical and may require further justification or be supported by further evidence.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests an architecture with a deterministic initialization which has only 0/1 values.
The reviewers were mostly (marginally) negative, mainly because of the low novelty and significance of this work. 

Specifically, the main novelty issues were:
1) Improving convergence speed and removing BatchNorm: was already done, in a quite similar manner, and it achieves better or similar results. (Fixup , ReZero: https://arxiv.org/abs/1901.09321, https://arxiv.org/abs/2003.04887, and few others as well)
2) Initializing a network with deterministic initialization: was also done (ConstNet, https://arxiv.org/abs/2007.01038). I think the main difference from the previous work is the additional Hadamard connections, which help break the symmetry. However, it is unclear what is the benefit of this modification, as the previous work could train without it (albeit on CIFAR).

Specifically, the main significance issues were:
1) Reducing standard deviation: The authors' response confirmed there is no statistically significant benefit (p=~ 0.1) for variance reduction when comparing with Kaiming initialization for ImageNet.
2) General network performance: The results do not seem better than the baseline (Xavier init is not a proper baseline in a network with ReLUs).
3) Sparsity claims: The network appears to be losing accuracy even with 20% sparsity, which isn't even useful for efficiency. For comparison, the lottery ticket hypothesis showed you can get to 90% sparsity and get better results. So, this is a nice observation, but not a major contribution.

Therefore, I recommend the authors to better distinguish themselves from previous works (What are the changes? Why are these important?), and improve their empirical results so they highlight the usefulness of the suggested method (e.g., improve the SOTA in some benchmark).
Decision: Reject