Identical Initialization: A Universal  Approach to Fast and Stable Training of Neural Networks
Authors: Yu Pan
Zekai Wu
Chaozheng Wang
Qifan Wang
Min Zhang
Zenglin Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Initialization
Idetity Matrix
Dynamical Isometry

Abstract: A well-conditioned initialization is beneficial for training deep neural networks. However, existing initialization approaches do not simultaneously show robustness and universality. Specifically, even though the widely-used Xavier and Kaiming initialization approaches can generally fit a variety of networks, they fail to train residual networks without Batch Normalization for calculating an inappropriate scale on data-flow. On the other hand, some literature design stable initialization (e.g., Fixup and ReZero) based on dynamical isometry,  an efficient learning mechanism. Nonetheless, these methods are specifically designed for either a non-residual structure or a residual block only, and even include extra auxiliary components, limiting their applicable range. Intriguingly, we find that the identity matrix is a feasible and universal solution to the aforementioned problems, as it adheres to dynamical isometry while remaining applicable to a wide range of models. Motivated by this, we develop Identical Initialization (IDInit), a sufficiently robust, universal, and fast-converging approach on the identity matrix. Empirical results on a variety of benchmarks show that IDInit is universal to various network types, and practically useful with good performance and fast convergence.

Under review as a conference paper at ICLR 2023
IDENTICAL INITIALIZATION: A UNIVERSAL APPROACH TO FAST AND STABLE TRAINING OF NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
A well-conditioned initialization beneficial for training deep neural networks. However, existing initialization approaches do not simultaneously show stability and universality. Specifically, even though the widely-used Xavier and Kaiming initialization approaches can generally fit variety of networks, they fail to train residual networks without Batch Normalization for calculating an inappropriate scale data-flow. On the other hand, some literature design stable initializa- tion (e.g.. Fixup and ReZero) based on dynamical isometry, an efficient learning mechanism. Nonetheless, these methods are specifically designed for either a non-residual structure or a residual block only, and even include extra auxiliary components, limiting their applicable range. Intriguingly, we find that the identity matrix is a feasible and universal solution to the aforementioned problems, as it adheres to dynamical isometry while remaining applicable to a wide range of mod- els. Motivated by this, we develop Identical Initialization (IDInit), sufficiently stable, universal, and fast-converging approach to the identity matrix. Empirical results on variety of benchmarks show that IDInit is universal to various network types, and practically useful with good performance and fast convergence.
INTRODUCTION
Deep Neural Networks (DNNs) have attracted significant attention due to their versatility in various applications. To obtain well-conditioned training, a suitable initialization is important (Sutskever et al., 2013; Arpit et al., 2019; Huang et al., 2020; Pan et al., 2022). Common initialization methods include Xavier (Glorot & Bengio, 2010) and Kaiming initialization (He al., 2015). Later, dynam- ical isometry (Saxe et al., 2014) is widely used for building stable starting status for very deep network (Mishkin & Matas, 2016; Burkholz Dubatovka, 2019), and even can stably train 10000- layered networks (Xiao et al., 2018). However, despite the successes, these aforementioned methods are unsuitable for residual blocks without Batch Normalization since calculating an improper scale on data-flow (see See. 4.2 and Sec. C.2). Addressing this issue, Bachlechner et al. (2021); Hardt & Ma (2017) proposed to stabilize the training of residual blocks according to dynamical isometry mechanism by transiting identity via multiplying residual stem with O. Nevertheless, the identity- maintaining methods can be only applied on residual networks, and some of them even require auxiliary components stabilize model training (Blumenfeld al., 2020; Bachlechner al., 2021; Zhao et al., 2021), leading to lost generality to other network structures. To overcome the above problems, in this paper, we propose stable, general, and convernngg initialization approach, based on the identity matrix.
Motivation on Identity Matrix. As mentioned above, implementing identity transition can natu- rally correspond to the isometric mechanism that is beneficial for fast convergence and improving performance (Bachlechner et al., 2021). To maintain this transition of both residual and non-residual modules, the identity matrix is potential solution. In detail, consider i-th block in DNN,
m
(++1) (r + II g(t.)) (0)))
where m is the number of weights in a block, ((i-)) means j-th weight in the i-th block, and r e (0, 7) indicates whether Fi.. (1) denotes residual layer. Usually when 0 and m 1, Eq. (1) is

---

Under review as a conference paper at ICLR 2023
1 0
0
0
0
0
0
c
0
X
Y
X
Y
Y
0
0 0
0
0
0
Figure 1: simple case of IDInit on a residual and non-residual network layer. e is set 6, therefore, y ss 0. Y of both left and right sub-figures is equal to X.
non-residual layer. Under this condition, achieve the identity transition, namely x(++1) x00, g(..1) 7 is the unique solution. When 7 and > 2, Eq. (1) denotes residual layer. At this moment, setting all (0(-.) == to I is not feasible. Therefore, it is usual to set the last weight (ti.) to 0 to maintain identity (Zhang et al., 2019; Zhao et al., 2021). Overall, without loss of generality, it is necessary and feasible to apply identity matrix initial status for maintaining identity.
Identical Initialization (IDInit). In this paper, we introduce the identity matrix into designing a novel and practical initialization, named Identical Initialization (IDInit). A simple case of IDInit is shown in Figure For a non-residual condition, squared identity matrices can easily achieve iden- tity transition, however, the most general condition, non-squared matrices cannot (Vaswani al., 2017). To adapt the non-squared situation, we modify the identity matrix by maintaining signal variance described in Sec. 3.2. For a residual condition, dead neuron problem will be caused by directly setting the last weight in a residual stem to O (Zhang et al., 2019; Zhao et al., 2021). Ad- dressing this, we simply select some elements an extremely small numerical value trainable neurons as in Figure 1. We also observe that an identical convolution layer that maintains identity transit degrades performance dramatically. Tackling the issue, we propose a modest change to an identical convolution layer by fusing spatial information, leading to significant improvement. Moreover, we find the model performance is uncertain with random initialization whose values are sampled from probability distribution (Glorot & Bengio, 2010; He et al., 2015). IDInit is exactly able to handle this situation of its determinacy that is unique, and uncorrelated with samplers.
increase
To our best knowledge, it is the first trial to put identical-lin like initialization into practice. Previous work, Bartlett et al. (2019). has used the identity matrix as a weight with excessive attention on approximate analysis, resulting in imprecise conclusions for the gap between reality and theory. Additionally, this work addresses only relatively simple scenarios without an activation function and ignores momentum in stochastic gradient descent (SGD), which optimize critical points to degree. And this method is incompatible with ResNets without Batch Normalization (Ioffe & Szegedy, 2015), since signals will explode if all weights are identically. Motivated by some of the observations and addressing shortcomings, we provide simple, practical, and efficient initialization method.
2 BACKGROUND AND RELATED WORK
Dynamical Isometry. Give an L-layer network with blocks formulated by Eq. (1), ( is input and (L) is the L-th layer's output. Assuming signal magnitude (e.g., o (x)) of each layer chang- ing in the scale the last signal magnitude can reach a² (e.g., a² (xx()) easy to cause signal explosion and diffusion, especially for large L. Introduced from mean-field theory (Pennington et al., 2017: 2018), the dynamical isometry is comparably reasonable mecha- nism to measure models' trainability. To utilize this paradigm, it usually considers the input-output Jacobian
(xz(((( making
(L)
J.. (r(0))
whose mean squared singular value is X- (Pennington et al., 2017) and (Bachlechner et al., 2021) show that x indicates the model in a chaotic phase, and prepppropgttted gradients will explode exponentially. By contrast, X < 1 means a model in an ordered manner that back-propagated gradients exponentially vanish. X is critical line of initialization, avoiding vanishing or
2

---

Under review as a conference paper at ICLR 2023
exploding gradients. Using the stable isometry condition can provide sufficiently stable training process for networks (Gilboa et al., 2019; Poole et al., 2016; Yang Schoenholz, 2017).
Related Initialization. Xavier (Glorot & Bengio, 2010) and Kaiming initialization (He et al,, 2015) is classical initialization. Specially for residual network efficiency, Hardt & Ma (2017) the- oretically demonstrates that network training benefits from keeping identity. Then, maintaining identity transit turns into promising way further improve model performance. Fixup (Zhang et al., 2019) and ZerO (Zhao et al., 2021) both set residual stem to (not residual connections) to guarantee the identity of signals, thereby initializing ResNets successfully. SkipInit (De & Smith, 2020) replaces Batch Normalization with a multiplier whose value is O. ReZero (Bachlechner et al., 2021) directly adds extra parameters of value O to keep identity, leading to fast convergence.
IDENTICAL INITIALIZATION (IDINIT)
In this section, we first use a descriptive example to elaborate on the mechanism of IDInit for fast convergence. Then, we describe the way to maintain the stability for non-square identity-like ma- trices. Next, we propose a zero preserving scheme to solve the dead neuron problem in identity transition. At last, use simple but efficient reshaping strategy to significantly enhance convolu- tion performance.
1.4
1.4
3.1 TOY EXAMPLE
Ga
Following Bachlechner et al. (2021), we utilize simple ex- ample of the mechanism that dynamical isometry helps IDInit to obtain a fast convergence. Con- sidering L-layer network with simplified Eq. (1):
64
0.1
62
0.2
M/
(a) Non-Residusl Plot.
(b) Residual Plot.
Figure 2: Contour plots of the log gradient norm log |@R|| on non-residual and residual networks. w(() and (() are both weights. The training process which is conducted dient descent using a training set of æg = {1., 1.1,..., 1.8). (a) shows (w = (1) = 1} can avoid poorly conditioned regions around 0, and converge to (()) = 2.19. (b) cares about two initial position (w)(1) = 0, w2 1} and (w 1, (!)) 0). The two points' trajectories do not also pass the poor regions around w((3) 11 w and
(L)) = (r (0 20 (1)
Bachlechner et al. (2021),
ground-truth
(L.) 50 x zo via gra-
where (1) and (2) denote the first weight and last weight a residual stem respectively, and x(*) the feature in layers. E [0, 1} deter- mines residual connection. Specif- ically, r 0 and = 1 represent non-residual and residual condi- tions respectively. The Jacobian of Eq. (3) is Jor = (r + (e()) (
converge the solution ())) 1.19.
Obviously, identity transition on both non-residual and residual settings, namely (r = 0, ( = wi) 1} and (r= = 1, w(1) = 1, (2) = 0) respectively, will achieve Jor. 1, which conforms to the dynamical isometry mechanism that helps improving training ability (Pennington et al., 2017).
Further, we delve into gradient update analysis. Following gradient descent, wi be updated by
w (1( (2) (X()) (r + (0
where R means the loss function, and i a learning rate. As (!)) and (?) are equivalent in Eq. (3), (2) can be updated similar to Eq. (4). When w(!)) 1, updates are required less than 1. Therefore, the learning rate is constrained to
2 T-11 (A x L- to 21)-111 if residual.
if non-residual,
(5)
For the non-residual condition, the learning rate polynomial to L,, thereby insensitive to the depth. By contrast, in the residual block, (2 >> will cause learning rate exponentially small and (2) -1 also cause gradient diffusion. On this condition, setting (2) 0 can be good solution
3

---

Under review as a conference paper at ICLR 2023
D:
DV
0
T-C
Pas
Dart
Dess D,
D
Days D.
Drs < Di
Dec Di
Dor 2 D.
Idantiy Iniiaaiization
Zero Preserving Iniialzation
Figure 3: Illustration IDInit for fully-connected layers.
for avoiding large output and restricting gradients in suitable norm. Besides, it feasible to update w(() with the first non-trial step
(2) 2000
(6)
and will converge with learning rate that polynomial in the depth L of the network. We plot the training dynamics in Figure 2, and use this simple example to illustrate the mechanism of IDInit, which is always well-conditioned position for training.
3.2 IDENTICAL FULLY-CONNECTED LAYERS
A standard identity matrix naturally satisfy identity transition illustrated in Figure 1. However, in non-square situation, this natural advantage is lost. To address this problem, we give small modification to non-square matrix. Specifically, for fully-connected layer transformed from Eq. (1) x z00) set the weight (11) PRD:11X22 to
((0)
if im= j (mod Dj), m.j (0 otherwise.
T,
(7)
The initialization formulated Eq. (7) is termed as IDI,, where IDI means the identical initial- ization function, and is calculated by considering the activation function, e.g. TReLU = and Tsigmoid for ReLU and sigmoid activation functions respectively. As in Figure 3, setting can form IDI, initialization. Moreover, We provide a trainability analysis on the non-squared con- dition of IDI, Sec. A.2 of the appendix.
Remark. ZerO constructs a dlimeniion-incrastie matrix (i.e., D++1 >> D2) by padding zero val- ues to an identity matrix, which is similar our method, namely padding identity matrices recur- rently. While both the two padding strategies can transit identity propagation, padding zero values will cause zero gradients, which is known as a rank constraint problem, and can severely affect the performance. By contrast, IDI, can derive non-zero gradients, thereby, inherently avoiding the rank constraint problem. Therefore, IDI, has significant improvement on the prior padding method.
3.3 IDENTICAL RESIDUAL BLOCKS
At present, residual blocks become the most popular module in almost all the neural network (c.g., Mixers (Liu et al., 202 la; Tolstikhin et al., 2021), Convolutions (He et al., 2016; Zhang et al., 2019), and Transformers (Liu al., 2021b Vaswani et al., 2017)). A residual block is usually constructed with a residual connection and several transformations the residual stem. Following Eq. (1), a simple residual case with two parameter matrices can be formulated as
(++1) = (I + g(..0) (6.1))). (1)
8
where (0) means an input of i-th residual block in network, I is an identity matrix denoting residual connection, and ((C0) and (ff.1) are weights in the i-th residual stem of residual block.
Recent research (Zhang et al., 2019; Zhao et al., 2021) directly sets (i) the last transformation in the residual stem to 0, i.e., g(t.0) = 0, thereby maintaining an identity as
(1+1) (I ()) = (1))
(9)
4

---

Under review as a conference paper at ICLR 2023
(ii) the last classification layer of the network to 0. Although the two settings work, interpretation is provided for the trainability. We consider the effect of (i) is from dynamical isometry, since the mean squared singular value of the input-output Jacobian of the network is 1. As for the setting (ii), the last layer can be trained with arbitrary values when transiting identity in networks, according to the trainability analysis in Sec. 3.2. Therefore, when setting all values of the last classification layer to O, this classification layer can still obtain non-zero gradients of suitable magnitude. Since the settings (i) keeps identity transition and (ii) helps improve training stability, we also consider adopting them to our initialization. However, while there is no problem with the setting (ii), the setting (i) contradicts the proposed identical initialization as generating dead neuron problem.
100
203
250
(a) Weight initialized with numerical value O.
(b) Weight initialized with IDIZ
Figure 4: The last weight in a residual block of a trained ResNet. More than half of elements in (a) not trained, which is known the dead neuron. By contrast, IDIZie-6 successfully solves the dead neuron problem and make all the elements in (b) trainable.
Dead Neuron Problem. Fixup (Zhang et al., 2019) only uses multiplier of after ((1.0) O. thereby obtaining non-zero gradients. However, in realistic implementation of neural networks, the the multiplier of Batch Normalization can set to (Goyal al., 2017), and down-sampling operation can also cause filled !eatures Under the implementations, ((+00) always acquires gradients with 0 values, known the dead neuron problem, causing failed weight updating.
To overcome the problem, we generate small values on ((3.0) to assist in training. Recall the goal of setting (i) that outputs 0. Therefore, consider building a calculation to get the expectation and variance of outputs approaching to O. Prior to that, we consider two i.i variables, and v2 whose variances are o² (11) = d² 02(22 & and means are p(v)) p(22) y. The variable e(vl- v2) have
((v)= 0, 0² (v) = 2
(10)
where 6 is a coefficient, and (v) will be limited to O when is sufficiently small. Assuming elements of (f)) are i.dd to each other, by applying subtraction on any two elements, the result has a mean of O, and a variance related to s. We also take ((2,0) E 12 Pp41 xD; as an instance. At first, we initialize gtt.0) with IDI, Then consider two cases: (i) if D++1 < D;, setting (( +; with
-e, when m%D; mi value. This method is termed as IDIZ,, In this paper, we set 1e- 6 everywhere. As shown in Figure 4, IDIZie-6 successfully initializes the last weight in a residual block.
IDI=;: (ii) if Di+1 2 Dj, set (00 variance of by setting & to a
1. Therefore, we can obtain
3.4 IDENTICAL CONVOLUTIONAL LAYERS
Besides the fully-connected layer, the convolutional layer is another important structure in deep neural networks. For constructing general initialization, we also explore an initialization pattern for convolution with identity transition. In this part, we consider convolution layer formulated
y C C& B
(11)
where E Bixxxxei an input tensor, y RRA 'x' Xcout is an output tensor, E R Xoout
a convolutional kernel, and B = Ia xw" Xeout is a bias and usually set to h, w and h', w' denote height and width of the input and the output, respectively. Cin. and Cout are channels of input and output separately. & means convolutional kernel size.
cifar:py
hhtps:/gtthhh.ccmaaaaaaseee///// resnet cifar (0/cddi//mmssttrrresseee.p
5

---

Under review as a conference paper at ICLR 2023
Ideniiyy Transitin
Cout
Features
Divendyy
Ozn
7
7
Iegus nunber
7
IDIC,
Snited
Inoweee
Ggut
7
7
Figure 5: A case of number "7" Identical Convolution Layer. The upper sub-figure maintains the identity transition. The under sub-figt sub-figure is IDIC, initialization that shifts features for increasing diversity. More feature diversity from IDIC, is beneficial for improving model performance.
Degeneration from Convolution Identity Transition. As shown upper sub-figure in Figure 5, similar to an identity matrix, a convolution layer can transit identity by setting O-filled C as
IDI (C.....))
(12)
where E N+ and k 2n 1. As convolutional kernel window size, k is usually an odd number. After Eq. (12), y X' if Can = Cout- y will samel and over-sample on X along channel when Cin Cout and Can Cout respectively. Keeping identity is usually considered an efficient way to improve model performance, however, we find that this setting can lead to a fatal performance degeneration (see Sec. 4.3).
Inspired by Han et al. (2020) that enhance model performance by increasing feature diversity, we propose fuse spatial information by simply reshaping matrix initialized IDI, Specifically, consider one inner product part of convolution feature patch (patch) E Bov as
(patck) C.X (C((paach)
(13)
where C' Cout Xefouttkkkzn is the convolutional kernel, and (patch) E R&kc. is input patch. By initializing C with IDI,, then reshaping into C E xgi XCout, our initialization for a convolu- tion is completed. As shown in the under sub-figure of Figure spatial features shifted, thereby increasing feature diversity. We utilize IDIC, to denote such a reshaping process. In addition, we IDIZC, to represent the similar operation IDIZg.
We conclude the whole DDnit as (1) Non-Residual Networks. Directly applying IDI, and IDIC, to all the fully-connected and convolutional layers, respectively; (2) Residual Networks. (i) Applying IDI, and IDIC, to all the fully-connected and convolutional layers, respectively: (ii) Applying IDIZ. and IDIZC, to the fully-connected and convolutional layers in the last position of residual blocks, and the position of last classification layer.
4 EXPERIMENT
In this section, we design set of experiments to validate the proposed DDnit.. To begin with, we conduct experiments non-residual convolution and residual convolution in Sec. 4.1 and Sec. 4.2 respectively. Then we implement an ablation experiment in Sec. 4.3 to show the effect of the pro- posed two modifications Sec. 3. Later we analyze the determinacy of initialization in Sec. 4.4. At last, we employ experiments tasks large-scale datasets ImageNet and WuDaoCorpora in Sec. 4.5 and Sec. 4.6 separately. We also analyze the variance amplification in Sec. C.2 Appendix.
4.1 VALIDATION ON NON-RESIDUAL CONVOLUTION
We use this experiment show the hardness of classical Kaiming initialization for training. This experiment is conducted on C0. We use nine convolutional layers named AllConv (Springen- berg et al., 2015). We show the structure of AllConv in Table 4 in the appendix. The optimizer is

---

Under review as a conference paper at ICLR 2023
Stochastic Gradient Descent (SGD) with momentum 0.9, weight decay 5e-4, and learning rate le- 1. The learning rate scheduler adopts a warm-up cosine reduction strategy. We run the model in 300 epochs on one Nvidia A100. We adopt Kaiming initialization and IDInit w/o IDIC, initialization for comparison.
Results are shown in Figure 6, without warm-up strategy which is a strong trick for training, both Kaiming and IDInit w/o IDII, fail to train the model. By con- trast, our initialization can train AllConv and maintain the highest performance in all situations, showing a strong effect on stability and performance. As IDInit w/o IDIC performs awfully, we demonstrate the degeneration from identity transition mentioned Sec. 3.4 is relieved by our new reshaping strategy.
0..
1.
0.6
12
0.2
0.5
ming
0.0
50 100
150 200 250 300 Epoch
0
15
70
30
40
50
05
Vormup Epoch
(a) Test Top-1
(b) Test Best
Figure 6: Results of AllConv Cifar10. The number behind the initialization denotes the warm-up epochs.
4.2 VALIDATION ON RESIDUAL CONVOLUTION
In this experiment, validate the proposed initialization with the comparison with existing initial- ization, including (1) Fixup: (2) SkipInit; (3) ReZero; (4) Kaiming: (5) Zero y (Setting the scale in Batch Normalization (BN) to 0) (Goyal et al., 2017). We use ResNet-56/1 10 as backbones on Cifar10. To verify the universality, we use two settings, namely w/ and w/o BN. For analyzing convergence, we adopt both SGD and Adam optimizer for updating models. We set SGD, with the momentum 0.9, the weight decay 5e-4, and the learning rate 0.2. For Adam, the learning rate is 0.001, 31 0.9 and 32 is 0.999. We train models for 200 epochs. The learning rate is reduced with cosine function.
Table 1: Results Cifar10. A multiplier is used for scaling the output of residual stem. For validating universality, we consider "ResNet w/o Multiplier" which is a tough condition for training.
56 Layer (SGD/Adam)
110 Layer (SGD/Adam)
Model
Init.
Acc.
Epochs to 80% Acc.
Acc.
Epochs to 8066 Acc.
Zero
92.32 sg/87/77
27
93.07 00.0720::
36.2/5627
Fixup
st.44:..s/89.50..00.
=
93.32
=3
ResNet
SkipInit
2/
92.67
Multiplier
ReZero
93.0 93.06 289.0.66//999......
33
94.03
35 35
Kaiming
93. 03.364..022 1877
34gs/50_2
94.06:a..s/87.894.r
/324/56ga
IDInit
93.41 00 to 20.01
26
27
ResNet
Kaiming
riverged/63.842..oo
Diverged//1.455
w/o Multiplier
DDInit
0 20.42
61 6125/8822
91.51
$7
Results are shown in Table 1. The proposed IDInit can achieve the least epochs to reach 80% accuracy in all settings, which shows good convergence ability. In the w/ BN setting, IDInit consistently performs well compared with baselines. In the w/o BN setting, Kaiming fails to train the models with SGD optimizer for calculating improper scales for data-flow. Even though using the adaptive Adam optimizer, Kaiming still cannot derive normal results. By contrast, DDInit can work normally both optimizers and derive comparably reasonable results.
4.3 ABLATION EXPERIMENT
We conduct this experiment to validate the effect of the proposed two improvements. The dataset is C0 and the backbone is ResNet-20. We run four times following settings: (i) IDInit w/o IDIC, and w/o IDIZC,; (ii) IDInt w/o IDIC, and w/ IDIZC,; (iii) IDInit w/ IDIC, and w/o IDIZC,; (iv) DInit. We choose SGD with momentum 0.9, weight decay 5e-4 and learning rate 0.1 to train

---

Under review as a conference paper at ICLR 2023
the models for 200 epochs. The learning rate is reduced with cosine function. And data auga-tt mixup applied.
As shown in Table 2, by applying the identity matrix directly, (i) obtains the lowest accuracy of 87.01% % among all cases. Regarding results of (ii) and (ii), both the two settings can make significant improvements of nearly 5.89% and 3.42% from (i), respectively. And IDIZC, make deeper effect than IDIC,- Equipping IDIC, and IDIZC,, IDInit will improve performance further, which demonstrates our modification efficient and practical.
Table 2: Results of the ablation experiment on ResNet-20..
Setting
(ii)
(iii)
(iv)
Accracy 87,01 = 0.29
92.9 0.18
90.43
0.14
93.22
+ 0.05
4.4 DETERMINACY ANALYSIS ON TEXT CLASSIFICATION
We conduct this experiment show random initialization can cause unacceptable uncertainty mea- sured with standard derivation, and IDInit can solve this problem for the inherent determinacy that seeds will not change initial weights when model is determined. We implement text classification on SST2 and SST5 (Socher et al., 2013) and select TexICNN (Kim, 2014), TextRNN (Lai et al., 2015) and Transformer (Vaswani et al., 2017) for comparison. For TexICNN and TextRNN, we use AdaDelta (Zeiler, 2012) optimizer with learning rate 1.0 and adopt Adam (Kingma & Ba, 2015) for Transformer with learning rate le-4. For the embedding layer, we utilize Glove (Pennington et al., 2014) and Word2Vec(Mikolov et al., 2013) to initialize the embedding weights. All models are trained up to 10 epochs. Fixing other the random initialization for 5 times.
except for the initialization, we run all
Table 3: Results of TexICNN and TextRNN SST2 and SST5. The subscript G denotes the embedding layer is initialized by Glove while W indicates Word2Vec. "Default" means the default initialization of models, specifically, Kaiming for TexCCN,, and Xavier for both TexRNN and Transformer. Std values larger than 1.0 are marked in red.
Datasets
Init.
TexCCNNaw
TextNNNow
TransformerGew
Default
81.40:0..e/44.64000
11.6990.se/84.2..a
80.97 1. 28..1..6///333...0
SST2
Orthogonal
48.444o. 41.8640..s/84.6111
82.22 48.229
IDInit
83.14/86.05
83.14/86.22
82.66/85.13
Default
44.68 44.68 16.6
44.27
41.81 41.8|
SST5
Orthogonal
44.91
44.61 44.61 1.6.41.1/46.133..m.mm
43.01 43.01
IDInit
45.66/48.05
47.12/48.83
45.22/46.24
As shown in Table 3, all the initialization methods can work normally. However, default random initialization obtains the lowest accuracy in both SST2 and SST5. Orthogonal initialization only performs slighter better than IDInit with TextRNN on SST2, and may cause a bad result as 42.35% with TexICNN on SST5. On the contrary, the proposed IDInit can achieve the highest accuracy in almost all the conditions, which shows the sufficient efficiency of IDInit. Furthermore, as deterministic method, IDInit offers the ability to develop fully deterministic training framework as well as improve performance for easy-to-analyze structures.
4.5 IMAGE CLASSIFICATION ON IMAGENET
In this experiment, we use ViT-B/32 (Dosovitskiy et al., 2021), ResNet-50/152 (RN-50/152) and Se-ResNet-50 (SRN-50) as backbones ImageNet. For ViT-B/32, the optimizer is AdamW with learning rate le-3 and a weight decay 5e-2. The training epochs 300. We use 30 epochs for warm- up. For RN-50/152 and SRN-50, the optimizer is SGD with learning rate le-1 and weight decay
8

---

Under review as a conference paper at ICLR 2023
le-4. The epoch for training is 90. We use epochs for warm-up. For all models, the batch size is 1024, and we apply data-augment including cutmix (Yun et al., 2019) with a = 1.0, mixup (Zhang et al., 2018) with 0.8, the switching probability is 0.5 and label smoothing with 0.1.
0.72
0.30
0.1
20.03
0.5
505
205
6-4
20
20.4
0.3
R00
802
033
350
150 350
29
40
(a) VIT-B/32
(b) RN-50 (Adamw)
(c) RN-50
(d) SRN-50
(e) RN-152
Figure 7: Results on ImageNet. "Default" means the default initialization of models. RN-50 (Adamw) means that ResNet-50 is trained with the same optimizer Adamw as the ViT-B/32.
Results are shown in Figure 7. On three types of networks, i.e., VIT, ResNet and Se-ResNet, and multiple depths, IDInit always achieves faster convergence and better performance than the baseline. And when training RN-50 with Adamw, the convergence of IDInit is still consistently fast. This experiment shows the good practicability and promising probability IDInit, which is beneficial to the artificial intelligence community.
4.6
ON WUDAOCORPORA
Pre-training plays an important role in various applications. In this experiment, we pre-train GPT with 8 and 96 layers from scratch on the large-scale text dataset WuDaoCorpora (Yuan et al., 2021) which has 200 GB training data and 72 billion Chinese characters. The optimizer is Adam with a learning rate 1e-3 and weight decay 5e-4. The batch size is 1536. For 8-layer GPT, steps. For 96-layer GPT, we run 13k steps. This experiment is conducted on 32 NVIDIA V100s.
70k
Stop
Stap
(a) 8-layer GPT Loss
(b) 96-layer GPT Loss
Figure 8: Results of GPT on WuDaoCorpora. "Default" means the default initialization of models for GPT. IDInit is our identical initialization.
As shown in Figure 8, IDInit achieves faster convergence on both 8 and 96 depths. Additionally, as Figure 8(a), the random initialization's 's loss unexpectedly increases sharply around 54k step, indicating unstable training. By contrast, IDInit steadily be trained normally. This experiment shows good convergence and stability of IDInit, which can benefit pre- treitraining to some extent.
5 CONCLUSION
In this paper, we introduce identical initialization (IDInit) that is based on the identity matrix. Addressing the problems encountered when developing IDInit, i.e.. dead neurons and performance degeneration, give two concise solutions, namely using zero to wipe off dead neurons and reshap- ing an identity-like matrix into a tensor thus increasing feature diversity, leading to performance wide generality, high stability, and fast convergence,
improvement. With good performance IDInit is promising to be applicable in practice. In the future, we hope that this identical design can motivate the Al community to implement more novel initialization methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust initial- ization for weightnorm & resnets. In NeurlPS, pp. 10900-10909, 2019.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cotrrell, and Ju- lian J. McAuley, Rezero is all you need: fast convergence large depth. In UAI, volume 161 of Proceedings of Machine Learning Research, pp. 1352-1361. AUAI Press, 2021.
Peter L. Bartlett, David P. Helmbold, and Philip M. Long. Gradient descent with identity initializa- tion efficiently learns positive-definite linear transformations by deep residual networks. Neural Comput., 31)3,, 2019.
Yaniv Blumenfeld. Dar Gilboa, and Daniel Soudry. Beyond signal propagation: feature diversity necessary in deep neural network initialization? In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 960-969. PMLR, 2020.
Rebekka Burkholz and Alina Dubatovka. Initialization of relus for dynamical isometry. In NeurlPS, pp. 2382-2392, 2019.
Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. NeurIPS, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang. Samuel S. Schoenholz, Ed H. Chi, and Jef- frey Pennington. Dynamical isometry and mean field theory of Istms and grus. CORR, Abs//190108887,, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org 2010.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training ima- genet in hour. CoRR, abs/ 11177706607777 2017.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, pp. 1577-1586. Computer Vision Foundation / IEEE, 2020.
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In ICLR (Poster). OpenReview.net, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In ICCV, pp. 1026-1034. IEEE Computer Society, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778. IEEE Computer Society, 2016.
Xiao Shi Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimiza- tion through better initialization. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 4475-4483. PMLR, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, volume 37 of JMLR Workshop and Conference Pro- ceedings, pp. 448-456. JMLR.org, 2015.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pp. ACI, 2014.
10

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In AAAI, pp. 2267-2273. AAAI Press, 2015.
Hanxiao Liu, Zihang Dai, David R. So, and Quoe V. Le. Pay attention to mlps. In NeurIPS, pp- 9204-9215, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9992- 10002. IEEE, 2021b.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In ICLR (Workshop Poster), 2013.
Dmytro Mishkin and Jiri Matas. All you need is good init. In ICLR (Poster), 2016.
Yu Pan, Zeyong Su, Ao Liu, Jingquan Wang. Nannan Li, and Zenglin Xu. A unified weight initializa- tion paradigm for tensorial convolutional neural networks. /CML, volume 162 of Proceedings of Machine Learning Research, pp.
PMLR, 2022.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, ACL, 2014.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NIPS, pp. 4785-4795, 2017.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. The emergence of spectral univer- sality in deep networks. In AISTATS, volume 84 of Proceedings of Machine Learning Research, pp. 1924-1932. PMLR, 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Expo- nential expressivity in deep neural networks through transient chaos. In NIPS, pp. 3360-3368, 2016.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning deep linear neural networks. ICLR, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631-1642. ACL, 2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR ( (Workshop), 2015.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pp. 1139-11477 JMLR.org, 2013.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. In NeurlPS, pp. 24261- 24272, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998 6008, 2017.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dicksted Samuel S. Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10, 000-layer vanilla convolutional neural networks. In ICML, volume 80 of Proceedings of Machine Learning Re- search, pp. 5389-5398. PMLR, 2018
Greg Yang and Samuel S. Schoenholz. Mean field residual networks: On the edge of chaos. In NIPS, pp. 7103-7114, 2017.
11

---

Under review as a conference paper at ICLR 2023
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang. and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 68, 2021.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. IEEE, 2019.
Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR (Poster). OpenReview.net, 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR (Poster). OpenReview.net, 2019.
Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing residual networks with only zeros and ones. CoRR, 10.21161..26,,, 2021.
12

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
A DInit Details
13
Full IDInit Scheme
A.2 Trainability Analysis Non-Squared Matrices.
A.3 Implementing IDInit Attention Layer in Transformer
14
B Detailed Settings of Experiments
14
B.I Details of Validation Non-Residual Convolution Experiment
14
Details Validation on Residual Convolution Experiment
B.3 Details Ablation Experiment
15
B.4 Details of Determinacy Analysis on Text Classification Experiment
15
B.5 Details of Image Classification on ImageNet Experiment
15
B.6 Details of Pre-Training WuDaoCorpora Experiments
16
Additional Experiments
C.I. Validation on the Lincar Structure
16
C.2 Analysis on Variance Propagation
16
C33 Analysis Weight Distribution
17
C.4 Analysis input-output Jacobian
18
A IDINIT DETAILS
A.1 FULL IDINIT SCHEME
Here, we show the full DDnnt scheme in Figure 9.
D
D
Desi
Des
2
Identity Intiaaizzatinn
Das
2D,.1
2D,t Dr
Dust <. D, 2Dus
Das Dr
Duv
Preserving Intiialization
Figure 9: Illustration of IDInit with all conditions.
13

---

Under review as a conference paper at ICLR 2023
A.2 TRAINABILITY ANALYSIS OF NON-SQUARER MATRICES.
The magnitude of signals transiting in layers is usually related to stability, and the variance of sig- nals be good indicator of the magnitude according the variance-control mechanism. Obvi- ously, IDI, can maintain forward signal variance. Therefore, we explore the trainability from the angle of the gradient backward procedure. Prior to that, consider a network x(() = f(x(0) = g((-1))((1-2) (00)) the gradient relationship can be formulated as OC
(z( = g(0) 0x(++1)
dc ((t)T g(i+t))
.(. Oc dc dC a
(0)
(14)
Oz(t))
Ozlit1)
where x((( E D g E IR D and L denotes loss of training. Since Vi, Di+1 Di. the training process will be stable as that gradients transit identically. Here, we pay attention to for there is activation function.
non-square conditions.
dc E (1,2..... 1}) contains elements of O, which indicates that must contain 0 values, causing failed updating. However, this is fake dead phenomenon, since all weights will be trained after several updating steps. At the first training step, apt2 Dc = For both Butty and ((4-1) not definitely containing O values, g(L-1) can be updated normally and deviates from identity-like form. Therefore, ac au!!-11 can get normal updating in the next step. As a result, all weights can be updated normally.
Vi, Di+1 < D, then azl: 86 BBaT
If Vi, Di+1 D, then the variance of gradients will increase from layer to layer. However, this will Dit1 0²(
ac not make trouble for training. For (r(() , its variance 0²( OC variance
(a )
dc Ba(t+n)) Therefore, the
ac can be calculated as 0r
0² Du
Uc
D1 D2 Do DI
(
DL DE
Oc dz(L)
dxlly) (
DL D Oc Do
(15)
Dr.-1
According to Eq. (15), the gradient will not explode in the backward procedure.
A.3 IMPLEMENTING IDINIT ON ATTENTION LAYER IN TRANSFORMER
In this part, show the way initialize the attention layer with IDInit. Prior that, formulating an attention layer as
(u, K, V) softmax QW K
Vd JWWWWO
(16)
where is the query matrix, K means the key matrix, V denotes the value matrix, Wº, WK and W represents the weights for Q, K, and V respectively, and 0 is the output transformation. Following the instruction of IDInit in Sec. 3, we firstly use IDI, to initialize WQ, W W and wo, And then, we use IDIZ, to initialize the last fully-connected layer Wo The t and are consistently set with the paper content and le-6, respectively.
B DETAILED SETTINGS OF EXPERIMENTS
In this paper, for ReLU activated networks, is set to v2 for the first layer in network and I for other IDI, IDIC, initializing layers, while for tanh-activated networks, all IDI, is set to 1, and is le 6 for all IDIZ./ / IDIZC, initializing layers.
B. DETAILS OF VALIDATION ON NON-RESIDUAL CONVOLUTION EXPERIMENT
In this experiment, we use AllConv (Springenherg et al., 2015) which consists of nine convolutional layers as the backbone network. We show the structure of AllConv in Table 4. The dataset is Cifar 10. The optimizer is Stochastic Gradient Descent (SGD) with momentum 0.9, weight decay 5e-4, and learning rate le-1. The learning rate scheduler adopts warm-up cosine reduction strategy. We run the model in 300 epochs on one Nvidia A100. We adopt Kaiming initialization and IDInit w/o IDIC, initialization for comparison. Since there is no residual connection, IDIZC, function in this experiment. For each initialization, we have run them with 0, 10, 20, 30, 40, 50, and 60 warm-up epochs. The experiment is conducted one Nvidia A 100.
do not consider the
14

---

Under review as a conference paper at ICLR 2023
Table 4: Architectures of the tensorial All-Conv networks. Window means the convolutional kernel
window size. Channels indicate Cin and Cout of a standard convolutional kernel C E R X Xkxk The avg pool denotes the average pooling operation.
Layer
Window
Channels
conv
3x3
3x 96
conv2
3x3
96x 96
conv3
3x3
96x 96
conv4
3x3
96x 192
conv5
3x3
192x 192
conv6
3x3
192x 192
conv7
3x3
192x 192
conv8
1x1
192x 192
192x 10
conv9
1x1
avg pool
B.2 DETAILS OF VALIDATION ON RESIDUAL CONVOLUTION EXPERIMENT
In this experiment, we validate the proposed initialization with the comparison with existing initial- ization, including (1) Fixup; (2) Skiplnit; (3) ReZero: (4) Kaiming: (5) Zero y (Setting the scale in Batch Normalization (BN) to O). We use ResNet-56/1 10 as backbones on Cifarl0. To verify the universality, we two settings, namely w/ and w/o BN. For analyzing convergence,
adopt both SGD and Adam optimizer for updating models. We set SGD, with the momentum 0.9, the weight decay 5e-4, and the learning rate 0.2. For Adam, the learning rate 0.001, 81 is 0.9 and B2 is 0.999, We train models for 200 epochs. The learning rate is reduced with cosine function. The experiment is conducted on one Nvidia A100.
B.3 DETAILS OF ABLATION EXPERIMENT
The dataset is Cifarl0 and the backbone is ResNet-20. We choose SGD with momentum 0.9, weight decay 5e-4, and learning rate 0.1 train the models for 200 epochs. The learning rate is reduced with a cosine function. And data-augment mixup is applied. The experiment is conducted on one Nvidia A100.
B.4 DETAILS OF DETERMINACY ANALYSIS ON TEXT CLASSIFICATION EXPERIMENT
We also explore performance networks on text classification datasets including SST2 and SST5 (Socher al., 2013) and we select TexICNN (Kim, 2014), TexRNN (Lai et al., 2015) and Trans- former (Vaswani et al., 2017) for comparison. For TexICNN and TextRNN, we use AdaDelta (Zeiler, 2012) optimizer with a learning rate 1.0 and adopt Adam (Kingma & Ba, 2015) for Transformer with a learning rate le-4. For the embedding layer, we utilize Glove (Pennington et al., 2014) and Word2Vec(Mikolo
al., 2013) to initialize the embedding weights. All models are trained up to 10 epochs. Fixing other hyper-parameters except for the initialization, we run all the random initialization for 5 times. The experiment conducted on one Nvidia A100.
B.5 DETAILS OF IMAGE CLASSIFICATION ON IMAGENET EXPERIMENT
In this experiment, we use ImageNet for validation. We use VIT-B/32 (Dosovitskiy et al., 2021), ResNet-50/152 (RN-50/152) and Se-ResNet-50 (SRN-50) as backbones. For ViT-B/32 that inputs 32 x 32 patch window, the optimizer is Adam W with a learning rate le-3 and weight decay of 5e-2. And the batch size is 1024. The epoch for training is 300. We use 30 epochs for warm-up. The input image size is 224 x 224. The dropout rates of the embedding layer and the network layer are all 0. For RN-50/152 and SRN-50, the optimizer is SGD with learning rate le-1 and a weight decay of le-4. And the batch size is 1024. The epoch for training is 90. We use 9 epochs for warm-up. The input image size 160 x 160 for the front 35 epochs and 224 x 224 for the remaining epochs. For all models, apply data-augment including cutmix (Yun et al., 2019) with 1.0, mixup (Zhang
15

---

Under review as a conference paper at ICLR 2023
et al., 2018) with 0.8, the switching probability is 0.5 and label smoothing with 0.1. The experiment is conducted on Nvidia A100.
B.6 DETAILS PRE-TRAINING ON WUDAOCORPORT EXPERIMENTS
In this experiment, conduct experiments on WuDaoCorpora (Yuan et al., 2021) which has 200 GB of training data and 72 billion Chinese characters. We use GPT with and 96 layers as backbones. The optimizer is Adam with learning rate le-3 and weight decay of 5e-4. And the batch size is 1536. We use 20k steps for warm-up. For 8-layer GPT, we run 70k steps. For 96-layer GPT, we run 13k steps. The experiment conducted on 32 Nvidia V100.
C ADDITIONAL EXPERIMENTS
We provide additional experiments further validate IDInit. and are set the same Sec. B.
C.I VALIDATION ON THE LINEAR STRUCTURE
This experiment is conducted on MNIST. use five linear layers named Liner-5 whose hidden layers all of dimension 512. The optimizer is SGD with momentum 0.9, weight decay 5e-4, and a learning rate le-1. The learning rate scheduler adopts a cosine reduction strategy. We run the model in 30 epochs which consist of Linear-5, and tanh and ReLU activation functions, respectively. The experiment is conducted on one Nvidia A A100.
Nvidia A100. We both consider Linear-5-tanh and Linear-5-ReLU
Table 5: Results of 5inear-5 on MNIST. "Default" means the default initialization of models where Xavier is for Linear-S-tande and Kaiming is adopted for Linear-5-ReLu
Init.
Linear-5-tanh
Linear-5-Rel.U
Default
98.26
98.21
IDInit 98.32
98.4
As shown in Table 5, IDInit can achieve the highest accuracy in both different tanh and ReLU conditions. The results show the ability of our proposed method to train a model with only fully- connected layers.
C.2 ANALYSIS VARIANCE PROPAGATION
Here conduct experiment Cifar10 demonstrate data-flow will keep stable. We types of networks: (1) FC: 10-layer fully-connected layers; (2) ResFC: 10 residual blocks (two fully- connected layers block); (3) Conv: 9-layer AllConv in Sec. B.I: (4) ResConv: 10 residual blocks (two convolutional layers in a block). For (I) and (2) two fully-connected networks, we reshape Cifar10 data as X E R22x96 input and does not use any activation function. For (I), hidden lengths are f200, 400, 600, 800, 1000, 1000, 800, 600. 400, 200). For (2), hidden lengths are all set 96. For (3) and (4) two convolution networks, we directly input images them, and use ReLU as the activation function. For (3), we directly use AllConv as shown in Table 4. For (4), we first use convolution to transfer an image to 16 channels, and then set the channels of all convolution within residual blocks to 16. For comparison, terms of the activation function. We also employ noises with 0 mean, and {0.00, 0.01, 0.10, 1.00} for comparing robustness. In the experiment, conducted on one Nvidia A100.
Xavier for (1) and (2), and Kaiming for (3) and (4) in
500 rounds for each model. The experiment is
Results shown in Figure 10. The regular methods Xavier and Kaiming can only work non- residual networks. On residual networks, they both cause giant standard derivation. leading to in- stability. By contrast, the proposed IDInit consistently transit data-flow in an appropriate scale
16

---

Under review as a conference paper at ICLR 2023
5.0
7.5
Actratton Wive
7.5
5.2
Brast Die
5.2
7.5
0141
5.0
25
2.5
2.5
0.0
-7.3
-7.
-20.0
-20.0
-10.0
-20.0
(a) FC-0.00
(b) ResFC-0.00
(c) Conv-0.00
(d) ResConv-O.00
10.0
20.0
Datact
Deleat
2.5
2.5
0.0
0.5
0.3
0.0
-25
-2.5
-2.5
-2.
-5.
-5.0
-7.5
-00.
-10.9
Layar
Caysr
Layo
Layre
(c) FC-0.01
(f) ResFC-0.01
(g) Conv-0.01
(h) ResConv-O.01
10.0
30.5
10.3
10,0
Actvetion 10.
7.5
5.0
Bivd
-25
-25
-2.5
-2.5
-5.0
-5.3
-5.3
-5.0
-7.5
-7.5
-7.5
-7.5
-10.0
10
Layer
Layor
Layar
Laysr
(i) FC-0.10
( ResFC-0.10
(k) Co
(I) ResConv-O...
10.0
10.5
10.5
10.0
Rcivadtinn Value
5.0
75
ACtvodion Wo.e
75
5.2
5.2
7.5
De
1841
5.0
7.5
Defiuit
1094:
2.5
2.5
-25
-5.0
-5.3
-5.0
-75
-73
-7.3
-7.5
10.0
10
-10.3
10
-0.0
-100
20
(m) FC-1.00
(n) ResFC-1.00
(0) Conv-1.00
(p) ResConv-1.00
Figure 10: Results of the analysis on variance propagation. The numerical value after the model name means the standard derivation of the noise. "Default" means the default initialization of models, specifically, Xavier for FC and ResFC, and Kaiming for Conv and ResConv. The default methods can only work on non-residual networks FC and Conv, however, fail on residual networks ResFc and ResConv, for cause instability with giant standard derivation. By contrast, IDInit can consistently transit data-flow in an appropriate scale on all models and various noises, which shows sufficient robustness, and can provide models with stable and efficient training.
on all models and various noises, which shows sufficient robustness, and provide models with stable and efficient training.
C.3 ANALYSIS ON WEIGHT DISTRIBUTION
In this experiment, we conduct an experiment on Cifar10 with ResNet-20 to show the weight dis- tribution of IDIni. We use SGD optimizer with learning rate 0.2, and weight decay 5e-4. The batch size 1024. Training epochs are 200. The learning rate is reduced with a cosine function. The experiment is conducted on one Nvidía A100.
The results are shown in Figure 11, weights initialized with IDInit are almost full of zero at the be- ginning, while Kaiming more zero values than Kaiming, which is beneficial for memory occupation since a 0 value will not cost memory space.
Gaussian distribution. At the end of the training, IDInit still contains
17

---

Under review as a conference paper at ICLR 2023
35
50
3.2
3.0
50
2.5
2.5
25
1.0
0.5
0.0
9.2
0.0
06
0.0g
Weight
Veigh Vakue
(a) Kaiming-Ei E4
(b) Kaiming E24
(c) Kaiming-Elnn E104
(d) Kaiming-EIIe E194
30
2.5
25
25
2.0
2.0
15
15
0.00
0.
09
Weighn Wase
Weight Vaice
WHig Wire
(eight Vale
(e) IDInit-E4
(I) IDInit-E24
(g) IDInit-E104
(h) IDInit-E194
Figure 1I: Histograms of the first convolution weights in ResNet-20. "E" means the epoch index. IDInit contains more zero values each epoch compared with Kaiming initialization.
C.4 ANALYSIS ON INPUT-OUTPUT JACOBIAN
Here we conduct an experiment on Cifarl0 with ResNet-130 to demonstrate IDInit follows the dynamical isometry. We remove batch normalization for the more clear difference between IDInit and Kaiming. We use an Adagrad optimizer with learning rate 0.1. The batch size is 100. The activation is ReLU. The experiment is conducted on one Nvidia A100.
Arg.s
0.8
0.6
0.
0.2
0.2
02
(g/4
0914
(a) Kaiming-E1
(b) Kaiming-E2
(c) Kaiming-Fi E3
(d) Kaiming-PFi
1.0
1.0
0.6
3.5
0.6
0.6
0,2
02
0
(g14.)
(c) [DInit-EI
IDInit-E2
(g) IDInit-E3
(h) IDInit-E4
Figure 12: Histograms log singular values (log(A..)) for the input-output Jacobian. "E" means the epoch index. Compared with Kaiming initialization, DDInit has significantly smaller squared singular value which can achieve a faster reduction of the loss.
As shown in Figure 12, Kaiming initialization cause high squared singular value x, reaching more than 2000. Compared to Kaiming, IDInit only derives around 1, indicating correspondence to the dynamical isometry. In addition, the loss of DDnnt decreases faster than Kaiming, which shows good convergent ability.
18


Reviews:
[{"review_id": "j6jgAZU9LX", "reviewer": "Reviewer_st9W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to initialize each linear layer of a deep neural network to be identity so as to accelerate training.", "summary_of_the_review": "This paper proposes identity initialization for deep neural networks to accelerate the training. I found the idea lack of novelty and the claim of solving the dead neuron problem insensible.", "strength_and_weaknesses": "Strengths: \n1. The experiments are done on many datasets.\n2. The code is attached for reproducibility.\n\nWeaknesses:\n1. It has been shown (Thereom 5 in [1]) that the identity initialization may not lead to convergence for deep linear neural networks. \n2. For residual layer, this work proposes to initialize the weight matrix with tiny values so as to avoid the dead neuron problem. This is not sensible. Because of the shortcut connections, weights with zero initialization still receive non-zero gradients.\n3. The overall idea, identity initialization, is not novel. For example, see [2]. The proposed variant in this paper is technically trivial.\n\nReferences:\n[1] Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks, Peter L. Bartlett, David P. Helmbold, Philip M. Long, ICML 2018.\n[2] A Simple Way to Initialize Recurrent Networks of Rectified Linear Units, Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton, arXiv 2005.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8tWPI2-j_L", "reviewer": "Reviewer_QUve", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new initialization scheme for both vanilla convnets and convnets with residual structures. This initialization is a simple and special case to maintain dynamic isometry at initialization time. Specifically, they initialize the weight matrix to be identity. The authors also propose a spatial shift scheme to improve the representation learning ability. Using this initialization, they can train nine-layer vanilla convnets and 110-layer residual networks on CIFAR. It is also shown this initialization method is helpful for transformer training.", "summary_of_the_review": "In summary, this paper is interesting as it shows simply initializing the kernels to be identity can make vanilla networks trainble. However, the writing, experiments, and presentation need to be improved a lot. I cannot recommend acceptance for this paper before this improvement.", "strength_and_weaknesses": "Strength:\n1) This paper is motivated by the dynamic isometry formulation and propose a very simple initialization (identity initialization). The simplicity is good and empirically helpful on varies settings.\n2) The proposed two techniques on improving the feature diversity are interesting.\n\nWeakness:\n1) Missing Reference: This paper misses discussion with DiracNet [1] and ISONet [2]. DiracNet reparameterize the convolutional kernals and essentially initialize it to be an identity. ISONet explicitly initialize the conv kernels to be identity both in vanilla convnets and residual convnets. It is necessary to discuss the differences and advantages over these previous works.\n2) Some terms and examples are confusing.\n- the robustness in this paper actually means insensitivity to hyperparameters. However, this term in machine learning usually means the network is insensitive to input noise.\n- Table 1 is also confusing. Most of the entries in the “w/ BN” category should be put in the “w/o” because Zero \\gamma, Fixup, Skip Init, ReZero are designed for training residual networks without normalization.\n- Figure 7 can be replaced by a Table becuase the current figure does not have caption and it is not straightforward to get the point from the figure.\n- The experiment setting and conclusion of section 4.4 is also confusing. Is the proposed initialization has any randomness? It seems the initialization is deterministic. In this case, why the std is not 0 in Table 1?\n3) The central message of the paper seems to be unclear. The authors mix two information together: 1) the trainability of network without batch norm; 2) the proposed method improves over the original initialization. The first part is well-motivated because BN has many disadvantages. However, the proposed method only shows the trainability of a network with 100 layers on CIFAR. The ImageNet experiments are based on networks with varies normalization layers and the proposed method only has marginal improvement over the baselines.\n\n[1] Zagoruyko et al. DiracNets: Training Very Deep Neural Networks Without Skip-Connections.\n[2] Qi et al. Deep Isometric Learning for Visual Recognition. ICML 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "72_rEjuIYzf", "reviewer": "Reviewer_Wd9d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose Identical Initialization (IDInit) that initializes weights using identity matrix and its variants. They consider how to initialize non-square matrices, residual structures and convolutional operations using identity-like methods. Empirical evaluation demonstrates its good performance and fast convergence. ", "summary_of_the_review": "Identical initialization is a very interesting topic and IDInit looks promising. My major concerns are its effectiveness (training degeneracy problem), necessity (motivations behind IDIZ and IDIC) and novelty (compared to ZerO). I would like to discuss these issues with the authors, and I am happy to raise my score if the authors can address my concerns.", "strength_and_weaknesses": "The paper is well-written and easy to follow. The authors also present extensive empirical evaluations to support their methods.\n\nHowever, I have a severe concern regarding the effectiveness of IDInit. The identical initialization has been well studied in ZerO paper [1], where the authors theoretically prove that a simple identical initialization leads to training degeneracy in practical networks. The major reason behind this training degeneracy is that for a dimension-increasing matrix (e.g., the $D_{i+1} > D_{i}$ case in the current paper), a simple \"identity + zero-padding\" causes the rank constraint ((ii) in Theorem 1 in ZerO). \n\nTo initialize dimension-increasing matrix, IDInit replicates the inputs to span over the output dimension. However, this causes a similar rank constraint (i.e., a symmetric problem) as each replica in the dimension-increasing matrix receives the same gradient (excludes some special cases). ConstNet paper [2] also shows the degeneration caused by insufficient feature diversity (i.e., the symmetry), which is related to the replication here. In addition, the authors only perform trainability analysis from a signal propagation perspective, without considering the potential degradation caused by the replication. \n\nI highly recommend the authors to empirically evaluate if IDInit can pass the simple rank verification in Figure 3 of ZerO paper. On the other hand, I suspect the surprisingly good empirical evaluation of IDInit comes from randomized operations (e.g., dropout) in networks, which directly breaks the training degeneracy. I suggest the authors to carefully consider this training degeneracy issue, as it affects the effectiveness and universality of the proposed method.\n\nOther comments:\n\n1. The need of zero preserving initialization (IDIZ) is unclear. The authors motivate IDIZ by arguing that Fixup or ZerO method will not work if the multiplier in batch normalization is initialized as 0. However, this is not a standard case because the multiplier in BN is usually initialized as 1. The motivation also requires a special case where BN is applied after the last conv layer. These reasons can not convince me why IDIZ is really needed. Do authors initialize the multiplier in BN as 0 in their empirical evaluation? If so, I suggest the authors to provide more details about this case.\n\n2. How does IDIC compare to the simple IDI in convolution case? The authors mention that IDIC is used to improve feature diversity but it's unclear why it is needed. ZerO paper [1] directly utilizes IDI in convolution (Algorithm 2 in ZerO) and it works well.\n\n3. Comparison to ZerO. Since ZerO is the most relevant work to IDInit I believe, I suggest the authors to empirically compare them, including both performance and determinacy analysis (since both methods claim the benefit as being deterministic).\n\n[1] https://arxiv.org/abs/2110.12661\n\n[2] https://proceedings.mlr.press/v119/blumenfeld20a.html", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-wUk1UcCWa9", "reviewer": "Reviewer_TjT6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The reviewed work proposes \"identical initialization\" as a novel initialization scheme for neural networks. The guiding principle of this initialization scheme is to ensure that each layer, at initialization, propagates activations identically.\n\nThe motivation behind this choice is that the resulting isommetry property of the network prevents vanishing or exploding gradients and thus allows for the stable training of deep architectures. \n\nNumerical experiments on classification for CIFAR10 and Imagenet show improvements due to the new initialization scheme. ", "summary_of_the_review": "In summary, the empirical results suggest that the authors may be up to an interesting result. However, the writing, presentation, and comparison to existing work need major improvement. In its present state, the paper is not ready to be published at a major ML conference.", "strength_and_weaknesses": "### Strength\nThe main strength of the paper are the empirical improvements apparently achieved by the proposed initialization scheme.\n\n### Weakness\nThe first-order weakness of the paper is the poor quality of its writing. There are numerous grammatical mistakes and vague language such as \"Therefore, if all values are set to 0, it can obtain non-zero gradients of a suitable magnitude.\" Due to the above, despite numerous attempts, I am still not entirely confident if I have correctly understood the approach proposed by the authors. \n\nA second weakness (possibly related to the first) is that the motivation for the proposed approach over FIXUP and ZerO is not clear. In particular the fixup approach, that similarly initializes the residual branches as products of identities and zero seems closely related. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject