IDInit: A Universal and Stable Initialization Method for Neural Network Training
Authors: Yu Pan
Chaozheng Wang
Zekai Wu
Qifan Wang
Min Zhang
Zenglin Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Initialization
Idetity Matrix
Dynamic Isometry
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: An initialization method for fast and stable training of deep neural networks based on the identity matrix.

Abstract: Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, initialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g., Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that is achieved only by a zero weight, which may be harmful to training. Addressing this concern, we introduce fully identical initialization (IDInit), a novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs a padded identity-like matrix to overcome rank constraints in non-square weight matrices. Furthermore, we show the convergence problem of an identity matrix can be solved by stochastic gradient descent. Additionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit is a straightforward yet effective initialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.

Published as a conference paper at ICLR 2025
IDINIT: A UNIVERSAL AND STABLE INITIALIZATION METHOD FOR NEURAL NETWORK TRAINING
Yu Pan' Chaozheng Wang² Zekai Wu? Qifan Wang' Min Zhang? Zenglin Xu
l Huawei Noah's 's Ark Lab The Hong Kong Polytechnic University *MetaAl Harbin Institute of Technology, Shenzhen Fudan University Shanghai Academy of AI for Science
"The Chinese University of Hong Kong
iperryuul gmail com com czwang238cse cuhk. edu. .kk
zenglinggmen gma
com com
ABSTRACT
Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, ini- tialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g.. Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that achieved only by zero weight, which may harmful to training. Ad- dressing this concern, introduce fully identical initialization (IDInit), novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs padded identity-like matrix to overcome rank con- straints in non-square weight matrices. Furthermore, we show the convergence problem of identity matrix can be solved by stochastic gradient descent. Addi- tionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit a straightforward yet effective ini- tialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.
INTRODUCTION
Deep neural networks have attracted significant attention due to their versatility in various applica- tions (He et al. 2016; Li et al. 2021 Wang et al. 2023). Behind these successes, initialization methods play crucial role in promoting stable and fast-convergent training processes for net- works (Sutskever al.) 2013] Arpit al., 2019; Pan et al. 2022 2024). Usually, initialization methods make effects by controlling the magnitude of signals. For example, Xavier (Glorot & Ben- gio 2010)) initialization originally proposed to maintain signals in the non-saturated region of the sigmoid activation function by restricting signal variances, which greatly solved the difficulty of training. Then, Poole et al. (0166) propose to initialize network weights by constraining signals on the edge of chaos through dynamical isometry, which can further benefit the network training. Later, Hardt & Ma (2077) analyzed the optimization landscape of linear residual networks, and found that weights that transit identity in layers can help networks converge fast as their F-norm is close that of the final converged weights. And identity transition also corresponds to isometry theory (Zhang al. 2019), thereby, contributing avoiding gradient explosion and diffusion.
An instance of preserving identity across neural network layers, known as "identity-control." is de- picted in Figure [ and formally expressed as Y = X. This type initialization can be implemented by setting specific weights (e.g., W₂) to O, thereby ensuring zero output in the sub-stem, as eluci- dated by Hardt & Ma (0117). This approach, how- ever, poses challenges in configuring the remaining
Y= W
Wi
W2 Pup
XER"
Y YE
Douu
Ideetity-onttok: Y
Figure 1: A case of identity-control initializa- tion, which sets W2 0 to satisfy Y = X.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Farder
0.14
0.05
1
0
Inderard
0.06
0.04
0
0.06
-0.08
0
Random
Hadamard
Partal Identity
IDint
Epoch
Epoch
(a) Initialization methods.
(b) Square Loss.
(c) Rectangle Loss.
Figure 2: Analyzing effect of initializing W1 while W2 0.
The experiment uses Cifar10 and (a) The initialization methods for W1 in
blocks in Figure [ and more details are in Appendix C.5
a rectangular format. Fixup: "Random": ZerO: "Hadamard"
And "Partial Identity" and "IDInit"
denote padding 0 and an identity matrix, respectively. (b) Set W1 2e2002200 and W₂ E 2
2440x200 as square matrices. "Identity-1" represents configuration where only one weight is initialized as O. Interestingly, while "Random" and "Hadamard" methods may outperform "Identity- I" in initial training epochs due to more network weights, they are hard to capture the inductive bias of "Identity-1". resulting in convergence difficulties. In contrast, IDInit can effectively leverage the training dynamics associated with "Identity-1" [c) Set W1 28x00220 and W2 E 2240x280 as rectangle matrices. "Default" means W1 and W2 are initialized with Xavier. However, "Default" proves ineffective for training, it conflicts with dynamical isometry. Furthermore, even though "Partial Identity" exhibits the capability to transmit partial signals, it performs poorly due to rank constraint issues. Finally, IDInit maintains well-training conditions by padding the identity matrix.
weight W1. Previous work such as Fixup (Zhang et al. 2019) and ZerO (Zhao et al. 2022) initial- ize W1 using the Xavier and Hadamard methods, respectively. These initializations can adversely affect the inductive bias already established by setting W₂ = O, a setting beneficial for training. As evidenced in Figure 2 both Xavier and Hadamard methods cause difficulties in achieving con- vergence. Observing this, we propose initializing W1 with an identity matrix 1, which retains the inductive bias 2W2 Wg. Moreover, 1 also achieves dynamical isometry in the sub-stem layer as discussed by Zhao et al. (2022). Figuree[demonstrates that using an identity matrix significantly aids in training convergence. Nonetheless, the practical application of an identity matrix faces two primary obstacles. First, an identity matrix requires equare-shaped weights, a condition seldom met practical networks. While partial identity matrix (by padding 0 to an identity matrix) offers workaround, it leads to rank constraints issues (Zhao et al.) 2022) when the output dimension exceeds the input dimension, impairing network generalization. The second obstacle concerns the convergence capability. As Bartlett et al. (2019) pointed out, weights initialized with an identity matrix are difficult to converge to the ground truth, of which eigenvalues contain negative values. This convergence problem is important it indicates limited universality of applying an identity matrix as an initialization method.
IDIii. In light of the preceding discussion, we aim to address these two major obstacles. To handle a non-square matrix, we pad a new identity matrix in adjacency to an identity matrix. We theo- retically demonstrate this operation can resolve the rank constraint problem. Then, alleviate the replica problem induced by this padding schemn, we impose loosening condition on the padded adentity-like matrix. Turning to the matter of convergence, we conduct an experiment to analyze it. Interestingly, we find that the convergence problem can be solved by adding a moment in an opti- mizer le. the stochastic gradient descent optimizer), which the most general setting for training neural networks. By introducing the identity-like matrix into the identity-contro framework, we implement fully identical initialization (IDInit), which ensures identity transition across both main and sub-stem layers. Moreover, we explore two additional techniques for improving the universality of IDInit and the identity framework:
(1) Higher-order Weights: An identity matrix is 2-D array and is necessary to consider an efficient method to transfer the identity matrix to a higher-order weight (c.g., a 4-D convolution). A previous strategy is to keep identity along the channel (see Sec. 3.1.3). However, this causes diversity loss in channels, which is harmful to performance. To remedy this shortage, we keep identity in patches alternatively for more diversity in channels to achieve improvement.
(2) Dead Neurons: As an identity-control method, IDInit sets the last layer of the sub-stem to for transiting identity in the main branch. However, dead neuron problem is possibly caused by
2

---

Published as a conference paper at ICLR 2025
this setting, especially for residual convolutional networks Zhang et al. 2019 Zhao et al. 2022 Addressing this, we select some elements to a small numerical value E to increase trainable neurons as in Figure 3
To our knowledge, IDInit the first successful trial to maintain identity in both main- and sub- stems by breaking the rank constraints, which promise the expressive power of IDInit. Then, address the replica problem by adding small noise while maintaining the dynamical isometry. By further proposing modifications to CNNs and solutions dead neuron problems, we have signifi- cantly improved accuracy of classifying Cifar10 on ResNet-20 by 3.42% and 5.89%, respectively. (see Section 4.2). Note that, although the identity matrix is used as initialization in prior work, was only used for square matrix, e.g., [L et al. (2015) set andenn-tohiidee layer in a recurrent neural network with an identity matrix for better performance. IDInit is novel for the consideration of non-standard situations, e.g., non-square matrix. On ImageNet, compared to the default ran- dom initialization, IDInit demonstrates superior performance, achieving an average improvement of 0.55%, and facilitates faster convergence across various settings, reducing the required training time by an average 7.4 epochs. IDInit can accelerate the training procedure of BERT-Base, manifesting an 11.3% reduction in computational cost. Therefore, our approach yields consistently significant advantages in the training of neural networks.
RELATED WORK
Consider an L-layer residual network, each residual block of which consists of a residual connection and residual stem that refers to the component excluding the residual connection. Assuming each residual stem contains two parameters, and the network's s input signal is denoted as x00), the i-th layer can be formulated as
(6+1) = a(I + ((:00)(()))))(())
(1)
where a(-) denotes the activation function, ( means an input of i-th residual block in network, I is an identity matrix denoting residual connection, and g(i.0) and g(i.1) are weights in the i-th residual stem of residual block.
Dynamical Isometry. Assuming the signal magnitude (e.g., (zx(1)) of each layer changing in
scale a, the last signal magnitude can reach a (e.g., (x(() a (z)1))) making it easy to cause signal explosion and diffusion, especially for large To mitigate this issue, dynamic isometry provides an effective solution. Considering the input-output Jacobian which is defined as
(L)
J.0
Or
the dynamical isometry is achieved when all the singular values of Jio are close to 1. Moreover, with the mean squared singular value of Ji noted as X, Pennington et al. (017)) and Bachlechner et al. (2021) show that X 1 indicates that the model is in a chaotic phase, and back-propagated gradients will explode exponentially. By contrast, X < 1 means a model in an ordered manner that propaapppe gradients vanish exponentially. X is a critical line of initialization, avoiding gradient vanishing or exploding. The isometry can provide sufficient robustness for the network
training (Gilboa et al. 2019 Poole et al. 2016 Yang & Schoenholz) 2017).
Network Initialization. Common initialization methods are Xavier (Glorot & Bengio 2010) and Kaiming initialization (He et al., 2015). Especially for residual networks efficiencc, Hardt & Ma (017) theoretically demonstrates that network training benefits from keeping identity. Le et al. 2015) set hidden-to-hidden layer in a recurrent neural network with an identity matrix for bet- ter performance. Fixup (Zhang et al. 2019) and ZerO (Zhao et al. 2022) successfully initialize ResNets by setting residual stem to (not residual connections) to guarantee the identity of sig- nals. SkipInit (De & Smith
2020) replaces Batch Normalization with multiplier whose value is O. ReZero (Bachlechner et al. 202l directly adds extra parameters value 0 to keep identity, leading
to fast convergence.
Identity-Contai Training Framework. depth by maintaining identity. DiracNet 7agoruyko & Komodakis 2017) maintains an identity
Net2Net (Chen et al. 2016) proposes to expand network
3

---

Published as a conference paper at ICLR 2025
Addition
Shorlout
X
D
D
D
D
Crantty-Presennve Infiiaization
Zero Preserving Initialization
IDI
IDI,
Figure 3: An overview of IDInit, which consists of identity-preserving preserving initialization IDI, and zero- preserving initialization IDIZ, of which dimensions denoted as DJ and 17 and usually set to and le-6 to maintain identity and transit zero. and + mean two adjacent layer indices.
for propagating information deeper into the network. However, suffers from reducing residual connection, causing performance loss. ISONet (Qi et al. (020) is an isometric learning framework that contains an identical initialization (i.e., the Dirac function that is also used in ZerO (Zhao et al.) 2022 by padding 0 in non-square matrix case), and isometric regulation in training. ISONet multiplies O to the residual stem like Fixup (Zhang et al. 2019). ISONet lacks the flexibility for various convolutions as specifies the without normalization, and requires SReLU.
FULLY IDENTICAL INITIALIZATION
The identity-control scheme serves practical initialization framework, with prior studies such as Fixup and ZerO demonstrating success within this paradigm. As depicted in Figure 3 DDInit achieves this scheme with two components: preetity-preserving initialization and zero-preserving initialization, aimed at transferring identity and zero, respectively. We elaborate on the identity- preserving initialization, which involves padding identity matrices, in Section 3.1 and discuss the zero- pere-preserving initialization, which addresses dead neurons, in Section 3.2
3.1 PRESERVING IDENTITY BY PADDING IDENTITY
standard identity matrix can naturally satisfy identity transition. However, non-square situa- tion, this natural advantage is lost. To address this problem, pad the identity matrix identity matrix to a non-square matrix. Specifically, for a fully-connected layer transformed from Eq. (T)
as x. (6+1) (1)) we set the weight g(() TR PRPP to
T, if m jj (mod D!),
(3)
miy
0, otherwise.
The initialization formulated as Eq. is termed as IDI, where IDI means the identical initializa- tion function, and 7 is calculated by considering the activation function, e.g., TReLU = =2 for the ReLU function. As shown in Figure setting 1 can form IDI, initialization.
3.1.1 ANALYSIS ON CONVERGENCE ABILITY OF THE IDENTITY MATRIX
As proposed by Bartlett et al. (2019), weights initialized with an identity matrix face difficulty in converging towards the target when eigenvalues include negative values. This implies potential constraint on the convergence efficacy of the IDInit method. Consequently, we will delve deeper into this issue in the following discussion. According to their study, when layers in neural network are initialized using the identity matrix, all the weight matrices of layers will be symmetric at each step of the training process. This persistent symmetry leads to the weights layers always being the same at any training step, causing the aforementioned convergence difficulty. Interestingly, find that this problem mainly caused by the gradient descent (GD) which uses all the data in one batch, and employing stochastic gradient descent (SGD) of which data in different batches
4

---

Published as a conference paper at ICLR 2025
can be different, effectively break the symmetry in gradients which facilitates convergence, and incorporating momentum can further accelerate the convergence process.
To elaborate this problem, y = 8z, where x E I represents the input, y E Id denotes the output, and 0 E Rdxd xd is the weight matrix. The weight matrix 0 is initialized to the identity matrix I, denoted as o1)) I. For our loss function, we employ the Mean Squared Error (MSE) and learning rate denoted by 7). Consider two training pairs {01, V1 J and {a2, 3/2 sampled from the same dataset D. The network is initially trained with {21, V1 J. and trained with {z2, V2 } in the next step.
present training case for single-layer network expressed as
Being updated after two steps, the final gradient A0() can be calculated as
σ24ž - nriaf 222z + ny1zf x222 - y242
(4)
While 0222 symmetric, x=23, yı m222". and y2uz can be asymmetric. To quantify the magnitude of the asymmetry in A let f2 -n + "//="//2222 y227 denote the asymmetric component. The magnitude of the asymmetry can be calculated as B(Il? - Ω 112)) Assuming 22, 31, 1/2 IR are random vectors with entries that are i.idd Gaussian random variables distributed as NO0 o²), the magnitude of the asymmetry is bounded as
(1+)) S dn + 3dfo4 dr df + 2f s E(Il - 22'11
(5)
As 17 is usually le- 1, and both training pairs {œ1, y1 } and { y2) can be generally normalized to N ~ (0, 1), thereby, the symmetry of the weight can be sufficiently influenced as
(22) = 0(0) 7000)
(6)
When introducing a momentum m initialized to A 0(2) will be updated as
m1) ym + 7))(11)
g(2) = (1) m (1) = ( - ym (0) -)
where y is the coefficient of Therefore, momentum can promote the weight to become asym- metric by accumulating the asymmetry of gradients in steps and impact more when samples are increased. We show that SGD) with momentum effectively resolve the issue of layers being the same in networks initialized with the identity matrix during training, which facilitates the conver- gence process. The completed derivation is provided in Sec. A.2 of the appendix.
As for networks of multiple layers, when their layers are asymmetric, each layer can be updated differently which breaks the convergence problem caused by the same gradients in each step (which is stated in Lemma 5 of Bartlett et al. (2019). ). As illustrated in Figure [0] of the appendix, it is evident that layers trained using SGD are different from each other, with the momentum component amplifying the degree of this difference.
3.1.2 ON RANK CONSTRAINT PROBLEM
ZerO (hhoo et al. 2022) identifies that Humension-inccrasing matrix may face rank constraint problem if padding zero values. In this analysis, we investigate whether padding with an identity matrix results in this constraint.
60-
Padding Zero (P2)
Pedding Idantity
(Pt)
2018
4:30
(a) Padding schemes.
(b) Rank plot.
Figure 4: Two padding schemes and their influ- ence ranks of a layer. We trained a 3-layer net- work on MNIST, and set Do = 768 and Dr 2048. We plot rank (onk))(())) shown in (b) padding identity can achieve more than rank of 768 like Hadamard, while padding zero is limited under 768. The loose condition can lead to better rank performance, however, cannot solve the rank constraint problem of padding zero weight g(0) by padding zeros (PZ) values, the rank constraint problem refers to
Rank Constraint Problem. Consider 3-layer network with weights {g(i)) == where g(0) E R DD ((1) E Dr x Dr,
De Da. where D2 > Do. Dr.- Given an input batch 2())) E DoxN with a size N, the formulation of the i-th layer is ((++)) (()) where e
g(2)
Dx D in (b) As
[2]. Define residual component A0(1) = ((1)
I. When initializing the dimension-increasinn
rank(A0(1) < Da-
(8)
5

---

Published as a conference paper at ICLR 2025
This rank constraint issue signifies performance limitation associated with the initialization method. Intriguingly, our findings indicate that the initialization method IDI, successfully avoids this rank constraint, as detailed in Theorem 3.1 The proof is deferred to Appendix [A.4
Theorem 3.1. If initializing all weights {0l) 322 by IDI, the rank of 20(1 can atlain
rank(A8(1) 2 Do.
(9)
which breaks the rank constraint.
Notably, Theorem I suggests that an IDInit initialized network can break this constraint through SGD without the help of non-linearity like ReLU which is mentioned as necessary in the prior study (Zhao et al. 2022). Specifically, when non-linearity like ReLU not applied, the rank of the middle weight being limited Da only happens at the beginning. After training for several steps, an IDInit-initialized network can break this constraint.
Replica Problem. When recurrently padding the identity matrix, the output features are still repli- cated. According to Blumenfeld et al. (2020). such a replica problem can be solved by adding noise to weights. Inspired by that, loosen the identity condition generate T (T,,,),, while keeping most identity. E, is small value and set to le-6 in this paper. With this loose condition, IDInit can give additional noise to output features and bring more feature diversity. Profiting from the feature diversity, IDInit therefore can increase the rank values as shown in Figure 4(b)
3.1.3 PATCH-MAINTAIN CONVOLUTION
Convolution layers are important structures in deep neural networks. Here, we will explore an initialization pattern for convolution with the identity transition. A convolution kernel is usually defined as C E B4xxxeex where Cin and Cout denote the number of channels of input and output, respectively, and denotes convolutional kernel size. Similar to an identity matrix. Zhao et al. (2022) propose channel-maintain convolution layer that transits identity by setting O-filled C through IDI, (Cn,n..... where N+ and k 2n + As convolutional kernel window size, is usually an odd number. When Cin Cout: the convolution maintains the identity. When Cin >> Cout or Cin < Coats will under-sample and over-sample on an input feature along channel respectively.
Keeping identity is usually considered as efficient way improve model performance, however, we find that this setting can lead to a fatal performance degeneration (see Sec. 4.2
Patch- Mainh-Mainntinn Convolution. Inspired by Han al. (2020) that enhance model performance by increasing channel diversity, we propose to fuse spatial information by simply reshaping a matrix initialized with IDI, Specifically, we reshape the convolutional kernel C into a matrix
C' E R Sous x xkkcu, We initialize C' as
(C),
(10)
Then by reshaping into C Rixxxxon XCout, our initialization for a convolution is completed. This reshaping strategy can shift spatial features, thereby increasing feature diversity. We utilize IDIC, to denote such a reshaping process. detailed description in Figure ii the Appendix.
3.2
PRESERVING ZERO BY TACKLING DEAD NEURONS
Given a residual network formulated by Eq. (1) prior identity-contre initialization Zhang al., 2019 Zhao et al. 2022) set the last transformation in the residual stem to 0, i.e., ((i.0) 0, thereby maintaining an identity as
(++1) (1 + 0)x = (
(II)
However, the setting can possibly cause dead neurons.
Dead Neuron Problem. The dead neuron problem occurs when a neuron's weight becomes zero and receives zero gradients, rendering it incapable of updating. This issue is harmful the training performance of models. Fixup /Zhang et al.) 2019) only thereby obtaining non-zero gradients. However, in a realistic implementation neural networks, the multiplier of Batch Normalization can be set 0 (Goyal al. 2017), and down-sampling operation
multiplier of I after g(+,0) 0,

---

Published as a conference paper at ICLR 2025
can also cause 0 filled features Under the implementations, ((.0)) always acquires gradients with O values, known as the dead neuron problem, which causes failed weight updating.
20-
(a) Weight initialized with numerical value O.
(b) Weight initialized with IDIZ
Figure 5: The last weight in a residual block of a trained ResNet. More than half of elements in are not trained, which is known as the dead neuron. By contrast, IDIZie-6 successfully solves the dead neuron problem and makes all the elements in (b) trainable.
Tackling this problem, we generate small values ((1.0) assist in training. Recall the goal of adentity-control initialization that outputs O. Therefore, we build a calculation to get the expectation and variance of outputs approaching Considering two ii.d variables, i1 and v2, whose variances are (11) = (12) ip and means are μ(v1) = [u(2) = the variable e = =(vl v2) have
[((v) 0, (²)()) = 24p=.
(12)
where is coefficient, and (v) will be limited to 0 when is sufficiently small. Assuming elements of (1) are i.i.d to each other, by applying subtraction any two elements, the result has a mean of and a variance related E. We also take ((1.0) E RP 1xD, D, as an instance. At first, we initialize g(a.0) with IDI. Then consider two cases: (i) if D < D. setting g(0) 1. with
IDIa;i (ii) if D 2 DJ. set m. = -E, when m%DD - j - 1. Therefore, we can obtain a variance of 0 by setting to a small value. This method is termed IDIZe, and we illustrate some cases in Figure 3 In this paper, we set ε = le - 6 everywhere. As shown in Figure S. IDIZ successfully initializes the last weight in residual block. In addition, we also transform IDIZg to convolution form IDICC, through the patch-maintain scheme.
(10)
The IDInit framework is characterized as follows: (1) For Non-Residual Networks: It involves di- rectly applying IDI r to fully-connected layers and IDIC r to convolutional layers. (2) For Residual Networks: This includes two steps: (i) Implementing IDI and IDIC across all fully colleccondetted and convolutional layers, respectively; (ii) Utilizing IDIZ& and IDICC e for fully-connected and convolutional layers positioned the end of residual blocks, and for the final classification layer.
EXPERIMENTS
In this section, we first analyze hyperparameters in Sec. 4.1 Then, we implement an ablation ex- periment in Sec. 4.2 to show the effect of the proposed two modifications in Sec. 3 We conduct experiments on residual convolution in Sec. 4.3 And we conduct image classification on ImageNet in Sec. 4.4 Later conduct text classification experiment in Sec. 4.5 At last, employ pre-training experiment on the large-scale dataset in |ec. separately. We conduct experiments on non-residual convolution in Sec. C.2 We also analyze the variance amplification in Sec. C.3
weight distribution in Sec, C.4 and dynamical isometry in Sec. C.5
4.1 EXPERIMENT FOR HYPERPARAMETER
In this experiment, we compare IDIni with Kaiming (He et al. 2015) by analyzing the training hyperparameters, i.e., the weight decay and the learning rate. We use C0far10. The backbone is ResNet-32, we use SGD with momentum of 0.9. The batch size is 1024. We train models for 200 epochs. The learning rate reduced with a cosine function. Each setting trained 3 times to calculate the standard deviation. More details and results are in Sec. B.] of the appendix.
ci firrr.p
resnet.py

---

Published as a conference paper at ICLR 2025
Table I: Results Cifar10. 7erO performs worse for down-sampling as mentioned in Sec. 3.2 IDInit consistently facilitates rapid convergence when employed with SGD and Adam.
56 Layer (SGD/Adam)
110 Layer (SGD/Adam)
Initialization
Acc.
Epochs to 80% Acc.
Acc.
Epochs to 80% Acc.
Zero 92.32 /// 87.37 40.43
57 / 6344
93.07
88.30
36+2 / 5647
ZerO
90.57 10.31 88,553
57 / 85
90.71
84.24
55 76
Fixup
93.24 10.550
31 / 55
93.32.20.23
/90.67 10.12
33 49
SkipInit
92.29 88 40.44
26 23
92.67 88..18
31+5 70
ReZero
93.0640.54 / 889.2640.00
33 / 4443
94.03
99.225 20.20
35 38
Kaiming
93.36+0.14 / 87 87.55 40.32
3443 / 50+2
94.06
87, 87.89 #0.41
33+4 56
IDInit
93.4110.10 90.01 10.32
2611 /34
94.04 21
90.53
27 /3622
Horing
Hill 304Maa
As shown in Figure 6 IDInit achieves a peak accuracy of 94.08% with weight decay of le- 3 and a learning rate of 1e-1. In comparison to Kaiming, IDInit demonstrates superior stabil- ity, maintaining high accuracy even when the learning rate is reduced below le-1. Overall, IDInit consistently delivers robust performance while maintaining stability, making a promis- ing candidate for practical applications.
Hillum
(a) IDInit
(b) Kaiming
Figure 6: The hyperparameter experiment on Ci- far10. IDInit demonstrates superior adaptability across a broader range of training configurations compared to Kaiming initialization, exhibiting no- table stability.
4.2 ABLATION EXPERIMENT
We conduct this experiment to validate the effect of the proposed two improvements. The dataset is Cifarl0 and the backbone is ResNet-20 (He et al., 2016). We run four times following settings: (i) IDInit w/o IDIC, and w/o IDIZC;; (ii) IDInit w/o IIIC and w/ IDIZC,; : (iii) IDInit w/ IDIC, and w/o IIIC,;; (iv) IDInit. For model training for 200 epochs, we employ SGD with momentum of 0.9, a weight decay of Se-5, and an initial learning rate of 0.1, which is adjusted using cosine annealing schedule. Additional details and results, including the Loose condition, can be found in Sec. B.3 of the appendix.
The results are shown in Table 2 By applying the identity matrix di- rectly, (i) obtains the lowest accu- racy of 87.01 9% among all cases. Re- garding results of (ii) and (iii), both the settings can make significant improvements of nearly 5.89% and 3.42% from (i), respectively. And IDIZC, can make a deeper effect than IDIC,. Equip- ping IDIC, and IDIZC,. IDInit will improve performance further, which demonstrates our modifi- cation is efficient.
Table 2: Results of the ablation experiment on ResNet-20.
Setting
(i)
(ii)
(iv)
Accracy 87.01 92.9-
90.4340.14
93.224q.08
4.3 IMAGE CLASSIFICATION ON CIFAR 10
In this experiment, validate IDInit with the comparison with existing initialization, including
(1) Fixup (Zhang et al. 2019): (2) SkipInit (De & Smith 2020); (3) ReZero (Bachlechner et al,, 202T: (4) Kaiming (He al. 2015): (5) Zero y (Setting the scale in Batch Normalization (BN) to 0) (Goyal et al. 2017]; ; (6) ZerO. We use ResNet-56/1 10 as backbones on Cifar10. For analyzing convergence, we adopt both SGD and Adam optimizer for updating models. We set SGD, with the momentum 0.9, the weight decay 5e-4, and the learning rate 0.2. For Adam, the learning rate is 0.001, 31 is 0.9 and B7 is 0.999. The training epoch is 200.
Results are shown in Table I Although ZerO uses the Hadamard matrix to break the rank constraint problem, it can be damaged by zero dawm-iamplin as mentioned in Sec. 3.2 Therefore, we reclaim the importance of using IDIZ, and IDIZC, for avoiding such potential damage. Compared with baselines, IDInit derives the best accuracies in most cases. addition, IDInit can achieve the least epochs reach 80% accuracy in all settings, which shows good convergence ability.
8

---

Published as a conference paper at ICLR 2025
Table 4: Results of text classification on SST2 and TREC-6. The subscript G denotes the embedding layer is initialized by Glove, while W indicates Word2Vee. "Default" means the default initialization of models, specifically, Kaiming for TexICNN, and Xavier for both TexIRNN and Transformer. Fixup is only applicable to the Transformer, as it is specifically designed for residual networks. Std values larger than 1.0 are marked in red. More results can be found in Table
Datasets
Init.
TexICNNaw
TextRNNow
Transfornerrww
Averageoww
Default
_1.40_oos 84.56
$1.6920.30 / 20.7
80.97 / 23.36
g0.35 /8.07 20.03
Orthogonal
82.24_o.4 84.37
81.86
82.22.:0.7 / 83.99
82.11 62 /8.33 20.00
SST2
Fixup
78.72++0.7 / 88, $1.25
ZerO
o.07 84,26.0.30
22.03 /8..80.0.c4
82.28 / 88..72-00.5
82.12 00.15 83.99
IDInit
82.60 85.67
82.66 85,49 20.33
82.48 / 18.451 20.24
82.58 200332
88.22 333
Default
90. st / /92.06
26.34 /90.52-1.b0
86.68 / 88..2
87.94
/90.59
Orthogonal
90. 00.34_..72 /92.72
85.86 00.8 / 89.88_1..8
86 1.51 / 89, 889.26::0.86
87.70
/90.62
TRHC-6
Fixup
86, 86.95-20.a5 89.35
ZorO
90.89 / /92.90 50
37.24 / 88.87 7
86.97 / 89,38 10.64
88.37 20 ex
/9..33 133.g.0
IDInit
91.22 0454 99.99400488
87.04
90.60-0..58
87.32 / 90.06::0.0
88,53 :0.53
/91.2000.5.
4.4
IMAGE CLASSIFICATION ON IMAGENET
0.0
We
validate ViT-B/32 (Doooviskkiy et al.
0.7
Jest Accweeyy
2021,
ResNet-50/152 (RN-50/152) (He et al.
0.6
0.66
0,5
2016 and Sc-ResNet-50 (SRN-50) (Hu et al.
2042
2020 as backbones on ImageNet in this exper- iment. For ViT-B/32, the optimizer is A with a learning rate le-3 and a weight decay Se-2. The training epochs is 300. We use 30 epochs for warm-up. For RN-50/152 and SRN- 50, we use SGD with learning rate 1e-1 and a weight decay lc-4 for 90-epoch training. We use 9 epochs for warm-up. For all models, the batch size 1024, and we apply data augment including cutmix (Yun et al., (019) with a 1.0, mixup Zhang et al.) 2018) with a = 0.8, the switching probability is 0.5 and a label smoothing with 0.1. More details and results can be found in Sec. [B.]
0.60
150 200 Epoch
250 300
20
Epoch
(a) VIT-B/32
(b) RN-50
Figure 7: Results ImageNet. "Default" means
the default initialization of models.
Results are shown in Fig- ure 7 and Table 3 On three types of networks, ie., ViT, ResNet and Se-ResNet, and multi- ple depths, IDInit always achieves faster conver-
Table 3: Results on ImageNet. The value in brackets means "Epochs to 60% Acc". On average. IDInit enhances accuracy by 0.55% compared to the baseline and expedites model convergence by 7.4 epochs.
Model VIT-B/32
RN-50 (Adamw)
RN-50
SRN-50
RN-152
(A)
Default 71.05 (44)
76.20 (20)
75.70
76.30
78.76 (28)
(
IDInit (42)
76.71 (14)
76.72 (24)
76.93 (22)
(23)
0.55 (7.4)
gence and better performance than the baseline. When training RN-50 with Adamw, the convergence of IDInit is consistently fast. Compared with RN-50, our initialization shows faster convergence speed. IDInit has an average improvement of 0.55%, which is significant to be in practice. This experiment shows the good practicability and promising probability of IDInit, which is beneficial to the artificial intelligence community.
4.5 TEXT CLASSIFICATION
We implement text classification on SST2 (Socher et al. 2013) and TREC-6 (Li Roth 2002) and select TexICNN (Kim 2014), TextRNN (Lai et al. 2015) and Transformer (Vaswani et al. 2017]) for comparison. For TextCNN and TextRNN, we use AdaDelta (Zeiler) 2012) optimizer with a learning rate 1.0 and adopt Adam (Kingma & Ba 2015) for Transformer with learning rate lc-4. For the embedding layer, we utilize Glove (Pennington et al. (0114) and Word2Vec initialize the embedding weights. All models are trained up to 10 epochs for 5 times.
et al. 2013) to
As shown in Table all the initialization methods can work normally. Default random initialization obtains the lowest accuracy in most cases both SST2 and TREC-6. Orthogonal initialization always derives modest results. By contrast to baselines, IDInit can achieve the highest accuracy in all conditions. In addition, IDInit always obtains the smallest std values, showing stable performance.
9

---

Published as a conference paper at ICLR 2025
4.6 PRE- TRAINNGG ON LANGUAGE MODEL
2.3
2.2
10ht
Pre-training plays an important role in various applications. We conduct the experiment show the fast convergence on BERT (Devlin et al. 2011).. The dataset is the concatenation of English Wikipedia and Toronto Book Corpus (Zhu al. 2013). We train the BERT-Base for 40 epochs with 768 batch size. The optimizer is AdamW with learning rate le- 4 and weight decay le-2. 32 NVIDIA V100s are used.
2.1
2.0
ALM
1.8
15
1.7
1.5
As shown Figure g "Default" means the default initial-
1.4
FLOPs (1e19)
ization of BERT-Base. IDInit achieves faster convergence. Specifically, IDInit shows an 11.3% acceleration ratio in terms of FLOPs. Moreover, IDInit can derive a lower loss of 1.46 in the end. As a result, IDInit is promising used in practice for enhancing convergence ability and performance.
Figure Results of BERT-Base.
5 DISCUSSION
The position ReLU. Pennington et al. (2017) pointed out that non-residual networks cannot achieve dynamical isometry when using the ReLU activation function. However, in residual net- works, such as residual block. As explored in Bartlett et al. (2019) and Hardt & Ma (2017). when the weights in the sub-stem are small, residual networks with ReLU can effectively approximate a linear network. This enables the model to follow dynamical isometry, as illustrated in Figure [8] where IDInit
W2ReLU X) + X, the non-linearity resides within the sub-stem of the
sults in most values being close to 1. Moreover, Tarnowski er al. (2019) provide a theoretical perspective, suggesting that any activation function within the residual stem support dynamical isometry. However, this behavior changes if ReLU is instead placed in the main stem, such as in Y ReLU(W, + X). In this configuration, the network fails maintain isometry, as noted by Pennington et al. 2017). This indicates that placing ReLU in the main stem is not advisable.
The mechanism behind the identical initialization. Figure hhighthhs the motivation behind the design of IDInit by demonstrating how identity-based initialization preserves structural bias while avoiding pitfalls such rank constraints. However, the exact mechanism behind this improvement still remains an open question. which is a promising area for future research. Investigating this mechanism further could provide valuable insights and pave the way for the development of more efficient initialization strategies, benefiting the broader research community.
Theoretical analysis regarding the convergence rate. Theoretical exploration of the conver- gence rate is a critical yet challenging aspect of initialization methods. The convergence process in deep neural networks is iterative and influenced by numerous factors beyond the initialization method, including the network architecture, optimization algorithm, learning rate, batch size, and data distribution. As this area holds significant importance, further research is necessary gain deeper insights, which will contribute to a more comprehensive understanding of initialization.
6 CONCLUSION
efficient initialization approach is crucial for training deep neural networks. this paper, we introduce fully identical initialization (IDInit) that is based on the identity matrix. Addressing the problems encountered when developing IDInit, i.e., dead neurons and performance degenera- tion, we give two concise solutions, namely using small numerical values wipe off dead neurons and reshaping an identity-like matrix into a tensor thus increasing feature diversity, leading to a performance improvement. With good performance wide generality, high stability, and fast vergence. IDInit promising to be applicable in practice. In the future, we hope that this identical design can motivate the AI community implement more novel initialization methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGE
We extend our sincere gratitude to all the reviewers whose insightful comments and constructive feedback during previous submissions greatly enhanced the quality of our paper. This research was partially supported by the CFFF platform at Fudan University.
IMPACT STATEMENTS & LIMITATION
Impact Statements. This paper introduces IDInit, initialization method designed enhance stability and convergence of the training process for neural networks. This method is unlikely to have negative societal impacts.
Limitation. While IDInit demonstrates notable advancements in convergence speed and perfor- mance enhancement, faces challenges in converging ground truths that include negative eigen- values. However, this drawback can be easily mitigated by incorporating momentum into the opti- mizer. Given that momentum is a commonly used setting, this limitation can be implicitly resolved as we show in the main context.
REFERENCES
Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust initial- ization for weightnorm & resnets. In NeurIPS, 10900-10909, 2019.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Ju- lian J. McAuley, Rezero is all you need: fast convergence at large depth. In UAI, volume 161 of Proceedings of Machine Learning Research, pp. 1352-1361. AUAI Press, 2021.
Peter L. Bartlett, David P. Helmbold, and Philip M. Long. Gradient descent with identity initializa- tion efficiently learns positive-definite linear transformations by deep residual networks. Neural Comput., 31)), 2019.
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: feature diversity necessary in deep neural network initialization? In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 960-969 PMLR, 2020.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. ICLR, 2016.
Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function deep networks. NeurIPS, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pp. 4171-4186. As- sociation for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, and Jef- frey Pennington. Dynamical isometry and mean field theory of Istms and grus. CoRR, abs/1901.00897, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org, 2010.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training ima- genet in hour. CoRR, abs/1 706.02677, 2017.
11

---

Published as a conference paper at ICLR 2025
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, pp. 1577-1586. Computer Vision Foundation IEEE, 2020.
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In ICLR (Poster). OpenReview.net, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pp. 1026-1034, IEEE Computer Society, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778. IEEE Computer Society, 2016.
Jie Hu, Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell., 42(8):2011 1023, 2020.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pp. 1746-1751.
ACL, 2014.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR (Poster), 2015.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In AAAI, pp. 2267-2273. AAAI Press, 2015.
Quoe V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. CoRR, (bs//500400094) 2015.
Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems, pp- 1-15, 2021.
Xin Li and Dan Roth. Learning question classifiers. COLING, 2002.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. ICLR (Workshop Poster), 2013.
Yu Pan, Zeyong Su, Ao Liu, Jingquan Wang, Nannan Li, and Zenglin Xu. A unified weight initializa- tion paradigm for tensorial convolutional neural networks. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 17238 -1257. PMLR, 2022.
Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. Reusing pre- trained models by multi-linear operators for efficient training. In NeurlPS, 2023.
Yu Pan, Ye Yuan, Yichun Yin, Jiaxin Shi, Zenglin Xu, Ming Zhang, Lifeng Shang, Xin Jiang, and Qun Liu. Preparing lessons for progressive training on language models. AAAI, pp. 18860- 18868. AAAI Press, 2024.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543 ACL, 2014.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NIPS, pp. 4785-4795, 2017.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Expo- nential expressivity in deep neural networks through transient chaos. In NIPS, pp. 3360-3368, 2016.
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma. and Jitendra Malik. Deep isometric learning for visual recognition. In ICML, volume 119 of Proceedings of Machine Learning Research, pp- 7824-7835. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631-1642. ACL, 2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (Workshop), 2015.
llya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pp. 1139-1147. JMLR.org, 2013.
Wojeiech Tarowwkk,, Piotr Warchol, Stanislaw Jastrzebski, Jacek Tabor, and Maciej A. Nowak. Dy- namical isometry is achieved in residual networks in a universal way for any activation function. In AISTATS, volume 89 of Proceedings of Machine Learning Research, pp. 2019.
PMLR,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, pp. 5998 6008, 2017.
Maolin Wang, Yu Pan, Zenglin Xu, Xiangli Yang, Guangxi Li, and Andrzej Cichocki. Tensor net- works meet neural networks: A survey and future perspectives. arXiv preprint arXiv:2302.09019, 2023.
Greg Yang and Samuel S. Schoenholz. Mean field residual networks: On the edge of chaos. In NIPS, pp. 7103-7114, 2017.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6022-6031. IEEE, 2019.
Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. CoRR, abs/1706.00388, 2017.
Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR (Poster). OpenReview.net, 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR (Poster). OpenReview.net, 2019.
Jiawei Zhao, Florian Tobias Schaefer, and Anima Anandkumar. Zero initialization: Initializing neural networks with only zeros and ones. Transactions Machine Learning Research, 2022.
ISSN 2835-8856. URL https: httpss//oppprreiieeee net. forum? 1itrmmmmm=1AAAApmmmmm TC
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19-27. IEEE Computer Society, 2015.
13

---

Published as a conference paper at ICLR 2025
A IDINIT DETAILS
A.1 FULL IDINIT SCHEME
Here, we show the full DDInit scheme in Figure
0
D
D
D
D
Identity Preserving Iniiialization
0
8
-
0
20 77
27%
Di
D2 <
DIn
12.
Zero Preserving Inttalization
Figure Illustration of IDInit with all conditions.
A.2 ANALYSIS ON CONVERGENCE
The issue of convergence proposed by Bartlett et al. (0019). According their study, when layers in a neural network are initialized using the identity matrix, all the weight matrices of layers will be symmetric each step of the training process. This persistent symmetry leads the weights of layers being the same as each other at any step, posing significant challenge in converging to the ground truth of which eigenvalues with negative values. Our findings indicate that employing stochastic gradient descent (SGD) approach can effectively break the symmetry which facilitates convergence, and incorporating momentum can further accelerate the convergence process. this context, provide formal proof demonstrating that SGD with momentum can alleviate the conver- gence issue.
Proof. First of all, we present training case for a single-layer network expressed as y dz, where IRd represents the input, E Rd denotes the output, and 0 E Rd is the weight matrix. The weight matrix is initialized to the identity matrix I, denoted as (()) = I. For our loss function, we employ the Mean Squared Error (MSE) and learning rate denoted by r,. Consider two training pairs (21, y1 and {Λ22 y/ sampled from the same dataset D. The network is initially trained with (#1, }1 and trained with {a2, 1/2 in the next step.
In the first step, we can get the prediction as
01 0(0)
(13)
The updated ((2) can be derived by
0) = (ύ1 - y1 yr = (0(0), 11 - y1)aT = (x1 - 31)=r
g(1) g(0) ()) p(0) 17(21 v1)=T I 17(02 y1)=T.
(14)
14

---

Published as a conference paper at ICLR 2025
Therefore, in the second step, the gradient AA(1) can be caleulated
A@ = (02 y2)=r2
(0(1) 2 1/2 )=
= ((1 - 11(311 yy ) x2 1/2 1/)222
x24 - 11t 11 22 + m191 1 2202 - y22z.
(15)
While 224z is symmetric, 2aLa 2245 5 z22 and y2 can be asymmetric. To calculate the magnitude of the asymmetry in A0(1), letting Ω = - x22ÿ + mylof æх y2 denotes the asymmetrie component, the magnitude of asymmetry that be calculated E(I Ω 1
Assuming F1, 22, y TEd are random vectors with entries that are i.i.d. Gaussian random variables, following N(0, o²), then the magnitude of asymmetry bounded as
4nj d 0 41 of + 22 < E(12 22 s
= d + 3d204
(16)
The proof can be found in Sec. A.3
As r is usually 1 1, and both training pairs ( } and {2 can be generally normalized to ~ (0, thereby, the symmetry of the weight can be
sufficiently influenced as
( 0(0) n1
(17)
When introducing momentum m( initialized to A assuming the coefficient of m is y, ((2) will be updated as
(1) = ym (0) + ,))0(1)
Ø = (1) m (1) = ( - ym (0) - -1)(11)
(18)
Therefore, momentum can promote the weight to become asymmetric by accumulating the asym- metry of gradients in steps and impact more when samples are increased.
As for networks of multiple layers, when their layers are asymmetric, each layer can be updated differently which breaks the convergence problem caused by the same gradients in each step (which stated in Lemma Bartlet et al. (2019).
This proof primarily demonstrates that SGD with momentum effectively resolve the issue of lay- ers being the same networks initialized with the identity matrix during training, which facilitates the convergence process. As illustrated in Figure I0 it is evident that layers trained using SGD are different from each other, with the momentum component amplifying the degree of this difference. By theoretically and empirically demonstrating that SGD with momentum can efficiently address this convergence problem, we hope this finding can offer valuable insights for the research com- munity, encouraging further investigation into identity initialization and its significant role in model training.
A.3 ANALYSIS ON ASYMMETRY
In this section, we analyze the magnitude of asymmetry in the gradient.
Setup and Target. Here, we assume 11, 22, V1, 3/ E Rd are random vectors with entries that are i..dd Gaussian random variables, following (0, According to Eq. (4), the asymmetry in the gradient arises from:
Ω = - 2242 + 1914 2222 - y2
(19)
Our target is compute the magnitude of asymmetry that can be calculated as
E [[1 - 11%))
= =E| | 2zna (yriafa227) + In 224 (myiafazat) x222)"""
シ2 + (12= 111
(20)
Lower Bound. Introducing substitutions u 91 2i, and of x2 2b rewrite:
15

---

Published as a conference paper at ICLR 2025
we momentum
w/ momentum
0.30
0.30
0.000
0.000
0.000
9.000
0.20
9.000
0.002
0.000
0.000
0.20
0.000
0.000
5.000
5.000
0.003
0.000
0.020
0.10
0.000
-0.05
0.000
0.05
Lager
Layer
Laper
Layer
Layer
Layer
Layer
(a) GD w/o momentum
(b) GD w/ momentum
SGID
momentum
SGD momentum
9.36
0.16
0.053
0.114
0.34
0.14
0.12
0.2
0.033
0.000
0.022
3.086
2.302
0.500
0.063
0.123
0.30
0.10
0.08
-0.05
0.063
0.06
Layer
0.066
0.056
-0.02
0.209
-0.0
Layer
Layer 1
Layer
Layer
Lyyer
Layer
Layer
Layer
(c) SGD w/o momentum
(d) SGD momentum
Figure 10: The distance between two layers in 4-layer network after training. In this experiment, we set the target matrix as -/ E R 10x 10 The weights are Wo, Wi, W2, W3 IR 10x 10 We randomly generated 4000 data pairs (X,, Y2} by Y, -Ix; &, where X,, Y; R210 and noise with mean O and std le-2. We use 2000 samples for training the network. We use the other 2000 samples for testing. Batch size is Mean squared error (MSE) used as the loss function. We calculate the distance by averaging the absolute value from the difference value of two layers. Layers trained using SGD display distinct differences from one another, and the incorporation of momentum significantly increases these differences, thereby accelerating the convergence speed.
B[I - (
E Insuuz xz4²) - (y22z - -22y3 )|b:)
(21)
Let w = qsu y2, then:
E(I 112) Bfllway 1
(22)
[[2([[[0|| 11 (w x2) )}
(23)
2 | |2|
El xz 222
(24)
Expanding and computing expectations:
6 = - -2T 2(1 d]o do do 2000,
(25)
2 208
2n dl 0 + 28 0
(26)
16

---

Published as a conference paper at ICLR 2025
Upper Bound. We derive the upper bound as - -
(27)
[[11-11011 2222 + (139 (ạ x2x + Im x24 (nm1
+ [-12222 + )
(28)
According to Relaxed Triangle Inequality, there is
<3(1/B111 =1 -ι 224 + x2 [[2]] Bllun xaa (y1 x222 11
+ +BIl y242 _ht + )
(29)
<3(7"d''* +nd ddo*)
(30)
(31)
This shows that higher learning rate promotes greater asymmetry, further explaining the observed differences. However, high learning rate can affect training stability. Therefore, while using higher learning rate reduce symmetry, crucial to carefully select its magnitude to maintain stability.
A.4 PROOF FOR THEOREM
Proof. Consider network with single hidden layer (i.e. L 3) and batch of linearly indepen-
dent samples, zi = { (0.1) ,7 (O.N)], with Do. Using II1 = Ei=1 au.3.1) [344
(0.1)
x
the
gradients for the first update step can be written as
UC
[1]
ac
[I|
III
dc 0012)
= (II IT).
(32)
00(0)
0
0011)
After updating the weights with learning rate 1) >> O, we have
( ( 7 111
g(D) (1 1j III -n 11
(
(I 1111
II
(33)
0
As result, the gradients of ((1) for the second update with second batch of linearly independent samples 2/2 = {x2 (0.1)
22
(0,N)
}, are given by
Oc
Σ
())
x2 10.00) x
0x, (3,3) ((2)
OC
(34)
20(1
i=1
-(U
ITT I1 (I - 17)
(I (I
(35)
Hl (I-
-1 I2
with 1T = M =Z=
8C
(0.3) Using the gradients of the second batch to update the parameters
0z (., x x2
with the same learning rate, we obtain
)() n7 Il
17 1 (I- II TI (I-
-1 - 1 - 112
I 1 II 1l
Consequently, the difference the weights after two updates to the initial value, I, given by
17 AO() = (-1.L: 17 11 II, ------ii IIIT) In (I-I 11 IIg
2)2 I Th2 (I TT
") TT 122
and ac are also linearly independent, rank([I)) = rank([])) (अT कि Da- Due to Sylvester's rank inequality, we can conclude that also rank Il)) = Do- As a result, the lower-right part of the difference has rank Dg. from which we can conclude that rank (ank(A0()) > Da.
Assuming that the gradients
17

---

Published as a conference paper at ICLR 2025
A.5 IMPLEMENTING IDINIT ON ATTENTION LAYER TRANSFORMER
In this part, we show the way to initialize the attention layer with IDInit. Prior to that, formulating attention layer as
At K, softmax QWOWKK K Vwwwo
(36)
Vd
where Q is the query matrix, K means the key matrix, V denotes the value matrix, W4, W and WY represents the weights for Q. K, and V respectively, and W is the output transformation. Following the instruction of IDInit in Sec. 3 we firstly use IDI, to initialize WQ, W W and
W And then, we use IDIZ, to initialize the last fully-connected layer W The T and are
consistently set with the paper content to and le-6, respectively.
A.6 DETAILS OF PATCH-MAINTAIN CONVOLUTIOON
We illustrate the figure to show the comparison between Stannel-mainterd convolution and patch- maintain convolution in Figure
Idertity Transition
Goal
Pooto no
Diarrig
4in
7
7
Iegut number
7
IDIC,
Snited
Incrwsiig
Geat
7
7
Figure 11: A case of number "7" Identical Convolution Layer. The upper sub-figure maintains the identity transition. The under sub-figure IDI, initialization that shifts features for increasing diversity. More feature diversity from IDIC, is beneficial for improving model performance.
B DETAILED SETTINGS OF EXPERIMENTS
In this paper, for ReLU activated networks, is set to v2 for the first layer in network and I for other IDI, IDIC, initializing layers, while for tanh-activated networks, IDI, is set to 1, and is le 6 for all IDIZ./ IDIZC, initializing layers.
B.1 EXPERIMENT FOR HYPERPARAMETERS
In this experiment, we compare IDInit with other initialization methods, including (1) Fixup: (2) ReZero; (3) Kaiming; and (4) Zero, by analyzing the training hyperparameters, i.e., the weight decay and the learning rate. We use Cifar10. The backbone is ResNet-32, we use SGD with momentum 0.9. The batch size 1024. We train models for 200 epochs. The learning rate is reduced with cosine function. Each setting is trained 3 times calculate the standard deviation.
We scanned the learning rate from 1e-3 lel and weight decay from 1e-8 to le-1, ensuring that the best-performing hyperparameters are not at the corners or edges of the grid. As shown in Figure 12 IDInit achieves a peak accuracy of 94.08% with a weight decay of le-3 and a learning rate of le-I. In comparison to other initialization methods including Kaiming, Fixup, and Rezero, IDInit demon- strates superior stability, maintaining high accuracy even when the learning rate is reduced below le-1. Although ZerO exhibits comparable stability lower learning rates owing to its Hadamard matrix's 's ability to sustain dynamics, it underperforms at higher learning rates due to the dead neu- rons caused by the weights in its residual stems. Fixup, the other hand, lacks stability by
18

---

Published as a conference paper at ICLR 2025
Millous
13
11.41:M
140.mm
10.00.m
13
3
W6-77
11.
9730mm
15.23
14 14
IDInit
(b) ZerO
(c) Kaiming
g Seakin 58 Mi. tidluc
58
stak
28 Mar
M44
3 1AR 26
18:20
(d) Fixup
(e) Rezero
Figure 12: The expanded hyperparameter experiment on Cifarl0 on ResNet-32.
eliminating batch normalization, rendering it unsuitable for high learning rates. Overall, IDInit con- sistently delivers robust performance while maintaining stability, making promising candidate for practical applications.
B.2 DETAILS OF IMAGE CLASSIFICATION ON CIFAR 10 EXPERIMENT
In this experiment, we validate the proposed initialization with the comparison with existing initial- ization, including (1) Fixup; (2) Skiplnit; (3) ReZero: (4) Kaiming; (5) Zero y (Setting the scale in Cifarl0. For analyzing
Batch Normalization (BN) O). We use ResNet-56/110 as backbones convergence, we adopt both SGD and Adam optimizer for updating models. We set SGD, with the momentum 0.9, the weight decay 5e-4, and the learning rate 0.2. For Adam, the learning rate is 0.001, 3, is 0.9 and 32 is 0.999. We train models for 200 epochs. The learning rate is reduced with a cosine function. The experiment conducted on one Nvidia A100.
We perform a detailed hyperparameter analysis for ResNet- 10, evaluating the learning rates {1, 2e-1, le-1) and weight decays (1e-4, 5e-4, 1e-3) on the standard baseline Kaiming and the more fragile Fixup method. As shown in Figure 13 both Kaiming and Fixup achieve optimal accuracy with learning rate of 2c-1 and a weight decay of 5e-4. However, Fixup fails train with learning rate of 1. Consequently, selecting learning rate of 2e-1 and weight decay of 5e-4 as the training hyperparameters in Section 4.3 justified.
B.3 DETAILS ABLATION EXPERIMENT
The dataset is Ci and the backbone is ResNet-20. We choose SGD with momentum 0,9, weight decay 5e-4, and learning rate 0.1 to train the models for 200 epochs. The learning rate is reduced with a cosine function. And aata-nugment mixup is applied. The experiment is conducted on one Nvidia A 1000
19

---

Published as a conference paper at ICLR 2025
20.
13-1
(a) Kaiming
(b) Fixup
Figure 13: The tuning hyperparameters on Cifarl0 on ResNet-
The extended analysis in Table 5 shows that the Loose condition, along with the components IDIC and IDIZ, contributes independently to performance improvements. Furthermore, the combination of these components yields the most significant results. Across all comparison pairs- Spirs--peciffiallly settings 1/4, 2/6, 3/7, and 5/8-the Loose condition consistently demonstrates performance improve- ments. This highlights its practical value and its role in enhancing the overall effectiveness of the initialization methods.
Table 5: Analysis of components.
Component
Setting
Setting
Setting
Setting 4
Setting
Setting
Setting 7
Setting
Loose
IDIC
IDIZ
Accuracy
16,121052
92.68 _0.0s
89.47 10.24
87.01
12.95
12.910.18
90.4310.14
93.22100
B.4 DETAILS OF TEXT CLASSIFICATION EXPERIMENT
We also explore performance networks text classification datasets including SST2, SST5 (Socher cet al.] 2013) and TREC-6, and we select TexCNN (Kim) 2014). TextRNN (Lai et al. 20155) and Transformer Vaswani al.) 2017) for comparison. For TexICNN and TextRNN, AdaDelta (Zeiler 2012) optimizer with a learning rate 1.0 and adopt Adam (Kingma & Ba 2015)
for Transformer with learning rate le-4. For the embedding layer, utilize Glove (Pennington
jt al. 2014) and Word2Vec (Mikolov et al.) (013) to initialize the embedding weights. All models
trained up to 10 epochs, and run all the random initialization 5 times. The experiment is conducted on one Nvidia A100.
2.22
116
205
355
2102
E0.7
3.0
zde
300
20
46
(a) VIT-B/32
(b) (Adamw)
RN-50
(c) RN-50
(d) SRN-50
(c) RN-155
Figure 14: Results ImageNet. "Default" means the default initialization of models. RN-50
(Adamw) means that ResNet-50 is trained with the same optimizer Adamw as the ViT-B/32.
B.5 DETAILS OF IMAGE CLASSIFICATION ON IMAGENET EXPERIMENT
In this experiment, we use ImageNet for validation. We use ViT-B/32 (Dooovitskiy et al. 2021 l.
ResNet-50/152 (RN-50/152) and Se-ResNet-50 (SRN-50) as backbones. For ViT-B/32 that inputs 32 x 32 patch window, the optimizer Adam W with a learning rate 1e-3 and weight decay of 5e-2. And the batch size is 1024. The epoch for training is 300. We use 30 epochs for warm-up. The input
20

---

Published as a conference paper at ICLR 2025
Table 6: Results of text classification SST2 and TREC-6. The subscript G denotes the embedding layer is initialized by Glove, while W indicates Werd2Vec. "Default" means the default initialization of models, specifically, Kaiming for TexICNN, and Xavier for both TexIRNN and Transformer. Fixup, ReZero and Skiplnit are only applicable to the Transformer, as it is specifically designed for residual networks. Std values larger than 1.0 are marked in red.
Datasets
Init.
TexICNNaw
Tex
Transformerenw
Averageaw
Default
o0s 84.56
81.69 20.30 884.29 81.86 g0.
21.20
23.36
21 84.07 20.03 82. 12.1120:e2 /8.32
Orthogonal
82.24_o 84.37
82.22.0087 / 83.99
SST2
Fixup
78.72.40.7 8 81.25
Re/ero
282.322 81.67
Skiplnit
88.112,u.75 82.30
ZerO
82.05-0.67
84.26
82.03 20.03
84, 84,800.0.44
82.28
88..72
82. 62.12
/83.93 20.55
111ini
22
85.67
22.66
20.33
82.48 184.51 =0.24
82.58.0.32
85.222
Default 90.80 99.006
26.34 /90.52.1.04
2..8 89.200
87
/90.59 2:220
Orthogonal
90.34_o.72
92.72
85, 85,86
889.88
86.90
18.926
87.70
/90.62 20.75
TREC-6
Fixup
86 86.95 88,35, 89.35-+0.55
Re/ero
86, 86.92220.55 89.36..0.22
Skiplait
83.59 / 87. 87.10
ZerO
90.89 /92.90
87.24
88.71
86.97 / 89.38 1...g....6
88.37
/90.33 20.00
IDInit
91.22 00.5 /92.94.0.a 20048
87.04 90.60.0..5
37.332.0.78/ 10.78/ 90.06
88.53 20.53 g9.220g0.5
image size is 224 224. The dropout rates the embedding layer and the network layer are all 0.1. For RN-50/152 and SRN-50, the optimizer is SGD with learning rate le-1 and weight decay of le-4. And the batch size is 1024. The epoch for training 90. We use epochs for warm-up. The input image size is 160 x 160 for the front 35 epochs and 224 x 224 for the remaining epochs. For all models, we apply data-augment including cutmix (Yun et al. 2019) with a = 1.0, mixup (Zhang 0.8, the switching probability 0.5 and label smoothing with 0.1. The
et al. 2018) with
experiment is conducted on Nvidia A 1100.
To further compare with other identity-contre methods, conducted experiments ResNet-50 using Fixup and Zero. As shown in Table IDInit outperforms both Fixup and ZerO, demonstrating its superior performance large-scale datasets.
Table 7: Comparison among Default, Fixup, ZerO and IDInit initialized ResNet-50 on ImageNet.
Init.
Epochs to 60% Accuracy
Accuracy
Default
38
75.70
Fixup
24
75.83
ZerO
30
75.64
IDInit
24
76.72
B.6 DETAILS OF
ON LANGUAGE MODEL
Pre-training plays an important role in various applications. We conduct the experiment to show the fast convergence on BERT (Devlin al. 2019). The dataset the concatenation of English Wikipedia and Toronto Book Corpus Zhu et al. (2015). We train the BERT-Base for 40 epochs with 768 batch size. The optimizer set to Adam with learning rate le-4 and weight decay le-2. 32 NVIDIA V100s are used.
C ADDITIONAL EXPERIMENTS
We provide additional experiments to further validate IDInit. and are set the same as Sec.
B
C.I VALIDATION ON THE LINEAR STRUCTURE
This experiment is conducted on MNIST. We use five linear layers named Liner-5 whose hidden layers are all of dimension 512. The optimizer is SGD with momentum 0.9, weight decay 5e-4,
21

---

Published as a conference paper at ICLR 2025
Table 8: Results of Linear-5 on MNIST. "Default" means the default initialization of models where Xavier is for Linear-5-tanh and Kaiming is adopted for Sinear---ReLU
Init.
Linear-5-tarr
Linear-5-ReLL.
Default
98.26
98.21
IDInit
98.32
98.4
and learning rate 1e-1. The learning rate scheduler adopts cosine reduction strategy. We the model 30 epochs on one Nvidia A100. We both consider Linear-5-tanh and Linear-5-Rel.U which consist of Linear-5, and tanh and ReLU activation functions, respectively. The experiment is conducted on one Nvidia A A100.
As shown in Table 8 IDInit can achieve the highest accuracy in both different tanh and ReLU conditions. The results show the ability of our proposed method to train a model with only fully- connected layers.
C.2 VALIDATION ON NON-RESIDUAL CONVOLUTION
We use this experiment to show IDInit can achieve good initial state for training on non-residual convolutional networks. In this experiment, we use AllConv (Springenberg et al. 2015) which con- sists nine convolutional layers as backbone network. WWe show the structure AllConv in Table 9 The dataset is C0 The optimizer is Stochastic Gradient Descent (SGD) with momen- tum 0.9, weight decay 5c-4, and learning rate le-1. The learning rate scheduler adopts a warm-up cosine reduction strategy. We the model in 300 epochs initialization and IDInit w/o IDIC, initialization for comparison. Since there is no residual connec- tion, we do consider the IDICC, function this experiment. For each initialization, run them with O, 10, 20, 30, 40, 50, and 60 warm-up epochs. The experiment is conducted on one Nvidia A 1000
Nvidia A 100. We adopt Kaiming
have
Table Architectures tensorial All-Conv networks. Window means the convolutional kernel window size. Channels indicate Cin and Cout of a standard convolutional kernel C RSin X xkxk The avg pool denotes the average pooling operation.
Layer
Window
Channels
conv
3x3
3x 96
conv2
3x3
96x 96
conv3
3x3
96x 96
conv4
3x3
96x 192
conv5
3x3
192x 192
conv6
3x3
192x 192
conv7
3x3
192x 192
conv8
1x 1
192x 192
192x 10 avg pool
conv9
1x1
Results are shown in Figure 16 without a warm-up strategy which a strong trick for training, both Kaiming and IDInit w/o IDIC, fail to train the model. By contrast, our initialization train AllConv and maintain the highest performance in all situations, showing strong effect on stability and performance. As DInnt w/o IDIC, performs poorly, we demonstrate the patch-maintain strategy
22

---

Published as a conference paper at ICLR 2025
5.0
7.5
Actiatt Wive
7.5
5.2
Brist Die
5.2
7.5
0141
5.0
25
2.5
2.5
2.5
0.0
-7.3
-7.
-20.0
-20.0
-10.0
-20.0
(a) FC-0.00
(b) ResFC-0.00
(c) Conv-0.00
(d) ResConv-O.00
10.0
20.0
Datact
Deleat
2.5
2.5
0.0
0.3
0.3
0.0
-25
-2.5
-2.5
-2.
-5.3
-5.0
-7.5
-00.
-10.0
-10.0%
Layor
Caysr
Layo
Layoe
(c) FC-0.01
(f) ResFC-0.01
(g) Conv-0.01
(h) ResConv-O.01
10.0
30.5
10.3
10.0
Actvestonn 10.
7.5
5.0
Bivd
7.5
1891
0.
-25
-25
-2.5
-2.5
-5.0
-5.3
-5.3
-5.0
-7.5
-7.5
-7.3
-7.5
-10.0
10
Layer
Layor
Layer
Layyr
(i) FC-0.10
( ResFC-0.10
(k) Conv-0.10
(I) ResConv-O...
10.0
10
10.5
30.0
Reivadtinn Value
5.0
75
AChvodion W0-e
7.
5.2
5.2
7,5
De
1841
5.0
7.5
Defiutt
1094:
2.5
2.5
2.5
-25
-2.5
-5.0
-3.3
-5.0
-75
-73
-7.3
-7.5
10.0
10
-10.3
10
-0.0
-100
0
20
(m) FC-1.00
(n) ResFC-1.00
(0) Conv-1.00
(p) ResConv-1.00
Figure 15: Results of the analysis on variance propagation. The numerical value after the model name means the standard derivation of the noise. "Default" means the default initialization of models, specifically, Xavier for FC and ResFC, and Kaiming for Conv and ResConv. The default methods can only work on non-residual networks FC and Conv, however, fail on residual networks ResFc and ResConv, for cause instability with giant standard derivation. By contrast, IDInit can consistently transit data-flow in an appropriate scale on all models and various noises, which shows sufficient robustness, and can provide models with stable and efficient training.
mentioned in Sec. 3.1.3 can be good for increasing feature diversity. This experiment shows the identical method can be a feasible initialization for non con-residual networks.
C.3 ANALYSIS ON VARIANCE PROPAGATION
Here we conduct an experiment on Cifarl0 to demonstrate data-flow will keep stable. We use types of networks: (1) FC: 10-layer fully-connected layers; (2) ResFC: 10 residual blocks (two fully- connected layers in block); (3) Conv: 9-layer AlIConv in Sec. C.2 (4) ResConv: 10 residual blocks (two convolutional layers in a block). For (1) and (2) two fully-connected networks, we reshape Cifar10 data as X E R32x96 as input and does not use any activation function. For (I), hidden lengths are {200, 400, 600, 800, 1000, 1000, 800, 600, 400, 200). For (2), hidden lengths are all set to 96. For (3) and (4) two convolution networks, we directly input images to them, and use ReLU as the activation function. For (3), directly use AllConv as shown in Table 9. For (4), first use convolution to transfer an image to 16 channels, and then set the channels of all convolution within residual blocks to 16. For comparison,
Xavier for (1) and (2), and Kaiming for (3) and (4) in
23

---

Published as a conference paper at ICLR 2025
1.0
IDlnt Kaiming
1.2
0.8
w/o IDIC
Test Accurac)
Test Accuracy
1.0
0.6
0.8
0.6
0.4
IDinit00
Dinit-10
0.4
0.2
Kaiming-BB
Kaiming-40
0.2
Kaiming-60
0.0
0.0
100 150 200 250 Epoch
0
10
20
30
40
50
60
Warmup Epoch
(a) Test Top-1
(b) Test Best
Figure 16: Results of AllConv on Cifar10. The number behind the initialization denotes the warm- up epochs.
terms of the activation function. We also employ noises with 0 mean, and {0.00, 0.01, 0.10, 1 1.00) for comparing robustness. In the experiment, we run 500 rounds for each model. The experiment is conducted on one Nvidia A100.
Results are shown in Figure 15 The regular methods Xavier and Kaiming only work non- residual networks. On residual networks, they both cause giant standard derivation, leading to in- stability. By contrast, the proposed IDInit consistently transit data-flow in appropriate scale on all models and various noises, which shows sufficient robustness, and can provide models with stable and efficient training.
C.4 ANALYSIS ON WEIGHT DISTRIBUTION
In this experiment, conduct experiment Cifarl0 with ResNet-20 to show the weight dis- tribution of IDInit We use an SGD optimizer with a learning rate 0.2, and weight decay 5e-4. The batch size is 1024. Training epochs 200. The learning rate is reduced with cosine function. The experiment is conducted on one Nvidia A100.
4.0
25
2.
2.5
2.0
2.0
1.5
15
1.0
0.0
Weigh: Vals
Weiht Wice
Wiggh Wiuss
Weigh: Vilee
(a) Kaiming-E4
(b) Kaiming-E22
(c) Kaiming-E104
(d) Kaiming-E199 -19m
4.0
4.0
4.0
4.0
3.5
35
200
15
1.5
15
15
1.0
1.3
1.0
1.0
0.5
05
0..00
-2.5
9.2
-05
-3.
0.0
0.
-05
0.0
05
Welphh Value
Weight
WNigh
Veight Value
(c) IDInit-E4
(f) IDInit-E24
(g) IDInit-E104
(h) IDInit-E194
Figure 17: Histograms of the first convolution weights in ResNet-20. "E" means the epoch index. IDInit contains more zero values in each epoch compared with Kaiming initialization.
The results are shown in Figure 17 ginning, while Kaiming more zero values than Kaiming, which is beneficial for memory occupation since a 0 value will not cost memory space.
weights initialized with IDIni are almost full of zero at the be- Gaussian distribution. At the end of the training, IDInit still contains
24

---

Published as a conference paper at ICLR 2025
C.5 ANALYSIS ON INPUT-OUTPUT JACOBIAN
Here we conduct an experiment on Cifarl0 with 64 blocks in Figure demonstrate IDInit follows the dynamical isometry. We the open-source code We remove batch normalization for the more clear difference between IDInit and Kaiming. We use an Adagrad optimizer with a learning rate 0.01. The batch size 100. The activation is ReLU. The experiment conducted on one Nvidia A100.
Awity 22 E1 1450
Avd2 226 INSOAN
Argg 22EL L
Awil 21T9 #y LE
0.0
0.8
0.0
0.0
0.4
0.4
0.0
0.4
0.0
0.0
0.0
(a) Default-E1
(b) Default-E2
(c) Default-E3
(d) Default-E4
1.0
Aug
0.
0.5
0.6
0.6
0.2
0.2
0.2
02
0914.)
(c) IDInit-BI
(f) IDInit-E2
(g) IDInit-E3
(h) IDInit-E4
Figure 18: Histograms of log singular values (log(A..)) for the input-output Jacobian. "E" means the epoch index. Compared with Default initialization, IDInit has a significantly smaller squared singular value x. which can achieve a faster reduction of the loss.
Default initialization cause a high squared singular value x. reaching more than 2000. Compared to Default, IDInit only derives X around 1, indicating correspondence to the dynamical isometry. In addition, the loss of IDInit decreases faster than Default, which shows good convergent ability.
As shown in Figure 18
C.6 FAILURE OF LONG RESIDUAL STEM
We conduct this experiment to show the failure case when the residual stem is long to show the importance of the stability of the residual stem. In this experiment, we conduct an experiment on Cifar10. We use a residual network named Res- 112 as in Table We set 109 layers in the residual stem. Batch normalization is not applied for fairly validating the stability of initialization methods. We use an SGD optimizer with a learning rate 0.2, and weight decay 1e-8. The batch size is 768. Training epochs are 35. The learning rate is reduced with cosine function. One Nvidia A100 is used.
Results are shown in Figure 19 When the network is trained for 4 epochs, both Kaiming and Fixup fail to train the network, since the standard derivations of their outputs explode. By contrast, IDInit successfully trains this network and the standard derivation of the output converges a stable value. This experiment demonstrates the ability of IDInit to stabilize the residual stem, which can benefit the training of the whole network.
C.7 EXPERIMENT ON MOP-BAAF-MOE:
We conducted experiments GPT- Base- MOE, modifying the GPT-Base with 8 experts. The train- ing settings mainly follow Pan et al. (003). The results, shown in Figure 20 indicate that IDInit can achieve 20% faster performance compared to the default random initialization, demonstrating the superior performance of IDInit.
Deee_ro-Deee__eee Fast_ N ipynb
25

---

Published as a conference paper at ICLR 2025
nan
60
Standard Derivation
5
IDnit
Kaiming
Test Accuracy
50
Foxup
40
IDinit Kaiming Fixup
3
30
20
1
10
0
5
10
15 Epoch
20
25
30
5
10
15 Epoch
20
25
30
(a) Test Accuracy
(b) Standard Derivation
Figure 19: Result of the experiment on the residual network with the long residual stem. Fig- ure [9(a) shows the accuracy of different initialization. Figure [9(b) shows the standard derivations of the outputs of networks with different initialization methods. The black dash line is the standard derivation of the network input.
Table 10: Architectures Res-112 Window means the convolutional kernel window size. Channels indicate and Cout of standard convolutional kernel C E CR XXxx The avg pool denotes the average pooling operation. Linear means linear layer.
Layer
Window
Channels
conv1
3x3
3x16
3x3
[16x 16] 18
16x32
3x3
Residual Block
[22 17
32x64
3x3
[64x64]x17
3x3
64x64
64x64 64
conv2
3x3
avg pool
Linear
64x 10
C.8 EXPERIMENT ON DIT
We train DIT-S/4 ImageNet using the provided code The experiment is conducted using the default training settings. As illustrated in Figure 21 IDInit consistently achieves faster convergence compared to the default initialization.
D DYNAMICAL ISOMETRY IN IDINIT
Following Bachlechner al. 2021)
utilize simple example of the mechanism that dynamical isometry helps DDIit to obtain a fast convergence. Considering T-llaeer network with a simple special case of Eq. (I:
(4) ())
(37)
26

---

Published as a conference paper at ICLR 2025
Detuut
6.5
6.0
5.0
4.5
3.5
6400 (1.78)
12608 (3.38)
(0.08)
step (token)
Figure 20: Pretraining on GPT-Base- MOE. IDlinit can achieve 20% after than Default initialization.
0.200
Default IDinit
0.195
0.190
Test Loss
0.185
0.180
0.175
0.170
0
10000
20000
Step
Figure 21: Training on DiT-S/4.
where (1) and (2) denote the first weight and last weight in a residual stem respectively, and (x()) is the feature in layers. E {0, 13 determines residual connection. Specifically, r and r = represent non-residual and residual conditions respectively. The Jacobian of Eq. (37 is Jor = (r + (m) Obviously, identity transition on both non-residual and residual settings, namely (r = 0, (2) = (1) = 1} and fr = 1, w((1) 1, (2 = 0} respectively, will achieve Joz. which conforms to the dynamical isometry mechanism that helps improving training ability (Pennington et al.] (2017). Further, we delve into a gradient update analysis. Following gradient descent, wi can be updated with
Am (2) + w (1)00))))))1))
(38)
where R means the loss function, and is a learning rate. As (1)) and (() are equivalent in Eq. (37 (2) can be updated similar to Eq. (38). When (1)) = 1, updates are required less than 1. Therefore, the learning rate is constrained to
A a 1xx 10 (2))-11 if residual.
if non-residual,
(39)
27

---

Published as a conference paper at ICLR 2025
188
1.6
16
14
14
12
0.8
0.6
0.6
0.1
0.4
0.2
5
(a) Non-Residual Plot.
(b) Residual Plot.
Figure 22: Contour plots the log gradient norm log [|0]||2 on non-residual and residual net- works. (1) and (2(2) are both weights. The training process set as Bachlechner et al. (2021), which conducted on ground-truth x(L) 50 25 via gradient descent using a training set of
25 (1., 1.1, ...y 1.8]. (a) shows {w((2) w()) 1} can avoid poorly conditioned regions around
0, and converge
w(()) 2.19. (b) cares about initial position (w(1) 0, w(() 1)
and {w]()) 1. (1) 0}. The points' trajectories do not also pass the poor regions around
w(!) 1, (2) 1 and converge to the solution w()) 1.19.
For the non-residual condition, the learning rate polynomial to I., thereby insensitive to the depth. By contrast, in the residual block, w.(2) >> will cause learning rate exponentially small and wx(2) 1 also cause gradient diffusion. On this condition, setting w((2) 0 can be good solution for avoiding large output and restricting gradients in a suitable norm. Besides, is feasible to update (2) with the first non-trial step
w (2)
XLa (1) (0) R()))))==2()
(40)
and will converge with learning rate that is polynomial in the depth L of the network. We plot the training dynamics in Figure 22 and use this simple example illustrate the mechanism IDInit, which is always well-conditioned position for training.
28


Reviews:
[{"review_id": "reBOuH0SH5", "reviewer": "Reviewer_5YxS", "summary": "This manuscript studies the initialization methods for deep neural networks. Through an elaborated problem identification, the manuscript proposes several treatments, including (1) padding a new identity matrix in adjacency to an identity matrix, (2) using optimizers with momentum to tackle the convergence issue, (3) improving the universality of IDInit and the identity-control framework from the perspectives of higher-order weights and dead neurons. Extensive numerical results over various neural architectures and datasets demonstrate the effectiveness of the proposed strategy.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The manuscript is well-structured and well-written. Most concepts are carefully explained and form a good story.\n* The numerical evaluations look extensive, considering several baselines and neural architectures on image classification, text classification, and llm pre-training.", "weaknesses": "1. The manuscript may need to elaborate the discussion on the momentum part, e.g., by providing some theoretical justifications. \n2. Some design/hyper-parameter choices are not justified by some references, e.g., many parts in Sec 4 (line 434-line 439, line 449-line 454, line 464-line 474, line 504-line 509, line 517-line 523)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOzBfXWsCq", "reviewer": "Reviewer_jXZf", "summary": "The authors propose an alternative method for neural network initialization that maintains the identity transition. The work uses the block identity-control initialization framework, and as such is composed of two blocks, one identity preserving  (which in case of non-square transformations repeats the identity along the sub-diagonals, instead of zero padding), and zero preserving transformation. The work also discusses how this concept can be used for convolutional architectures, and how its setting may mitigate the dead-neurons problem in identity control.", "questions": "1) In Appendix A3, what is the upper index in $x_i^{(0)}$ ? Why in $\\Pi_1$ in the derivative it changes to 3? Also, should not Theorem 3.1  also include some constraints on the batch size, since this will influence the rank of the gradients?   \n2) In Figure 2, I am not sure if I understand what Identity-1 is? Is it simply initializing W1 to Identity (since the matrices  for plot 2b are squared)?   \n3) In Equation (3), how is $\\tau$ determined for different activation functions? It does not seem to be specified in the paper.   \n4) I liked the hyperparameter experiments from Section 4.1 and actually maybe even wished for this part to be more extended. If IDInit improves the stability over other initialization schemes, it would be nice to highlight this more (e.g. repeat Figure 7 on more datasets, by commenting on the standard deviation of the loss during the runs, etc.)     \n\n**Minor**:\n\n- How do the results in Figure 2 change with varying matrix sizes?  \n- A suggestion: remove the word “Some” from the caption of Figure 8, as it diminishes the perceived significance of the experiment.  \n- The term “residual stem” was initially unclear. Introducing and defining this term earlier in the paper might help readers understand it more quickly.  \n- Lines 134–135: The phrase, “In this paradigm, the input-output Jacobian is calculated as…” is somewhat confusing. Equation 2 simply defines the input-output Jacobian, and its definition does not change based on whether it is viewed through the lens of dynamical isometry. I recommend removing the phrase “In this paradigm” for clarity.\n\n\nSince currently the weaknesses outweigh the strengths of the paper, I am slightly leaning more towards rejection rather than acceptance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-  The method is a simple addition to the family of identity-control initialization, which is easy to implement and use. The obtained results are in most studied cases competitive to, or better than other identity-control schemes   \n-  The investigation into the ways and mitigating the dead neuron problem with the use of small perturbations in the IDInit method seems interesting and potentially valuable to the community.   \n-  While the experiments on various hyperparameters (Section 4.1) are limited in the scope of datasets and models examined, they are potentially insightful, showing that switching to IDInit initialization can lead to a broader range of hyperparameter configurations performing effectively.", "weaknesses": "The are places in the paper in which I am not sure what the empirical evaluation serves to demonstrate, or whether the obtained results are  significant:\n\n1) Firstly, I have concerns regarding Figure 2.  I understand that it serves as a motivation plot for the later claims and I believe that comparing the impact of the initialization of matrix $W_1$ on the performance of various control-initialization setups could be a valuable study by itself. However, my issue is with the presentation of the results. It’s not clear a) What is the model used in this study? b) What is the dataset and task being evaluated? c) What is the type of loss measured and what does “Rectangle Loss” mean? Do the words “Square” and “Rectangle” in the description of the plots refer to the type of matrices used? Additionally, are the lines in Figures 2a and 2b averaged over multiple runs? If so, what is the standard deviation (this is also an opportunity to say something about the stability of those different methods)? Including this information would help demonstrate the stability of the different methods. Finally, where is “Random” in these plots? Is it obscured by another method? Since Figure 2 seems to be an important plot for establishing the importance of introducing the IDInit, I would wish for this plot to be very clear.\n\n\n2) Figures 8 and Table 3 are potentially valuable, as they involve the largest networks tested in the paper and might capture readers' interest. IDInit appears to outperform the default initialization in these cases, but a comparison with other identity-control methods would be useful to determine if the performance gain is due specifically to IDInit or identity-control in general. Although these experiments are resource-intensive, reporting the mean and standard deviation over multiple runs would help evaluate whether the performance differences are statistically significant.\n\nOverall, the paper is well-written, but there are sections with vague claims or unclear intent:\n\n3) Lines 110–114 state, “By further proposing modifications to CNNs and solutions to dead neuron problems we have significantly improved accuracy by…”. It is unclear what task, network, or dataset this refers to, or what it is being compared to—state-of-the-art methods or alternative identity-control techniques? Since this is part of the main contributions section (end of Section 1), this ambiguity is distracting, particularly as similar wording appears in lines 115–116: “On ImageNet, IDInit can achieve almost all the best performance…”. Again, compared to what? I suggest being specific or moving such claims to the experimental results section, where precise differences in accuracy can be reported.  \n4) The definition and understanding of dynamical isometry from the related work seems to be slightly inadequate. I.e. we say that a network attains dynamical isometry if all the singular values of the input-output Jacobian are equal to 1. This is not equivalent with operating at the criticality regime $X=1$. In fact, for certain initialization and activations (e.g. orthogonal with ReLU), it is possible to operate in the criticality regime, yet never obtain the dynamical isometry property (see Section 2.5.2 of Pennington 2017). I know it does not seem to be a huge issue, since the dynamical isometry is not the center of attention of this paper, but it raises doubts whenever dynamical isometry is mentioned in the paper.  \n5) In Section 3.1.1's convergence analysis, the statement, “as $\\eta$ is usually 1e-1 and both training pairs…” is unclear. It is not accurate to label any learning rate as “standard” or “usual,” which makes the comment about the “magnitude of the asymmetric component” vague. Instead, it would be more informative to discuss the constraints or boundaries for the $\\eta$ parameter that ensure broken symmetry.  \n\nThese issues, while not critical individually, collectively (together with some other issues -- see section “Questions”) make the paper challenging to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTKKpfPkOf", "reviewer": "Reviewer_bfvR", "summary": "In conclusion, this paper introduces Fully Identical Initialization (IDInit), a novel approach that leverages an identity-like matrix to effectively preserve identity across both main and sub-stem layers of residual networks. This method addresses the rank constraints in non-square weight matrices through a padded identity-like matrix and improves upon traditional identity matrix convergence issues using stochastic gradient descent. By processing higher-order weights and tackling dead neuron problems, IDInit enhances stability and performance, demonstrating improved convergence in various settings, including large-scale datasets and deep models.", "questions": "- Could you include additional experimental results involving self-supervised learning methods and diffusion models?\n- Could you provide further theoretical analysis on the convergence rate of IDInit?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces IDInit, a novel initialization method that utilizes an identity-like matrix to preserve identity across both the main and sub-stem layers of residual networks. IDInit accelerates the training process across diverse datasets and neural architectures.\n- An effective initialization method is crucial for the deep learning community.\n- The paper offers a clear presentation, including well-demonstrated experimental results and a robust technical solution.\n- Several claims are theoretically supported. Notably, the paper demonstrates how this method can address the rank constraint problem.", "weaknesses": "- In certain instances, IDInit remains suboptimal. For example, in Table 1, the final accuracy of a 110-layer neural network does not surpass the baselines.\n- The proposed IDInit has not been validated across diverse learning paradigms, such as self-supervised learning methods [1] and diffusion models [2].\n- The paper lacks theoretical analysis regarding the convergence rate of IDInit.\n\n[1] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in neural information processing systems 33 (2020): 6840-6851.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vEEBvpZXQW", "reviewer": "Reviewer_96yw", "summary": "This paper proposes a new initialization method for deep neural networks. It aims to initialize residual branches to have output close to zero, but keep an identity-like initialization in linear transformations on the residual except for the final one. They focus on the case where the feature dimension of the residual differs from that of the skip connection (typically wider), covering both convolutional and MLP style residuals. The authors analyze their initialization and experimentally validate its performance across various neural networks.\n\n**Recommendation:** I lean towards rejection. The core idea is interesting and the results seem promising but I think the paper would benefit from a significant rewrite to improve the presentation clarity as well as better controlled comparison experiments.", "questions": "- Why change the notation from Figure 1 to Equation 1? What do you mean when applying the activation function to the weights / transformation in Equation 1? This notation is not defined or explained.\n- Many aspects of Figure 2 are unclear. Does this network have activation functions? Why is the learning rate not tuned per method? Why is the rank issue so significant here despite the matrices being almost square? I find it hard to believe that a difference of 40 features out of 280 can cause such a large difference between the methods. Why use Adagrad and not a more standard optimizer like SGD or Adam?\n- Figure 3: I think this would be clearer if you used one symbol for the number of features on the skip path and another for the intermediate dimension. Here D_i is used twice to mean different dimensions.\n- Figure 4: I feel that when comparing momentum to no momentum the learning rate must be taken into account. Using momentum in SGD increases the effective learning rate so it is not surprising that the difference is larger.\n- Lines 314-318: Here you introduce the “loose” condition where you add random noise to your leading matrices. However, this is not mentioned again. Are the later experiments conducted with this change and does it make a difference in practice?\n- Section 4.1: I don’t find this convincing. The granularity of the sweep is too high and the best performance for the Kaiming initialization occurs at the edge of the sweep.\n- Section 4.2: I also don’t find this very convincing. In general the hyperparameters need to be tuned per method for a fair comparison. How do we know that your hyperparameters don’t favor your method?\n- Section 4.3: Here I feel you could explain the exact setup a bit better, e.g. what exactly do you do without IDIC?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Problem tackled is practical and of interest to the community \n- The core idea is interesting\n- Authors claim decent performance improvements", "weaknesses": "- This paper is very hard to follow. It frequently relies on citing prior works without summarizing or explaining their results sufficiently. The setup is not explained well, it is often hard to tell if there are non-linearities between the matrices, if normalization layers are used etc. The notation is not consistent throughout the paper. Finally the paper is full of spelling and grammar mistakes as well as poorly phrased sentences.\n- Some of the experiments are not convincing e.g. in Section 4.1 and 4.2. In general the hyperparameter tuning might not provide a fair comparison between methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)