Towards Theoretically Inspired Neural Initialization Optimization
Authors: Yibo Yang
Hong Wang
Haobo Yuan
Zhouchen Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: initialization optimization

Abstract: Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters. In the domain of neural initialization, however, similar automated techniques have rarely been studied. Most existing initialization methods are handcrafted and highly dependent on specific architectures. In this paper, we propose a differentiable quantity, named GradCoisne, with theoretical insights to evaluate the initial state of a neural network. Specifically, GradCosine is the cosine similarity of sample-wise gradients with respect to the initialized parameters. By analyzing the sample-wise optimization landscape, we show that both the training and test performance of a network can be improved by maximizing GradCosine under gradient norm constraint. Based on this observation, we further propose the neural initialization optimization (NIO) algorithm. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost compared with the training time. With NIO, we improve the classification performance of a variety of neural architectures on CIFAR10, CIFAR-100, and ImageNet. Moreover, we find that our method can even help to train large vision Transformer architecture without warmup. 

Towards Theoretically Inspired Neural Initialization Optimization
Yibo Yang', Hong Wang", Haobo Yuan Zhouchen Lin
JD Explore Academy, Beijing, China
Keyy Lab. of Machine Pereeption (MoE), School of Intelligence Science and Technology, Peking University IIstiiute of Artificial Intelligence and School of Computer Science, Wuhan University Institute for Artificial Intelligence, Peking University Pazhou Laboratory, Guangzhou, China
Abstract
Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters. In the domain of neural initialization, however, similar automated techniques have rarely been studied. Most existing initialization methods are handcrafted and highly dependent on specific architectures. In this paper, we propose differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. Specifically, GradCosine is the cosine similarity of sample-wise gradients with respect to the initialized parameters. By analyzing the sample- wise optimization landscape, we show that both the training and test performance of network can be improved by maximizing GradCosine under gradient norm constraint. Based on this observation, we further propose the neural initialization optimization (NIO) algorithm. Generalized from the sample-wise analysis into the real batch setting, NIO able to automatically look for better initialization with negligible cost compared with the training time. With NIO, we improve the classification performance of a variety neural architectures on CIFAR- 10, CIFAR-100 100, and ImageNet. Moreover, find that our method can even help train large vision Transformer architecture without warmup.
Introduction
For deep neural network, architecture [19, 20, 21] and parameter initialization [18, 16] are two initial elements that largely account for the final model performance. Lots human efforts have been devoted to finding better answers with respect to the two aspects. To automatically produce better architectures, neural architecture search [62, 38, 5] has been research focus. However, on the other hand, using automated techniques for parameter initialization has rarely been studied.
Previous initialization methods are mostly handcrafted. They focus on finding proper variance patterns of randomly initialized weights [18, 16, 40, 36] or rely on empirical evidence derived from certain architectures [58, 23, 15, 7J. Recently, [60, 8] propose learning-based initialization that learns to tune the norms of the initial weights so as minimize a quantity that is intimately related to favorable training dynamics. Despite being architecture-agnosti and gradient- based, these methods do not consider sample-wise landscape. Their derived quantities lack theoretical supports for being related model performance. It is unclear whether optimizing these quantities can indeed lead to better training generalization performance.
In order find a better initialization, theoretically sound quantity that intimately evaluates both the training and test performance should be designed. Finding such quantity is non- non-trivial problem and
corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

shows many challenges. First, the test performance is mostly decided by the converged parameters after training, while the training dynamic is more related to the parameters at initialization or during training. Second, to efficiently find a better starting point, the quantity is expected to be differentiable to enable the optimization in the continuous parameter space. To this end, we leverage the recent advances in optimization landscape analysis [2] to propose novel differentiable quantity and develop a corresponding algorithm for its optimization at initialization.
Specifically, our quantity is inspired by analyzing the optimization landscapes of individual training samples [59]. Through generalizing prior theoretical results on batch-wise optimization [2] to sample- wise optimization, we prove that both the network's 's training and generalization error are upper bounded by a theoretical quantity that correlates with the cosine similarity of the sample-wise local optima. Moreover. this quantity also relates to the training dynamic since it reflects the optimization path consistency [35, 37] from the starting point. Unfortunately, the sample-wisg local optima are intractable. With the hypothesis that the sample-wise local optima can be reached by the first-order approximation from the initial parameters, we can approximate the quantity via the sample-wist gradients at initialization. Our final result shows that, under a limited gradient norm, both the training and test performance of a network can be improved by maximizing the cosine similarity of sample-wise gradients, named GradCosine, which is differentiable and easy to implement.
We then propose the Neural Initialization Optimization (NIO) algorithm based on GradCosine to find a better initialization agnostic of architecture. We generalize the algorithm from the sample-wise analysis into the batch-wise setting by dividing a batch into sub-batches for friendly implementation. We follow [8, 60] using gradient descent to learn a set of scalar coefficients of the initialized parameters. These coefficients are optimized to maximize GradCosine for better training dynamic and expected performance while constraining the gradient norm from explosion.
Experiments show that for variety of deep architectures including ResNet [19], DenseNet [21], and WedeResNNt [56], our method achieves better classification results on CIFAR-I0/10 [27] than prior heuristic [18] and learning-base [8, 60] initialization methods. We can also initialize ResNet-50 [19] on ImageNet [9] for better performance. Moreover, our method is able to help the recently proposed -ran-Transformer [32] achieve stable training and competitive results on ImageNet even without warmup [17]. which is crucial for the successful training of Transformer architectures [31, 52].
2
Related Work
2.1
Network Initialization
Existing initialization methods are designed to control the norms of network parameters via Gaussian initialization [16, 18] or orthonormal matrix initialization [40, 36] with different variance patterns. These analyses are most effective for simple feed-forward networks without skip connections or normalization layers. Recently, initialization techniques specified for some complex architectures are proposed. For example, [58] studied how to initialize networks with skip connections and [23] generalized the results into Transformer architecture [49]. However, these heuristic methods are restricted to specific architectures. Automated machine learning has achieved success in looking for hyperparameters [3, 13] and architectures [62, 38. 5, 54, 55, 22]. while similar techniques for neural network initialization deserve more exploration. Current learning based initialization methods [8, 60] optimize the curvature [8] or the loss reduction of the first stochastic step [60] using gradient descent to tune the norms of the initial parameters. However, these methods lack theoretical foundations to be related to the model performance. Different from these methods, our proposed GradCosine is derived from a theoretical quantity that is the upper bound of both training and generalization error.
2.2 Evaluating Model Performance at Initialization
Evaluating the performance of network at initialization has been an important challenge and widely applied in zero-shot neural architecture search [1, 34, 6, 42, 59, 41, 29] and pruning [50, 28, 44]. The evaluation quantities in these studies are mainly based on the initial gradient norm [44, 41], the eigenvalues of neural tangent kernel [6, 41J, and the Fisher information matrix [47, 48, 45). However, these quantities cannot reflect optimization landscape, which is crucial for training dynamic and generalization [4, 12, 14, 43, 30, 2]. [2] provided theoretical evidence that for a sufficiently large neighborhood of random initialization, the optimization landscape is nearly convex and semi-
2

---

( I
01
65
0
05 62
0
(d)
Figure I: (a) Optimization landscape with sparser sample-wise local optima corresponds a worse 0" (larger loss 1). (b) Optimization landscape with denser sample-wise local optima corresponds to a better 0* (smaller loss I). However, the density of sample-wise local optima cannot reflect training path. We further leverage the cosine similarity of sample-wisg local optima. Under the same local optima density, (d) corresponds to a better training dynamic than (c), since (d) enjoys better optimization path consistency (smaller cosine distance between 0 Gg and 0 0o). We give more detailed explanations and discussions in Appendix C.
smooth. Based on the result, [59] proposed to use the density of sample-wise local optima to evaluate and rank neural architectures. Our study also performs sample-wise landscape analysis, but differs from [59] in that our proposed quantity is differentiable and reflects optimization path consistency, while the quantity in [59] is non- con-differentiable so cannot serve for initialization optimization. We make a comparison in more details in Appendix C.
3
Theoretical Foundations
3.1 Sample-wise Optimization Landscape Analysis
Conventional optimization landscape analyses [30, 2] mainly focus on the objective across mini- batch of samples and miss potential evidence hidden in the optimization landscapes of individual samples. As recently pointed out in [59], by decomposing mini-batch objective into the summation of sample-wise objectives across individual samples in the mini-batch, they find that the network with denser sample-wise local optima tends to reach a better local optima for the mini-batch objective, as shown in the Figure (1) (a)-(b). Based on this insight, they propose to use sample-wise local optima density to judge model performance at initialization.
Density of sample-wise local optima. For batch of training samples S = {(z,, Bi)))(()) loss function 1(0i, ys), and a network fo(-) parameterized by 0 E TR.", the sample-wise local optima density V s. (-)) is measured by the averaged Manhattan distance between the pair-wise local optima {0; hiel)) of all samples near the random initialization 0g [59], i.e.
VH n
ss.((fo,())
10 Σ 1
-8111. 6.j E [1.n],
(1)
where H is the smoothness upper bound: Vk E [m], [n), (V=(((())( v.))a. S H and H always exists with the smoothness assumption that for a neighborhood Te, of random initialization Oo. the sample-wise optimization landscapes are nearly convex and semi-smooth [2]. Based on this assumption, the training error of the network can be upper bounded by " (z.(foo Moreover, with probability 1 1-5, the generalization error measured by population loss (<.....)> (f yn)l is upper bounded by = (f,,(,, (-)) (ff,
+ Vin where D = {(x... V/. hee(r) is the underlying data distribution of test samples and o² is the upper bound of Var
17 [59].
Although the sample-wise local optima density is theoretically related to both the network training and generalization error, we point out that it may not be suitable quantity to evaluate network initialization due to the following reasons. First, it ignores the optimization path consistency from initialization Og each sample-wise local optimum 07 As shown in Figure/ (c)-(d), while both (c) and (d) have the same local optima density, in Figure (d) the optimization path from the initialization

---

to the local optima of the two samples are more consistent. Training networks on samples with more consistent optimization paths using batch gradient descent naturally corresponds to a better training dynamic that enjoys faster and more stable convergence [39, 37]. Second, the sample-wise local optima in Eq. (1) are intractable. can approximated by measuring the consistency of sample-wise gradient signs [59]. But it is non-differentiable and cannot serve for initialization optimization.
Based on this observation, we aim derive a new quantity that directly reflects optimization path consistency and is a differentiable function of the initialization Op. Our proposed quantity is based on the cosine similarity of the paths from initialization to sample- -iise local optima.
Cosine similarity of sample-wise local optima. Concretely, our quantity can be formulated as:
2
@s,(foo()))
cos <(0; - 0o. 07 00.00-00)) i,j E [1,
3
where a and 3 the maximal and minimal Ez-norms of the sample-wise optimization paths, i.e. = max 0olla), 8 = min(1 0)112), Vi E n.]. and cos <(0 - 0. 03 0o) refers to the cosine similarity of the paths from initialization 0g sample-wise local optima, 0ī and 07. The cosine term in Eq. (2) reflects the optimization path consistency. Together with the distance term 3. is also able measure the density of sample-wise local optima. In the ideal case, when all the local optima are located at the same point, (fo, (-)) = V (fo, = 0. Hence, compared with V our θ is more suitable for evaluating the initialization quality.
3.2 Main Results
In this subsection, we theoretically illustrate how minimizing the quantity in Eq. (2) corresponds to better training and generalization performance. Similar to [59], our derivations are also based on the evidence that there exists neighborhood for random initialization such that the sample- wise optimization landscapes are nearly convex and semi-smooth [2].
There exists no saddle point in a sample-wise optimization landscape and every local optimum is global optinum [59].
Lemma 1.
Based on Lemma 1, we can draw relation between the training error and Os, (fe. we show that the proposed quantity is also related to generalization performance as the upper bound of population error. We present the following two theoretical results.
(-)). Moreover,
Theorem 2. The training loss L = 1 E. 1ff- (x:), 3i) of a trained network fo- on a dataset S = (I,, ve) Hietn] is upper bounded by Os. (a, (-)), and the bound is tight when @sa(fo, (-)) O.
Theorem 3. Suppose that o² is the upper bound of V ar (=.... 1110~ - 0 1:). where 0% is the local optimum in the comvex neighborhood of Go for test sample (T Bu). With probability 8, the population loss ((r...))Dll (Iw), yw)] is upper bounded by (, (-)) + Vno
We provide proofs of these theorems in Appendix A. Combining both theorems, can conclude that @s.(fe, (-)) is the upper bound of both training and generalization errors of the network fe-. Therefore. minimizing & theoretically helps to improve the model performance.
Albeit theoretically sound, (fo (.)) requires sample-wise optima 0: which are intractable at initialization. To this end, we will show how to develop differentiable and tractable objective based on Eq. (2) in Section 4, and introduce the initialization optimization algorithm in Section 5.
GradCosine
4.1 First-Order Approximation of Sample-Wise Optimization
Since we are dealing with saiple-wise optimization, is reasonable to calculate each sample-wise optimum by the first-order approximation. We hypothesize that each sample-wise optimum can be reached via only one-step gradient descent from the initialized parameters. Its rationale lies in that it is very easy for deep neural network to learn the optimum for only one training sample with gradient descent. Based on this hypothesis, we can approximate each local optimum as:
0 0o 79t, i [1,
(3)

---

Algorithm 1 GradCosine (GC) and gradient norm (GN)
Require:
Initial network parameters 8u. and batch of samples S {(2,, 84 B.]BE B 18).
I: for i=1 do
2: Compute the sample-wise gradient:
f- (z,( y6)l gi E IRm
3: end for
4: Compute the average of gradient norm:
GN(S, 0o) t B -i= ||9:||2
5: Compute the cosine similarity of gradients:
Pi. ll. l2- les 11 I,.., B.j = ,,... B
Compute average of gradient cosine:
GC(S, 0o) < 82 ΣE Σ Φi.
7: Output GN and GC.
where gi Vol(fo(x.)) el is the sample-wise gradient at initialization, and 17 is the learning rate. With Eq. (3), the upper bound quantity in Eq. (2) can be simplified as:
Hymaz
Σ
Imaz cos ((x. i.j E [1,n)
(4)
71
Imin
where Imaz and Imin are the maximal and minimal sample-wise gradient norms at initialization, i.e., Imax max( (9:l12), Imin = (2nn))9:112)) Vi e [1, n].
4.2 Guidance for Initialization
Suppose that the sample-wise gradient at initialization is upper bounded by constant 7, i.e., Imax S y, and then we have:
Hy²
7 - cos cos <(r i.j ((9
θ
[1,
(5)
n2
Jmin
From Eq. (2) to Eq. (5), we have almost converted the upper bound into quantity that can be easily calculated by the sample-wise gradient at initialization, except for the term Jmin and the constraint Imar S 7. We will explicitly add corresponding optimization objective and constraint in neural
initialization optimization algorithm in Section to ensure small 7//min-
From Eq. (5), can draw some useful guidance for a better initialization:
(1) The sample-wise gradient norms of the initialized parameters should be large and close to the maximal value bounded by to induce small 7/9mini
(2) The cosine similarity of the sample-wise gradients should be large as possible.
Relations to Favorable Training Dynamic. There are significant evidences in existing studies that support the two rules for better training dynamic. The first rule is intimately related to the neural tangent kernel (NTK) [25] that has recently been shown to determine the early learning behavior [51, 41, 6, 42]. Specifically, [41] finds that the training dynamic of neural network can be characterized by the trace norm of NTK at initialization and further approximates via the initial gradient norm gauge to search for neural architectures. [60] initializes network by maximizing the loss reduction of the first gradient descent step, which also corresponds to gradient norm in the first-order approximation of the loss function. The two rules together prefer a model whose sample-wise gradients have close & and cosine distances. is in line with prior observations that the initial gradient variance should be small enable large learning rate [31, 61]. However, gradient variance is not a proper optimization objective due to its high sensitivity to gradient norm.
While the first rule of improving the initial gradient norm has been suggested for better training dynamic in prior studies [60, 41], the second rule completely new. We refer this novel quantity, i.e. cos < 9j), for evaluating the network initialization as GradCosine. Intuitively. in the ideal case where the initial gradients of all samples are identical, we have the smallest upper bound of training and generalization error. Moreover, both gradient norm and GradCosine are differentiable and thus enable optimization to find better initialization. We provide the calculation of gradient norm (GN) and GradCosine (GC) for initialized network in Algorithm 1.
5

---

Algorithm 2 Batch GradCosine (B-GC) and batch gradient norm (B-GN)
Require: Initialized network parameters 0b. batch of samples S {(2,, 34) [e)) the number of sub-batue batches D. and overlap ratio
1: for = 1 to do
2: Compute the batch-wise gradient:
Id N Ejes, Vol(fo(a;)) V1)00
3: end for
4: Compute B-GN((S, 8o) and B-GC(S, 0a) with batch-wise gradients following Lines (4-6) in Algorithm 1:
5:
Output B-GN and B-GC.
B; y=0 0 Ace 96.03 (saple-wwise)
4; 0, Acc 95.80
(batch-setingg)
0.25, 95.94 (batch-setting))
(batch-setting:)
Figure 2: illustration sample-wise inputs and batch-wise inputs with different batch settings. The example experiment is conducted on CIFAR-10 with ResNet-110.
5 Neural Initialization Optimization
Based on the GradCosine, we propose our Neural Initialization Optimization (NIO) algorithm. Our algorithm follows [60, 8] to rectify the variance of the initialized parameters in network via gradient descent. Concretely, we introduce set learnable scalar coefficients denoted as M {w1, ... W. The initial parameters 0,- [...... Wm rectified by these coefficients can be formulated as 0M = for W,, .mm m.) ). The NIO algorithm mainly solves the following constrained optimization problem:
maximize M
GC(S, A + (N(SSNN0)
subject to max S i = 1-,1... 1.... B,
where GC and GN refer to GradCosine and gradient norm, respectively, as calculated Algorithm 1, and B is the batchsize of the batch data S. With the maximal sample-wise gradient norm bounded by y, we maximize both GradCosine and gradient norm.
Although the problem in Eq. (6) is differentiable and tractable, inferring GradCosine and gradient norm in the sample-wise manner can be very time-consuming especially for networks trained on large datasets such as ImageNet. Moreover, using gradient-basse methods to solve Eq. (6) requires deriving the second-ordes gradient with respect to the network parameters, which leads to unbearable memory consumption in the sample-wise manner. To this end, we introduce the batch-wise relaxation.
5.1 Generalizing to Batch Settings
To reduce the time and space complexity of gradient calculation, further generalize GC and GN from the sample-wise manner into a more efficient batch setting. Specifically, for mini-batch of training samples 5 {(r,, [e()) we split it into sub-batches S1,.., Sd, ..., Sp with an overlap ratio of that is used to relieve the violation of the sample-wist setting. The number of samples in each sub-batch is = D- 1. and each sub-batch can be denoted as:
S = { (4-
where d - 1,.,, D. The sample-wise input can be regarded as the special case of the batch split when setting D= B, = O. We further illustrate some split examples in Figure 2. We empirically find that making the sub-batches overlapped with each other stabilize the optimization and results in better initialization. As shown Figure 2, the test accuracies of the models trained using our initialization with different batch settings further confirm this practice.
The GradCosine and gradient norm under the batch setting are illustrated Algorithm 2. The main difference is that, in the batch setting, the gradient is computed as the average of gradients in

---

Algorithm 3 Neural Initialization Optimization
Require: Initialized parameters 0p E R"" learning rate for scale coefficients M, upper bound of gradient norm 7, lower bound of the scale coefficients sub-batches D, and overlap ratio r.
0.01, total iterations T, batch size B, the number of
1: M 1. where denotes the all-ones vector in R";
2: for f = 1 T do
3: Sample St from the training set;
4: Compute 9t. B-GC(S,, 9 xrie) ), and (-GN( A ((t)) by Algorithm 2;
5: Compute the 9mar max: (19:1)))))))). i 1, L,... D:
6:
if gmuzz ? then
7:
((++1 - M -V Miti B-GN
8:
else
9: M M0 -V
-VVM Min (B-GC B-GN)
end if
11: Clamp ((++1) using a;
12:
end for
13: Output the rectified initialization parameters 0xr (T)
sub-batch. Our optimization problem in the batch setting can be formulated as:
maximize B-GC(SS 0M) + B-GN(S, 0M), max([19](1) < 7, = 1,.... D,
M
subject to
where B-GC and B-GN refer to the batch-wise GradCosine and gradient norm as calculated in Algorithm 2.
5.2 Main Algorithm
The final neural initialization optimization is illustrated in Algorithm 3. It rectifies the initialized parameters by gradient descent solve the constrained optimization problem in Eq. (8).
Concretely, we iterate for iterations. At each iteration, a random batch data S. is sampled from the training set, and divided into sub-batches according to D and We calculate B-GN and B-GC by Algorithm If the maximal gradient norm is larger than the predefined upper bound y, we minimize the averaged batch gradient satisfied, we maximize the GradCosine (B-GC) and gradient norm (B-GN) objectives simultaneously. in order to minimize the upper bound of the quantity θ in Eq. (5), which intimately corresponds to not only a lower training and generalization error, but also a better training dynamic.
to avoid gradient explosion at initialization. When the constraint is
6 Experiments
6.1 Initialization for CNN
Dataset and Architectures. We validate our method on three widely used datasets including CIFAR10/100 [27] and ImageNet [9]. We select three kinds of convolution neural networks includ-
ing ResNet [19], DenseNet [21], and WideResNet [56]. On CIFAR we adopt ResNetl 10, DenseNet100, and the 28-layer Wide ResNet with Widen Factor (WRN-28-10) as three main architectures for evaluation. To further show that our initialization method helps training dynamic for better training stability, we also conduct experiments on the same architectures but remove the batch normalization (BN) [24] layers. Moreover, to illustrate that our method can be extensible to large-scale benchmarks, we test our proposed NIO on ImageNet using ResNet-50. The detailed settings for different architectures and datasets are described in Appendix B.
Experiment Results on CIFAR-10/100, We select four different initialization methods for com- parison: (1) Kaiming Initialization [18]; (2) First train the network for one epoch with a linear warmup learning rate, denoted as Warmup; (3) Metalnit [8]; and (4) GradInit [60]. (3) and (4) are learning- based initialization as ours. We re-implement their methods using the code provided in [60]. For Metalnit, GradInit, and our proposed NIO, the initialization is rectified based on the Kaiming initialized parameters. After initialization, we train these models for 500 epochs with the same
7

---

Table 1: Test accuracies of three architectures on
Table 2: Test accuracies of three architectures on CIFAR-100. Best results are marked in bold.
CIFAR-10. 10. Best results are marked in bold.
Model
ResNet-110 DenseNet- 100 WRN-28-10
Model
ResNet- DenseNet- 100
WRN-28-10
w/ BN
w/ BN
Kaiming 95.53+ 0.19
95.75+ 0.13
97.27 0.27
Kaiming
74.54 0.21
76.58+ 0.23
81.40=+ 0.28
Warmup 95.56. 0.12
95.73 0.27
97.30=
0.18
Warmup
74.63 0.33
76.60 0.32
81.45 0.25
Metainit
95.45- 0.33
95.75+
0.11
97.26+
0.17
Metainit
74.32+ 0.26
76.23+
0.28
81.46 0.20
Gradinit 95.811 0.29
95.77 0.22
97.341 0.15
Gradinit
0.17 75.40
76.14:1 0.21
81.35 0.31
NIO
96.13 0.16
96.07 0.18
97.50
0.21
NIO
75.72 0.15
76.86-t 0.26
81.83 0.20
w/o BN
w/o BN
Kaiming 94.83++ 0.24
94.78+ 0.16
97.15+
0.33
Kaiming
73.03+ 0.23
71.27+ 0.25
79.28+ 0.26
Warmup 94.75.1 0.21
94.8610.11
97.20
Warmup
73.10:1 0.14
71.50:1 0.24
79.58 0.13
Metainit
94.79=00.15
95.15+
97.27
0.25
Metainit
72.60+ 0.17
71.68+
0.37
79.15 0.24
Gradinit
95.12.4:00 0.16
95.31
0.37
97.12 0.13
Gradinit
72.06
0.31
71.334
0.21
79.64 0.23
NIO
95.271 0.19
95.62 0.18
97.35 0.19
NIO
73.17 0.15
72.79
0.22
79.76 0.26
OPEi, gitic ring!
coOl, git0mm))
2.0
greainingl juun
21.0
1.4
-0.2
u.u
Coola LA
coolo vA Ho
1.0
1.
Layers
Randoo Batches
Wanidoon datches
64)
Figure 3: Empirical comparisons between the Kaiming initialization (red) and our NIO (blue) using ResNet-110 10 on CIFAR-10. (a) and (b) show the ratio of the maximal sample-wise gradient norm to the minimal one. (c) and (d) depict the averaged wiir-wise cosine similarities between sample-wise gradients. (a) and (c) are calculated for each layer with random batch. (b) and (d) are calculated for all parameters using 100 randomly sampled batches. Zoom in for a better view.
training setting. Each model is trained four times with different seeds. We report the average and standard deviation numbers of the accuracies on the test set.
As shown in Table 1 and 2, compared with the Kaiming initialization, observe that Metalnit and GradInit do not achieve improvement for some cases. As a comparison, our proposed NIO consistently improves upon the Kaiming initialization on both CIFAR-10 and CIFAR-100 with and without BN. Especially for R 10-BN and DenseNet-100- 100-nBNN,, the performance improvements upon the Kaiming initialization are more than 0.5% CIFAR- 10 and 1.0% on CIFAR-100. Compared with GradInit, we also achieve better results by more than 1.0% for ResNet-11O-noBI and DenseNet- 10- noBN CIFAR-100.
We also make some empirical comparisons between the Kaiming initialization and our NIO with ResNet- 110 CIFAR-10 10. As shown in Figure compared with the Kaiming initialization, NIO enjoys significantly smaller ratios of the maximal gradient norm to the minimal one, and larger cosine similarities of sample-wise gradients. Therefore, after NIO, the gradient norm ratio is reduced towards 1.0, while the cosine similarity is improved towards 1.0, which indicates that the quantity in Eq. (2) is minimized, and the sample-wise gradients close with respect to length and direction.
Ablation Studies on Batch Setting. We further test the sensitivity of our method to the batch setting hyper-paranne paremeeresssss in Eq. (7), the number of sub-batches D and overlap ratio We show the model performances and the time and space complexities of NIO with ResNet- 110 on CIFAR- using different D and in Tabel 3. The performances do not vary significantly with the change of D and r. The worst accuracy (when D = and r = 0) 75.56% in Table is still higher than the Kaiming initialization result 74.54% by 1 16. As D increases, the time and memory consumption of NIO also grows, because more batch gradients need to be calculated by back propagation. When is small, a larger should be adopted relieve the violation of the sample-wise hypothesis.
Experiment Results on ImageNet. To verify that our method can extensible to large-scale task, further evaluate our approach on the ImageNet dataset. Using different initialization methods,
8

---

Table 3: Model performance and complexity of ResNet-110 10 on CIFAR- with different batch set- tings. NIO is performed with batchsize of 128. smaller number of sub-batches D leads faster speed and less memory consumption, and almost does not harm the final performance. "Time" refers to the time consumed for each iteration in NIO.
Table 4: Top-1 accuracy of ResNet-50 on ImageNet. The results FixUp and Metalnit are taken from their papers [58, 8], and the other methods are re- implemented by us under the same training setting. FixUp removes the BN layers as their default and the other models are trained with BN layers.
Method
Accuracy (%)
Init. Train time
Accuracy (%)
D
r=0
r=0.2 r=0.4
r=0.6
Time
GPU Mem.
Kaiming [18]
76.43=00.05
-/8.5h
FixUp [58]
76.0
- -/8,5h
2
75.56
75.71
75.96
75.72
3.75s
5007M
Metalnit [8]
76.0
0.81h 8.5h
3
75.60
76.00
75.97
75.76
4.07s
6001M
GradInit [60]
76.50-10.005
0.21h 8.5h
4
75.58
76.03
75.71
75.88
4.68s
7067M
NIO (ours)
76.71-10.07
0.03h / 8.5h
Table 5: Top-1 accuracy of Swin- Transfornofrrr (tiny) on ImageNet with and without warmup. "fail" represents that the model cannot converge. Warmup takes up the first 20 epochs when enabled.
Kaiming
Truma
GradInit
NIO (ours)
w/ warmup
79.4 fail
81.3 fail
80.4
81.3
w/o warmup
79.9
80.9
we train ResNet-50 for 100 epochs with a batchsize of 256. Alll models are trained under the same training setting. As shown in Table 4, we improve upon the Kaiming initialization by 0.3%. We also re-implement GradInit using the default configurations their paper [60], and have performance close the Kaiming initialization baseline. Morcover, NIO takes the shortest initialization time compared with the other two learning-based initialization methods, Metalnit and GradInit. It is because only 100 iterations are needed for NIO in our implementation, while Metalnit and GradInit require 1000 and 2000 iterations, respectively, as their default settings [8, 60]. Compared with the training time, the initialization time of NIO is negligible.
6.2 Initialization for Transformer
Vision Transformer architectures have been popular and proven be effective for many vision tasks (11,46, 32, 53J. But Transformer architectures suffer from unstable training due to the large gradient variance [31, 61] in the Adam optimizer [26], so have to resort to the warmup trick, which adopts small learning rate at initialization to stabilize the training process. In order to test the effectiveness of NIO to improve the training stability, we perform NIO for Transfrrasfrrmer [32] with and without warmup on ImageNet. Detailed training and initialization settings are described in Appendix B. As shown in Table 5, when warmup not enabled for training, both Kaiming and Truncated normal initialized models cannot converge. As comparison, the model with NIO achieves top-1 accuracy of 80.9%, which is very close to the standard training result with warmup. These results indicate that NIO is able to produce better initialization agnostic of architecture and dataset.
6.3 Discussions
Why not Gradient Variance? As indicated by Eq. (2) and Figure 3, the aim of NIO is to reduce the variability of sample-wise gradients. But why do we not directly minimize the gradient variance or just the pair-wise Euclidean distances? We point out that gradient variance is highly sensitive to gradient norm. Directly minimizing gradient variance would lead to a trivial gradient norm near zero, which disables the training. So the ability of reducing gradient variance while keeping gradient norm at a proper range is one crucial property of our proposed NIO.
"TTrain time" is tested on an NVIDIA A100 server with a batchsize of 256 among 8 GPUs. "Init. time" of Metalnit and GradInit is tested for 1000 and 2000 iterations, respectively, according the default configurations their papers [8, 60]. NIO is performed for 100 iterations so enjoys fast initialization process.
Graddnit did not perform on SwinTransformer in their paper. We adopt their initialization settings for ResNet-S0 50 to perform GradInit SwinTransfornn and then train the model.

---

Limitations and Societal Impacts. Similar to [60], the choice of y has not been determined from theoretical point yet. Calculating GradCosine accurately requires much memory consumption for large models, so it needs the batch setting relaxation. Besides, performing NIO without dataset will be also promising breakthrough that deserves future exploration. Our study is general and not concerned with malicious practices with respect to privacy, public health, fairness, etc. The proposed algorithm is lightweight and does not bring much environment burden.
7 Conclusion
In this paper, we propose a differentiable quantity with theoretical insight to evaluate the initial state of a neural network. Based on our sample-wise landscape analysis, we show that maximizing the quantity with the maximal gradient norm upper bounded is able to improve both training dynamic and model performance. Accordingly, we develop an initialization optimization algorithm to rectify the initial parameters. Experimental results show that our method is able automatically produce better initialization for variant architectures improve the performance on multiple datasets with negligible cost. It also helps a Transformer architecture train stably even without warmup.
Acknowledgments and Disclosure of Funding
Z. Lin was supported by the NSF China (No.s 62276004 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606), and Project 2020BD006 supported by PKU-Baidu Fund.
References
[1] S. Abdelfattah, A. Mchrotra, L. Dudziak, and Lane. Zero-cost proxies lightweight nas. In ICLR, 2021.
[2] Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over over-ovvemeene parameterization. In ICML, pages 242 252. PMLR, 2019,
[3] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In NeurlPS,
volume 24, 2011.
[4] A. Brutzkus and A. Globerson. Globally optimal gradient descent for a convnet with gaussian inputs. In
ICML. pages 605-614. PMLR, 2017.
[5] L.-C. Chen, M. Collins, Y. Zhu, G. Papandreou. B. Zoph, Schroff, H. Adam, and Shlens. Searching efficient multi-scale architectures for dense image prediction. In NeurlPS, volume 31, 2018. [6] W. Chen, X. Gong, and Z. Wang. Neural architecture search on imagenet in four gpu hours: A theoretically
inspired perspective. In ICLR, 2020.
[7] D. Das, Bhalgat, and F. Porikli. Data-driven weight initialization with sylvester solvers. arXiv preprint
arXiv:2105. 10335, 2021.
Y. N. Dauphin and S. Schoenholz. Metainit: Initializing learning by learning initialize. In NeurlPS, volume 32, 2019.
[9] J. Deng, W. Dong, R. Socher, L.-J Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248 255, 2009.
[10] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout, arXiv preprint arxiv: 10XXv:/778,0055555 04552, 2017.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, Gelly, et al. image worth 16x words: Transformers for image recognition at scale. In ICI.R, 2020.
[12] S. Du, J. Lee, Y. Tian, A. Singh, and Poezos. Gradient descent learns one-hiddeen-lyer cnn: Don't be afraid spurious local minima. JCML, pages 1339-133000 1348. PMLR, 2018
[13] M. Feurer and F. Hutter. Hyperparameter optimization. In Automated machine learning, pages 3-33. Springer, Cham, 2019.
[14] R. Ge, J. D. Lee, and T. Ma. Learning one-hidden-Beee layer neural networks with landscape design. In ICLR,
2018.
[15] Gilmer, B. Ghorbani, A. Garg, S. Kudugunta, B. Neyshabur, D. Cardoze, Dahl. Z. Nado, and O. Firat. loss curvature perspective on training instability deep learning. arXiv preprint arXiv:2 10.04369,
2021.
[16] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS. pages 249-256, 2010.
[17] P. Goyal, P. Dollár. R. Girshick, P. Noordhuis. L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint 1706:1706000677 2017.
10

---

[18] K. He, X. Zhang. S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[20] A. G. Howard. M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Audreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 170..046444 2017.
[21] G. Huang, 7. Liu, I. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks.
In CVPR. pages 4700 2017.
[22] T. Huang. S. You. Y. Yang. Z. Tu, F. Wang, C. Qian, and C. Zhang. Explicitly learning topology for diffcrentiable neural architecture search. arXiv preprint 20Xiv:20/1.09.00 2020.
[23] X. S. Huang. F. Perez, J. Ba, and M. Volkovs, Improving transformer optimization through better initialization. In /CML. pages 4475 4483. PMLR, 2020.
[24] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICMI., pages 448-456. PMLR, 2015.
[25] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural
networks. NeurlPS, volume 31, 2018.
[26] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. In ICLR, 2015.
[27] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. In
ICLR, 2019.
[29] H. Li, Y. Yang, D. Chen, and Z. Lin. Optimization algorithm inspired deep neural network structure design.
In ACML, 2018.
[30] Y. Li and Y. Yuan. Convergence analysis of two-layer neural networks with relu activation. In NeurlPS,
volume 30, 2017.
[31] L. Liu, H. Jiang. P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2020.
[32] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[33]
Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[34] J. Mellor, J. Turner, A. Storkey, and E. J. Crowley. Neural architecture search without training. In ICML,
pages 7588-7598. PMLR, 2021.
[35] N. Mculeau and M. Dorigo. Ant colony optimization and stochastic gradient descent. Artificial life, 8(2): 121, 2002. [36] D. Mishkin and J. Matas. All you need is a good init. arXiv preprint 10XX::5511.06422. 2015.
[37] S. Oymak and M. Soltamolkotabi. Overparameterized nonlinear learning: Gradient descent takes the shortest path? In /CML, pages 1951-4960. PMLR, 2019.
[38] E. Real, A. Aggarwal, Y. Huang, and Q V. Le. Regularized evolution for image classifier architecture search. In AAAI, volume 33, pages 4780-4788 2019.
[39] S. Ruder. An averview of gradient descent optimization algorithms. arXiv preprint arXiv: 1609.04747, 2016.
[40] A. M. Saxe, L. McClelland, and S. Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
[41] Y. Shu, S. Cai, Z. Dai, B. C. Ooi, and B. K. H. Low. Nasi: Label-and data-agnostic neural architecture search at initialization. In ICIR, 2022.
[42] J. B. Simon, M. Dickens, and M. R. DeWeese. Neural tangent kernel cigenvalues accurately predict 2021.
generalization. arXiv preprint arXiv:2110.0342
[43] M. Soltanolkotabi, Learning relus via gradient descent. In NearlPS, volume 30, 2017.
[44] H. Tanaka, D. Kunin, D. L. Yamins, and Ganguli. Pruning neural networks without any data by itcratively conserving synaptic flow. In NewrIPS, volume 33, pages 6377-6389, 2020.
[45] L. Theis, I. Korshunova, A. Tejani, and F. Huszár. Faster gaze prediction with dense networks and fisher
pruning. arXiv preprint arXiv: 100:11100.07777 2018
[46] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML,, pages 10347-10357. PMLR, 2021.
[47] J. Turner, E. J. Crowley, M. O' O'Boyle, A. Storkey, and G. Cray. Blockswap: Fisher-guided block substitution for network compression on budget. In ICLR, 2020.
[48] J. Turner, F. J. Crowley. and M. F. O' O'Boyle. Neural architecture search as program transformation exploration. In Proceedings of the 26th ACM International Conference Architectural Support for Programming Languages and Operating Systems, pages 915-927, 2021.
[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, In NeurlPS, pages 5998 6008, 2017.
11

---

[50] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020. [51] L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. In ICML. pages 10462-10472. PMLR. 2020.
[52] R. Xiong, Y. Yang, D. He, K. Zheng. S. Zheng. C. Xing, Zhang. Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In JCML, pages 10524-10533, PMLR, 2020.
[53] Y. Xu, Q. Zhang. J. Zhang. and D. Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive
bias. In NeurlPS, volume 34, 2021.
[54] Y. Yang. H. Li, S. You, F. Wang. C. Qian, and Z. Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding. In NeurlPS, 2020.
[55] Y. Yang. S. You, H. Li, F. Wang, C. Qian, and Z. Lin. Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. In CVPR, 2021.
[56] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint /60x:/600.07144: 2016.
[57] H. Zhang. M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In ICIR, 2018.
[58] H. Zhang, Y. N. Dauphin, and T. Ma. Residual learning without normalization via better initialization. In ICI,R, 2019.
[59] 7. 7hang and 7. Jia. Gradsign: Model performance inference with theoretical insights. In ICI.R, 2022.
[60] C. Zhu, R. Ni, Z. Xu, K. Kong. W. R. Huang, and T. Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training. In NeurIPS, volume 34, 2021.
[61] Zhuang, T. Tang, Y. Ding, C. Tatikonda, N. Dvornek X. Papademetris, and J. Duncan, Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In NewrIPS, volume 33, pages 18795- 18806, 2020.
[62] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the last three paragraphs in Section 1.
(b) Did you describe the limitations of your work? [Yes] See Section 6.3.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.3.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We are ensured.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table 1 and 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Table 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the corre- sponding papers.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
12

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "a8HrRb-465T", "reviewer": "Reviewer_Jy3Y", "summary": "Authors study the problem of finding (learning) the best neural network initialization, and propose GradCosine - a measure of fitness for neural network initialization, based the similarity between individual sample gradients. This measure can then be optimized using and iterative procedure denoted as NIO (neural initialization optimization). Authors explain how GradCosine relates to model training and show a the relation between this quantity and the density of sample-wise local optima ([52] in the paper).\nFrom a theoretical point of view, the paper demonstrates that minimizing GradCosine results in favourable bounds on the training loss. For practical applications, authors offer a sub-batch version of GradCosine that can be computed more efficiently.", "questions": "Note: the question / suggestion below bears little significance. If authors are constrained by response time, please ignore it.\n\nTo the best of my understanding, even batched GradCosine currently requires more GPU memory than the objectives of MetaInit or GradInit (per model parameter). This could be a limitation when applying NIO for large models, e.g. transformer language models.\nI wonder if it is possible to reduce the memory usage algorithmically.\n\nSuppose that most of that memory is contributed by gradients w.r.t. weight matrices in linear layers, whether in convolutions or transformer projections.\n\nFor linear layers, one can observe that __one-sample gradients w.r.t. weight matrices are low-rank__.\nIn the simplest case, an MLP linear layer with batch size 1 will always receive rank-1 gradient - due to the fact that the gradient w.r.t. weight matrix is a product of two matrices of shape (1 x in_features) and (out_features x 1), where both \"1\"s correspond to batch size.\nFor convolution and attention layers, this will similarly result in low-rank gradients.\n\nThis trick is described in more detail in [1], though i believe that it was invented prior to that work.\n\nPut simply, if your gradient w.r.t. weight matrix was computed with batch size 1, you can compute the pairwise products for GradCosine without computing the gradients w.r.t. weight, using only the gradients w.r.t. activations. Furthermore, you could avoid storing activations for all layers by re-computing them on the fly[2].\n\n\n[1] https://arxiv.org/abs/2110.11309\n\n[2] https://arxiv.org/abs/1604.06174v2\n\n\n", "limitations": "To the best of my knowledge, authors have sufficiently addressed the limitations of their work.\nAs for the societal impact, this specific paper contains fundamental research in deep learning, thus it is hard to foresee its societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To the best of my understanding, the theoretical analysis appears sound, based on reasonable assumptions. [That said, I am no expert in the area of learned initialization.] The main experiments look reasonable, though I would recommend making some extra comparisons.\n\n1. in Table 5, you evaluate how NIO trains SWIN transformer without warmup. At least one baseline (GradInit[53]) also claims to work in that setting. Perhaps it would be best to compare NIO against that.\n\n2. Table 4 does not report standard deviations, while all prior experiments do. It might be useful to include them - or explain why they are missing.\n\n\nOn an unrelated note, I must applaud authors for supplying a Dockerfile in their supplementary code. Publishing the corresponding docker container will make it easier for future researchers to reproduce this work and build on it, even if the required libraries break compatibilities.\n\n\n### Typos / nitpicking\n\n> L215  making the sub-batches overlapped with each other **stables** the optimization\n\nperhaps a typo? \"stables\" -> \"stabilizes\"\n\n\n> L250 Each model is trained **for** four times with different seeds. \n\nconsider removing \"for\"\n\n\n> NIO is able to produce a better initialization that benefits model performance agnostic of architecture and dataset.\n\n[nit] this is concluded at an early section, where the only evidence is training cnn-only models on CIFAR-10/100. As a weak suggestion, I would recommend making this conclusion later, once you demonstrate NIO performance for transformers and ImageNet.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "URG-X4GXfRX", "reviewer": "Reviewer_u98c", "summary": "This paper works on the initialization of neural network parameter based on theoretically inspired optimization algorithm. It is to substitute the previous network initialization algorithm, such as Kaiming's method, [52], etc. The proposed approach is closely related to [52] by introducing the cosine similarity of sample-wise gradient for optimizing the initial network parameter. The proposed approach is applied to Resnet, DenseNet, WideResNet, and transformer. The results show that the proposed initialization method achieved better final network training results than the Kaiming's method, and the other compared learning-based methods.  ", "questions": "Please see my questions on the novelty, experimental justifications, etc., in the weakness. ", "limitations": "The paper clearly state the limitations of this approach. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper theoretically dig into the network initialization based on the difference of network parameter initialization and sample-wise local minima, and derive a quantity to approximate the upper-bound of training and test error bounds. The paper further approximate and minimize  the bound with constraint on gradient magnitude.  The proposed method is reasonble and inspired by theoretical analysis.\n\n2. The proposed method is evaluated on network training and show improved results with the proposed network parameter initialization method.\n\nWeakness:\n\nMy major questions and concerns are on the novelty over [52] and insufficient evaluation details. \n\n1. The  theoretical analysis of this work is similar to the analysis of [52], though the conclusion of this paper is different measuring by the cosine similarity of sample-wise gradients. \n\n2. The proposed approach is subject to an optimization problem of (6) adaptive to mini-batch based implementation. For feasibility in implementation, it makes some assumption and introduced the hyper-parameters, e.g., \\lambda. How is the performence affected by the hyper-parameters of the optimization algorithm, including, e.g., \\lambda, and number of iterations, etc ?\n\n3.   Due to the randomness, e.g., the mini-batch, the performance should be reported by running a same task in several times, and the mean the variance of the performance should be more insightful to compare the different methods. \n\n4. Because of the introduced additional computational cost in optimization, the approach is reported by running 100 iterations. How about the comparisons (speed and performance) if all the compared methods are aligned in number of running iterations for initialization?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Txm4DVqI58A", "reviewer": "Reviewer_JX6Q", "summary": "The paper first introduces a quantity, the cosine similarity of sample-wise local optima to evaluate the model performance at the initialization. They theoretically proved that their proposed quantity is the upper bound of both the training and generalization error under certain assumptions. Based on this theoretical finding, they approximate the sample-wise optimum with the first-order approximation to make the quantity differentiable and tractable. As a result, they simplify the upper bound quantity and achieve the initialization by maximizing the quantity with the gradient-based method. Their empirical results show that they can achieve better performance on various datasets and network structures compared to other initialization methods.", "questions": "1. Can we have some experiments to show that the proposed cosine similarity of sample-wise local optima is more suitable for evaluating the initialization quality?\n\n2. Does the initialization framework work for multiple optimization methods? For example, like sharpness-aware-minimization?\n\n3. It seems the method is dataset-dependent, will we have the same benefits if we initialize the parameter of the networks with a different dataset? For example, training on cifar-10 but initialized with cifar100 or ImageNet?\n\n4. Does this only work for initialization? Can we repeatedly use it during the training and will that be better?\n", "limitations": "The authors fairly addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well organized and easy to follow. The theoretical analysis seems correct and motivated. Their empirical results are quite good, especially on CIFAR datasets. \n\nWeaknesses:\nIt will be better if we can have some experiments to show that the proposed cosine similarity of sample-wise local optima is useful or better than the previous methods. Since the initialization method is motivated by this quantity, it’s better to make the usefulness of this quantity clear. \n\nThe citation in Appendix A for the proof of Lemma 1 seems wrong. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I_Zls_qdBc6", "reviewer": "Reviewer_ugxF", "summary": "This work proposes a new initialization method for neural network. The authors first use Fig. 1 to illustrate drawbacks of the sample-wise local optima density $\\Phi_{S,l}$ that adopts Manhattan distance between the pair-wise local optima, and lead to a Cosine similarity of sample-wise local optima $\\Theta_{S,l}$. Then the authors introduce to approximate local optimum by one step gradient descent and approximate $\\Theta_{S,l}$ by Eq. 4 (GradCosine). Finally, the initialized weights $\\theta_0$ are obtained by maximizing GradCosine and GradNorm. Extensive experiments are conducted to verify the efficacy of this approach, showing that GradCosine surpasses MetaInit and GradInit on multiple datasets in both accuracy and speed.", "questions": "1. The authors claim that (Line 114-115): The optimization path from the initialization to the local optima of the two samples in Fig. 1(d) are more consistent compared to that in Fig. 1(c). However, I notice that $\\theta_0$ in Fig. 1(c) is closer to $\\theta_1^*, \\theta_2^*$ than $\\theta_0$ in Fig.1 (d), resulting in a larger angle $\\angle(\\theta_1^*-\\theta_0, \\theta_2^*-\\theta_0)$. My point is that the angle cannot reflect the quality of optimization path since an initial point that is closer to the optimal point might as a larger angle.\n2. I have one concern on the approximation in Eq. 3. For initialized weights $\\theta_0$, such an approximation seems not reasonable. I hope the authors could discuss the rationality of it.\n", "limitations": "The limitations are addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The writing is logical and fluent.\n2) The experimental results are good.\n3) The motivation is interesting. Nevertheless, I have one question about it, please see the first point in Question.\n\nWeakness (Question):\n1) In Line 28, the authors claim that “these methods merely use the first order training dynamic as the main optimization objective”. However, MetaInit[8] and GradInit[53] both adopts gradient descent method.\n2) It seems that the theoretical analysis in Sec. 3.2 cannot show that $\\Theta_{S,l}$ has a tighter bound than $\\Phi_{S,l}$. Then I think Sec. 3.2 is a little redundant, which can be put into the supplementary.\n3) Typo: ‘GradCoisne’ in Line 6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept