Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
Authors: Xin Yuan
Pedro Henrique Pamplona Savarese
Michael Maire
Venue: NeurIPS 2023 poster (2023)
Keywords: network growing
efficient network training

Abstract: We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics.  Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network.  To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents.  Experiments show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original training computation budget.  We demonstrate that these gains translate into real wall-clock training speedups.

Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
Xin Yuan University of Chicago yuanx@uchicago.. edu
Pedro Savarese
Michael Maire University of Chicago mmaire@uchicago edu
TTI-Chicago savarese@ttic.ed
Abstract
We develop an approach to efficiently grow neural networks, within which param- eterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics utilize auxiliary gradient- basedd local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functional- ity of the network. To address the optimization difficulty resulting from imbalanced training effort distributed subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribu- tion of these separate subcomponents. Experiments show that our method achieves comparable or better accuracy than training large fixed-size models, while saving substantial portion the original training computation budget. We demonstrate that these gains translate into real wall-clock training speedups.
Introduction
Modern neural network design typically follows "larger better" rule of thumb, with models consisting of millions of parameters achieving impressive generalization performance across many tasks, including image classification [22, 32, 30, 46), object detection [13, 26, 11]. semantic segmen- tation [27, 3, 24] and machine translation [34, 7]. Within class of network architecture, deeper or wider variants of base model typically yield further improvements accuracy. Residual networks (ResNets) [15] and wide residual networks [45] illustrate this trend in convolutional neural network (CNN) architectures. Dramatically scaling up network size into the billions of parameter regime has recently revolutionized transformer-hasee language modeling [34, 7, 1].
The size of these models imposes prohibitive training costs and motivates techniques that offer cheaper alternatives to select and deploy networks. For example, hyperparameter tuning is notoriously expensive as it commonly relies on training the network multiple times, and recent techniques aim circumvent this by making hyperparameters transferable between models of different sizes, allowing them to tuned on small network prior training an original large model once [41].
Our approach incorporates these ideas, but extends the scope of transferability to include the pa- rameters of the model itself. Rather than view training small and large models separate events, we grow a small model into a large one through many intermediate steps, each of which introduces additional parameters to the network. Our contribution is do in manner that preserves the function computed by the model at each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training large static model from scratch.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Spiltt Neuroro
Add Nouzors
Feromettraation
Noc
Byy: did
Split J Add
Y/2
7/2
p.y:
aay Trander
17/2:
Spilt
Cucy Snall Optiniationn
Moc
0
1/1
Nlo
Optisbttinn
8
+
(a) Existing Methods: Splitting Init, Global LR
(b) Ours: Function- Preeiiving Init, Stagewise LR
Figure I: Dynamic network growth strategies, Different from (a) which rely on either splitting [4, 25, 39] or adding neurons with auxiliary local optimization [38, 10), our initialization (b) of new neurons is random but function paaetioonpreservingg Additionally, our separate learning rate scheduler governs weight updating to address the discrepancy in total accumulated training between different growth stages.
Competing recent efforts to grow deep models from simple architectures [4, 23, 25, 39, 37, 38, 44, 10J draw inspiration from other sources, such as the progressive development processes of biological brains. In particular, Net2Net [4] grows the network by randomly splitting learned neurons from previous phases. This replication scheme, shown in Figure (a) is a common paradigm for most existing methods. Gradient craddeet-bbsed methods [38, 39] determine which neurons to split and how to split them by solving a combinatorial optimization problem with auxiliary variables.
At each growth step, naive random initialization of new weights destroys network functionality and may overwhelm any training progress. Weight rescaling with a static constant from a previous step not guaranteed be maintained as the network architecture evolves. Gradient-based methods outperform these simple heuristics but require additional training effort in their parameterization schemes. Furthermore, all existing methods use global LR scheduler to govern weight updates, ignoring the discrepancy among subnetworks introduced in different growth phases. The gradient itself and other parameterization choices may influence the correct design for scaling weight updates.
We develop growing framework around the principles of enforcing transferability of parameter settings from smaller to larger models (extending [4]]). offering functional continuity, smoothing opti- mization dynamics, and rebalancing learning rates between older and newer subnetworks. Figure 1(b) illustrates key differences with prior work. Our core contributions
We propose a parameterization scheme accounting for the variance transition among networks of smaller and larger width a single training process. Initialization of new weights is gradient-free and requires neither additional memory nor training. Improved Optimization with Learning Rate Adaptation: Subnetworks trained for different lengths have distinct learning rate schedules, with dynamic relative scaling driven by weight norm statistics.
Parameterization using Variance Transfer:
Better Performance and Broad Applicability: Our method not only trains networks fast, but also yields excellent generalization accuracy, even outperforming the original fixed-size models. Flexibility in designing a network growth curve allows choosing different trade-offs between training resources and accuracy. Furthermore, adopting acceleration in terms of wall-clock training time. We demonstrate results on image classification and machine translation tasks, across various network architectures.
adaptive batch size schedule provides
2 Related Work
Network Growing. diverse range of techniques train models by progressively expanding the network architecture [36, 9, 5, 37, 44]. Within this space, the methods of [4, 25, 39, 38, 10] are most relevant to our focus incrementally growing network width across multiple training stages. Net2Net [4] proposes a gradient- feeeientt-feeee neuron splitting scheme via replication, enabling knowledge transfer from previous training phases; initialization of new weights follows simple heuristics. Liu al.'s 's splitting approach [25] derives a gradient-
scheme for duplicating neurons by formulating a combinatorial optimization problem. FireFly [38] gains flexibility by also incorporating brand new neurons. Both methods improve Ner2Net's 's initialization scheme by solving optimization problem with auxiliary variables, at the cost of extra training effort. GradMax [10], consideration training dynamics, performs initialization via solving singular value decomposition (SVD) problem.

---

Neural Architecture Search (NAS) and Pruning, Another subset of methods mix growth with dynamic reconfiguration aimed at discovering or pruning task-optimized architectures. Network Morphism [36] searches for efficient networks by extending layers while preserving the parameters. AutoGrow [37] takes an AutoMI. approach governed by heuristic growing and stopping policies. Yuan et al. [44] combine learned pruning with sampling strategy that dynamically increases or decreases network size. Unlike these methods, we focus on the mechanics of growth when the target architecture is known, addressing the question of how to best transition weight and optimizer state to continue training an incrementally larger model. NAS and pruning are orthogonal to, though potentially compatible with, the technical approach we develop.
Hyperparameter Transfer. Multiple works [42, 29, 16] explore transferable hyperparameter (HP) tuning. The recent Tensor Program (TP) work of [40] and [41] focuses on zero-shot HP transfer across model scale and establishes principled network parameterization scheme to facilitate HP transfer. This serves required to account for dynamic growth.
anchor for our strategy, though, as Section 3 details, modifications are
Learning Rate Adaptation. Surprisingly, the existing spectrum of network growing techniques utilize relatively standard learning rate schedules and do not address potential discrepancy among subcomponents added at different phases. While general-purpos adaptive optimizers, e.g. Ada- Grad [8], RMSProp [33], Adam [20], or AvaGrad [31], might ameliorate this issue, we choose to explicitly account for the discrepancy. As layer-adaptive learning rates (LARS) [12, 43] benefit in some contexts, we explore further learning rate adaption specific both layer and growth stage.
3 Method
3.1 Parameterization and Optimization with Growing Dynamies
Functionality Preservation. We grow network's 's capacity by expanding the width of computational units (e.g., hidden dimensions in linear layers, filters in convolutional layers). To illustrate our scheme, consider a 3-layer fully-connected network with ReLU activations & at a growing stage l:
(I) where E RC is the network input, Yt RC is the output, and u, E C h, E RC are the hidden activations. In this case, W.r is a CT x C matrix, while W is C. x C7:² and W is C Cp. Our growing process operates by increasing the dimensionality of each hidden state, iee. from CF and C C²_ and Ch effectively expanding the size of the parameter tensors for the next growing stage ++ 1. The layer parameter matrices W, have their shapes changed accordingly and become W1+.. Figure 2 illustrates the process for initializing Wi+1 from W, at a growing step.
U Φ0 xW
h; = =((W"u.)
yr W hr,
Following Figure 2(a), we first expand W along the output dimension by adding two copies of new weights V," of shape Chu C generating new features oi (V,*)). The first of activations become
(2) where concat denotes the concatenation operation. Next, we expand W.'' across both input and
U++1 concat (u,, (VV" x). <(V"x))
output dimensions, as shown in Figure 2(b). We initialize new weights Zí of shape C.2 C -h and add to W.' two copies of it with different signs: + +2, and Z," This preserves the output of the layer since o( 6W, uu + 2 "V'" x) + (-Z "Z (VV(')()))))
& (W)" he We then add two -2. x C yielding activations copies of new weights V,", which has shape 2
h++1 = concat(h,, (VV. (V,*u++1,, o( v())).
(3)
We similarity expand Wh with new weights Z match the dimension of h441, as shown in Figure 2(c). The network's 's output after the growing step is:
yt+1 = W h4 + Z (0
(V," u + (- )oV,
h = ye
which preserves the original output features in Eq. 1. Appendix provides illustrations for more layers.
We defer the transformation between W; and W to the next subsection. It involves rescaling by constant factors, does not affect network functionality, and is omitted in Eq. 1- 4 for simplicity.
3

---

C
C
W W?
W
Ct
W7
0+1
V (0011/0*)
V
V
(a) Input Layer
C411
C
C
W."
0
W
W
2, 44
C
W."
1+1
2 ~ 1/C,1
V22
Vi* ((0,1//1))).)
V."
(b) Hidden Layer
C
CA
CA WM
W
C
W
W
C
28 NO 1/00,13
(c) Output Layer
Figure 2: Initialization scheme. In practice, we also add noise to the expanded parameter sets for symmetry breaking.
Weights Initialization with Variance Transfer (VT). Yang el [41] investigate weight scaling with width at initialization, allowing hyperparameter transfer by calibrating variance across model size. They modify the variance of output layer weights from the commonly used I to 1 We Fance
Fan?
adopt this same correction for the added weights with new width: W and 2h are initialized with variances of and
However, this correction considers training differently-sizel models separately. which fails to accom- modate the training dynamics in which width grows incrementally. Assuming that the weights of the old subnetwork follow h ~ NO,,, cxa (which holds at initialization), we make them compatible
with weight tensor parameterization by rescaling it with the fan,n ratio W.M. W.a C C
See Table (top). Appendix A provides detailed analysis.
This parameterization rule transfers modern CNNs with batch normalization (BN), Given weight scaling ratio of c, the running mean and variance a of BN layers are modified as cp and c²a, respectively.
Stage-wise Learning Rate Adaptation (LRA). Following [41], we employ a learning rate scaling factor of x 1 fan:.. the output layer when using SGD, compensating for the initialization scheme. However, subnetworks from different growth stages still share global learning rate, though they have trained for different lengths. This may cause divergent behavior among the corresponding weights, making the training iterations after growing sensitive to the scale of the newly-initialized weights. Instead of adjusting newly added parameters via local optimization [38, 10], we govern the update of each subnetwork in stage-wise manner.
Let W, denote the parameter variables of layer at a growth stage I, where we W, and W/ correspond to the same set of variables such that W W, denotes the new parameter variables whose values are initialized with Z, and V. Moreover, let Wa and Gat denote the values and gradients of W. \ Wp-1. We adapt the learning rate used to update each sub-weight W. for

---

Table I: Parameterization and optimization transition for different layers during growing. C₁ and Ce+1 denote the input dimension before and after growth step.
Input Layer
Hidden Layer
Output Layer
Old Re-scaling
/C//C4 1/C141
C/ 1C/C 1/(C)/1)
Init.
New Init.
1/CT
Stage
1/Co I
Adapt.
f-th Stage
I Wa
I W Ao
W
S S I, as follows:
f(W Ak) Th = 1Jo f(Wao)
WAR + WAs - Thk Ga
(5)
where Y/ is the base learning rate, f is a function that maps subnetworks of different stages to corresponding train-time statistics, and WAo are the layer's parameter variables the first growth stage. Table (bottom) summarizes our LR adaptation rule for SGD when f is instantiated as weight norm, providing stage-wise extension the layer-wise adaptation method LARS [12], i.e. LR < Wl|. Alternative heuristics are possible; see Appendices C and D.
3.2 Flexible and Efficient Growth Scheduler
We train the model for Trotal epochs by ex- panding the channel number of each layer to C final across N growth phases. Existing meth- ods [25, 38] fail derive systemic way for distributing training resources across growth trajectory. Toward maximizing efficiency, we experiment with coupling between model size and training epoch allocation.
Growing using Var. Transfer and Learning Rate Adapt. with Flexible Scheduler
Algorithm 1
Input: Data X, labels Y, task loss T. Output: Grown model W
Initialize: Wo with Co, To, Bg, 100 do
for t 0 to
if t > 0 then
Init. S, from Sn using VT in Table 1. Update C, and T, using Eq. and Eq. 7.
Architectural Scheduler. We denote initial channel width Co and expand exponentially:
Update B, using Eq. 8 (optional)
Itertotal = T. * len(X)//BB
C. = C CC-11 lp,Cr.ll if t N<N-1 -1 Cfinnl
end if
(6)
if t=N-1
for iter =1 to Iter do
Forward and calculate I.(((
where L-12 rounds to the nearest even number and pe the growth rate between stages.
Back propagation with 1.
Update each sub-component using Eq. 5.
end for
Epoch Scheduler. We denote number of epochs assigned to t-th training stage Tr, with IN- Tiotal. We similarly adapt T; via an exponential growing scheduler:
end for return WN-1
T + 1p:Ti-11 if << N -1 T.
- (7)
(Trtall - - if
Wall-clock Speedup via Batch Size Adaptation. Though the smaller architectures in early growth stages require fewer FLOPs, hardware capabilities may still restrict practical gains. When growing width, in order to ensure that small models fully utilize the benefits of GPU parallelism, adapt the batch size along with the exponentially-grovee architecture in a reverse order:
Bouse
if t N if t t<N
B
Br
where Brase is the batch size of the large baseline model. Algorithm summarizes our full method.
5

---

4 Experiments
We evaluate image classification and machine translation tasks. For image classification,
CIFAR-10 [21], CIFAR- [21] and ImageNet [6]. For the neural machine translation, we use the IWSLT' 14 dataset [2] and report the BLEU [28) score German English (De-En) translation.
Large Baselines via Fixed-size Training. We use VGG-11 [32] with BatchNorm [19], ResNet- 20 [15], MobileNetVI [17] for CIFAR-10 and VGG-19 with BatchNorm, ResNet-18, Mo- bileNetV1 for CIFAR-I00. We follow [18] for data augmentation and processing, adopting random shifts/mirroring and channel-wise normalization. CIFAR-10 10 and CIFAR-100 models are trained for 160 and 200 epochs respectively. with a batch size of 128 and initial learning rate (LR) of 0.1 using SGD. We adopt a cosine LR schedule and set the weights decay and momentum 5e-4 and 0.9. For ImageNet, we train the baseline ResNet-50 and MobileNetV1 [17] using SGD with batch sizes of 256 and 512, respectively. We adopt the same data augmentation scheme as [14], the cosine LR scheduler with initial LR of 0.1, weight decay of le-4 and momentum of 0.9.
For IWSLT 14, we train an Encoder-Decoder Transformer (6 attention blocks each) [34]. We set width as dmmoddl = 1/ f/ = 512, the number of heads Thead = 8 and dy = dg = dy = dmmodee/ Theed = 64. We train the model using Adam for 20 epochs with learning rate 1e-3 and (3) 32) (0.9, 0.98). Batch size is 1500 and we use 4000 warm-up iterations.
Table 2: Growing ResNet-20, VGG- 11, and MobileNetV1 CIFAR-10. 10.
ResNet-20
VGG-11
MobileNetvl
Train Cost(%) 1 Accuracy(%)) f Cost( ())
Test
Train
Train Accuracy (%) t Cost(%)
Test
Test
Method
| Accuracy (%)
Large Baseline
100
92.62
0.15
100
92.14 0
100
92.27 0.11
Net2Net
54.90
91.60
0.21
52.91
91.78
53.80
90.34 + 0.20
Splitting
70.69
91.80 + 0.10
63.76
01.88
65.92
91.50
FireFly-split
58.47
$1.7840.1
56.18
91.11
56.37
91.56 0.06
FireFily
68.96
92.10
60.24
62.12
91.69 0.07
Ours
54.90
92.53 0.11
52.91
92.34 0.15
53.80
92.01 + 0.10
Table 3: Growing R8sNet-18, VGG-19, and MobileNetVi CIFAR-100.
ResNet-18 18
VGG-19
MobileNetvl
Train Cost(%) 4 Accuracy (%) f Cost(%) (%)
Test
Train
Test A.cuurcc(%% t Cost(%) 4 Accuracy (%)
Train
Test
Method
Large Baseline
100
78.36 + 0.12
100
72.59 = 0.23
100
72.13 + 0.13
Net2Net
52.63
76.48
0.20
52.08
71.88 = 0.24
52.90
70.01
+ 0.20
Splitting
68.01
77.01 + 0.12
60.12
71.96 - 0.12
58.39
70.45 + 0.10
FireFly-split
56.11
77.22
0.11
54.64
72.19
+ 0.14
54.36
70.69 0.11
FireFly
65.77
77.25
+ 0.12
57.48
72.79
0.13
56.49
70.99 + 0.10
Ours
52.63
78.12 0.15
52.08
73.26
+ 0.14
52.90
71.53 + 0.13
Implementation Details. We compare with the growing methods Net2Net [4], Splitting [25], FireFly- split, FireFly [38] and GradMax [10]. In our method, noise for symmetry breaking is 0.001 to the norm of the initialization. We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers (e.g., Adam, AvaGrad).
For image classification, we run the comparison methods except GradMax alongside our algorithm for all architectures under the same growing scheduler. For the architecture scheduler, we set Pe as 0.2 and Co as 1/4 of large baselines in Eq. 6 for all layers and grow the seed architecture within N = 9 stages towards the large ones. For epoch scheduler, we set Pt. as 0.2, To as 8, 10, and 4 in Eq. 7 on CIAFR-10. 10, CIFAR- 100, and ImageNet respectively. Total training epochs Tiotat are the same as the respective large fixed-size models. We train the models and report the results averaging over 3 random seeds.

---

For machine translation, we grow the encoder and decoder layers' widths while fixing the embedding layer dimension for a consistent positional encoding table. The total number of growing stages 4, each trained for epochs. The initial width is 1/8 of the large baseline (i.e. demodel = 64 and d...gz = 8). We set the growing ratio Pe as 1 so that dmnode evolves as 64, 128, 256 and 512.
We train all the models on an NVIDIA 2080Ti 1IGB GPU for CIFAR-IO, CIFAR-I00, and IWSLT" 14, and two NVIDIA A40 48GB GPUs for ImageNet.
4.1 CIFAR Results
All models grow from a small seed architecture to the full-sized one in stages. each trained for 98,9. 11, 13, 16, 19, 23, 28, 33} epochs (160 total) on CIFAR-10, and {10, 12, 14, 17, 20, 24, 29, 35, 39} (200 total) on CIFAR-100. Net2Net follows the design of growing by splitting via simple neuron replication, hence achieving the same training efficiency our gradient- free method under the same growing schedule. Splitting and Firely require additional training effort for their neuron selection schemes and allocate extra GPU memory for auxiliary variables during the local optimization stage. This is computationally expensive, especially when growing large model.
ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10. Table 2 shows results in terms of test accuracy and training cost caleulated based on overall FLOPs. For ResNet-20,, Splitting and Firefly achieve better test accuracy than Net2Net, which suggests the additional optimization benefits neuron selection at the cost of training efficiency. Our method requires only 54.9% of the baseline training cost and outperforms all competing methods, while yielding only 0.09p.p (percentage points) performance degradation compared to the static baseline. Moreover, our method even outperforms the large fixed-size VGG-11 by 0.20p.p test accuracy, while taking only 52.91 % of its training cost. For MobileNetV1, 1, our method also achieves the best trade-off between training efficiency and test accuracy among all competitors.
ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100. We also evaluate all methods on CIFAR- 100 using different network architectures. Results in Table 3 show that Firely consistently achieves better test accuracy than Firefly-split, suggesting that adding new neurons provides more flexibility for exploration than merely splitting, Both Firely and our method achieve better performance than the original VGG-19, suggesting that network growing might have an additional regularizing effect. Our method yields the best accuracy and largest training cost reduction.
Table 4: ResNet-50 and MobileNetV1 on ImageNet.
ResNet-50
MobileNet-vl
Train Cost 4
Test A%c
Train Cost %))
Test (%)
Method
Large
100
76.72
0.18
100
70.80 + 0.19
Ner2Net
60.12
74.89
0.21
63.72
66.19 + 0.20
FireFly
71.20
75.01 + 0.11
86.67
66.40 + 0.14
GradMax
86.67
68.60 = 0.20
Ours
60.12
75.90 + 0.14
63.72
69.91 + 0.16
Table 5: Transformer on IWSLT'14 14.
Transformer
Method
Train Cost(%) 4
BLEU
Large
100
32.82 + 0.21
Net2Net
64.64
30.97 0.35
Ours-w/o buffer
64.64
31.44
Ours-w buffer
64.64
31.62 + 0.16
Ours-w buffer-RA
64.64
32.01 + 0.16
7

---

4.2
ImageNet Results
We first grow ResNet-50 on ImageNet and compare the performance of our method to Net2Net and FireFly under the same epoch schedule: (4, 5, 6, 8, 9, 11, 14, 29) (90 total) with 9 growing phases. We also grow MobileNetV1 from small seed architecture, which is more challenging than ResNet-50. We train Net2Net and our method uses the same scheduler as for ResNet-50. We also compare with Firefly-Opt (a variant of FireFly) and GradMax and report their best results from [10]. Note that both methods not only adopt additional local optimization but also train with a less efficient growing scheduler: the final full-sized architecture needs to be trained for a 75% fraction while ours only requires 32. 32. Table 4 shows that our method dominates all competing approaches.
4.3 IWSLT14 De-En Results
We grow a Transformer from dinodet epochs using Adam. Applying gradient-based growing methods to the Transformer architecture is nontrivial due to their domain specific design of local optimization. As such, we only compare with Net2Net. We also design variants of our method for self-comparison, based the adaptation rules for Adam Appendix C. As shown in Table 5, our method generalizes well to the Transformer architecture.
dmoddl 512 within stages, each trained with
4.4 Analysis
Ablation Study. We show the effects of turning on/off each of our modifications to the baseline optimization process of Net2Net (1) Growing: adding neurons with functionality preservation. (2) Growing+VT. VT: only applies variance transfer. (3) Growing+RA: only applies LR rate adaptation. (4) Full method. We conduct experiments using both ResNet-20 CIFAR-10 and ResNet- 18 CIFAR- 100. As shown in Table 6, different variants of our growing method not only outperform Net2Net consistently but also reduce the randomness (std. over 3 runs) caused by random replication. We also see that, both RA and VT boost the baseline growing method. Both components are designed and woven accomplish the empirical leap. Our full method bests the accuracy.
Variant
Res-20 on C-10 (%)
Res- 18 on C-100 (%)
Net2Net
91.60 0.21
76.48 + 0.20(+0.00)
Growing
91.62
76.82 + 0.17
Growing+ VT
92.00 0.10(+0.40)
77.27 + 0.14(
Growing+RA
92.24 + 0.11(+064))
77.74 + 0.16(+1.22)
Full
92.53 + 0.11(+0.9)
78.12 + 0.15(+1.64)
Table 6: Ablation study VT and RA components.
Frndart2
16
Large Tran Out
16
Growing Standord
wowhhgluure Tordfer-Sondd
(66)
1.4
55
Ttain Loss
08
66
Lorge STrondentxx Tran Cost
64
GrowingrOurs Tenster xssubtbrri
35
2-1
2-
2.
Learning Rate
Learning Rate
(a) Train Loss
(b) Test Accuracy
Figure 3: Baselines of 4-yer simple CNN.

---

31-4
Stnndrr mil 550 Trester+ Sandard
Tander+ SED 4dept.
7 Stage
8-th
Mi Epocrslast Stage!
Us
m
Eoochslast Teo
Epechslaast Two Stggest
(a) Test Accuracy
(b) Global LR
(c) Rate Adaptation
Figure 4: (a) Performance with Var. Transfer and Rate Adaptation growing ResNet-20. (b) and (c) visualize the gradients for different cub-components along training in the last two stages.
Justification for Variance Transfer. We train a simple neural network with 4 convolutional layers on CIFAR-10. The network consists resolution-preserving convolutional layers; each convolution has 64, 128, 256 and 512 channels, 3 x 3 kernel, and is followed by BatchNorm and ReLU activations. Max-pooling is applied to each layer for a cesolutin-ddoossappee
of 4, 2, 2, and 2. These CNN layers are then followed by linear layer for classification. We first alternate this network into four variants, given by combinations of training epochs E {13(1x), 30(2.3x)) }} and initialization methods E (standard, jutansfer [41]]. We also grow from a thin architecture within 3 stages, where the channel number of each layer starts with only 1/4 of the original one, i.e., {16, 32, 64, 128] - (32, 64, 128, 256} {64, 128, 256, 512), each stage is trained for 10 epochs.
For network growing, we compare the baselines with standard initialization and variance transfer. We train all baselines using SGD, with weight decay set as O and learning rates sweeping over {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 0.8, 1.0, 1.2, 1.5, 2.0). In Figure 3(b), growing with Var. Transfer (blue) achieves overall better test accuracy than standard initialization (orange). Large baselines with merely ptransfer in initialization consistently underperform standard ones, which validate that the compensation from the LR rescaling is necessary in [41]. We also observe, in both Figure (a) and 3(b), all growing baselines outperform fixed-size ones under the same training cost, demonstrating positive regularization effects. We also show the effect of our initialization scheme by comparing test performance on standard ResNet-22 on CIFAR- 10. As shown in Figure 4(a), compared with standard initialization, our variance transfer not only achieves better final test accuracy but also appears more stable. See Appendix F for a fully-connected network example.
Justification for Learning Rate Adaptation. We investigate the value of our proposed stage-wise learning rate adaptation as an optimizer for growing networks. As shown in the red curve in Figure 3, rate adaptation not only bests the train loss and test accuracy among all baselines, but also appears to be more robust over different learning rates. In Figure 4(a), rate adaptation further improves final test accuracy for ResNet-20 on CIFAR-10, under the same initialization scheme.
Figure 4(b) and 4(c) visualize the gradients of different sub-components for the 17-th convolutional layer of ResNet-20 during last two growing phases of standard SGD and rate adaptation, respectively. Our rate adaptation mechanism rebalances subcomponents' gradient contributions to appear in lower divergence than global LR, when components are added at different stages and trained for different durations. In Figure 5, we observe that the LR for newly added Subnet-8 (red) in last stage starts around 1 1.8x the base LR, then quick adapts to a smoother level. This demonstrates that our method is able to automatically adapt the updates applied to new weights, without any additional local optimization costs [39, 10]. All above show our method has positive effect in terms of stabilizing training dynamics, which is lost one attempts to train different subcomponents using a global LR scheduler. Appendix D provides more analysis.
Flexible Growing Scheduler. Our growing scheduler gains the flexibility to explore the best trade- offss between training budgets and test performance in a unified configuration scheme (Eq. 6 and Eq. 7). We compare the exponential epoch scheduler (p. E {0.2, 0.25, 0.3, 0.355) to a linear one (p. = 0) in ResNet-20 growing on CIFAR-10, 10, denoted as 'Exp.' and "Linear" baselines in Figure 6. Both baselines use the architectural schedulers with P. E (0.2, 0.25, 0.3, 0.35), each generates trade-offs between train costs and test accuracy by alternating To. The exponential scheduler yields better

---

13.00
Rate Adaptation fectore
3.5
02.55
Test Accuracy
2.
201 pi meadi
Burter
97.39
92.02
Epochs Stage)
Triin Cost (%6
Figure 5: Visualization of our adaptive LR.
Figure 6: Comparison growing schedules.
Model
Fix BatchSize
RatchSize
GPU Memory (M)
Train Time/Epach
300
500
400
200
Stages
Stages
(a) GPU memory allocations
(b) Training time
Figure 7: Track of GPU memory and wall clock training time for each growing phase of ResNet- 18.
overall trade-off's than the linear one with the same Pc. addition to different growing schedulers, we also plot a baseline for fixed-size training with different models. Growing methods with both schedulers consistently outperform the fixed-size baselines, demonstrating that the regularization effect of network growth benefits generalization performance.
Wall-clock Training Speedup. We benehmark GPU memory consumption and wall-clock training time on CIFAR-100 for each stage during training single NVIDIA 2080Ti GPU. The large baseline ResNet-18 18 trains for 140 minutes achieve 78.36% accuracy. As shown in the green bar of Figure 7bb), the growing method only achieves marginal wall-clock acceleration, under the same fixed batch size. As such, the growing ResNet-18 18 takes 120 minutes to achieve 78.12% accuracy. The low GPU utilization in the green bar in Figure 7(a) hinders FLOPs savings from translating into real-world training acceleration. In contrast, the red bar of Figure 7 shows our batch size adaptation results in large proportion of wall clock acceleration by filling the GPU memory, and corresponding parallel exccution resources, while maintaining test accuracy. ResNet-15 18 trains for minutes (1. speedup) and achieves 78.0 78.01% accuracy.
Conclusion
We tackle a set of optimization challenges network growing and invent a corresponding set of techniques, including initialization with functionality preservation, variance transfer and learning rate adaptation to address these challenges. Each of these techniques can be viewed as 'upgrading' original part for training static networks into corresponding one that accounts for dynamic growing. There is one-to-one mapping of these replacements and a guiding principle governing the formulation of each replacement. Together, they accelerate training without impairing model accuracy a result that uniquely separates our approach from competitors. Applications to widely- used architectures on image classification and machine translation tasks demonstrate that our method bests the accuracy of competitors while saving considerable training cost.
Acknowledgments and Disclosure of Funding
This work was supported by the National Science Foundation under grant CNS-195511 18 and the University of Chicago CERES Center.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berer, Sam McCandlish, Alec Radford, liya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[2] Mauro Cettolo, Jan Nichues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, 2014.
[3] Liang- Chich Chen, George Pupandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv: 05587.0557.200 2017.
Liang-Chieh
[4] Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2Net: Accelerating learning via knowledge transfer. In ICLR, 2016.
[5] Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. NeST: A neural network synthesis tool based on a grow-and-prune paradigm. IEEE Trans. Computers, 2019.
[6] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. CVPR. 2009.
[7] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidircctional transformers for language understanding. In NAACL, 2019.
[8] John C. Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In ICLR Workshop. 2018
[10] Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. GradMax: Growing neural networks using gradient information. In ICLR, 2022.
[11] Golnaz Ghiasi, Tsung-Yr Yi Lin, Ruoming Pang, and Quoe V. Lc. NAS-FPN: Learning scalable feature
pyramid architecture for object detection. 190.009004 2019.
[12] Boris Ginsburg, Igor Gitman, and Yang You. Large batch training of convolutional networks with layer-wise adaptive rate scaling. 2018.
[13] Ross B. Girshick. Fast R-CNN. In ICCV, 2015.
[14] Sam Gross and Michael Wilber. Training and investigating residual nets. http://torch.hh// 2016/02/04/resnett html, 2016.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016
[16] Samuel Horváth, Aaron Klein, Peter Richtárik, and Cédric Archambcau. Hyperparameter transfer learning with adaptive complexity. In AISTATS, 2021.
[17] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications.
1704.0470.4866 2017.
[18] Gao Huang, Yu Sun, Zhuang Liu, Danicl Sedra, and Kilian Weinherger. Deep networks with stochastic depth. In ECCV, 2016.
[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[21] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. http://www. toronto.
adh//Kriz/cifar html, 2014.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In NeurlPS, 2012.
[23] Xilai Li, Yingho Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. arXiv: 003100 2019.
[24] Chenxi Liu, Liang-Chien Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto- DeepLae: Hicrarchical neural architecture search for semantic image segmentation. In CVPR, 2019.
[25] Qiang Liu, Wu Lemeng, and Wang Dilin. Splitting steepest descent for growing neural architectures. In NewrIPS, 2019.
11

---

[26] Wei Liu, Dragomir Anguelov. Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[28]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. ACL, 2002.
[29] Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. In NeurIPS, 2018.
[30] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
[31] Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire, Domain- Somaiin-ndeepnneet dominance of adaptive methods. In CVPR, 2021.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. ICLR, 2015.
[33] Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[35] Chengeheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire. Orthogonalized SGD and nested architectures for anytime neural networks. In ICML, 2020.
[36] Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In ICML, 2016.
[37] Wei Wen, Feng Yan, Yiran Chen, and Hai Li. AutoGrow: Automatic layer growing in deep convolutional networks. In KDD, 2020.
[38] Lemeng Wa, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In NewrIPS, 2020.
[39] Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture optimiza- tion: Escaping local optimum with signed neural splitting. arXiv:2003. 10392, 2020.
[40] Greg Yang and Edward J. Hu. Tensor Programs IV: Feature learning in infinite-width neural networks. In ICML, 2021.
[41] Greg Yang. Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In NewrlPS, 2021.
[42] Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In AISTATS, 2014.
[43] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh, Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR, 2020.
[44] Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In ICLR, 2021.
[45] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv: 2016.
[46] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In
CVPR, 2022.
12

---

A More Analysis on Variance Transfer
Our fixed rescaling formulation, regarding relative network width, is extension to the principled zero-shot HP transfer method [40, 41], based the stability assumption, denoted VT. A dynamic rescaling based actual old weight values an alternative plausible implementation choice, denoted as VT-constraint.
Theorem A.I. Suppose the goal is enforce the unit variance feature, then the scaling factor an input layer with weights W with input shape C v coolway. while for a hidden layer with weights W and input
shape C is c"v(W"]]
Proof. Consider a hidden layer that computes u W x followed by another layer that computes h W u (ignoring activations for simplicity). At growth step, the first layer's outputs change from
u W7
(9)
to
U4+1 [ua' u." , " W. Vi V/ where 5" denotes the scaling factor applied to W The second layer's outputs change from h: W."
(10)
(11)
to
h++1 [he's he" h. h," = [s W u., V." uet1. V u(ut+
(12)
where s" denotes the scaling factor applicd W
The variance the features after the growth step arc:
Vuu'' = (8 0 W."
(13)
V/u." V[V.'"]]
(14)
= V
(15)
Vhh." VVV (Ci+1 - C. V.,""",)))
(16)
Given the goal of enforcing unit-variance features for across all four vectors, we get:
V CoV(W?]
V(u''== =1
(17)
v(h,1= =1
(18)
1 ~ = Viu "]
VIV." =
(19)
1
VVV
V(hi" =
(u. + (C C V(u, 'l) =1.
(20)
C
C++1
This differs from the default formulation Section 3.1, which corresponds to scaling factors s" and
We compare the default VT with VT-constraint by growing ResNet-20 on CIFAR- 10. As shown in Table 7, both VT and VT-constraint outperform the standard baseline. which suggests standard initialization is a suboptimal design in network growing. We also note that involving the weight statistics is not better than our simpler design, which suggests that enforcing the old and the new features to have the same variance is not a gond choice.
Table 7: Comparisons among standard initial- ization, VT-constraint (Theorem A.I) and the default VT (Section 3.1) for growing ResNet-20 on CIFAR-10.
Variants
Test Acc. (%)
Standard
91.62 0.12
VT-constraint
91.93 + 0.12
VT
92.00 + 0.10
13

---

[+1
O0
W
Growing CNN Hidden Layer
W
C
CA
+2
-2
(a) Growing CNN hidden layer
Residual Black
tt #1 x1x1
Reeiiual Black
Height
Withh Growth
Care
Come
of of aicc
of Ot
x W Ouey thaee
dt Irou Shepe
c Waee then
ot of acko Wiel thee
dt Rx
Whage thee
Weep thewe
Ouiput thaee
(b) Growing residual block
Figure 8: Illustration for growing other layers.
B General Network Growing in Other Layers
We have shown network growing for 3-layer fully connected networks as a motivating example in Section 3.1. We now show how to generalize network growing with K (>3) layers, with conv layers, residual connections.
Generalization to K (> 3) Layers. In network-width growing formulation. layers may be expanded in 3 patterns. (1) Input layer: output channels only; (2) Hidden layer: Both input (due expansion of the previous layer) and output channels. (3) Output layer: input channels only. As such, 3-layer case sufficient to serve as a motivating example without loss of generality. For K (> 3) layer networks, the 2nd to (K 1)th layers simply follow the hidden layer case defined in Section 3.1.
Generalization to Convolutional layers. In network width growth, we only need to consider the expansion along input and output channel dimensions matter for fully connected networks or CNNs. Equations 1-4 still hold for CNN layers. Specifically, when generalizing to CNNs (from 2-d Cout Cun weight matrices 4-d Cout x C x ls ones), we only consider the matrix operations on the first two dimensions since we not change the kernel size k. For example, in Figure 2(b), newly added weights + Z2 in linear layer can be indexed - :.
W[0: Oh,, O C + Gh1 C
In CNN layer, it is simply W C. CF C.
Residual Connections. We note that differently from plain CNNs like VGG, networks with residual connections do require that the dimension of activation and its residual match since these will be added together. Our method handles this by growing layers at same rate, Hence, at each growth step, the shape of the two tensors to be added by the skip connections always matches.
Concrete Example of Adding Units to a Network. In model width growth, our common practice is to first determine the output dimension expansion from the growth scheduler. If previous layer's output channels are be grown, then the input dimension of the current layer should be also expanded to accommodate that change.
Let's consider concrete example for CNN with more layers (> 3) and residual blocks. Without loss of generality, we omit the kernel size k and denote cach convolutional layer channel shape (C... Cout). We denote the input feature with 1 output dimension as æ. Initially, we have the input layer W' with channel dimensions of (1,2) to generate ba (2-dimensional), followed by a 2-layer residual block (identity mapping for residual connection) with weights (22) (2, 2) and (33) (2, 2), followed by an output layer with weights (4)
The corresponding computation graph (omitting BN and ReLU for simplification) is
y (4)) (w + w())).
In more detail, rewrite the computation in the matrix formulation:
14

---

wi (1)
hay /
(1)
w (2)
1
h:00 = w(w0
h (2)
221
2.222
(3)
wi
(3)
111
A(00 (09) +100
h1j
h
y (10
wi,1
(4)
(4) W1.
Now assume that we want to grow the dimension of the network's hidden activations from 2 to 4 (i.e. a (,2) (3) (4) which are 2-dimensional, should become 4-dimensional each).
h
k
100 2
(1)
w
w
(2)
A (1)
1 (2)
(1)
(2)
(. (1)
wa
(1)
w
wig
20
(3)
(8)
w. (3)
(2)
wi.1
(3)
(8)
(0 (2)
W2.1
102
w (3)
(8)
W3.2
(3)
(8)
244.2
w
(3)
h
(1)
(3)
1000 (0 +100
h
h
(3)
A(47 (4)
(4)
y (1)
wi,2
w
w1.4]
(4)
h (4)
((
We added 2 the input layer's weights (1)) (to increase its output dimension from 2 4), added rows and 2 columns to the hidden layer's weights ((²), (3) (to increase its output dimensions from 2 to 4, and to increase its input dimension from 2 4 they consistent with the previous layers), and added 2 columns the output layer's 's weights (4() (to increase its input dimension from 2 to 4 so is consistent with the increase
15

---

Table 8: Concrete growing example.
Layer
Initial Arch
After Growth
New weights added
WO)
(1,2)
(1,4)
4 2
W(2)
(2,2)
(4,4)
4 2
W(3)
(2,2)
(4,4)
4 2 12
W(1)
(2,1)
(4,1)
4 1 - 2 =
in dimensionality of h ).) Note that (3) and (2) still maintain matching shapes (to be added up) in the residual block since we grow (1) and W(3) output dimensions with the same rate.
We summarize architectural growth terms of C.n. and Cout (omitting kernel kc) in Table 8. We also show the growing for CNN hidden layer in Figure 8(a) and residual blocks in Figure 8(b) for better illustration.
Note that this example shows how growing works in general: our specific method also includes particulars as to what values are used to initialize the newly-added weights, as well as modifications to optimizer state.
C Generalization to Other Optimizers
We generalize our LR adaptation rule Adam [20] and AvaGrad [31] in Table 9. Both methods are adaptive optimizers where different heuristics adopted to derive parameter-wise learning strategy, which provides primitives that can extended using our stage-wise adaptation principle for network growing. example, vanilla Adam adapts the global learning rate with running estimates of the first moment m. and the second moment et, of the gradients. where the number of global training steps i is an integer when training fixed-size model. When growing networks, tracks each subcomponent's 'age" (i.e., number steps it been trained for). As such, subcomponent stage 0, t/i] starts as and the learning rate is adapted from m./ v (global) to myglmaae (stage-wise). Similarly, we also generalize our approach to AvaGrad by adopting de, m. of /"ec (eate1]
learning rate adaptation instead considers vector t which
newly grown
the original paper as stage-wise variables.
Preserving Optimizer State/Buffer. Essential to adaptive methods are training-time statistics (e.g. running averages and in Adam) which are stored as buffers and used to compute parameter-wise learning rates. expanded when growing networks, which in practice
Different from fixed-size models, parameter requires ce-instantiating a new optimizer each growth step. Given that our initialization scheme maintains functionality of the network, we are also able to preserve and inherit buffers from previous states, effectively maintaining the optimizer's 's state intact when adding new parameters. We experimentally investigate the effects of this state preservation.
Table 9: Rate adaptation rules for Adam [20] and AvaGrad [31].
Our LR Adaptation
O-th Stage
m / /000
Adam
i-th Stage
majd (
AvaGrad
0-th Stage
[Teuu]/ / / Zotuy la
Nejo)
myjo
i-th Stage
c (maji (m((-1))
[[at /Mate Mate 4/111111
16

---

adopt batch size of 128 when growing ResNet-20 on CIFAR-10. We search the initial learning rate (LR) for LARS over (1e-3, 2e-3, 5e-3, 1e-2, 2e-2, 5e-2, 1e-1, 2e-1, 5e-1] and observe that value of 0.02 yields the best results. adopt the default initial learning rate of 0.1 for both standard SGD and our method. As shown in Table 11, LARS underperforms both standard SGD and our adaptation strategy, suggesting that layer-wise learning rate sufficient for successful Table 10: Generalization to Adam and AvaGrad for ResNet-20 on CIFAR-10.
Optimizer
Training Method
Preserve Opt. Buffer
Train Cost (%)
Test Acc. (%)
Adam
Large fixed-size
NA
100
92.29
Adam
Growing
No
54.90
91.44
Adam
Growing
Yes
54.90
91.61
Aou RA.
Growing
Yes
54.90
92.13
AvaGrad
Large fixed-size
NA
700
52.45
AvaGrad
Growing
No
54.90
90.71
AvaGrad
Growing
Yes
54.90
91.27
AvaGrad+our RA.
Growing
Yes
54.90
91.72
Table 11: Comparisons among standard SGD, LARS, and our adaptation method for growing ResNet-20 on CIFAR-10.
Optimizer
Test Acc. (%)
Standard SGD SGD with Layer-wise Adapt. (LARS) Ours
91.95 + 0.09
91.32 + 0.11
92.53 + 0.11
Results with Adam and AvaGrad. Table 10 shows the results of growing ResNet-20 on CIFAR-10 with Adam and Avagrad. For the large, fixed-size baseline, train Adam with Ir 0.1, 0.1 and AvaGrad with
= 0.5, = 10.0, which yields best results for ResNett-20 following [31]. We consider different settings for comparison: (1) optimizer without buffer preservation: the buffers are refreshed at each new growing phase; (2) optimizer with buffer preservation: the buffer/state is inherited from the previous phase, hence being preserved at growth steps; (3) optimizer with buffer and rate adaptation (RA): applies
rate adaptation strategy described Table 9 while also preserving internal state/butfers. We observe that (1) consistently underperforms (2), which suggests that preserving the state/huffers in adaptive optimizers is crucial when growing networks. Option (3) hests the other settings for both Adam and AvaGrad, indicating that our rate adaptation strategy generalizes effectively to Adam and AvaGrad for the growing scenario. Together, these also demonstrate that our method has the flexibility incorporate different statistics that tracked and used distinct optimizers, where take Adam AvaGrad as examples. Finally, our proposed stage- wise rate adaptation strategy can be employed with virtually any optimizer.
Comparison with Layer-wise Adaptive Optimizer. We also consider LARS [12, 43], a layer-wise adap- tive variant of SGD, to compare different adaptation concepts: layer-wise versus layer + stage-wise (ours). Note that although LARS was originally designed for training with large batches,
adaptation by itself Le.. without accounting for stage wise discrepancies growing of networks.
D More Analysis on Rate Adaptation
We show additional plots of stage-wise rate adaptation when growing ResNet-20 CIFAR-IO. Figure shows the of adaptation factors based on the LR of the seed architecture from to 8th stages (the stage index starts at O), We see an overall trend that for newly-added weights, its learning rate starts at > 1x of the base LR then quickly adapts to a relatively stable level. This demonstrates that our approach is able to efficiently and automatically adapt new weights to gradually and smoothly fade in throughout the current stage's optimization procedure.
We also that rate adaptation is general design that different subnets should not share global learning The RA formulation designed empirically. maz(1, W. W,-1ll) a plausible implementation choice,
Table 12: Comparisons among different RA implementation choices for growing ResNet-20 on CIFAR-10.
RA Implementation Choice
Test Ace. (%)
NA (Standard SGD)
91.62 0.12
max W Wi-lll)
91.42 + 0.12
Ours
92.53 + 0.11
17

---

Supd
54
Eods Sapa
(a) 1-st Stage
(b) 2-nd Stage
(c) 3-rd Stage
(d) 4-th Stage
Epcde Se Segt
(e) 5-th Stage
(f) 6-th Stage
(g) 7-th Stage
(h) 8-th Stage
Figure 9: Visualization of rate adaptation factor dynamics across all growing stages (except O-th)
based the assumption that new weights must have higher learning rate. We conducted experiments by growing ResNet-20 on CIFAR-10. As shown Table 12, we see that this alternative does not work better than our original design, and even underperforms standard SGD.
E More Visualizations on Sub-Component Gradients
We further compare global LR and rate adaptation showing additional visualizations of sub compomentttt gradients different layers and stages when growing ResNet-20 20 CIFAR-I0. We select the 2nd (layerl- blockl-conv/ 1) and 17th (layer3-blowe (lok--bloo22coo22) convolutional layers and plot the gradients cach sub- cobpommmnnntt at the 3rd and 5th growing stages, respectively. in Figures 10, 11, 12, 13. These demonstrate that our rate adaptation strategy able to re-balance and stabilize the gradient's 's contribution of different suhcomponents, hence improving the training dynamics compared global scheduler.
F Simple Example Fully-Connected Neural Networks
Additionally, we train simple fully-connected neural network with 8 hidden layers on CIFAR- 10 each hidden
layer has 500 neurons and is followed by ReLU activations. The network is has final lincar layer with 10 neurons for classification. Note that each CIFAR- 10 image is flattened to 3072-dimensional (32 32 x 3) vector prior being given input the network. We consider adopting training epochs (costs) E (25(1x), 50(2x)} (20(1x).50(2)), We also grow from thin architecture to original one within 10 stages, each stage consisting of epochs, where the number of units of each hidden layer grows from 50 to 100, 150, 500. The total training cost is equivalent to the fixed-size one trained for 25 epochs. We train all basclines using SGD, with weight decay set 0 and learning rates sweeping (0.01, 0.02, 0.05, 0.1, 0.2, 0.53: results are shown Figure 14(a). Compared to standard initialization (green),
variants of this baseline network by
M// MMM0
(a) Using Global
(b) Using RA
(a) Using Global
(b) Using RA
Figure 10: Gradients of 2nd cony 3rd stage.
Figure 11: Gradients of 2nd conv at 5th stage.
18

---

Epeds Dridteg
Qpecka Dr Saye
Epods S40sage
0c5 Say
(a) Using Global
(b) Using RA
(a) Using Global
(b) Using
Figure 12: Gradients of 17th conv at 3rd stage. Figure 13: Gradients of 17th cony at 5th stage.
08
35
Coaing Elinte:
54
Test Accuracy
Train Loss
43
0.5
82
51
02
Ligg Mual Vaa Eae
Lsign Muaniia
01
50
Goviing Sunder
Growngg Dus baaee a thil lespater
2.
2-1
2
21
Leaming Rate
Learning Rate
(a) Train Loss
(b) Test Accuracy
Figure 14: Results of simple fully-connected neural network.
the loss curve given by growing variance transfer (blue) is more similar to the curve of the large baseline all using standard SGD which is also consistent with the observations when training model of different scales separately [41]. Rate adaptation (in red) further lowers training loss. Interestingly,
observe in Figure 14(b) that test accuracy behavior differs from the training loss one given Figure 14(a), which may suggest that regularization is missing due to, for example, the lack of paramcter-sharing schemes (like CNN) in this fulle-coneected network.
G More Comparisons with GradMax [10]
For CIFAR-10 and CIFAR-100, GradMax used different models for growing and did not re-implement GradMax both datasets. Also, generalizing such gradient-based growing methods the Transformer architecture nontrivial. GradMax. Our accuracy outperforms GradMax by 1.3 while lowering training costs, which is significant to demonstrate the benefit of our method. We also trained our method to grow WRN-28-1 (w/wo BatchNorm, used in GradMax paper) on CIFAR-10 and 100 and compare it with GradMax in Table 13. We see that ours still consistently outperforms GradMax.
such, we only cover MobileNet on ImageNet which used in both ours and
Table 13: Comparison with GradMax.
CIFAR-10 (w BN)
CIFAR- 10 (w/o BN)
LIFAR-100 (w/o BN)
Train Cost (%) J
Test Acc. (%) f
Train
Test
Train
Test
Method
Cost (%) $ Acc. (%) f
Cost (%) 4
Acc. (%) t
Large Baseline
100
93.40 + 0.10
100
92.90 + 0.20
100
69.30 +: 0.10
GradMax [10]
77.32
93.00 + 0.10
77.32
92.40
+ 0.10
77.32
66.80
0.20
Ours
58.24
93.29 + 0.12
58.24
92.61 + 0.10
58.24
67.83
0.15
19

---

H Extension to Continuously Incremental Datastream
Another direct and intuitive application for our method is to fit continuously incremental datastream where Do .i... C Dn... Dx-1. The network complexity scales up together with the data so that larger capacity can be trained on more data samples. Orthogonalized SGD (OSGD) [35] address the optimization difficulty in this context, which dynamically re-balances task-spocific gradients via prioritizing the specific loss influence. We further extend our optimizer by introducing a dynamic variant of orthogonalized SGD, which progressively adjusts the priority of tasks on different subnets during network growth.
Suppose the data increases from D to D,. we first accumulate the old gradients Gn-1 using one additional epoch Dn-1 and then grow the network width. For each batch of D., first project gradients of the new architecture (n-th stage), denoted as Gn. onto the parameter subspace that is orthogonal to G zero-pane zero-padded version Gn-1 with desirable shape. The final gradients G are then calculated by re-weighting the original Gn and its orthogonal counterparts:
G = Gn A projoped (Gn); A:1-0
(21)
where is a dynamic hyperparameter which weights the original and orthogonal gradients. When X 1, subsequent outputs do not interfere with carlier directions of parameters updates. We then anneal À to O so that the newly-introduced data and subnetwork can smoothly fade in throughout the training procedure.
Implementation Details. We implement the task in two different settings, denoted 'progressive class' and "progressive data' CIFAR-100 dataset within stages. In the progressive class setting, we first randomly select 20 classes the first stage and then add new classes each growing stage. In the progressive data setting, we sequentially sample fraction of the data with replacement for cach stage, Le., 20%, 30%, 100%.
18 on Continuous 1-000 We evaluate our method on continuous datastreams by growing a ResNet- 18 on CIFAR- 100 and comparing the final test accuracies. As shown in Table 14, compared with the large baseline, our growing method achieves 1.53x cost savings with slight performance degradation in both settings. The dynamic OSGD variant outperforms the large baseline with 1.46x acceleration, demonstrating that the new extension improves the optimization on continuous datastream through gradually re-balancing task-specific gradients of dynamic networks.
Table 14: Growing ResNet- on incremental CIFAR-100.
Progressive Class
Progressive Data
Train Cost (%) 4
Test Acc. (%) f
Train Cost (%) +
Test Acc. (%) 1
Method
Large fixed-size Model
100
76.80
100
76.65
Ours
65.36 68.49
76.50
65.36
76.34
Ours-Dynamic-OSGD
77.53
68.49
77.85
20


Reviews:
[{"review_id": "iXi41BmIYf", "reviewer": "Reviewer_Tyrx", "summary": "This paper presents a method for incrementally growing network width across stages. The reader is shown the details of the growing process using a 3-layer feedforward network as an example, with the appendix providing details for other types of networks. Additional algorithmic details, including learning rate and batch size scheduling are provided. The paper then presents experimental results over 3 repeats, including an ablation study.\n\n**Rebuttal:** I have read the rebuttal and it addresses my questions. Based on the novelty and contribution of this work, I maintain my original rating of 6.", "questions": "- In Table 1, I'm not entirely sure I see a clear pattern: if we had one more layer, for example, would the last layer (whose dimensions I'll still call $C^h$) at initialization have the old weights rescaled as $\\left(\\frac{C^h_t}{C^h_{t+1}}\\right)^{(3/2)}$, and the new inits be $\\frac{1}{(C^h_{t+1})^3}$?\n- In L143, you say using a global learning rate causes divergent behavior: have you seen this in practice? Can you show loss plots under this scheme?\n- In Eq. 5, what is the motivation for setting $f$ to the norm function? Is the Appendix Section 4 what the paper refers to when discussing alternative heuristics (L156)?\n- In Table 2, what do the underscores represent?\n- In Tables 2-5, what statistical tests did you run to compare your methods? I'm also not sure 3 repeats is sufficient to capture variance in performance effectively.\n- In Tables 4-5, please change the training costs of Net2Net to bold (since they were in bold in Tables 2-3).\n- In Table 4, did Net2Net and Firefly achieve the same performance across all 3 runs, or did you only run each once?\n- In Tables 2-5, how is training cost computed? Is it FLOPS/wall time/etc?", "limitations": "The paper's limitations are addressed by providing details for how to overcome them. For example, an ablation study shows the necessity of the learning rate adaptation scheme; the appendix provides details on expansion to residual blocks and convolutional blocks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Section 3 of this paper is perhaps the strongest part of this paper. There is no ambiguity in the presentation, and any questions the reader may have (generalization to Adam and other network architectures) are presented either in the appendix or later in the paper. The paper also does not limit itself to a single architecture, and shows details for two other optimizers, as well as other architecture types. It also provides an ablation study showing how each component of the overall approach is useful.", "weaknesses": "A few parts of the paper are a bit terse. In particular, Section 4 of the Appendix could benefit from additional exploration of, e.g., more alternatives or motivation/derivation for using the proposed alternative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer Tyrx\n\nThank you for the review and comments. We address your points individually below.\n\n**Q: A few parts of the paper are a bit terse. In particular, Section 4 of the Appendix could benefit from additional exploration of, e.g., more alternatives or motivation/derivation for using the proposed alternative.**\n\nA: Thanks for the suggestion.  Since the learning rate adaption (LRA) formulation is designed empirically, one has the flexibility to implement other plausible options.  The motivation of the alternative design choice $max(||W\\_{i} \\setminus W\\_{i-1}||)$ is to assign new weights with higher learning rates.  However, it does not work as well as baselines.\n\n**Q: In Table 1, I'm not entirely sure I see a clear pattern: if we had one more layer, for example, would the last layer (whose dimensions I'll still call $C\\^h$), at initialization have the old weights rescaled as $(\\frac{C\\_t\\^h}{C\\_{t+1}\\^{h}})\\^{3/2}$, and the new inits be $\\frac{1}{(C\\_{t+1}\\^{h})^2}$?**\n\nA: The patterns for initialization are as in Table 1, regardless of the number of layers.  For a network with K layers (K $\\geq$ 3), the initialization rules still fall in three classes as depicted in Tabel 1: (1) 1st layer follows the 'Input Layer' pattern; (2) 2nd to (K-1)-th layers follow 'Hidden Layer' pattern; (3) K-th layer follows 'Output Layer' pattern.  Note that the initialization formulation in Table 1, regarding relative network width, is an extension to the principled zero-shot HP transfer method.  Table 1 takes a three-layer network as a motivating example and the patterns/rules generalize to arbitrary depth ($> 3$).\n\n**Q: you say using a global learning rate causes divergent behavior: have you seen this in practice? Can you show loss plots under this scheme?**\n\nA: We observed the divergent behavior of gradients caused by the global learning rate and plotted them in Figure 4(b) and also in Appendix Figures 5 and 6.  Note that, as shown in the ablation study in Table 6, even with a global learning rate, growing networks is still able to train (loss will not be divergent) but cannot achieve good performance.\n\n**Q: In Table 2, what do the underscores represent?**\n\nA: Underscores represents the second best results.  We will add this explanation to the Table 2 caption in the final version.\n\n**Q: In Tables 2-5, what statistical tests did you run to compare your methods? I'm also not sure 3 repeats is sufficient to capture variance in performance effectively.**\n\nA: We repeatedly train the models 3 times with different random seeds and then report the mean accuracy and standard deviation on the test set using these 3 models.  The observed standard deviations for results in Table 2-5 with 3 repeating runs are small, and imply that our method's performance improvements over competitors is statistically significant.\n\n**Q: In Tables 4-5, please change the training costs of Net2Net to bold (since they were in bold in Tables 2-3).**\n\nA: We will change it to bold in the final version.\n\n**Q: In Table 4, did Net2Net and Firefly achieve the same performance across all 3 runs, or did you only run each once?**\n\nA: For Net2Net and Firefly, we only run each once.  Note that our method's reported mean test accuracy and standard deviation establishes it as a significant improvement over these methods.  For completeness, we will repeat the ImageNet experiments with different random seeds for Net2Net and Firefly in the final version of the paper and include their standard deviations.\n\n**Q: In Tables 2-5, how is training cost computed? Is it FLOPS/wall time/etc?**\n\nA: In Table 2-5, the training cost is computed in FLOPS.  We also conducted a wall-clock time analysis in Figure 7."}, {"review_id": "pMPoyJUnri", "reviewer": "Reviewer_iBKk", "summary": "This paper developed an approach to efficiently grow neural network architecture by parameterization scheme with regard to weight, activation, and gradient scaling. It employed a learning rate adaptation mechanism rebalancing the gradient contribution of the subnetworks. The experimental results show efficiency in wall-clock training time.", "questions": "- In Figure 7 (b), why is it measured over time/epoch?\n- In (6) and (7), what are the base cases?\n- Why was the certain learning rate adaptation rule selected?", "limitations": "The authors did not mention anything about limitations. It would have been nicer if they were upfront about any limitations to share. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-\tThe main technical section is described in detail, although there are some missing points (please refer to the Questions section.)\n-\tThe paper is easy to follow.", "weaknesses": "-\tWondering if the authors considered the original training cost of the original backbone network. For example, if the original network costs XXX for training from scratch, how much will the proposed approach will cost when all steps’ costs are aggregated, apples to apples? \n-\tThere is no description of how to calculate train costs.\n-\tIt is not clear if the subnetworks are trained at a global learning rate or a different learning rate: at line 58, the authors claimed that subnetworks with different lengths have distinct learning rate schedules, while at line 142, it is noted that subnetworks from different growth stages will share a global learning rate. \n-\tIt would have been better if the authors provided justification for why they picked the LR adaptation rule and comparisons with other LR adaptation rules. That is because a learning rate schedule (although, in many cases, it is overlooked) can have a high impact on the final performance and result. So it would have been nicer if the authors explained more or showed more clues on that.\n-\tIn (6), C_0 (base case) is not defined.\n-\tIn (7), the base case is not defined either. Also, are the parentheses (with floor opening and ceiling closing) for rounding to the nearest odd number?\n-\tIt needs to be clarified why Figure 7 (b) is measured over time/epoch. In other words, why the time spent per epoch matters. It would have been nicer if the authors had added an explanation for that.\n-\tIt is not clear the overall benefit of growing a network gradually while preserving the function computed by the model at each growth step. Is there any potential possibility/intention to use such an intermediate model? Otherwise, it is not clear to preserve the functions at each step.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer iBKk\n\nThank you for the review.  However, it appears there is a serious misunderstanding as many of your questions make incorrect assumptions or are directly answered in the paper itself.  Please see our clarifications in response to your individiual points.\n\n**Q: Wondering if the authors considered the original training cost of the original backbone network. For example, if the original network costs XXX for training from scratch, how much will the proposed approach will cost when all steps’ costs are aggregated, apples to apples?**\n\nA: Training costs in Table 2-5 are apples-to-apples comparisons.  We train all methods from scratch and costs of all steps are aggregated in the comparison.  Growing methods do not require the training of an original backbone.  We do compare with the original backbone network training cost, denoted as Large Baseline, with 100\\% costs in all tables.  For example, 54.90\\% means the whole training process of network growing takes 54.90\\% the cost of training a fixed-size full-scale architecture.\n\n**Q: There is no description of how to calculate train costs.**\n\nA: In Table 2-5, training costs are measured by accumulated forward and backward FLOPs.  We report the relative percentage to fixed-size baselines, demonstrating the training efficiency of network growing over full-scale architectures.  We will add this explanation to the revised version.\n\n**Q: It is not clear if the subnetworks are trained at a global learning rate or a different learning rate: at line 58, the authors claimed that subnetworks with different lengths have distinct learning rate schedules, while at line 142, it is noted that subnetworks from different growth stages will share a global learning rate.**\n\nA: In our method, subnetworks are trained with different learning rates.  Line 142 is not to describe our approach; instead, Line 142 points out that sharing a global learning rate may incur optimization difficulties.  Hence, we propose a learning rate adaptation approach to address this concern.\n\n**Q: It would have been better if the authors provided justification for why they picked the LR adaptation rule and comparisons with other LR adaptation rules.**\n\nA: Our learning rate adaptation (LRA) is a general design and one has the flexibility to design different rules empirically.  We choose the weight norm scaling rule which empirically works best with SGD.  As mentioned in Line 156, we do provide comparison to other LR adaptation rules in Appendix Tables 5 and 6 (see the supplementary material).\n\n**Q:  In (6), $C\\_0$ (base case) is not defined; In (7), the base case is not defined either. Also, are the parentheses (with floor opening and ceiling closing) for rounding to the nearest odd number?**\n\nA: The base values vary for different baseline architectures.  We define them in the implementation details: Line 200 to 212.\n\nIn Line 168, we define $\\lfloor \\cdot \\rceil\\_2$ as rounding to the nearest even number for Eq.(6). For Eq.(7), $\\lfloor \\cdot \\rceil$ by default denotes rounding to the nearest integer.\n\n**Q: It needs to be clarified why Figure 7 (b) is measured over time/epoch.**\n\nA: In network growing, the network sizes, training epochs, and training costs vary in different stages. \nHowever, the network maintains the same size within each stage and time/epoch best represents the wall clock training efficiency for each stage.\n\n**Q: It is not clear the overall benefit of growing a network gradually while preserving the function computed by the model at each growth step. Is there any potential possibility/intention to use such an intermediate model? Otherwise, it is not clear to preserve the functions at each step.**\n\nA: Functionality preservation is a principle rule in all network growing methods, including Net2Net, Splitting, Firefly, and Gradmax.  This is to avoid the loss of information already learned from the previous training stages.  Disrupting functionality is equivalent to disrupting the learned behavior of the model, which means some additional training effort must be spent to relearn that behavior.  Adding new weights without functionality preservation leads to longer training time for equivalent performance."}, {"review_id": "lwCW0tbcZQ", "reviewer": "Reviewer_j2wb", "summary": "The authors study the network growth problem and propose a novel problem to stabilize the optimization difficulty. The authors conduct experiments on various model architectures (including VGG, ResNet, MobileNet, and Transformer) and observe consistent performance improvements. ", "questions": "Is the Transformer used in Section 4.3 constructed in the Post-LN or Pre-LN manner? ", "limitations": "I suggest conducting experiments with pre-training tasks like BERT, which would provide great value to this study. Also, as to the machine translation task, the IWSLT14 dataset is not as reliable as the WMT14 EN-De/Fr dataset. Some additional related work on this topic are: 1. Shallow-to-Deep Training for Neural Machine Translation; 2. On the transformer growth for progressive bert training. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The studied problem is important. The proposed approach is novel. Experiments with various model architectures are conducted and demonstrate consistent performance improvements. ", "weaknesses": "The most important application for the proposed algorithm is large-scale training. However, the experiments discussed in this study are based on relatively small models, which left doubts on the applicability of the proposed method on large-scale training. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer j2wb\n\nThank you for the review and comments.  We address your points individually below.\n\n**Q: The most important application for the proposed algorithm is large-scale training. However, the experiments discussed in this study are based on relatively small models, which left doubts on the applicability of the proposed method on large-scale training.**\n\nA: This paper demonstrates the proof-of-concept with results up to ImageNet scale.  We are contributing a new method, explaining the reasoning behind its design choices, and providing ablation studies to analyze its behavior at the largest scale our compute resources will allow.  We agree that extremely large-scale training (e.g., billion-scale parameter models) is a target for future work; we will need partners with access to that level of computing resources. We also appreciate the suggestions and will look into BERT pre-training and WMT14 for the final version.\n\n**Q. Is the Transformer used in Section 4.3 constructed in the Post-LN or Pre-LN manner?**\n\nA: It is constructed in the Pre-LN manner.  We will add this clarification to the final version.\n"}, {"review_id": "dQanx1rZ3q", "reviewer": "Reviewer_GgUs", "summary": "Starting from an optimization and training dynamics perspective, this paper proposes a new method to efficiently growing neural networks, based on a parameterization that dynamically stabilizes weight, activation, and gradient scaling as the architecture grows over time and different subnetworks take the lion's share of the gradient contribution. To achieve this while preserving functional continuity, the approach  presented here grows the network by adding new random untrained neurons with initial net 0 functional contribution, and involves a learning rate scheduling mechanism that rebalances the gradient contribution to different subcomponents of the network added over time. The functional preservation reduces training disruptions and instabilities otherwise introduced by this step-function change in the network.This achieves wall-clock improvement at even or better accuracy than training a large fixed-size model.", "questions": "Just a suggestion: since Figure 3 is too small to be legible, I'd recommend moving it to the appendix.", "limitations": "There are no major concerns about societal impact of this work. Authors do not over-claim impact in this paper, and they are quite matter-of-fact about the applicability of their work. The scale limitation of their experiments, however, prevents this work from being more impactful.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The biggest strength of the paper is that it takes a reasonable, justified, principled approach to improving upon currently available methodologies in network growth, clearly hypothesizes that it might result in certain expected improvements, and then sets out to test this hypothesis through empirical experimentation. This thought process and methodology is clearly outlined in this well-written paper.\n\nThe paper compares to various competing baselines, not only to the large full-size model, thus providing an empirical comparison among similar solutions in the literature. In particular, the authors break down the performance of their method into incremental performance gains provided by the various components in the system they designed, as show in Table 6. These ablations help answer questions about the relative important of each design choice.\n\nI also found the empirical investigation of the growing schedule quite compelling, as I imagine it would be a point of exploration for any practitioner wanting to implement this methodology.\n\n", "weaknesses": "The study is limited to simple, low scale image classification and machine translation tasks, which aren't representative of the application domains often facing researchers and practitioners these days. Therefore, it remains unproved whether this method would help in the context of large scale deep learning, which is the context in which the problems the authors are trying to solve actually occur. The disconnect between the problem statement and the regime in which the solution is tested weakens the point of the paper, as many promising approaches at low scales have proven completely ineffective at large scales.\n\nOne general concern with progressively grown architectures, especially with respect to whether the data distribution is / isn't changed with each growing iteration, is about the generalization properties of the final configuration, whether it loose the generalization properties of the equivalent full-scale architecture when the training data distribution stays constant, or whether it affects the task learning abilities in a continual learning setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer GgUS\n\nThank you for the review and comments.  We address your points individually below.\n\n**Q: The study is limited to simple, low-scale image classification and machine translation tasks, which aren't representative of the application domains often facing researchers and practitioners these days. ... The disconnect between the problem statement and the regime in which the solution is tested weakens the point of the paper, as many promising approaches at low scales have proven completely ineffective at large scales.**\n\nA: We demonstrate the proof-of-concept with results up to ResNet-50 on ImageNet scale.  We are contributing a new method, explaining the reasoning behind its design choices, and providing ablation studies to analyze its behavior at the largest scale our compute resources will allow.  We agree that extremely large-scale training (e.g., billion-scale parameter models) is a target for future work; we will need partners with access to that level of computing resources.\n\n**Q: One general concern with progressively grown architectures, especially with respect to whether the data distribution is / isn't changed with each growing iteration, is about the generalization properties of the final configuration, whether it loose the generalization properties of the equivalent full-scale architecture when the training data distribution stays constant, or whether it affects the task learning abilities in a continual learning setup.**\n\nA: Appendix Section 8 provides an extension to our approach under a continual learning setup.  Specifically, for the continuous data stream where data distribution may shift, we proposed a new Dynamic-OSGD method to accommodate the change along both data and architecture aspects.  In Appendix Table 8, we show that our growing method outperforms the equivalent full-scale architecture under the continual learning setting.  This is an initial step to bridging the dynamics of architectural configuration and data distribution during training.\n\n**Q: since Figure 3 is too small to be legible, I'd recommend moving it to the appendix.**\n\nA: We will move Figure 3 to the Appendix in the final version.\n"}, {"review_id": "5yHM8tT91U", "reviewer": "Reviewer_NBgL", "summary": "This paper studies the growth of neural networks in order to reduce training time. Authors study various optimization challenges when training dynamically grown neural networks and propose some novel techniques to address these challenges. The paper is well written in general, though it requires few clarifications. ", "questions": "- Will the code be made public?\n- How many epochs do you train for ImageNet?\n- In Equation (5) what is f? Does it return momentum statistics? If so, it might be better to use a different letter instead of a function.\n- Algorithm 1, what is S_n? Do you mean W_n?\n- In Tables 2-5, how do you calculate the training cost? It would be nice to add this calculation to the appendix?\n- In Tables 2-5, how do you generate Net2Net and Firefly results. Do you use the same schedule? If different schedules are used, do you do hyper parameter search for these baselines?\n- Can you also report training loss in the appendix? Often small datasets have overfitting problem as shown in Figure 3, which is often not the case for larger datasets and it would be nice to know where the gains are.\n- \"We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers\" Why optimizer states have different treatments in Adam and SGD? Can you do an ablation on this? Also do you do the same for the baselines, i.e. Net2Net and Firefly?", "limitations": "There is no limitation section at the moment. I think inability to match baseline numbers for larger experiments can be mentioned here.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Important research topic with a lot of technical complexity. The community would benefit a lot from results and techniques given assuming authors plan to open-source their code and make their results reproducible.\n- This paper studies the optimization for growing networks, which is an understudied, yet very important topic.\n- This paper proposes a new initialization technique for growing NNs and combines it with a version of LARS optimizer.\n- Experimental evaluation include transformers trained on natural language.\n- Authors achieve real-time speed-ups through additional optimization (i.e. increasing batch size).\n", "weaknesses": "## Major\n- Comparison with other methods in ImageNet is not ideal since baseline numbers might be different in different codebases. Also ideally one would only change the initialization and keep others parts the same to demonstrate that the proposed initialization is better than the alternatives. Looking at the results, it is not clear whether the alternatives would perform worse with the adaptive learning rate. For example one can run Gradmax with the proposed adaptive learning rate and growth schedule and compare the results. In general it would be nice to give more details about the Imagenet experiments and try applying the proposed improvements to other methods and see whether they also improve performance for them.\n- “Together, they accelerate training without impairing model accuracy – a result that uniquely separates our approach from competitors.” This is not supported as baselines are not as good as the original baseline for the MobileNet-Cifar100 and Imagenet experiments. Can we train longer to get better results? It would be nice to run these experiments and check whether growing methods can match baseline performance with additional training. \n- Contributions can be more accurate. (1) Function preserving growth is proposed by many (Openai, gradmax, net2net). (2) Stable training dynamics also exist for these methods. If what authors meant is to be more robust to different learning rates, I believe this is a common feature for adaptive optimizers.  (3) “Existing methods [26, 38] fail to derive a systemic way for distributing training resources across a growth trajectory.” Not clear this is true. Authors suggest a different heuristics than the ones used in previous work. I don't think there is enough motivation to claim this particular way being more systematic than the others. At the end they are all heuristics and the one proposed by this paper seem to perform better. I would focus on this part.\n\n## Minor\n- This paper seems to suggest a new initialization technique for neuron adding. You can also cite Gradmax showing that adding neurons to be more effective than splitting neurons. \n- I would move VGG models to the appendix. For any practical purposes VGGs are a bit irrelevant at this point. \n- Table 4: Firefly->FireflyOpt\n- Figure 3 is very difficult to read in printed form, can you make these plots bigger?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer NBgL\n\nThank you for the review and comments.  We address your points individually below.\n\n**Q: Comparison with other methods in ImageNet is not ideal since baseline numbers might be different in different codebases; There is no limitation section at the moment. I think inability to match baseline numbers for larger experiments can be mentioned here.**\n\nA: For ImageNet, we reimplemented all the methods except Gradmax and compare them under the same setting, including large fixed-size baseline architecture, hyperparamers, optimizers and software/hardware.  For Gradmax on ImageNet, we use the reported results from the respective paper.  Our full-scale fixed-size MobileNet-V1 baseline is of the same performance (70.8) as reported in the Gradmax paper.  On the same network architectures, we achieve the same performance as existing codebases; our comparisons are fair.\n\n**Q: This is not supported as baselines are not as good as the original baseline for the MobileNet-Cifar100 and Imagenet experiments. Can we train longer to get better results? It would be nice to run these experiments and check whether growing methods can match baseline performance with additional training.**\n\nA: During the rebuttal, we trained MobileNet-Cifar100 longer and achieve equivalent performance while still saving on total training cost, as shown in the table below.\n\n|Method |Train Cost  (\\%) $\\downarrow$|  Test Acc. (\\%) $\\uparrow$ | \n|---  |--- |--- |\n|Large Baseline |   100 | $72.13\\pm0.13$|\n|Ours |   52.90 | $71.53\\pm0.13$|\n|Ours|   65.54 | $72.06\\pm0.11$|\n\nWe will add ImageNet results and adjust the discussion of the results accordingly in the final version.\n\n**Q: Contributions can be more accurate.**\n\nA: We will revise our writing to refer to specific technical methods and contributions rather than any overarching claim about high-level concepts.\n\n**Q: This paper seems to suggest a new initialization technique for neuron adding. You can also cite Gradmax showing that adding neurons to be more effective than splitting neurons.**\n\nA: We will add this in the final version.\n\n**Q: I would move VGG models to the appendix. For any practical purposes VGGs are a bit irrelevant at this point.**\n\nA: Thanks for the suggestion. We will move VGG experiments to Appendix.\n\n**Q: Table 4 typo; Figure 3.**\n\nA: We will fix the typo in Table 4; We will make Figure 3 bigger and move it to the appendix in the final version.\n\n**Q: Will the code be made public?**\n\nA: Yes.\n\n**Q: How many epochs do you train for ImageNet?**\n\nA: We follow the standard setting and train 90 epochs on ImageNet.\n\n**Q: In Equation (5) what is f? Does it return momentum statistics? If so, it might be better to use a different letter instead of a function.**\n\nA: $f$ denotes a general function\n\n**Q: Algorithm 1, what is $S\\_n$? Do you mean $W\\_n$?**\n\nA: Yes, it is $W\\_n$. We will fix the typo in the revised version.\n\n**Q: In Tables 2-5, how do you calculate the training cost? It would be nice to add this calculation to the appendix?**\n\nA: We calculated the training costs by accumulating forward and backward FLOPs.  We then report the relative percentage to large fixed-size baselines, demonstrating the training efficiency.  We will add this explanation to the Appendix.\n\n**Q: In Tables 2-5, how do you generate Net2Net and Firefly results. Do you use the same schedule? If different schedules are used, do you do hyper parameter search for these baselines?**\n\nA: Yes, we use the same schedule for Net2Net and Firefly.\n\n**Q: Can you also report training loss in the appendix?**\n\nA: Thanks for the suggestion.  We will revise the appendix to include training loss.\n\n**Q: Why optimizer states have different treatments in Adam and SGD? Can you do an ablation on this? Also do you do the same for the baselines, i.e. Net2Net and Firefly?**\n\nA: In Appendix Section 3, 'Preserving Optimizer State/Buffer', we provide the justification for why we preserve the states during growing.  Appendix Table 4 provides the corresponding ablation study on Adam and Avagrad.\n\nFor SGD, we find preserving the momentum buffer during growing does not improve the final performance. As such, we simply reinitialize it.  We did the same for all Net2Net and Firefly baselines."}, {"review_id": "tNOb8cJ5Ta", "reviewer": "Reviewer_8dnv", "summary": "This paper introduces multiple synergistic tricks that allow for structurally growing networks to be competitive with and even sometimes outperform static networks. These methods include a batched schedule of layer widening, initializing opposing neurons, adapting the scale of the existing weights, handling optimizer  and using subnetwork-specific dynamic learning rates. ", "questions": "Figure 1(b), notably the input weights of $\\delta+\\epsilon_1$ and $\\delta+\\epsilon_2$, and the caption of Figure 2 suggest you are doing “symmetry breaking”, but the rest of your methods do not mention this and rather emphasize that the new weights are two identical copies. Can you clarify this point? Does symmetry breaking help empirically?\n\nWhen including results of methods from related works, did you reimplement/run them yourself or copy their reported results? In the latter case, you must report any differences in experimental protocols beyond methods, including hyperparameters, optimizers, and software/hardware.\n\nThe learning rate adaptation method based on weight norm is not very clearly motivated. How was this selected: just on the previous works, or is there further justification? How is this affected by hyperparameter choices such as initialization scheme and weight decay? Other strategies exist that may be more intuitive, such as scaling by gradient norm [1]. Simply applying a subnet-wise cosine annealing may also suffice. \n\nLee, Yoonho, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. \"Surgical Fine-Tuning Improves Adaptation to Distribution Shifts.\" In The International Conference on Learning Representations. 2023.\n\nCheck the bibtex entry for Chen et al. (2016).\n\nWhenever you refer to the appendix (Lines 116, 156, etc.), please refer to a specific appendix section.\n\nLine 204 has a typo: CIAFR -> CIFAR.\n", "limitations": "No limitations nor potential negative societal impacts are explicitly addressed. Including these discussions would be appreciated.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces a novel synergy of methods to support growing a neural network, some of which are novel themselves. \n\nQuality: The paper provides experimental results demonstrating that the proposed method achieves comparable or better accuracy than large fixed-size models while saving computation time during training across image classification and machine translation benchmarks and backbones architectures.\n\nClarity: Overall, the authors effectively communicate their ideas, methodologies, and results, making it easier for readers to understand the research. Some inconsistencies are noted below.\n\nSignificance: This is also the most thorough and practical work on growing models to my knowledge, from shallow and deep CNNs to transformers. \n", "weaknesses": "Please check and fix the formatting of your paper. Table 5 violates the right margin. Table 6 is hard to read without sufficient spacing between the columns. Many figures contain text that is too small and completely illegible when printed: for accessibility, please ensure all fonts within figures are legible at only 100% zoom.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer 8dnv\n\nThank you for the review and comments.  We address your points individually below.\n\n**Q: Formatting, margins, fonts.**\n\nA: In the final version, we will correct margins and spacing in Tables 5 and 6, and increase font size in figures.\n\n**Q: the caption of Figure 2 suggest you are doing “symmetry breaking”, but the rest of your methods do not mention this and rather emphasize that the new weights are two identical copies. Can you clarify this point? Does symmetry breaking help empirically?**\n\nA: We adopt symmetry breaking as suggested in Net2Net.  As indicated in Line 197, the noise for symmetry breaking is 0.001 to the norm of the initialization, which is negligible and does not affect our claim of functionality preservation during growth.  Symmetry breaking is necessary to enable the network to eventually learn to utilize all new parameters; otherwise identically initialized new parameters would receive exactly the same gradients and remain identical.\n\n**Q: When including results of methods from related works, did you reimplement/run them yourself or copy their reported results? In the latter case, you must report any differences in experimental protocols beyond methods, including hyperparameters, optimizers, and software/hardware.**\n\nA: We reimplemented all the methods except Gradmax-ImageNet and compare them under the same setting, including large fixed-size baseline architecture, hyperparamers, optimizers and software/hardware. For Gradmax on ImageNet, we use the reported results from the respective paper.  Our own trained large fixed-size MobileNet-V1 baseline yields the same performance (70.8) as that reported in the GradMax paper for the same architecture.  As we have consistent results between both codebases, our results are a fair comparison.  In Appendix Section 7, we also reimplement and compare with Gradmax on CIFAR-10 and CIFAR-100 under the same settings (and our method again outperforms Gradmax).\n\n**Q: No limitations nor potential negative societal impacts are explicitly addressed. Including these discussions would be appreciated.**\n\nA: We will add these discussions to the final version.\n\n**Q: The learning rate adaptation method based on weight norm is not very clearly motivated. How was this selected: just on the previous works, or is there further justification? How is this affected by hyperparameter choices such as initialization scheme and weight decay? Other strategies exist that may be more intuitive, such as scaling by gradient norm [1]. Simply applying a subnet-wise cosine annealing may also suffice.**\n\nA: Our learning rate adaptation (LRA) is a general design that different subnets should not share a global learning rate.  Our LRA formulation is designed empirically and we find scaling by weight norm works best for network growing with SGD.  Note that the LRA design choice is independent of the choice of other optimizer hyperparameters (HPs).  In our method, we choose the same set of HPs for a fair comparison.\n\nWe do have some further investigation about the LRA design choices, as shown in Appendix Section 3 Table 5 with LARS (scaling by both weight norm and gradient norm) and Table 6 (new weights with higher learning rates).\n\nDuring the rebuttal, we also conducted experiments for ResNet-20 growth on CIFAR-10 using the other two LRA designs as suggested, with the same set of other optimizer HPs.  As shown in the table below, we find our method works best with SGD for network growing.\n\n |LRA Impl. Choice|   Test Acc. (\\%) $\\uparrow$ | \n|--- |--- |\n|   NA (Standard SGD)\t| $91.62\\pm0.12$|\n|  Scaling Gradient\t| $91.23\\pm0.10$|\n|   Subnet-wise cosine annealing\t| $91.98 \\pm 0.11$|\n|  Ours | $92.53\\pm0.11$|\n\n\n**Q: Check the bibtex entry for Chen et al. (2016); Refer to specific appendix section; Line 204 typo.**\n\nA: We will fix in the final version.\n"}]

Author rebuttal:
# General Reply

We thank the reviewers for the detailed comments and answer specific questions in individual responses to each review below.  We also conducted additional experiments on learning rate adaption (prompted by Reviewer 8dnv) and slightly longer training (prompted by Reviewer NBgL).  Please see these results in the respective replies below.

Meta-review: This paper introduces an efficient method for expanding neural network architectures through parameterization, encompassing weight, activation, and gradient scaling. It utilizes a learning rate adaptation mechanism to rebalance gradient contributions from subnetworks. Experimental results demonstrate notable reductions in wall-clock training time. All the reviewers unanimously accepted the paper, and so did the final decision. The authors should also revise their paper according to the reviewers' suggestions in the final version.
Decision: Accept (poster)