Accelerated Training via Principled Methods for Incrementally Growing Neural Networks
Authors: Xin Yuan
Pedro Henrique Pamplona Savarese
Michael Maire
Venue: Submitted to ICLR 2023 (2023)

Abstract: We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics.  Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network.  To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents.  Experimental results show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original computation budget for training.  We demonstrate that these gains translate into real wall-clock training speedups.

Under review as a conference paper at ICLR 2023
ACCELERATED TRAINING VIA PRINCIPLED METHODS FOR INCREMENTALLY GROWING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We develop approach efficiently grow neural networks, within which param- eterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-basee local optimization, we craft parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference func- tionality of the network. To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents. Experimental results show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original computation budget for training. We demonstrate that these gains translate into real wall-clock training speedups.
INTRODUCTION
Modern neural network design typically follows a "larger is better" rule of thumb, with models consisting of millions of parameters achieving impressive generalization performance across many tasks, including image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Real al., 2019; Zhai et al., 2022), object detection (Girshick, 2015; Liu al., 2016; Ghiasi al., 2019), semantic segmentation (Long et al., 2015; Chen et al., 2017; Liu et al., 2019a) and machine translation (Vaswani et al., 701;; Devlin et al., 2019). Within a class of network architecture, deeper or wider variants of base model typically yield further improvements accuracy. Residual networks (ResNets) (He et al., 2016b) and wide residual networks (Zagoruyko Komodakis, 2016) illustrate this trend in convolutional neural network (CNN) architectures. Dramatically scaling up network size into the billions of parameter regime has recently revolutionized transformer-base language modeling (Vaswani al., 2017: Devlin et al., 2019; Brown et al., 2020).
The size of these models imposes prohibitive training costs and motivate techniques that offer cheaper alternatives to select and deploy networks. For example, hyperparameter tuning is notoriously expensive as it commonly relies on training the network multiple times, and recent techniques aim to cireumvent this by making hyperparameters transferable between models of different sizes, allowing them to be tuned on small network prior to training the original model once (Yang et al., 2021).
Our approach incorporates these ideas, but extends the scope of transferability rameters of the model itself. Rather than view training small and large models as separate events, we grow a small model into a large one through many intermediate steps, each of which introduces additional parameters to the network. Our contribution is to do so in a manner that preserves the function computed by the model each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training large static model from scratch.
include the pa-
Motivating this general strategy, we view aspects prior works as hinting that deep network training may naturally be amenable to dynamically growing model size. For example, residual connections (He et al., 2016b) introduce depth-wise shortcuts, solving a gradient vanishing issue and thereby making very deep networks end-to-end trainable. Prior to ResNet, manually circumventing this issue involved

---

Under review as a conference paper at ICLR 2023
Spitt Neurrone
Grow Neurons
Mac
Th Y/2
Paremetertaation
Tac
M
no
1/1
Split / Add
1r/2:
OMl PiP:
Split
Grow
V. Trunsfer 6.6: Grnd-iie--n Int.
Grad Intt
Ps Small Optiniatioon
Moc
0+ c
Po f2 Small hoss
Noo
11
0
no
0
Fss Seperate LI
Global
(a) Existing Methods: Splitting Init, Global LR
(b) Ours: Function-Preservingg Init, Stagewise LR
Figure 1: Dynamic network growth strategies. Different from (a) which reply either split- ting (Chen et al., 2016; Liu et al., 201 9b; Wu et al., 2020b) or adding neurons with auxiliary local optimization (Wu et al., 2020a; Evci et al., 2022), our initialization (b) of new neurons is random but function-preserving Additionally, our separate learning rate (LR) scheduler governs weight updating in order address the discrepancy in total accumulated training between different growth stages.
adding outputs and losses to intermediate layers (Szegedy al., 2015), effectively causing a shallower subnetwork to train first and bootstrap the training of the full network. Larsson et al. (2016) show that end-to-end training of FractalNet, an alternative shorteut architecture, implicitly trains shallower subnetworks first. If such phenomena, though perhaps difficult to analyze, occurs more broadly, it suggests that one might achieve computational advantage by adopting an explicit growth strategy that matches the implicit subnetwork training schedule which occurs within large static network.
While overparameterization benefits generalization, a more detailed view suggests possible compati- bility between the desire to maintain overparameterizee deep network and to dynamically grow such a network. The "double-descent" varsinarinnce trade-off curve (Belkin et al., 2019) indicates that large model capacity may be safe strategy to ensure operation in the modern interpolating regime with consequently low test error. Small models, as we take for a starting point in dynamic growth, might not be sufficiently overparameterized and incur higher test error. However, Nakkiran et al. (2020) experimentally observe double-deseent occurs with respect to both model size and number of training epochs. To remain in the interpolating regime, model must be overparameterized relative to the amount it has been trained, which can be satisfied by an appropriate growth schedule.
Competing recent efforts to grow deep models from simple architectures (Chen al., 2016; Li et al., 2019; Dai et al., 2019; Liu et al., 2019b; Wu et al., 2020b; Wen et al., 2020; Wu et al., 2020a; Yuan et al., 2021; Evei et al., 2022) draw inspiration from other sources, such as the progressive development processes of biological brains. In particular, Net2Net (Chen et al., 201 6) grows the network by randomly splitting learned neurons from previous phases. This replication scheme, shown in Figure ((a) is common paradigm for most existing methods. Splitting steepest descent (Wu et al., 2020b) determines which neurons to split and how to split them by solving a combinatorial optimization problem with auxiliary variables. Firefly (Wu et al., 2020a) further improves flexibility by incorporating optimization for adding new neurons. Both methods outperform simple heuristics, but require additional training effort in their gradient-based based parameterization schemes. Furthermore, all existing methods use a global learning rate scheduler to govern weight updates, ignoring the discrepancy in total training time among subnetworks introduced in different growth phases.
We develop growing framework around the principles of enforcing transferability of parameter settings from smaller to larger models (extending Yang et al. (2021)), offering functional conti- nuity, smoothing optimization dynamics, and rebalancing learning rates between older and newer subnetworks. Figure 1(b) illustrates key differences with prior work. Our core contributions are:
Parameterization using Variance Transfer:
We propose a parameterization scheme accounting for the variance transition among networks of smaller and larger width in a single training process. Initialization of new weights is gradient- and requires neither additional memory nor training.
Improved Optimization with Rate Adaptation: Subnetworks trained for different lengths have distinct learning rate schedules, with dynamic relative scaling driven by weight norm statistics.
Better Performance and Broad Applicability: Our method not only trains networks fast, but also yields excellent generalization accuracy, even outperforming the original fixed-size models. Flexibility in designing a network growth curve allows choosing different trade-offs between training resources and accuracy. Furthermore, adopting acceleration in terms of wall-clock training time. We demonstrate results on image classification and machine translation tasks, across diverse set of network architectures.
adaptive batch size schedule provides
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Network Growing. A diverse range of techniques train models by progressively expanding the network architecture (Wei et al., 2016; Elsken et al., 2018; Dai et al., 2019; Wen et al., 2020; Yuan et al., 2021). Within this space, the methods of Chen et al. (2016); Liu et al. (2019b); Wu et al. (2020b;a); Evei et al. (2022) are most relevant to focus incrementally growing network width across multiple training stages. Net2Net (Chen et al., 2016) proposes a gradient- free neuron splitting scheme via replication, enabling knowledge transfer from previous training phases; initialization of new weights follows simple heuristics. Liu et al. (2019b)'s Splitting approach derives gradient-based scheme for duplicating neurons by formulating a combinatorial optimization problem. FireFly (Wu et al., 2020a) gains flexibility by also incorporating brand new neurons. Both of these methods improve Net2Net's initialization scheme by solving an optimization problem with auxiliary variables, at the cost of extra training effort. GradMax (Evci et al., 2022), in consideration of training dynamics, performs initialization via solving a singular value decomposition (SVD) problem.
Neural Architecture Search (NAS) and Pruning. Another subset of methods mix growth with dynamic reconfiguration aimed at discovering or pruning task-optimized architectures. Network Morphism (Wei et al., 2016) searches for efficient networks by extending layers while preserving the parameters. Autogrow (Wen et al., 2020) takes an AutoML approach governed by heuristic growing and stopping policies. Yuan et al. (2021) combine learned pruning with sampling strategy that dynamically increases or decreases network size. Unlike these methods, we focus on the mechanics of growth when the target architecture is known, addressing the question of how to best transition weight and optimizer state to continue training an incrementally larger model. NAS and pruning are orthogonal to, though potentially compatible with, the technical approach we develop.
Hyperparameter Transfer. Yogatama & Mann (2014); Perrone et al. (2018); Horváth et al. (2021) explore transferrable hyperparameter (HP) tuning. The recent Tensor Program (TP) work of Yang Hu (2021); Yang et al. (2021) focuses on zero-shot HP transfer across model scale and establishes a principled network parameterization scheme to facilitate HP transfer. This serves as an anchor for our strategy, though, as Section 3 details, modifications are required to account for dynamic growth.
Learning Rate Adaptation. Surprisingly, the existing spectrum of network growing techniques utilize relatively standard learning rate schedules and do not address potential discrepancy among subcomponents added at different phases. One might expect that newer weights should have higher learning rates than older weights. While general-purpose adaptive optimizers (e.g., AdaGrra (Duchi et al., 2011), RMSProp (Ticleman et al., 2012), Adam (Kingma & Ba, 2015), AvaGrad (Savarese et al., 2021)) might ameliorate this issue, we choose to explicitly account for the discrepancy. As layer-adaptive learning rates (LARS) (Ginsburg et al., 2018; You et al., 2020) benefit in some contents, we explore further learning rate adaption specific to both layer and growth stage.
3 METHOD
3.1 PARAMETERIZATION AND OPTIMIZATION WITH GROWING DYNAMICS
Functionality Preservation. We grow network capacity by expanding the width of computational units (e.g., hidden dimensions in linear layers, filters in convolutional layers). To illustrate our scheme, consider a 3-layer fully-connected network with ReLU activations Φ::
(1) where x E RC the network input, y E RC.t is the output, and hn E Hio ,hh E Hous are the hidden activations. In this case, W is a Hi, x C.n matrix, while /h is Hout x Hin and Wour Cout x Hout- After training the network for a few epochs, we increase its capacity by increasing the dimensionality of each hidden state, i.e. from Hin and Hout to Hin and Hout, respectively. The layer parameter matrices W have their shapes changed accordingly and become W.
ón ((W'" x)) h2 <(W"h"))
y y= W'owh" h"
Figure 2 illustrates the process for initializing W. As Figure 2(a) shows, we first expand Win along the output dimension by adding two copies of new weights Win of shape H.-HH. x Cin.
2
'ee defer the transformation between Wold and Weew to the next subsection. It involves rescaling by constant factors, does not affect network functionality, and is omitted in Eq. 1-4 for simplicity.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Paramoterization and optimization transition for different layers during growing.
Input Layer
Hidden Layer
Output Layer
Old Weighs Scaling
1
Hi/ Hm
Hout/ Hout
Init.
New Weighs Init.
1/Cin
1/Hm
1/(Hou)) out) )
0-th Stage
1
1/H out W W
LR Adapt.
[W wi | [
i-th Stage
W
generating new features ft = o( 6W x) and changing the first set of activations from h to
h am concat(h" (h" hn (()))
Next, we expand W across both input and output dimensions, shown in Figure 2(b). We initialize new weights W of shape Hout x 2 + +W and - W This preserves the output of the layer since
- and add to W two copies of it with different signs:
@(""h" tn + Whh"" + (- )-,"" s(W"h h²
We then add two copies of new weights W which has shape Hant Hons x Hin, yielding activations
h concal(h", (W" h o0 hW"hh"))
We similarity expand Wout to match the dimension of hhg As Figure 2(c) shows, the final output is:
o Wo ho + Wout o( ((W.h"") + (- W )o (W
which preserves the original output features in Eq. 1.
GG.
C₁
Weights Initialization with Variance Transfer (VT). Yang et al. (2021) investigate weight scaling with width at initialization, allowing hyperparameter transfer by calibrat- ing variance across model size. They modify the variance of output layer weights from the commonly used 1/fammn to 1 L/fan We adopt this same correction for the added weights with new width: Wout and Wont are initialized
Witt
Him
Ain
ald
Win
Win
(a) Input Layer
H
with variances of 1/Hous and 1/Hoout respectively.
Moxs
u 1/ 19 12--00811/8 (...
However, this correction considers training differently- sized models separately, which fails to accommodate the training dynamics in which width grows incrementally. To make the weights of the old subnetwork old (0, 1 (H )2) compatible with the entire weight ten- parameterization, rescale it with the fan;, ratio
(b) Hidden Layer
Hest
With
new Want How Hout- Also see Table (top).
(c) Output Layer
This parameterization rule transfers to modern convolu- tional networks with batch normalization (BN). Given a weight scaling ratio of the running mean μ and variance o of BN layers are modified as cp and c² c²,,, respectively.
Figure 2: Initialization scheme. In prac- tice, we also add noise to the expanded parameter sets for symmetry breaking.
Stage-wiss wise Learning Rate Adaptation (RA). Following (Yang et al., 2021), employ learning rate scaling factor of x 1/fan;n, on the output layer when using SGD, compensating for the initial- ization scheme. However, subnetworks from different growth stages still share global learning rate, though they have trained for different lengths. This may cause divergent behavior among the corresponding weights, making the training iterations after growing sensitive the scale of the newly-initialized weights. Instead of adjusting newly added parameters via local optimization (Wu et al., 2020a; Evei et al., 2022), we govern the update of each subnetwork in a stage-wise manner.
Suppose layer scales up with width at different growing stages So C S1 C C Sy-1 with associated weight and gradient tensors as Wo C W, C... C WN-1 and Go C C C... C GN-1, respectively. We adapt the learning rate and update the i-th sub-weights W, W,-1 as:
= no0 * f(S \ S,-1)
W
Wi-1 + - * (G. \ G.-1), i> 0
(5)
f(S))
4

---

Under review as a conference paper at ICLR 2023
where 7o is the base learning rate and is implicit function that maps subnetworks of different stages to corresponding train-time statistics. Table (bottom) summarizes our LR adaptation rule for SGD when is instantiated as weight norm. Alternative heuristics possible; see Appendix A.I.
3.2 FLEXIBLE AND EFFICIENT GROWTH SCHEDULER
We train the model for nntal epochs by expand- ing the channel number of each layer to final across N growth phases. Existing methods (Liu et al., 2019b; Wu et al., 2020a) fail to derive systemic way for distributing training resources across a growth trajectory. Toward maximizing efficiency, we experiment with a coupling be- tween model size and training epoch allocation.
Algorithm 1 Growing using Var. Transfer and Learning Rate Adapt. with Flexible Scheduler
Input: Data X, labels Y. task loss L Output: Grown model S
Initialize: So with Co. To. Bo. ?/0
for n to N do
if > 0 then
Init. S, from Sn using VT in Table 1, Update Cn and Tú using Eq. 6 and Eq. 7.
Architectural Scheduler. We denote initial channel width as Co and expand exponentially:
Update B,, using Eq. 8 (optional)
IteIcotal Tn (xn(X)//e))))
+ 11-22-112 if N C. 1C
[ Cyiial
end if
(6)
n
== N
for i= to Iterrotal do
Forward and calculate /
where L-12 rounds to the nearest even number and Pe is the growth rate between stages.
Back propagation with I.
Update each sub-component using Eq. 5. end for end for return SN-1
Epoch Scheduler. We denote number of epochs assigned to n-th training stage as Tn. with Enzo Tn = Tiots.. We similarly adapt Tn via an exponential growing scheduler:
Tn-1 1p-Ta-11 if n<N-1 N-2
T₂
Teotal Limo T if n n= N
Wall-clock Speedup via Batch Size Adaptation. Though the smaller architectures in early growth stages require fewer FLOPs, hardware capabilities may still restrict practical gains. When growing width, in order to ensure that small models fully utilize the benefits GPU parallelism, we adapt the batch size along with the exponentially-grovee architecture in reverse order:
{ Buase
if n N n <
Bn-1
Bn
Pb B,
if
where Brase is the batch size of the large baseline model. Algorithm summarizes our full method.
EXPERIMENTS
We evaluate on image classification and machine translation tasks. For image classification, use CIFAR-I0 (Krizhevsky et al., 2014), CIFAR-100 100 (Krizhevsky et al., 2014) and ImageNet (Deng et al., 2009). For the neural machine translation, we use the IWSLT' 14 dataset (Cettolo al., 2014) and report the BLEU (Papineni et al., 2002) score on German to English (De-En) translation task.
Large Baselines via Fixed-size Training, We use VGG- (Simonyan Zisserman, 2015) with BatchNorm (Ioffe & Szegedy, 2015), ResNet-20 (He et al., 2a), MobileNetVI (Howard et al., 2017) for CIFAR-10 and VGG-19 with BatchNorm, ResNet- 18, MobileNetV1 for CIFAR-100. We follow Huang et al. (2016) for data augmentation and processing, adopting random shifts/mirroring and channel-wise normalization. CIFAR-I0 and CIFAR-100 models are trained for 160 and 200 epochs respectively, with a batch size of 128 and initial learning rate (LR) of 0.1 using SGD. We adopt a cosine LR schedule and set the weights decay and momentum as 5e-4 and 0.9. For ImageNet, train the baseline ResNet-50 and MobileNetVi (Howard et al., 2017) using SGD with batch sizes of 256 and 512, respectively. We adopt the same data augmentation scheme as Gross & Wilber (2016), the cosine L.R scheduler with initial LR of 0.1, weight decay of 1e-4 and momentum of 0.9.
5

---

Under review as a conference paper at ICLR 2023
Table 2: Growing ResNet-20, VGG-I1, and MobileNetVI CIFAR-10.
ResNet-20
VGG-11
MobileNetvl
Train
Test
Train
Test
Train
Test
Method
(ost(%) 4 Accuracy(%) t Cost(%)
A.ccuracy(%) (%) t Cost(%)
4 Accuracy (%) 1
Large Baseline
100
92.62 + 0.15
100
92.14 0.22
100
92.27 0.11
Net2Net
54.90
91.60 + 0.21
52.91
91.78 0.27
53.80
90.34 + 0.20
Splitting
70.69
91.80 0.10
63.76
91.88 + 0.15
65.92
91.50 + 0.06
FireFly-split
58.47
91.78 0.11
56.18
91.91 + 0.15
56.37
91.56 + 0.06
FireFly
68.96
92.10 0.13
60.24
92.08 = 0.16
62.12
91.69 - 0.07
Ours
54.90
92.53 + 0.11
52.91
92.34 + 0.15
53.80
92.01 + 0.10
Table 3: Growing ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100.
ResNet-18
VGG-19
MobileNetvl
Train
Test
Train
Test
Train
Test
Method
Cost(%) 4. Accuracy( (%) t Cost + Accuracy(%) f Cost(% (%)
Accuracy( (%)
Large Baseline
100
78.36 + 0.12
100
72.59 + 0.23
100
72.13 = 0.13
Net2Net
52.63
76.48 + 0.20
52.08
71.88 + 0.24
52.90
70.01 = 0.20
Splitting
68.01
77.01 = 0.12
60.12
71.96 0.12
58.39
70.45 = 0.10
FireFly-split
56.11
77.22 0.11
54.64
72.19 0.14
54.36
70.69 0.11
FireFly
65.77
77.25 + 0.12
57.48
72.79 0.13
56.49
70.99 = 0.10
Ours
52.63
78.12 0.15
52.08
73.26 + 0.14
52.90
71.53 + 0.13
For IWSLT' 14, we train an Encoder-Decoder Transformer (6 attention blocks each) (Vaswani al., 2017). We set width as dmodel 1/4d = 512, the number of heads Thead 8 and d3 = dg = d, = dmodel/ Mhead = 64. We train the model using Adam for 20 epochs with learning rate 1e-3 and (3,, 52) = (0.9, 0.98). Batch size 1500 and we use 4000 warm up iterations.
Implementation Details. We compare with the growing methods Net2Net (Chen et al., 2016). Splitting (Liu al., 2019b), FireFly-split, FireFly (Wu et al., 2020a) and GradMax (Evei et al., 2022).
For image classification, we run the comparison methods except GradMax alongside our algorithm for all architectures under the same growing scheduler. For the architecture scheduler, we set Pe as 0.2 and Co as 1/4 of large baselines in Eq. 6 for all layers and grow the seed architecture within N = 9 stages towards the large ones. For epoch scheduler, we set Pt. as 0.2, To as 8, 10, and 4 in Eq. 7 on CIAFR-10. 10, CIFAR- 100, and ImageNet respectively. Total training epochs Trotal the same as the respective large fixed-size models. For CIFAR- 10 and CIFAR- 100, we train the models and report the results averaging over 3 random seeds.
For machine translation, we grow the encoder and decoder layers' widths while fixing the embedding layer dimension for consistent positional encoding table. The total number of growing stages is 4, each trained for epochs. The initial width is 1/8 of the large baseline (i.e. dmodee = 64 and dx.g. 8). We the growing ratio Pe. as 1.0 that dmooed evolves 64, 128, 256 and 512.
We train all the models on an NVIDIA 2080Ti 12GB GPU for CIFAR-10, CIFAR-100, and IWSLT' and two NVIDIA A40 48GB GPUs for ImageNet.
4.1 CIFAR RESULTS
All models grow from a small seed architecture to the full-sized one in 9 stages, each trained for (8.9, 11, 13, 16, 19, 23, 28,33) 33} epochs (160 total) in CIFAR-10, and {10, 12, 14, 20, 24, 29, 35, 39) (200 total) in CIFAR-100. Net2Net follows the design of growing by splitting via simple neuron replication, hence achieving the same training efficiency as our gradient- free method under the same growing schedule. Splitting and Firely require additional training effort for their neuron selection schemes and allocate extra GPU memory for auxiliary variables during the local optimization stage. This is computationally expensive, especially when growing large model.
ResNet-22, VGG-11, and MobileNetV1 on CIFAR-10. Table shows results in terms of test accuracy and training cost calculated based on overall FLOPs. For ResNet-20, Splitting and Firefly

---

Under review as a conference paper at ICLR 2023
Table 4: ResNet-50 and NobileNerVi ImageNet.
Table 5: Transformer on IWSLT 14,
ResNet-50
MobileNet-vi
Transformer
Train
Test (%c.
Train
Test
Train Cost(%) 4 Score 1
BLEU
Method
Method
Cost(%) (%)
Cost(%)
+ Acc.(%)
Large
100
76.72
100
70.80
Large
100
32.84
Net2Net
60.12
74.89
63.72
66.19
Net2Net
64.64
30.94
FireFly 71.20
75.01
86.67
66.40
Ours-w/o buffer
64.64
31.42
GradMax
86.67
68.60
Ours-w buffer 64.64
31.66
Ours 60.12
75.90
63.72
69.92
Ours-w buffer-RA
64.64
32.08
achieve better test accuracy than Net2Net, which suggest the additional local optimization benefits neuron selection at the cost of training efficiency. Our method requires only 54.9% of the baseline training cost and outperforms all competing methods, while yielding only 0.09p.p (percentage points) performance degradation compared to the static baseline. Moreover, our method can even outperform the large fixed-size VGG-11 by 0. 0.20p.p test accuracy, while taking only 52.91% % of its training cost. For MobileNetV1, 1, our method also achieves the best trade-off between training efficiency and test accuracy among all competitors.
R8sNet-18, VGG-19, and MobileNetV1 on CIFAR-100. We also evaluate all methods on CIFAR- 100 using different network architectures. Results in Table show that Firely consistently achieves better test accuracy than Firefly-split, suggesting that adding new neurons provides more flexibility for exploration than merely splitting. Both Firely and our method achieve better performance than the original VGG-19, suggesting that network growing might have additional regularizing effect. Our method yields the best accuracy and largest training cost reduction.
4.2 IMAGENET RESULTS
We first grow ResNet-50 on ImageNet and compare the performance of our method to Net2Net and FireFly under the same epoch schedule: (4, 4, 5, 6, 8, 9, 11, 14, 29} (90 total) with 9 growing phases. We also grow MobileNetVI from a small seed architecture, which is more challenging than ResNet-50. We train Net2Net and method using the same scheduler for ResNet-50. We also compare with Firefly-Opt (a variant of FireFly) and GradMax and report their hest results from Evei al. (2022). Note that both methods not only adopt additional local optimization, but also train with less efficient growing scheduler: the final full-sized architecture needs to be trained for a 75% fraction while ours only requires 32.2%. Table shows that our method dominates all competing approaches.
4.3 IWSLT14 DE-EN RESULTS
We grow a Transformer from dmodet = 64 to dmodel = 512 within 4 stages, each trained with 5 epochs using Adam. Applying gradient-based growing methods to the Transformer architecture is non-trivial due to their domain specific design of local optimization. As such, we only compare with Net2Net. We also design variants of our method for self-comparison, based on the adaptation rules for Adam in Appendix A. As shown in Table 5, our method generalizes well to the Transformer architecture for the machine translation task. Comparison among variants is also consistent with Table 7, demonstrating the benefit of learning rate adaptation.
4.4 ANALYSIS
Variance Transfer. We train a simple neural network with 4 convolutional layers on CIFAR-10 10. The network consists of 4 -ereuurron--eeevvingg convolutional layers; each convolution has 64, 128, 256 and 512 channels, a 3 x 3 kernel, and is followed by BatchNorm and ReLU activations. Max-pooling is applied to each layer for a resolution-dowwsampiin of 4, 2, 2, and These CNN layers are then followed by a linear layer for classification. We first alternate this network into four variants, given by combinations methods {standard, jatrnnsfer ( (Yang et al., 2021)). We also grow from thin architecture within 3 stages, where the channel number of each layer starts with only 1/4 of the original one, i.e. (16, (16,32, 32, 64, 128) - (32, 64, 128, 256) {64, 1 128, 256, 512), each stage is trained for 10 epochs.
training epochs E {13(1x), 30(2.3))) and initialization

---

Under review as a conference paper at ICLR 2023
+ Stanie Handert222 Celll
18
Lage jirendent la Zeer
pfrerslortx froe Gosl
GewigBlander
14
Gowiggbba Ner Turate Stancer squt ++ fiwwreitinn aa ty Tardie-S00 with bev Araparer)
Tost Acccua Accuracy/9o
Thain
30
Lag FardariD Cart
Lage Sarderili
Lagg altunslet22 Gaitl
66
Large atranstertic Call
Growag Torde--ttrdde 500)
Alsplatin
24
2
2-
2
Learning Rate
Learning Rate
(a) Train Loss
(b) Test Accuracy
Figure 3: Different baselines of 4-layers simple CNN on CIFAR-10.
Screet
Sureet
Teet setsccuurry (N)
Schaed
Init + SGD
Trander Stnddnd
Stage
8-th Stage
7:th
8-th
Epocholas: Stage)
Epochstlast Two Stagest
EsschoLast Neo Strgest
(a) Test Accuracy
(b) Gradients with Global LR
(c) Gradients with Rate Adaptation
Figure 4: (a) Performance with Var. Transfer and Rate Adaptation growing ResNet-20. (b) and (c) visualizes the gradients for different sub-compoents along training in the last two stages.
For network growing, we compare the baselines with standard initialization and variance transfer. We train all baselines using SGD, with weight decay set as O and learning rates sweeping over (0.01, 0.02, 0.05, 0.1, 1, 0.2, 0.5, 0.8, 1.0, 1.2, 1.5, 2.0]. In Figure 3(b), growing with Var. Transfer (blue) achieves overall better test accuracy than standard initialization (orange). Large baselines with merely jutransfer in initialization consistently underperform standard ones, which validate that the compensation from the LR re-scaling is necessary in Yang et al. (2021). We also observe, in both Figure 3(a) and 3(b), all growing baselines outperform fixed-size ones under the same training cost, demonstrating positive regularization effects. We also show the effect of our initialization scheme by comparing test performance standard ResNet-20 CIFAR- 10. As shown in Figure 4(a), compared with standard initialization, our variance transfer not only achieves better final test accuracy but also appears more stable. See Appendix A.4 for fully-connected network example.
Learning Rate Adaptation. We investigate the value of our proposed stage-wise learning rate adaptation an optimizer for growing networks. As shown in the red curve in Figure 3, rate adaptation not only bests the train loss and test accuracy among all baselines, but also appears to be more robust over different learning rates. In Figure 4(a), rate adaptation further improves final test accuracy for ResNet-20 on CIFAR-10, under the same initialization scheme.
Figure 4(b) and 4(c) visualize the gradients of different sub- cobponenssn for the 17-th convolutional layer of ResNet-20 during last two growing phases of standard SGD and rate adaptation, respectively, Our rate adaptation mechanism rebalances subcomponents' gradient contributions to appear in lower divergence than global LR, when components are added at different stages and trained for different durations. In Figure 5, we observe that the LR for newly added Subnet-8 (red) in last stage starts around 1.8x x the base LR, then quick adapts to a smoother level. This demonstrates that our method is able to automatically adapt the updates applied to new weights, without any additional local optimization costs (Wu et al., 2020b; Evei al., 2022). All above show method has positive effect in terms of stabilizing training dynamics, which is lost if one attempts to train different subcomponents using global LR scheduler. Appendix A.2 and A.3 provide more visualizations.
8

---

Under review as a conference paper at ICLR 2023
1.
K SEUFA adeet
19.00
Pakee Adaptation factors
3.6
Test Accuracy (96)
1.4
32.33
02.65
50 we -020 - -022)
3.2
92.99
Bertow
Lncer Spae (v
Fataline
Epochs (Last Stage)
Triin Cost (86]
300
Figure 5: Visualization of our adaptive LR.
Figure Comparison of growing schedules.
Large Model
Ste-BatchSize
BabchSbbbbbbeeee Grow
Large Model
Fiu FiscbatchSikee Grow
Ado BatchSize Grow
1200
Train Time/ Tine/Epoch (6]
43
GPU Memory (MO
1208
800
32
260
Stages
Stages
(a) ResNet-18 GPU memory allocations
ResNet-18 training time
Figure 7: Track of GPU memory allocations and wall clock training time for each growing phase.
Flexible Growing Scheduler. Our growing scheduler gains the flexibility to explore the best trade- off' between training budgets and test performance in a unified configuration scheme (Eq, 6 and Eq. 7). We compare the exponential epoch scheduler (p: E {0.2, 0.25, 0.3. 0.35}) to a linear one (p: 0) in ResNet-22 growing on CIFAR-10, denoted as 'Exp." and 'Linear' baselines in Figure Both baselines use the architectural schedulers with P. E {0.2, 0.25, 0.3, 0.35], each generates trade-offs between train costs and test accuracy by alternating To. The exponential scheduler yields better overall trade-offs than the linear with the same P.. In addition different growing schedulers, we also plot a baseline for fixed-size training with different models. Growing methods with both schedulers consistently outperforms the fixed-size baselines, demonstrating that the regularization effect of network growth benefits generalization performance.
Wall-clock Training Speedup. We benchmark GPU memory consumption and wall-clock training time on CIFAR-100 for each stage during training on single NVIDIA 2080Ti GPU. The large baseline ResNet-18 trains for 140 minutes achieve 78.36% accuracy. As shown in the green bar of Figure 7(b), the growing method only achieves marginal wall-clock acceleration, under the same fixed batch size. As such, the growing ResNet-18 takes 120 minutes to achieve 78.12% accuracy. The low GPU utilization in the green bar in Figure 7(a) hinders FLOPs savings from translating into real-world training acceleration. In contrast, the red bar of Figure shows that our batch size adaptation results in large proportion of wall clock acceleration by filling the GPU memory, and corresponding parallel execution resources, while maintaining test accuracy. ResNet- 188 trains for 84 minutes (1.7x speedup) and achieves 78.01% % accuracy.
5 CONCLUSION
We propose an efficient and accurate method for network growing, based on principled rules regarding both parameterization and optimization. Our parameter transition from older to newer subnetworks is general and quick to execute when expanding the network. Our carefully designed learning rate adap- tation mechanism improves optimization dynamics in networks consisting of subcomponents with heterogeneous training durations. Applications to widely-used architectures image classification and machine translation tasks demonstrate that our method bests the accuracy of competitors, even outperforming the original fixed-size models, while saving considerable training cost.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical varinnar trade-off. Proceedings of the National Academy of Sciences, 116632):15849-1588 1584,, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voon Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Mauro Cettolo. Jan Nichues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, 2014.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587, 2017.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2016.
Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. Nest: A neural network synthesis tool based on grow-and-prune paradigm. IEEE Trans. Computers, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In ICLR Workshop, 2018
Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. Gradmax: Growing neural networks using gradient information. In ICLR, 2022.
Golnaz Ghiasi, Tsung- Lin, Ruoming Pang, and Quoc Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. arXiv: 1904.07392 2019.
Boris Ginsburg. Igor Gitman, and Yang You. Large batch training of convolutional networks with layer-wise adaptive rate scaling, 2018
Ross B. Girshick. Fast R-CNN. In ICCV, 2015.
Sam Gross and Michael Wilber. Training and investigating residual nets. http://torch.cht blog/2016/02/ //02///2///// /ttp///////eeeeee/// htm1,, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016a.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016b.
Samuel Horváth, Aaron Klein, Peter Richtárík, and Cédric Archambeau. Hyperparameter transfer learning with adaptive complexity. In AISTATS, 2021.
Andrew G. Howard. Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: /704.0486/ 2017.
10

---

Under review as a conference paper at ICLR 2023
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In NeurlPS, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. http: httt:://ww... cs. toronto, http//kkii//cca
htmm,, 2014.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv: 2016.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv preprint arXiv:194.000310 2019.
Chenxi Liu, Liang-Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019a.
Qiang Liu, Wu Lemeng, and Wang Dilin. Splitting steepest descent for growing neural architectures. 2019b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, 2016.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR, 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. ACL, 2002.
Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyperpa- rameter transfer learning. In NeurIPS, 2018
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire. Domain-indeppndeen dominance of adaptive methods. In CVPR, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9, 2015.
Tijmen Tieleman, Geoffrey Hinton, al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Chengeheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire. Orthogonalized SGD and nested architectures for anytime neural networks. In ICMI, 2020.
11

---

Under review as a conference paper at ICLR 2023
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In JCML, 2016.
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. Autogrow: Automatic layer growing in deep convolu- tional networks. In KDD, 2020.
Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), NeurlPS, 2020a.
Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture opti- mization: Escaping local optimum with signed neural splitting. arXiv preprint arXiv:2003.10392, 2020b.
Greg Yang and Edward Hu. Tensor programs IV: feature learning in infinite-widht neural networks. In ICML, 2021.
Greg Yang. Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), NeurlPS, 2021.
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In AISTATS, 2014.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. ICLR, 2020.
Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In ICLR, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 2016.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Table 6: Rate Adaptation Rule for Adam (Kingma & Ba, 2015) and AvaGrad (Savarese et al., 2021).
Our LR Adaptation
Adam
0-th Stage
Pejo
i-th Stage
mald [0
[xjs]
Pe[t-]]
AvaGrad
0-th Stage
[ijoo//
?ejo]
/djo
12
Majo
i-th Stage
This Wet
(maj) (m((-i))
[Tee 1aaps-1y/// 4a(s
defs
111l2
Table 7: Generalize Adam and AvaGrad for ResNet-20 CIFAR-10.
Optimizer
Training Method
Preserve Opt. Buffer
Train Cost (%)
Test Acc. (%)
Adam
Large fixed-size
NA
100
92.29
Adam
Growing
No
54.90
91.44
Adam
Growing
Yes
54.90
91.61
Adam+tour RA.
Growing
Yes
54.90
92.13
AvaGrad
Large fixed-size
NÄ
TO0
92.45
AvaGrad
Growing
No
54.90
90.71
AvaGrad
Growing
Yes
54.90
91.27
AvaGrad+our RA.
Growing
Yes
54.90
91.72
A.1 GENERALIZATION TO OTHER OPTIMIZERS
We generalize LR adaptation rule Adam (Kingma & Ba, 2015) and AvaGrad (Savarese et al., 2021) in Table 6. Both methods are adaptive optimizers where different heuristics are adopted to derive a parameter-wee
learning rate strategy, which provides primitives that can be extended using our stage-wise adaptation principle for network growing. For example, vanilla Adam adapts the global learning rate with running estimates of the first moment m, and the second moment ei the gradients, where the number of global training steps is an integer when training fixed-size model. When growing networks, our learning rate adaptation instead considers a vector t which tracks each subcomponent's 'age" (i.e. number of steps it has been trained for). As such, for newly grown subcomponent at stage 0, t[]] starts as and the learning rate is adapted from m./ /v, (global) metg (Mafe11) (stage-wise). Similarly, also generalize our approach to AvaGrad by adopting Vorg [e]e-1]]
M,, de, m. of the original paper as stage-wise variables.
Preserving Optimizer State/Buffer Essential to adaptive methods training-time statistics (e.g. running averages my and v2 in Adam) which are stored as buffers and used to compute parameter- wise learning rates. Different from fixed-size models, parameter sets expanded when growing networks, which in practice requires re- re-instantiating a new optimizer at each growth step. Given that our initialization scheme maintains functionality of the network, we are also able preserve and inherit buffers from previous states, effectively maintaining the optimizer's 's state intact when adding new parameters. We investigate the effects this state preservation experimentally.
Results with Adam and AvaGrad Table 7 shows the results growing ResNet-20 on CIFAR-10 10 with Adam and Avagrad. For the large, fixed-size baseline, we train Adam with 1r 0.1, AvaGrad with Ir 0.5, c 10.0, which yields the best results for ResNet-20 following Savarese et al. (2021). We consider different settings for comparison, (1) optimizer without buffer preservation: the buffers are refreshed at each new growing phase (2) optimizer with buffer preservation: the buffer/state inherited from the previous phase, hence being preserved at growth steps (3) optimizer with buffer and rate adaptation (RA): applies our rate adaptation strategy described in Table 6 while also preserving internal state/buffers. We observes that (1) consistently underperforms (2), which suggests that preserving the state/buffers in adaptive optimizers is crucial when growing
and
13

---

Under review as a conference paper at ICLR 2023
Table 8: Comparisons among Standard SGD, LARS and Ours for ResNet-20 Growth CIFAR-10.
Optimizer
Test Acc. (%)
Standard SGD
91.95 + 0.09
SGD with Layer-wise Adapt.(LARS)
91.32 = 0.11
Ours
92.53 + 0.11
networks. (3) bests the other settings for both Adam and AvaGrad, indicating that our rate adaptation strategy generalizes effectively to Adam and AvaGrad for the growing scenario. Together, these also demonstrate that our method has the flexibility to incorporate different statisties that are tracked and used by distinct optimizers, where we take Adam and AvaGrad as examples. Finally, our proposed stage-wise rate adaptation strategy can be employed to virtually any optimizer.
Comparison with Layer-wise Adaptive Optimizer We also consider LARS (Ginsburg et al., 2018; You et al., 2020), layer-wise adaptive variant of SGD, to compare different adap- tation concepts: layer-wise versus layer + stage-wise (ours). Note that although LARS was originally designed for training with large batches, adopt batch size of 128 when growing ResNet-20 on CIFAR-IO. We search the initial learning rate (LR) for LARS over (le-3, 2e-3, 5e-3, le-2, 2c-2, 5e-2, 2o-1,2e-1, 5e-1) and observe that value of 0.02 yields the best results. We adopt the default initial learning rate of 0.1 for both standard SGD and our method. As shown in Table 8, LARS underperforms both standard SGD and our adaptation strategy, suggesting that layer-wise learning rate adaptation by itself - i... without accounting for stage-wise discrepancies not sufficient for successful growing of networks.
A.2 MORE VISUALIZATIONS ON RATE ADAPTATION
We show additional plots of stage-wise rate adaptation when growing ResNet-20 on CIFAR-10 10. Figure 8 shows the of adaptation factors based on the LR the seed architecture from 1st to 8th stages (the stage index starts at 0). We see an overall trend that for newly-added weights, its learning rate starts at > 1 x of the base LR then quickly adapts relatively stable level. This demonstrates that our approach is able to efficiently and automatically adapt new weights to gradually and smoothly fade in throughout the current stage's optimization procedure.
Aleil
Se Sapel
Bodd Pagu
Bodo 20 Supt
(a) 1-st Stage
(b) 2-nd Stage
(c) 3-rd Stage
(d) 4-th Stage
cotaeaaiin Facter
Tpate Sapal
Bodds Dagi
Stgel
(e) 5-th Stage
(6) 6-th Stage
(g) 7-th Stage
(h) 8-th Stage
Figure 8: Visualization of Rate Adaptation Factor Dynamics across All Growing Stages (except 0-th)
14

---

Under review as a conference paper at ICLR 2023
A.3 MORE VISUALIZATIONS ON SUB- COMPONNNNN GRADIENTS
We further compare global LR and our rate adaptation by showing additional visualizations of sub- component gradients of different layers and stages when growing ResNet-20 on CIFAR-IO. We select the 2nd
convolutional layers and plot the gra- dients of each sub-component at the 3rd and 5th growing stages, respectively, in Figures 10, 11, 12. These demonstrate that our rate adaptation strategy is able to re-balance and stabilize the gradient's s contribution of different subcomponents, hence improving the training dynamics compared global scheduler.
-convl) and 17th layer3-block2-cone
(a) Using Global Figure 9: Gradients of 2nd conv at 3rd stage.
(b) Using RA
(a) Using Global
(b) Using RA
Figure 10: Gradients of 2nd conv at 5th stage.
(a) Using Global
(b) Using RA
(a) Using Global
(b) Using RA
Figure 11: Gradients of 1 17th conv at 3rd stage. Figure 12: Gradients of 17th conv at 5th stage.
A.4 SIMPLE EXAMPLE ON FULLY-CONNECTED NEURAL NETWORKS
Additionally, train simple calnecco neural network with hidden layers CIFAR-10 10 each hidden layer has 500 neurons and is followed by ReLU activations. The network is has final linear layer with 10 neurons for classification. Note that each CIFAR-10 image is flattened to
(loward
0.2
Siwilig Plus Bonder Stadue sKt)
54
se Accuracy(%)
Train Lo5s
0.4
03
La moll Fanttee Nooatee
Leaming Rate
Learning Rate
(a) Train Loss
(b) Test Accuracy
Figure 13: Results of Simple Fully-connected Neural Network.
15

---

Under review as a conference paper at ICLR 2023
3072-dimensional (32 32 x 3) vector prior to being given input the network. We consider two variants of this baseline network by adopting training epochs (costs) E {25 (002x))) We also grow from a thin architecture to the original one within 10 stages, each stage consisting of epochs, where the number of units of each hidden layer grows from 50 to 100, 150, 500. The total training cost is equivalent the fixed-size one trained for 25 epochs. We train all haselines using SGD, with weight decay set as O and learning rates sweeping over {0.01, 0.02, 0.05, 0.1, 0.2, 0.5]: results are shown in Figure 13()). Compared standard initialization (green), the loss curve given by growing variance transfer (blue) is more similar to the curve of the large baseline- all using standard SGD which is also consistent with the observations when training model of different scales separately (Yang et al., 2021). Rate adaptation (in red) further lowers training loss. Interestingly, we observe in Figure 13(b) that the test accuracy behavior differs from the training loss one given in Figure 13(a), which may suggest that regularization is missing due to, for example, the lack of paramoter-sharing schemes (like CNN) in this fully- cully-connected network.
Table 9: Growing ResNet-18 using Incremental CIFAR- 100.
Progressive Class
Progressive Data
Train Cost (%) + Accuracy (%) f Cost (%) & Accuracy (%) f
Test
Train
Test
Method
Large fixed-size Model
100
76.80
100
76.65
Ours
65.36
76.50
65.36 68.49
76.34
Ous-Dynnmic-OSGG
68.49
77.53
77.85
A.5 EXTENSION TO CONTINUOUSLY INCREMENTAL DATASTREAM
Another direct and intuitive application for our method is to fit continuously incremental datastream where Do C. DI.... C Dy.... C DN The network complexity scales up together with the data so that larger capacity can be trained on more data samples. Orthogonalized SGD (OSGD) (Wan et al., 2020) address the optimization difficulty in this context, which dynamically re- beballacess task-specific gradients via prioritizing the specifie loss influence. We further extend our optimizer by introducing dynamic variant of orthogonalized SGD, which progressively adjusts the priority of tasks on different subnets during network growth.
Suppose the data increases from Dn to Dn, we first accumulate the old gradients Gn-1 using one additional epoch on Dn and then grow the network width. For each batch of D,. we first project gradients of the new architecture (n-th stage), denoted as Gn, onto the parameter subspace that is orthogonal to Grad 1. zero- pero-ppdded version of Gn-1 with desirable shape. The final gradients G are then calculated by re- weightinn the original G, and its orthogonal counterparts:
G = Gn * projerad x-1 (Gn), 111+0
where > is dynamic hyperparameter which weights the original and orthogonal gradients. When A = 1, subsequent outputs do not interfere with earlier directions of parameters updates. We then anneal that the newly-introduced data and subnetwork can smoothly fade in throughout the training procedure.
Implementation Details. We implement the task in two different settings, denoted as* progressive class' and 'progressive data' on CIFAR-100 100 dataset within 9 stages. In the progressive class setting, we first randomly select 20 classes in the first stage and then add 10 new classes each growing stage. In the progressive data setting, we sequentially sample a fraction of the data with replacement for each stage, i.e. 20%, 30%, ..., 100%.
ResNet-18 on Continuous CIFAR-100: We evaluate our method on continuous datastreams by growing a ResNet-18 on CIFAR- and comparing the final test accuracies. As shown in Ta- ble 9, compared with the large baseline, our growing method achieves 1.53x cost savings with slight performance degradation in both settings. The dynamic OSGD variant outperforms the large baseline with 1.46x acceleration, demonstrating that the extension improves the optimization on continuous datastream through gradually re- ce-balancing the task-specific gradients of dynamic networks.
16


Reviews:
[{"review_id": "isf4UEOp4JH", "reviewer": "Reviewer_ePbb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of growing neural networks during training.  The authors propose a parameterization and optimization scheme that pays close attention to weight and gradient scaling while reacting to training dynamics.  They further refine their original proposal to mitigate the problem of newly grafted subnetworks, that are relatively un-trained relative to the existing network, by an adaptive gradient scheme to ensure the contribution to the gradient is more balanced across new and older components.  They provide experimental evidence to show the efficacy of their method, paying particular attention to the savings in training time (or computation budget) accrued relative to larger monolithic networks.\n", "summary_of_the_review": "While the authors have proposed what is likely an advance in model growth, they have been hampered somewhat by the choices they made in their experiments.  \n\nIncremental model growing is at its most useful when growing to very large models (think large language models circa 2021), )that would trouble even industry research labs.  Demonstrating a convincing win here in terms of performance versus compute (and time) saved would represent a more effective contribution, yet they focused much of their experiments on smaller vision models that do not offer a compelling reason to be grown stage-wise.\n\nFor this reason, I think that the paper offers some value, if rather marginal.", "strength_and_weaknesses": "### Strengths\n- The authors have tried hard to present a more principled and stable method for growing networks from smaller to larger capacities, which is an important problem especially in large language models.\n- Figure 5 and 6 are refreshing to see, as is much of section 4.4.  The authors are attempting to help us better understand how each different component of Algorithm 1 works to yield the final results in Tables 1-5.\n- The authors do a good job summarizing and surveying the related work in network growing, and also explain the details of each component of their method in section 3 very clearly, using Figure 2 to help explain the growth stage layer-wise.\n\n### Weaknesses\n\n- The worth of some of the core contributions aren't very clearly established in the paper.  For instance, the value of the variance transfer parameterization achieves comparable performance with competing methods (some gains in table 5, but tables 1-3 show mostly marginal gains over Net2Net and Firefly).  \n- Also, the last claim is too broad.  The authors adaptive method bests the large model baseline for the VGG models.  The phrase \"even outperforming the original fixed-size models\" makes it seem as if this is often the case. \n- In section 4.1, the paragraph describing table 2 is a nice result, but somewhat overstated.  On two models the authors' method outperform the baseline large model.  And on most comparisons (CIFAR 10/100), the gain is on par with Net2Net.  If the authors could find a test where their method was substantially better in test accuracy than Net2Net, it would really bolster their claims that Net2Net’s heuristic splitting method needs improving. Though the Transformer results in Table 5 display a somewhat more substantial difference in BLEU score.\n- Is there a reason that distributions were reported in tables 1 through 3 but not tables 4 and 5? \n- In the description of the results for Figure 3b, Blue is sometimes dominating orange in Figure 3b, but it’s not consistent especially at the lower learning rates (which are more consistent with what are used in practice)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cR1M1RVAq_6", "reviewer": "Reviewer_qzqu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates methods to grow the width of linear neural network layers during training. Their proposed method consists of a variety of techniques:\n\n* The new layer has a repeated block of new weights, which are multiplied by new submatrices in the next layer which only differ in sign, so that the final contribution of these weights is zero (or at least, almost).\n* The old weights are scaled so that their expected variance is as if the weights were initialized using the fan-in of the new layer\n* Throughout training, the learning rates for the new weights are scaled by the ratio of their norm and the norm of the original weights\n* The layer sizes grow exponentially as a function of the growing stages\n* The growing stages have an exponentially growing number of epochs\n* The batch size decreases exponentially\n\nThe suggested method is compared on a variety of vision models (ResNet, VGG, MobileNet) and datasets (CIFAR-10, CIFAR-100, ImageNet) as well as a transformer trained for machine translation. They compare against several other methods (Net2Net, splitting, FireFly, GradMax). The results are encouraging, with the final test accuracy usually being very close to the large model trained from scratch while reducing training cost between 30% and 50%.", "summary_of_the_review": "A paper that introduces some nice ideas, but it tries to do too many things and in the process fails to investigate the hard questions. The resulting paper feels like a long list of experiments and proposals without hypotheses that are tested. I would argue to reject this paper.", "strength_and_weaknesses": "This paper has some interesting ideas: I like the way that the layers are grown. It feels similar to splitting (i.e., duplicating weights and then halving them) while avoiding changing the scale of the weights, which should help with the learning dynamics. The learning rate scaling is also interesting. One of the main problems with growing networks is that it can take a long time for the new capacity to be used. Perhaps this scaling of the learning rate helps with this somehow.\n\nMy main issue with this paper is that it proposes a lot of things (a function-preserving growing method, scaling of weights, learning rate scaling, growing schedules, batch size schedules, etc.). Many of these are not very theoretically grounded, and the ablations are limited. This makes it hard to disentangle which part of the methods are working and why. It gives a feeling of the authors having thrown the kitchen sink at the problem until something stuck.\n\nThe result is that this paper raises many questions for me:\n\n* Is the variance transfer really useful? The effect seems minimal in figure 3. There are several learning rates for which the variance transfer and non-variance transfer curves overlap. Especially given that this is on CIFAR-10 with a 4-layer network, it seems like a bad idea to draw conclusions from these results. Without confidence intervals, figure 4 also is hard to draw definitive conclusions from (e.g., if training would have stopped at epoch 150 instead of 160, the methods would have been indistinguishable).\n* Does variance transfer actually make sense? The scaling of the weights would only make sense if we know that the magnitude of the old weights hasn't changed much during training, but this isn't validated.\n* What is the motivation for the learning rate scaling proposed in table 1? The intuition I would have is that you want to have a bigger learning rate for the new weights, for two reasons: they have to catch up, and the old weights can't be allowed to get too distorted by the initially random signals of the new weights. But the current learning rate scaling actually scales down the learning rate of the new weights when their norm is small, which seems like it might get them stuck? Would $\\max(1, \\lVert W_i \\setminus W_{i-1} \\rVert)$ maybe make more sense?\n* Figure 4(c) shows that the gradient norms are closer together. In fact, the gradient norms of the oldest subnets seem to be lower, suggesting that they are better trained. But figure 5 shows that their weight norms (and hence learning rates) are still higher towards the end of training. This seems counterintuitive to me (usually you reduce the learning rate when you get closer to the optimum). And why is it that the weights of these old subnets stay so high compared to the initial weights? Is that because the variance transfer keeps scaling the old weights down?\n* In figures 4(b), 4(c) and 5 you can see a very common dynamic of growing networks (look at subnets 7 and 8): When new weights are introduced they initially just produce noise, so the first thing the network does is to unlearn them (usually by driving them to zero). After that, the network takes some time to integrate this new capacity, at which point the gradient on these weights becomes similar to the rest of the network. My concern is that the proposed learning rate scheduling is slowing down this process by lowering the learning rate. \n\nI could probably keep going and come up with more questions. (For example, I am wondering if on one hand the scheduling is helping by increasing the learning rate of older subnets compared to newer subnets, but on the other hand hurting by integrating new subnets slower in the network. Or maybe the lower learning rate is actually helping by giving the older network time to adapt to the newer weights before those settle?) My concern is that at the end, this paper provides a few interesting data points, but raises more questions than it answers. To argue for acceptance, I would like to see a paper with more focus (e.g., only the learning rate scheduling, or only the function-preserving growing method) and thoroughly study this through thoughtful ablations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3nGZydjIjq", "reviewer": "Reviewer_3rDh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train neural network models incrementally to reduce training time. To this end it proposes two heuristics, the first being variance transfer, which involves re-parameterizing weights to account for their variance, and second being rate adaptation, which adjust the learning rate of newly added weights different from the weights already present in the model. Experiments show that these strategies result in more accurate neural networks that are trained faster than existing methods for incremental training.", "summary_of_the_review": "I am currently providing a recommendation of reject. Overall, this paper does not really propose any interesting technical insights or ideas, rather it consists of heuristics that nonetheless achieve SOTA results, making it borderline. However, I am willing to raise my score given that the authors are willing to show ablation experiments on realistic CIFAR10 / CIFAR100 models which will show the exact contribution of the various components of the method. \n\n---- Update ----\nBased on the additional experiments showed in the rebuttals, I am changing my recommendation to that of accept. My original review still largely stands, that the paper proposes a collection of heuristics that achieve SOTA results, making it still borderline. However, the strong results with ablation experiments indicate that these techniques may be useful to practitioners interested in this problem. Further, as I mention in my response, I hope that paper tempers down claims of being principled in an update of the draft.", "strength_and_weaknesses": "**Strengths**:\n\n+ The paper tackles an important problem of accelerating training via incrementally growing neural networks, and a good solution to this problem can have huge implications for large-scale training of neural networks.\n\n+ The heuristics proposed in this paper, i.e., variance transfer and rate adaptation seem like reasonable approaches, and the experimental results (esp those on Imagenet) agree with that observations.\n\n\n**Weaknesses**:\t\n\n**Proposed methods are heuristics but claimed otherwise; lacking justification** \n\n- The title and several other parts of the paper make the claim that the proposed methods are principled. This seems to be because the methods are inspired from the scalings used in the mutransfer paper (Yang et al., 2021). However, it is worth stressing that a principled strategy in one setting does not immediately imply it being a principled strategy in another setting without appropriate justification, as I elaborate in the next two points. I thus request the authors to remove claims regarding the principled nature of proposed methods. \n\n- The justification for variance transfer is \"to make the weights of the old subnetwork compatible with the entire weight tensor parameterization\", where an assumption is made that the old weights are drawn from a gaussian distribution with std = (1 / h_out^2). However, this is true only at initialization, and the distribution of weights can diverge in principle during training. Question: Why use the variance at initialization as opposed to the empirical variance at the time of growing the model? What is the precise justification for variance transfer in the first place?\n\n- The learning rate adaptation method has no justification provided (even an intuitive one) and is purely a heuristic. Suggestion: Perhaps some intuition can be obtained by analyzing the dynamics of a simple one hidden layer MLP model?\n\n- Missing detail: How are SGD momentum vectors handled at the time of growing a model? Are they re-initialized from scratch for each stage of re-training, or are they carried over from previous stages of optimization?\n\n**Missing ablations on realistic models**\n\n- While the analysis experiments in Section 4.4 are appreciated, it is important to have ablations presented for the more realistic Resnet-20 / Resnet-18 / Mobilenetv1 models used in Tables 1 and 2. In particular, it would be interesting to compare the following baselines - (1) Growing method + LR adaptation (LRA) + Variance transfer (VT) (aka the \"full\" method, already presented), (2) Method + LRA only, (3) Method + VT only, (4) Method only, which involves simply adding the new rows / columns to weights according to the prescribed schedule. In the absence of principled justification, these thorough ablations are necessary to evaluate the usefulness of the method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject