Title: Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: network growing

efficient network training
Abstract: We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics.  Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network.  To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents.  Experiments show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original training computation budget.  We demonstrate that these gains translate into real wall-clock training speedups.

Accelerated Training via Incrementally Growing Neural Networks using

Abstract

We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network. To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents. Experiments show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original training computation budget. We demonstrate that these gains translate into real wall-clock training speedups.

1 Introduction

Modern neural network design typically follows a 'larger is better' rule of thumb, with models consisting of millions of parameters achieving impressive generalization performance across many tasks, including image classification [22, 32, 30, 46], object detection [13, 26, 11], semantic segmentation [27, 3, 24] and machine translation [34, 7]. Within a class of network architecture, deeper or wider variants of a base model typically yield further improvements to accuracy. Residual networks (ResNets) [15] and wide residual networks [45] illustrate this trend in convolutional neural network (CNN) architectures. Dramatically scaling up network size into the billions of parameter regime has recently revolutionized transformer-based language modeling [34, 7, 1].

The size of these models imposes prohibitive training costs and motivates techniques that offer cheaper alternatives to select and deploy networks. For example, hyperparameter tuning is notoriously expensive as it commonly relies on training the network multiple times, and recent techniques aim to circumvent this by making hyperparameters transferable between models of different sizes, allowing them to be tuned on a small network prior to training an original large model once [41].

Our approach incorporates these ideas, but extends the scope of transferability to include the parameters of the model itself. Rather than view training small and large models as separate events, we grow a small model into a large one through many intermediate steps, each of which introduces additional parameters to the network. Our contribution is to do so in a manner that preserves the function computed by the model at each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, we use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training a large static model from scratch.

𝜼

𝟎

𝜼

𝟎

𝜼

𝟎

𝜼

𝟎

𝜽

𝜸

𝜸/𝟐

𝜸/𝟐

𝜽 + 𝝐

𝟏

𝜽 + 𝝐

𝟐

𝜹

𝝈

Split / Add

Add Neurons

𝜽, 𝜸

:         Old

𝜸/𝟐

:        Split

𝝈, 𝜹

:   Grad-based  Init.

𝝐

𝟏

, 𝝐

𝟐

:  Small Noise

Parameterization

Optimization

𝜼

𝟎

Global LR

𝜼

𝟎

𝜼

𝟎

𝜼

𝟎

𝜽

𝜸

𝜽′

𝜹 + 𝝐

𝟐

-𝝈

Grow Neurons

𝜼

𝟎

𝜹 + 𝝐

𝟏

+𝝈

𝜸′

𝜼

𝟏

𝜼

𝟏

𝜼

𝟏

𝜼

𝟏

𝜼

𝟎

𝜼

𝟏

Separate LR

𝜽, 𝜸

:      Old

𝜽

#

, 𝜸

#

: Var. Transfer

𝝈, 𝜹

:   Grad-Free Init.

𝝐

𝟏

, 𝝐

𝟐

:    Small Noise

Parameterization

Optimization

Grow

(a) Existing Methods: Splitting Init, Global LR

(b) Ours: Function-Preserving Init, Stagewise LR

Competing recent efforts to grow deep models from simple architectures [4, 23, 5, 25, 39, 37, 38, 44, 10] draw inspiration from other sources, such as the progressive development processes of biological brains. In particular, Net2Net [4] grows the network by randomly splitting learned neurons from previous phases. This replication scheme, shown in Figure 1(a) is a common paradigm for most existing methods. Gradient-based methods [38, 39] determine which neurons to split and how to split them by solving a combinatorial optimization problem with auxiliary variables.

At each growth step, naive random initialization of new weights destroys network functionality and may overwhelm any training progress. Weight rescaling with a static constant from a previous step is not guaranteed to be maintained as the network architecture evolves. Gradient-based methods outperform these simple heuristics but require additional training effort in their parameterization schemes. Furthermore, all existing methods use a global LR scheduler to govern weight updates, ignoring the discrepancy among subnetworks introduced in different growth phases. The gradient itself and other parameterization choices may influence the correct design for scaling weight updates.

We develop a growing framework around the principles of enforcing transferability of parameter settings from smaller to larger models (extending [41]), offering functional continuity, smoothing optimization dynamics, and rebalancing learning rates between older and newer subnetworks. Figure 1(b) illustrates key differences with prior work. Our core contributions are:

Parameterization using Variance Transfer: We propose a parameterization scheme accounting for the variance transition among networks of smaller and larger width in a single training process. Initialization of new weights is gradient-free and requires neither additional memory nor training.

Improved Optimization with Learning Rate Adaptation: Subnetworks trained for different lengths have distinct learning rate schedules, with dynamic relative scaling driven by weight norm statistics.

Better Performance and Broad Applicability: Our method not only trains networks fast, but also yields excellent generalization accuracy, even outperforming the original fixed-size models. Flexibility in designing a network growth curve allows choosing different trade-offs between training resources and accuracy. Furthermore, adopting an adaptive batch size schedule provides acceleration in terms of wall-clock training time. We demonstrate results on image classification and machine translation tasks, across various network architectures.

2 Related Work

Network Growing. A diverse range of techniques train models by progressively expanding the network architecture [36, 9, 5, 37, 44]. Within this space, the methods of [4, 25, 39, 38, 10] are most relevant to our focus - incrementally growing network width across multiple training stages. Net2Net [4] proposes a gradient-free neuron splitting scheme via replication, enabling knowledge transfer from previous training phases; initialization of new weights follows simple heuristics. Liu et al. 's splitting approach [25] derives a gradient-based scheme for duplicating neurons by formulating a combinatorial optimization problem. FireFly [38] gains flexibility by also incorporating brand new neurons. Both methods improve Net2Net's initialization scheme by solving an optimization problem with auxiliary variables, at the cost of extra training effort. GradMax [10], in consideration of training dynamics, performs initialization via solving a singular value decomposition (SVD) problem.

Neural Architecture Search (NAS) and Pruning. Another subset of methods mix growth with dynamic reconfiguration aimed at discovering or pruning task-optimized architectures. Network Morphism [36] searches for efficient networks by extending layers while preserving the parameters. AutoGrow [37] takes an AutoML approach governed by heuristic growing and stopping policies. Yuan et al. [44] combine learned pruning with a sampling strategy that dynamically increases or decreases network size. Unlike these methods, we focus on the mechanics of growth when the target architecture is known, addressing the question of how to best transition weight and optimizer state to continue training an incrementally larger model. NAS and pruning are orthogonal to, though potentially compatible with, the technical approach we develop.

Hyperparameter Transfer. Multiple works [42, 29, 16] explore transferable hyperparameter (HP) tuning. The recent Tensor Program (TP) work of [40] and [41] focuses on zero-shot HP transfer across model scale and establishes a principled network parameterization scheme to facilitate HP transfer. This serves as an anchor for our strategy, though, as Section 3 details, modifications are required to account for dynamic growth.

Learning Rate Adaptation. Surprisingly, the existing spectrum of network growing techniques utilize relatively standard learning rate schedules and do not address potential discrepancy among subcomponents added at different phases. While general-purpose adaptive optimizers, e.g. , AdaGrad [8], RMSProp [33], Adam [20], or AvaGrad [31], might ameliorate this issue, we choose to explicitly account for the discrepancy. As layer-adaptive learning rates (LARS) [12, 43] benefit in some contexts, we explore further learning rate adaption specific to both layer and growth stage.

3 Method

3.1 Parameterization and Optimization with Growing Dynamics

Functionality Preservation . We grow a network's capacity by expanding the width of computational units ( e.g., hidden dimensions in linear layers, filters in convolutional layers). To illustrate our scheme, consider a 3-layer fully-connected network with ReLU activations ϕ at a growing stage t :

where x ∈ R C x is the network input, y t ∈ R C y is the output, and u t ∈ R C u t , h t ∈ R C h t are the hidden activations. In this case, W x t is a C u t × C x matrix, while W u t is C h t × C u t and W h t is C y × C h t . Our growing process operates by increasing the dimensionality of each hidden state, i.e., from C u t and C h t to C u t +1 and C h t +1 , effectively expanding the size of the parameter tensors for the next growing stage t +1 . The layer parameter matrices W t have their shapes changed accordingly and become W t +1 . Figure 2 illustrates the process for initializing W t +1 from W t at a growing step. 1

Following Figure 2(a), we first expand W x t along the output dimension by adding two copies of new weights V x t of shape C u t +1 -C u t 2 × C x , generating new features ϕ ( V x t x ) . The first set of activations become

where concat denotes the concatenation operation. Next, we expand W u t across both input and output dimensions, as shown in Figure 2(b). We initialize new weights Z u t of shape C h t × C u t +1 -C u t 2 and add to W u t two copies of it with different signs: + Z u t and -Z u t . This preserves the output of the layer since ϕ ( W u t u t + Z u t ϕ ( V x t x ) + ( -Z u t ) ϕ ( V x t x )) = ϕ ( W u t u t ) = h t . We then add two copies of new weights V u t , which has shape C h t +1 -C h t 2 × C u t +1 , yielding activations

We similarity expand W h t with new weights Z h t to match the dimension of h t +1 , as shown in Figure 2(c). The network's output after the growing step is:

which preserves the original output features in Eq. 1. Appendix B provides illustrations for more layers.

(c) Output Layer

Weights Initialization with Variance Transfer (VT). Yang et al. [41] investigate weight scaling with width at initialization, allowing hyperparameter transfer by calibrating variance across model size. They modify the variance of output layer weights from the commonly used 1 fan in to 1 fan 2 in . We adopt this same correction for the added weights with new width: W h and Z h are initialized with variances of 1 C h t 2 and 1 C h t +1 2 .

However, this correction considers training differently-sized models separately, which fails to accommodate the training dynamics in which width grows incrementally. Assuming that the weights of the old subnetwork follow W h t ∼ N (0 , 1 C h t 2 ) (which holds at initialization), we make them compatible with new weight tensor parameterization by rescaling it with the fan in ratio as W h ′ t = W h t · C h t C h t +1 . See Table 1 (top). Appendix A provides detailed analysis.

This parameterization rule transfers to modern CNNs with batch normalization (BN). Given a weight scaling ratio of c , the running mean µ and variance σ of BN layers are modified as cµ and c 2 σ , respectively.

Stage-wise Learning Rate Adaptation (LRA). Following [41], we employ a learning rate scaling factor of ∝ 1 fan in on the output layer when using SGD, compensating for the initialization scheme. However, subnetworks from different growth stages still share a global learning rate, though they have trained for different lengths. This may cause divergent behavior among the corresponding weights, making the training iterations after growing sensitive to the scale of the newly-initialized weights. Instead of adjusting newly added parameters via local optimization [38, 10], we govern the update of each subnetwork in a stage-wise manner.

Let W t denote the parameter variables of a layer at a growth stage t , where we let W t and W ′ t correspond to the same set of variables such that W t +1 \ W t denotes the new parameter variables whose values are initialized with Z t and V t . Moreover, let W ∆ k and G ∆ k denote the values and gradients of W k \ W k -1 . We adapt the learning rate used to update each sub-weight W ∆ k , for

0 ≤ k ≤ t , as follows:

where η 0 is the base learning rate, f is a function that maps subnetworks of different stages to corresponding train-time statistics, and W ∆0 are the layer's parameter variables at the first growth stage. Table 1 (bottom) summarizes our LR adaptation rule for SGD when f is instantiated as weight norm, providing an stage-wise extension to the layer-wise adaptation method LARS [12], i.e., LR ∝ || W || . Alternative heuristics are possible; see Appendices C and D.

3.2 Flexible and Efficient Growth Scheduler

We train the model for T total epochs by expanding the channel number of each layer to C final across N growth phases. Existing methods [25, 38] fail to derive a systemic way for distributing training resources across a growth trajectory. Toward maximizing efficiency, we experiment with a coupling between model size and training epoch allocation.

Algorithm 1 : Growing using Var. Transfer and Learning Rate Adapt. with Flexible Scheduler

Architectural Scheduler. We denote initial channel width as C 0 and expand exponentially:

where ⌊·⌉ 2 rounds to the nearest even number and p c is the growth rate between stages.

Epoch Scheduler. We denote number of epochs assigned to t -th training stage as T t , with ∑ N -1 t =0 T t = T total . We similarly adapt T t via an exponential growing scheduler:

Wall-clock Speedup via Batch Size Adaptation. Though the smaller architectures in early growth stages require fewer FLOPs, hardware capabilities may still restrict practical gains. When growing width, in order to ensure that small models fully utilize the benefits of GPU parallelism, we adapt the batch size along with the exponentially-growing architecture in a reverse order:

where B base is the batch size of the large baseline model. Algorithm 1 summarizes our full method.

4 Experiments

We evaluate on image classification and machine translation tasks. For image classification, we use CIFAR-10 [21], CIFAR-100 [21] and ImageNet [6]. For the neural machine translation, we use the IWSLT'14 dataset [2] and report the BLEU [28] score on German to English (De-En) translation.

Large Baselines via Fixed-size Training. We use VGG-11 [32] with BatchNorm [19], ResNet20 [15], MobileNetV1 [17] for CIFAR-10 and VGG-19 with BatchNorm, ResNet-18, MobileNetV1 for CIFAR-100. We follow [18] for data augmentation and processing, adopting random shifts/mirroring and channel-wise normalization. CIFAR-10 and CIFAR-100 models are trained for 160 and 200 epochs respectively, with a batch size of 128 and initial learning rate (LR) of 0.1 using SGD. We adopt a cosine LR schedule and set the weights decay and momentum as 5 e4 and 0 . 9 . For ImageNet, we train the baseline ResNet-50 and MobileNetV1 [17] using SGD with batch sizes of 256 and 512, respectively. We adopt the same data augmentation scheme as [14], the cosine LR scheduler with initial LR of 0.1, weight decay of 1 e4 and momentum of 0 . 9 .

For IWSLT'14, we train an Encoder-Decoder Transformer (6 attention blocks each) [34]. We set width as d model = 1 / 4 d ffn = 512 , the number of heads n head = 8 and d k = d q = d v = d model /n head = 64 . We train the model using Adam for 20 epochs with learning rate 1 e3 and ( β 1 , β 2 ) = (0 . 9 , 0 . 98) . Batch size is 1500 and we use 4000 warm-up iterations.

Implementation Details. We compare with the growing methods Net2Net [4], Splitting [25], FireFlysplit, FireFly [38] and GradMax [10]. In our method, noise for symmetry breaking is 0.001 to the norm of the initialization. We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers ( e.g., Adam, AvaGrad).

For image classification, we run the comparison methods except GradMax alongside our algorithm for all architectures under the same growing scheduler. For the architecture scheduler, we set p c as 0.2 and C 0 as 1/4 of large baselines in Eq. 6 for all layers and grow the seed architecture within N = 9 stages towards the large ones. For epoch scheduler, we set p t as 0 . 2 , T 0 as 8 , 10 , and 4 in Eq. 7 on CIAFR-10, CIFAR-100, and ImageNet respectively. Total training epochs T total are the same as the respective large fixed-size models. We train the models and report the results averaging over 3 random seeds.

For machine translation, we grow the encoder and decoder layers' widths while fixing the embedding layer dimension for a consistent positional encoding table. The total number of growing stages is 4, each trained for 5 epochs. The initial width is 1/8 of the large baseline ( i.e., d model = 64 and d k,q,v = 8 ). We set the growing ratio p c as 1.0 so that d model evolves as 64, 128, 256 and 512.

Wetrain all the models on an NVIDIA 2080Ti 11GB GPU for CIFAR-10, CIFAR-100, and IWSLT'14, and two NVIDIA A40 48GB GPUs for ImageNet.

4.1 CIFAR Results

All models grow from a small seed architecture to the full-sized one in 9 stages, each trained for { 8 , 9 , 11 , 13 , 16 , 19 , 23 , 28 , 33 } epochs ( 160 total) on CIFAR-10, and { 10 , 12 , 14 , 17 , 20 , 24 , 29 , 35 , 39 } ( 200 total) on CIFAR-100. Net2Net follows the design of growing by splitting via simple neuron replication, hence achieving the same training efficiency as our gradientfree method under the same growing schedule. Splitting and Firely require additional training effort for their neuron selection schemes and allocate extra GPU memory for auxiliary variables during the local optimization stage. This is computationally expensive, especially when growing a large model.

ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10. Table 2 shows results in terms of test accuracy and training cost calculated based on overall FLOPs. For ResNet-20, Splitting and Firefly achieve better test accuracy than Net2Net, which suggests the additional optimization benefits neuron selection at the cost of training efficiency. Our method requires only 54 . 9% of the baseline training cost and outperforms all competing methods, while yielding only 0 . 09 p.p (percentage points) performance degradation compared to the static baseline. Moreover, our method even outperforms the large fixed-size VGG-11 by 0 . 20 p.p test accuracy, while taking only 52 . 91% of its training cost. For MobileNetV1, our method also achieves the best trade-off between training efficiency and test accuracy among all competitors.

ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100. We also evaluate all methods on CIFAR100 using different network architectures. Results in Table 3 show that Firely consistently achieves better test accuracy than Firefly-split, suggesting that adding new neurons provides more flexibility for exploration than merely splitting. Both Firely and our method achieve better performance than the original VGG-19, suggesting that network growing might have an additional regularizing effect. Our method yields the best accuracy and largest training cost reduction.

4.2 ImageNet Results

We first grow ResNet-50 on ImageNet and compare the performance of our method to Net2Net and FireFly under the same epoch schedule: { 4 , 4 , 5 , 6 , 8 , 9 , 11 , 14 , 29 } ( 90 total) with 9 growing phases. We also grow MobileNetV1 from a small seed architecture, which is more challenging than ResNet-50. We train Net2Net and our method uses the same scheduler as for ResNet-50. We also compare with Firefly-Opt (a variant of FireFly) and GradMax and report their best results from [10]. Note that both methods not only adopt additional local optimization but also train with a less efficient growing scheduler: the final full-sized architecture needs to be trained for a 75% fraction while ours only requires 32 . 2% . Table 4 shows that our method dominates all competing approaches.

4.3 IWSLT14 De-En Results

We grow a Transformer from d model = 64 to d model = 512 within 4 stages, each trained with 5 epochs using Adam. Applying gradient-based growing methods to the Transformer architecture is nontrivial due to their domain specific design of local optimization. As such, we only compare with Net2Net. We also design variants of our method for self-comparison, based on the adaptation rules for Adam in Appendix C. As shown in Table 5, our method generalizes well to the Transformer architecture.

4.4 Analysis

Ablation Study. We show the effects of turning on/off each of our modifications to the baseline optimization process of Net2Net (1) Growing: adding neurons with functionality preservation. (2) Growing+VT: only applies variance transfer. (3) Growing+RA: only applies LR rate adaptation. (4) Full method. We conduct experiments using both ResNet-20 on CIFAR-10 and ResNet-18 on CIFAR-100. As shown in Table 6, different variants of our growing method not only outperform Net2Net consistently but also reduce the randomness (std. over 3 runs) caused by random replication. We also see that, both RA and VT boost the baseline growing method. Both components are designed and woven to accomplish the empirical leap. Our full method bests the test accuracy.

2

6

2

4

2

2

2 0

Learning Rate

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Train Loss

Large Standard(2.3x Cost)

Large Standard(1x Cost)

Large with

Transfer(2.3x Cost)

Large with

Transfer(1x Train Cost)

Growing(Standard)

Growing(Ours Var. Transfer+Standard SGD)

Growing(Ours Var. Transfer+SGD with Rate Adaptation)

2

6

2

4

2

2

2 0

Learning Rate

55

60

65

70

75

80

85

Test Accuracy(%)

Large Standard(2.3x Cost)

Large Standard(1x Cost)

Large with

Transfer(2.3x Cost)

Large with

Transfer(1x Train Cost)

Growing(Standard)

Growing(Ours Var. Transfer+Standard SGD)

Growing(Ours Var. Transfer+SGD with Rate Adaptation)

(a) Train Loss

(b) Test Accuracy

130

135

140

145

150

155

160

Epochs(Last Stage)

88

89

90

91

92

Test Accuracy (%)

Standard Init. + SGD

Var. Transfer + Standard SGD

(a) Test Accuracy

7-th Stage

8-th Stage

(b) Global LR

7-th Stage

8-th Stage

(c) Rate Adaptation

Justification for Variance Transfer. We train a simple neural network with 4 convolutional layers on CIFAR-10. The network consists of 4 resolution-preserving convolutional layers; each convolution has 64, 128, 256 and 512 channels, a 3 × 3 kernel, and is followed by BatchNorm and ReLU activations. Max-pooling is applied to each layer for a resolution-downsampling of 4, 2, 2, and 2. These CNN layers are then followed by a linear layer for classification. We first alternate this network into four variants, given by combinations of training epochs ∈ { 13(1 × ) , 30(2 . 3 × ) } and initialization methods ∈ { standard, µ transfer [41] } . We also grow from a thin architecture within 3 stages, where the channel number of each layer starts with only 1/4 of the original one, i.e., { 16 , 32 , 64 , 128 } → { 32 , 64 , 128 , 256 } → { 64 , 128 , 256 , 512 } , each stage is trained for 10 epochs.

For network growing, we compare the baselines with standard initialization and variance transfer. We train all baselines using SGD, with weight decay set as 0 and learning rates sweeping over { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 5 , 2 . 0 } . In Figure 3(b), growing with Var. Transfer (blue) achieves overall better test accuracy than standard initialization (orange). Large baselines with merely µ transfer in initialization consistently underperform standard ones, which validate that the compensation from the LR rescaling is necessary in [41]. We also observe, in both Figure 3(a) and 3(b), all growing baselines outperform fixed-size ones under the same training cost, demonstrating positive regularization effects. We also show the effect of our initialization scheme by comparing test performance on standard ResNet-20 on CIFAR-10. As shown in Figure 4(a), compared with standard initialization, our variance transfer not only achieves better final test accuracy but also appears more stable. See Appendix F for a fully-connected network example.

Justification for Learning Rate Adaptation. We investigate the value of our proposed stage-wise learning rate adaptation as an optimizer for growing networks. As shown in the red curve in Figure 3, rate adaptation not only bests the train loss and test accuracy among all baselines, but also appears to be more robust over different learning rates. In Figure 4(a), rate adaptation further improves final test accuracy for ResNet-20 on CIFAR-10, under the same initialization scheme.

Figure 4(b) and 4(c) visualize the gradients of different sub-components for the 17-th convolutional layer of ResNet-20 during last two growing phases of standard SGD and rate adaptation, respectively. Our rate adaptation mechanism rebalances subcomponents' gradient contributions to appear in lower divergence than global LR, when components are added at different stages and trained for different durations. In Figure 5, we observe that the LR for newly added Subnet-8 (red) in last stage starts around 1 . 8 × the base LR, then quick adapts to a smoother level. This demonstrates that our method is able to automatically adapt the updates applied to new weights, without any additional local optimization costs [39, 10]. All above show our method has a positive effect in terms of stabilizing training dynamics, which is lost if one attempts to train different subcomponents using a global LR scheduler. Appendix D provides more analysis.

Flexible Growing Scheduler. Our growing scheduler gains the flexibility to explore the best tradeoffs between training budgets and test performance in a unified configuration scheme (Eq. 6 and Eq. 7). We compare the exponential epoch scheduler ( p t ∈ { 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 } ) to a linear one ( p t = 0 ) in ResNet-20 growing on CIFAR-10, denoted as 'Exp.' and 'Linear' baselines in Figure 6. Both baselines use the architectural schedulers with p c ∈ { 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 } , each generates trade-offs between train costs and test accuracy by alternating T 0 . The exponential scheduler yields better

130

135

140

145

150

155

160

Epochs (Last Stage)

1.0

1.2

1.4

1.6

1.8

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

Subnet-6

Subnet-7

Subnet-8

Better

0

1

2

3

4

5

6

7

8

Stages

0

200

400

600

800

1000

1200

Large Model

Fix-BatchSize Grow

Ada-BatchSize Grow

(a) GPU memory allocations

0

1

2

3

4

5

6

7

8

Stages

0

10

20

30

40

Train Time/Epoch (s)

Large Model

Fix-BatchSize Grow

Ada-BatchSize Grow

(b) Training time

overall trade-offs than the linear one with the same p c . In addition to different growing schedulers, we also plot a baseline for fixed-size training with different models. Growing methods with both schedulers consistently outperform the fixed-size baselines, demonstrating that the regularization effect of network growth benefits generalization performance.

Wall-clock Training Speedup. We benchmark GPU memory consumption and wall-clock training time on CIFAR-100 for each stage during training on single NVIDIA 2080Ti GPU. The large baseline ResNet-18 trains for 140 minutes to achieve 78.36 % accuracy. As shown in the green bar of Figure 7(b), the growing method only achieves marginal wall-clock acceleration, under the same fixed batch size. As such, the growing ResNet-18 takes 120 minutes to achieve 78 . 12% accuracy. The low GPU utilization in the green bar in Figure 7(a) hinders FLOPs savings from translating into real-world training acceleration. In contrast, the red bar of Figure 7 shows our batch size adaptation results in a large proportion of wall clock acceleration by filling the GPU memory, and corresponding parallel execution resources, while maintaining test accuracy. ResNet-18 trains for 84 minutes ( 1 . 7 × speedup) and achieves 78 . 01% accuracy.

5 Conclusion

We tackle a set of optimization challenges in network growing and invent a corresponding set of techniques, including initialization with functionality preservation, variance transfer and learning rate adaptation to address these challenges. Each of these techniques can be viewed as 'upgrading' an original part for training static networks into a corresponding one that accounts for dynamic growing. There is a one-to-one mapping of these replacements and a guiding principle governing the formulation of each replacement. Together, they accelerate training without impairing model accuracy - a result that uniquely separates our approach from competitors. Applications to widelyused architectures on image classification and machine translation tasks demonstrate that our method bests the accuracy of competitors while saving considerable training cost.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science Foundation under grant CNS-1956180 and the University of Chicago CERES Center.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign , 2014.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587 , 2017.

Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2Net: Accelerating learning via knowledge transfer. In ICLR , 2016.

Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. NeST: A neural network synthesis tool based on a grow-and-prune paradigm. IEEE Trans. Computers , 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR , 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019.

John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR , 2011.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In ICLR Workshop , 2018.

Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. GradMax: Growing neural networks using gradient information. In ICLR , 2022.

Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V. Le. NAS-FPN: Learning scalable feature pyramid architecture for object detection. arXiv:1904.07392 , 2019.

Boris Ginsburg, Igor Gitman, and Yang You. Large batch training of convolutional networks with layer-wise adaptive rate scaling. 2018.

Ross B. Girshick. Fast R-CNN. In ICCV , 2015.

Sam Gross and Michael Wilber. Training and investigating residual nets. http://torch.ch/blog/ 2016/02/04/resnets.html , 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Samuel Horváth, Aaron Klein, Peter Richtárik, and Cédric Archambeau. Hyperparameter transfer learning with adaptive complexity. In AISTATS , 2021.

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 , 2017.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV , 2016.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , 2015.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. http://www.cs.toronto. edu/~kriz/cifar.html , 2014.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In NeurIPS , 2012.

Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv:1904.00310 , 2019.

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-DeepLab: Hierarchical neural architecture search for semantic image segmentation. In CVPR , 2019.

Qiang Liu, Wu Lemeng, and Wang Dilin. Splitting steepest descent for growing neural architectures. In NeurIPS , 2019.

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV , 2016.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL , 2002.

Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. In NeurIPS , 2018.

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI , 2019.

Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire. Domain-independent dominance of adaptive methods. In CVPR , 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 2012.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Chengcheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire. Orthogonalized SGD and nested architectures for anytime neural networks. In ICML , 2020.

Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In ICML , 2016.

Wei Wen, Feng Yan, Yiran Chen, and Hai Li. AutoGrow: Automatic layer growing in deep convolutional networks. In KDD , 2020.

Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In NeurIPS , 2020.

Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture optimization: Escaping local optimum with signed neural splitting. arXiv:2003.10392 , 2020.

Greg Yang and Edward J. Hu. Tensor Programs IV: Feature learning in infinite-width neural networks. In ICML , 2021.

Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In NeurIPS , 2021.

Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In AISTATS , 2014.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR , 2020.

Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In ICLR , 2021.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07146 , 2016.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR , 2022.

A More Analysis on Variance Transfer

Our fixed rescaling formulation, regarding relative network width, is an extension to the principled zero-shot HP transfer method [40, 41], based on the stability assumption, denoted as VT. A dynamic rescaling based on actual old weight values is an alternative plausible implementation choice, denoted as VT-constraint.

Theorem A.1. Suppose the goal is to enforce the unit variance feature, then the scaling factor of an input layer with weights W x with input shape C x is √ 1 C x V [ W x ] , while for a hidden layer with weights W u and input

Proof. Consider a hidden layer that computes u = W x x followed by another layer that computes h = W u u (ignoring activations for simplicity). At a growth step, the first layer's outputs change from

where s x denotes the scaling factor applied to W x . The second layer's outputs change from

to to

where s u denotes the scaling factor applied to W u .

The variance of the features after the growth step are:

Given the goal of enforcing unit-variance features for across all four vectors, we get:

This differs from the default VT formulation in Section 3.1, which corresponds to scaling factors of s x = 1 and s u = √ C u t C u t +1

We compare the default VT with VT-constraint by growing ResNet-20 on CIFAR-10. As shown in Table 7, both VT and VT-constraint outperform the standard baseline, which suggests standard initialization is a suboptimal design in network growing. We also note that involving the weight statistics is not better than our simpler design, which suggests that enforcing the old and the new features to have the same variance is not a good choice.

Growing

(a) Growing CNN hidden layer

Conv 1

Conv 2

Input

Weight Shape

Weight Shape

+

Shortcut

Weight Shape

Output

Output Shape

Residual

Block

Width Growth

with Same Rate

Conv 1

Conv 2

Input

Weight Shape

Weight Shape

+

Shortcut

Weight Shape

Output

Output Shape

Residual

Block

(b) Growing residual block

B General Network Growing in Other Layers

We have shown network growing for 3-layer fully connected networks as a motivating example in Section 3.1. We now show how to generalize network growing with K (>3) layers, with conv layers, residual connections.

Generalization to K ( > 3 ) Layers. In our network-width growing formulation, layers may be expanded in 3 patterns. (1) Input layer: output channels only; (2) Hidden layer: Both input (due to expansion of the previous layer) and output channels. (3) Output layer: input channels only. As such, the 3-layer case is sufficient to serve as a motivating example without loss of generality. For K ( > 3 ) layer networks, the 2nd to ( K -1) th layers simply follow the hidden layer case defined in Section 3.1.

Generalization to Convolutional layers. In network width growth, we only need to consider the expansion along input and output channel dimensions no matter for fully connected networks or CNNs. Equations 1-4 still hold for CNN layers. Specifically, when generalizing to CNNs (from 2-d C out × C in weight matrices to 4-d C out × C in × k × k ones), we only consider the matrix operations on the first two dimensions since we do not change the kernel size k . For example, in Figure 2(b), newly added weights + Z u t in linear layer can be indexed as W [0 : C h t , C u t : C u t + C u t +1 -C u t 2 ] . In CNN layer, it is simply W [0 : C h t , C u t : C u t + C u t +1 -C u t 2 , : , :] .

Residual Connections. We note that differently from plain CNNs like VGG, networks with residual connections do require that the dimension of an activation and its residual match since these will be added together. Our method handles this by growing all layers at the same rate. Hence, at each growth step, the shape of the two tensors to be added by the skip connections always matches.

Concrete Example of Adding Units to a Network. In model width growth, our common practice is to first determine the output dimension expansion from the growth scheduler. If a previous layer's output channels are to be grown, then the input dimension of the current layer should be also expanded to accommodate that change.

Let's consider a concrete example for a CNN with more layers ( > 3 ) and residual blocks. Without loss of generality, we omit the kernel size k and denote each convolutional layer channel shape as ( C in , C out ). We denote the input feature with 1 output dimension as x . Initially, we have the input layer W 1 with channel dimensions of (1,2) to generate h 1 (2-dimensional), followed by a 2-layer residual block (identity mapping for residual connection) with weights W (2) (2 , 2) and W (3) (2 , 2) , followed by an output layer with weights W (4) (2 , 1) .

The corresponding computation graph (omitting BN and ReLU for simplification) is

In more detail, we rewrite the computation in the matrix formulation:

Now assume that we want to grow the dimension of the network's hidden activations from 2 to 4 ( i.e., h (1) , h (2) , h (3) , h (4) , which are 2-dimensional, should become 4-dimensional each).

We added 2 rows to the input layer's weights W (1) (to increase its output dimension from 2 to 4), added 2 rows and 2 columns to the hidden layer's weights W (2) , W (3) (to increase its output dimensions from 2 to 4, and to increase its input dimension from 2 to 4 so they are consistent with the previous layers), and added 2 columns to the output layer's weights W (4) (to increase its input dimension from 2 to 4 so it is consistent with the increase

in dimensionality of h (4) ). Note that h (3) and h (1) still maintain matching shapes (to be added up) in the residual block since we grow W (1) and W (3) 's output dimensions with the same rate.

We summarize the architectural growth in terms of C in and C out (omitting kernel size k ) in Table 8. We also show the growing for CNN hidden layer in Figure 8(a) and residual blocks in Figure 8(b) for better illustration.

Note that this example shows how growing works in general; our specific method also includes particulars as to what values are used to initialize the newly-added weights, as well as modifications to optimizer state.

C Generalization to Other Optimizers

We generalize our LR adaptation rule to Adam [20] and AvaGrad [31] in Table 9. Both methods are adaptive optimizers where different heuristics are adopted to derive a parameter-wise learning rate strategy, which provides primitives that can be extended using our stage-wise adaptation principle for network growing. For example, vanilla Adam adapts the global learning rate with running estimates of the first moment m t and the second moment v t of the gradients, where the number of global training steps t is an integer when training a fixed-size model. When growing networks, our learning rate adaptation instead considers a vector t which tracks each subcomponent's 'age' ( i.e., number of steps it has been trained for). As such, for a newly grown subcomponent at a stage i > 0 , t [ i ] starts as 0 and the learning rate is adapted from m t / √ v t (global) to m t [ i ] \ m t [ i -1] √ v t [ i ] \ v t [ i -1] (stage-wise). Similarly, we also generalize our approach to AvaGrad by adopting η t , d t , m t of the original paper as a stage-wise variables.

Preserving Optimizer State/Buffer. Essential to adaptive methods are training-time statistics ( e.g., running averages m t and v t in Adam) which are stored as buffers and used to compute parameter-wise learning rates. Different from fixed-size models, parameter sets are expanded when growing networks, which in practice requires re-instantiating a new optimizer at each growth step. Given that our initialization scheme maintains functionality of the network, we are also able to preserve and inherit buffers from previous states, effectively maintaining the optimizer's state intact when adding new parameters. We experimentally investigate the effects of this state preservation.

Table 11: Comparisons among standard SGD, LARS, and our adaptation method for growing ResNet-20 on CIFAR-10.

Results with Adam and AvaGrad. Table 10 shows the results of growing ResNet-20 on CIFAR-10 with Adam and Avagrad. For the large, fixed-size baseline, we train Adam with lr = 0 . 1 , ϵ = 0 . 1 and AvaGrad with lr = 0 . 5 , ϵ = 10 . 0 , which yields the best results for ResNet-20 following [31]. We consider different settings for comparison: (1) optimizer without buffer preservation: the buffers are refreshed at each new growing phase; (2) optimizer with buffer preservation: the buffer/state is inherited from the previous phase, hence being preserved at growth steps; (3) optimizer with buffer and rate adaptation (RA): applies our rate adaptation strategy described in Table 9 while also preserving internal state/buffers. We observe that (1) consistently underperforms (2), which suggests that preserving the state/buffers in adaptive optimizers is crucial when growing networks. Option (3) bests the other settings for both Adam and AvaGrad, indicating that our rate adaptation strategy generalizes effectively to Adam and AvaGrad for the growing scenario. Together, these also demonstrate that our method has the flexibility to incorporate different statistics that are tracked and used by distinct optimizers, where we take Adam and AvaGrad as examples. Finally, our proposed stage-wise rate adaptation strategy can be employed with virtually any optimizer.

Comparison with Layer-wise Adaptive Optimizer. We also consider LARS [12, 43], a layer-wise adaptive variant of SGD, to compare different adaptation concepts: layer-wise versus layer + stage-wise (ours). Note that although LARS was originally designed for training with large batches, we adopt a batch size of 128 when growing ResNet-20 on CIFAR-10. We search the initial learning rate (LR) for LARS over { 1 e3 , 2 e3 , 5 e3 , 1 e2 , 2 e2 , 5 e2 , 1 e1 , 2 e1 , 5 e1 } and observe that a value of 0 . 02 yields the best results. We adopt the default initial learning rate of 0 . 1 for both standard SGD and our method. As shown in Table 11, LARS underperforms both standard SGD and our adaptation strategy, suggesting that layer-wise learning rate adaptation by itself i.e., without accounting for stage-wise discrepancies - is not sufficient for successful growing of networks.

D More Analysis on Rate Adaptation

We show additional plots of stage-wise rate adaptation when growing a ResNet-20 on CIFAR-10. Figure 9 shows the of adaptation factors based on the LR of the seed architecture from 1st to 8th stages (the stage index starts at 0). We see an overall trend that for newly-added weights, its learning rate starts at > 1 × of the base LR then quickly adapts to a relatively stable level. This demonstrates that our approach is able to efficiently and automatically adapt new weights to gradually and smoothly fade in throughout the current stage's optimization procedure.

We also note that rate adaptation is a general design that different subnets should not share a global learning rate. The RA formulation is designed empirically. max (1 , || W i \ W i -1 || ) is a plausible implementation choice,

Table 12: Comparisons among different RA implementation choices for growing ResNet-20 on CIFAR-10.

9

10

11

12

13

14

15

16

17

Epochs (1-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

(a) 1-st Stage

18

20

22

24

26

28

Epochs (2-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

(b) 2-nd Stage

30

32

34

36

38

40

Epochs (3-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

(c) 3-rd Stage

42

44

46

48

50

52

54

56

Epochs (4-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

(d) 4-th Stage

58

60

62

64

66

68

70

72

74

76

Epochs (5-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

(e) 5-th Stage

77.5

80.0

82.5

85.0

87.5

90.0

92.5

95.0

97.5

Epochs (6-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

Subnet-6

(f) 6-th Stage

100

105

110

115

120

125

Epochs (7-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

Subnet-6

Subnet-7

(g) 7-th Stage

130

135

140

145

150

155

160

Epochs (8-th Stage)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Rate Adaptation Factors

Subnet-0(Seed Arch.)

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

Subnet-6

Subnet-7

Subnet-8

(h) 8-th Stage

based on the assumption that new weights must have a higher learning rate. We conducted experiments by growing ResNet-20 on CIFAR-10. As shown in Table 12, we see that this alternative does not work better than our original design, and even underperforms standard SGD.

E More Visualizations on Sub-Component Gradients

We further compare global LR and our rate adaptation by showing additional visualizations of sub-component gradients of different layers and stages when growing ResNet-20 on CIFAR-10. We select the 2nd (layer1block1-conv1) and 17th (layer3-block2-conv2) convolutional layers and plot the gradients of each sub-component at the 3rd and 5th growing stages, respectively, in Figures 10, 11, 12, 13. These demonstrate that our rate adaptation strategy is able to re-balance and stabilize the gradient's contribution of different subcomponents, hence improving the training dynamics compared to a global scheduler.

F Simple Example on Fully-Connected Neural Networks

Additionally, we train a simple fully-connected neural network with 8 hidden layers on CIFAR-10 - each hidden layer has 500 neurons and is followed by ReLU activations. The network is has a final linear layer with 10 neurons for classification. Note that each CIFAR-10 image is flattened to a 3072-dimensional ( 32 × 32 × 3 ) vector as prior to being given as input to the network. We consider two variants of this baseline network by adopting training epochs (costs) ∈ { 25(1 × ) , 50(2 × ) } . We also grow from a thin architecture to the original one within 10 stages, each stage consisting of 5 epochs, where the number of units of each hidden layer grows from 50 to 100 , 150 , ..., 500 . The total training cost is equivalent to the fixed-size one trained for 25 epochs. We train all baselines using SGD, with weight decay set as 0 and learning rates sweeping over { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } : results are shown in Figure 14(a). Compared to standard initialization (green),

28

30

32

34

36

38

40

Epochs (3-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

3

Subnet-0

Subnet-1

Subnet-2

Subnet-3

(a) Using Global

28

30

32

34

36

38

40

Epochs (3-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

3

Subnet-0

Subnet-1

Subnet-2

Subnet-3

(b) Using RA

57.5

60.0

62.5

65.0

67.5

70.0

72.5

75.0

Epochs (5-th Stage)

1

2

3

4

5

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

(a) Using Global

57.5

60.0

62.5

65.0

67.5

70.0

72.5

75.0

Epochs (5-th Stage)

1

2

3

4

5

6

7

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

(b) Using RA

28

30

32

34

36

38

40

Epochs (3-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

(a) Using Global

28

30

32

34

36

38

40

Epochs (3-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

(b) Using RA

57.5

60.0

62.5

65.0

67.5

70.0

72.5

75.0

Epochs (5-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

(a) Using Global

57.5

60.0

62.5

65.0

67.5

70.0

72.5

75.0

Epochs (5-th Stage)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gradient Norm

1e

4

Subnet-0

Subnet-1

Subnet-2

Subnet-3

Subnet-4

Subnet-5

(b) Using RA

2

6

2

5

2

4

2

3

2

2

2

1

Learning Rate

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Train Loss

Large Model 1x Train Cost

Large Model 2x Train Cost

Growing (Standard)

Growing (Ours Var. Transfer + Standard SGD)

Growing (Ours Var. Transfer + SGD with Rate Adaptation)

(a) Train Loss

2

6

2

5

2

4

2

3

2

2

2

1

Learning Rate

50

51

52

53

54

55

Test Accuracy(%)

Large Model 1x Train Cost

Large Model 2x Train Cost

Growing (Standard)

Growing (Ours Var. Transfer + Standard SGD)

Growing (Ours Var. Transfer + SGD with Rate Adaptation)

(b) Test Accuracy

the loss curve given by growing variance transfer (blue) is more similar to the curve of the large baseline - all using standard SGD - which is also consistent with the observations when training model of different scales separately [41]. Rate adaptation (in red) further lowers training loss. Interestingly, we observe in Figure 14(b) that the test accuracy behavior differs from the training loss one given in Figure 14(a), which may suggest that regularization is missing due to, for example, the lack of parameter-sharing schemes (like CNN) in this fully-connected network.

G More Comparisons with GradMax [10]

For CIFAR-10 and CIFAR-100, GradMax used different models for growing and we did not re-implement GradMax on both datasets. Also, generalizing such gradient-based growing methods to the Transformer architecture is nontrivial. As such, we only cover MobileNet on ImageNet which is used in both ours and GradMax. Our accuracy outperforms GradMax by 1 . 3 while lowering training costs, which is significant to demonstrate the benefit of our method. We also trained our method to grow WRN-28-1 (w/wo BatchNorm, used in GradMax paper) on CIFAR-10 and CIFAR-100 and compare it with GradMax in Table 13. We see that ours still consistently outperforms GradMax.

H Extension to Continuously Incremental Datastream

Another direct and intuitive application for our method is to fit continuously incremental datastream where D 0 ⊂ D 1 , ... ⊂ D n ... ⊂ D N -1 . The network complexity scales up together with the data so that larger capacity can be trained on more data samples. Orthogonalized SGD (OSGD) [35] address the optimization difficulty in this context, which dynamically re-balances task-specific gradients via prioritizing the specific loss influence. We further extend our optimizer by introducing a dynamic variant of orthogonalized SGD, which progressively adjusts the priority of tasks on different subnets during network growth.

Suppose the data increases from D n -1 to D n , we first accumulate the old gradients G n -1 using one additional epoch on D n -1 and then grow the network width. For each batch of D n , we first project gradients of the new architecture ( n -th stage), denoted as G n , onto the parameter subspace that is orthogonal to G pad n -1 , a zero-padded version of G n -1 with desirable shape. The final gradients G ∗ n are then calculated by re-weighting the original G n and its orthogonal counterparts:

where λ is a dynamic hyperparameter which weights the original and orthogonal gradients. When λ = 1 , subsequent outputs do not interfere with earlier directions of parameters updates. We then anneal λ to 0 so that the newly-introduced data and subnetwork can smoothly fade in throughout the training procedure.

Implementation Details. We implement the task in two different settings, denoted as 'progressive class' and 'progressive data' on CIFAR-100 dataset within 9 stages. In the progressive class setting, we first randomly select 20 classes in the first stage and then add 10 new classes at each growing stage. In the progressive data setting, we sequentially sample a fraction of the data with replacement for each stage, i.e., 20% , 30% , ..., 100% .

ResNet-18 on Continuous CIFAR-100. We evaluate our method on continuous datastreams by growing a ResNet-18 on CIFAR-100 and comparing the final test accuracies. As shown in Table 14, compared with the large baseline, our growing method achieves 1 . 53 × cost savings with a slight performance degradation in both settings. The dynamic OSGD variant outperforms the large baseline with 1 . 46 × acceleration, demonstrating that the new extension improves the optimization on continuous datastream through gradually re-balancing the task-specific gradients of dynamic networks.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces multiple synergistic tricks that allow for structurally growing networks to be competitive with and even sometimes outperform static networks. These methods include a batched schedule of layer widening, initializing opposing neurons, adapting the scale of the existing weights, handling optimizer  and using subnetwork-specific dynamic learning rates.\n\nSTRENGTHS:\nOriginality: The paper introduces a novel synergy of methods to support growing a neural network, some of which are novel themselves. \n\nQuality: The paper provides experimental results demonstrating that the proposed method achieves comparable or better accuracy than large fixed-size models while saving computation time during training across image classification and machine translation benchmarks and backbones architectures.\n\nClarity: Overall, the authors effectively communicate their ideas, methodologies, and results, making it easier for readers to understand the research. Some inconsistencies are noted below.\n\nSignificance: This is also the most thorough and practical work on growing models to my knowledge, from shallow and deep CNNs to transformers.\n\nWEAKNESSES:\nPlease check and fix the formatting of your paper. Table 5 violates the right margin. Table 6 is hard to read without sufficient spacing between the columns. Many figures contain text that is too small and completely illegible when printed: for accessibility, please ensure all fonts within figures are legible at only 100% zoom.\n\nQUESTIONS:\nFigure 1(b), notably the input weights of $\\delta+\\epsilon_1$ and $\\delta+\\epsilon_2$, and the caption of Figure 2 suggest you are doing “symmetry breaking”, but the rest of your methods do not mention this and rather emphasize that the new weights are two identical copies. Can you clarify this point? Does symmetry breaking help empirically?\n\nWhen including results of methods from related works, did you reimplement/run them yourself or copy their reported results? In the latter case, you must report any differences in experimental protocols beyond methods, including hyperparameters, optimizers, and software/hardware.\n\nThe learning rate adaptation method based on weight norm is not very clearly motivated. How was this selected: just on the previous works, or is there further justification? How is this affected by hyperparameter choices such as initialization scheme and weight decay? Other strategies exist that may be more intuitive, such as scaling by gradient norm [1]. Simply applying a subnet-wise cosine annealing may also suffice. \n\nLee, Yoonho, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. \"Surgical Fine-Tuning Improves Adaptation to Distribution Shifts.\" In The International Conference on Learning Representations. 2023.\n\nCheck the bibtex entry for Chen et al. (2016).\n\nWhenever you refer to the appendix (Lines 116, 156, etc.), please refer to a specific appendix section.\n\nLine 204 has a typo: CIAFR -> CIFAR.\n\nLIMITATIONS:\nNo limitations nor potential negative societal impacts are explicitly addressed. Including these discussions would be appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the growth of neural networks in order to reduce training time. Authors study various optimization challenges when training dynamically grown neural networks and propose some novel techniques to address these challenges. The paper is well written in general, though it requires few clarifications.\n\nSTRENGTHS:\n- Important research topic with a lot of technical complexity. The community would benefit a lot from results and techniques given assuming authors plan to open-source their code and make their results reproducible.\n- This paper studies the optimization for growing networks, which is an understudied, yet very important topic.\n- This paper proposes a new initialization technique for growing NNs and combines it with a version of LARS optimizer.\n- Experimental evaluation include transformers trained on natural language.\n- Authors achieve real-time speed-ups through additional optimization (i.e. increasing batch size).\n\nWEAKNESSES:\n## Major\n- Comparison with other methods in ImageNet is not ideal since baseline numbers might be different in different codebases. Also ideally one would only change the initialization and keep others parts the same to demonstrate that the proposed initialization is better than the alternatives. Looking at the results, it is not clear whether the alternatives would perform worse with the adaptive learning rate. For example one can run Gradmax with the proposed adaptive learning rate and growth schedule and compare the results. In general it would be nice to give more details about the Imagenet experiments and try applying the proposed improvements to other methods and see whether they also improve performance for them.\n- “Together, they accelerate training without impairing model accuracy – a result that uniquely separates our approach from competitors.” This is not supported as baselines are not as good as the original baseline for the MobileNet-Cifar100 and Imagenet experiments. Can we train longer to get better results? It would be nice to run these experiments and check whether growing methods can match baseline performance with additional training. \n- Contributions can be more accurate. (1) Function preserving growth is proposed by many (Openai, gradmax, net2net). (2) Stable training dynamics also exist for these methods. If what authors meant is to be more robust to different learning rates, I believe this is a common feature for adaptive optimizers.  (3) “Existing methods [26, 38] fail to derive a systemic way for distributing training resources across a growth trajectory.” Not clear this is true. Authors suggest a different heuristics than the ones used in previous work. I don't think there is enough motivation to claim this particular way being more systematic than the others. At the end they are all heuristics and the one proposed by this paper seem to perform better. I would focus on this part.\n\n## Minor\n- This paper seems to suggest a new initialization technique for neuron adding. You can also cite Gradmax showing that adding neurons to be more effective than splitting neurons. \n- I would move VGG models to the appendix. For any practical purposes VGGs are a bit irrelevant at this point. \n- Table 4: Firefly->FireflyOpt\n- Figure 3 is very difficult to read in printed form, can you make these plots bigger?\n\nQUESTIONS:\n- Will the code be made public?\n- How many epochs do you train for ImageNet?\n- In Equation (5) what is f? Does it return momentum statistics? If so, it might be better to use a different letter instead of a function.\n- Algorithm 1, what is S_n? Do you mean W_n?\n- In Tables 2-5, how do you calculate the training cost? It would be nice to add this calculation to the appendix?\n- In Tables 2-5, how do you generate Net2Net and Firefly results. Do you use the same schedule? If different schedules are used, do you do hyper parameter search for these baselines?\n- Can you also report training loss in the appendix? Often small datasets have overfitting problem as shown in Figure 3, which is often not the case for larger datasets and it would be nice to know where the gains are.\n- \"We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers\" Why optimizer states have different treatments in Adam and SGD? Can you do an ablation on this? Also do you do the same for the baselines, i.e. Net2Net and Firefly?\n\nLIMITATIONS:\nThere is no limitation section at the moment. I think inability to match baseline numbers for larger experiments can be mentioned here.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStarting from an optimization and training dynamics perspective, this paper proposes a new method to efficiently growing neural networks, based on a parameterization that dynamically stabilizes weight, activation, and gradient scaling as the architecture grows over time and different subnetworks take the lion's share of the gradient contribution. To achieve this while preserving functional continuity, the approach  presented here grows the network by adding new random untrained neurons with initial net 0 functional contribution, and involves a learning rate scheduling mechanism that rebalances the gradient contribution to different subcomponents of the network added over time. The functional preservation reduces training disruptions and instabilities otherwise introduced by this step-function change in the network.This achieves wall-clock improvement at even or better accuracy than training a large fixed-size model.\n\nSTRENGTHS:\nThe biggest strength of the paper is that it takes a reasonable, justified, principled approach to improving upon currently available methodologies in network growth, clearly hypothesizes that it might result in certain expected improvements, and then sets out to test this hypothesis through empirical experimentation. This thought process and methodology is clearly outlined in this well-written paper.\n\nThe paper compares to various competing baselines, not only to the large full-size model, thus providing an empirical comparison among similar solutions in the literature. In particular, the authors break down the performance of their method into incremental performance gains provided by the various components in the system they designed, as show in Table 6. These ablations help answer questions about the relative important of each design choice.\n\nI also found the empirical investigation of the growing schedule quite compelling, as I imagine it would be a point of exploration for any practitioner wanting to implement this methodology.\n\nWEAKNESSES:\nThe study is limited to simple, low scale image classification and machine translation tasks, which aren't representative of the application domains often facing researchers and practitioners these days. Therefore, it remains unproved whether this method would help in the context of large scale deep learning, which is the context in which the problems the authors are trying to solve actually occur. The disconnect between the problem statement and the regime in which the solution is tested weakens the point of the paper, as many promising approaches at low scales have proven completely ineffective at large scales.\n\nOne general concern with progressively grown architectures, especially with respect to whether the data distribution is / isn't changed with each growing iteration, is about the generalization properties of the final configuration, whether it loose the generalization properties of the equivalent full-scale architecture when the training data distribution stays constant, or whether it affects the task learning abilities in a continual learning setup.\n\nQUESTIONS:\nJust a suggestion: since Figure 3 is too small to be legible, I'd recommend moving it to the appendix.\n\nLIMITATIONS:\nThere are no major concerns about societal impact of this work. Authors do not over-claim impact in this paper, and they are quite matter-of-fact about the applicability of their work. The scale limitation of their experiments, however, prevents this work from being more impactful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors study the network growth problem and propose a novel problem to stabilize the optimization difficulty. The authors conduct experiments on various model architectures (including VGG, ResNet, MobileNet, and Transformer) and observe consistent performance improvements.

STRENGTHS:
The studied problem is important. The proposed approach is novel. Experiments with various model architectures are conducted and demonstrate consistent performance improvements.

WEAKNESSES:
The most important application for the proposed algorithm is large-scale training. However, the experiments discussed in this study are based on relatively small models, which left doubts on the applicability of the proposed method on large-scale training.

QUESTIONS:
Is the Transformer used in Section 4.3 constructed in the Post-LN or Pre-LN manner?

LIMITATIONS:
I suggest conducting experiments with pre-training tasks like BERT, which would provide great value to this study. Also, as to the machine translation task, the IWSLT14 dataset is not as reliable as the WMT14 EN-De/Fr dataset. Some additional related work on this topic are: 1. Shallow-to-Deep Training for Neural Machine Translation; 2. On the transformer growth for progressive bert training.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper developed an approach to efficiently grow neural network architecture by parameterization scheme with regard to weight, activation, and gradient scaling. It employed a learning rate adaptation mechanism rebalancing the gradient contribution of the subnetworks. The experimental results show efficiency in wall-clock training time.\n\nSTRENGTHS:\n-\tThe main technical section is described in detail, although there are some missing points (please refer to the Questions section.)\n-\tThe paper is easy to follow.\n\nWEAKNESSES:\n-\tWondering if the authors considered the original training cost of the original backbone network. For example, if the original network costs XXX for training from scratch, how much will the proposed approach will cost when all steps’ costs are aggregated, apples to apples? \n-\tThere is no description of how to calculate train costs.\n-\tIt is not clear if the subnetworks are trained at a global learning rate or a different learning rate: at line 58, the authors claimed that subnetworks with different lengths have distinct learning rate schedules, while at line 142, it is noted that subnetworks from different growth stages will share a global learning rate. \n-\tIt would have been better if the authors provided justification for why they picked the LR adaptation rule and comparisons with other LR adaptation rules. That is because a learning rate schedule (although, in many cases, it is overlooked) can have a high impact on the final performance and result. So it would have been nicer if the authors explained more or showed more clues on that.\n-\tIn (6), C_0 (base case) is not defined.\n-\tIn (7), the base case is not defined either. Also, are the parentheses (with floor opening and ceiling closing) for rounding to the nearest odd number?\n-\tIt needs to be clarified why Figure 7 (b) is measured over time/epoch. In other words, why the time spent per epoch matters. It would have been nicer if the authors had added an explanation for that.\n-\tIt is not clear the overall benefit of growing a network gradually while preserving the function computed by the model at each growth step. Is there any potential possibility/intention to use such an intermediate model? Otherwise, it is not clear to preserve the functions at each step.\n\nQUESTIONS:\n- In Figure 7 (b), why is it measured over time/epoch?\n- In (6) and (7), what are the base cases?\n- Why was the certain learning rate adaptation rule selected?\n\nLIMITATIONS:\nThe authors did not mention anything about limitations. It would have been nicer if they were upfront about any limitations to share.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for incrementally growing network width across stages. The reader is shown the details of the growing process using a 3-layer feedforward network as an example, with the appendix providing details for other types of networks. Additional algorithmic details, including learning rate and batch size scheduling are provided. The paper then presents experimental results over 3 repeats, including an ablation study.\n\n**Rebuttal:** I have read the rebuttal and it addresses my questions. Based on the novelty and contribution of this work, I maintain my original rating of 6.\n\nSTRENGTHS:\nSection 3 of this paper is perhaps the strongest part of this paper. There is no ambiguity in the presentation, and any questions the reader may have (generalization to Adam and other network architectures) are presented either in the appendix or later in the paper. The paper also does not limit itself to a single architecture, and shows details for two other optimizers, as well as other architecture types. It also provides an ablation study showing how each component of the overall approach is useful.\n\nWEAKNESSES:\nA few parts of the paper are a bit terse. In particular, Section 4 of the Appendix could benefit from additional exploration of, e.g., more alternatives or motivation/derivation for using the proposed alternative.\n\nQUESTIONS:\n- In Table 1, I'm not entirely sure I see a clear pattern: if we had one more layer, for example, would the last layer (whose dimensions I'll still call $C^h$) at initialization have the old weights rescaled as $\\left(\\frac{C^h_t}{C^h_{t+1}}\\right)^{(3/2)}$, and the new inits be $\\frac{1}{(C^h_{t+1})^3}$?\n- In L143, you say using a global learning rate causes divergent behavior: have you seen this in practice? Can you show loss plots under this scheme?\n- In Eq. 5, what is the motivation for setting $f$ to the norm function? Is the Appendix Section 4 what the paper refers to when discussing alternative heuristics (L156)?\n- In Table 2, what do the underscores represent?\n- In Tables 2-5, what statistical tests did you run to compare your methods? I'm also not sure 3 repeats is sufficient to capture variance in performance effectively.\n- In Tables 4-5, please change the training costs of Net2Net to bold (since they were in bold in Tables 2-3).\n- In Table 4, did Net2Net and Firefly achieve the same performance across all 3 runs, or did you only run each once?\n- In Tables 2-5, how is training cost computed? Is it FLOPS/wall time/etc?\n\nLIMITATIONS:\nThe paper's limitations are addressed by providing details for how to overcome them. For example, an ablation study shows the necessity of the learning rate adaptation scheme; the appendix provides details on expansion to residual blocks and convolutional blocks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}