SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Accelerated Training via Incrementally Growing Neural Networks using
Variance Transfer and Learning Rate Adaptation
Xin Yuan
University of Chicago yuanx@uchicago.edu
Pedro Savarese
TTI-Chicago savarese@ttic.edu
Michael Maire
University of Chicago mmaire@uchicago.edu
We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network. To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents. Experiments show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original training computation budget. We demonstrate that these gains translate into real wall-clock training speedups.
Modern neural network design typically follows a "larger is better" rule of thumb, with models consisting of millions of parameters achieving impressive generalization performance across many tasks, including image classification [22, 32, 30, 46], object detection [13, 26, 11], semantic segmentation [27, 3, 24] and machine translation [34, 7]. Within a class of network architecture, deeper or wider variants of a base model typically yield further improvements to accuracy. Residual networks (ResNets) [15] and wide residual networks [45] illustrate this trend in convolutional neural network (CNN) architectures. Dramatically scaling up network size into the billions of parameter regime has recently revolutionized transformer-based language modeling [34, 7, 1].
The size of these models imposes prohibitive training costs and motivates techniques that offer cheaper alternatives to select and deploy networks. For example, hyperparameter tuning is notoriously expensive as it commonly relies on training the network multiple times, and recent techniques aim to circumvent this by making hyperparameters transferable between models of different sizes, allowing them to be tuned on a small network prior to training an original large model once [41].
Our approach incorporates these ideas, but extends the scope of transferability to include the parameters of the model itself. Rather than view training small and large models as separate events, we grow a small model into a large one through many intermediate steps, each of which introduces additional parameters to the network. Our contribution is to do so in a manner that preserves the function computed by the model at each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, we use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training a large static model from scratch.
(a) Existing Methods: Splitting Init, Global LR
Figure 1: Dynamic network growth strategies. Different from (a) which rely on either splitting [4, 25, 39] or adding neurons with auxiliary local optimization [38, 10], our initialization (b) of new neurons is random but function-preserving. Additionally, our separate learning rate scheduler governs weight updating to address the discrepancy in total accumulated training between different growth stages.
(b) Ours: Function-Preserving Init, Stagewise LR
Competing recent efforts to grow deep models from simple architectures [4, 23, 5, 25, 39, 37, 38, 44, 10] draw inspiration from other sources, such as the progressive development processes of biological brains. In particular, Net2Net [4] grows the network by randomly splitting learned neurons from previous phases. This replication scheme, shown in Figure 1(a) is a common paradigm for most existing methods. Gradient-based methods [38, 39] determine which neurons to split and how to split them by solving a combinatorial optimization problem with auxiliary variables.
At each growth step, naive random initialization of new weights destroys network functionality and may overwhelm any training progress. Weight rescaling with a static constant from a previous step is not guaranteed to be maintained as the network architecture evolves. Gradient-based methods outperform these simple heuristics but require additional training effort in their parameterization schemes. Furthermore, all existing methods use a global LR scheduler to govern weight updates, ignoring the discrepancy among subnetworks introduced in different growth phases. The gradient itself and other parameterization choices may influence the correct design for scaling weight updates.
We develop a growing framework around the principles of enforcing transferability of parameter settings from smaller to larger models (extending [41]), offering functional continuity, smoothing optimization dynamics, and rebalancing learning rates between older and newer subnetworks. Figure 1(b) illustrates key differences with prior work. Our core contributions are:
- Parameterization using Variance Transfer: We propose a parameterization scheme accounting for the variance transition among networks of smaller and larger width in a single training process. Initialization of new weights is gradient-free and requires neither additional memory nor training.
- Improved Optimization with Learning Rate Adaptation: Subnetworks trained for different lengths have distinct learning rate schedules, with dynamic relative scaling driven by weight norm statistics.
- Better Performance and Broad Applicability: Our method not only trains networks fast, but also yields excellent generalization accuracy, even outperforming the original fixed-size models. Flexibility in designing a network growth curve allows choosing different trade-offs between training resources and accuracy. Furthermore, adopting an adaptive batch size schedule provides acceleration in terms of wall-clock training time. We demonstrate results on image classification and machine translation tasks, across various network architectures.
Network Growing. A diverse range of techniques train models by progressively expanding the network architecture [36, 9, 5, 37, 44]. Within this space, the methods of [4, 25, 39, 38, 10] are most relevant to our focus - incrementally growing network width across multiple training stages. Net2Net [4] proposes a gradient-free neuron splitting scheme via replication, enabling knowledge transfer from previous training phases; initialization of new weights follows simple heuristics. Liu et al.'s splitting approach [25] derives a gradient-based scheme for duplicating neurons by formulating a combinatorial optimization problem. FireFly [38] gains flexibility by also incorporating brand new neurons. Both methods improve Net2Net's initialization scheme by solving an optimization problem with auxiliary variables, at the cost of extra training effort. GradMax [10], in consideration of training dynamics, performs initialization via solving a singular value decomposition (SVD) problem.
Neural Architecture Search (NAS) and Pruning. Another subset of methods mix growth with dynamic reconfiguration aimed at discovering or pruning task-optimized architectures. Network Morphism [36] searches for efficient networks by extending layers while preserving the parameters. AutoGrow [37] takes an AutoML approach governed by heuristic growing and stopping policies. Yuan et al. [44] combine learned pruning with a sampling strategy that dynamically increases or decreases network size. Unlike these methods, we focus on the mechanics of growth when the target architecture is known, addressing the question of how to best transition weight and optimizer state to continue training an incrementally larger model. NAS and pruning are orthogonal to, though potentially compatible with, the technical approach we develop.
Hyperparameter Transfer. Multiple works [42, 29, 16] explore transferable hyperparameter (HP) tuning. The recent Tensor Program (TP) work of [40] and [41] focuses on zero-shot HP transfer across model scale and establishes a principled network parameterization scheme to facilitate HP transfer. This serves as an anchor for our strategy, though, as Section 3 details, modifications are required to account for dynamic growth.
Learning Rate Adaptation. Surprisingly, the existing spectrum of network growing techniques utilize relatively standard learning rate schedules and do not address potential discrepancy among subcomponents added at different phases. While general-purpose adaptive optimizers, e.g., AdaGrad [8], RMSProp [33], Adam [20], or AvaGrad [31], might ameliorate this issue, we choose to explicitly account for the discrepancy. As layer-adaptive learning rates (LARS) [12, 43] benefit in some contexts, we explore further learning rate adaption specific to both layer and growth stage.
3.1 Parameterization and Optimization with Growing Dynamics
Functionality Preservation. We grow a network's capacity by expanding the width of computational units (e.g., hidden dimensions in linear layers, filters in convolutional layers). To illustrate our scheme, consider a 3-layer fully-connected network with ReLU activations  $\phi$  at a growing stage  $t$ :
$$
\boldsymbol {u} _ {t} = \phi \left(\boldsymbol {W} _ {t} ^ {x} \boldsymbol {x}\right) \quad \boldsymbol {h} _ {t} = \phi \left(\boldsymbol {W} _ {t} ^ {u} \boldsymbol {u} _ {t}\right) \quad \boldsymbol {y} _ {t} = \boldsymbol {W} _ {t} ^ {h} \boldsymbol {h} _ {t}, \tag {1}
$$
where  $\pmb{x} \in \mathbb{R}^{C^x}$  is the network input,  $\pmb{y}_t \in \mathbb{R}^{C^y}$  is the output, and  $\pmb{u}_t \in \mathbb{R}^{C_t^u}$ ,  $\pmb{h}_t \in \mathbb{R}^{C_t^h}$  are the hidden activations. In this case,  $\pmb{W}_t^x$  is a  $C_t^u \times C^x$  matrix, while  $\pmb{W}_t^u$  is  $C_t^h \times C_t^u$  and  $\pmb{W}_t^h$  is  $C^y \times C_t^h$ . Our growing process operates by increasing the dimensionality of each hidden state, i.e., from  $C_t^u$  and  $C_t^h$  to  $C_{t+1}^u$  and  $C_{t+1}^h$ , effectively expanding the size of the parameter tensors for the next growing stage  $t+1$ . The layer parameter matrices  $\pmb{W}_t$  have their shapes changed accordingly and become  $\pmb{W}_{t+1}$ . Figure 2 illustrates the process for initializing  $\pmb{W}_{t+1}$  from  $\pmb{W}_t$  at a growing step. $^1$
Following Figure 2(a), we first expand  $W_{t}^{x}$  along the output dimension by adding two copies of new weights  $V_{t}^{x}$  of shape  $\frac{C_{t+1}^{u} - C_{t}^{u}}{2} \times C^{x}$ , generating new features  $\phi(\mathbf{V}_{t}^{x}\mathbf{x})$ . The first set of activations become
$$
\boldsymbol {u} _ {t + 1} = \operatorname {c o n c a t} \left(\boldsymbol {u} _ {t}, \phi \left(\boldsymbol {V} _ {t} ^ {x} \boldsymbol {x}\right), \phi \left(\boldsymbol {V} _ {t} ^ {x} \boldsymbol {x}\right)\right), \tag {2}
$$
where  $\mathrm{concat}$  denotes the concatenation operation. Next, we expand  $\pmb{W}_{t}^{u}$  across both input and output dimensions, as shown in Figure 2(b). We initialize new weights  $\pmb{Z}_{t}^{u}$  of shape  $C_t^h\times \frac{C_{t + 1}^u - C_t^u}{2}$  and add to  $\pmb{W}_{t}^{u}$  two copies of it with different signs:  $+Z_{t}^{u}$  and  $-Z_{t}^{u}$ . This preserves the output of the layer since  $\phi (\pmb{W}_t^u\pmb {u}_t + \pmb{Z}_t^u\phi (\pmb{V}_t^x\pmb {x}) + (-\pmb{Z}_t^u)\phi (\pmb{V}_t^x\pmb {x})) = \phi (\pmb{W}_t^u\pmb {u}_t) = \pmb {h}_t$ . We then add two copies of new weights  $\pmb{V}_{t}^{u}$ , which has shape  $\frac{C_{t + 1}^{h} - C_{t}^{h}}{2}\times C_{t + 1}^{u}$ , yielding activations
$$
\boldsymbol {h} _ {t + 1} = \operatorname {c o n c a t} \left(\boldsymbol {h} _ {t}, \phi \left(\boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1}\right), \phi \left(\boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1}\right)\right). \tag {3}
$$
We similarity expand  $\mathbf{W}_t^h$  with new weights  $\mathbf{Z}_t^h$  to match the dimension of  $h_{t+1}$ , as shown in Figure 2(c). The network's output after the growing step is:
$$
\begin{array}{l} \boldsymbol {y} _ {t + 1} = \boldsymbol {W} _ {t} ^ {h} \boldsymbol {h} _ {t} + \boldsymbol {Z} _ {t} ^ {h} \phi \left(\boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1}\right) + \left(- \boldsymbol {Z} _ {t} ^ {h}\right) \phi \left(\boldsymbol {V} _ {t} ^ {u} \boldsymbol {u} _ {t + 1}\right) \tag {4} \\ = \boldsymbol {W} _ {t} ^ {h} \boldsymbol {h} _ {t} = \boldsymbol {y} _ {t}, \\ \end{array}
$$
which preserves the original output features in Eq. 1. Appendix B provides illustrations for more layers.
<sup>1</sup>We defer the transformation between  $\mathbf{W}_t$  and  $\mathbf{W}_t'$  to the next subsection. It involves rescaling by constant factors, does not affect network functionality, and is omitted in Eq. 1-4 for simplicity.
(a) Input Layer
(b) Hidden Layer
(c) Output Layer
Figure 2: Initialization scheme. In practice, we also add noise to the expanded parameter sets for symmetry breaking.
Weights Initialization with Variance Transfer (VT). Yang et al. [41] investigate weight scaling with width at initialization, allowing hyperparameter transfer by calibrating variance across model size. They modify the variance of output layer weights from the commonly used  $\frac{1}{\mathrm{fan}_{in}}$  to  $\frac{1}{\mathrm{fan}_{in}^2}$ . We adopt this same correction for the added weights with new width:  $W^{h}$  and  $Z^{h}$  are initialized with variances of  $\frac{1}{C_t^{h,2}}$  and  $\frac{1}{C_{t + 1}^{h,2}}$ .
However, this correction considers training differently-sized models separately, which fails to accommodate the training dynamics in which width grows incrementally. Assuming that the weights of the old subnetwork follow  $\pmb{W}_t^h \sim \mathcal{N}(0, \frac{1}{C_t^{h2}})$  (which holds at initialization), we make them compatible
with new weight tensor parameterization by rescaling it with the fan $_{in}$  ratio as  $\boldsymbol{W}_t^{h'} = \boldsymbol{W}_t^h \cdot \frac{\boldsymbol{C}_t^h}{\boldsymbol{C}_{t+1}^h}$ . See Table 1 (top). Appendix A provides detailed analysis.
This parameterization rule transfers to modern CNNs with batch normalization (BN). Given a weight scaling ratio of  $c$ , the running mean  $\mu$  and variance  $\sigma$  of BN layers are modified as  $c\mu$  and  $c^2\sigma$ , respectively.
Stage-wise Learning Rate Adaptation (LRA). Following [41], we employ a learning rate scaling factor of  $\propto \frac{1}{\mathrm{fan}_{in}}$  on the output layer when using SGD, compensating for the initialization scheme. However, subnetworks from different growth stages still share a global learning rate, though they have trained for different lengths. This may cause divergent behavior among the corresponding weights, making the training iterations after growing sensitive to the scale of the newly-initialized weights. Instead of adjusting newly added parameters via local optimization [38, 10], we govern the update of each subnetwork in a stage-wise manner.
Let  $\mathcal{W}_t$  denote the parameter variables of a layer at a growth stage  $t$ , where we let  $\boldsymbol{W}_t$  and  $\boldsymbol{W}_t'$  correspond to the same set of variables such that  $\mathcal{W}_{t+1} \setminus \mathcal{W}_t$  denotes the new parameter variables whose values are initialized with  $\boldsymbol{Z}_t$  and  $\boldsymbol{V}_t$ . Moreover, let  $\boldsymbol{W}_{\Delta k}$  and  $\boldsymbol{G}_{\Delta k}$  denote the values and gradients of  $\mathcal{W}_k \setminus \mathcal{W}_{k-1}$ . We adapt the learning rate used to update each sub-weight  $\boldsymbol{W}_{\Delta k}$ , for
Table 1: Parameterization and optimization transition for different layers during growing.  $C_t$  and  $C_{t+1}$  denote the input dimension before and after a growth step.
Input Layer	Hidden Layer	Output Layer
Init.	Old Re-scaling	1	√Ct^u/Ct+1	Ct^h/Ct+1
New Init.	1/Ct^x	1/Ct+1	1/(Ct+1)^2
Adapt.	0-th Stage	1	1	1/C0
t-th Stage	||WΔt^x||	||WΔt^u||	||WΔt^h||
||WΔ0^x||	||WΔ0^u||	||WΔ0^h||
$0 \leq k \leq t$ , as follows:
$$
\eta_ {k} = \eta_ {0} \cdot \frac {f \left(\boldsymbol {W} _ {\Delta k}\right)}{f \left(\boldsymbol {W} _ {\Delta 0}\right)} \quad \boldsymbol {W} _ {\Delta k} \leftarrow \boldsymbol {W} _ {\Delta k} - \eta_ {k} \boldsymbol {G} _ {\Delta k}, \tag {5}
$$
where  $\eta_0$  is the base learning rate,  $f$  is a function that maps subnetworks of different stages to corresponding train-time statistics, and  $W_{\Delta 0}$  are the layer's parameter variables at the first growth stage. Table 1 (bottom) summarizes our LR adaptation rule for SGD when  $f$  is instantiated as weight norm, providing an stage-wise extension to the layer-wise adaptation method LARS [12], i.e.,  $LR \propto ||W||$ . Alternative heuristics are possible; see Appendices C and D.
3.2 Flexible and Efficient Growth Scheduler
We train the model for  $T_{total}$  epochs by expanding the channel number of each layer to  $C_{final}$  across  $N$  growth phases. Existing methods [25, 38] fail to derive a systemic way for distributing training resources across a growth trajectory. Toward maximizing efficiency, we experiment with a coupling between model size and training epoch allocation.
Architectural Scheduler. We denote initial channel width as  $C_0$  and expand exponentially:
$$
C _ {t} = \left\{ \begin{array}{l l} C _ {t - 1} + \lfloor p _ {c} C _ {t - 1} \rfloor_ {2} & \text {i f} t <   N - 1 \\ C _ {\text {f i n a l}} & \text {i f} t = N - 1 \end{array} \right. \tag {6}
$$
where  $\lfloor \cdot \rfloor_{2}$  rounds to the nearest even number and  $p_c$  is the growth rate between stages.
Epoch Scheduler. We denote number of epochs assigned to  $t$ -th training stage as  $T_{t}$ , with  $\sum_{t=0}^{N-1} T_{t} = T_{total}$ . We similarly adapt  $T_{t}$  via an exponential growing scheduler:
Algorithm 1: Growing using Var. Transfer and Learning Rate Adapt. with Flexible Scheduler
Input: Data  $X$ , labels  $Y$ , task loss  $L$
Output: Grown model  $\mathcal{W}$
Initialize:  $\mathcal{W}_0$  with  $C_0, T_0, B_0, \eta_0$
for  $t = 0$  to  $N - 1$  do
if  $t > 0$  then
Init.  $S_{n}$  from  $S_{n - 1}$  using VT in Table 1.
Update  $C_t$  and  $T_{t}$  using Eq. 6 and Eq. 7.
Update  $B_{t}$  using Eq. 8 (optional)
$\operatorname{Iter}_{total} = T_t * \text{len}(X) // B_t$
foriter  $= 1$  toItertotal do
Forward and calculate  $l = L(\mathcal{W}_t(\pmb{x}), \pmb{y}))$ .
Back propagation with  $l$ .
Update each sub-component using Eq. 5.
return  $\mathcal{W}_{N - 1}$
$$
T _ {t} = \left\{ \begin{array}{l l} T _ {t - 1} + \lfloor p _ {t} T _ {t - 1} \rceil & \text {i f} t <   N - 1 \\ T _ {\text {t o t a l}} - \sum_ {i = 0} ^ {N - 2} T _ {i} & \text {i f} t = N - 1 \end{array} \right. \tag {7}
$$
Wall-clock Speedup via Batch Size Adaptation. Though the smaller architectures in early growth stages require fewer FLOPs, hardware capabilities may still restrict practical gains. When growing width, in order to ensure that small models fully utilize the benefits of GPU parallelism, we adapt the batch size along with the exponentially-growing architecture in a reverse order:
$$
B _ {t - 1} = \left\{ \begin{array}{l l} B _ {\text {b a s e}} & \text {i f} t = N \\ B _ {t} + \lfloor p _ {b} B _ {t} \rceil & \text {i f} t <   N \end{array} \right. \tag {8}
$$
where  $B_{base}$  is the batch size of the large baseline model. Algorithm 1 summarizes our full method.
We evaluate on image classification and machine translation tasks. For image classification, we use CIFAR-10 [21], CIFAR-100 [21] and ImageNet [6]. For the neural machine translation, we use the IWSLT'14 dataset [2] and report the BLEU [28] score on German to English (De-En) translation.
Large Baselines via Fixed-size Training. We use VGG-11 [32] with BatchNorm [19], ResNet-20 [15], MobileNetV1 [17] for CIFAR-10 and VGG-19 with BatchNorm, ResNet-18, MobileNetV1 for CIFAR-100. We follow [18] for data augmentation and processing, adopting random shifts/mirroring and channel-wise normalization. CIFAR-10 and CIFAR-100 models are trained for 160 and 200 epochs respectively, with a batch size of 128 and initial learning rate (LR) of 0.1 using SGD. We adopt a cosine LR schedule and set the weights decay and momentum as 5e-4 and 0.9. For ImageNet, we train the baseline ResNet-50 and MobileNetV1 [17] using SGD with batch sizes of 256 and 512, respectively. We adopt the same data augmentation scheme as [14], the cosine LR scheduler with initial LR of 0.1, weight decay of 1e-4 and momentum of 0.9.
For IWSLT'14, we train an Encoder-Decoder Transformer (6 attention blocks each) [34]. We set width as  $d_{model} = 1 / 4d_{ffn} = 512$ , the number of heads  $n_{head} = 8$  and  $d_k = d_q = d_v = d_{model} / n_{head} = 64$ . We train the model using Adam for 20 epochs with learning rate 1e-3 and  $(\beta_1, \beta_2) = (0.9, 0.98)$ . Batch size is 1500 and we use 4000 warm-up iterations.
Table 2: Growing ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10.
Method	ResNet-20	VGG-11	MobileNetv1
Train Cost(%)	Test Accuracy(%) ↑	Train Cost(%)	Test Accuracy(%) ↑	Train Cost(%)	Test Accuracy(%) ↑
Large Baseline	100	92.62 ± 0.15	100	92.14 ± 0.22	100	92.27 ± 0.11
Net2Net	54.90	91.60 ± 0.21	52.91	91.78 ± 0.27	53.80	90.34 ± 0.20
Splitting	70.69	91.80 ± 0.10	63.76	91.88 ± 0.15	65.92	91.50 ± 0.06
FireFly-split	58.47	91.78 ± 0.11	56.18	91.91 ± 0.15	56.37	91.56 ± 0.06
FireFly	68.96	92.10 ± 0.13	60.24	92.08 ± 0.16	62.12	91.69 ± 0.07
Ours	54.90	92.53 ± 0.11	52.91	92.34 ± 0.15	53.80	92.01 ± 0.10
Table 3: Growing ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100.
Method	ResNet-18	VGG-19	MobileNetv1
Train Cost(%) ↓Accuracy(%) ↑	Test Cost(%) ↓Accuracy(%) ↑	Train Cost(%) ↓Accuracy(%) ↑	Test Cost(%) ↓Accuracy(%) ↑	Train Cost(%) ↓Accuracy(%) ↑	Test Cost(%) ↓Accuracy(%) ↑
Large Baseline	100	78.36 ± 0.12	100	72.59 ± 0.23	100	72.13 ± 0.13
Net2Net	52.63	76.48 ± 0.20	52.08	71.88 ± 0.24	52.90	70.01 ± 0.20
Splitting	68.01	77.01 ± 0.12	60.12	71.96 ± 0.12	58.39	70.45 ± 0.10
FireFly-split	56.11	77.22 ± 0.11	54.64	72.19 ± 0.14	54.36	70.69 ± 0.11
FireFly	65.77	77.25 ± 0.12	57.48	72.79 ± 0.13	56.49	70.99 ± 0.10
Ours	52.63	78.12 ± 0.15	52.08	73.26 ± 0.14	52.90	71.53 ± 0.13
Implementation Details. We compare with the growing methods Net2Net [4], Splitting [25], FireFly-split, FireFly [38] and GradMax [10]. In our method, noise for symmetry breaking is 0.001 to the norm of the initialization. We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers (e.g., Adam, AvaGrad).
For image classification, we run the comparison methods except GradMax alongside our algorithm for all architectures under the same growing scheduler. For the architecture scheduler, we set  $p_c$  as 0.2 and  $C_0$  as 1/4 of large baselines in Eq. 6 for all layers and grow the seed architecture within  $N = 9$  stages towards the large ones. For epoch scheduler, we set  $p_t$  as 0.2,  $T_0$  as 8, 10, and 4 in Eq. 7 on CIAFR-10, CIFAR-100, and ImageNet respectively. Total training epochs  $T_{total}$  are the same as the respective large fixed-size models. We train the models and report the results averaging over 3 random seeds.
For machine translation, we grow the encoder and decoder layers' widths while fixing the embedding layer dimension for a consistent positional encoding table. The total number of growing stages is 4, each trained for 5 epochs. The initial width is  $1/8$  of the large baseline (i.e.,  $d_{model} = 64$  and  $d_{k,q,v} = 8$ ). We set the growing ratio  $p_c$  as 1.0 so that  $d_{model}$  evolves as 64, 128, 256 and 512.
We train all the models on an NVIDIA 2080Ti 11GB GPU for CIFAR-10, CIFAR-100, and IWSLT'14, and two NVIDIA A40 48GB GPUs for ImageNet.
4.1 CIFAR Results