SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} h ^ {(1)} = W ^ {(1)} x = \left[ \begin{array}{c} w _ {1, 1} ^ {(1)} \\ w _ {2, 1} ^ {(1)} \end{array} \right] x = \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \end{array} \right] \\ h ^ {(2)} = W ^ {(2)} h ^ {(1)} = \left[ \begin{array}{c c} w _ {1, 1} ^ {(2)} & w _ {1, 2} ^ {(2)} \\ w _ {2, 1} ^ {(2)} & w _ {2, 2} ^ {(2)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(2)} \\ h _ {2} ^ {(2)} \end{array} \right] \\ h ^ {(3)} = W ^ {(3)} h ^ {(2)} = \left[ \begin{array}{c c} w _ {1, 1} ^ {(3)} & w _ {1, 2} ^ {(3)} \\ w _ {2, 1} ^ {(3)} & w _ {2, 2} ^ {(3)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(2)} \\ h _ {2} ^ {(2)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(3)} \\ h _ {2} ^ {(3)} \end{array} \right] \\ h ^ {(4)} = h ^ {(3)} + h ^ {(1)} = \left[ \begin{array}{c} h _ {1} ^ {(3)} \\ h _ {2} ^ {(3)} \end{array} \right] + \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(4)} \\ h _ {2} ^ {(4)} \end{array} \right] \\ y = W ^ {(4)} h ^ {(4)} = \left[ \begin{array}{c c} w _ {1, 1} ^ {(4)} & w _ {1, 2} ^ {(4)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(4)} \\ h _ {2} ^ {(4)} \end{array} \right] \\ \end{array}
$$
Now assume that we want to grow the dimension of the network's hidden activations from 2 to 4 (i.e.,  $h^{(1)}, h^{(2)}, h^{(3)}, h^{(4)}$ , which are 2-dimensional, should become 4-dimensional each).
$$
\begin{array}{l} h ^ {(1)} = W ^ {(1)} x = \left[ \begin{array}{c} w _ {1, 1} ^ {(1)} \\ w _ {2, 1} ^ {(1)} \\ w _ {3, 1} ^ {(1)} \\ w _ {4, 1} ^ {(1)} \end{array} \right] x = \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \\ h _ {3} ^ {(1)} \\ h _ {4} ^ {(1)} \end{array} \right] \\ h ^ {(2)} = W ^ {(2)} h ^ {(1)} = \left[ \begin{array}{c c c c} w _ {1, 1} ^ {(2)} & w _ {1, 2} ^ {(2)} & w _ {1, 3} ^ {(2)} & w _ {1, 4} ^ {(2)} \\ w _ {2, 1} ^ {(2)} & w _ {2, 2} ^ {(2)} & w _ {2, 3} ^ {(2)} & w _ {2, 4} ^ {(2)} \\ w _ {3, 1} ^ {(2)} & w _ {3, 2} ^ {(2)} & w _ {3, 3} ^ {(2)} & w _ {3, 4} ^ {(2)} \\ w _ {4, 1} ^ {(2)} & w _ {4, 2} ^ {(2)} & w _ {4, 3} ^ {(2)} & w _ {4, 4} ^ {(2)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \\ h _ {3} ^ {(1)} \\ h _ {4} ^ {(1)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(2)} \\ h _ {2} ^ {(2)} \\ h _ {3} ^ {(2)} \\ h _ {4} ^ {(2)} \end{array} \right] \\ h ^ {(3)} = W ^ {(3)} h ^ {(2)} = \left[ \begin{array}{c c c c} w _ {1, 1} ^ {(3)} & w _ {1, 2} ^ {(3)} & w _ {1, 3} ^ {(3)} & w _ {1, 4} ^ {(3)} \\ w _ {2, 1} ^ {(3)} & w _ {2, 2} ^ {(3)} & w _ {2, 3} ^ {(3)} & w _ {2, 4} ^ {(3)} \\ w _ {3, 1} ^ {(3)} & w _ {3, 2} ^ {(3)} & w _ {3, 3} ^ {(3)} & w _ {3, 4} ^ {(3)} \\ w _ {4, 1} ^ {(3)} & w _ {4, 2} ^ {(3)} & w _ {4, 3} ^ {(3)} & w _ {4, 4} ^ {(3)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(2)} \\ h _ {2} ^ {(2)} \\ h _ {3} ^ {(2)} \\ h _ {4} ^ {(2)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(3)} \\ h _ {2} ^ {(3)} \\ h _ {3} ^ {(3)} \\ h _ {4} ^ {(3)} \end{array} \right] \\ h ^ {(4)} = h ^ {(3)} + h ^ {(1)} = \left[ \begin{array}{c} h _ {1} ^ {(3)} \\ h _ {2} ^ {(3)} \\ h _ {3} ^ {(3)} \\ h _ {4} ^ {(3)} \end{array} \right] + \left[ \begin{array}{c} h _ {1} ^ {(1)} \\ h _ {2} ^ {(1)} \\ h _ {3} ^ {(1)} \\ h _ {4} ^ {(1)} \end{array} \right] = \left[ \begin{array}{c} h _ {1} ^ {(4)} \\ h _ {2} ^ {(4)} \\ h _ {3} ^ {(4)} \\ h _ {4} ^ {(4)} \end{array} \right] \\ y = W ^ {(4)} h ^ {(4)} = \left[ \begin{array}{c c c c} w _ {1, 1} ^ {(4)} & w _ {1, 2} ^ {(4)} & w _ {1, 3} ^ {(4)} & w _ {1, 4} ^ {(4)} \end{array} \right] \left[ \begin{array}{c} h _ {1} ^ {(4)} \\ h _ {2} ^ {(4)} \\ h _ {3} ^ {(4)} \\ h _ {4} ^ {(4)} \end{array} \right] \\ \end{array}
$$
We added 2 rows to the input layer's weights  $W^{(1)}$  (to increase its output dimension from 2 to 4), added 2 rows and 2 columns to the hidden layer's weights  $W^{(2)}$ ,  $W^{(3)}$  (to increase its output dimensions from 2 to 4, and to increase its input dimension from 2 to 4 so they are consistent with the previous layers), and added 2 columns to the output layer's weights  $W^{(4)}$  (to increase its input dimension from 2 to 4 so it is consistent with the increase
Table 8: Concrete growing example.
Layer	Initial Arch	After Growth	New weights added
W(1)	(1,2)	(1,4)	4 × 1 − 2 × 1 = 2
W(2)	(2,2)	(4,4)	4 × 4 − 2 × 2 = 12
W(3)	(2,2)	(4,4)	4 × 4 − 2 × 2 = 12
W(4)	(2,1)	(4,1)	4 × 1 − 2 × 1 = 2
in dimensionality of  $h^{(4)}$ . Note that  $h^{(3)}$  and  $h^{(1)}$  still maintain matching shapes (to be added up) in the residual block since we grow  $W^{(1)}$  and  $W^{(3)}$ 's output dimensions with the same rate.
We summarize the architectural growth in terms of  $C_{in}$  and  $C_{out}$  (omitting kernel size  $k$ ) in Table 8. We also show the growing for CNN hidden layer in Figure 8(a) and residual blocks in Figure 8(b) for better illustration.
Note that this example shows how growing works in general; our specific method also includes particulars as to what values are used to initialize the newly-added weights, as well as modifications to optimizer state.
C Generalization to Other Optimizers
We generalize our LR adaptation rule to Adam [20] and AvaGrad [31] in Table 9. Both methods are adaptive optimizers where different heuristics are adopted to derive a parameter-wise learning rate strategy, which provides primitives that can be extended using our stage-wise adaptation principle for network growing. For example, vanilla Adam adapts the global learning rate with running estimates of the first moment  $m_t$  and the second moment  $v_t$  of the gradients, where the number of global training steps  $t$  is an integer when training a fixed-size model. When growing networks, our learning rate adaptation instead considers a vector  $t$  which tracks each subcomponent's 'age' (i.e., number of steps it has been trained for). As such, for a newly grown subcomponent at a stage  $i > 0$ ,  $t[i]$  starts as 0 and the learning rate is adapted from  $m_t / \sqrt{v_t}$  (global) to  $\frac{m_{t[i]}\backslash m_{t[i-1]}}{\sqrt{v_{t[i]}\backslash v_{t[i-1]}}}$  (stage-wise). Similarly, we also generalize our approach to AvaGrad by adopting  $\eta_t, d_t, m_t$  of the original paper as a stage-wise variables.
Preserving Optimizer State/Buffer. Essential to adaptive methods are training-time statistics (e.g., running averages  $m_t$  and  $v_t$  in Adam) which are stored as buffers and used to compute parameter-wise learning rates. Different from fixed-size models, parameter sets are expanded when growing networks, which in practice requires re-instantiating a new optimizer at each growth step. Given that our initialization scheme maintains functionality of the network, we are also able to preserve and inherit buffers from previous states, effectively maintaining the optimizer's state intact when adding new parameters. We experimentally investigate the effects of this state preservation.
Table 9: Rate adaptation rules for Adam [20] and AvaGrad [31].
Our LR Adaptation
Adam	0-th Stage	mt[0]/√vt[0]
i-th Stage	mt[i] \ mt[i-1] / √vt[i] \ vt[i-1]
AvaGrad	0-th Stage	ηt[0] / [|ηt[0] / √dt[0]|2] ⊙ mt[0]
i-th Stage	ηt[i] \ nt[i-1] / [|ηt[i] \ nt[i-1] / √dt[i] - dt[i-1]|2] ⊙ (mt[i] \ mt[i-1])
Table 10: Generalization to Adam and AvaGrad for ResNet-20 on CIFAR-10.
Optimizer	Training Method	Preserve Opt. Buffer	Train Cost (%)	Test Acc. (%)
Adam	Large fixed-size	NA	100	92.29
Adam	Growing	No	54.90	91.44
Adam	Growing	Yes	54.90	91.61
Adam+our RA.	Growing	Yes	54.90	92.13
AvaGrad	Large fixed-size	NA	100	92.45
AvaGrad	Growing	No	54.90	90.71
AvaGrad	Growing	Yes	54.90	91.27
AvaGrad+our RA.	Growing	Yes	54.90	91.72
Optimizer	Test Acc. (%)
Standard SGD	91.95 ± 0.09
SGD with Layer-wise Adapt. (LARS)	91.32 ± 0.11
Ours	92.53 ± 0.11
Results with Adam and AvaGrad. Table 10 shows the results of growing ResNet-20 on CIFAR-10 with Adam and Avagrad. For the large, fixed-size baseline, we train Adam with  $lr = 0.1$ ,  $\epsilon = 0.1$  and AvaGrad with  $lr = 0.5$ ,  $\epsilon = 10.0$ , which yields the best results for ResNet-20 following [31]. We consider different settings for comparison: (1) optimizer without buffer preservation: the buffers are refreshed at each new growing phase; (2) optimizer with buffer preservation: the buffer/state is inherited from the previous phase, hence being preserved at growth steps; (3) optimizer with buffer and rate adaptation (RA): applies our rate adaptation strategy described in Table 9 while also preserving internal state/buffers. We observe that (1) consistently underperforms (2), which suggests that preserving the state/buffers in adaptive optimizers is crucial when growing networks. Option (3) bests the other settings for both Adam and AvaGrad, indicating that our rate adaptation strategy generalizes effectively to Adam and AvaGrad for the growing scenario. Together, these also demonstrate that our method has the flexibility to incorporate different statistics that are tracked and used by distinct optimizers, where we take Adam and AvaGrad as examples. Finally, our proposed stage-wise rate adaptation strategy can be employed with virtually any optimizer.
Comparison with Layer-wise Adaptive Optimizer. We also consider LARS [12, 43], a layer-wise adaptive variant of SGD, to compare different adaptation concepts: layer-wise versus layer + stage-wise (ours). Note that although LARS was originally designed for training with large batches, we adopt a batch size of 128 when growing ResNet-20 on CIFAR-10. We search the initial learning rate (LR) for LARS over  $\{1\mathrm{e} - 3,2\mathrm{e} - 3,5\mathrm{e} - 3,1\mathrm{e} - 2,2\mathrm{e} - 2,5\mathrm{e} - 2,1\mathrm{e} - 1,2\mathrm{e} - 1,5\mathrm{e} - 1\}$  and observe that a value of 0.02 yields the best results. We adopt the default initial learning rate of 0.1 for both standard SGD and our method. As shown in Table 11, LARS underperforms both standard SGD and our adaptation strategy, suggesting that layer-wise learning rate adaptation by itself – i.e., without accounting for stage-wise discrepancies – is not sufficient for successful growing of networks.
D More Analysis on Rate Adaptation
We show additional plots of stage-wise rate adaptation when growing a ResNet-20 on CIFAR-10. Figure 9 shows the of adaptation factors based on the LR of the seed architecture from 1st to 8th stages (the stage index starts at 0). We see an overall trend that for newly-added weights, its learning rate starts at  $>1\times$  of the base LR then quickly adapts to a relatively stable level. This demonstrates that our approach is able to efficiently and automatically adapt new weights to gradually and smoothly fade in throughout the current stage's optimization procedure.
We also note that rate adaptation is a general design that different subnets should not share a global learning rate. The RA formulation is designed empirically.  $\max(1, ||\boldsymbol{W}_i \setminus \boldsymbol{W}_{i-1}||)$  is a plausible implementation choice,
Table 11: Comparisons among standard SGD, LARS, and our adaptation method for growing ResNet-20 on CIFAR-10.
RA Implementation Choice	Test Acc. (%)
NA (Standard SGD)	91.62 ± 0.12
max(1, ||Wi \ Wi-1||)	91.42 ± 0.12
Ours	92.53 ± 0.11
Table 12: Comparisons among different RA implementation choices for growing ResNet-20 on CIFAR-10.
(a) 1-st Stage
(b) 2-nd Stage
(c) 3-rd Stage
(d) 4-th Stage
(e) 5-th Stage
(f) 6-th Stage
(g) 7-th Stage
Figure 9: Visualization of rate adaptation factor dynamics across all growing stages (except 0-th)
(h) 8-th Stage
based on the assumption that new weights must have a higher learning rate. We conducted experiments by growing ResNet-20 on CIFAR-10. As shown in Table 12, we see that this alternative does not work better than our original design, and even underperforms standard SGD.
E More Visualizations on Sub-Component Gradients
We further compare global LR and our rate adaptation by showing additional visualizations of sub-component gradients of different layers and stages when growing ResNet-20 on CIFAR-10. We select the 2nd (layer1-block1-conv1) and 17th (layer3-block2-conv2) convolutional layers and plot the gradients of each sub-component at the 3rd and 5th growing stages, respectively, in Figures 10, 11, 12, 13. These demonstrate that our rate adaptation strategy is able to re-balance and stabilize the gradient's contribution of different subcomponents, hence improving the training dynamics compared to a global scheduler.
F Simple Example on Fully-Connected Neural Networks
Additionally, we train a simple fully-connected neural network with 8 hidden layers on CIFAR-10 - each hidden layer has 500 neurons and is followed by ReLU activations. The network is has a final linear layer with 10 neurons for classification. Note that each CIFAR-10 image is flattened to a 3072-dimensional  $(32 \times 32 \times 3)$  vector as prior to being given as input to the network. We consider two variants of this baseline network by adopting training epochs (costs)  $\in \{25(1 \times), 50(2 \times)\}$ . We also grow from a thin architecture to the original one within 10 stages, each stage consisting of 5 epochs, where the number of units of each hidden layer grows from 50 to 100, 150, ..., 500. The total training cost is equivalent to the fixed-size one trained for 25 epochs. We train all baselines using SGD, with weight decay set as 0 and learning rates sweeping over  $\{0.01, 0.02, 0.05, 0.1, 0.2, 0.5\}$ : results are shown in Figure 14(a). Compared to standard initialization (green),
(a) Using Global
Figure 10: Gradients of 2nd conv at 3rd stage. Figure 11: Gradients of 2nd conv at 5th stage.
(b) Using RA
(a) Using Global
Figure 12: Gradients of 17th conv at 3rd stage. Figure 13: Gradients of 17th conv at 5th stage.
(a) Train Loss
Figure 14: Results of simple fully-connected neural network.
the loss curve given by growing variance transfer (blue) is more similar to the curve of the large baseline - all using standard SGD - which is also consistent with the observations when training model of different scales separately [41]. Rate adaptation (in red) further lowers training loss. Interestingly, we observe in Figure 14(b) that the test accuracy behavior differs from the training loss one given in Figure 14(a), which may suggest that regularization is missing due to, for example, the lack of parameter-sharing schemes (like CNN) in this fully-connected network.
G More Comparisons with GradMax [10]
For CIFAR-10 and CIFAR-100, GradMax used different models for growing and we did not re-implement GradMax on both datasets. Also, generalizing such gradient-based growing methods to the Transformer architecture is nontrivial. As such, we only cover MobileNet on ImageNet which is used in both ours and GradMax. Our accuracy outperforms GradMax by 1.3 while lowering training costs, which is significant to demonstrate the benefit of our method. We also trained our method to grow WRN-28-1 (w/wo BatchNorm, used in GradMax paper) on CIFAR-10 and CIFAR-100 and compare it with GradMax in Table 13. We see that ours still consistently outperforms GradMax.
Table 13: Comparison with GradMax.
Method	CIFAR-10 (w BN)	CIFAR-10 (w/o BN)	CIFAR-100 (w/o BN)
Train Cost (%) ↓	Test Acc. (%) ↑	Train Cost (%) ↓	Test Acc. (%) ↑	Train Cost (%) ↓	Test Acc. (%) ↑
Large Baseline	100	93.40 ± 0.10	100	92.90 ± 0.20	100	69.30 ± 0.10
GradMax [10]	77.32	93.00 ± 0.10	77.32	92.40 ± 0.10	77.32	66.80 ± 0.20
Ours	58.24	93.29 ± 0.12	58.24	92.61 ± 0.10	58.24	67.83 ± 0.15
H Extension to Continuously Incremental Datastream
Another direct and intuitive application for our method is to fit continuously incremental datastream where  $D_0 \subset D_1, \ldots \subset D_n, \ldots \subset D_{N-1}$ . The network complexity scales up together with the data so that larger capacity can be trained on more data samples. Orthogonalized SGD (OSGD) [35] address the optimization difficulty in this context, which dynamically re-balances task-specific gradients via prioritizing the specific loss influence. We further extend our optimizer by introducing a dynamic variant of orthogonalized SGD, which progressively adjusts the priority of tasks on different subnets during network growth.
Suppose the data increases from  $D_{n-1}$  to  $D_n$ , we first accumulate the old gradients  $G_{n-1}$  using one additional epoch on  $D_{n-1}$  and then grow the network width. For each batch of  $D_n$ , we first project gradients of the new architecture (  $n$ -th stage), denoted as  $G_n$ , onto the parameter subspace that is orthogonal to  $G_{n-1}^{pad}$ , a zero-padded version of  $G_{n-1}$  with desirable shape. The final gradients  $G_n^*$  are then calculated by re-weighting the original  $G_n$  and its orthogonal counterparts:
$$
\boldsymbol {G} _ {n} ^ {*} = \boldsymbol {G} _ {n} - \lambda * \operatorname {p r o j} _ {\boldsymbol {G} _ {n - 1} ^ {p a d}} \left(\boldsymbol {G} _ {n}\right), \quad \lambda : 1 \rightarrow 0 \tag {21}
$$
where  $\lambda$  is a dynamic hyperparameter which weights the original and orthogonal gradients. When  $\lambda = 1$ , subsequent outputs do not interfere with earlier directions of parameters updates. We then anneal  $\lambda$  to 0 so that the newly-introduced data and subnetwork can smoothly fade in throughout the training procedure.
Implementation Details. We implement the task in two different settings, denoted as 'progressive class' and 'progressive data' on CIFAR-100 dataset within 9 stages. In the progressive class setting, we first randomly select 20 classes in the first stage and then add 10 new classes at each growing stage. In the progressive data setting, we sequentially sample a fraction of the data with replacement for each stage, i.e.,  $20\%$ ,  $30\%$ , ...,  $100\%$ .
ResNet-18 on Continuous CIFAR-100. We evaluate our method on continuous datastreams by growing a ResNet-18 on CIFAR-100 and comparing the final test accuracies. As shown in Table 14, compared with the large baseline, our growing method achieves  $1.53 \times$  cost savings with a slight performance degradation in both settings. The dynamic OSGD variant outperforms the large baseline with  $1.46 \times$  acceleration, demonstrating that the new extension improves the optimization on continuous datastream through gradually re-balancing the task-specific gradients of dynamic networks.
Table 14: Growing ResNet-18 on incremental CIFAR-100.
Method	Progressive Class	Progressive Data
Train Cost (%) ↓	Test Acc. (%) ↑	Train Cost (%) ↓	Test Acc. (%) ↑
Large fixed-size Model	100	76.80	100	76.65
Ours	65.36	76.50	65.36	76.34
Ours-Dynamic-OSGD	68.49	77.53	68.49	77.85