GradMax: Growing Neural Networks using Gradient Information
Authors: Utku Evci
Bart van Merrienboer
Thomas Unterthiner
Fabian Pedregosa
Max Vladymyrov
Venue: ICLR 2022 Poster (2022)
Keywords: efficient training
efficient
computer vision
architecture search

Abstract: The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights  and  efficiently  find  the  optimal  initialization  by  means  of  the  singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures. We open sourced our code at https://github.com/google-research/growneuron

Published as a conference paper at ICLR 2022
GRADMAX: GROWING NEURAL NETWORKS USING GRADIENT INFORMATION
Utku Evei, Bart van Merriënboer, Thomas Unterthiner, Max Vladymyrov, Fabian Pedregosa Google Research, Brain Team (evcu, bartvm, unterthi heer, mxv, wer,,,,,,,edeeeeeeeee pedreppdeeegsseeeeooo hempeeddeeee......eeeeeeeggoo/go (poogeggeesss)eogllle..oom
ABSTRACT
The architecture and the parameters of neural networks are often optimized in- dependently, which requires costly retraining of the parameters whenever the ar- chitecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients the new weights and efficiently find the optimal initialization by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and archi- tectures!.
INTRODUCTION
The architecture of deep learning models influences model's inductive biases and has been shown to have a crucial effect on both the training speed and generalization (dÁscoli et al., 2019; Neyshabur, 2020). Searching for the best architecture for given task is an active research area with diverse approaches, including neural architecture search (NAS) (Elsken et al., 2019), pruning (Liu et al., 2018), and evolutionary algorithms (Stanley & Mikkullinen, 2002). Most of these ap- proaches are costly, as they require large search spaces or large architectures to start with. In this work consider alternative approach: Can we start with small network and learn efficient architecture without ever needing a large network or exceeding the size of our final architecture?
The idea of incrementally increasing the size of model has been used in many settings such as boosting (Friedman, 2001), continual learning (Rusu et al., 2016), architecture search (Elsken et al., 2017: Cortes al., 2017), optimization (Fukumizu & Amari, 2000; Caccia al., 2022), and rein- forcement learning (Berner et al., 2019). Despite the wide range of applications of growing neural networks, the initialization of newly grown neurons is rarely studied. Existing work on growing neu- ral networks either adds new neurons randomly (Chen et al., 2016; Berner et al., 2019) or chooses them with the aim of decreasing the training loss (Bengio et al., 2006; Liu al., 2019; Wu al., 2020a). In this work we take a different approach. Instead of improving the training objective im- mediately through growing, we focus on improving the subsequent training dynamics. As we will show, this has a longer-lasting effect than the greedy approach of improving loss during growing.
Our main contribution a new growing method that maximizes the gradient norm of newly added neurons, which we call GradMax. We show that this gradient maximization problem can be solved in a closed form using the singular value decomposition (SVD) and initializing the new neurons using the top singular vectors. We show that the increased gradient norm persists during future steps, which in return yields faster training.
2 GROWING NEURAL NETWORKS
Neural network learning is often formalized as the following optimization problem:
arg min (( [L(f(z),)) ((e=))-([((((()..))) EES
We open source code https: / /gi cott:to.oo//////// www.://oooee-essrrrrreeeeeeeeeee e-rrsearcc//rrooeeurr

---

Published as a conference paper at ICLR 2022
where S is set of neural networks, L is loss function and D is data set consisting of inputs and outputs y). Most often the set S is constrained to a single architecture (e.g., ResNet-18 (Zagoruyko & Komodakis, 2016)) and only the parameters of the model are learned. However, hand-crafted architectures are often suboptimal and various approaches aim to optimize the architecture itself together with its parameters.
Methods such as random search (Bergstra & Bengio, 2012), NAS, and pruning search a larger space that contains variety of architectures. Growing neural networks has the same goal: It learns the architecture and weights jointly. It aims to achieve this by incrementally increasing the size of the model. Since this approach never requires training a large model from scratch it needs less memory and compute than pruning methods (Yuan et al., 2021). Reducing the cost of finding efficient archi- tectures important given the ever-growing size of architectures and the accompanying increase in environmental footprint (Thompson et al., 2021).
Growing neural networks: When, where and how? Algorithms for growing neural networks start training with a smaller seed architecture. Then over the course of the training new neurons are added to the seed architecture, either increasing the width of the existing layers or creating new layers. Algorithms for growing neural networks should address the following questions:
1. When to add new neurons? For instance, some methods (Liu et al., 2019; Kilcher et al., 2019) require the training loss to plateau before growing, whereas others grow using predefined schedule.
2. Where to add new capacity? We can add new neurons to the existing layers or create new layers among the existing ones.
3. How initialize the new capacity?
In this work we mainly focus on the question of how and introduce a new initialization method for the new neurons. Our approach can also be used to guide when and where to grow new neurons. However, in order to make our comparison with other initialization methods fair we keep the growing schedule (where and when) fixed.
How to grow new neurons? Suppose that during training we would like to grow i new neurons layer as depicted in Figure After the growth, new neurons are appended to the existing weight matrices as follows:
We Whew
W
W E+1
We+1
Waew
(3)
A desirable property of growing algorithms is preserve the information that the network has leamned. Therefore the initialization of new neurons should ensure our neural network has the same outputs before and after growth.
We describe two main approaches to growing: splitting and adding. Splitting (Chen et al., 2016; Liu et al., 2019) duplicates existing neurons and adjusts outgoing weights so that the output in the next layer is unchanged. However, splitting has some limitations: (1) It creates neurons with the same weights and small changes required to break symmetry; (2) It can't 't be used for growing new layers it requires existing neurons to begin with.
Another line of work focuses on adding new neurons while ensuring that the output does not change by setting either Weew or W to zero. This approach doesn't 't have the limitations mentioned above and often results in better performance (Wu et al., 2020a). Below we summarize two methods that use this approach:
Random Berner et al. (2019) initialize new neurons randomly when growing new layers.
Firefly Wu et al. (2020a) combine splitting and adding random neurons. Random neurons are trained for a few steps to reduce loss directly and the most promising neurons are selected for growing.
Previous work focused either on solely keeping the output of the neural network the same (Random) or immediately reducing the training objective (Firefly). In this work take different approach
2

---

Published as a conference paper at ICLR 2022
We+1
W
Wnew
Whew
t-1
W
W
Before
Growth
GradMax
Figure 1: Schematic view of the GradMax algorithm. Growing new neurons requires initializing incoming (Wgr)) and outgoing #+1 weights for the new neuron. GradMax incoming weights to zero (dashed lines) in order to keep the output unchanged, and initializes outgoing weights us- ing SVD (equation I1). This maximizes the gradients on the incoming weights with the aim of accelerating training.
and focus on improving the training dynamics. We initialize the weights that the network's output remains unchanged while trying to improve the training dynamics by maximizing the gradient norm. The hypothesis is that this accelerates training and leads better model in the long run.
3 GRADMAX
This section describes our main contribution: GradMax, method that maximizes the gradient norm of the new weights. We constrain the norm of the new weights to avoid the trivial solution where the weight norm tends to infinity. We also require that the network output unchanged when the neuron is added. Note that, due the latter, gradients of the existing weights are unchanged during growth and therefore we maximize the gradients on the new weights introduced.
arg max Whi
ED dW
OL
+
ED
OL
s.t.
W Wiew
F
W E+1
S c
(4)
WW E+1
211
Motivation: Large gradients lead to large objective decrease Con- sider running gradient descent function that is differentiable with 3-Lipschitz gradient. A classical result for this class of functions is that the decrease in objective function after one step of gradient descent with step-size 1/B is upper bounded by ((W) I L(W (Nesterov, 2003). This bound decreases as the norm of the gradient increases.
((W) |
This observation implies that the larger the gradient, the larger the ex- pected decrease (assuming constant Lipschitz constant). Hence, in- terleaving gradient maximizing steps within training will lead to an im- proved decrease in later iterations. use this observation the context of growing neural networks and propose method that interleaves the normal training process with steps that maximize the gradient norm by adding new units, which now describe in detail.
L(W))
W
The general maximization problem in eq. (4) is non-trivial to solve. However, we show that with some simplifying assumptions we can find an approximate solution using singular value decomposition (SVD). Here we derive this solution using fully connected layers and share derivation for convolutional layers at Appendix B.
GradMax
Consider 3 fully connected layers denoted with indices é- 1, l and t+ 1 and the following recursive definition:
Wchr-1 f(ze),
(5)
(6) where subscripts denote layer-indices. Let M, denote the number of units in layer L, and f the activation function. The vectors zı, h Mr are pre-activations and activations respectively, with hg = æ being an input sampled from the dataset D. The entries of We E R R M are the parameters of the layer.
When growing lo neurons the weight matrices W, and W are replaced by W and W E+1 as defined in eqs. (2) and (3). We denote the pre-activations and activations of the new neurons with
3

---

Published as a conference paper at ICLR 2022
26 =/e and h The gradients of the weights can be derived using the chain rule:
aL
aL Ōze+1
(=
+
neww E+1
h
(7)
DWw aL
ÖL θze+1
h
new,
(8)
DW
E+1
The simplifying assumptions that we will now make are that W = 0 and that f(0) = 0 with gradient (0) = 1. Note that this guarantees that W Wayy h 0, independent of the training data. Moreover, simplifies the gradients to
ØL OWnew aL 0,
neww T OL
h
(9)
0ze+1
(10)
w 4+1
which reduces our problem to
aL
arg max
W
new, E+1
ED
0zz+1
h
[-1
s.t.
Wiew [+1 F S c.
(11)
The solution to this maximization problem is found in closed form by setting the columns of Waey as the top & left-singular vectors of the matrix ED 88 h- and scaling them by [(.r)....))
aL
(where is the i-th largest singular value). In order to make fair comparison between different methods we scale each initialization such that their norm is equal to the same value, i.e., mean norm of the existing neurons (similar to Liu et al. (2017)).
Note that is feasible to also use the singular values to guide where and when to grow, since the singular values are equal to the value of the maximized optimization problem above. For example, neurons could be added when the singular values meet a certain threshold, and layers to grow could be chosen depending which have the largest singular values. We leave this for future work.
Non-linearities and normalization Note that assumptions that f(0) 0 and f'(0) to common activation functions such as rectified linear units (if zero is included in the linear part) and the hyperbolic tangent. Other functions could be adapted to fit this definition as well (e.g.. by shifting the output of the logistic function by -
apply
In fact our derivation holds for any such that f0( a, * 0. This means that batch nor- malization can also be used, since it meets these assumptions. However, care must be taken with activation functions (or their derivatives) that are unstable around zero. For example, batch normal- when all inputs are zero, which can lead to large jumps in parameter
ization scales the gradients by = space (Lewkowyez et al., 2020, and Appendix C).
GradMax-66 -ppiimiiee (GradMaxOpt) One can also directly optimize eq. (11) using an iterative method such as projected gradient descent. Our experiments show that this approach generally does not work as well as using the SVD, highlighting the benefit of having a closed-form solution.
However, if the outgoing weights set to zero (Wheew 0) instead of the incoming weights then the solution can no longer be found using SVD and direct optimization of eq. (4) could provide a so- lution. This could be preferable in some situations since it removes the constraints the activation function. Moreover, it can avoid the unstable behaviour of functions such as batch normalization.
aL h over the Full gradient estimation Solving eq. (11) requires calculating the gradient 8z241 full dataset. This can be expensive so in practical applications we propose using large minibatch instead. the minibatch gradient sufficiently close to the full gradient then the top singular vectors are likely be close well (Stewart, 1998). We validate this experimentally in Section 4.2.
EXPERIMENTS
We evaluate gradient maximizing growing (GradMax) using small multilayer perceptrons (MLPs) and popular deep convolutional architectures. In Section 4.1 we focus verifying the effectiveness
4

---

Published as a conference paper at ICLR 2022
Bondom GrocMax GrachaacOpt
Adjusted Gr Gradient Vorm
0.55
Random
Croet
0.04
Growth
4x10-1
Loss
Cruwth
3x10-1
2x10
0.00
330 Step Number
Growth Number
Training Steps
(a) After Growth
(b) During Training
(c) After Steps
Figure 2: (a) We measure the norm of the gradients with respect to W after growing a single neu- ron starting from checkpoints generated during the Random (Random) growth experiments. Since Random and GradMaxOpt are stochastic we repeat those experiments 10 times. (b) Gradient norm of the flattened parameters (both layers) throughout training. (c) Similar to (a), we load checkpoints from each growth step (Growth 1-5) and grow a new neuron using GradMax (Sg) and Random (fr)- Then continue training for 500 steps and plot the difference in training loss (i.c., L(fr) (fg))) The confidence intervals are defined over 5 repetition of the experiment described above.
of GradMax and in Section 4.2 evaluate GradMax on common image classification benchmarks and show the effect of different hyper-parameter choices.
We implement GradMax using Tensorflow (Abadi et al., 2015) and modify the implementation of the standard ReLU activation to output sub-gradient at O. In Appendix A we share the implementation details of GradMax and show how it can be used for growing new layers. We will open-source our implementation upon publication.
4.1 TEACHER- TEACHER-STUDENT EXPERIMENTS WITH MLPs
We have main goals in this section. First, we empirically verify that the gradient norm is sig- nificantly increased using method. Then, verify our hypothesis that increasing the gradient norm at a given step improves the training dynamics beyond the growing step. Finally, we gather experimental evidence that networks grown with GradMax can achieve better training loss and gen- eralization compared to baselines and other methods in a teacher-student setting.
Teacher-Student task We use teacher-student task (Lawrence et al., 1997) to compare different growing algorithms. In this setting student network must lear the function of a given teacher network. Our teacher network fr consists of m, input nodes, m, hidden nodes and m, output nodes (denoted with m, m7, m,)) We initialize weights of St randomly from the range [-1, 1] and then sample N = 1000 training points D, from Gaussian distribution with zero mean and unit variance and pass through ft to obtain target values Dy. With this training data we train various student networks (f,) minimizing the squared loss between (D2) and Dy.
A key property this teacher-student setting is that when the student network has the same archi- tecture the teacher network, the optimization problem has multiple global minima with 0 training loss. Here we highlight results using fully connected layers alone. In Appendix D we show addi- tional validation experiments well as repeat experiments from this section with convolutions and batch normalization.
Verifying GradMax In our initial experiments we use teacher network of size 20 10: 10. All growing student networks begin with a smaller hidden layer of 20 10. We grow the hidden layer to match the teacher architecture in 5 growth steps performed every 200 training steps. We also train two baseline networks with sizes 20 from scratch. After the final growth we perform 500 more steps resulting in 1500 training steps in total. We also run the version of GradMax in which we optimize eq. (1I) directly using gradient descent (GradMaxOpt). We start with a random initialization and use the Adam optimizer. The weight matrix is scaled to the target norm after each gradient step.
(Baseline-Bip and 20 5 10 (BaselineSmaall)
In Figure 2a show that in this setup GradMax is able initialize the neurons with sig- nificantly higher gradient norm compared to random growing. The results for GradMaxOpt show that naive direct optimization of the gradient norm does not recover the solution found by GradMax
5

---

Published as a conference paper at ICLR 2022
10
Randem
GradMaa
GrodNox
Training LoSs
Beseline (Smal)
Training Loss
157
Bood Smmal
Beselne-tigg
Baoelin
Auorcee Secco
10"
200
400
Training Steps
6:0
503
10.0
1200
1403
1000
Training Steps
1500
2000
2:000
3000
3500
Cranka
Lige
(a) Small (K 1, 20 10: 10)
(b) Large (K = 5, ft 100 50 10)
(c) Run time
Figure 3: Training curves averaged over 5 runs and provided with 80% confidence intervals. both settings GradMax improves optimization over Random. (right) Run time of the growing algorithms.
using SVD. This highlights the benefit of having formulated problem that can be directly solved. In Figure 2b we plot the total adjusted gradient norm² and observe that the larger gradient norm after growing persists for future training steps. Finally, in Figure 2c we plot the difference in train- ing loss when GradMax is used for growing compared to randomly growing. We observe consistent improvements in the training dynamics which last for over 500 training steps. This supports our hypothesis that increasing the gradient norm accelerates training and leads to greater decrease in the training loss for subsequent steps.
Training Curves We plot the training curves for two different teacher networks in Figure 3. For Figure 3a we trained teacher network of 20 10 10 whereas Figure 3b uses larger teacher network of size 100 50 : 10. For the large teacher network setting we begin with student networks of size 100 25 10 that grown every 500 steps (first growth on step 500), adding 5 neurons
a time resulting for 5 growth steps. We train the grown networks for an additional 1000 iterations resulting in 3500 iterations in total. Further experimental details are shared in Appendix D.
In these experiments the initial student models require about half the number of FLOPS required by the teacher models. Therefore training and growing student models with a linear growing schedule costs only 75% of the FLOPS required to train Baseline-Bit Big. The extra operations required by Grad- Max run faster than a single training step and therefore their cost is negligible as shown in fig. 3c. In all cases GradMax achieves lower training loss compared to the random and GraddMx--Optmiine (GradMaxOpt) methods. For the larger teacher network (Figure 3b), GradMax even matches the performance of network trained from scratch (Big- Baig-Baslline). However for small teacher networks (Figure 3a), all growing methods fall short of matching the Baseline-Big performance similar to Berner et al. (2019); Ash & Adams (2020), in which the authors show that training the final network from scratch works better than warm warm--tartingggroone growing an existing network. GradMax narrows down this important gap by maximizing gradients.
4.2 IMAGE CLASSIFICATION EXPERIMENTS
In this section we benchmark GradMax using various growing schedules and architectures on CIFAR-10, CIFAR-100, and ImageNet. GradMax can easily be applied to convolutional networks (see Appendix B for implementation details). Similar to results in the previous section, share baselines where we train the seed architecture (Small-Bi (aaall-Bseelin)) and the target architecture (Big-Baseline) from scratch without growing. As additional baseline version of Firefly which initializes new neurons by minimizing the training loss directly (without the extra candidates used by the original method). We refer to this baseline as Firefly-Optimized (Firefly-Opt). Firefly requires non-zero outgoing weights in order to optimize the loss. There- fore we initialize the outgoing weights for all methods to small random values (/Weer e for 8 10-4) We use batch size of 512 for ImageNet (128 for CIFAR) experiments and calculate the SVD using the same batch size used in training. At every growth iteration. add new neurons to all layers with reduced initial width, proportional to their target widths. Here our goal is not to obtain state of art results, but to assess how the initialization of new neurons affects the final performance. All networks are trained for the same total number of steps, and thus Baseline-Sman and the grown
implement a simpler
We divide the gradients by the training loss factor out the linear scaling effect of the training loss the gradient norms.

---

Published as a conference paper at ICLR 2022
107
139
10%
3x 10
Paramneee Court
Parameter Court
Training Loos
Training Loss
10tl
x
1ma Boselrn
Small Bisdire
Fg. Bood ane
10-
Ricty Opt
10
10-2
Deathtie
Gracmes
Kandiom
10P
40000 Training & Steps
60000
80000
20000
10003 Training Steps
60000
80000
Wide-Resnet-285 CIFAR-10
(b) VGG-11 CIFAR-10
4x10
4X
3K10P
Training Loos
Parameter Count
Trainnn Lees
Sma Baceline
xx
Parsmister Count
Big Baseiin
109
2x10x
3x10P
Firsty
Small Sasa ins
2x100
2x100
Ny-ox
Gradmee
Batddom
20000
Training Steps
40020
60200
90000
50000 130000
Traning Steps
150300
200000
(d) MobileNet-vl ImageNet
(c) Wide-Resnet-22
CIFAR-100
Figure 4: We plot training loss over time, each experiment averaged over experiments. Number of parameters of the network trained increases over time. Red lines indicate number of parameters over training.
Dataset
Architecture WRN-28-1
Baseline-S
Baseline-B
Random
Firefly
Gradmax
89.
92.910.2
90.6::022
90.
91.11:0.1
CIFAR-10
VGG11
84.1#0.1
86.6:10.3
83.8
84.0++0.2
84.4++0.4
CIFAR-100
WRN-28-1
63.7+00.0
69.3:10.1
66.7:10.4
66.5:10.1
6..8:10.2
ImageNet
Mobilenet-VI
55.0+0.0
70.8:10.0
66.9-F0.3
66.4410.1
68.6.10.2
Table 1: Test accuracy different baselines and growing methods on different tasks. All results are averaged over 3 random seeds (used for training).
models require less FLOPS than Baseline-Big to train. We share our results in Table and observe consistent improvements when GradMax is used.
CIFAR-10/100 We perform experiments on CIFAR-10 and CIFAR-100 using two different ar- chitectures: VGG-11 (Simonyan & Zisserman, 2015) and WRN-28-1 (Zagoruyko & Komodakis, 2016). For both architectures we reduce number of neurons in each layer by 1/4 to obtain the seed architecture (width multiplier=0.25). For the ResNet architecture we only shrink the first con- volutional layer at every block to prevent mismatch when adding skip-connections. We train the networks for 200 epochs using SGD with momentum of 0.9 and decay the learning rate using cosine schedule. In both experiments we use batch size of 128 and perform a small learning rate sweep find learning rates that give best test accuracy for baseline training. We find 0.1 for Nede-ResNet and 0.05 for VGG to perform best and use the same learning rates for all different methods. In Figures 4a to 4c we plot the training loss for different growing strategies and share the test accuracy in Table 1.
ImageNet We grow MobileNet-vl architectures on ImageNet using a seed architecture of width 0.25 (i.e. all layers have one forth the channels). The MobileNet-vl architecture contains depth- wise convolutions between each convolutional layers. When growing convolutional layers, ini- tialize the depth-wise convolution in between to identity. Training loss for this settting shared in Figure 4d.
7

---

Published as a conference paper at ICLR 2022
BN
Inverse
Baseline-S
Baseline-B
Random
Firefly
Gradmax(-Opt)
X
90.610.2
90.810.3
91.1=0.1
89.910.3
92.911.22
X
92.1
92.211.2
92.411.1
92.9
92.910.1
93.0.10.1
90.2+10.3
93.4
92.8+0.1
92.8+00.2
92.9-10.2
Table Average test accuracy when growing WRN-28 on CIFAR-10 with batch normalization and outgoing weights set to zero. BN refers to batch normalization and inverse indicates that set the outgoing weights to zero. When the outgoing weights are set to zero, we use GradmaxOpt.
066
0.600
0.875
0.5
0.6
0
366
Statherd 1e
0.6
10000
06775
2000
oli grotno
0.0
0.670
Food
10
10
10
0.2
04
0.E
0.8
1.3
Grow Prequency
Inita toston Scale
Figure 5: We grow MobileNet-vl networks during the ImageNet training to investigate the effect of (left) growing schedule, (center) &, and (right) scale used in initialization the test accuracy.
Aiggment left- --iigullar vectors
Start of training
Nianment left singular vectors
End of training
1.0
1.0
0.9
0.3
0.8
0.8
0.5
k-5
2048
512
Minibatch size
Minibatch size
Figure 6: The alignment of the top-k left-singular vectors for WRN-28-1 during the CIFAR-10 training. The alignment calculated between the full gradient and minibatches of varying sizes. We do not apply random cropping or flips to the inputs. A total of 10 experiments are run and confidence intervals of 95% are plotted.
Batch Normalization Results and Inverse Formulation We perform our main set of experi- ments using networks without batch norm and setting the incoming weights zero. However, in some cases using batch norm or setting outgoing weights to zero might be useful. Table 2 shows the results for such alternatives when growing residual networks on CIFAR-10. For this particular setting observe that batch norm has limited effect results, however we observe consistent im- provements when outgoing weights are set to zero. We use GradmaxOpt in this setting and observe improvements over both random and Firefly.
Hyperparameters In Figure 5 we show the effect of various hyperparameters on the performance of GradMax when growing MobileNet-vl on ImageNet. First on Figure 5-left we compare different growing schedules. Growing neurons early in the training and growing every 5000 step achieves the best results. Subfigure Figure 5-center shows the effect of e used in our experiments. As expected, larger values of e (and therefore preserving the output of the network during the growing) harm the final test accuracy. Finally, Figure 5-right shows the effect of the scale used when initializing the new neurons. We compare normalizing the scales using the mean norm of existing neurons (mean) to using no normalization (fixed) and observe no visible different in performance. We observe relatively robust performance for values larger than 0.5 and therefore simply set the scale 0.5 in all of our experiments.
8

---

Published as a conference paper at ICLR 2022
Effect of Minibatch Size To validate that we can approximate the SVD of the full gradient with the SVD of a large minibatch we calculate both and plot the alignment of the lint-lin vectors. We define alignment as the absolute value of the Frobenius trace of the two matrices of singular vectors, normalized with K. The plots in Figure 6 show that at the beginning of training the sin- gular vectors strongly align. At the end of training the gradients are less correlated and hence the alignment decreases. However, it is still reasonable estimate.
RELATED WORK
Guscade-Correlatin (CasCor) learning architecture (Fahlman & Lebiere, 1990; Shultz & Rivest, 2000), to our knowledge, is the first algorithm that grows a neural networks during training. Cas- Cor adds new layers that consists of single neurons that maximize the correlation between centered activations and eπrоr signals at every growth step by resulting in a very deep but narrow final ar- chitecture. Platt (1991) allocates new neurons whenever the per-sample error is high. Fukumizu & Amari (2000) show that local minima can turned into saddle points by adding neurons. Chen al. (2016) proposed splitting neurons and Wei et al. (2016) presented more general study of performance-preservinn architecture transformations. Elsken et al. (2017) apply these transforma- tions to NAS. Wen al. (2019) proposed growing neurons and discovered that it is best to grow layers early on. Lu al. (2018) also add new layers, but they sparsify the resulting network. Convex neural networks frames the learning of miehhdden layer neural networks as an diimite-dimesioonll convex problem (Bengio et al., 2006; Rosset et al., 2007; Bach, 2017) and then approximates the solution by an incremental algorithm that adds a neuron at each step.
Kilcher et al. (2019) show that the scaling of the outhound weights can be optimized with respect gradient norm when the original network is at local minima. The solution is given by the closed form using a matrix of outbound partial derivatives. Liu et al. (2019) propose an algorithm for splitting neurons. For each neuron it computes an approximate reduction in loss if this weight were cloned and adjusted by given offset. The solution this problem is given by the eigendecomposition of a "splitting matrix". The splitting is most effective when the network is very close to convergence. Calculating the splitting matrix is expensive so Wang et al. (2019) the authors propose faster approximation. Wu et al. (2020b) expand the set of splitting directions which helps avoid local minima in architecture space. In the context of manifold learning, Vladymyrov (2019) also proposed to grow the number of dimensions of the embedding manifold in order to escape local minima. Similar to our approach, new parameters are found in closed form.
Gradient based growth criteria is used in the context of growing sparse connections (Liu al., 2017; Dettmers& & Zettlemoyer, 2019; Evci et al., 2020; ab Tessera et al., 2021; Evci et al., 2022) and when initializing neural networks (Dauphin & Schoenholz, 2019). Most relevant et al., 2017) which aims to maximize gradients when growing sparse neurons by wiring neurons with highest correlations. However their method is limited growing single neuron and modifies the output of existing neurons due to non-zero initialization on both sides.
work is NeST (Liu
6 DISCUSSION
Limitations GradMax requires activation functions to map zero to zero and have non-zero deriva- tive the origin. Some activation functions, such as radial basis functions, don't satisfy these requirements. We studied fully connected layers and convolutional layers, but did not consider the combination of the two other architectures such transformers.
Future work We are looking to address these limitations in future work. Moreover, in this work we didn't address the questions as to when and where networks should be grown, although we outlined ways in which our method could be adapted to do so. Our method could also be useful network morphism to be used as part of a more complex NAS method. These are further avenues for exploration.
CONCLUSION
In this work we presented GradMax, approach for growing neural networks that aims to maximize the gradient norm when growing. We verified our hypothesis and demonstrated that provides faster training in variety of settings.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We like to thank members of the Google Brain team for their useful feedback. Specifically we like thank Mark Sandler, Andrey Zhmoginov, Elad Eban, Nicolas Le Roux, Laura Graesser and Pierre-Antoine Manzagol for their feedback on the project and the preprint.
REFERENCES
Kale ab Tessera, Sara Hooker, and Benjamin Rosman. Keep the gradients flowing: Using gradient flow to study sparse network optimization. ArXiv, abs/2102.016777 670, 2021.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning on heterogeneous systems, 2015. URL http: http://etssoffo...oooo/ org/.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. arXiv preprint arXiv: 181: 18/. 03980098,,1 2018
Jordan Ash and Ryan P Adams. On warm-starting neural network training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3884-3894. Curran Asso- http://s.....rr...eeeeeeeeeeee hctpps/cc/capp/222//1/2. ciates, Inc., 2020. URL Pape:a pdf.
288cd2567953 (88025579510044443391156aaa
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 2017.
Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. In Advances in Neural Information Processing Systems, vol- ume 18, 2006. URL https: /ttp://diccceeeeeeeeeee. Rcct7eecbbfFlaatt22eeeeeeeessss
httpppsccccppppe/////////s// pdf.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor. Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. ArXiv, 2019. URL http: /ttp//aaxvvvv//v////v//eee/.a.va org abs / 1912. 06680.
Lucas Caccia, Jing Xu, Myle Ott, MarcAurelio Ranzato, and Ludovic Denoyer. On anytime learning at macroscale, 2022. URL https: httpp//////peeeeeeeeeeeeeeeeeeee di wwt..milii-iini..... YolllpYriis...
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. 4th International Conference on Learning Representations, 2016.
Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mchryar Mohri, and Scott Yang. AdaNet: Adaptive structural learning artificial neural networks. Proceedings of the 34th International Conference Machine Learning, pp. 883, 2017.
Stéphane dÁscoli, Levent Sagun, Giulio Biroli, and Joan Bruna. Finding the needle in the haystack with convolutions: on the benefits of architectural bias. In Advances Neural Information Pro- cessing Systems 32, 2019.
10

---

Published as a conference paper at ICLR 2022
Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning to initialize. In Advances in Neural Information Processing Systems, 2019.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. ArXiv, 2019. URL hwtp:htttaaattte org. wwwp/rrg////ooooo..ee//...ee.bee absaabas//// /907 04840.
Thomas Elsken, lan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528, 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of Machine Learning and Systems 2020. 2020.
Utku Evci, Yani Andrew Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. AAAI Conference on Artificial Intelligence, 2022.
Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In
D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1990. URL https: /ttp:///c/ccccceeeeoeeeee wwwp:/ceeeedinnn...iii.....e.... hcc/pappr//999/E/ (ateceeee177777033377aaaaaaaaaaa
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, 1189-1232, 2001.
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 2000.
Yannic Kilcher, Gary Bécigneul, and Thomas Hofmann. Escaping flat areas via function-preeserving structural network modifications, 2019. URL https: http::::///ppervvieeee http://wwwweeoeeeeeeeeee HleadiOcFO.
Steve Lawrence, C. Giles, and Ah Tsoi. Lessons in neural network training: Overfitting may be harder than expected. In Proceedings of the National Conference on Artificial Intelligence, 1997.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning; the catapult mechanism. ArXiv, abs/2003.022111 2020.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning, ArXiv, abs/1910.07454, 2020.
Qiang Liu, Lemeng Wu, and Dilin Wang. Splitting steepest descent for growing neural architectures. ArXiv, 2019, URL https: httpaaaiii....aaaaaa/a//aa 1910, 910.0..00. 02366.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prne Paradigm. ArXiv, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the Value of Network Pruning. ArXiv, 2018
Jun Lu, Wei Ma, and Boi Faltings. Compnet: Neural networks growing via the compact network morphism. arXiv preprint arXiv:1804.1 1XXx:18844.10166 2018.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2003.
Behnam Neyshabur. Towards learning convolutions from scratch. ArXiv, 2020. URL https: 13657.
John Platt. A resource-allocating network for function interpolation. Neural Computation, 1991.
Saharon Rosset, Grzegorz Swirszez, Nathan Srebro, and Ji Zhu. ε regularization in infinite dimen- sional feature spaces. In International Conference on Computational Learning Theory. Springer, 2007.
11

---

Published as a conference paper at ICLR 2022
Andrei A. Rusu, Neil C. Rabinowitz, G. Desjardins, Hubert Soyer, J. Kirkpatrick, K. Kavukcuoglu, Razvan Pascanu, and R. Hadsell. Progressive neural networks. ArXiv, 2016.
Thomas Shultz and François Rivest. Knowledge-based cascade-correlation. volume 5, 07 2000. doi: 10.1109/JCNN.2000. 20w.NN..000..611155
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/ hass/1409.1566. 2015.
Kenneth Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evol. Comput., 2002. URL https: httpp////./...11///6666622222200 69811.
Gilbert W Stewart. Perturbation theory for the singular value decomposition. Technical report, 1998.
Neil Thompson. Kristjan Greenewald, Keeheon Lee, and Gabriel Manso. Deep learning's diminish- ing returns. IEEE Spectrum, 58(10), 2021.
Aravind Vasudevan, Andrew Anderson, and David Gregg. Parallel multi channel convolution using general matrix multiplication. In 2017 IEEE 28th international conference on application-specife systems, architectures and processors (ASAP). pp. 19-24. IEEE, 2017.
Max Vladymyrov. No pressure! addressing the problem of local minima in manifold learning algorithms. arXiv preprint 1rXiv:190..11398 2019.
Dilin Wang, Meng Li, Lemeng Wu, Vikas Chandra, and Qiang Liu. Energy-aware neural architec- ture optimization with fast splitting steepest descent. arXiv preprint arXiv: 2019.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In International Conference on Machine Learning. PMLR, 2016.
Wei Wen, Feng Yan, and Hai Helen Li. Autogrow: Automatic layer growing in deep convolutional networks. Arxiv, 2019. URL htttt::::::aiiii org/ /tsrg/bb////re /190 02909.
Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: gen- cral approach for growing neural networks. In Advances in Neural Information Process- ing Systems, 2020a. URL https: http://eooeeeeignneeeee. nwwrips.ss.... wwtppppr///ee.ee/eeecc httpapprr220//11eee pdf.
Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture opti- mization: Escaping local optimum with signed neural splitting. arXiv preprint arXiv:2003.10392. 2020b.
Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In International Conference Learning Representationss 2021. URL https: www.reveew.eeeee////pe/e net forum? d.w.wbwxbbbbbbb
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv, abs/ abs//600....... 2016.
A IMPLEMENTATION DETAILS
GradMax requires the calculation of the quantity 0z hT_, (eq. (9)). The user could perform this
0L
outer product manually, but in most machine learning frameworks the following approach is easier to implement.
We note that when Wnew 0 and f(0) 0 we can write
W f(W new h = W h W E Me-Mixx 1x
(12)
and
Ze+1 W he + W h
(13)
Note that W 0. The advantage of this formulation is that
aL
aL
xz h
(14)
dwaEx
12

---

Published as a conference paper at ICLR 2022
Hence GradMax easily be implemented in any framework with automatic differentiation by OL- temporarily introducing this auxiliary matrix W in order to calculate 8xc+1 h After the columns of With are calculated using SVD, the matrix Wex is replaced with the new layer (Zt+1 = Wetihh (WW hg-1)).
Growing Layers Matrix decomposition point of view provides a novel insight into the growing problem since it can be applied to any 2 layers and can help grow new one between the two. Note that, our initial derivation we studied the problem adding new neurons to an existing layer. Now we can choose any layers k and l and create new layer between them through growing. If there is already a layer between the two (i.e. k appended to the existing layers.
or = + 2), then the new neurons are
B CONVOLUTIONS
The derivation of GradMax for fully connected layers can readily be extended to convolutional
layers:
Waew T ÒL (1(()
h1-1 ÔL
(15)
awnew
0ze+1
Whew
he-1
(16)
Òzı+1
aT
ØT. Uzet1
h!?"
(17)
DW
8+1
(18)
where 2D convolution and T is the transposed 2D convolution (i.e., the gradient of the convolution).
As before, the required gradient is practice most easily calculated by introducing an auxiliary convolutional layer such that 81. BL.
h We let
OW!
322+1
(19) where W = 0 are filters of the appropriate size. Namely, if We filters of size (he, we) with i input channels and W++1 contains filters of size (he41, W8+1) with 0++1 output channels, then Walx will have filters of size (he he+1 we we+1 -1l 1) with in input and 0 output channels.
z1-1 = W4+1 * he + W h
We now have the equivalent of equation 11 to solve for convolutional layers:
arg max
Ep 8+1
h
S.l. [ cc
(20)
Wiee
0zi+1
++1
We express the 2D convolution as a matrix multiplication (Vasudevan et al., 2017) using the im2col method. This means that Wnew is flattened to matrix of size R 11 hei 1 1 1 x The matrix
AL hT is of size Riex 8zets size Rhe+ -0e4 Xie by extracting the appropriate patches (depending on the filter sizes, padding, and strides) which are flattened and then concatenated. SVD can then be readily applied to this reformulated problem and the resulting top-k left-singular vectors can be reshaped to form the filters of the new layer.
ED
but turned into larger matrix of
C BATCH NORMALIZATION
Batch normalization scales the activations to have unit variance. This makes the incoming weights scale-free (Arora et al., 2018; Li & Arora, 2020), which is usually a desirable property. However, be disruptive when growing neurons. For the first step after growing, batch normalization keeps the zero activations and scales the gradients with J. After the first gradient step the incoming weights will be non-zero, resulting in non-zero activations which will be scaled to unit variance. This can
13

---

Published as a conference paper at ICLR 2022
10
022
0.2
0.1
0.1
Training Loss
0.05
Training Loss
0.05
0.02
0.02
0.01
0.01
10
109
109
250
500
750
1000
1250
1500
1000
2000
3000
4000
5000
Training Steps
Training Steps
(a) Large (ft = 20: 10 10)
(h) Large (fr = 400 200 10)
Figure 7: Effect of learning rate the optimization speed and quality. We repeat each experiment 3 times with a different seed and report the average values with 80% confidence intervals.
disrupt learning the outgoing weights have not been sufficiently trained yet. We observed that using small values for the outgoing weights can be useful to reduce the negative effect this jump. Alternatively, propose in the main text, it might be preferable outgoing connections to zero when batch normalization layers and use GradMaxOpt.
D ADDITIONAL EXPERIMENTAL DETAILS FOR STUDENT-TEACHER EXPERIMENTS
Learning rate. For both experiments we perform small hyper parameter search over the learning rate. In Figure 7 we share learning curves using different learning rates that range between 0.2 and 0.01. Higher learning rates bring faster convergence, however best results are obtained with the second highest learning rate equal to 0.1. To avoid confounds related to the optimizer, restrict the experiments to gradient descent with constant learning rate.
Correlation between singular values and the final decrease. We know from the theory that the norm the gradient is proportional to the singular values given by the solution of equation 11. In other words, solution corresponding to the top singular values should give the largest nomm of the gradient for new weights and therefore should have the best training dynamics in the long run. In Figure 8 demonstrate this on simple Student/Teacher set up we used in the main paper. We first train own network without any growing (blue curve on the top graph). We then grow the network at different stages of optimization using any of the five singular values and let the network converge to some value. As we can see, darker colors corresponding to the larger singular values generally converge faster and correspond the lower values of the objective function.
At the lower portion of the figure, we quantify this result more precisely by repeating the experiment five times and computing the Pearson correlation coefficient between the vector of singular values and the loss decrease some iterations after growing. There z-axis represents how many iterations have passed since the growth and the y-axis shows the correlation between the loss values at that iteration and the singular values. The lower the values the more (anti-) correllcore the values are and, therefore, the more we can trust the GradMax algorithm. The iteration always negative and gets worse we iterate. However, if we use GradMax at the later stages of the algorithm close to convergence, the correlation holds for a while.
Student-Teecher Experiments with Convolutional Networks. Similar to the experiments in Fig- ure 2a, here we show the results of the Student-Teacher task on a convolutional model. As visible on Figure 9, the same conclusions norm the gradient after each growth. Interestingly, the gradient norm somewhat larger for the Random growth, but in Figure 10 we see that GradMax achieves better results overall.
be made for this case well. GradMax achieves the best
14

---

Published as a conference paper at ICLR 2022
Grow
Grow @ 30
Grow
Grow @
Grow 1000
04.17
5-195
1-1P
giwwing
0012
Sigain
3+115
291
the
vite
195
<on
127
1001
Number of iterations
Number of iterations
Number of iteraaionns
Number of iterations
Number of iterations
103
Pharser car cost
120
65
102
6-2
Stops 2PCW 08
500ps 200WIng
5101ing
Staps
Stops pICwing
Figure 8: Long-term effect of growing. Top: Growing once with one of 5 singular values starting at different iteration (20th, 50th, 100th, 500th 1000th). Blue curve represents no growing. Bottom: The correlation between the singular values and the loss function decrease after certain growing iteration.
Adjustdd Gradient Norm
10
Growd
10
Gradidax
Sancom
Growt
2.0
liMMal)
A LOSS
Browt
1
Gradiixx
6:5
GradMcxcppt
2
1680
1250
Growch Number
Training Steps
Step Number
(a) After Growth
(b) During Training
(c) After N Steps
Figure 9: (a) In this plot we load checkpoints from each growth step generated during the Random experiments. We grow single neuron and measure the norm of the gradients with respect Wnew Since Random and GradMaxOpt are stochastic we repeat those experiments 10 times. GradMax provides the maximum gradient norm (b) We measure the the training. GradMax improves gradient norm over Random. (c) Similar to (a), we load checkpoints from each growth step and grow new neuron using GradMax (fg) and Random (fr). Then continue training for 500 steps and plot the difference in training loss (i.e. L(fr) L(fy)). All experiments are repeated 5 times.
of flattened parameters throughout
Student-Teacher Experiments with Batch Norm Similar to the MLP and convolution cases, Figures 11 and 12 provide results for the GradMax performance for the model with BatchNorm.
Setting both incoming and outgoing weights to zero In these experiments we set both side of the new neuron to zero and initialize the bias This non-zero bias provides unit activation. which provides non-zero gradients for the outgoing weights. After the first step, outgoing weights become non-zero and learning takes off. Results in Figure 13a show that this approach performs worse than GradMax.
Effeet of start iteration and initial width Here, first, we look at the effect of growing new neu- rons during different parts of the training. We repeat small the stucent-teacher experiments Sec- tion 4.1 using GradMax, but adjust the iteration the first neuron is grown. We grow 5 neurons in total with 200 step intervals. also adjust the training steps after the last growth so that all experiments match in training FLOPs. Results presented in Figure 13b. We observe that growing early in the training works best, which highlights a different role for growing algorithms than what is presented in the literature (Fukumizu & Amari, 2000): Neural networks can be grown during training with the goal of improving training dynamics. Our results show the potential of such an approach. Next, in Figure 13c, study the effect of initial network size while keeping total FLOPS used during train-
15

---

Published as a conference paper at ICLR 2022
GraitAee
GrodMbx
Training Loss
Basetine (mmsh)
Training Loss
6x10
gimall
1x10
3x10
0
1002
2103
3303
4200
5,00
6:00
Training Steps
Training Steps
(a) Small (K 1, 20 10 10)
(b) Large (K 5, ft 100 50 10)
Figure 10: Training curves averaged over 5 different runs and provided with 80% confidence in- tervals. In both settings GradMax significantly improves optimization over Random. Split-based methods seems to cause some instability large network settings causing frequent jumps in the training objective.
Graditox
Sennom
Adjusted Gradient Norm
10"
Gradilax
Dencrm
0.0
Grooth
Gradd:
101
GradMaxOpp
GradMaaxpp
Crnsth
Growth
0 Loss
0.10
Growdh
000
1000
1250
200
Growth Number
Training Steps
Step Number
(a) After Growth
(b) During Training
(c) After N Steps
Figure 11: (a) this plot we load checkpoints from each growth step generated during the Random experiments. We grow single neuron and measure the norm of the gradients with respect to W Since Random and GradMaxOpt is stochastic we repeat those experiments 10 times. GradMax provides the maximum gradient norm (b) We measure the norm of flattened parameters throughout the training. GradMax improves gradient norm over Random. (c) Similar to (a), we load checkpoints from each growth step and grow new neuron using GradMax (fg) and Random (fr). Then continue training for 500 steps and plot the difference in training loss (i.e. L(fe) L(f,)). All experiments are repeated 5 times.
ing constant. As expected, starting with larger student model helps optimization and obtains better training loss for both GradMax and Random, while GradMax consistently achieves better results than Random in all settings.
Adding More Neurons In our student-teacher experiments we stop growing new neurons when the student model matches the teacher model in terms of number of the student model beyond the size of the teacher model and observe with around 50% more neurons (1.5x), the student model exceeds the baseline (big) performance.
neurons. Figure 13d we grow
16

---

Published as a conference paper at ICLR 2022
Randam GrodMbx
10
Rondom
Bese neveliee Sinw
Training Loss
Training Loss
Baseline
Eacd (B4g))
10
10
200
6.0
600
500
1000
1200
1600
200
400
500
300
1006
1200
1400
Training Steps
Training Steps
(a) Regular
Outgoing zero E+1
Training curves averaged over 5 different runs and provided with 80% confidence in- tervals. In both settings GradMax significantly improves optimization over Random. Split-based methods seems to cause some instability in large network settings causing frequent jumps in the training objective.
Figure 12:
Rondom
Rendio
200
Training Loss
Training Loss
Basaline (Smal))
800
10P
Basaline [5gg]
10F
19a
1000
1500
2003
Training Steps
Training nn Steps
(a) All-zero initialization
(b) Start iteration
GradMaa (1)
10
Training Loss
GradMaa
Training Loss
10F
(1)
10P
Randorr (3)
Ramdor (5)
GradManx;11
GradM4bx 5i
GradMar
Deseline (Smaal)
(3g)
500
1000
1500
1030
2000
3000
4000
Training Steps
Training in Steos
(c) Seed Architecture
(d) More Neurons
Figure 13: (a) We compare setting all-weights to zero (and having unit bias) to our approach of setting only the incoming weights to zero (e.g. Random and GradMax). (b) Effect of growing during different parts of the training for GradMax. Labels correspond to the iteration when the first neuron is grown. We grow neurons in total with 200 step intervals. Training steps adjusted so that all runs have same amount of FLOPs. (c) Effect of initial student network size. As before we adjust total number of steps that the total training FLOPs match for all experiments. Numbers in parentheses represent the size of the hidden layer at initialization. We grow neurons starting from first training iteration. (d) Growing beyond teacher network shows that with larger capacity, grown networks exceed the baseline performance.
17


Reviews:
[{"review_id": "e1BW57XrFNA", "reviewer": "Reviewer_LUg4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+ This paper is well-written and neat. The motivation and the method are clear.\n\n+ The method is simple but somewhat novel, I like simple ideas. \n\n**Cons:**\n\n- It seems many prior works (as described in section 2) are missing in experiments. \n\n- Can authors list the time complexity of GradMax and GradMaxOpt?\n\n", "summary_of_the_paper": "In this paper, the authors propose the GradMax algorithms, in order to add neurons that will cause better optimization results in latter iterations.", "summary_of_the_review": "My weakness is listed above. \n\n----\n### Post rebuttal \n\nAfter reading the author's response, I feel my questions are addressed. I will increase my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SCi-8_c8JJ4", "reviewer": "Reviewer_igLv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is somewhat confusingly written, and it could be better organized. For example, the loss curves are given for the training set in Figure 5 for convnets on CIFAR-10, CIFAR-100, and Imagenet. This is appropriate, given that the method is intended to accelerate training by focusing on the gradients. However, the performance of these networks on the test set are in Table 1, on the previous page. Table 1 should be moved so it shows up *after* the training curves. I give some more suggestions below. \n\nStrengths:\n\nThe work provides a novel, practical approach to growing networks.\n\nThe method is cost-effective, as solving the SVD problem is fast. Its motivation and derivation are presented (mostly) clearly.\n\nThis provides a new alternative to Neural Architecture Search (NAS).\n\nThe method applies to modern convolutional networks.\n\nWeaknesses, with concrete, actionable feedback\n\nI find the writing confusing at times. \n\nThe second paragraph in Section 4 mentions three goals. It doesn't mention that you are going to test this on convolutional networks (Section 4.2). This introductory paragraph should include that as a fourth goal, as it is an introduction for the whole section 4, not just 4.1. This reader at least, was surprised when I got to Section 4.2, as I was beginning to think you weren't going to try anything relevant to deep convolutional networks. It is best to set expectations.\n\nFigure 3C displays the relative difference in loss reduction between GradMax and randomly-initialized units added at the same point in training. Another baseline would be to add units with 0 weights (both input and output), with a bias such that they have some small activation, and thus get incorporated into the network.\n\nIn general, while the experiments are somewhat systematic, they start in the middle somehow. Would you always start with a network of some size? Can you apply the method from the beginning of training, starting one unit or channel per layer? In a practical setting, how would you decide to stop adding units? What happens to these networks if you keep adding units? \n\n*Minor issues/typos/wording suggestions*\n\nFirst paragraph of Section 3: Change:\n\nWe also require that the network output before and after the growth remains unchanged.\n\nto:\n\nWe also require that the network output is unchanged when the neuron is added.\n\nIn the paragraph after equation 4, you say that: \n\n\"A classical result for this class of functions is that the *decrease in objective function* after one step of gradient descent with step-size 1/β is upper bounded by L(Wl) − β2 ∥∇L(Wl)∥2 (Nesterov, 2003). This is an expression that decreases as the norm of the gradient increases.\" [emphasis added]\n\nThis says that the *decrease of the loss* gets smaller as the gradient gets bigger, so obviously, if that were true, you would want a *smaller* gradient, which makes no sense. What I think you meant to say was that the *loss* is upper bounded by this expression, so you want the gradient to get bigger. Is that right?\n\nBottom of page 4, regarding Figure 2. Here, you don't say (in the text, anyway), that this is WRN 28-1 (or describe what that network is), what the task is, or the training set. Have you defined k at this point?\n\nSection 4.1, second paragraph: \n\nteacher-student settings -> \n\nteacher student setting\n\nThe sentence: \"The baselines Baseline-Small and Baseline-Big are trained using the fixed architectures 20:5:10 and 20:10:10.\" is confusing because you define them after you use their name. \n\nBetter: We define two baseline networks trained from scratch. Baseline-Big is the size of the final network, 20:10:10. Baseline-Small is the size of the initial student network, 20:5:10. \n\nSame page, second line from the bottom: the SVD -> SVD.\n\nPage 6, middle: \n\nYou measure the norm of the parameters? Don't you mean the norm of the gradient?\n\nThis experiments is repeated 5 times -> \n\nThis experiment is repeated 5 times.\n\nTraining curves paragraph: When do you *start* adding nodes? 500 training steps? \n\nSection 4.2, first sentence ends with CIFAR-10. You should end it with \"CIFAR-10, CIFAR-100, and ImageNet.\" (no need to add \"2012\"). \n\nNext sentence:\n\nNote that GradMax can be applied... ->\n\nGradMax can easily be applied...\n\nline or two down:\n\nby minimizing the training loss directly ->\n\nby minimizing the training loss with a single new neuron\n\nFor both architectures we start using a width multiplier of 0.25 for our seed architecture. ->\n\nFor both architectures we start with a layer 1/4 the original width for our seed architecture.\n\nThen you say you apply this to the first layer of resnet, but you don't say what layer you apply it to for VGG-11.\n\nSimilarly, for MobileNet-v1, you don't say what layer you shrink.\n\nTable 2: What dataset is this for? What networks?\n\nBottom of page 6: \n\nWe observe relatively robust for values...->\n\nWe observe relatively robust performance for values...\n\nRelated work: You don't reference the canonical example of growing networks: Scott Fahlman's Cascade-Correlation networks from 1992 or so. I notice that there are recent video lectures posted online of Scott talking about cascade correlation and deep learning. I also found a blog post entitled: \"Cascade-Correlation, a Forgotten Learning Architecture\"! So you aren't the only one...\n\nPage 9: real-basis functions -> radial basis functions?\n\nVarious places: \n\nfig X -> Figure X or Fig. X\n\ntable y -> Table y\n\nsubfigure 4b -> Figure 4(b)\n\nFor the larger network, GradMax even matches...->\n\nFor the larger teacher network (Figure 4(b)), GradMax even matches...\n\nHowever for small teacher networks, all growing methods...\n\nHowever for small teacher networks (Figure 4(a)), all growing methods...\n\n\"This highlights an important gap to fill in future research.\" Ok, but in the current environment, most people care about big networks! \n\nBottom of page 6: You mention the number of flops to train - it would probably be good to squeeze in a bar plot of flops of the various methods. This is a big selling point of the method. \n\nBottom of page 13: If the norm of the gradient is inversely proportional to the singular values, you would want to use the smallest singular values, but obviously, you want the biggest ones, right? Can you clear this up for me?\n \n", "summary_of_the_paper": "This paper presents a new method (as far as I know) of growing neural networks. This can be viewed as an approach to finding the meta-parameter corresponding to network size. The main idea is to add units such that they maximize the norm of the gradient. This is suggested to be superior to methods that grow by adding units that minimize the loss, as maximizing the gradient will lead to learning benefits in subsequent training. With some simplifying assumptions, this turns out to be an optimization problem solvable by SVD. Although it is a full batch method, using a large minibatch size provides a practical algorithm. The results show that the intuition is correct: The change in loss over following epochs continues to be larger than if randomly-initialized units are added (see Figure 3(c)). The networks are able to grow to the size of a reference network trained from scratch, and achieve performance that is not far from this standard in some cases. It is not expected here that growing the network will achieve equivalent performance to a properly-sized network trained from scratch, but it uses less compute due to starting with smaller networks. The approach is first tested in fully-connected, shallow networks, but generalizes to convolutional networks, including resnets, hence, it is a practical approach to growing networks.\n\nI have read the authors' response and am happy to raise my score a notch. I also think this is a worthwhile direction for research.", "summary_of_the_review": "This is a novel idea (as far as I know) for growing networks, with a good intuition: add units so that they maximize the gradient, rather than minimize the loss. This has good effects on the learning dynamics. Unfortunately, the writing and organization dampens enthusiasm for the paper, and the experiments seem a bit haphazard. I think this paper could be improved considerably by addressing these issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Gfve0VknSAc", "reviewer": "Reviewer_1m2b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- The idea of the proposed weight initialization method, maximizing the gradient norm of the new weights, is intuitively reasonable. Also, the proposed SVD-based algorithm seems to be technically sound.\n- The advantage of the proposed GradMax against random initialization is shown through the numerical experiments.\n\n[Weaknesses]\n- Although GradMax works well when growing neurons in an existing layer, the reviewer thinks that GradMax is not applicable when adding a new layer to networks. Perhaps, it may be possible by adding a layer together with skip connections.\n- Although adding neurons gradually in model training can reduce the total training cost, the performance degradation compared to training the big model from scratch (Baseline (Big)) is not negligible.\n- This paper only focuses on the weight initialization when adding new neurons to the network. However, as the author pointed out, when and where new neurons should be added is important in practice. Also, when we should stop the neuron growing is important in terms of finding a good network structure. The proposed GradMax should be combined with an architecture search method and evaluated to show the effectiveness in the practical use case.", "summary_of_the_paper": "This paper proposes a weight initialization method when growing neural networks. The key idea is to initialize the new neuron's weights to maximize their gradient norm. The proposed initialization is realized by the singular value decomposition (SVD) under some assumptions. The effectiveness of the proposed gradient maximizing growth (GradMax) is verified on the simple artificial tasks and image classification tasks in the scenario for increasing the number of neurons.", "summary_of_the_review": "The proposed weight initialization method is reasonable when adding neurons in model training. However, the experimental evaluation assumes a limited situation. The effectiveness of the proposed method in practical usage is not apparent.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Auq2T0VWBYW", "reviewer": "Reviewer_LS3D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. Although I am not familiar with the literature, the idea seems novel and the experiments on common image datasets are comprehensive and well-executed. The improvements seem marginal but are consistent throughout the paper. \n\nNevertheless, the motivation of the paper is not well explained and needs more mathematical and empirical justification. As far as I understand, the authors want to introduce (neurons and therefore) weights such that the overall gradient of the loss wrt to all the weights is minimized. I find this idea interesting but not well reflected in the considered optimization problem, equation 4. Since the introduction of the new weights also change the backward/gradient computation of layers k < l, it could be that although optimization problem 4 is well solved, the overall gradient decreases - which would go against the motivation/ method proposed. \nPlease elaborate on this point and explain why this can't happen. The teacher-student experiments are not sufficient because no weights before the newly introduced W_l^new exist and I cant see anything about this for the larger models (with batchnorm). \n\nOn top of this, this could be turned around and part of the method: How can we add new neurons such that we change/improve the backward pass in way that improves the gradient norm of the whole network e.g. by backward computation that circumvents vanishing gradients. In general, the motivation of the paper should be much more investigated and could be made much stronger.\n\nAnother important point that is missing in my oppionin is the careful handling of batchnorm layers. \nScaling the norm of the (newly added) weights has no effect in the forward pass but scales the norm of the gradient in the backward pass, see e.g. (https://openreview.net/forum?id=rkxQ-nA9FX, https://arxiv.org/pdf/1910.07454.pdf). \nThis goes hand in hand with your motivation but is never discussed for models when batchnorm is used. Please elaborate on this and show that your motivation and the proposed method still make sense with batchnorm layers. \n\nIf the authors manage to convince me / improve my mathematical understanding and intuition of what the method is actually doing, I would consider raising my score. \n", "summary_of_the_paper": "The paper proposes a method to grow neural networks by adding neurons in a specific way. While requiring that the forward pass stays unchanged, the authors want to initialize the new weights such that the gradient wrt to these weights is maximized. They show small but consistent improvements compared to the baselines although the proposed method seems considerably more complex than baselines, as far as I can see.\n", "summary_of_the_review": "Not sufficient and convincing evidence that the motivation of the paper make sense and is met in practice. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper looks into growing neural networks, and finds an improved approach to the initialisations of new layers, viz by maximising the gradient norm.  Simple, straightforward, neat, and no good reason to reject.  It will benefit those who are using growing NNs.
Decision: Accept (Poster)