Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates methods to grow the width of linear neural network layers during training. Their proposed method consists of a variety of techniques:

* The new layer has a repeated block of new weights, which are multiplied by new submatrices in the next layer which only differ in sign, so that the final contribution of these weights is zero (or at least, almost).
* The old weights are scaled so that their expected variance is as if the weights were initialized using the fan-in of the new layer
* Throughout training, the learning rates for the new weights are scaled by the ratio of their norm and the norm of the original weights
* The layer sizes grow exponentially as a function of the growing stages
* The growing stages have an exponentially growing number of epochs
* The batch size decreases exponentially

The suggested method is compared on a variety of vision models (ResNet, VGG, MobileNet) and datasets (CIFAR-10, CIFAR-100, ImageNet) as well as a transformer trained for machine translation. They compare against several other methods (Net2Net, splitting, FireFly, GradMax). The results are encouraging, with the final test accuracy usually being very close to the large model trained from scratch while reducing training cost between 30% and 50%.
strength_and_weaknesses: This paper has some interesting ideas: I like the way that the layers are grown. It feels similar to splitting (i.e., duplicating weights and then halving them) while avoiding changing the scale of the weights, which should help with the learning dynamics. The learning rate scaling is also interesting. One of the main problems with growing networks is that it can take a long time for the new capacity to be used. Perhaps this scaling of the learning rate helps with this somehow.

My main issue with this paper is that it proposes a lot of things (a function-preserving growing method, scaling of weights, learning rate scaling, growing schedules, batch size schedules, etc.). Many of these are not very theoretically grounded, and the ablations are limited. This makes it hard to disentangle which part of the methods are working and why. It gives a feeling of the authors having thrown the kitchen sink at the problem until something stuck.

The result is that this paper raises many questions for me:

* Is the variance transfer really useful? The effect seems minimal in figure 3. There are several learning rates for which the variance transfer and non-variance transfer curves overlap. Especially given that this is on CIFAR-10 with a 4-layer network, it seems like a bad idea to draw conclusions from these results. Without confidence intervals, figure 4 also is hard to draw definitive conclusions from (e.g., if training would have stopped at epoch 150 instead of 160, the methods would have been indistinguishable).
* Does variance transfer actually make sense? The scaling of the weights would only make sense if we know that the magnitude of the old weights hasn't changed much during training, but this isn't validated.
* What is the motivation for the learning rate scaling proposed in table 1? The intuition I would have is that you want to have a bigger learning rate for the new weights, for two reasons: they have to catch up, and the old weights can't be allowed to get too distorted by the initially random signals of the new weights. But the current learning rate scaling actually scales down the learning rate of the new weights when their norm is small, which seems like it might get them stuck? Would $\max(1, \lVert W_i \setminus W_{i-1} \rVert)$ maybe make more sense?
* Figure 4(c) shows that the gradient norms are closer together. In fact, the gradient norms of the oldest subnets seem to be lower, suggesting that they are better trained. But figure 5 shows that their weight norms (and hence learning rates) are still higher towards the end of training. This seems counterintuitive to me (usually you reduce the learning rate when you get closer to the optimum). And why is it that the weights of these old subnets stay so high compared to the initial weights? Is that because the variance transfer keeps scaling the old weights down?
* In figures 4(b), 4(c) and 5 you can see a very common dynamic of growing networks (look at subnets 7 and 8): When new weights are introduced they initially just produce noise, so the first thing the network does is to unlearn them (usually by driving them to zero). After that, the network takes some time to integrate this new capacity, at which point the gradient on these weights becomes similar to the rest of the network. My concern is that the proposed learning rate scheduling is slowing down this process by lowering the learning rate. 

I could probably keep going and come up with more questions. (For example, I am wondering if on one hand the scheduling is helping by increasing the learning rate of older subnets compared to newer subnets, but on the other hand hurting by integrating new subnets slower in the network. Or maybe the lower learning rate is actually helping by giving the older network time to adapt to the newer weights before those settle?) My concern is that at the end, this paper provides a few interesting data points, but raises more questions than it answers. To argue for acceptance, I would like to see a paper with more focus (e.g., only the learning rate scheduling, or only the function-preserving growing method) and thoroughly study this through thoughtful ablations.
clarity,_quality,_novelty_and_reproducibility: Overall, I don't think this paper is well written. The introduction is thorough in its coverage of related work, but lacks structure. It is not entirely clear what the distinction between the related work and introduction sections are. Section 3 is at times too dense because it tries too hard to formalize the method without providing an explanation or intuition for it.

The mathematical notation is messy: The weights are indexed by stage $i$ ($W_{i-1}$ becomes $W_i$), but their sizes are not ($H_{in}$ becomes $\widehat{H_{in}}$). Why use $H_{out}$ (where $H$ is for hidden) to denote the size of the output layer? $concat$ is not a mathematical operator. Multiplication is at times written as $\times$ and sometimes as $\ast$ (the latter is usually used for convolutions). Set notation is used on matrices, which doesn't really make sense. I also find the notation of $W_i \setminus W_{i-1}$ a bit convoluted. Additionally, some minor typographic things in the mathematical writing: I would prefer $x^{-1}$ instead of $1/x$, and super- and subscripts should be wrapped in `\mathrm`.

The figures are too busy. For example, I would use a figure like 1(b) that just shows the growing method that uses positive/negative weights, but leave out the variance transfer, learning rate scaling, noise, etc. Figure 2 could use colours, and shouldn't have all the formulae from the text repeated. 

Are figures 4(b) and 4(c) plots of the gradient norms, or plots of the step norms (i.e., is this before or after being scaled with the learning rate)? I am assuming before (since figure 5 seems to suggest that the gradients for earlier subnets would actually be scaled up). Why are the gradients for the seed architecture weights not plotted?

What is the noise that is added to the new weights for symmetry breaking? How does the norm of this noise compare to the norm of the initialization?

Minor comment in the text: The title "stage-wise learning rate adaptation" made me think that the learning for each subnetwork is only changed once per stage. It was only when I saw figure 5 that I realized that the learning rate is actually changing throughout training.

All in all, I find the paper lacks clarity and some details are missing for reproduction. The paper introduces some novel techniques that could be interesting, but it fails to provide novel insights due to a lack of ablations and careful investigation of the underlying dynamics.
summary_of_the_review: A paper that introduces some nice ideas, but it tries to do too many things and in the process fails to investigate the hard questions. The resulting paper feels like a long list of experiments and proposals without hypotheses that are tested. I would argue to reject this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of growing neural networks during training.  The authors propose a parameterization and optimization scheme that pays close attention to weight and gradient scaling while reacting to training dynamics.  They further refine their original proposal to mitigate the problem of newly grafted subnetworks, that are relatively un-trained relative to the existing network, by an adaptive gradient scheme to ensure the contribution to the gradient is more balanced across new and older components.  They provide experimental evidence to show the efficacy of their method, paying particular attention to the savings in training time (or computation budget) accrued relative to larger monolithic networks.

strength_and_weaknesses: ### Strengths
- The authors have tried hard to present a more principled and stable method for growing networks from smaller to larger capacities, which is an important problem especially in large language models.
- Figure 5 and 6 are refreshing to see, as is much of section 4.4.  The authors are attempting to help us better understand how each different component of Algorithm 1 works to yield the final results in Tables 1-5.
- The authors do a good job summarizing and surveying the related work in network growing, and also explain the details of each component of their method in section 3 very clearly, using Figure 2 to help explain the growth stage layer-wise.

### Weaknesses

- The worth of some of the core contributions aren't very clearly established in the paper.  For instance, the value of the variance transfer parameterization achieves comparable performance with competing methods (some gains in table 5, but tables 1-3 show mostly marginal gains over Net2Net and Firefly).  
- Also, the last claim is too broad.  The authors adaptive method bests the large model baseline for the VGG models.  The phrase "even outperforming the original fixed-size models" makes it seem as if this is often the case. 
- In section 4.1, the paragraph describing table 2 is a nice result, but somewhat overstated.  On two models the authors' method outperform the baseline large model.  And on most comparisons (CIFAR 10/100), the gain is on par with Net2Net.  If the authors could find a test where their method was substantially better in test accuracy than Net2Net, it would really bolster their claims that Net2Net’s heuristic splitting method needs improving. Though the Transformer results in Table 5 display a somewhat more substantial difference in BLEU score.
- Is there a reason that distributions were reported in tables 1 through 3 but not tables 4 and 5? 
- In the description of the results for Figure 3b, Blue is sometimes dominating orange in Figure 3b, but it’s not consistent especially at the lower learning rates (which are more consistent with what are used in practice)

clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality

The writing is clear enough, but some critical points are less clear than they should be.  For instance, the method(s) herein are motivated in the introduction as:
>Our contribution is to do so in a manner that preserves the function computed by the model at each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, we use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training a large static model from scratch.

Yet after reading the paper, apart from the gains in efficiency, it's unclear which of the other advances really help contribute to stability in training.  

- Figure 3a has interesting information about the variability of the different growing methods, but this is obscured by the over-plotting of each method at the sampled learning rate points.  Could this be re-plotted with jitter?  Or as a dodged bar graph with whiskers (since the line segments connecting the distributions carry no information)?
- It’s hard to know what to take away from Figure 3b.  On the surface it looks like the red is most resistant to a larger learning rate, but this is likely because the growth method is adaptively adjusting the learning rate, as detailed in section 3.  The condition that seems to be missing is a large network with an adaptive learning rate
- In Figure 4a, it’s true that  the rate adaptation *seems* more stable, but this can be measured and reported to be more precise.  You could treat this trace as a time series and report the autocorrelation for each method

### Minor questions
Question in section 3.1: does the $\nu_{0}$ global learning rate get adjusted also?
Question in section 4 (Large Baselines via Fixed-size Training): Does the initial learning rate correspond to $\nu_{0}$?

### Comments

The problem of incrementally growing networks seems like it’s almost dual with the problem of pruning trained networks (or training networks);  A question for the authors is have they thought about what the literature on neural network pruning (especially LTH-flavoured work on finding subnetworks) has to say about the problem of growing networks?

Similarly, I wonder how the authors would say this work connects  with the aims of distillation?  There, a smaller model is trained to approximate the function of a larger model.  Here, a larger  model is adaptively grown in a way that preserves the function of the smaller model. 

A final point is that figure 7 showcases some nice results for saving time  (and memory for fixed batch size grow), but for ResNet18 models, it’s a bit artificial, as this is a very small model that will take a maximum of one GPU to train anyhow. This would be much more convincing if it were measuring the time /epoch of a much larger model, like modern LLMs (e.g T5-3B), that take a very long time to train 
summary_of_the_review: While the authors have proposed what is likely an advance in model growth, they have been hampered somewhat by the choices they made in their experiments.  

Incremental model growing is at its most useful when growing to very large models (think large language models circa 2021), )that would trouble even industry research labs.  Demonstrating a convincing win here in terms of performance versus compute (and time) saved would represent a more effective contribution, yet they focused much of their experiments on smaller vision models that do not offer a compelling reason to be grown stage-wise.

For this reason, I think that the paper offers some value, if rather marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to train neural network models incrementally to reduce training time. To this end it proposes two heuristics, the first being variance transfer, which involves re-parameterizing weights to account for their variance, and second being rate adaptation, which adjust the learning rate of newly added weights different from the weights already present in the model. Experiments show that these strategies result in more accurate neural networks that are trained faster than existing methods for incremental training.
strength_and_weaknesses: **Strengths**:

+ The paper tackles an important problem of accelerating training via incrementally growing neural networks, and a good solution to this problem can have huge implications for large-scale training of neural networks.

+ The heuristics proposed in this paper, i.e., variance transfer and rate adaptation seem like reasonable approaches, and the experimental results (esp those on Imagenet) agree with that observations.


**Weaknesses**:	

**Proposed methods are heuristics but claimed otherwise; lacking justification** 

- The title and several other parts of the paper make the claim that the proposed methods are principled. This seems to be because the methods are inspired from the scalings used in the mutransfer paper (Yang et al., 2021). However, it is worth stressing that a principled strategy in one setting does not immediately imply it being a principled strategy in another setting without appropriate justification, as I elaborate in the next two points. I thus request the authors to remove claims regarding the principled nature of proposed methods. 

- The justification for variance transfer is "to make the weights of the old subnetwork compatible with the entire weight tensor parameterization", where an assumption is made that the old weights are drawn from a gaussian distribution with std = (1 / h_out^2). However, this is true only at initialization, and the distribution of weights can diverge in principle during training. Question: Why use the variance at initialization as opposed to the empirical variance at the time of growing the model? What is the precise justification for variance transfer in the first place?

- The learning rate adaptation method has no justification provided (even an intuitive one) and is purely a heuristic. Suggestion: Perhaps some intuition can be obtained by analyzing the dynamics of a simple one hidden layer MLP model?

- Missing detail: How are SGD momentum vectors handled at the time of growing a model? Are they re-initialized from scratch for each stage of re-training, or are they carried over from previous stages of optimization?

**Missing ablations on realistic models**

- While the analysis experiments in Section 4.4 are appreciated, it is important to have ablations presented for the more realistic Resnet-20 / Resnet-18 / Mobilenetv1 models used in Tables 1 and 2. In particular, it would be interesting to compare the following baselines - (1) Growing method + LR adaptation (LRA) + Variance transfer (VT) (aka the "full" method, already presented), (2) Method + LRA only, (3) Method + VT only, (4) Method only, which involves simply adding the new rows / columns to weights according to the prescribed schedule. In the absence of principled justification, these thorough ablations are necessary to evaluate the usefulness of the method.

clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is mostly clear to read, however the description in section 3 feels cluttered with too much unnecessary notation. Nonetheless, Figure 2 is a good addition that conveys the message compactly.

Originality: The methods in this paper are moderately original. While the overall strategy does not deviate too broadly from those used in prior works (e.g.: Net2Net), the inspiration from the scaling used in the mutransfer paper makes the methods novel.

Quality: The paper scores moderate to low on quality. The methods come with little to no justifications, and the experiments have some ablations missing. Despite this, the large scale experiments on Imagenet are appreciated and show potentially that the proposed methods can scale to larger models.

Reproducibility: The method seems mostly reproducible from description, however, some details like that of optimizer resetting, are missing.

summary_of_the_review: I am currently providing a recommendation of reject. Overall, this paper does not really propose any interesting technical insights or ideas, rather it consists of heuristics that nonetheless achieve SOTA results, making it borderline. However, I am willing to raise my score given that the authors are willing to show ablation experiments on realistic CIFAR10 / CIFAR100 models which will show the exact contribution of the various components of the method. 

---- Update ----
Based on the additional experiments showed in the rebuttals, I am changing my recommendation to that of accept. My original review still largely stands, that the paper proposes a collection of heuristics that achieve SOTA results, making it still borderline. However, the strong results with ablation experiments indicate that these techniques may be useful to practitioners interested in this problem. Further, as I mention in my response, I hope that paper tempers down claims of being principled in an update of the draft.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a method for incrementally growing a neural network. The layers and weight matrices are gradually expanded, and the new weights are initialized by flipping the original weights. The main technical novelty consists of heuristics for rescaling parameters, adapting the learning rates, choosing the length of each training stage, and so on. On various image and language benchmarks, the proposed method is able to nearly match the test error of ordinary training while cutting the training cost by about a third.

The scores for the paper are somewhat borderline. Reviewers felt like the problem of incrementally growing networks is a useful one and the ideas seem fairly reasonable. On the other hand, they felt like the methods were somewhat of a bag of tricks with heuristic justifications, and the experiments didn't disentangle what was causing the improvement. In the revision, the authors introduced several ablation experiments which helped to address the latter criticism (and made some reviewers raise their scores). However, the contribution itself still feels incremental and heuristic. This is ok -- cleverness isn't a requirement in a paper -- but there needs to be enough analysis that readers will walk away having gained new insights. I think this could be served by finding cases where there are more substantial gains from incremental growing, or picking the tricks that seem most promising and subjecting them to deeper empirical and theoretical analysis.
justification_for_why_not_higher_score: This paper could reasonably be accepted, but I lean towards rejection for the reasons outlined above.
Decision: Reject