Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a weight initialization method when growing neural networks. The key idea is to initialize the new neuron's weights to maximize their gradient norm. The proposed initialization is realized by the singular value decomposition (SVD) under some assumptions. The effectiveness of the proposed gradient maximizing growth (GradMax) is verified on the simple artificial tasks and image classification tasks in the scenario for increasing the number of neurons.
main_review: [Strengths]
- The idea of the proposed weight initialization method, maximizing the gradient norm of the new weights, is intuitively reasonable. Also, the proposed SVD-based algorithm seems to be technically sound.
- The advantage of the proposed GradMax against random initialization is shown through the numerical experiments.

[Weaknesses]
- Although GradMax works well when growing neurons in an existing layer, the reviewer thinks that GradMax is not applicable when adding a new layer to networks. Perhaps, it may be possible by adding a layer together with skip connections.
- Although adding neurons gradually in model training can reduce the total training cost, the performance degradation compared to training the big model from scratch (Baseline (Big)) is not negligible.
- This paper only focuses on the weight initialization when adding new neurons to the network. However, as the author pointed out, when and where new neurons should be added is important in practice. Also, when we should stop the neuron growing is important in terms of finding a good network structure. The proposed GradMax should be combined with an architecture search method and evaluated to show the effectiveness in the practical use case.
summary_of_the_review: The proposed weight initialization method is reasonable when adding neurons in model training. However, the experimental evaluation assumes a limited situation. The effectiveness of the proposed method in practical usage is not apparent.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose the GradMax algorithms, in order to add neurons that will cause better optimization results in latter iterations.
main_review: **Pros:**

+ This paper is well-written and neat. The motivation and the method are clear.

+ The method is simple but somewhat novel, I like simple ideas. 

**Cons:**

- It seems many prior works (as described in section 2) are missing in experiments. 

- Can authors list the time complexity of GradMax and GradMaxOpt?


summary_of_the_review: My weakness is listed above. 

----
### Post rebuttal 

After reading the author's response, I feel my questions are addressed. I will increase my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method (as far as I know) of growing neural networks. This can be viewed as an approach to finding the meta-parameter corresponding to network size. The main idea is to add units such that they maximize the norm of the gradient. This is suggested to be superior to methods that grow by adding units that minimize the loss, as maximizing the gradient will lead to learning benefits in subsequent training. With some simplifying assumptions, this turns out to be an optimization problem solvable by SVD. Although it is a full batch method, using a large minibatch size provides a practical algorithm. The results show that the intuition is correct: The change in loss over following epochs continues to be larger than if randomly-initialized units are added (see Figure 3(c)). The networks are able to grow to the size of a reference network trained from scratch, and achieve performance that is not far from this standard in some cases. It is not expected here that growing the network will achieve equivalent performance to a properly-sized network trained from scratch, but it uses less compute due to starting with smaller networks. The approach is first tested in fully-connected, shallow networks, but generalizes to convolutional networks, including resnets, hence, it is a practical approach to growing networks.

I have read the authors' response and am happy to raise my score a notch. I also think this is a worthwhile direction for research.
main_review: The paper is somewhat confusingly written, and it could be better organized. For example, the loss curves are given for the training set in Figure 5 for convnets on CIFAR-10, CIFAR-100, and Imagenet. This is appropriate, given that the method is intended to accelerate training by focusing on the gradients. However, the performance of these networks on the test set are in Table 1, on the previous page. Table 1 should be moved so it shows up *after* the training curves. I give some more suggestions below. 

Strengths:

The work provides a novel, practical approach to growing networks.

The method is cost-effective, as solving the SVD problem is fast. Its motivation and derivation are presented (mostly) clearly.

This provides a new alternative to Neural Architecture Search (NAS).

The method applies to modern convolutional networks.

Weaknesses, with concrete, actionable feedback

I find the writing confusing at times. 

The second paragraph in Section 4 mentions three goals. It doesn't mention that you are going to test this on convolutional networks (Section 4.2). This introductory paragraph should include that as a fourth goal, as it is an introduction for the whole section 4, not just 4.1. This reader at least, was surprised when I got to Section 4.2, as I was beginning to think you weren't going to try anything relevant to deep convolutional networks. It is best to set expectations.

Figure 3C displays the relative difference in loss reduction between GradMax and randomly-initialized units added at the same point in training. Another baseline would be to add units with 0 weights (both input and output), with a bias such that they have some small activation, and thus get incorporated into the network.

In general, while the experiments are somewhat systematic, they start in the middle somehow. Would you always start with a network of some size? Can you apply the method from the beginning of training, starting one unit or channel per layer? In a practical setting, how would you decide to stop adding units? What happens to these networks if you keep adding units? 

*Minor issues/typos/wording suggestions*

First paragraph of Section 3: Change:

We also require that the network output before and after the growth remains unchanged.

to:

We also require that the network output is unchanged when the neuron is added.

In the paragraph after equation 4, you say that: 

"A classical result for this class of functions is that the *decrease in objective function* after one step of gradient descent with step-size 1/β is upper bounded by L(Wl) − β2 ∥∇L(Wl)∥2 (Nesterov, 2003). This is an expression that decreases as the norm of the gradient increases." [emphasis added]

This says that the *decrease of the loss* gets smaller as the gradient gets bigger, so obviously, if that were true, you would want a *smaller* gradient, which makes no sense. What I think you meant to say was that the *loss* is upper bounded by this expression, so you want the gradient to get bigger. Is that right?

Bottom of page 4, regarding Figure 2. Here, you don't say (in the text, anyway), that this is WRN 28-1 (or describe what that network is), what the task is, or the training set. Have you defined k at this point?

Section 4.1, second paragraph: 

teacher-student settings -> 

teacher student setting

The sentence: "The baselines Baseline-Small and Baseline-Big are trained using the fixed architectures 20:5:10 and 20:10:10." is confusing because you define them after you use their name. 

Better: We define two baseline networks trained from scratch. Baseline-Big is the size of the final network, 20:10:10. Baseline-Small is the size of the initial student network, 20:5:10. 

Same page, second line from the bottom: the SVD -> SVD.

Page 6, middle: 

You measure the norm of the parameters? Don't you mean the norm of the gradient?

This experiments is repeated 5 times -> 

This experiment is repeated 5 times.

Training curves paragraph: When do you *start* adding nodes? 500 training steps? 

Section 4.2, first sentence ends with CIFAR-10. You should end it with "CIFAR-10, CIFAR-100, and ImageNet." (no need to add "2012"). 

Next sentence:

Note that GradMax can be applied... ->

GradMax can easily be applied...

line or two down:

by minimizing the training loss directly ->

by minimizing the training loss with a single new neuron

For both architectures we start using a width multiplier of 0.25 for our seed architecture. ->

For both architectures we start with a layer 1/4 the original width for our seed architecture.

Then you say you apply this to the first layer of resnet, but you don't say what layer you apply it to for VGG-11.

Similarly, for MobileNet-v1, you don't say what layer you shrink.

Table 2: What dataset is this for? What networks?

Bottom of page 6: 

We observe relatively robust for values...->

We observe relatively robust performance for values...

Related work: You don't reference the canonical example of growing networks: Scott Fahlman's Cascade-Correlation networks from 1992 or so. I notice that there are recent video lectures posted online of Scott talking about cascade correlation and deep learning. I also found a blog post entitled: "Cascade-Correlation, a Forgotten Learning Architecture"! So you aren't the only one...

Page 9: real-basis functions -> radial basis functions?

Various places: 

fig X -> Figure X or Fig. X

table y -> Table y

subfigure 4b -> Figure 4(b)

For the larger network, GradMax even matches...->

For the larger teacher network (Figure 4(b)), GradMax even matches...

However for small teacher networks, all growing methods...

However for small teacher networks (Figure 4(a)), all growing methods...

"This highlights an important gap to fill in future research." Ok, but in the current environment, most people care about big networks! 

Bottom of page 6: You mention the number of flops to train - it would probably be good to squeeze in a bar plot of flops of the various methods. This is a big selling point of the method. 

Bottom of page 13: If the norm of the gradient is inversely proportional to the singular values, you would want to use the smallest singular values, but obviously, you want the biggest ones, right? Can you clear this up for me?
 

summary_of_the_review: This is a novel idea (as far as I know) for growing networks, with a good intuition: add units so that they maximize the gradient, rather than minimize the loss. This has good effects on the learning dynamics. Unfortunately, the writing and organization dampens enthusiasm for the paper, and the experiments seem a bit haphazard. I think this paper could be improved considerably by addressing these issues.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to grow neural networks by adding neurons in a specific way. While requiring that the forward pass stays unchanged, the authors want to initialize the new weights such that the gradient wrt to these weights is maximized. They show small but consistent improvements compared to the baselines although the proposed method seems considerably more complex than baselines, as far as I can see.

main_review: The paper is well written and easy to follow. Although I am not familiar with the literature, the idea seems novel and the experiments on common image datasets are comprehensive and well-executed. The improvements seem marginal but are consistent throughout the paper. 

Nevertheless, the motivation of the paper is not well explained and needs more mathematical and empirical justification. As far as I understand, the authors want to introduce (neurons and therefore) weights such that the overall gradient of the loss wrt to all the weights is minimized. I find this idea interesting but not well reflected in the considered optimization problem, equation 4. Since the introduction of the new weights also change the backward/gradient computation of layers k < l, it could be that although optimization problem 4 is well solved, the overall gradient decreases - which would go against the motivation/ method proposed. 
Please elaborate on this point and explain why this can't happen. The teacher-student experiments are not sufficient because no weights before the newly introduced W_l^new exist and I cant see anything about this for the larger models (with batchnorm). 

On top of this, this could be turned around and part of the method: How can we add new neurons such that we change/improve the backward pass in way that improves the gradient norm of the whole network e.g. by backward computation that circumvents vanishing gradients. In general, the motivation of the paper should be much more investigated and could be made much stronger.

Another important point that is missing in my oppionin is the careful handling of batchnorm layers. 
Scaling the norm of the (newly added) weights has no effect in the forward pass but scales the norm of the gradient in the backward pass, see e.g. (https://openreview.net/forum?id=rkxQ-nA9FX, https://arxiv.org/pdf/1910.07454.pdf). 
This goes hand in hand with your motivation but is never discussed for models when batchnorm is used. Please elaborate on this and show that your motivation and the proposed method still make sense with batchnorm layers. 

If the authors manage to convince me / improve my mathematical understanding and intuition of what the method is actually doing, I would consider raising my score. 

summary_of_the_review: Not sufficient and convincing evidence that the motivation of the paper make sense and is met in practice.
review_title: null

Meta-review: This paper looks into growing neural networks, and finds an improved approach to the initialisations of new layers, viz by maximising the gradient norm.  Simple, straightforward, neat, and no good reason to reject.  It will benefit those who are using growing NNs.
Decision: Accept (Poster)