Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a new principled approach to growing deep network architectures based on continuous relaxation of discrete structure optimization combined with a sparse subnetwork sampling scheme. It starts from a simple seed architecture and dynamically grows/prunes both the layers and filters during training. Through extensive experiments, the authors show that this method produces more efficient networks while reducing the computational cost of training, still maintaining good validation accuracy, compared to other NAS or pruning/growing methods. 

Strength: 
(+) The proposed idea of formulating the problem as a continuous relaxation of discrete structure optimization is interesting. It seems to be a more principled approach than previous NAS or separate pruning/growing approaches. 
(+) Extensive experimental results are provided to verify the superiority over recent other methods and also to show the performance behavior of the proposed method. The overall experimental setup is systematic and comprehensive. 
The experiments were done on widely used deep networks on various tasks.

I only have concerns about the clarity of the notation and the representation of the figures. Specific examples are as follows: 

In Eq. (3), it is said that f is the operation in Eq. (1). However, I couldn’t find f in Eq. (1).

It should be clarified how many temperature parameters β are in the proposed model. Only one or as many as channels and layers? If it is only one, it does not seem reasonable that all the probabilities growing or pruning channels and layers are the same.  Equation (7) seems to imply β to be a vector, but earlier notations (e.g. in Algo 1, Equation 6, etc.) seem to present it as a scalar.

Overall, there are confusing symbols, whether it’s a scalar or a vector. I recommend the channel and layer indicators are denoted as vectors. It seems that each channel and each layer has its unique indicator, respectively. Also, notations should include channel and layer index if they are different depending on channels and layers.

Additionally, all the experimental results shown in the main manuscript are on convolutional neural networks while the abstract mentions recurrent neural networks. The appendix has some, but very little has the main manuscript. If it’s an important part of this manuscript, the authors should include at least a brief summary of the results. 

Some figures (and the text inside) are too small while containing many details, probably because of the space limit. For example, Figure 3 has many lines that are hard to analyze and texts that are not readable. 

In Table 1, what’s the meaning of the underlines? I guess the second best results, but for RestNet-20, the method with the second-best FLOPs is SoftNet, not Provable. And the explanation about the boldface and underlines should be included. 

review_title: Principled approach to growing deep networks 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel NAS method that searches the model architectures by grows the networks. This searching strategy determines the channel and layer configurations by assigning a binary learnable parameter m for each channel or layer. The objective is to optimize a trade-off between the model performance on the given task and the regularization on the binary indicator m.

Pros:
1. The general idea of searching the architectures by growing the networks sounds very interesting. The authors propose a novel framework to achieve their idea, and also apply some tricks to speed up and simplify the optimization (e.g. budget-aware growing and learning by continuation). 
2. The paper is well-written and easy to follow.
3. The authors conduct a series of solid experiments to verify the effectiveness of their proposed methods. The experiments show the performance of channel pruning, the remarkable improvement on AutoGrow model, and the comparsion with other NAS methods.

Cons:
1. Compared to ProxylessNet, the proposed model can reduce half of the training time but does harm to the model performance.

Questions:
1. What's the exact meaning of "Top-1 valiadation accuracy"? What's the different with Top-1 accuracy? Is this metric evaluated on the valiadation set?
review_title: Official Blind Review #3

final_score: rating: 7: Good paper, accept
review_content: **Pros**:

1. This paper nicely unifies two different classes of approaches (NAS + sparsity) for determining the topology of neural networks. They are combined into a single optimization problem, with binary indicators on network components and connections.

2. Experiments illustrate the behavior of the method. It is good to see that the experiments dig a big deeper than end-result accuracy. For instance, the "budget-aware growing" is shown well to work as described by Fig 3.

**Cons**:

3. No attention to random seeding.

The sparsification dynamics seem likely to change somewhat from one run to the next. The submission does not describe how random seeding was done for training. Multiple runs with different seeds are not shown, and the distribution of accuracies across runs is unknown. Attention to randomness for this kind of training process seems especially important given the variances in results in the Lottery Ticket hypothesis paper.

4. No comparison to simple random baseline.

A large portion of the method consists of a search method over the space of possible sparse networks, combining it with growing the network to get a NAS-like method. It has been observed, though, that in a sufficiently general space of this kind one can randomly sample connections and see high accuracies. So to identify the sources of empirical gains, it is good to consider experimental baselines, such as random sampling, that separate the contribution of the search space and the search method:

  * Xie et. al. "Exploring randomly wired neural networks for image recognition"
  * Li & Talkwaker "Random search and reproducibility for neural architecture search"
  * Yu et. al. "Evaluating the Search Phase of Neural Architecture Search"
  * Radosavovic et. al. "On Network Design Spaces for Visual Recognition."

Note that the submission's method also is randomly choosing connections, through a somewhat involved process that also accounts for the observed sparsity of G during training. The simplest baseline seems to be the "uniform pruning" described in Section 4.4. This only ablates part of the method that doesn't seem to meet the same criterion here.

5. Incomplete illustration of the cost/accuracy tradeoff.

The gold standard for comparison in both sparse-neural-network papers and NAS is to consider the accuracy at a range of different model costs. See for example:

  * Blalock et. al. "What is the state of neural network pruning?" Figs 1, 3
  * (Yang et. al., 2018) Figs 5-9

This clearly illustrates whether a method is overall better (i.e. produces better models across the entire pareto frontier), or is only better for some ranges or on one metric. For a result like the first one in Table 5 in the submission, it is unclear which model is better: they may simply be considering different points on the same cost/accuracy curve.

6. As a less important aside, "budget-aware growing" seems to be an ad-hoc reinvention of something similar to an Augmented Lagrangian method. Explicitly describing the differences from standard optimization techniques might be good.

**Reasoning for rating**:

While the experiments are extensive, I think they miss the key comparisons that show how useful the method and each of its components is. Given that many different innovations are included in the submission, it may be a muddle for follow-up research to sort out how good each individual one is.

**Misc comments**

  * Check spacing around (6) in Algorithm 1
  * Colon instead of comma after "trainable variables" in §4.1
  * "For better analyze the growing patterns" -> "To better analyze the growing patterns" on page 14
  * Wortsman et. al. "Discovering Neural Wirings" is another closely related work at the intersection of NAS and pruning. (with major differences from the submission)

**After rebuttal**

The authors have gone above and beyond in providing additional experimental results. All of the points raised above that deal with methodological issues are completely addressed.

The sole significant weakness that remains is the lack of the kind of ablation/component studies that would justify individual design decisions. I do not disagree with the authors that this will be difficult for this work, but I still feel they would have been helpful for researchers who will be building upon this method. 
review_title: Big idea, with extensive experiments

final_score: rating: 7: Good paper, accept
review_content: Summary:

  This paper proposes a NAS-type work for growing a small network to a large network by adding channels and layers gradually. The authors apply the method to both CNN and LSTM networks. 

Strong points:

  1. This paper is well-written and shows good results.

  2. The proposed algorithm is sound and effective. E.g. use less wall-time as compared to other NAS approaches.


Weak points:
  
  1. It seems that the number of channels and number of layers still need to be predefined (the mask size).


Questions:


  1. How does the FLOPs reduction translate to runtime saving?

  2. What is the target sparsity u in the experiments?

  3. When growing with layers, does the author observe any middle layer is dropped and then recovered? If so, does it happen frequently? 

  4. At section 4.2 and 4.3, what is the size of the channel/layer mask? I believe the author still needs to define the upper bound of the network can grow. If so, does the upper bound affect the optimization?  Or the proposed method gradually expands the mask?

  5. In table 4, I think Efficient-B0 should be taken into consideration as it is a recent representative approach.
  
After rebuttal:

The authors' rebuttal addressed all my questions and I upgrade my rating.


review_title: Reviews

Meta-review: The paper proposes a method to grow deep network architectures over the course of training. The work has been extremely well received and has clear novelty and solid experiment validation.
Decision: Accept (Oral)