Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies a very interesting topic: automatically grow filters and layers in neural networks and find an "optimal" width and depth for neural networks. The method is motivated by SPLITLBI, and its effectiveness is verified by experiments and comparison with AutoGrow. I tend to accept this paper, before the following questions can be answered:
1. I guess the major issue in this paper is that the method is not clearly explained and rigorously formulated, although it's an extension of SPLITLBI.
-- 1.1. what's Γ? Is it a copy of W or not? What's the exact mathematical function of loss L() w.r.t. W and Γ? How does the neural architecture change after adding Γ?
-- 1.2. why Γ can be approximated by gradients in Eq. (3)? What's the intuition behind?
-- 1.3. why Γ is necessary? How does it compare with enforcing group Lasso on W directly, like what was done in Nonparametric Neural Networks [1]?
Without clarifying those, people can hardly learn from and use this paper.

2. Experiments
-- 2.1. Include the learned width in Table 1.
-- 2.2. In comparison with AutoGrow, the pairs of ResNet is fair, but the pairs of PlainNet is hard to judge because different neural architectures are used. AutoGrow uses 4 blocks while this paper uses 5 blocks. It's unclear if the benefit comes from the method or just from an additional block.
-- 2.3. In the experiments of layer growing, please clarify if filter growth is also applied or not.
-- 2.4 clarify "their growing process is not efficient." If I read the AutoGrow paper correctly, efficiency is one of the their claims and they showed that the growing process is as fast as "training a single DNN", and they scaled to ImageNet, which is not covered in this paper.

Minors:
1. networks with "20 filters" and "100 neurons" are used as the seeds. How critical are they?
2. "To the best of our knowledge, this is the first algorithm for BoN that can simultaneously learn the network structures and parameters from training data." is over-claimed. Lots of pruning methods can do it, although they start from a large one and prune it down.

[1] Philipp, George, and Jaime G. Carbonell. "Nonparametric neural networks." arXiv preprint arXiv:1712.05440 (2017).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes an architecture search method for deep convolutional neural network models that progressively increases the number of filters per layer as well as the number of layers, and the authors refer to this general approach as boosting networks. The algorithm for increasing the number of filters is based on split linear Bregman iteration, and the algorithm for increasing the number of layers proceeds block by block, increasing the layers per block until the accuracy does not increase. The experiments convincingly demonstrate gains in performance and smaller network sizes compared to baseline models, naive boosting methods, and a related method called Autogrow.

In my view, there are two main areas for further improvement for this work. First, the GT-Layers algorithm can be better motivated. Why is GT-Filters only run once at the beginning, rather than iteratively as the number of layers increases? Why does the procedure go from bottom blocks to up blocks (and, by the way, what are bottom blocks and up blocks)? Why measure training accuracy to determine when to add layers? I understand that these are all fairly heuristic choices, but nevertheless there needs to be proper motivation for all of the above.

Second, an additional effort should be made to compare to additional prior work in architecture search. It seems like the authors are suggesting that this method should be more computationally efficient and find smaller architectures, and demonstrating this empirically would greatly strengthen the paper. In particular, the existing results depicting the final number of parameters in the learned model are particularly striking to me. I appreciate that the authors included an experiment showing that a standard ResNet cannot be trained with the same number of FLOPs as the network found by your method. Can a similar analysis be made for wall clock time, i.e., how long the models actually take to train? A similar study (FLOPs, wall clock time, etc.) would also be very useful for the current comparison to Autogrow, as these metrics are often just as important, if not more important, than the number of parameters of the final model.

A thorough pass through the paper for spelling and grammar would be very useful.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper focuses on topic of searching for the optimal architecture for the deep network. Building on the split linearized bregman iteration strategy, the authors propose two practical algorithms to boost network, namely GT-filters Alg and GT-layers Alg. The proposed algorithms can simultaneously grow and train a network by progressively adding both convolutional filters and layers. The experiments conducted on VGG and ResNets display the comparable accuracies between the BoN and the standard big models, but with much more compact representations and balanced computational cost.

Strengths:
1 The authors introduce two simple but practical algorithms for augmenting the architectures of deep network. The quite promising results are achieved on baselines, w.r.t. the balance of prediction performance and model complexity with the budgeted computational resources.
2 The paper is clearly written and easy to follow. 

Weaknesses:
1 The proposed BoN is built upon the existing SplitLBI algorithm that can identify the sparse approximation of the weight structure. The contributions are mainly from incorporating the adaptive criteria for adding filters and layers. Thus contribution is incremental and novelty is limited. 
2 The projection operator in equation (4) is a key step of the boosting procedure, but is not clearly defined and explained.
3 For experiments, results shows good results on simple baselines, more complicated or large-scale datasets should be included for evaluations. Also, only Autogrow is compared to the proposed method, to make the results more convincing, other architecture searching approaches (e.g. NAS) should be added to comparisons.

Other questions:
1 For the GT-layers Alg, would it more efficient to boost layer first before boosting the filters? 
2 Will GT-layers be robust to the case when the number of filters is overly specified?
review_title: Official Blind Review #1

Meta-review: This paper considers how to learn the structure of deep network by beginning with a simple network and then progressively adding layers and filters as needed. The paper received three reviews by expert working in this area. R1 recommends Weak Reject due to concerns about novelty, degree of contribution, clarity of technical exposition, and experiments. R2 recommends Weak Accept and has some specific suggestions and questions. R3 recommends Weak Reject, also citing concerns with experiments and writing. The authors submitted a response that addressed many of these comments, but R1 and R3 continue to have concerns about contribution and the experiments, while R2 maintains their Weak Accept rating. Given the split decision, the AC also read the paper. While we believe the paper has significant merit, we agree with R1 and R3 on the need for additional experimentation, and believe another round of peer review would help clarify the writing and contribution. We hope the reviewer comments will hep authors prepare a revision for a future venue.
Decision: Reject