Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: N/A (Reproducibility Review)
overall_review: N/A (Reproducibility Review)
review_summary: N/A (Reproducibility Review)
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper introduces a software framework (BootstrapNAS) for supernet-based NAS. Given a pre-trained model, it provides an easy-to-use interface for converting it to a super-net, training the super-net with different algorithms, and searching for good sub-networks under different constraints (MACs, latency). The authors also promise to release the code to the research community. 

overall_review: [pros]
1. This paper is well-written.
2. The authors promise to release their code to the research community.
3. This paper provides extensive experiments on different datasets (ImageNet, CIFAR10, CIFAR100, VFX), showing the generalization ability of their framework.

[cons]
1. This paper does not present new algorithms/methods. But, I think it is fine for a software framework paper. 
review_summary: Overall, I think this is a good software framework paper that will be useful for many practitioners.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This work describes BootstrapNAS, a software framework for one-shote NAS. Bootstrap supports the (1) generating and training a supernet based on a pre-trained model (e.g., ResNet-50), (2) and search for the high-performing subnets from the trained supernet. BootstrapNAS also provides APIs for the users to develop their own training strategy, search algorithms, or performance estimators.

The authors provide results on several supernets (e.g., ResNet-50, MobileNetV3, ..) and several tasks (image classification and segmentation) to demonstrate the usefulness of BootstrapNAS.
overall_review: Overall I think this is a nice work as the proposed software BootstrapNAS would make it much easier to apply one-shot NAS to existing architectures/search space. The flexibility of the APIs in BootstrapNAS will also make it easier for future research to develop new methods.

My main concerns are about how the “pre-trained” weights are used in BootstrapNAS (see Clarity for more details). This is not just for clarity, as I think they are important technical details that need to be very clear in the paper.

The x-axis in Fig. 6 needs to be re-designed. Now it looks like the  MACs are reduced by more than 10x, although it is just 12% fewer for EfficientNet-B0.

review_summary: I recommend acceptance due to the usefulness of the proposed software framework.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors present a method and framework for automatically constructing a super-network / one-shot model from a pre-trained architecture, overcoming the need for carefully and manually defining one-shot models. This is achieved by varying the given, pre-trained architecture across 3 dimensions: depth, filter and kernel sizes. 
The authors build upon the Network Compression Framework (NNCF) (Kozlov et al., 2020). 
Once the super-network is build, different strategies can be employed to train the super-network and to search for optimal architectures within the super-network.  The authors additionally consider multi-objective optimization and search for architectures that are optimal w.r.t., e.g., accuracy and latency. For this, off-the-shelf optimizers such as NSGA-II can be used once the super-model is trained.

The authors evaluate their framework with various pre-trained architectures (ResNet, MobileNetV2, V3, EfficientNet) and on multiple problems (image classification for Imagenet and Cifar10 + video segmentation). The proposed framework consistently improves over the baseline architectures, either in terms of accuracy or efficiency or both.
overall_review: + practical framework for automatically constructing super-model given some baseline architecture
+ addressing problem of high practical relevance with little prior work
+ different strategies for training super-model and searching can be employed --> flexible and adaptable framework

- super-network is a bit limited as only depth, filters, kernel sizes are varied (see review summary for more details)
- no major methodological contributions

A study on the computational costs for training and searching the super-model (relative to the cost of pre-training the baseline architecture) would be interesting. What would happen if the baseline architecture is not pre-trained? How would this affect the computational cost and performance of the method? 

Minor comment: the reference for the 'once for all' paper (Cai et al.) seems broken. In general, citation style is a bit inconsistent and unclean.
review_summary: The authors propose an interesting framework for automatically constructing a super-model for architecture search given some baseline architecture. The framework seems flexible, easy to use and easy to adapt. The authors address an interesting problem and thus the framework could be of great interest for the community.

The only major concern I have is that the super-model is fairly limited since it only varies depth, filter and kernel sizes. Also, the given baseline is the architecture with maximum size within the super-model, i.e., no larger architecture can be discovered by definition. It would be interesting to have more options here, e.g., for also varying the layer types / operations and for also finding larger architectures. E.g., one could employ network morphisms [1,2,3] to expand the baseline architecture. A similar idea as presented in this work is 
 employed by [4], which should therefor also be discussed in this work.

Minor final comment: [5] might be worth adding in the discussion on available frameworks in the related work.


[1] Net2Net: Accelerating Learning via Knowledge Transfer, Chen et al.
[2] Network Morphism, Wei et al.
[3] Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution, Elsken et al.
[4] Fast Neural Network Adaptation via Parameter Remapping and Architecture Search, Fang et al.
[5] NASLib: A Modular and Flexible Neural Architecture Search Library, Ruchte et al.
review_title: null

Meta-review: metareview: The paper proposes a software framework to help ease the process of training supernets for NAS. The paper also demonstrates the utility of the proposed framework for multi-objective NAS on both classification and dense-prediction tasks. The paper has received 1 strong accept and 3 accept ratings. All the reviewers agree that the paper would be impactful and the software framework would be valuable to the research community. I fully concur with the reviews. The paper is well-written and clear for the most part.

Therefore, I recommend acceptance for the paper.
confidence: 5: The area chair is absolutely certain
Decision: Accept