Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes new methods of Once-for-all(OFA) network training methods, which significantly reduce the training time as compared to the existing OFA methods. Specifically, this work:
- Eliminate the need of pretraining the teacher network with in-place distillation
- Develop an upper-attentive sampling method which always sample the full-sized sub-net at each iteration to help co-train the rest sub-networks
- Use an upper-attentive warmup technique which trains only the full sized sub-net for a few epochs before co-training to further improve the performance

With these optimizations, this work can decrease the number of sampled sub-networks in each iteration of training, further reducing the total training cost.

main_review: Strength:
- Overall this paper is well written and easy to follow.
- The reduction in training time is indeed decent

Weakness:
- Limited novelty.
- Lacking the deeper explanation of the improved results.

summary_of_the_review: - **Limited novelty**.
It seems to me the two major contributions in-place distillation and upper-attentive sampling are already proposed by the existing works. It is a little unclear what further modifications the authors did other than stitching these two techniques together. Would be great if the authors can give some stress here considering the improvements in the results.

- **Lacking the deeper explanation of the improved results.** 
In the results part, it seems to me that the proposed methods just magically improve as compared to on the baselines. Not clear the underlying reasons and insights for them. Would be nice for the authors to point out as I might miss them.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript aims to reduce the training cost of once-for-all networks. The proposed method is built upon CompOFA that reduces the number of sub-networks within the OFA network to 243 by only considering networks whose dimensions are coupled. This paper introduces several techniques to further reduce the training cost, including in-place distillation, upper-attentive sampling, and upper-attentive warmup.
main_review: Strengths
1. The training cost saving is clear and significant while maintaining the same level of accuracy as OFA and CompOFA.
2. The proposed method can potentially benefit real-world deep learning applications. 

Weaknesses:
[major]
1. Limited novelty. Although this paper presents good results, the novelty of the proposed method is a bit weak. For example, in-place distillation is not new. Upper-attentive sampling and upper-attentive warmup are more like engineering tricks. 
2. This paper only presents results on one design space (mobilenetv3). But, in practice, MobileNetV3 is not supported or not efficient in many cases (e.g., on FPGA, GPU, etc). Showing the generalizability of the proposed method on other design spaces (e.g., MobileNetV2) is important. 

[minor]
1. On ImageNet, it seems that only supporting 243 sub-networks is sufficient for NAS. But it is unclear whether this strategy also works under the transfer learning setting. 

summary_of_the_review: This paper presents several simple methods to reduce the training cost of once-for-all networks, showing clear and significant training cost saving without losing accuracy. But, I find the novelty of the proposed method is a bit weak, and the generalizability of the method on other design spaces and transfer learning is unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this manuscript, the authors propose a new framework for training the once-for-all (OFA) networks. This framework uses an in-place distillation schema to replace the teacher training in the former OFA training fashion and makes the overall training time faster. The paper is overall well-written with good performance.
main_review: In this manuscript, the authors propose a new framework for training the once-for-all (OFA) networks. This framework uses an in-place distillation schema to replace the teacher training in the former OFA training fashion and makes the overall training time faster. The paper is overall well-written with good performance.

My concerns are as follows.

- The novelty for this paper is limited: this paper claims four contributions they made, but the idea for training without the teacher network borrowed from the in-place distillation paper without much modification. The upper-attentive schema simply samples the biggest network in each phase, which can be regarded as "merging“ the training for "teacher network" and "elastic network" in one phase, but the author does not show the performance boost for the schema (compared to without using upper-attentive).
- The OFA aims to alleviate search cost across different hardware platforms, but the comparison in the experiment only include little platform. The author should prepare more experiments across different platforms (mobile, CPU, GPU, FPGA, etc...).
- It is not clear how each component in this paper contributes to the final performance for the reduction of training time, the author should do an ablation study to justify this.

I will raise my vote if the concerns are addressed.
summary_of_the_review: The paper is overall well-written, but the novelty for this paper is kind of limited. The author should prepare more experiments to justify the performance and address the technical contribution for this paper.

I will raise my vote if the concerns are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a method to train a once-for-all network, where one network can run at different resource constraints. The method is based on previous methods, and the author further improved the training speed by around 1.5x - 1.8x without loss of performance. The method is evaluated on ImageNet classification.
main_review: Strength:
1. The proposed method improved the training speed without loss of accuracy.
2. The paper is easy to follow.

Weakness:
1. The proposed method is of very limited novelty and technical contribution. Looking at the contribution part, (1) in-place distillation is used in Slimmable Network. (2) The upper-attentive sampling is basically Sand-wich rule in Slimmable Network, the only difference is that the author doesn’t sample the smallest sub-network. (3) The warm-up training is also a very incremental and commonly-used technique. The proposed method is basically a combination of previous techniques.
2. The results seem to be not consistent with that reported in the paper. For example, in Table 3, the performance of OFA seems to be lower than that reported in the original paper.
3. Some missing related works: \
[1] NestedNet: Learning Nested Sparse Structures in Deep Neural Networks \
[2] MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution \
[3] Dynamic Slimmable Network 

summary_of_the_review: Overall, I think the paper has limited technical contribution, and the training speedup is not surprising (less than 2x). If it is a combination of techniques but significantly improved the training speed (say 5x-10x), which makes such network training easily-doable for everyone, then I would say it is a large contribution. Currently, I think the author need to further improve their method.

======== Post Rebuttal ========

Thank the author for the response. My concern about the novelty and significance of the method and results still remain. I agree that combining existing techniques may need more tuning. But this is hard to be considered as enough contribution, especially given that the performance is not significantly improved. Therefore, I will keep my score.
review_title: null

Meta-review: This paper exposes a method to reduce the training cost of once-for-all networks.
Overall this paper is well written and easy to follow, and the experimental section shows a clear reduction of training time on the examples used.
However, the reviewers point out that the experimental section could benefit from adding more design spaces, and have a better explanation of the results. More importantly, three out of four reviewers agree that the novelty of this work is too low for the submission to be accepted, with the fourth one only giving a score of 6 (and also noting the lack of novelty). I therefore recommend reject for this paper.
Decision: Reject