Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this manuscript, authors propose an OFA NAS framework. They train a supernet first and then finetune the elastic version of the large network. After training, the sub-networks derived from the supernet can be applied for different scenarios directly without retraining. The motivation is clear and interesting. My concerns are as follows.
1.	When sampling sub-networks, a prediction model is applied to predict the accuracy of networks. It is interesting to show the accuracy of the prediction model itself and how it will influence the final selection.
2.	The results compared in Table 2 are outdated. Authors should at least add the result of MobileNetV3.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper tries to tackle the problem of searching best architectures for specialized resource constraint deployment scenarios. The authors basically take a two-step approach: First train a large network including all the small networks with weight sharing and some specially designed trick (e.g., progressive shrinking). Second, use prediction based NAS method to learn the performance/inference prediction module, from which the good sub architecture corresponding to a particular scenario is obtained. The experiments show that the proposed method is promising.

Pros:


1. It is an interesting new paradigm that tries to solve AutoML for different deployment scenarios “once for all”.  AFAIK there is no prior works thinking in this way.
2. It is useful and encouraging to see the proposed method achieves satisfactory performances, on par with the current best method specially designed for different deployment environment, while the computational cost is reduced by a large margin. 
3. Paper is clearly written and easy to understand.

Cons:

1. The motivation towards “progressive shrinking (PS)” is not that clear. It seems natural to train a large network, and from it to train sub structures, since overparameterization helps NN training. However, it is hard to imagine that training from large to small could eliminate the “interfering” of subnetworks, let alone “while maintaining the same accuracy as independently trained networks”.  To me it is neither theoretically nor empirically supported (Please note the training of subnetworks definitely affect the learnt weights of the big one through weight sharing). In particular, the subnetworks with weight sharing could achieve the same, or even better performances compared with those non shared counterparts, which seems too good to be true.
    1. A possible explanation might be that the overparameterization brings additional gain in the optimization process of each small network, especially with the help of knowledge distillation. If that is true, an additional ablation study should be done to separate the benefits of PS, and the disadvantage of weight sharing (i.e., interfering). 
2. I see no statements about code release. If a clear, and TIMELY code (for the SEARCH phase, not only for the Eval phase) release could be done, then at least from the perspective of application, the impact of this paper could be further enhanced.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this papers, the authors learn a Once-for-all net. This starts as a big neural network which is trained normally (albeit with input images of different resolutions). It is then fine-tuned while sampling sub-networks with progressively smaller kernels, then lower depth, then width (while still sampling larger networks occasionally, as it reads). This results in a network from which one can extract sub-networks for various resource constraints (latency, memory etc.) that perform well without a need for retraining.

This paper is well written, and the results are very good. However there are serious problems that need addressing.

The method as described *is not reproducible*. The scheduling of sampling subnetworks is alluded to on page 4, and that's it. It is essential that the authors include their exact subnet sampling schedule e.g. as pseudocode with hyperparameters. There is no point doing good work if other researchers cannot build off it. 

On another reproducibility note, as far as I can tell, the original model isn't given. There would be no harm in adding this to the appendix. 

Figure 1 is misleading, as we don't find out until later in the paper that Once For All #25 means that each of these points was finetuned for a further 25 epochs (which on ImageNet is non-trivial). This defeats the narrative of the paper (once-for-all plus some fine-tuning isn't exactly once-for-all).

Is there a reason why the progressive shrinking goes resolution->kernel->depth->width? Was this just the permutation that worked best? I would be curious as to why this is.

For elastic width, I wasn't sure why the "channel sorting operation preserves the accuracy of larger sub-networks". Could you please elaborate?

Kudos on adding CO2 emissions in Table 2, I hope this gets reported more often.

In the introduction, the authors talk about iPhones and then the hardware considered is Samsung and Google. A minor note, but it seems inconsistent.

Another minor note, in Table 2, (Strubell et al) should be out of the brackets, as it is part of the sentence.

Given that there are 10^19 subnetworks that can be sampled, it would be nice to see more than 3-4 appear on a plot. This makes it seem like they might have been cherry-picked. Sampling a few 100/1000 subnets and producing some Pareto curves would be both interesting and insightful.

Pros
-------
- Good results
- Well written
- Neat idea

Cons
-------
- Training details are obfuscated. This paper should not be accepted without them.
- Very few subnetworks of the vast quantity that exist are observed.

In conclusion, I am giving this paper a weak reject, as it is currently impossible to reproduce, and as such, is of no use to the community. If the authors remedy this I will gladly raise my score.
review_title: Official Blind Review #2

Meta-review: The authors propose a new method for neural architecture search, except it's not exactly that because model training is separated from architecture, which is the main point of the paper. Once this network is trained, sub-networks can be distilled from it and used for specific tasks.

The paper as submitted missed certain details, but after this was pointed out by reviewers the details were satisfactorily described by the authors. 

The idea of the paper is original and interesting. The paper is correct and, after the revisions by authors, complete. In my view, this is sufficient for acceptance.
Decision: Accept (Poster)