Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper Overview:
This paper is aiming at optimizing the OFA method in neural network model searching and training. Though CNN models perform well for many prolems, there is a serious shortage of this method. Researchers need to build and train a new model for every new problem, which will cost a lot of time and resources. A recent work, OFA, proposed a new method to partly solve this problem by training a family of models at a same time by parameter sharing. However, the OFA still have some problems. The searching space of OFA is too huge so the training process still takes too long and cost lots of resources. Thus this paper addressed a new solution, CompOFA, to speed up the training and try to get a balance between the accuracy and latency by building constraints between dimensions of model searching space and "progressive shrinking" approach.

Strengths:
1. This paper gives out obvious evidence of their basic insight. They use a heatmap to show the trade-off relationship between the width and depth of different models.
2. This paper uses real data and SOTA works to do evaluation, makes their result more convincing.

Weaknesses:
1. The design idea and motivation of this paper is not well addressed. How do you decide to build the constraints? Why the trade-off is necessary? As far as I know, most of AI models care most about their accuracy, not latency, Unless the model is really too huge and slow (obviously the evaluation part of this paper only includes very small models with ms level latency). Because the inference speed can be easily increased by adding more GPUs. Since the inference job does not need to do synchronization, its is easily to achieve the linear scale out ratio. The paper just use a very arbitrary way to determine the constraints between these dimensions. How do you know that this is a good trade-off that researchers want?
2. The design part is too brief. The first subsection just tell us "We decide to build a constraint like this", but why? The second subsection about speeding up the training is too ambiguous. Using some pseudo-code or program chart can help readers to understand your work's logic and innovations.
3. The evaluation part is not solid enough to support this papers claim at the introduction. It claims to reduce searching space from 10^19 to 243. However, it still costs 50% training times compared to the original OFA method. And the training results also does not show apparent gains. The accuracy gain in most cases is lower than 0.4% while the overall accuracy does not exceed 80%. It will be better to add a classical light-weight CNN to be compared with, such as AlexNet, VGG or ResNet.

Other Comments:
This paper's idea is quite creative and valuable that adding some constraints to the searching space will help us to remove many unnecessary models and improve the overall efficiency. However, the current design of the solution is not mature enough. It is necessary to add theoretical proofs for important design choices and convince readers that the choice is reasonable. And the evaluation part could also be improved by adding some classic models to compare.
review_title: A good idea but poor motivation, execution, and presentation

final_score: rating: 5: Marginally below acceptance threshold
review_content: #### Summary
This paper proposes to reduce the search space of OFA (a training scheme for obtaining networks for various deployment requirements) by scaling or shrinking the depth and width dimensions in NAS search space together. Because of the reduction, the multiple training phases of OFA is also less important and can be simplified to reduce training time.


#### Technical
* (+) The key observation made in this paper makes sense to me that the search space of OFA is over-sufficient and might be unnecessary for the target problem -- deployment in various environment with different latency/hardware constaints
* (+) the experiments of this paper looks good, the improvement over OFA in terms of training speedup and energy saving is impressive.

* (-)  I do appreciate the good engineering and intensive empirical results revealed by this paper but the technical novelty of this paper seems to me very increamental compared to OFa, though the arguments made in section 3.2 makes total sense me. For examples, the authors argued the model dimensions are not orthogonal, and the latency requirements only need to be satisfied in a way (granularity) that are below many thresholds. I think a better way to materialize these arguments is to explicitly characterize how these two observations should be incorporated into the architecture space, e.g., establishing direct correspondences between the design of the architecture space and the latency requirement, or by revealing how different dimension of architecture space are coupled (either theoretically or empirically ) and affect the design of the space. However, the paper ends up with a (not so intuitive) approach  -- that slightly modifies the space by coupling of two dimensions of the space, and starts discussing that by coupling these two dimensions, progressive multil-phase training can be eliminated to improve training.



review_title: This paper reduces the space of OFA with two tricks while still satisfying deployment requirements

final_score: rating: 7: Good paper, accept
review_content: This paper presents a compounding strategy for constraining the search spaces for once-for-all (OFA) network training framework. This is motivated by observations in prior work (particularly the EfficientNet by Mingxing Tan and Quoc Le) that certain compound relations exist between network dimensions (widths, depths, resolutions) for achieving optimal accuracy-latency trade-offs. In this work, network depths and widths are compoundly coupled. The resulting method called CompOFA reduces the search spaces significantly from 10^19 candidates to 3^5 (CompOFA) or 10^10 (CompOFA-Elastic) candidates, thus cutting the training costs by 50% or 30%. The models found with CompOFA/Elastic for specific hardware latencies are also more accurate than those found with OFA, despite requiring less training costs.

Paper's strengths
- Both smaller search spaces and the avoidance of progressive shrinking further reduce the training costs of an already efficient framework (OFA) while not sacrificing prediction performance is impressive. This makes it more practically useful.
- The proposed compounding strategy is well-motivated by existing findings and nicely integrated to the OFA framework. It also clearly demonstrates that (human) prior knowledge is still very relevant in automated search for efficient networks.
- The models searched by CompOFA match the performance of (or in some cases, outperform) OFA models, both in terms of best models given certain latencies and sampled models at population levels.
- The paper is generally well-written and includes sufficient references to prior work.

Paper's weaknesses
- The pre-defined coupling configurations [D:2, W:3], [D:3, W:4], [D:4, W:6] are heuristically chosen based on the default search space originally proposed by the OFA paper. CompOFA works well because potentially these numbers happened to quite effective. In the more realistic settings (e.g., other datasets, tasks) where we do not have access to the prior numbers, it may be difficult to achieve good performance with a small number of coupling configurations, and OFA may work better to explore different combinations in a larger search space.

- The compound coupling only considers network depths and widths, and only a small modification to the design space is proposed to achieve that. If the proposed method considered image resolution, the paper would be more complete in the context of compound scaling and it would have a stronger contribution.

- In the OFA paper, smaller models are generated by shrinking the larger models, such that the smaller models (partially) retain the weights of the larger models. In this paper, all the smaller models are generated and trained at the same time. However, there is no information in this paper on how they are derived from the teacher network, e.g., whether they partially share the weights with teacher or they are distinctly initialized.

- While it is interesting that CompOFA models could outperform OFA models in some cases, this paper does not make it clear why this is the case. CompOFA's search space is supposedly a subset of OFA's search space. The models trained by CompOFA should be covered by OFA. 

- Code is not provided in the submission for reproducibility and there is no promise of code release.

Minor comment
- Sec 5.2 mentions "by using just one stage of training after the teacher model" but the paper should explicitly mention that this applies only to CompOFA (fixed kernel). CompOFA (elastic kernel) in Table 4 actually still requires multiple stages.

This paper's ideas and contributions are nice-to-have but they are far from being groundbreaking.

##post-rebuttal##

I share the concerns of others reviewers that this paper has a limited novelty and narrowed scope but I think the authors have addressed other issues/concerns quite well. In my opinion, the key contributions mostly come from the insight and experimental results, and less so on the heuristic itself. Thus, I would adjust my rating to 7.
review_title: Pretty impressive results with a minimal (but important) search space modification

final_score: rating: 6: Marginally above acceptance threshold
review_content: Deploying models on multiple target end points with different hardware spec requires training model variants of an underlying architecture that meet the latency and other resource constraints on the device. OFA is a framework that helps this search for architectures like mobilenetv3 by training a model with large components and then training multiple instances of smaller models wherein components have reduced size so as to fit multiple target devices. Due to the large state space of parameter sizes for subcomponents, this search can take a while.

This paper proposes a heuristic to bring down the size. The main premise here is that searching for various subcomponents size parameters as if they were orthogonal design choices is not optimal. Good choices for parameters, for example the depth and the width are correlated. Therefore in place of searching as though these were orthogonal, the authors propose a correlated search to reduce the number of possibilities. This halves the overall search time, which is substantial saving given the time of the overall search. 

The experiments are comprehensive on backing up the premise for the limited choice of data set and architecture considered. They show consistent improvements in overall training time  while showing the quality of resulting architectures is comparable to OFA methodology. This is done by showing the accuracy and size distribution of models obtained with this method vs OFA. 

My concern with this paper is that the heuristic, while useful in this case, is not particularly interesting. Nor does the paper establish that this is applicable to other architectures. The result is that the scope of this paper is narrow. I would like more evidence to demonstrate that there are generalizable principles here. 
review_title: a heuristic for reducing architectural search time for compact models based on mobilenet architectures

Meta-review: This paper focuses on  once-for-all (OFA) network training towards developing accurate models for different hardware platforms and varying latency constraints. The paper proposes an approach to significantly reducing model search space and thus training costs without losing in predictive performance. The paper is well-written, and the authors provide a thorough and convincing response to the concerns raised by some of the reviewers.
Decision: Accept (Poster)