Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a NAS method in the context of many-task many-hardware few-shot learning. To this end, the authors introduce many techniques, including supernet meta-training strategy, layer-wise profiling for latency evaluation, and etc. The experimental results show the effectiveness of the proposed method in some cases.

After rebuttal: I tend to give Borderline, but in system, I only can choose from *Borderline Accept* and *Borderline Reject*. Thus, I keep the scoring Borderline Reject. But I lower the confidence level from 4 to 2.
strengths_and_weaknesses: 1. The authors consider a NAS problem for many-task many-hardware learning, which may be useful for real-world applications.
2. To address the issue the architecture may be over-fitting, the authors propose global expansion and adaptive number of layers in the backbone design.

questions: 1. The novelty of this paper seems to be limited. The proposed method seems to be more like a simple stacking of different techniques. More critically, some components in the proposed method have been studied in existing methods, such as super-net meta-training strategy, layer-wise profiling.

2. In Introduction, the authors emphasize the efficiency of the proposed method, i.e., reducing the search complexity from O(THC) to O(1). Compared with OFA, I am confusing which proposed component make the search cost to reduce to O(1).

3. Layer-wise profiling has been proposed and investigated in FBNet. In my opinion, this is not a good way to evaluate the latency of candidate architectures. In practice, the inference framework (e.g., TensorRT) would perform layer/operation fusion. For instance, TensorRT will fuse conv and batch norm layer into a single layer. In this case, the overall latency of the architecture is not equal to the sum of that of each layers. The reality is often more complicated than the above case. Thus, I think it would be difficult to accurately estimate the latency.

4. The compared methods in Tables 3 and 4 seem to be outdated. Most of the compared methods/papers are published before 2020, which make it hard to verify the effectiveness of the proposed method.

5. I think the word “Adaption” in the paper title means adapting from an existing architecture to another. But after reading the paper, the proposed method is more likely to search for new promising architectures rather than adaption. I am not very sure about this. It may be my misunderstanding.

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a unified framework, namely rapid model architecture adaption, for joint searching the best architecture given multiple tasks (in T), various hardware (in H) and various constraints (in C). Traditional NAS methods have a complexity of O(THC) where this work has a complexity of O(1)

In essense, this paper glues the existing techniques from three domains, i) neural architecture search such as global expansion and adaptive number of layers, ii) a naive adaption meta learning strategy to improve the 1shot and 5-shot ability, and iii) look-up table like performance benchmarking technique in FB-Net NAS paper to profile the hardware latency. 
strengths_and_weaknesses: The strength of this paper is obvious, it does not try to complicate the current literature by introducing some articulated complex search strategy but try to formulate a novel problem, that minimizing the multi-task multi-hardware and multi-search space in one step. The experiments demonstrate the effectiveness of this method. 

On the other hand, it is obvious that this paper does not have much technical novelty. The entire methods can be separated into independent procedures that aim to solve a specific problem. Compared to the relevant baseline Once-For-All, which brings the a novel solution for multiple device and constraints neural architecture search by leveraging the super-net and proposing those tricks like progressive shrinking and depth pruning, this work seems to incoporate several existing approach to tackle a similar problem. In short, the engineering contribution of this paper is significantly larger than the algorithmic one.  I am not sure if this paper introduces any new message to the current community. 



questions: As this paper is clearly written, the only question I can raise now is if we should accept a paper with many engineering efforts. In my humble opinion, this paper seems like a moderate modification of OFA, replacing the latency predictor by a straight-forward baseline, layer-wise profiling that is introduced in FBNet long time ago. Then adapts another simple baseline from meta-learning domain to minimize the O(T) into O(1). 

Post rebuttal comment:

This paper did find an interesting question in meta-learning, and I agree that this is critical and should be deemed as a solid contribution to the current research community. I raised my score to weak accept. 
limitations: I do not see direct negative societal impact of this work so far. 
As for the limitation of the method itself, to use the latency profiler, it requires manual benchmarking the latency on multiple devices and might be hard for researchers that has insufficient resources.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a method for many-task many-hardware NAS. The approach combines existing hardware-aware NAS with MAML. A layer-wise profiling strategy is introduced to improve profiling time, and modifications to the NAS algorithm to prevent overfitting in the few-shot setting is also proposed. The method achieves competitive experimental results on omniglot and mini-imagenet whilst being very efficient.
strengths_and_weaknesses: **Strengths:** 
The paper is clear and well written. The proposed layer-wise profiling is a simple solution that greatly outperforms latency predictor baseline in this setting. The proposed Global Expansion and Adaptive Number of Layers are also simple changes that I like. The experimental results are very strong.

**Weaknesses:** 
Layer-wise profiling is ultimately a heuristic and could have larger errors for strange computation graphs where the total runtime is not equal to the sum of the runtimes of each layer. That being said, experimentally this approach has a lower MSE than baseline label predictor so it's a great step up. 

The need to manually add a final pooling layer feels strange, as this is a NAS method one would have hoped that this could have been discovered automatically.

questions: Have the authors experimented with discovering how much pooling to add in the end rather than manually fixing it?
limitations: The authors have addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The author proposes a very interesting new method, that is, the adaptive adaptation problem of multi-devices in the domain of NAS. The idea of meta-learning is introduced into the NAS framework to search the underlying optimal neural network model. The contributions of this paper mainly focus on two aspects: one is the generality of the method, the network architecture obtained by the method in this paper can be adapted to multiple devices at the same time, without the need to search the optimal structure for each device separately; the second is the superiority of the proposed method, the search time is shorter than those comparison methods.
strengths_and_weaknesses: Strengths:
This paper provides a solution for complex task, which is novel and has important meaning in the field of NAS research.

Weaknesses:
1 The layout of Method section seems unreasonable. First, the Super-net meta-training strategy subsection introduces the training optimization strategy for architecture search, while the following two strategies (Layer-wise profiling and Adaption strategy) are some skills for network structure search. The order of the two seems to be reversed. Second, Tables 1 and 2 present partial results, and it seems more appropriate to move them to the Evaluation section.
2 The Method section lacks the focus of the core issue: when there are multiple different hardware devices, the author does not give the detailed description of how to set the hardware constraints. 

questions: but their final classification accuracy is not the same. If the results on three devices are different, then the goal of this paper seems unachieved. Please refer to sentence in the Introduction section: “Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this method offers a chance to reduce the hardware search complexity from O(HC) to O(1)”. How to explain this sentence?
limitations:  It seems that the author has only solved some of the existing problems, and has not provided the relevant results for the optimization goal of multiple devices. It is necessary to supplement the relevant experimental results in the revised version.
review_title: null

Meta-review: metareview: Reviewers agree that the proposed many-task many-hardware few-shot learning setup is well motivated and valuable in real-world applications. The main novelty of this paper lies in the meta-learning side, and their results reveal that metric-based MAML methods are not latency-friendly, and the community should look back into optimization-based MAML methods. Authors have devoted extensive engineering efforts into this paper, as recognized by several reviewers.

Previously raised concerns are mainly about the evaluation of overall novelty (e.g., similarity with OFA), possibly inaccurate layer-wise profiling due to layer/operation fusion, and missing comparison with more recent methods. Most of them have been well resolved in the author feedback; therefore, AC recommends acceptance.
confidence: Certain
Decision: Accept