Rapid Model Architecture Adaption for Meta-Learning

Abstract: Network Architecture Search (NAS) methods have recently gathered much attention. They design networks with better performance and use a much shorter search time compared to traditional manual tuning. Despite their efficiency in model deployments, most NAS algorithms target a single task on a fixed hardware system. However, real-life few-shot learning environments often cover a great number of tasks ($T$) and deployments on a wide variety of hardware platforms ($H$). 	

The combinatorial search complexity $T \times H$ creates a fundamental search efficiency challenge if one naively applies existing NAS methods to these scenarios. To overcome this issue, we show, for the first time, how to rapidly adapt model architectures to new tasks in a \emph{many-task many-hardware} few-shot learning setup by integrating Model Agnostic Meta Learning (MAML) into the NAS flow. The proposed NAS method (H-Meta-NAS) is hardware-aware and performs optimisation in the MAML framework. MetaNAS shows a Pareto dominance compared to a variety of NAS and manual baselines in popular few-shot learning benchmarks with various hardware platforms and constraints. In particular, on the 5-way 1-shot Mini-ImageNet classification task,  the proposed method outperforms the best manual baseline by a large margin ($5.21\%$ in accuracy) using $60\%$ less computation.

Reviews and author responses:

 Dear reviewer,

The discussion period is ending soon. We appreciated your previous thoughtful review and made the suggested layout changes in our Method section. We also added an answer to your question in our previous response. 

Please feel free to let us know if you have any further questions, so that we may at least has a final chance to clear up some of them before the discussion period ends. 

If you do not feel there are any futher questions, we would greatly appreciate if you can raise the score to full acceptance.

Thanks.

 Dear reviewer,

The discussion period is ending soon. We appreciated your previous thoughtful review and believe that our response has covered most of the issues that you have raised. 

Please feel free to let us know if you have any further questions, so that we may at least has a final chance to clear up some of them before the discussion period ends. 

If you do not feel there are any futher questions, we would greatly appreciate if you can re-consider the score.

Thanks.

 We thank the reviewer's supportive comments on our novelty. 

This is in fact in line with the logic behind our novelty claim: we first doubted the ideological shift to Metric-based MAML in the meta-learning community, and then decided to challenge the their performance by applying NAS on Optimization-based MAML.

If there are any further actions we can do to help the reviwer to raise the score, please feel free to let us know before the discussion deadline. 

Many thanks.

 I agree the author's point, and showing a novel insight on meta-learning side is much more important than developing some trivial but complicated technical modifications. 

Hope this can encourage people to discover more interesting topics in the future. I raised my score to weak accept. 

 Thanks authors for running this experiment, and apologies for the late response.
I'm happy with the results, to me this was the biggest weakness and has been elegantly resolved.

Regarding other reviewers' concerns about novelty, I believe Prof Michael Black has a great point on novelty in general ([link](https://medium.com/@black_51980/novelty-in-science-8f1fd1a0a143)):
>>The idea is obvious because the authors just combined two well known ideas.
>
>Obvious is the opposite of novelty. So, if an idea is obvious after you’ve heard it, reviewers quickly assume it isn’t novel. The novelty, however, must be evaluated before the idea existed. The inventive novelty was to have the idea in the first place. If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea. 

Now I believe that the idea was novel before it was evaluated, or at least, the degree of improvement is large enough to be surprising, else why have no one else done this before?

 Dear Reviewer 7Ry8, we appreciated the thoughtful review, and available to answer any questions.


 Dear Reviewer DeFC, we appreciated the thoughtful review, and available to answer any questions.


 Dear Reviewer Wnsk, we appreciated the thoughtful review, and available to answer any questions.


 Dear Reviewer AK57, we appreciated the thoughtful review, and available to answer any questions.

 Many thanks for your comments and noting that our problem setup and the proposed method is novel!



### Answer to Question 1 
We thank the reviewer for the suggestion on the layout. We are happy to put Layer-wise profiling and Adaption strategy first, and then Super-net meta training strategy. Regarding the partial results in this section (Table 1 and 2), we are also happy to restructure and move them to the evaluation section. Please see the updated paper with these layout changes.

### Answer to Question 2
We pre-train the large super-net and run the adaption algorithm (Appendix D) to pick the suitable network with meta-trained parameters to deploy to the specific hardware-constraint-task combination. We have to mention that this adaption algorithm is very fast because it only contains inference runs (no training involved). We only trained the super-net once (O(1) training cost), and the actual training time is only slightly longer than standard meta-training, resulting in  a huge saving when compared to training networks – O(HTC) times for different deployments. We hope this is clarifies things and will add more details to the paper. 


 Many thanks for your comments and noting that our work is easy to follow!

The reviewer is correct that we did run a manual tunning on the pooling mechanism. In our experiments, we test pooling from $1\times1$ to $7 \times 7$ and concluded that $5 \times 5$ is optimal. We didn't include this originally in our search space because existing NAS algorithms do not really put the last stage pooling in the search space. The reviewer is also correct that this pooling strategy can be added to the search space, so we quickly ran our algorithm again with an additional search space of pooling ($1\times1$, $3\times3$, $5\times5$ and $7\times7$).

| Method | Accuracy |
| --- | ----------- |
| 
H-Meta-NAS + 5 × 5 Pool + GE + ANL | $56.35 \pm 0.73$ |
| H-Meta-NAS + searched Pool + GE + ANL | $56.42 \pm 1.02$ |

Unsurprisingly, the network picked $5\times5$ pooling. We also elaborated on this experiment further and added it to our Appendix (Section K) in the updated paper, we thank the reviewer for raising this question. 



 Many thanks for your comments and we are very glad to hear that you liked our evaluation on various hardware platfroms.

On the NAS side, we combined canonical NAS algorithms with Optimization-based meta-learning algorithms and worked on a novel problem setup that is both multi-device and multi-task, which can be seen as an extension to existing NAS algorithms.

**We would like to highlight again our novelty on the meta-learning side**, as we note in which is our last contribution mentioned at the bottom of page 2. The meta-learning community, as discussed in our related work, had an ideological shift and moved from Optimization-based methods to Metric-based ones. 

One major contribution of this work, as explained in our evaluation in page 9, is that with the correct network architecture Optimization-based approaches can have better or comparable accuracy to metric-based methods, but **with >1000X reductions in MACs** (Table 5, when compared to Meta-Baseline and DeepEMD). 

We suggest this is a useful knowledge for the meta-learning community going beyond simply proposing a NAS algorithm. **We hope this finding will encourage researchers in this field to look back into Optimization-based MAML.**


 Many thanks for your comments! If anything is unclear still after reading the responses, please do not hesitate to reach out.


### Answer to Question 1 

On the NAS side, we combined canonical NAS algorithms with Optimization-based meta-learning algorithms and worked on a novel problem setup that is multi-device and multi-task. 

We do use some established components from existing NAS frameworks. We would like to emphasise the novelty on the meta-learning side,which is also our last contribution mentioned at the bottom of page 2. The meta-learning community, as discussed in the related work section, had an ideological shift and moved from Optimization-based methods to Metric-based ones. 

One major contribution of this work, as explained in our evaluation on page 9, is that with the correct network architecture Optimization-based approaches can have better or comparable accuracy to metric-based methods , but **with >1000X reductions in MACs** (Table 5, when compared to Meta-Baseline and DeepEMD). We suggest that this is a useful knowledge for the meta-learning community and goes beyond simply proposing a NAS algorithm. 

### Answer to Question 2 and 5
OFA focused on finding the suitable network across different hardware platforms with different hardware constraints, this shrinks the complexity from O(THC) to O(T). The meta-learning framework helps to adapt to different tasks so that O(T) complexity is reduced to O(1). We no longer need to train from scratch O(THC) times, but instead we only train the super-net once and adapt the network to the downstream task, hardware, hardware constraint combination. 

###  Answer to Question 3
We agree and are aware of the fact that inference engines such as TensorRT would perform certain graph fusion optimizations. In fact, we deployed networks to Raspberry Pis and Pi Zeros using TVM, which is a popular inference compiler for various hardware backends and have performed not only graph-level optimizaitons but also hardware-level optimizations using the TVM runtime. As our Table 1 results suggest, the MSE error on Pi zero for our latency estimator is still fairly low. We already took some considerations about this graph fusion when building the profiling strategy, for instance, we bundle Convolution with BN and ReLU as a single ‘layer’ to profile. Most of the graph fusion optimizations on these frameworks are fairly static, a smart profiling strategy in our case is enough to get accurate readings for building a layer-wise estimator. It is also worth mentioning that, in theory, if the layerwise profiling is perfect, our MSE error should be zero. We suggest that some of the more dynamic optimizations on the hardware backends (eg. low-level fusion in LLVM) are causing these small MSE errors that we see. We thank the reviewer for pointing it out and will add the details about our profiling in the revised version of this paper.

###  Answer to Question 4
Regarding the compared papers, as we suggest in our Answer to Question 1, the meta-learning field has experienced an ideological shift. The papers compared in Table 3 and 4 are still important mainstream Optimization-based meta-learning algorithms. With this paper we hope to bring more attention to Metric-based meta-learning **because of its’ run-time efficiency**. Our empirical results suggest that Optimization-based MAML methods with well-tuned architectures can achieve comparable accuracy with significantly less latency overhead. **We hope this finding will encourage researchers in this field to look back into Optimization-based MAML.**


 This paper presents a NAS method in the context of many-task many-hardware few-shot learning. To this end, the authors introduce many techniques, including supernet meta-training strategy, layer-wise profiling for latency evaluation, and etc. The experimental results show the effectiveness of the proposed method in some cases.

After rebuttal: I tend to give Borderline, but in system, I only can choose from *Borderline Accept* and *Borderline Reject*. Thus, I keep the scoring Borderline Reject. But I lower the confidence level from 4 to 2. 1. The authors consider a NAS problem for many-task many-hardware learning, which may be useful for real-world applications.
2. To address the issue the architecture may be over-fitting, the authors propose global expansion and adaptive number of layers in the backbone design.
 1. The novelty of this paper seems to be limited. The proposed method seems to be more like a simple stacking of different techniques. More critically, some components in the proposed method have been studied in existing methods, such as super-net meta-training strategy, layer-wise profiling.

2. In Introduction, the authors emphasize the efficiency of the proposed method, i.e., reducing the search complexity from O(THC) to O(1). Compared with OFA, I am confusing which proposed component make the search cost to reduce to O(1).

3. Layer-wise profiling has been proposed and investigated in FBNet. In my opinion, this is not a good way to evaluate the latency of candidate architectures. In practice, the inference framework (e.g., TensorRT) would perform layer/operation fusion. For instance, TensorRT will fuse conv and batch norm layer into a single layer. In this case, the overall latency of the architecture is not equal to the sum of that of each layers. The reality is often more complicated than the above case. Thus, I think it would be difficult to accurately estimate the latency.

4. The compared methods in Tables 3 and 4 seem to be outdated. Most of the compared methods/papers are published before 2020, which make it hard to verify the effectiveness of the proposed method.

5. I think the word “Adaption” in the paper title means adapting from an existing architecture to another. But after reading the paper, the proposed method is more likely to search for new promising architectures rather than adaption. I am not very sure about this. It may be my misunderstanding.
 N/A

 This paper proposes a unified framework, namely rapid model architecture adaption, for joint searching the best architecture given multiple tasks (in T), various hardware (in H) and various constraints (in C). Traditional NAS methods have a complexity of O(THC) where this work has a complexity of O(1)

In essense, this paper glues the existing techniques from three domains, i) neural architecture search such as global expansion and adaptive number of layers, ii) a naive adaption meta learning strategy to improve the 1shot and 5-shot ability, and iii) look-up table like performance benchmarking technique in FB-Net NAS paper to profile the hardware latency.  The strength of this paper is obvious, it does not try to complicate the current literature by introducing some articulated complex search strategy but try to formulate a novel problem, that minimizing the multi-task multi-hardware and multi-search space in one step. The experiments demonstrate the effectiveness of this method. 

On the other hand, it is obvious that this paper does not have much technical novelty. The entire methods can be separated into independent procedures that aim to solve a specific problem. Compared to the relevant baseline Once-For-All, which brings the a novel solution for multiple device and constraints neural architecture search by leveraging the super-net and proposing those tricks like progressive shrinking and depth pruning, this work seems to incoporate several existing approach to tackle a similar problem. In short, the engineering contribution of this paper is significantly larger than the algorithmic one.  I am not sure if this paper introduces any new message to the current community. 


 As this paper is clearly written, the only question I can raise now is if we should accept a paper with many engineering efforts. In my humble opinion, this paper seems like a moderate modification of OFA, replacing the latency predictor by a straight-forward baseline, layer-wise profiling that is introduced in FBNet long time ago. Then adapts another simple baseline from meta-learning domain to minimize the O(T) into O(1). 

Post rebuttal comment:

This paper did find an interesting question in meta-learning, and I agree that this is critical and should be deemed as a solid contribution to the current research community. I raised my score to weak accept.  I do not see direct negative societal impact of this work so far. 
As for the limitation of the method itself, to use the latency profiler, it requires manual benchmarking the latency on multiple devices and might be hard for researchers that has insufficient resources. 

 This paper introduces a method for many-task many-hardware NAS. The approach combines existing hardware-aware NAS with MAML. A layer-wise profiling strategy is introduced to improve profiling time, and modifications to the NAS algorithm to prevent overfitting in the few-shot setting is also proposed. The method achieves competitive experimental results on omniglot and mini-imagenet whilst being very efficient. **Strengths:** 
The paper is clear and well written. The proposed layer-wise profiling is a simple solution that greatly outperforms latency predictor baseline in this setting. The proposed Global Expansion and Adaptive Number of Layers are also simple changes that I like. The experimental results are very strong.

**Weaknesses:** 
Layer-wise profiling is ultimately a heuristic and could have larger errors for strange computation graphs where the total runtime is not equal to the sum of the runtimes of each layer. That being said, experimentally this approach has a lower MSE than baseline label predictor so it's a great step up. 

The need to manually add a final pooling layer feels strange, as this is a NAS method one would have hoped that this could have been discovered automatically.
 Have the authors experimented with discovering how much pooling to add in the end rather than manually fixing it? The authors have addressed the limitations.

 The author proposes a very interesting new method, that is, the adaptive adaptation problem of multi-devices in the domain of NAS. The idea of meta-learning is introduced into the NAS framework to search the underlying optimal neural network model. The contributions of this paper mainly focus on two aspects: one is the generality of the method, the network architecture obtained by the method in this paper can be adapted to multiple devices at the same time, without the need to search the optimal structure for each device separately; the second is the superiority of the proposed method, the search time is shorter than those comparison methods. Strengths:
This paper provides a solution for complex task, which is novel and has important meaning in the field of NAS research.

Weaknesses:
1 The layout of Method section seems unreasonable. First, the Super-net meta-training strategy subsection introduces the training optimization strategy for architecture search, while the following two strategies (Layer-wise profiling and Adaption strategy) are some skills for network structure search. The order of the two seems to be reversed. Second, Tables 1 and 2 present partial results, and it seems more appropriate to move them to the Evaluation section.
2 The Method section lacks the focus of the core issue: when there are multiple different hardware devices, the author does not give the detailed description of how to set the hardware constraints. 
 but their final classification accuracy is not the same. If the results on three devices are different, then the goal of this paper seems unachieved. Please refer to sentence in the Introduction section: “Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this method offers a chance to reduce the hardware search complexity from O(HC) to O(1)”. How to explain this sentence?  It seems that the author has only solved some of the existing problems, and has not provided the relevant results for the optimization goal of multiple devices. It is necessary to supplement the relevant experimental results in the revised version.

Meta-review: Reviewers agree that the proposed many-task many-hardware few-shot learning setup is well motivated and valuable in real-world applications. The main novelty of this paper lies in the meta-learning side, and their results reveal that metric-based MAML methods are not latency-friendly, and the community should look back into optimization-based MAML methods. Authors have devoted extensive engineering efforts into this paper, as recognized by several reviewers.

Previously raised concerns are mainly about the evaluation of overall novelty (e.g., similarity with OFA), possibly inaccurate layer-wise profiling due to layer/operation fusion, and missing comparison with more recent methods. Most of them have been well resolved in the author feedback; therefore, AC recommends acceptance.

Decision: Accept