Rapid Model Architecture Adaption for Meta-Learning
Authors: Yiren Zhao
Xitong Gao
Ilia Shumailov
Nicolo Fusi
Robert D. Mullins
Venue: NeurIPS 2022 Accept (2022)
Keywords: NAS

Abstract: Network Architecture Search (NAS) methods have recently gathered much attention. They design networks with better performance and use a much shorter search time compared to traditional manual tuning. Despite their efficiency in model deployments, most NAS algorithms target a single task on a fixed hardware system. However, real-life few-shot learning environments often cover a great number of tasks ($T$) and deployments on a wide variety of hardware platforms ($H$). 	

The combinatorial search complexity $T \times H$ creates a fundamental search efficiency challenge if one naively applies existing NAS methods to these scenarios. To overcome this issue, we show, for the first time, how to rapidly adapt model architectures to new tasks in a \emph{many-task many-hardware} few-shot learning setup by integrating Model Agnostic Meta Learning (MAML) into the NAS flow. The proposed NAS method (H-Meta-NAS) is hardware-aware and performs optimisation in the MAML framework. MetaNAS shows a Pareto dominance compared to a variety of NAS and manual baselines in popular few-shot learning benchmarks with various hardware platforms and constraints. In particular, on the 5-way 1-shot Mini-ImageNet classification task,  the proposed method outperforms the best manual baseline by a large margin ($5.21\%$ in accuracy) using $60\%$ less computation.

Rapid Model Architecture A Adaption for Meta-Learning
Yiren Zhao" Imperial College London a. www.@imperiallccmmmmm ac. uk
Xitong Gao"
Shenzhen Institute of Advanced Technology, CAS xt st.gao@siat..n an
Ilia Shumailov
Nicolo Fusi
University of Oxford
Microsoft Research fus i @mi crosoft. com
ilia, shuaiiov@chch ox ac.uk uk
Robert D Mullins University of Cambridge
robert mullins@cl...mmmmmmmmmmmmmmmmm cam. ac uk
Abstract
Network Architecture Search (NAS) methods have recently gathered much atten- tion. They design networks with better performance and use much shorter search time compared to traditional manual tuning. Despite their efficiency in model deployments, most NAS algorithms target single task fixed hardware system. However, rial-life few-shot learning environments often cover great number tasks (I) and deployments
a wide variety of hardware platforms (H).
The combinatorial search complexity T H. creates fundamental search effi- ciency challenge if one naively applies existing NAS methods to these scenarios. To overcome this issue, we show, for the first time, how to rapidly adapt model ar- chitectures to new tasks in a many-task many-hardwen few-shot learning setup by integrating Model Agnostic Meta Learning (MAML) into the NAS flow. The pro- posed NAS method in the MAMI. framework. H-Meta-NAS. shows Pareto dominance compared to a variety of NAS and manual baselines in popular few-shot learning benchmarks with various hardware platforms and constraints. In particular, on the 5-way 1-shot Mini- ImageNet classification task, the proposed method outperforms the best manual baseline by a large margin (5.21% in accuracy) using 60% less computation.
is hardware-awart and performs optimisation
1 Introduction
Existing Network Architecture Search (NAS) methods show promising performance on image Zophi
and Le 2016 Liu et al.] 2018). language (Guo et al. 2019 So et al. 2019 and graph data iZao
ct al.] 2020bl. The automation not only reduces the human effort required for architecture tuning but
also produces architectures with tate-of-thhe-rt performance in domains like image classification
Zoph and Le 2016) and language modeling [So et al., 2019 Most NAS methods today focus on a single task with fixed hardware system, yet real-life model deployments covering multiple tasks and various hardware platforms will significantly prolong this process. As illustrated in Figure [ common design flow is re-engineer the architecture and train for the different task( (hardTTr-arwwaret/// pairs with different constraints (C). The architectural engineering phase can be accomplished whether
*Correspondence Yiren Zhao and Xitong Gao.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Architectural Engineering Train from scratch or Re-train
Task 1
Hordware
Constraint
Task
Hardwara
Constraint
Figure 1: Deploying networks in a many-task many-device few-shot learning setup. This implies a large search complexity O(THC).
manually by using established NAS procedure. The challenge is designing efficient method to overcome the quickly scaling (TTIC) search complexity described in Figure |
Few-shot learning systems follow exactly this many-lask many-device setup, when considering deployments on different user devices on key applications such as facial [Guo al. 2020) and speech recognition [Hsu al. 2020]. A task in few-shot learning normally takes N-way K K-shot formulation, where it contains N classes with K support samples and Q query samples in each class. Model-Agnostic Meta-Learning (MAML), incorporating the idea of learning learn, builds a meta-model using great number of training tasks, and then adapts the meta-model to unseen test tasks using only very small number of gradient updates (Finn et al. 2017 MAML then becomes a powerful and elegant approach for few-shot learning its ability to quickly adapt to new tasks can potentially shrink the O(THC) complexity illustrated in Figure I to O(HC). In the meantime, hardware-aware NAS methods (Cai et al., [019 2018 Xu et al. 2020], e.g. the train-once-for-all technique (Cai al. 2019], support deployments of searched models to fit different hardware platforms with various latency constraints. These hardware-aware NAS techniques further
reduce the search complexity from O(THC) to (T) CCa et al. 2018).
In this paper, we propose novel Hardware-an aware Meta Network Architecture Search (H-Meta-NAS) Integration of the MAML framework into hardware-aware NAS theoretically reduces the search complexity from O(THC) to O(1), allowing for rapid adaption of model architectures to unseen tasks new hardware systems. However, identified the following challenges in this integration:
Classic NAS search space contains many over pereparameterized sub-models, this makes it hard to tackle the over-fitting phenomenon in few-shot learning.
Hardware-awaae NAS profiles latency for sub-networks on each task-hardware pair, this profiling can be prolonged significantly with a great number of tasks and, more importantly, if the targeting device has scarce computation resources.
To tackle these challenges, then propose to Global Expansion (GE) and Adaptive Number of Layers (ANL) to allow drastic change in model capabilities for tasks with varying difficulties. Our experiments later demonstrate that such changes alleviate over-fitting in few-shot learning and improve the accuracy significantly. We also present novel layer-wise profiling strategy to allow the reuse of profiling information across different tasks. We make the following contributions:
We propose a novel Hardware-aware Network Architecture Search for Meta learning (H- Meta-NAS). H-Meta-NAS quickly adapts meta-architecture to new tasks with hardware- awareness and can be conditioned with various device-specific latency constraints. The proposed NAS reduces search complexity from O(THC) to O in realistic many-task many-device few-shot learning setup. We extensively evaluate H-Meta-NAS on various hardware platforms (GPU, CPU, mCPU, IoT, ASIC accelerator) and constraints (latency and model size), our latency- performance generally outperforms various MAMI. baselines and other NAS competitors.
We propose a task-agnostic layer-wise profiling strategy that reduces the profiling run- time from around 10 hours to 1.2 hours when targeting hardware with limited capabilities (e.g. IoT devices).

---

MAML models are prone to over-fitting in the few-shot learning setup Antoniou et al. 2018 We show several design options for the NAS algorithm, namely Global Expansion and Adaptively Number of Layers respectively. These methods help the NAS to overcome the over-fitting problem by optimizing the architectural design space.
We reveal that popular Metric-based MAMI methods not latency-friendly Our empirical results suggest that Optimization-banl baseed MAML method with well-tuned architec- tures achieve comparable accuracy with significantly less latency overhead (+= x on Mini-Imagell 5-way 1-shot classification).
2 Background
2.1 Few-shot learning in the MAMI framework
Inspired by human's 's ability to learn from only a few tasks and generalize the knowledge to unseen problems, meta learner is trained over distribution of tasks with the hope of generalizing its learned knowledge to new tasks Finn et al. [017]]
arg min (BTer(Co(TT))
Equation captures the optimization objective of meta-learning, where optimal parameters are obtained through optimizing on set of meta-training tasks. Current approaches of using meta- learning to solve few-shot learning problems can be roughly categorized into three types: Memory- based, Metric-based and Optimization-base
Memory-based method utilizes meeoor-aaggmenter neural network [Munkhdalai and Yu 2017
Gidaris and Komodakis 2018 to memorize meta-knowledge for fast adaption to new tasks, Metric- based methods aim meta-learn a high-dimensional feature representation of samples, and then apply certain metrics to distinguish them. For instance, Meta-Baseline utilizes the cosine nearest-centrood metric [Chen et al. 2020] and DeepEMD applies the Wasserstein distance (Zhang et al., 2020]. Optimization- based method, on the other hand, focuses on learning good parameter initialization (also known as meta- metaa-aaraesees or meig-weights) from great number of training tasks, such that these meta- perampprsseeerss adapt to new few-shot tasks within a few gradient updates. The most well- established Optimization-baste based method is Agnel-Agnostic Meta-Learning (MAML.) (Finn et al.. 2017]. MAML is a powerful yet simple method to tackle the few-shot learning problem, since its adaption relies solely on gradient updates. Antoniou et al. later demonstrate MAML++, a series of modifications that improved MAML's performance and stability. Baikk et al. introduce an additional network for generating adaptive parameters for the inner-loop optimization.
Despite the rise in popularity of the meta-learning framework applied to few-shot learning, little attention has been paid the runtime efficiency of these approaches. In this work, we reveal that the current mainstream Mletic-based methods all suffer from severe latency overhead, since Metric- based approaches rely on metric comparisons and have to use multiple inference runs to generate these metrics (at least two) for single image classification. With the help of H-Meta-NAS, we show how Optimization-ban based methods can achieve a similar level of accuracy but using significantly less computation at inference time.
2.2 Network architecture search
Architecture engineering tedious and complex process requiring lot of effort from human experts. Network Architecture Search (NAS) focuses on reducing the amount of manual tuning in this design space. Early NAS methods use evolutionary algorithms and reinforcement learning to traverse the search space [oph and Le 2016 Real et al. 2017). These early methods require scoring architectures trained to a certain convergence and thus use a huge number of GPU hours. Two major directions of NAS methods, Gradient- based and Evolution-based methods, are then explored in parallel in order to make the search cost more affordable. Gradient-bard based NAS methods use Stochastic Gradient Descent (SGD) to optimize set of probabilistic priors that are associated with architectural choices [Liu et al. 2018 Casale et al. 2019 Although these probabilistic priors can be made latency-awar [Wu et al.) 2019 Xu et al. 2020 it is challenging to make them follow hard latency constraint. Evolution super-net and use evolutionary algorithms reinforcement learning to pick best-suited sub-networks
NAS, on the other hand, operates on top of pre-trained
3

---

making it easier to be constrained by certain hardware metrics. For instance, Once-for-all (OFA) an Evolution-based NAS method and its searched networks are not only optimized for a specific hardware target but also constrained by pre-defined latency budget [Cai ar. 2019]. Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this method offers chance to reduce the hardware search complexity from (HC) to (1)) Latency is key metric in mobile applications, and obviously can be also improved through other techniques such
Ca et al. 2018 2019]
as quantization (Zhao et al.) 2019a Hönig et al. 2022] and sparsification [Gao et al. 2018 Wang
ct al., 2019] in various learning setups, however, many of these optimizations might be incffectual
Nikollić et al. 2019] unless the underlying hardware has special support for it (Su et al. 2018 Zhao et al. 2019b Parashar et al. 2017]. NAS algorithm is promising approach to address this problem, since it can directly optimize the network topology; recent NAS methods have also been extended to
Transformers (So al. 2019] and Graph Neural Networks (Zao et al. 2020b a].
Several NAS methods are proposed under the MAMI framework [Kim et al. 2018 Shaw et al. 2018
Lian et al. 2019]. these methods successfully reduce the search complexity from ((T) to O(1).
However, some of these methods do not show significant performance improvements compared
to carefully designed MAML methods (e.g. MAML++) [Kim et al., 2018 Shaw et al. 2018]. In the meantime, some of these MAML-based NAS methods follow the Gradient-basee approach and operate on complicated cell-based structures [Lian et al. [019]. We illustrate later how cell-baeed NAS causes an undesirable effect on latency, and also meets fundamental scalability challenges when trying to deploy in a many-task many-device few-shot learning setup.
3 Method
Problem formulation In the MAML setup, we consider a set of tasks T and each task Ti E T contains a support set Dj and query set Dj. The support set is used for task-level learning while the query set is in charge of evaluating the model. All tasks are divided into three sets, namely meta-training (Terain), valaa-vali (Tval) and meta-testing (T test) sets.
Equation formally states the objective of the pre-training stage illustrated in Figure 3a The
objective of this process is optimize the parameters @ of the super-net for various sub-networks sampled from the architecture set A. This will ensure the proposed H-Meta-NAS to have both the peta-parameters and meta-architecttres ready for the adaption to new tasks.
arg min Ea))(() [TeI.r [Co a]]]
Equation describes how MMMta-NAS adapts network architectures to a particular task T with a given hardware constraint Cn. In practice, using the support set data Dj from a target task T, we apply genetic algorithm for finding the optimal architectures a". We discuss further how this process in detail in later sections.
a" = aim Co((D, act
(3)
s.t. C(a) S Ch
Architecture search space H-Meta-NAS considers a search space composed of different kernel sizes, the number of channels, and activation types. We mostly consider VGG9-based NAS backbone, that is a 5-layer CNN model with the last layer being a fully connected layer. We chose this NAS backbone because both MAML Finn al. [2017) and MAML++ Antoniou al. 2018 used a VGG9 model architecture. The details of this backbone are in Appendix A
We allow kernel sizes to be picked from {1. 3, 5), channels to be expanded with set of scaling factors (0.25, 0.5, 0.75, 1, 1.5, 2, 2.25] and also six different activation functions (details in Appendix). For a single layer, there are 7 6 126 search options. H-Meta-NAS also contains an Adaptive Number of Layers strategy, the network is allowed to use a subset of the total layers in the supernet with maximum usage of 4 layers. The whole VGG9-based backbone then gives in total 126* x 4 z 10 possible neural network architectures.
In addition, to demonstrate the ability of H-Meta-NAS on more complex NAS backbone. We also studied an alternative ResNet1 1 22-baeed NAS backbone, that has approximately 2 x 1021 possible sub-networks.

---

Lakency Profile
Aophing Lubeny Profler
Lapar
Pictiie at layeewbe Nrrhllecuun continstoons
Lakeo Prolke
information nech Jo givnn hordware
AUTT
2ire
Figure 2: An illustration of how Latency Proflerr works. The left side shows how the profiler is built by recording the run-time of all configurations of each layer. The right side shows this profier then can sum the layer-wise information to provide hardware-aware runtime for whole network.
que
Averaged accuracy
Ateel beiing the. mookkeeeee
Mads kalli padio ooid
0.6
0.4
50
0.3
100
0.2
500
50
1000
150
200
250
300
Number of Iterations (M)
(a) Super-net meta-training.
(b) The effect of different pool size and different number iterations (M).
Figure 3: Figure 3a is overview of the super-net meta-training. Figure 3b shows how different
parameters the Adaption strategy can affect accuracy.
Layer-wise profiling Hardware-aware NAS needs the run-time of sub-networks the targeting hardware to guide the search process Cai et al. 2019 Xu et al. [2020). However, the profiling stage can be time-consuming if given low-end hardware as the profiling target and the search space large. For instance, running a single network inference of VGG9 on the Raspberry Pi Zero with 1GHz single-core ARMv6 CPU takes around 2.365 seconds to finish. If we assume this is the average time needed for profiling a sub-network, given that the entire search space includes around 10 sub-networks, a naive traverse will take formidable amount of time which approximately 6 x 10 hours. More importantly, the amount of profiling time scales with the number of hardware devices (O(H)). Existing hardware-ane aware NAS schemes build predictive methods estimate the run-time of sub-networks Cai et al. 2019]. Xu et al. (2020) and have relatively significant error. An illustration of profiler shown Figure and latency predictor's illustration in Figure S
Adaption strategy The adaption strategy uses genetic algorithm Whitley [994] pick the best-suited sub-network with respective to a given hardware constraint, the full a algorithm is detailed in Appendix D In general, the adaption algorithm randomly samples set of tasks from Teat- and uses the averaged loss value and satisfaction to the hardware constraints as indicators for the genetic algorithm. The genetic algorithm has pool size P and number of iterations M.
then run tyyer-pprraeter analysis in Figure 3b to determine the values of and M. The full adaption algorithm makes use of these hyper-parameters is in our Appendix. The horizontal axis shows the number of iterations and the vertical axis shows averaged accuracy on the sampled tasks for all architectures in the pool. Figure 3b shows that the accuracy convergence is reached after
5

---

Existing NAS methods fixed number of charnels layers 'edge' each searchable
GE allows theee edge layere to shrinklexpend
output
H
ANL allows the network to use fewer layers Figure 4: A graphical illustration of GE and ANL. Both methods will allow more drastic change in model capabilities, allowing the searched model to deal with tasks with varying difficulties.
around 150 iterations, and running for additional iterations only provides marginal accuracy gains. For this reason, picked the number of iterations to be 200 for balance between accuracy and run-time. In the meantime, we notice in general a higher pool size will give better- petter-adapted accuracy. However, this does mean the final searched accuracy is affected the same degree. The final re-trained accuracy of searched architectures show an accuracy gap of 0.21% % between P = 100 and P 200 and 0.32% between P 100 and P 500. An increase in pool size can prolong the run-time significantly, we thus picked a pool size of 100 since it offers the best balance between accuracy and run-time.
NAS backbone design, Global Expansion (GE) and Adaptive Number of Layers (ANL) One particular problem in few-shot learning is that models are prone to over- fver-fitting. This is because only small number of training samples are available for each task and the network normally iterate these samples many times Antoniou et al. 2018). We would like to explore the architectural space to help models overcome over-fitting and conduct a case study for different design options available for the backbone network. We identify the following key changes to the NAS backbone to help the models to have high accuracy in few-shot learning:
X pooling: Pooling that applied the final convolutional operation, X indicates the height and width of feature maps after pooling.
Global Expansion (GE): Allowing the NAS to globally expand or shrink the number of channels of all layers.
Adaptive Number of Layers (ANL): Allowing the NAS to arbitrary number of layers, the network then able to early stop using only a fewer number of layers.
Figure further illustrates how GE and ANL can allow a much smaller model compared to existing NAS backbones. We also discuss in Section how the pooling strategy can join the NAS process.
Super-net meta meta-training strategy As illustrated by prior work Cai et al. 2019 progressively shrinking the super-net during meta-training can reduce the interference between sub-networks. We observe the same phenomenon and then use similar progressive shrinking strategy in H-Meta-NAS, the sampling process p(A) will pick the largest network with a probability of p. and randomly pick other sub-networks with probability of - p. We apply an exponential decay strategy to p:
c e
p= Pe (p: Pe) erp(-a
cm E,
Pe and p; are the end and initial probabilities. e is the current number of epochs, and e, and em are the starting and end epochs of applying this decaying process. determines how fast the decay is. graphical illustration is shown in Figure Sa In our experiment, we pick p; = 1.0 and e, 30, because the super-net reaches a relatively stable training accuracy at that point.
We then start the decaying process, and the value a = 5 is determined through hyper-parameter study shown in our Appendix B

---

Table 1: Comparing latency predictor with our proposed profiling, MSE Error is the error between estimated and measured latency, Time is the total time taken to collect and build the estimator.
Hardware
Metric
Latency Prodictor
Layer-wise Profiling
2080 Ti GPU
MSE Error
0.0188
0.00690
Time
16.09 mins
6.216 sees
MSE Error
0.165
0.0119
Intel 19 CPU
Time
21.92 mins
16.41 sees
MSE Error
N/A
0.00742
Pi Zero
Time
N/A (Approx. 220 hours)
82.41 mins
Evaluation
We evaluate H-Meta-NAS on a range of popular few-shot learning benchmarks. For each dataset, we search for the meta-architecture and meta-pan meta-parameters. We then adapt the meta-architecture with respect to a target connttee-conttaaint pair. In the evaluation stage, we then re-train the obtained hardware-awe aware task-specific architecture to convergence and report the final accuracy. We consider three popular datasets in the few-shot learning community: Omniglot, Mini- Mini-ImageNet and Few-shot CIFAR 100. We use the PytorchMeta framework to handle the datasets (Deleu et al.
2019].
Omniglot a handwritten digits recognition task, containing 1623 samples [ Lake et al. 2015 We use the meta train/validationn/est splits originally used Vinyals et al. Vinyals et al. 2016]. These
splits are over 1028/172/423 classes (characters).
Mini-ImageNet is first introduced by Vinyals et al. This dataset contains images of 100 different classes from the ILSVRC-12 dataset (Deng et al. 2009], the splits are taken from Ravi et al. Ravi and Larochelle 2016
Table 10 in our appendix details the systems and representative devices considered. We use the ScaleSIM cycle-accurate simulator et al. 2018) for the Eyeriss Chen et al. (0166) accelerator. This simulation and more datasets and search configurations information are in Appendix
Evaluating layer-wise profiling We re- re-implemented the latency predictor in OFA Cai et al. 2019 as a baseline and compare it to our layer-wise profiling. We pick 16K training samples and 10K validation samples to train and test the latency predictor, which is the same setup used in OFA. We use another 10K testing samples to evaluate the performance of OFA-based latency predictor against our layer-wise profiling on different hardware systems terms of MSE (measuring the latency estimation quality) and Time (measuring the efficiency). As illustrated in Table [ layer-wise profiling saves not only time but also has smaller MSE егтоr compared a predictor-based strategy that is very popular in today's 's evolutionary -ased NAS frameworks (Cai et al.) 2019 2018). In addition, layer-wise profiling shows orders of magnitude better run-time when targeting hardware devices with scarce computational resources. If we consider an IoT class device as target (i.e the Raspberry Pi Zero), it requires an unreasonably large amount of time generate training samples for latency predictors, making them an infeasible approach in real life. For instance, the total time consumed by the latency predictor is infeasible to execute on Pi Zero (last row in Table [ Of course, reality, there is also a great number of IoT devices using more low-end CPUs compared to Pi Zero (ARMV5 or ARMV4), making the latency predictor even harder to be deployed on these devices. Also in a many- hary-hardwaar setup considered in this paper, this profiling is executed O(H) times.
Most existing layer-wise profile/look-un up approaches consider at most mobile systems as targeting platforms [Xu et al. 2020 Yang et al. 2018). These systems are in general more capable than a great range of IoT devices. In this paper, we demonstrate the effectiveness of this approach on more low-end systems (Raspberry Pi and Pi Zeros), illustrating this is the more scalable approach for hardware-aware NAS operating on constrained hardware systems.
Evaluating GE and ANL Our results in Table suggest that correct pooling strategy, GE and ANL can change the NAS backbone to allow the search space to reach much smaller models and thus provide better accuracy. In addition, Tablegalso illustrates that x 5 pooling necessary for higher accuracy. We hypothesize this is because a relatively large fully-connected layer after the
7

---

Table A case study of different design options for the NAS backbone network. Experiments are
executed with model size constraint of 70K on the Mini- Mini-ImageNet 5-way 1-shot classification task.
Design options
Accuracy
MAML
48. 70%
MAML+
52.15%
H-Meta-NNAS + x Pool
42.28%
H-MMetaNNAS + 5 x Pool
46.13%
H-Meta-NAS + x Pool + GE
53.09%
H-Meta-NAS + 5 x 5 Pool + GE + ANL
56.35%
pooling is required for the network to achieve good accuracy in this few-learning setup. We then demonstrate using case study in our evaluation of how combination of these techniques can help H-Meta-NAS: the final searched model can have an up to 14. 14.28% accuracy increase on the 5-way 1-shot Mini-ImageNet classification if using these optimization tricks. Table[aals showed us that a well-tuned architecture can help MAML models overcome the over-fitting phenomenon in the few-shot learning setup.
Table 3: Results of Omniglot 20-way few-shot classification. We keep two decimal places for our experiments, and keep the decimal places as was reported for other cited work. reports a MAMI. replication implemented by Antoniou et al.
Accuracy
Method
Size
MACs
1-shot
5-shot
Stamese Net: Kech et al
2015
35.96.M
1.36C
99.2%
97.0%
Matching
Vievals
20.29M
93.8%
98.3%
Meta-SGU
er Lree
2017
119.86K
16.21M
95 93% = +00+00.000 0.38%
98. 97% # O. 1.1%
Meta-NAS Elsken
2020
100.00
96. 06.20%
0.16%
99.20%
0.07%
MAMI Finn et al
2017
K13.21K
10.07M
95.8% 0.3%
98.9% + 0.2%
(Replication Troen Anlondu
2018
10.07M
98.78% 99.35 99.33% = O 03%
MAML++ Antoniou et 2018
113.211
10.07AM
97 65%
1 0.05%
MAML+ (Lozal Reptentioo)
113:21K
10.07M
96 60%
0.28%
0.07%
H-Meta-NAS
110.73K
4 4.55M
97.61 0.00%
99.1 10.11%% + 0.09%
Table Results Mini-Imagen ImtgeNet 5-way classification. We use two decimal places for our experi- ments, and keep the decimal places of cited work as they were originally reported. T-NAS uses the complicated DARTS cell Lian al. 20199. has a smaller size but a large MACs usage.
Accuracy
I-shot
5-shot
Matching Nets Vinvals
228 23A
200.31M
13.41 0.77%
35.31 73%
CompareNets Sung df 2018
337.95K
318.38.
50.44 + 0.82%
65.32 + 0. 0.70%
MAML
57.38M
48.70
1.84%
63.11 0.92%
MAML++ Aniomoo el
2018
70.09K
57.38M
52.15 1 0.26%
68.32 1 0.44%
L2T
70.09K
OFA Cai et al, 2019 (1 (Lacal Replication
MAMI+
82.20K
33.1
51.32 + 0.07%
68.22 + 0.12%
Auto-Meta Kim et 2018
98.70K
61.16
08.18 0.14%
RASE: (Softmax)
Staww ef ai 2018
1200
65.4 + 0.7%
(Gumbell)
0.7%
Meta-NAS
Eisken et s1
2020
100K
1..2:.0..4%
67.8 = 0. 7%
T-NAS
24.3/26 223///5KKKK
37 97.26/ 63M
52.81 1.11%
.29%%
T-NAS++
Lran et
2019
24.3 226.5
37. 37.96/33 62..633
54.11 + 1.35%
69. 09 = 0.85%
H-Meta.NAS
70.28
57.36
1.11%
Evaluating H-Meta-NAS searched architectures Table displays the results of H-Meta-NAS on the Omniglot 20- way 1 1-shot and 5-shot classification tasks. We match the size of H-Meta-NAS to MAML and MAML+ for fair comparison. H-Meta-NAS outperforms all competing methods apart from the original MAML++. MAML+ uses a special evaluation strategy. it creates an ensemble of models with best validation-set performance. MAML++ then picks the best model from the ensemble based on support set loss and report accuracy on the query set. We then locally replicated MAML+ without this trick, and show that H-Meta-NAS outperforms by a significant margin (+ (+1.01% on s-shot and -0.11% on 5-shot) with around half of the MACs (4.95M compared to 10.07M..
20-way

---

40001025
AccurocS
Inference Lacency on
64
24 24 10
Inference Latoncy Eyores imst
Inference Latency Raspberry Ims)
Vilg
Figure 5: Target a GPU
Figure 6: Target an ASIC
Figure 7: Target an IoT device
Table shows the results of running the 5-way I-shot and 5-shot Mini-ImageNet tasks, simi- lar to the previous results, we match the size of searched networks to MAML, MAML+ and ALFA+MAML++22F Table 3 not only displays results on MAML methods with seed-architecctress it also shows the performance of searched networks including Auto-Meta Kim et al. [2018), BASE Shaw et al. 2018 and T-NAS Lian et al.
[019]] H-Meta-NAS shows interesting results when compared to T-NAS and T-NAS++. 1--MetaNNAS has much higher accuracy (+3.26%% in 1-shot and 7.94% 5-shot) and a smaller MAC count, but uses a greater amount parameters. T-NAS and T-NAS+++ use DARTS cells Liu et al. (2018). This NAS cell contains a complex routing of computational blocks, making it not suitable for latency critical applications. We will demonstrate later how this design choice gives worse on-device latency performance.
H-Meta-NAS for diverse hardware platforms and constraints In addition to using the model sizes as a constraint for H-Meta-NAR we use various latency targets on various hardware platforms as the optimization target. Figure [5] Figure 5 Figure 7 show how GPU, ASIC and IoT device latency can be used as constraints. The smaller model sizes of T-NAS do not provide a better run-time on GPU devices (Figure 5), fact, T-NAS based models have the worst run-time on GPU devices due to the complicated dependency of DARTS cells. We only compare to MAML and MAML++ when running on Eyeriss due to the limitations of the ScaleSIM simulator Samajdar et al. 2018 In Appendix D we provide more latency vs. accuracy plots using various hardware platforms' latency as constraints and observe the same pareto dominance.
Table 5: Applying H-Meta-NAS to different NAS backbones/algorithms for the Mini-ImageNNe 5-way 1-shot classification.
Method
Network Backhone
Inference Style
Accuracy
MAML
al.
2017
VGG-based
Single Pass
70.09K
57.38M
48.70 +: 1.84%
MAML++
Anfornou et al
2018
VGG-based
Single Pass
70.0KK
57.38M
52.15 1: 0.26%
Meta- Baseing liva
ResNet-based
Multi Pass
12.11M
56.48G
63.17 = 0.23%
DecpEMD Ziang ot A
A120
ResNet-based
Multi Pass
12.44M
56.38G
05.91
+ 0.82%
MetaNAS Eisken
A120
DAKTS-ccll-bsedd
Single Pass
1M
61.8 0.1%
H Meta NAS
VGG based
VGG-based
Single Pass
70.28
24.094f
57.36
1.11%
HA-Meta-NAS
ResNet-bused
Single Pass
70662
28.19M
64.67
2.03%
Table 6: Comparing the NAS search complexity with N tasks, И hardware platforms and C constraints. Search time is estimated for deployment scenario with 500 tasks and 10 hardware- constraint pairs, estimation details are discussed in Appendix.
Method
Style
Hardwase-wwae
Scarch complexity
Scarch time (GIU hrs)
DARIS
Gradient- single task
OVTCC)
Once for La dl 2019 TNAS & TNAS- Lun di 2019
Evolution based, single task
Yes
((N)
Gradient based. multi uak
No
:: 10'
M-MeIEANAS
Evalution-basod, multi task
Yes
O13)
40
A more complex NAS backbone and a comparison to Metrie-based MAML Table 5 shows how H-Meta-NAS performs with more complicated NAS backbone. In previous experiments, build the NAS on top of a VGG9 backbone since it is the architecture utilized in the MAML++ algorithm. To have a fair comparison, in this section, we additionally show that H-Meta-NAS can be applied with a more complicated backbone and it shows better final accuracy expected.
did not manually pick complex NAS backbone. However,
We compare the proposed approach with state-of-the-art Metric-based meta-learning methods Zhang al. 2020 Chen et al. 20200. There is current trend that Metric-basee MAML approaches are

---

becoming the mainstream in few-shot learning. Chen et al. shows that simple cosine-similane measurement can be used as a metric and outperforms standard Optimization-biin based approaches by significant margin. In this experiment, we demonstrate that Metric-based methods suffer from a significant runtime overhead. Optimization methods, on the other hand, have significantly less MACs usage. Although using only single inference pass (our method does not conduct inference runs on the support set when deployed), H--Meta-NAS shows competitive results with SOTA Metric-based methods while having much smaller MACs usage (around 2000x), ), showing that well-tuned network architecture can help Optimization methods to close the accuracy gap. We hope this finding will encourage researchers in this field to look back into Optimization-basee MAML. Search complexity and search time In Table 6 show comparison between H-Meta-NAS and various NAS schemes in the many-task many-device setup. Specifically, we consider a scenario with 500 tasks and 10 different hardware- consttriit hansware-constraint pairs. Our results in Table 6 suggest that H-Meta-NAS is the most efficient search method because of its low search complexity.
Conclusion
In this paper, show H-Meta-NAS, NAS method focusing fast adaption of not only model weights but also model architectures in a many-task many-device few-shot learning setup. H-Meta- NAS outperforms wide range of MAML baselines effectiveness of H-Meta-NAN on different hardware systems and constraints, and demonstrate its superior performance real devices using orders of magnitude shorter search time compared to existing NAS methods.
set of few-shot learning tasks. We study the
Acknowledgements
Xitong Gao is supported in part by Shenzhen Science and Technology Innovation Commission (No. JCY1201908121100033
References
A. Antoniou, H. Edwards, and A. Storkey. How to train your MAML. arXiv preprint arXiv:1810.09502, 2018
S. Baik, M. Choi, J. Choi, H. Kim, and K. M. Lee. Meta-learning with adaptive hyperpa- rameters. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20755-20765. Cur- ran Associates, Inc., 2020. URL Paper pdf
H. Cai, L. Zhu, and S. Han. ProxylessNAS: Direct neural architecture search target task and hardware. arXiv preprint 1rXiv:1812.000 2018.
H. Cai, C. Gan, Т. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:190S, 09791, 2019.
P. Casale, Gordon, and N. Fusi. Probabilistic neural architecture search. arXiv preprint MXKiv:9902.01111 2019.
T. Chen, T. Moreau, Z. Jiang, I.. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, et al. TVM: An automated End-to-End optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578-594, 2018.
Y. Chen, X. Wang, Z. Liu, Xu, and T. Darrell. new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.00 04399, 2020.
Y.-H. Chen, T. Krishma, J. S. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE journal of solid-state circuits, 52(1):127-133, 2016.
T. Deleu, T. Würfl, M. Samiei, J. P. Cohen, and Y. Bengio. Torchmeta: A Meta-Learning library for PyTorch, 2019. URL. https:///axxv... org/ 11.9.99996677 https://gtthbbbooommttstudddooor
Available at:
10

---

J. Deng. W. Dong, R. Socher, L., Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. IEEE, 2009.
T. Elsken, B. Staffler, J. H. Metzen, and F. Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12365-12375, 2020.
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
X. Gao, Y. Zhao, L. Dudziak, R. Mullins, and C.-z. Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.
S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4367-4375, 2018.
J. Guo, X. Zhu, C. Zhao, D. Cao, Z. Lei, and S. Z. Li. Learning meta face recogmition in unseen domains. CoRR, 0bs/2200.07733. 2020. URL https: :ettpa//aarte /tapaav//////v////aa www.givv..rrraeee
Y. Guo, Y. Zheng, M. Tan, Q. Chen, J. Chen, P. Zhao, and J. Huang. NAT: Neural architecture transformer for accurate and compact architectures. arXiv preprint arXiv:1900. 14488, 2019.
R. Hönig, Y. Zhao, and R. Mullins. DAdaQuant: Doubly-adaptive quantization for communication- efficient federated learning. In International Conference on Machine Learning, pages 8852-8866. PMLR, 2022.
J.-Y. Hsu, Y.-J. Chen, and H.-y. Lee. Meta learning for end-to-end low-resource speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 7844-7848. IEEE, 2020.
J. Kim, S. Lee, S. Kim, M. Cha, J. K. Lee, Y. Choi, Y. Choi, D.-Y. Cho, and J. Kim. Auto-meta: Automated gradient based meta learner search. arXiv preprint arXiv: 1806.069277 2018.
G. Koch, R. Zemel, and R. Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 350(6266):1332- 1338, 2015.
Z. Li, F. Zhou, F. Chen, and H. Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint arxiv 09835, 2017.
D. Lian, Y. Zheng, Y. Xu, Y. Lu, I.. Lin, P. Zhao, Huang, and S. Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations, 2019.
H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv: /8066000090 2018.
T. Munkhdalai and H. Yu. Meta networks. In International Conference on Machine Learning, pages 2564-2563. PMLR, 2017.
M. Nikolić, M. Mahmoud, A. Moshovos, Y. Zhao, and R. Mullins. Characterizing sources of ineffectual computations in deep learning networks. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 165-176. IEEE, 2019.
B. N. Oreshkin, Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. arXiv preprint arXiv: 1805 10/23, 2018.
A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally. SCNN: An accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH computer architecture news, 45(2):27-40, 2017.

---

S. Ravi and H. Larochelle. Optimization as model for fho-shot learning, 2016.
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q V. Le, and A. Kurakin. Large-scale evolution image classifiers. In International Conference on Machine Learning, pages 2902-2911. PMLR, 2017.
A. Samajdar, Y. Zhu, P. Whatmough, M. Mattina, and T. Krishna. Scale-sim: Systolic accelrraoor simulator. arXiv preprint arXiv: arXiv:/8/1.02883, 2018.
A. Shaw, W. Wei, W. Liu, L. Song, and B. Dai. Meta architecture search. arXiv preprint 19140544.,.09584, 2018.
D. So, Q. Le, and C. Liang. The evolved transformer. In International Conference on Machine Learning, pages 5877-5886. PMLR, 2019.
J. Su, J. Faraone, J. Liu, Y. Zhao, D. B. Thomas, P. H. Leong, and P. Y. Cheung. Redundancy-reduced MobileNet acceleration on reconfigurable logic for ImageNet classification. International Symposium Applied Reconfigurable Computing, pages 16 28- Springer. 2018.
F. Sung. Y. Yang. L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 119--120,, 2018.
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning, arXiv preprint arXiv: 04080, 2016.
K. Wang, X. Gao, Y. Zhao, X. Li, D. Dou, and C.-Z. Xu. Pay attention to features, transfer learn faster CNNs. In International conference on learning representations, 2019.
D. Whitley, genetic algorithm tutorial. Statistics and computing, 4(2):65-85, 1994.
B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer. FBNet: Hardware- aware efficient ConvNet design via differentiable neural architecture search. In CVPR, 2019.
Y. Xu, L. Xie, X. Zhang, X. Chen, B. Shi, Q. Tian, and H. Xiong. Latency-aware differentiable neural architecture search. arXiv preprint arXiv:2001. 063992, 2020.
T.-. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V. Sze, and H. Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), pages 285-300, 2018.
C. Zhang, Y. Cai, G. Lin, and C. Shen. Deepemd: Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12203-12213, 2020.
Y. Zhao, X. Gao, D. Bates, R. Mullins, and C.-Z. Xu. Focused quantization for sparse CNNs. Advances in Neural Information Processing Systems, 32, 2019a.
Y. Zhao, X. Gao, X. Guo, J. Liu, Wang, Mullins, P. Y. Cheung, G. Constantinides, and C.-Z. Xu. Automatic generation of multi-precision multi-arithmetic enn accelerators for FPGAs. In 2019 International Conference on Field- Programmable Technology (ICFPT), pages 45-53, IEEE, 2019b.
Y. Zhao, D. Wang, D. Bates, R. Mullins, M. Jamnik, and P. Lio. Learned low precision graph neural networks, arXiv preprint 2000::0009.09232.. 2020a.
Y. Zhao, D. Wang, X. Gao, R. Mullins, P. Lio, and M. Jamnik. Probabilistic dual network architecture search graphs. arXiv preprint arXiv:2003.00 09676, 2020b.
B. Zoph and Q V. Le. Neural architecture search with reinforcement learning arXiv preprint 1rXiv:1611.01572 2016.
12


Reviews:
[{"review_id": "tNQSm_uo_xo", "reviewer": "Reviewer_AK57", "summary": "This paper proposes a unified framework, namely rapid model architecture adaption, for joint searching the best architecture given multiple tasks (in T), various hardware (in H) and various constraints (in C). Traditional NAS methods have a complexity of O(THC) where this work has a complexity of O(1)\n\nIn essense, this paper glues the existing techniques from three domains, i) neural architecture search such as global expansion and adaptive number of layers, ii) a naive adaption meta learning strategy to improve the 1shot and 5-shot ability, and iii) look-up table like performance benchmarking technique in FB-Net NAS paper to profile the hardware latency. ", "questions": "As this paper is clearly written, the only question I can raise now is if we should accept a paper with many engineering efforts. In my humble opinion, this paper seems like a moderate modification of OFA, replacing the latency predictor by a straight-forward baseline, layer-wise profiling that is introduced in FBNet long time ago. Then adapts another simple baseline from meta-learning domain to minimize the O(T) into O(1). \n\nPost rebuttal comment:\n\nThis paper did find an interesting question in meta-learning, and I agree that this is critical and should be deemed as a solid contribution to the current research community. I raised my score to weak accept. ", "limitations": "I do not see direct negative societal impact of this work so far. \nAs for the limitation of the method itself, to use the latency profiler, it requires manual benchmarking the latency on multiple devices and might be hard for researchers that has insufficient resources. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this paper is obvious, it does not try to complicate the current literature by introducing some articulated complex search strategy but try to formulate a novel problem, that minimizing the multi-task multi-hardware and multi-search space in one step. The experiments demonstrate the effectiveness of this method. \n\nOn the other hand, it is obvious that this paper does not have much technical novelty. The entire methods can be separated into independent procedures that aim to solve a specific problem. Compared to the relevant baseline Once-For-All, which brings the a novel solution for multiple device and constraints neural architecture search by leveraging the super-net and proposing those tricks like progressive shrinking and depth pruning, this work seems to incoporate several existing approach to tackle a similar problem. In short, the engineering contribution of this paper is significantly larger than the algorithmic one.  I am not sure if this paper introduces any new message to the current community. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oY1YXILtlM", "reviewer": "Reviewer_Wnsk", "summary": "This paper presents a NAS method in the context of many-task many-hardware few-shot learning. To this end, the authors introduce many techniques, including supernet meta-training strategy, layer-wise profiling for latency evaluation, and etc. The experimental results show the effectiveness of the proposed method in some cases.\n\nAfter rebuttal: I tend to give Borderline, but in system, I only can choose from *Borderline Accept* and *Borderline Reject*. Thus, I keep the scoring Borderline Reject. But I lower the confidence level from 4 to 2.", "questions": "1. The novelty of this paper seems to be limited. The proposed method seems to be more like a simple stacking of different techniques. More critically, some components in the proposed method have been studied in existing methods, such as super-net meta-training strategy, layer-wise profiling.\n\n2. In Introduction, the authors emphasize the efficiency of the proposed method, i.e., reducing the search complexity from O(THC) to O(1). Compared with OFA, I am confusing which proposed component make the search cost to reduce to O(1).\n\n3. Layer-wise profiling has been proposed and investigated in FBNet. In my opinion, this is not a good way to evaluate the latency of candidate architectures. In practice, the inference framework (e.g., TensorRT) would perform layer/operation fusion. For instance, TensorRT will fuse conv and batch norm layer into a single layer. In this case, the overall latency of the architecture is not equal to the sum of that of each layers. The reality is often more complicated than the above case. Thus, I think it would be difficult to accurately estimate the latency.\n\n4. The compared methods in Tables 3 and 4 seem to be outdated. Most of the compared methods/papers are published before 2020, which make it hard to verify the effectiveness of the proposed method.\n\n5. I think the word “Adaption” in the paper title means adapting from an existing architecture to another. But after reading the paper, the proposed method is more likely to search for new promising architectures rather than adaption. I am not very sure about this. It may be my misunderstanding.\n", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The authors consider a NAS problem for many-task many-hardware learning, which may be useful for real-world applications.\n2. To address the issue the architecture may be over-fitting, the authors propose global expansion and adaptive number of layers in the backbone design.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "f4jRuGn0wfX", "reviewer": "Reviewer_DeFC", "summary": "The author proposes a very interesting new method, that is, the adaptive adaptation problem of multi-devices in the domain of NAS. The idea of meta-learning is introduced into the NAS framework to search the underlying optimal neural network model. The contributions of this paper mainly focus on two aspects: one is the generality of the method, the network architecture obtained by the method in this paper can be adapted to multiple devices at the same time, without the need to search the optimal structure for each device separately; the second is the superiority of the proposed method, the search time is shorter than those comparison methods.", "questions": "but their final classification accuracy is not the same. If the results on three devices are different, then the goal of this paper seems unachieved. Please refer to sentence in the Introduction section: “Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this method offers a chance to reduce the hardware search complexity from O(HC) to O(1)”. How to explain this sentence?", "limitations": " It seems that the author has only solved some of the existing problems, and has not provided the relevant results for the optimization goal of multiple devices. It is necessary to supplement the relevant experimental results in the revised version.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper provides a solution for complex task, which is novel and has important meaning in the field of NAS research.\n\nWeaknesses:\n1 The layout of Method section seems unreasonable. First, the Super-net meta-training strategy subsection introduces the training optimization strategy for architecture search, while the following two strategies (Layer-wise profiling and Adaption strategy) are some skills for network structure search. The order of the two seems to be reversed. Second, Tables 1 and 2 present partial results, and it seems more appropriate to move them to the Evaluation section.\n2 The Method section lacks the focus of the core issue: when there are multiple different hardware devices, the author does not give the detailed description of how to set the hardware constraints. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LcUSxkUSSw0", "reviewer": "Reviewer_7Ry8", "summary": "This paper introduces a method for many-task many-hardware NAS. The approach combines existing hardware-aware NAS with MAML. A layer-wise profiling strategy is introduced to improve profiling time, and modifications to the NAS algorithm to prevent overfitting in the few-shot setting is also proposed. The method achieves competitive experimental results on omniglot and mini-imagenet whilst being very efficient.", "questions": "Have the authors experimented with discovering how much pooling to add in the end rather than manually fixing it?", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \nThe paper is clear and well written. The proposed layer-wise profiling is a simple solution that greatly outperforms latency predictor baseline in this setting. The proposed Global Expansion and Adaptive Number of Layers are also simple changes that I like. The experimental results are very strong.\n\n**Weaknesses:** \nLayer-wise profiling is ultimately a heuristic and could have larger errors for strange computation graphs where the total runtime is not equal to the sum of the runtimes of each layer. That being said, experimentally this approach has a lower MSE than baseline label predictor so it's a great step up. \n\nThe need to manually add a final pooling layer feels strange, as this is a NAS method one would have hoped that this could have been discovered automatically.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept