Hardware-adaptive Efficient Latency Prediction for NAS via Meta-Learning
Authors: Hayeon Lee
Sewoong Lee
Song Chong
Sung Ju Hwang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural architecture search
meta-learning

Abstract: For deployment, neural architecture search should be hardware-aware, in order to satisfy the device-specific constraints (e.g., memory usage, latency and energy consumption) and enhance the model efficiency. Existing methods on hardware-aware NAS collect a large number of samples (e.g., accuracy and latency) from a target device, either builds a lookup table or a latency estimator. However, such approach is impractical in real-world scenarios as there exist numerous devices with different hardware specifications, and collecting samples from such a large number of devices will require prohibitive computational and monetary cost. To overcome such limitations, we propose Hardware-adaptive Efficient Latency Predictor (HELP), which formulates the device-specific latency estimation problem as a meta-learning problem, such that we can estimate the latency of a model's performance for a given task on an unseen device with a few samples. To this end, we introduce novel hardware embeddings to embed any devices considering them as black-box functions that output latencies, and meta-learn the hardware-adaptive latency predictor in a device-dependent manner, using the hardware embeddings. We validate the proposed HELP for its latency estimation performance on unseen platforms, on which it achieves high estimation performance with as few as 10 measurement samples, outperforming all relevant baselines. We also validate end-to-end NAS frameworks using HELP against ones without it, and show that it largely reduces the total time cost of the base NAS method, in latency-constrained settings.

HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning
Hayeon Lee Sewoong Lee Song Chong' Sung Hwang
KAIST AITRICS", Seoul, South Korea
(hayeon926, dltpdnd21, songchong, sjbwange22
Abstract
For deployment, neural architecture search should be hardware-aware, in order satisfy the device-specific constraints (e.g.. memory usage, latency and energy con- sumption) and enhance the model efficiency. Existing methods on hardware-aware NAS collect large number of samples (e.g., accuracy and latency) from a target device, either builds lookup table a latency estimator. However, such approach is impractical in real-world scenarios as there exist numerous devices with different hardware specifications, and collecting samples from such a large number of de- vices will require prohibitive computational and monetary cost. To overcome such limitations, we propose Hardware-adee adaptive Efficient Latency Predictor (HELP)) which formulates the device-specific latency estimation problem as meta-learning problem, such that we can estimate the latency of a model's performance for a given task on an unseen device with few samples. To this end, we introduce novel hardware embeddings to embed any devices considering them as black-box func- tions that output latencies, and meta-learn the hardware-adaptic latency predictor in a device-dependent manner, using the hardware embeddings. We validate the proposed HELP for its latency estimation performance which it achieves high estimation performance with as few as 10 measurement samples, outperforming all relevant baselines. We also validate end-to-end NAS frameworks using HELPP against ones without it, and show that largely reduces the total time cost of the base NAS method, in attenyycoonstrained settings. Code is available at
unseen platforms, on
1 Introduction
Neural Architecture Search (NAS) [39, 1, 25, 22, 21, 36, 5], which aims search for the optimal architecture for given task, has achieved a huge practical success by overcoming the sub-optimality of manual architecture designs. However, for NAS be truly practical in real-world scenarios, should be hardware-aware. That is, we need to search for the neural architectures that satisfy diverse device constraints (e.g., memory footprint, inference latency, and energy consumption). Due to the practical importance of the problem, many existing works propose to take into account the hardware efficiency constraints (mostly latency) the search process [29, 3, 34, 32, 6, 2, 37, 4, 33].
However, one of the main challenges with such hardware-awar NAS is that collecting training samples (e.g., seaitectuur-latency pairs on each target device) build reliable prediction models for the efficiency metrics, is computationally costly requires the knowledge of the hardware devices. Existing hardware-aware NAS methods usually require a large number of samples (e.g., 5k) and train metric predictors for each device from scratch. Additionally, most works [29, 3, 34, 32, 6) are task-spe,ific, and thus such collection performance samples should done from scratch, for new task. Thus, the sample collection process is prohibitively costly when we consider real- world
Thhese authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Unseen Devices
Collection Pipeline
HELP (Ours)
Latency constrained
Small Latency Samples
Meta-Training Pool
Search
Possible Architactures 2 spf
Messurement Enough
Meta-
Queryy
HW--AS
HELP
Knowledge
Algorithm
Transfer
Feedback
Architecturel Pyfordh Compliation Device
Conventional Latency Predictor
Quarry
Obtained Architecturo
Need Large Latency Samples Measuroment
Task-Specific
Feedback
Application Build
Predictor
Figure I: Concept. Conventional latency estimation methods require large number of mechitettue-latenne pairs to build a prediction model separately for each device, which is inefficient. Contrarily, the proposed HELP uses a single meta-latency predictor that can fast adapt to any unknown device by collecting only a few latency measurements from it, by utilizing the meta-knowledge of the source device pool.
deployment of a model large number of hardware devices, for any tasks. OFA [4] alleviate the search cost by utilizing a high-performance network trained on ImageNet as the supernet, which does not require training the models from scratch. Yet, they are still sub-optimal since building device-specific predictors for metric still requires a large number of samples to be collected, to build layer-wise latency predictor for each device. This could take multiple hours depending on the task and the device, and becomes a bottleneck for latency-constraine NAS. BigNAS [38] considers FLOP as the efficiency metrie, but this is highly inaccurate proxy since latency of an architecture could differ based on its degree of parallelism and memory access cost, for the same FLOP.
To overcome such limitations, we propose a novel sample-efficient latency predictor, namely Harpware-ddaptive Efficient Latency Predictor (HELP), which supports the latency estimation for multiple devices with single model, by allowing it rapidly adapt unseen devices with only few latency measurements collected from each device. Specifically, we formulate the latency prediction problem as a few-shot regression task of estimating the latency given an architecture-devic pair, and propose a novel hardware embedding that can embed any devices by utilizing the latencies of the reference architectures on each device its embeddings. Then, we propose meta-learning framework which combines amortized meta-learning with gradient- -ased meta-learning, to learn the latency predictor to generalize across multiple devices, utilizing the proposed hardware embeddings. This allows the model to transfer knowledge learned from known devices to new device, and thus to achieve high sample efficiency when estimating the latency of unseen devices.
HELP is highly versatile and general, as it is applicable to any hardware platforms and architecture search spaces, thanks to device-agnostic hardware embeddings. Also, HELP can be coupled with any NAS frameworks to reduce its computational bottleneck in obtaining satency-constraing architectures. Especially, when coupled with rapid NAS methods such as MetaD2A [16], OFA [4] and HAT [33], HELP can perform the entire latency-constrained NAS process for a new device almost instantly. We validate the latency estimation performance of HELP on the NAS-Bench-201 space [9] with various devices from different hardware platforms, utilizing the latency dataset for an extensive device pool in WW--NAS-Bende dataset [18]. The results show that our meta-learned predictor successfully generalize to unseen target devices, largely outperforming baseline latency estimation methods using at least 90x less measurements. Then, we combine HELP with existing NAS methods [4, 33] and show that our meta- meta-latency predictor largely reduce their total search cost. To summarize, the main contributions of this paper are as follows:
We formulate the latency estimation of neural architecture for given device as a few-shot regression problem, which outputs the latency given an architecture-device pair as the input.
To represent heterogeneous hardware devices, we propose novel device-agnostic hardware embedding, which embeds a device by its latencies on reference architectures.
We propose a novel latency predictor, HELP, which meta-learns few-shot regression model to generalize across hardware devices, that can estimate the latency of an unseen device using only few measurements. HELP obtains significantly higher latency estimation performance over baselines with minimal total time cost.
We further combine HELP with existing NAS frameworks to show that it leads to find latency- constrained architectures extremely fast, eliminating the latency-estimation bottleneck in the hardware-constrained NAS.
2

---

Meta-Training
Meta-Test
Source Pool
Unseen Device
Unseen Device
HW Adaptive Init. Params,
Transfer {0,00 Φ
Few-SSor Samples
X H sAms
Fest
T
Few Shot Samples
0
Meta Training
redlicioo
0
Latemcy
{0, }}
Unseon
Unseen
Arch.
Predict Latency
Figure 2: Overview. For the hardware-adaptive latency estimation of an unseen device for latency-conssrriined NAS, we introduce latency- based hardware embedding and a modulator of the initial parameters. By formulating the sample-efficies efficiefn NAS problem as a few-shot regression problem under the meta-learning framework, our learnee predictor successfully exploits meta- knowllnn 0 from the source device pool, to achieve high sample efficiency on unseen devices.
2 Related Work
Latency Prediction in NAS Hardware-awars NAS [29, 3, 4, 34, 32, 13, 33, 26] aims to design neural architectures that achieves a good trade-off between accuracy and latency for efficient deploy- ment to target devices. While they need to consider the actual latencies of the architectures device, evaluating them while searching for architectures is costly. Thus, most existing works replace it with proxies, such as FLOPs [38], but they are inaccurate measure of latencies. Earlier hardware- aware NAS methods have been used a layer-wise latency predictor (lookup table) [29, 3, 4, 34] which sums up the latencies measured for each operation in the given neural networks. While this is better than FLOPs, they do not accurately capture the complexities of multiple layer execution on real hardware devices. Thus, recent methods use an end-to-end latency predictor [10, 33] that trained with the latency measurements from the target device, to improve the latency prediction accuracy. BRP-NAS [10], which is GCN-based model, is currently the ofate-of-the-art end-to-end latency pre- dictor. However, this method is also limited in that it requires large number of architecture-latene pairs for each device. Since the latency estimator cannot generalize to a new device, whenever a new device is given, the NAS system needs to build a new latency estimator, which may take hours to finish. The proposed method significantly reduces the total building cost the latency predictor for each device, by using single meta-learned predictor and adapting it to new device with only few measurements from it.
target
Meta-learning and Meta-NAS Meta-lcarning (learning to learn) [30] aims to learn a model that generalizes over distribution of tasks rather than single task, such that the model meta-learned over a large number of tasks rapidly adapts to an unseen task. The performance of existing methods [31, 28, 12, 24, 17] is usually evaluated on few-shot classification tasks, where the model classifies between instances of unseen classes given only few training examples. Recently, several works have proposed to utilize meta-learning for NAS. Most of them focus on few-shot classification tasks search for the architectures and parameters that can generalize well to new task [11, 20, 27] with gradient- based learnnne. However, they have limited practical applicability since the computational cost of meta-learning is prohibitively large, for NAS under standard many-shot setting. A recently proposed meta-NAS framework with amortized meta-learning, MetaD22 [16), which utilizes a set encoder to encode task and uses graph decoder to generate task-adaptive architecture, obtained state-of-the-art performance on unseen datasets, with minimal search cost. We also propose a similar amortized meta-leamiig framework, for aeltwarre-ddapiiv NAS, based on a novel hardware device embedding. However, after obtaining the conice-conditionedd initialization parameters, we perform further inner gradient steps for device-specific adaptation, unlike MetaD2A [16].
3 Method
Our goal is to design a prediction model that can accurately predict the efficiency metric for a novel architecture-device pair, using small number of performance samples from the device. While our method is generally applicable to any efficiency metries that can be measured from the device (e.g. latency, energy consumption, and memory footprint) we focus on the latency prediction in this work.
3

---

3.1 Problem Definition
{h",X' X", Y where h" E H hardware device, X C X' is set of neural architectures, and Y C C is set of latencies of X" directly measured on the hardware device ht. Then, our goal is to learn a regression model f(x;; 0) parameterized by 0 that estimates the latency E Y of a neural architecture x E X for given hardware device h" by minimizing empirical loss C (e.g. mean squared error) on the predicted values f(x"; 0) and actual measurements Y as follows:
Assume that we are given a task specification
IR
min (X':()))))
(1)
Learning such a regression model seems like a simple problem, since we can collect any number of measurements from any devices. However, this is a more challenging problem than it seems:
1. Since cannot generalize across devices, we need to learn N predictors (-; 0T) for devices, collecting performance samples and training the performance predictor separately for each device, which requires prohibitive computational costs with large number of devices.
2. Even when assuming that learn device specific predictor, in order not to overfit the regression model, we need to collect a large number of sranteectur-lltenge sample pairs for each device (e.g. 2k [33], 5k [23] samples) to achieve reliable prediction performance.
device-specific
3. With lack of generalization ability across devices and architectures, the NAS framework needs to repeat the time-consuming sample collection process whenever a new device is given, that may take hours to complete. This will become computational bottleneck even for rapid meta-NAS framework such as MetaD2A [16].
To overcome such limitations, we propose a single predictor f(:: 0) which can generalize across devices and architectures, by fast adapting new target device and architecture that are unseen during training. by collecting only small number of archieetture-ltenn (X" < X Y < Y"). We achieve this goal with a meta meta-learning framework that can transfer knowledge obtained from the device and architecture pool p(T).
pairs from the device
3.2 Hardware-adaptive Latency Prediction with Device Embeddings
While the measured latency y + (x, h) is dependent on both the device type h and the architecture x, existing latency prediction models takes the form of (a: 0), ignoring the device-speciff constraints, since the latency predictor is learned for each device separately. This results in poor performance when learning single latency predictor perform metric estimation multiple devices, including unseen ones, which is our main objective. Thus, we propose hardware-conditionee prediction model:
f(x,, h:; 0) X' x H
(2)
that can predict the latency differently depending on the device type h. even for the same architecture A crucial challenge of our handware-conditione prediction model is how to represent the hardware device h, for all devices regardless of their platform types. This is not trivial problem since the physical architecture of hardware devices could be very different (e.g. CPU vs FPGA). Thus, we simply consider the device as a black box function which outputs the inference latency given an architecture, instead of directly modeling the hardware devices. Then, we obtain the latencies of the device on fixed set of reference neural architectures as follows:
VA = {ui (#1, h), (I2, (t.,... (ā
(3)
where E is the set of the reference neural architectures {I1, I2,..., C. X, fixed across all tasks for both meta--raining and meta-test, and d the number of the reference architectures; in our experi- ments, we set d = 10. Further, yf( (zi, h) = {xe(x,, h) h)- (i))( (V/ - (V,(0) } are standardized latency values ranging from Since this set of reference devices should be representative, we select them to be diverse and heteroge- neous. As for the reference architecture,, we randomly sample them from the search space. For more detailed descriptions of the reference devices and architectures, please see the supplementary file. The proposed black-box treatment of hardware devices, and the latency-based hardware embedding allows us to embed new device without considering its detailed hardware specification.
1, where Vi( {y1 (x1, h), (2222,,))) .... Va h))-

---

3.3 Meta-Learning the Hardware-adaptive Latency Predictor
To tackle the few-shot regression problem for multiple devices by utillizing the collected pool of devices and architectures P(T), we propose a novel hardware- hardware-adaptive meta-learning framework of the latency predictor that meta-learns faz h; 0) across a task distribution p(T) to rapidly adapt the predic- tor [(ar, h; 0*) to an unseen neural architecture x given the task specification = (h", Vv, X", Y"). During the meta-training phase, we leverage the episodic training strategy, which we simulate a few-shot regression task for each iteration by randomly sampling task from the device-architecture pool p(T) and splitting 7 as training set D = {h", X", Y and test set D = {hh { YY where X C. X and X C X denote sets of neural architecture samples, X is the set of few-shot samples [XT] < X Note that there is overlap between them; that is, X n = 0. Y C Y and Y Y denote the sets of corresponding latency values of neural architectures X and XT, measured on device h", respectively. Basically, for each task т, we obtain the hardware-adapting prediction model [(X, ; (T) function of V/Z- Formally, meta-train the latency predictor to minimize the test loss [(: DT) by optimizing the following task adaptive meta-learning objective:
Σ ((I(X",VI:'0") VZ:0"), TTP(()
min
We can simply use the task embedding VE to obtain a task conditoodd latency predictor, in which case we are using an amortized meta-learming framework similar to one proposed in Lee et. al. [16], which aims to meta-learn a dataset- dataset-adaptive performance predictor and NAS framework. However,
we can further perform fho-shot adaptation with the few latency measurements collected from the target device, by conducting inner gradient updates with them as follows:
0 ((+1) To o L(S(X,, (())
YT) for t= 1=
(5)
where denotes the teh inner gradient step, I' is the total number of inner gradient steps, and a denotes the multi-dimensional global learning rate vector [19]. This meta-learning formulation allows us to adapt to a new device rapidly, by using the knowledge of the devices used for meta-training. However, when we are encountered with a completely new device that has little relatedness to any devices from the meta-device pool, it will be helpful to deviate more from the meta- knowledge captured by (o) [15]. To this end, we introduce a hardware- adaative modulator x = 5(V; Φ) : T + Rd parameterized by Φ to modulate the initial parameter as 0go) = 0 zT, where 0 ((0) is the new initialization for hardware hT, Following [15], we set ((o) + 0 z" for weights and 0()) + 0 + 2T for biases with an element-wise multiplication operator o. This leads to the following update rule:
0 (0)
0 * T
0 pe+
io Vo.
(.) L(f(X", V% 12 ())) for t= 1, 1......T T
where T is the number of inner gradient steps. Then, the final meta-learning objective is as follows:
min 0.6
Σ c(f(X", V. :
(8)
Tmp(TT
Thus, we meta-learn both the model parameters for the hardware-adaptive latency predictor, and the modulator for the shared initial parameters.
Few-shot Adaptation to Unseen Devices (Meta-Test) We now describe how use our meta- learned latency prediction model f(; 0) to estimate the latency of an architecture on a novel device h" that is unseen during veta-training. The task-specific predictors [33, 4, 23] need a large amount of latency measurements from an unseen device h², over diverse architectures in order not to overfit, which may take an excessive time to collect. However, our model is able to measure the latency values y" of a new architecture x" by collecting only few latency measurements from it (we use 10 or 20), using the device-conditioned meta-learning. Given latency prediction task of an architecture z" on a novel device ht" v (h" X"), we first obtain its hardware device embedding Vn, by obtaining its latencies on a fixed set of reference architectures, following Equation (3). Then we use the device embedding Vn to obtain the device-optimized parameters of the latency predictor 0% (r following Equation (6) and (7). Then, we use the device-optimized latency predictor f(-, Vis: (/r+)) to measure the latency of z". We can further combine this meta-latency predictor with a NAS method to perform latency-cosstrained NAS for novel device.
5

---

Avg. Corr. of Meta-Tes Test
1.0
Method
Unseen Platform
0.8
Respi4
ASIC
FPGA
MAML [12]
0.568
0.602
0.541
0.6
ANP [14]
0.801
0.657
0.884
Meta-SGD [19]
0.844
0.831
0.882
0.4
HELP (Ours)
Init. Meta SGD
Amertization
0.568
0.604
0.2
Scratch
HELP (Ours)
HW-Condition
0.853
0.904
0.361
Shot Adapt
0.913
10
100
200
500
1000
1500
2000
2500
3000
= Modulator
0,885
0.942
0.889
The Number of Samples
Table 2: The correlation of the estimated latency with
Figure 3: Latency estimation performance as
HELP to the actual latency, unseen devices with 10 measurement samples from each device (FBNet space).
function of the number of samples collected.
Computational Complexity of HELP. The meta- training of the latency predictor is done only once, and once done, we can adapt the meta-latency estimator for the latency estimation of any number of devices. Since conventional approaches require to collect a large num- ber of samples and train a device-specific latency estima-
Method
Azchitecture Search Cost
Latency Estimator Building Cost
Tesk-specific NAS
OID) O(1)
O(DN) O(DN)
MetaD2A [16]
MetaDZA HELP
O(D)
Table 1: Cost of NAS and latency estimation.
tor for each target device, while HELP only needs to collect 10 samples per device. HELP reduces the time complexity of obtaining latency estimations from (DDN) to O where D is the number of devices and N is the number of samples to sufficiently train each latency estimator.
4 Experiment
In this section, we first verify the efficacy of our meta-learning scheme the latency prediction of architectures on unseen devices, in Section 4.1. In this section, we also validate the sample-efficieney and the performance of HELP against existing latency prediction models, and predictor trained with conventional meta-learning. Then, in Section 4.2, we validate HELP's effectiveness and efficiency on the full Latency-constraiin NAS for novel devices, by combining it with existing NAS methods.
Search Space Following the evaluation procedure of HW-NAS- [18], we consider two search spaces, 1AS-Bench-201 [9] and FBNet [34]. Additionally, we consider MobileNetV3 [13, 4] and HAT [33] search space for end-to-end latency conctyyconstrrined NAS in Section 4.2. For a detailed description, refer to the supplementary file.
Meta-Training Pool/Meta-Test Pool To construct the Peta-Training Pool, we collect the latency measurements from 18 heterogeneous devices, including GPUs, CPUs, mobile devices (NVIDIA 1080ti, Titan X, Titan XP, RTX 2080ti, Xeon Silver 4114, Silver 4210r, Samsung A50, S7, Google Pixel3, Essential Ph 1). For GPUs, we consider three different batch sizes [1, 32, 256(64) and for all other hardware devices, we use the batch size of 1. We collect 900/4000 (architecture, latency) pairs of each training device for NAS-Bench-201 and FBNNt search space, respectively. As for the Meta-Test Pool, we consider Unseen Devices and Unseen Platforms. Unseen Devices include NVIDIA GPU Titan RTX, Intel CPU Xeon Gold 6226, and Google Pixel2, which are different from the devices in the meta- training pool but belong to the same categories (GPU, CPU, mobile device). On the other hand, Unseen Platforms include Jetson AGX Xavier, Raspi4, ASIC-Eyeriss, and FPGA, which are completely unseen categories of devices. For Raspi4, ASIC-Eyeriss, FPGA and Pixel3, we use the latency measurements provided in the HW-NAS-Bende [18]. For the implementation details of our model, please refer to the supplementary file.
Baselines We compare our framework against relevant baselines. 1) MAML [12]: A few-shot regres- sion baseline which meta-learns the initial parameters over multiple tasks via bi-level optimization. 2) Meta-SGD [19]: An extension of MAML with the meta-learned learning rate for the inner-gradient step. 3) ANP [14]: few-shot regression model implemented with Attentional Neural Processes, which uses differentiable attentions to attend to the relevant contexts for the given query. 4) BRP- NAS' [10]: A predictor- NAS method with a graph convolution neural network-bascd latency predictor. This baseline achieves the previous ofate-of-the-ar performance on latency prediction in the NAS-Bench-200 search space. 5) MetaD2A [16]: This is a meta-NAS framework without a latency predictor which enables rapid architecture search (a few GPU seconds) for unseen datasets, which obtains ofate-of-the-ar art performance on multiple datasets in the MAS-Bench-201 search space.
For experiment on LatBench provided by BRP-NAS, please refer to the supplementary file.

---

Table 3: Comparison of the latency estimators unseen devices and platforms on NAS-Bench-201.
Unseen Device
Unseen Platform
Method
Transfer
Sample
GPU
CPU
Pixe12
Raspi4
ASIC
FPGA
Mean
FLOPS
0,950
0.826
0.765
0.846
0.437
0.900
0.787
Layer-wise Predictor
0.667
0.866
0.767
BRP-NAS [10]
900
0.814
0.796
0,666
0.847
0.811
0.801
0.789
-ee-taat samples)
3200
0.822
0.805
0.693
0.853
0.830
0.828
0.805
HELP (Ours)
10
0.987
0.989
0.802
0.890
0.940
0.985
0.932
Proxy
Layerwise
BRP-NAS
HELP (Ours!
Proxy
Layer-wise
BRP-NAS
HELP (Ours)
Nestura
Massaned
Messure
Heaured
1LOPs (M)
Predicted
Predicted Lat.
Precicted Ltt.
1L0Ps
Predicted
Predicted
Predicted
(a) Titan RTX GPU
Intel Xeon Gold CPU
Figure 4: Comparison of the estimated and measured latencies Titan RTX GPU and Intel Xeon Gold CPU. While BRP-NAS requires 900 samples train the latency predictor, meta-latency predictor requires only 10 samples for adaptation, and significantly outperforms in the estimation performance.
4.1 Efficacy of HELP Few-shot Latency Estimation for Novel Devices
We first validate whether the transferring the meta-knowledge obtained over meta-training pool to an unseen meta-test device helps to improve the sample efficiency and prediction performance of the latency predictor. We adapt the meta-latency predictor on 10 architecture-latency pairs of 6 unseen meta-test devices, and report the Spearman's 's rank correlation (higher the better) between the estimated latencies and actual latencies on 1,000 neural architectures from the test set in FBNet search space, over random runs. (Figure 3, Table 2, and Figure 5).
Latency Estimation Performance for Unseen Devices Figure 3 reports the average value of corre- lation scores of different predictors as function of the number of architecture-latency pairs from meta-test devices. Shaded regions are the range of standard deviation of 5 runs with random seeds. HELP and Meta-SGD uses the initial parameters that are meta-learned over large meta-training pool, and fine-tunes its parameter with given training samples. On the other hand, Scratch means to simply train a regression model from scratch with a given samples. The result shows that using meta-knowledge (HELP .P and Init. of Meta-SGD) consistently outperforms the model trained from scratch. Specifically, HELP's s lateney predictor with hardware-adaptive initial parameter Og achieves significantly larger performance gain over baselines when the number of samples is smaller (e.g.. 10 and 50). Such sample-efficiency allows HELP to search for architectures that satisfy the latency constraints with orders of magnitudes shorter time compared to existing methods.
Effect of the Hardware-adaptive Meta-learning We analyze the effect of hardware-adaee adaptive meta- leamning in Table 2. We observe that the meta-latency predictor using the proposed modules largely outperforms the other ones trained with other meta-learning baselines that are hardware-indepente which shows the effectiveness of our hardware-adaptive meta-learning unseen platforms. This due to the heterogeneity of the tasks (devices) in the meta-training dataset, in which the task- conditioning becomes more important.
Effect of the Size of the Device Pool We further analyze the effect of
Aug. Dor, Mota--eed
the size of the meta-training pool, on the performance of the meta-latency predictor. Figure 5 reports the performance of our latency predictor over different sizes of the randomly sampled device pool. In particular, when the number of devices in the meta-training pool is 10 or more, our model achieves over 0.9 correlation on unseen devices using only 10 samples of the unseen target device, regardless of the device types of the meta-training pool. Contrarily, Meta-SGD does not yield meaningful performance gains even with large meta-training pools.
0.6
0.5
HELP (ours
0.4
- SeD
6 5 10 12 Training
The
Figure 5: Effect of the meta-training pool size.
Sample-efficiency of HELP To demonstrate the sample-efficinncy of our meta-learned latency predictor, we compare against three baselines: 1) proxy predictor using number of FL.OPs 2) layer-wise predictor and 3) the latency predictor from BRP-NAS [10] (Table and Figure 4). The results show that FLOPs, although easy to compute, is inaccurate proxy for latency estimation.
7

---

Table 4: Performance comparison of different latency estimators combined with MetaD2A for latency- constrained NAS, on CIFAR- 100 dataset with MAS-Bench-200 search space, For the building time and the total NAS cost of MetaDZA+HELP, we report only time and cost during the meta-test time. The meta- meta-training training time of HELP is 25 hours and the time to meta-train the MetaD2A is 46 GPU hours, which is conducted only once across all unseen devices.
Model
Const (ms)
Lateney [ms)]
Accurccy (9%
MACs (M)
Latency Model Building Time
Total NAS Cost (Wall Clock)
Speed Up
Sample
MetaD2A+ BRI-NAS [10]
66.9
1220s
MetaD2A HELP (Ours)
13
67.4
47
10
25s
125s
9.8x
Unseen Devie:
Google Pixel2
MctaDZA BRP-NAS [10]
Meta22A + HELP (Ours)
22
34
19
73.5
70.5
185
55
900
10
1120%
25s
1220s
125s
9.8%
1,0x
Meta2AA HELP (Ouns)
Mcta2ZA + BRP-NAS [10]
34
34
34
73.5
73.5
185
185
900
10
1120s
25s
1220s
125s
9,8x
1.0x
Metal2ZA Layer-wise Pred.
73.2
121
900
9986
10M8s
1,0x
MetaD2A BRP.NAS [10]
18
21
67.0
86
900
940k
1040s
1.1x
MctaDZA HELP (Ouns)
69.3
51
10
11s
111s
9.9 x
Unseen Device
MetaDA Layer-wise Pred.
41
73.5
184
900
998s
1098s
L.xx
Titan RTX
Mctal2ZA BRP-NAS [10)
71.5
(Batch 256)
MemalZAA HELP (Ours)
71.6
9.9%
Meta)ZAA Layer-wise Pred.
41
73.5
184
900
9986
10N8s
1.0x
MetaD2A RRP.NAS [10]
25
23
70.7
82
900
94l1s
1040s
1.1%
MetaDZA HELP (Ouns)
25
71.8
86
10
11s
111s
9,9x
(k) Sononnn
49 Titan RTX Latency (ms)
I5 183 125 15 115 259
13
130
Gold 6226 Latency (M6)
Poxe22 Labency (ms)
Raspi Latency (mm))
Figure 6: The accuracy-latency trade-off of the NAS framework with an oracle accuracy predictor, combined with the oracle latency estimator (yellow star), BRP-NAS (green square), layer-wise predictor (blue triangle), and HELP (Ours- red circle). on various devices in HAS-Bench-201 space. HELP, with its accurate latency estimation, obtains prrettrrronter models, while baselines yield sub-optimal architectures.
The layer-wise predictor also achieves poor performance, because it cannot reflect the complexity and the holistic effect of the network architecture. The latency predictor from BRP-NAS [10], which is a 4-layer GCN with 600 hidden units followed by fully connected layer to produce scalar output, achieves significantly better estimation compared to the first two baselines. However, this model requires 900 samples from each architecture-device pair, as described in [10]. Finally, our HELP predictor achieves the best performance, achieving the Spearman's 's rank correlations of 0.987 on GPU and 0.989 on CPU, using only 10 latency measurements of the architecture each device. This shows the clear advantage of our method, in terms of the estimation accuracy and sample-efficieney
4.2 End-to-end Latency-constraine NAS with HELP
To show that HELP does help NAS frameworks rapidly obtain sonnny--oonttrrnee/ooppnadl archi- tectures for a novel device, we combine HELP with existing NAS methods, namely MetaD2A [16), OFA [4] and HAT [33], and validate the performance on the latency-constrained NAS tasks.
In Table and Table 5, besides latency and accuracy, we additionally report the üime-fffiiency of the latency estimators with three different measures. First, we use the number of samples which are the number of architecture-latency pairs obtained from the targel device, that are used to build or train the latency estimator. We also report the building time, which is the total wall clock time to build the latency estimator, including the time required for sample collection, architecture compilation on the target devices, transmitting the architecture to the target device, and measuring the latency on the device. Finally, we report the total NAS cost, which is the sum of both the estimator building time and the architecture search time, on target task. After obtaining the architecture, we measure the actual latency of the architecture on the target device and report it as latency (ms). For the building time and the total NAS cost, exclude the cost of any procedures that not done during the meta-test time, such as the meta- meta-training of MetaD2A model (46 GPU hours) and HELP (25/18 hours for MetaD2A and OFA), as well as the time to train the supernet for OFA (1,200 GPU hours).
We first combine HELP with MetaD2A and compare it with the MetaD2A combined with other latency predictors. We conduct NAS on MAS-Bench-201 [9] benchmark for the CIFAR- 100 dataset,
8

---

Table 5: The results of the latency constrained-NA. experiment for ImageNet- Ik with MobileNetV3 search space. For building time and total NAS cost OFA+HELP, we report only time and cost during the meta-test time. The meta- veeantraining time of HELP is 18 hours and the time to train the supernet for OFA is 1,200
GPU hours, which is conducted only once across all unseen devices.
Device
Model
MACS (M)
Larency (ms)
Accuracy (%)
Sample
Latency
Building Time
Total
(Wall Clock)
Speed Up
MobileNatV3-Lane [13]
219M
22.1
75.2
MassNet-Al [29)
312M
20.0
75.2
40,004.5h
1.0x
Unsen Device
(BNet-C1341
375M
27.5
74.9
7.5k
4.2h
580.2h
69x
Titan XTX
ProxyleSNAS-GPU
465M
22.0
75.1
.KK
2.8h:
502.8h
80x
(Batch 64)
OFA+Layer Pred. [4]
397M
21.5
76.4
2667x
ORA HELP (20ms)
230M
20.3
76.0
OFA HFJ.P H23m)(23ms)
268
23.1
76.8
10
26s
0.007h (266)
8.7Mx
OFA HELP (28ms)
346M
28.6
77.9
MohsleNatV--Laree [13]
219M
132
75.2
MaasNet-A1 [29)
32M
212
75.2
8k
35.6h
40.035.6h
1.0x
Unseen Platform
FBNes-B [34]
295M
212
74.1
7.5k
33.3h
609.3h
66x
Intel Xeon
ProxylesNNAS-CPJ [3]
465M
200
75.1
5k
22.2h
522.2h
77x
6226
OFA+Layc-wiis Pred. H]
301M
167
DFA HELP (170ms)
336
147
77.6
OFA HEEL (190ms) (190p()90ms))
375M
171
78.1
20
300s
0.08h (300s)
0.5Mx
MobileNerV3-Lnm [133
219M
70.8
Unseen Platform
MaisNet-AA [29]
312M
71.6
75.2
8k
24.9h
40,024.98 9h
1.0x
Jebson AGX Xavier
ProxylesNAS GPU
465M
82.6
75.1
5k
15.6h
515.6h
78x
(Batch 16)
OFA-Layert-wie Pred, 14|
349M
69.2
75.8
27k
84h
84hh
476x
OFA HELP (65ms)
243M
67.4
75.9
OFA HELP (70mm)
279M
76.4
76.7
10
112s
0.03h (112s)
1.3Mx
using Google Pixe12 mobile phone and NVIDIA Titan RTX GPU as the target devices The results in Table show that HELP largely outperforms BRP-NAS [10], the previous state-of-the art latency predictor, with 90x sample efficiency and 9.8x, 9.9x computational efficiency on Pixel2 and Titan RTX, respectively. Specifically, HELP MetaD2A can efficiently retrieve an optimal lafency-constrained architecture in 125s/1 Is for the given dataset on Pixel2/Titan RTX, respectively, while BRP-NAS [10]'s predictor with large building time (1 (1100//90s) becomes a bottleneck for trade-off of HELP
MetaD2A's rapid NAS process. Further, against the bascline latency estimators, by combining them with the oracle accuracy predictor on NAS-Benche Bench- 201 space (in Figure 6). With HELP, oracle NAS obtains near Pareto-optimal models in most cases, while combining it with other latency estimators yield sub-optimal models.
validate the accuracy-efficiene
In Table 5, we validate HELP on large-scale dataset, Ima- geNet [7], against state-of-the the--rt NAS methods. In this exper- iment, we only consider GPUs with various batch sizes (1, 32, 64) as the meta-training devices, and Titan RTX GPU, Xeon CPU and Jetson Edge GPU as unseen devices. We combine HELP with OFA by training the accuracy predictor on the pre-trained OFA network, and conducting predictor-guice evo- lutionary architecture search to the latency constraint. The result shows that, while searching for competitive architectures, HELP significantly reduces the total NAS cost of the baseline NAS methods which builds new latency estimator for each target device. Figure shows that OFA+HELP reduces the total NAS cost on a target device by 2140x, when compared with the OFA + layer-wise predictor. This allows to benefit from the rapid search speed of OFA, since the total NAS cost is only tens or hundreds of seconds, while the OFA + layer-wise predictor takes 15 hours, which is impractical. The details of the searched architectures are provided in the supplementary file.
Total NAS Cost
MresNet
FBNet
OFA
OFA+HELP
2140k
10
10"
10
Figure HELP reduces the total NAS cost by 2140x on Titan RTX. The total NAS cost is represented on a scalec
ProxylessNAS has roughly 42 unique blocks (7 different operations per 6 different input shapes) and use 5k architecture samples to build layer-wis, wise latency predictor. Thus, we proportionally estimate the number of samples to train its latency estimator for the FBNet and OFA supernet as 7.5k and 27k, which have 63 and 225 unique blocks, respectively. MnasNet does not have latency estimator, but directly measures the latency of every architecture in the search process (8k architectures).
Hardware-awan Transformer Architecture Search To demonstrate the task-level generality of HELP, we further conduct end-to-end latency-constrained Transformer architecture search experi-
For more results various devices, please check the supplementary file.

---

ments on machine translation task, WMT" 14 En-De, by combining HELP with HAT [33], which is hardware-aware NAS method for Transformer. For WMT 14 En-De, we follow [33, 35] for training,
Table 6: Results of hardware-awe aware Transformer architecture search on WMT' 14 En-De. By combining HAT with HELP, x fewer samples are used to train latency predictor while achieving competitive performance.
Target Device
Model
Constraint
Latency
Number of Samples
BLEU score
HAT+End-io- -nd Pred.
90ms
73.9ms
2000
27.08
Unseen Device NVIDIA Titan RTX
HAT+HELP (Ours)
90ms
74.0ms
10
27.19
HAT+Fnd-to- -nn Pred.
150ms
108. 4ms
2000
27.04
HAT+HELP (Ours)
150ms
106.5ms
27.44
HAT-End-to-End Pred.
200ms
159.6ms
2000
27.20
Unseen Platform CPU Intel Xeon Gold6240
HAT+HELP (Ours)
200ms
159 6ms
10
27.20
HATT+End-to-Enu Pred.
400ms
369.4ms
2000
28.09
HAT+HELLP (Ours)
400ms
2ms
27.52
validation, test setting of datasets. The meta-training device pool is configured only with GPUs such as NVIDIA Titan X, 1080ti, 2080ti, and the unseen devices are Titan RTX GPU and Intel Xeon CPU. As baseline model, train the end-to-end latency predictor (End-to-End Pred.) using 2000 architecture-latency pair samples for each device following HAT [33]. Table 6 shows the results of NAS with different latency constraints, and BLEU score of searched models. HAT+HELP, which replaces a latency predictor of HAT with HELP,, successfully obtains the competitive Transformer models while using x00x fewer samples for training latency predictor than the original HAT.
5 Discussion
Limitation The proposed hardware-conditionel meta-learning framework allows HELP obtain an optimal
network within few seconds, when combined with rapid NAS methods such as [16, 4, 33], since building a latency estimator is often bottleneck for them. However, combining HELP with slower NAS methods based on RL, gradient-baset search, and evolutionary algorithms will be less effective since the total NAS cost is dominated by the architecture search cost, rather than the time required to build the latency estimator. Yet, since mainstream NAS research nowadays is focusing more on the reduction of the architecture search cost [25, 22, 8, 4, 16], believe that the latency estimation will become more of a bottleneck, and our sample-efficient latency estimator will become even more useful for centtryc NAS.
Societal Impact Since our method requires to build meta-training pool only once, and meta-latency estimator can rapidly adapt to new device with as few as 10 samples, we can largely reduce the waste in the computational resources required for obtaining the latency measurements. Since the repeated measurements require large energy consumption that also yields high CO2 emissions, and reduce devices' lifetime, our method is more setroomment-fiendd] than existing methods that require a large number of measurements from each device.
6 Conclusion
We proposed a novel meta-learned latency predictor, that can estimate the latency of an architecture on a novel device, using only few measurements from it. While conventional latency prediction methods are inefficient since they cannot generalize across devices, and require a large number of latency measurements for each device, our latency predictor is meta-learned to rapidly adapt to an unseen device by utilizing the meta-knowledge accumulated over device pool. Using novel hardware embedding function that embeds each device based on its latencies on a set of reference architectures, we conducted nardware-conditions
meta-learning to obtain a device-specific initial parameters, and further took inner gradient steps to adapt to a new device. We validated our meta- latency predictor by measuring its latency estimation performance it outperforms baselines, using only 10 to 20 samples per device. Furthermore, we combined our latency predictor with three rapid NAS methods, to show that it performs latency-constrained NAS on unseen devices extremely fast and accurately.
unseen devices, which
Acknowledgements This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075), Samsung Research Funding Center of Samsung Electronics (No. 10201214-08145-44
0000120000800-01 and the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the We thank Seul Lee for providing helpful feedbacks in
Korean Govermment MSIT
10

---

preparing an earlier version of the manuscript and NMSL laboratory of KAIST for supporting various mobile devices. We also thank anonymous reviewers their insightful comments suggestions.
References
[1] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforeement learning. In In International Conference Learning Repre- sentations (ICLR), 2017.
[2] Maxim Berman, Leonid Pishchulin, Ning Xu, Matthew B Blaschko. and Gérard Medioni. Aows: Adaptive and optimal network width search with latency constraints. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11217-11226, 2020.
[3] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
[4] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URL hattps://arkeei org/ /ptp///f////// 1908. 09791 pdf.
[5] Xiangning Chen, Ruochen Wang. Minhao Cheng. Xiaocheng Tang, and Cho-Jui Hsich. Dr(nas): Dirichlet neural architecture search. In International Conference Learning Representations, 2021.
[6] Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, et Chamnet: Towards efficient network design through platform-aware model adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1
2019.
[7] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[8] Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference on computer vision and pattern recognition (CVPR), 2019.
[9] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
[10] Eukasz Dudziak, Thomas Chau, Mohamed S Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D Lane. Brp-nas: Prediction-basse nas using gens. Advances neural information processing systems (NeurIPS), 2020.
[11] Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2020.
[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning (ICML), 2017.
[13] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiet Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the JEFF/CVF International Conference on Computer Vision pages 1314-1324, 2019.
[14] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gareelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. In International Conference Learning Representations, 2019.
[15] Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of- distribution tasks. In International Conference on Learning Representations (ICLR), 2020.
11

---

[16] Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learn- ing to generate graphs from datasets. In International Conference on Learning Representations. 2021.
[17] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[18] Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. Hw-nas-bench: Hardware-aware neural architecture search benchmark. In International Conference on Learning Representations, 2021.
[19] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 09835, 2017.
[20] Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations (ICLR), 2019.
[21] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference Computer Vision (ECCV), 2018.
[22] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In In International Conference on Learning Representations (ICLR), 2019.
[23] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Liu. Neural architecture optimiza- tion. In Advances in neural information processing systems (NeurIPS), 2018.
[24] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint arXiv: 10XX:v:8003000 00099, 2018
[25] Hicu Pham, Melody Y Guan, Barret Zoph. Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICML), 2018
[26] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[27] Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[28] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NIPS), 2017.
[29] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828. 2019.
[30] Sebastian Thrun and Lorien Pratt, editors. Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
[31] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems (NIPS), 2016.
[32] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12965-12974, 2020.
[33] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-awarr transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
12

---

[34] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[35] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less atten- tion with lightweight and dynamic convolutions. In International Conference Learning Representations (ICLR), 2019.
[36] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-effciiett architecture search. In International Conference on Learning Representations (ICLR), 2020.
[37] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Bowen Shi, Qi Tian, and Hongkai Xiong. Latency-awen differentiable neural architecture search. arXiv preprint arXiv:2001.06392. 2020.
[38] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In European Conference on Computer Vision, pages 702-717. Springer, 2020.
[39] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In In International Conference Learning Representations (ICLR), 2017.
13


Reviews:
[{"review_id": "p5HJQLnHEZY", "reviewer": "Reviewer_RCcn", "summary": "Modern neural architecture search methods often require us to train a *latency prediction model* which can predict the latency of any candidate network architecture within a human-defined search space. These are commonly obtained by (i) sampling a set of random network architectures from the search space, (ii) benchmarking each one on the target inference hardware device, and (iii) training a regression model on these <architecture, latency> pairs in order to generalize to new, unseen architectures. The submission explores the idea of (i) training a single model which can generate predictions for many different hardware devices, and (ii) using meta-learning to allow the model to generalize to a new hardware device with little additional data (e.g., 10 samples).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Note**: I'll focus on the Neural Architecture Search / latency aspects of the submission (rather than the meta-learning aspects), since that's what I'm most familiar with.\n\n# Overview\nThe paper's central claim is that meta-learning can be used to train a latency prediction model which can generalize to new (unseen) hardware devices with minimal training data (e.g., benchmarks of 10 random architectures from a large search space). The main steps are to (i) select a \"seed\" set of hardware devices, (ii) collect benchmark latencies for ~900 - 4000 random architectures for each device, and (iii) use the resulting <architecture, device, latency> triples to train a model which can predict latencies for architectures and hardware devices not seen in the training dataset (Section 4).\n\nOne of the inputs to the model is a representation of the target hardware device. This is generated in an interesting/clever way. The authors propose sampling/fixing 10 random architectures from the search space, and using the vector `<latency of architecture 1, latency of architecture 2, ...>` as their representation of the hardware device (Section 3.2). The submission proposes a \"z-modulator\" (weight adaptation mechanism) to adjust the network weights for a target hardware device.\n\nThe submission provides two sets of evaluations to test the proposed method. First (Figure 3, Figure 4, Figure 5, Table 2): evaluating how well the method performs on new hardware devices that were not seen during training, and comparing against a set of standard baselines (e.g. using FLOPS instead of on-device performance, comparing against a previously proposed method called BRP-NAS which trains a separate latency predictor for each device). In the authors' experiments, the newly proposed method significantly outperforms these baselines. And second (Table 3, Table 4, Table 5, Figure 6): plugging the proposed latency estimator into previously proposed NAS algorithms (e.g., Once-for-All, MetaD2A) and measuring the end-to-end quality impact on NASBench-201 or the Once-for-All search space. Both sets of experiments provide evidence that the latency predictor, once trained, can generalize reasonably well to new hardware devices with only ~10 benchmark samples.\n\n**Clarity**: The submission seems clear and well-written.\n\n**Significance**: The problem of trying to reduce the cost of training latency models for new hardware-devices seem well-motivated, and this is the first paper I've seen which tries to directly address this problem. It also seems like a nice application of ideas from few-shot learning. The results might be useful for practitioners who wish to simultaneously target a large number of different hardware devices.\n\n**Incorrect Use of Big-O Notation**: The submission makes the following claim on Line 209 and Table 1 (\"cost of NAS and latency estimation\"):  \"Since conventional approaches require to collect a large number of samples and train a device-specific latency estimator for each target device, while HELP only needs to collect 10 samples per device, HELP reduces the time complexity of obtaining latency estimations on N devices from O(N) to O(1).\"\n\nThis seems like an incorrect use of big-O notation. The cost of training models for N hardware devices is reduced from `900*N` to `10*N+C`, where C is the cost of training the meta-model. Both of these costs are technically `O(N)`, since the cost grows linearly with the number of target devices as N goes to infinity. **Because the claims about big-O complexity are technically incorrect, I would not feel comfortable accepting the paper until the issue is addressed, or until the authors clarify their meaning (if I misunderstood their claims).**\n\n**Caveats (different devices in same family may be strongly correlated)**: While the ability to adapt to a new hardware device with only 10 samples is impressive, it's possible that this is possible because many of the devices in the submission's pool are quite similar to each other. For example, it might be reasonable to expect Intel Xeon Silver and Intel Xeon Gold CPUs to have strongly correlated performance. This is significant because (i) it's not obvious to me that we actually need separate neural network architectures for the two CPUs, and (ii) it's possible that we could get an excellent predictor for Intel Xeon Gold by taking latency predictions for Xeon Silver and rescaling them a bit, without any kind of fancy meta-learning tricks.\n\nOverall, the target devices the authors evaluate can be broken down into the following categories (Appendix A.2):\n1. Desktop GPU (5 devices, all made by NVIDIA).\n2. Edge GPU (1 device, made by NVIDIA).\n3. Server CPU (3 devices, all made by Intel).\n4. Mobile Phone CPU (5 devices, made by Samsung and Qualcomm) \n5. Raspberry Pi (1 device).\n6. ASIC (1 device, made by Eyeriss).\n7. FPGA (1 device).\n\nI have mixed feelings about this dataset. On the positive side, it covers a diverse set of hardware devices of different types/sizes. On the negative side, if we split the list of devices randomly, we might end up with an overly optimistic estimate of the model's performance. Given a random split, with high probability, at least one mobile desktop GPU will end up in the training set and another very similar device will end up in the test set.\n\nFor this reason, I think that evaluating the latency predictor's ability to generalize across device families is importance. The submission takes a first step towards addressing this in Table 5, where the meta-model is trained only on CPUs but is evaluated on GPUs + Intel CPUs. This suggests that in terms of end-to-end performance (plugging in the latency predictor to a previously proposed NAS algorithm), the current method gets reasonable generalization across hardware families. But it would be helpful to see a more careful analysis of the predictor's raw performance as well.\n\n# Additional Detailed Comments\nFigure 3: Each line shows a shaded error region, but I didn't notice an explanation for how these shaded regions were derived (e.g., worst run vs. 1 standard deviation below the mean).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors' discussion of limitations and societal impact in Appendix D adequately addresses the NeurIPS review requirements.\n\nI would be interested in seeing a more detailed study of the meta-model's generalization performance within vs. across hardware families, but I think the submission already has enough interesting results for me to recommend its acceptance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SXgTt6FbqLH", "reviewer": "Reviewer_rNFs", "summary": "This paper aims to improve the estimation of latency of neural models on unseen hardware devices. Its contributions are:\n- Utilize a collected pool of devices and architectures to meta-train the predictor.\n- Adapt the predictor to unseen hardware devices/platforms by considering hardware device embedding -- the latencies of the unseen devices on a fixed set of reference neural architectures.\n- Introduce a \"hardware-adaptive modulator\" to improve the prediction.\n- Reduce the latency estimator building cost of N devices from O(N) to O(1).\n- Apply the latency estimator to latency-constrained NAS.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe contributions of this paper are quite outstanding:\n- Improve SOTA latency predictor by formulating it as a few-shot regression problem.\n- Use latencies on reference architectures as hardware embeddings.\n- Combine with exisiting NAS framework to accelerate latency-constrained NAS.\n- Good combination of existing techniques: GCN and MLP to encode architectures and devices, meta learning to adapt the predictor to unseen devices.\n- Improve number of samples required to adapt to an unseen device.\n\n**Quality**\n- The proposed methods are sound and convincing.\n- The empirical results support the arguments.\n- The meta-learning approach looks simple and effective.\n- The hardware-adaptive modulator needs more explanation.\n\n**Clarity**\nSome explanations in the paper are not clear. I hope the author can address the problems and make it a solid paper.\n- Table 2: what are HW-condition, few-shot adapt and z modulator? It will be clearer to relate to the sections explicitly.\n- Hardware-adaptive modulator seems to be an important contribution. However the explanation is not very clear -- How is the hardware-adaptive modulator be designed, implementated and trained?\n- Why is the latency model builing time so long for ProxylessNAS and OFA-layerwise? They use layer-wise predictor which are the summation of profiled per-layer latency. Should be fast.\n- Check mathematical notations. For example: f vs f^{\\tau}, V_h^{(0)} -- what is (0)? 2) \n- Figure 3 and 5: What is the unseen device? How to select the device in the pool for each point in the x-axis? \n\n**Significance**\n- The paper has reduced the latency estimator building cost by meta-learning a few-shot regression model that can generalize across hardware devices.\n- The source code is available to reproduce the results. It would be more helpful if the dataset (latency measurement from the device pool) is also available.\n\n**Post-rebuttal updates**\nThank you to the authors for answering my questions and clarifying the technical details. I am convinced that meta learning + hardware performance predictor is a solid contribution. I will increase my score to accept this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GffTkqyHESA", "reviewer": "Reviewer_TRAs", "summary": "This work proposes a method which is able to learn a hardware performance predictor for a wide range of devices with only a few training samples(such as 10), through first one-time meta-learning a hardware embedding with the samples from a large device-architecture-performance pool and then adapt to the seen/unseen devices through inner-gradient manner with only a few training samples; throughout the process, this work treats the hardware devices as black boxes outputing the performance metric, thus not holding the statistical assumption about hardware devices' performance.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this work \n1. Proposes a good method to predict the hardware performance of many devices while only requiring a few training samples without the statistical assumption between the hardware devices and their performance. \n2.This work will certainly expedite and facilitate the process of hardware-aware NAS. \n3.The writing is clear and to the points, pleasant to read\n\nHowever, I have a couple of concerns which would be great if the authors can address them.\n\n1. Algorithm-task level generality: It would be better to show some end-to-end results on more datasets/tasks, or model types(if possible), e.g., GNN, transformers...\n\n2. More device variety and metric types: Different devices may have different calculation process for some performance metric, like throughput. For latency metric, most of the devices have a accumulative relation between layer(component)-wise latency and the system-latency. However, for throughput on architectures like cell-phones, Eyeriss or the FPGA in HW-NAS-Bench, the final throughput equals the inverse of system-latency, as they execute the workload layers sequentially. However, some architectures, such as the one in DNNBuilder,  execute the workload layers in a pipelined fashion. Thus the throughput will instead be the inverse of the bottleneck-layer latency. It would great to show some results/insights on how meta-learning on one type of devices generalize to the other type.\n\nDNNBuilder: an Automated Tool for Building High-Performance DNN Hardware Accelerators for FPGAs", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, it's addressed", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "0IjqrMG-zET", "reviewer": "Reviewer_FZ6V", "summary": "This paper presents a hardware latency modeling and prediction methodology that adapts itself to multiple devices through few shot meta-learning. The overall goal of the paper is to create a hardware predictor for use within hardware-aware neural architecture search (NAS). \n\nA graph convolutional network (GCN) is first trained using latency measurements from a number of devices (18) then few-shot learning (with as little as 10 measurements) is done to adapt the predictor to predict the latency of unseed devices (3 devices are similar to the devices used for training, and 3 devices are different). The problem addressed with this work is important as it is often impractical to get numerous measurements from real hardware devices. Few-shot metalearning is one way to adapt a latency predictor for a new device and this is the first time I have seen metalearning applied to the specific problem of hardware latency for different devices. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the results in the paper are strongly in favour of the proposed methodology compared to prior work, there are many questionable assumptions in those comparisons and missing ablations that need to be addressed to prove the efficacy of the proposed meta-learning approach. I am strongly in favour of seeing this work published but I find it hard to give a high score without addressing the major issues listed below.\n\n### Strengths:\nThe topic is important and interesting and the use of meta-learning is spot-on. This is a great way to formulate the hardware latency prediction problem and provide a low-cost way to adapt to different devices.\n\n### Major issues:\n- Without code release, it is difficult to trust the results.\n- Choice of \"unseen device\" and \"unseen platform\" seems to be hand-picked for good results. An ablation would really help in understanding the pros/cons of using HELP. For example, what happens with fewer train devices? What happens when you swap train and test devices? What happens when your \"completely unseen platforms\" more complex? Latency prediction for ASIC, FPGA and MCU devices is *very* simple and could usually be modeled analytically by counting the number of compute cycles. Additionally, the \"unseed devices\" CPU/GPU/Pixel are **very** similar to devices in the meta-train pool for which many training samples are provided.\n- Some results are overstated. When comparing the number of samples used to train each predictor, all training points should be reported in some way. For HELP in Table 3 for example, it only has '10' measurements needed but the actual number is 900*18+10=16210. I do understand that most of this is a fixed cost that is not incurred with each new device but it still needs to be clearly spelled out -- I found the current presentation of results slightly misleading in some places.\n- Again in Table 3, it is difficult to discern **why** HELP is better than BRP-NAS? Both predictors are based on GCNs. The only reason I can think of has to be related to the fact that HELP is trained on more data points (18 devices * 900 points) and so that data allows it to achieve a better prediction. I would've expected HELP to be an *approximation* of BRP-NAS so to speak. A discussion of the reason behind this improvement is warranted.\n\n### Minor issues:\n- Equation 3 is odd. It combines set notation and function notation. It is unclear to me what it is saying.\n- Why are some results missing? For example in Table 3 the layerwise predictor is not present for 4 devices.\n- In Table 3, why is the FLOPs correlation coefficient so high for the GPU when Fig. 4(a) suggests that it should be very low?\n- I don't view this as the main contribution of the work, but in the comparison to baseline metalearning approaches like meta-SGD in Fig. 5, the baseline seem to be completely failing - is it just untuned? or is this a trivial baseline? Or is the fact that you use a modulator making all the difference? A discussion on this would help.\n- Table 1 is misleading in my opinion.\n\n## Post-rebuttal\nAll of my concerns were addressed - please see the conversation below with the authors. I will increase my score to reflect that I am satisfied with their responses. As mentioned below: With the code release and additional ablations (especially ones that show that this method generalizes with different devices in test vs. train pool), I would like to vote for accepting this paper. I think it is a solid contribution that straddles the boundary between two important areas: metalearning and hardware device modeling. I would suggest adding many of the ablations in this rebuttal to the submitted paper (or its appendix).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitations of the paper are stated in the main review. I do not have comments on the societal impact as this work is quite technical and abstracted away from the important societal issues in the machine learning community.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a timely paper. It addresses a relevant problem using novel meta-learning methods and obtains strong results.
All reviewers were in favour of acceptance.

There was a long internal discussion with dozens of posts about the comparison with BRP-NAS. The authors first misleadingly claimed that the results shown in the paper are consistent with the ones in the BRP-NAS paper, but they are clearly not (with a huge difference in Spearman correlation of roughly 0.8 vs. 0.99!). During the review process, the authors' code for running BRP-NAS was checked and found to be fine, so that is not the reason for the performance difference. 
The most likely explanations for the different performance of BRP-NAS are (1) The data may be measured differently and/or (2) BRP-NAS may need different hyperparameters for new data. Concerning the latency measurement methodology, the BRP-NAS paper described their detailed methodology for measurements in detail: they pruned NAS-Bench-201 graphs and optimized them before running them on the device, then discarded the first few measurements and averaged multiple runs. The current paper doesn't give full details about the measurement pipeline to the paper, and I strongly encourage the authors to add more details. In the further discussion, the authors indeed explained that they use a less controlled (and arguably, more realistic) noise than in the BRP-NAS paper, which reduces the predictability. 

The authors also added a convincing comparison to BRP-NAS on BRP-NAS' own benchmark LatBench.

An orthogonal concern that was brought up in the reviewer discussion was that Table 5 is overclaiming: OFA + HELP obviously does *not* take a total time of 26 seconds, as OFA takes very substantial time to construct the one-shot model. I strongly encourage the authors to fix this by reporting the total time spent, in order to present their contribution realistically.
Another concern that came up was that authors were given an additional page this year and encouraged to use this page to discuss the broader impact and societal impact; the authors instead deferred that discussion to the appendix, which could be seen as cheating the page limit. I strongly encourage them to include a discussion in the main paper for the final version and move some less important details to the appendix instead.

Overall, the work addresses an important and relevant problem; the use of meta-learning for NAS is much underexplored and thus very timely. I am therefore clearly in favour of acceptance.
Decision: Accept (Spotlight)