NAHAS: Neural Architecture and Hardware Accelerator Search
Authors: Yanqi Zhou
Xuanyi Dong
Daiyi Peng
Ethan Zhu
Amir Yazdanbakhsh
Berkin Akin
Mingxing Tan
James Laudon
Venue:  (2021)
Keywords: neural architecture search
systems
hardware

Abstract: Neural architectures and hardware accelerators have been two driving forces for the rapid progress in deep learning.
Although previous works have optimized either neural architectures given fixed hardware, or hardware given fixed neural architectures, none has considered optimizing them jointly. In this paper, we study the importance of co-designing neural architectures and hardware accelerators. To this end, we propose NAHAS, an automated hardware design paradigm that jointly searches for the best configuration for both neural architecture and accelerator. In NAHAS, accelerator hardware design is conditioned on the dynamically explored neural networks for the targeted application, instead of fixed architectures, thus providing better performance opportunities. Our experiments with an industry-standard edge accelerator show that NAHAS consistently outperforms previous platform-aware neural architecture search and state-of-the-art EfficientNet on all latency targets by 0.5% - 1% ImageNet top-1 accuracy, while reducing latency by about 20%. Joint optimization reduces the search samples by 2x and reduces the latency constraint violations from 3 violations to 1 violation per 4 searches, compared to independently optimizing the two sub spaces.

Under review as a conference paper at ICLR 2021
NAHAS: NEURAL ARCHITECTURE AND HARDWARE ACCELERATOR SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Neural architectures and hardware accelerators have been two driving forces for the rapid progress deep learning. Although previous works have optimized either neural architectures given fixed hardware, or hardware given fixed neural architectures, none has considered optimizing them jointly. In this paper, we study the importance of co-designing neural architectures and hardware accelerators. To this end, we propose NAHAS, an automated hardware design paradigm that jointly searches for the best configuration for both neural architecture and accelerator. In NAHAS, accelerator hardware design is conditioned on the dynamically explored neural networks for the targeted application, instead fixed architectures, thus providing better performance opportunities. Our experiments with an industry-standia edge accelerator show that NAHAS consistently outperforms previous platform-aware neural architecture search and state-of-the- art EfficientNet on latency targets by 0.5% 1% ImageNet top-1 accuracy, while reducing latency by about 20%. Joint optimization reduces the search samples by 2x and reduces the latency constraint violations from 3 violations to 1 violation per 4 searches, compared to independently optimizing the two sub spaces.
INTRODUCTION
Conventional hardware design has been driven by benchmarks (e.g. SPEC (PE)) where selected set of workloads are evaluated and the average performance is optimized. For example, CPUs are optimized for sequential workloads such as desktop applications and GPUs are designed for massive parallel workloads such as gaming. graphics rendering, scientific computing, etc. Generalization over wide set of representative workloads is the traditional method for hardware design and optimization. However, the selected workloads can stay fixed for a substantially long time, which makes the hardware design lag behind the algorithmic changes.
As result of hitting the end of Moore's 's Law in the recent decade, the focus has switched to hardware specialization provide additional speedups and efficiency for narrowed-down application domain. Google's TPU (TPU) and Intel's Nirvana NNP (Yang) (219)) are two representative accelerators specialized for deep learning primitives and MLPerf (MLP) has become prevalent for benchmarking the state-of-the-art design of ML accelerators. However, rapid progress in deep learning has given birth numerous more powerful, expressive, and efficient models in a short time, which results in both benchmarking and accelerator development lagging behind. For example, squeeze-and and-ex with global pooling and SiLU/Swish non-linearity (Ramachandran et al. [017] Elfwing et al. 2018)) 8) are found to be useful in EfficientNet (Tan & Le) (019), however, neither can currently execute efficiently even on a highly specialized accelerator. We need evolve the accelerator design more rapidly.
the other hand, platform-aware neural architecture search TTan et al. 2019 Wu et al. 2019 Cai et al.] 2018) optimizes the neural architectures for target inference device. The target device has a fixed hardware configuration that can significantly limit NAS flexibility and performance. For example, the target device may have sub-optimal compute and memory ratio for the target application combined with a inference latency target, which can shift the optimal NAS model distributions and result in underperformance.

---

Under review as a conference paper at ICLR 2021
We propose NAHAS, new paradigm of software and hardware co-design, by parameterizing neural architecture search and hardware accelerator search in a unified joint search space. We use a highly parameterized mudustry-standan ML accelerator our target device, which has tunable set of important hardware parameters. These knobs fundamentally determine hardware characteristics such number of compute units, amount of parallelism, compute to memory ratio, bandwidth, etc., which we found very critical to model performance. We formulate the optimization problem as bi-level optimization with hardware resource constraints on chip area and model latency.
Unlike conventional hardware optimization, NAHAS is a lask driven approach, where the task problem (e.g. image classification, object detection) or a domain of problems (e.g. vision, NLP), not set of fixed programs or graphs (e.g. ResNet, Transformers). This effectively creates generalization across the vertical stack, making the hardware evolve with the applications. NAHAS can be practically used to design customized accelerators for autonomous driving and mobile SoC (system-on-chip) where a set of highly optimized accelerators are combined into a system.
We also propose latency-driven optimization that maximizes model accuracy while meeting latency constraint under a chip area budget. Conventional platform-aware NAS typically focuses on searching efficient NAS models with higher accuracy, lower parameters and FLOPs (number of multiply-add operations). However, optimizing for lower parameters and FLOPs is not necessarily good for performance (Wu et al. 2019). For example, multi-branch network like NasNet (Zoph) et al. 2017) has lower parameters and FLOPPs compared to layer-wise network such as ResNet and MobileNet, but its fragmented cell-level structure is not hardware friendly and can be evaluated very slowly on the target device. As a matter of fact, the number of parameters and FLOPs are only indirect metrics that could affect a direct metric such latency and power negatively if not considered. For example, the model can run out of memory if the number of parameters is too large. However, a direct optimization on indirect metrics won't necessarily improve the direct metrics. Aocotorator Confguration
Acosleratoor corfigt
Accelerator config1
Latency
Latency
model
Accelorator config2
Accelaritor
Controler
Traine
NAS model 2
Accelerator vonfig3
Acouracy
Latency
NAS model 3
Joint search
Corfigunntion
Reward
Engane
Error
Error
(a) NAHAS Extends the Pareto Frontier
(b) NAHAS Work Flow
Motivating example (a) and high level work flow (b). In (a) different accelerator configurations have different pareto frontiers consisting of different NAS models (left) and joint search effectively extends the pareto frontier by joining multiple frontiers (right).
Figure 1:
Figure shows a motivating example of using joint search and high level workflow of NAHAS. While conventional platform-aware NAS selects models along the pareto frontier with different latency and accuracy tradeoff's for one target device, as indicated in Figure Ta left, NAHAS further expands the pareto frontier by enabling different hardware accelerator configurations. To summarize our contributions:
We develop a fully automated framework that can jointly optimize neural architectures and hardware accelerators. For the first time, we demonstrate the effectiveness of co- optimizing neural architecture search with the parameterization of a highly optimized industry-standard accelerator.
We propose a drivee search method, which is aandware-agnosti and achieves state-of-the-an results across multiple search spaces. NAHAS outperforms MnasNet and EfficientNet on all latency targets by 0.5%- 1% in ImageNet accuracy and 20% in latency.
We observe that different model sizes combined with different latency targets require completely different hardware accelerator configurations. Customizing accelerators for different model sizes and latency targets becomes essential when co-designing neural architectures and accelerators for domains such autonomous driving.
2

---

Under review as a conference paper at ICLR 2021
We compare different optimization strategies and find that joint search consistently outperforms optimizing NAS and HAS in an alternating fashion. Oneshot search can significantly reduce search cost (total number of samples and search time), however, it is less suitable for large models when constructing the super network is too costly.
RELATED WORKS
ML-driven architecture search: Design space exploration in computer systems has become more
erucial due the surge of specialized hardware (Parsa et al.] [2020 Igbal ct al. [020 Sun et al. 2020
Nardi et al. 2019 Cong et al. 2018 Koeplinger et al.] 2018 Balaprakash et al. 2016 Ansel et al,
2014). Mierarchical-PABO (Parsa et al. 2020) and FlexiBO (Iqbal al., (020)
multi-objective Bayesian optimization for neural network accelerator design. In order to reduce computational cost,
(Sun al. 2020) apply genetic algorithm to design CNN models without modifying the underlying
architecture. HyperMapper (Nardi et al. 2019) uses random forest in automatic tuning of hardware
accelerator parameters multi-objective setting.
Platform-aware neural architecture search: MnasNet (Tan et al. 2019) pioncered platform-and aware neural architecture search where for the first time NAS models are tailored for target hardware device and resource-efficient models are identified. ProxylesNNAS (Cai et al. 2018) directly learns the architectures for ImageNet by proposing a gradient-based approach train binarized parameters. FBNet (Wu et al.) (211)) proposes differentiable platform-awarr NAS using Gumbel Softmax and ChamNet (Dai et 2019 proposes a neural architecture adaptation method using efficient accuracy and resource predictors. Accelerator-awe
NAS (Gupta & Akin 2020) for the first time targets cellustry-standale accelerators and identified SoTA models for an edge TPU. However, none of these work optimizes the underlying hardware accelerators together with NAS.
Co-design: There is a growing body of work exploring neural architecture search and hardware design (Jang al., 2020a Choi et al., 2020 Jiang al.] 2020b) Achararit al. 2020 Yang jet al. 2020] Kwon et al. 2018 Yang et al.) (018). However, most of the work target FPGAs or academic accelerators such Eyeriss Chen al. (2016), where the hardware is less optimized on real workloads (e.g. ImageNet) and the performance model is less accurate. NAHAS, however, targets highly optimized mndustry-standaan ML accelerator and the performance simulator is cycle- accurate that has been validated. Moreover, NAHAS demonstrates effectiveness on real ImageNet workload, outperforming SoTA models on two different search spaces.
METHOD
In this section, we will formulate the joint optimization problem of NAHAS and present two different approaches.
3.1 FORMULATION
The objective for NAHAS it to find neural architecture parameter a and hardware accelerator parameter l such that the validation accuracy on ImageNet classification task can be maximized while meeting chip area and latency target.
min L(a, Jt, w Draat s.l. w = arg min L(o,, h, War Dirain) a,
Wa
Latency(a, h)) S Tiateneyy Area(h) S Tarea-
where L indicates the objective function of the tasks (e.g., cross-entropy for classification) and w. denotes the weights of the architecture a. NAHAS introduces a broader search space than NAS (neural architecture search) or HAS (hardware accelerator search) alone, with the flexibility to fix either a or h therefore, the optimization problem reduced to NAS or HAS. We empirically compared different optimization strategies in Section 4.4
3.2 NAS SEARCH SPACE
MobileNetV22: We build the architecture search space S based on the standard MobileNetV2. The search space is tailored for mobile edge processors, therefore, consists mostly of efficient operations
3

---

Under review as a conference paper at ICLR 2021
such mobile inverted bottleneck convolution (MBConv). Specifically, search for the kernel size from {3. 5, 7} for each MBConv, and we also search for the expansion ratio from (3. 6) for each block except for the first one, which has the default expansion ratio of 1. In MobileNetV2, there are 17 inverted residual blocks, and thus the cardinality of Si is about 8.4e12.
EfficientNee:: In order create larger NAS models and to better leverage modern edge accelerators which has larger number of compute units and memory capacities, we build the architecture search space S2 based on the standard EfficientNet-BO. Similar to S1, also search for the kernel size from (3, 5, 7) and the expansion ratio from {3. 6). Since there are 16 inverted residual blocks in EfficientNet-Bee the cardinality of S2 about 1.4e12.
3.3 HAS SEARCH SPACE
The target device is an industry-standarr, highly parameterized edge accelerator which allows us to create various configurations in large design space with tradeoffis between performance, power, area, and cost. The accelerator features set of parallel processing elements (PE) organized in a 2D tile. The number of PEs in each dimension determines the aspect ratio of the chip. each PE there are multiple compute lanes that shares local memory and each lane has register file and series of single-instruccio multiple-data (SIMD) style multiply-accummn
(MAC) compute units. Each of these architecture components provide degree of parallelism and corresponding area cost. With a fixed chip area budget, HAS optimizes the on-chip resource allocation, balancing the compute and memory and searching for the best chip parameterization for given application.
Table Edge Accelerator Search Space
parameters
type
search space
parameters
type
search space
PEs in_x dimension
int
1, 2, 4, 6, 8
local momlmmmmrr.... MB int
0.5, 1, 2, 3,
PEs in y dimension
int
1, 4, 6,
compute lanes
int
1, 2, 4, 8
SIMD_units
int
32, 64, 128
ho_bandwidth ghps float
5, 10, 15, 20, 25
.BB
int 8. 16, 32, 64, 128
As baseline we take the default accelerator configuration which is optimized while considering series of production workloads from multiple domains. The baseline configuration features 4x4 PEs where each PE has 2 MB local memory and compute lanes. Each compute lane has 32 KB register file and 64 4-way SIMD units. Since the accelerator is targeted for edge use cases, SIMD units sustain the peak throughput for 8- quantized operations. This baseline configuration can deliver a peak throughput of 26 TOPS/s at 0.8 GHz.
Unlike the NAS search space, the HAS search space contains many invalid points, which makes training a cost model or joint search with the in-house simulator more challenging. Invalid configurations can be caused by many reasons. For example, the created accelerator configuration in combined with the NAS model is not supported by the compiler or a NAS model is created too large for the generated HAS configuration, etc.
3.4 SEARCH OBJECTIVE
While power and area can be other important metrics to be considered for efficient accelerator design, we focus on maximizing model accuracy while meeting an inference latency constraint target device. We also impose a chip area constraint, which is set equal to the baseline accelerator design. The paper does not minimize chip area or latency, as in practise they don't need to be minimized. However, the chip resource constraint can be set to area, power, energy, or a combination of many, however, this paper focuses on an area constraint only.
(Tan et al. 2019), we use customized weighted product to encourage Pareto optimal solutions. More specifically, we have optimization metric of model accuracy and two hardware constrain metrics of model latency and chip area. The optimization goal is maximize model accuracy while meeting the latency and area constraints.
Similar to Mnasnet
Latency(a, h) Area(h) pn
max Accuracyyya x a,
wwo x
T Tiatency
Tarea
4

---

Under review as a conference paper at ICLR 2021
where wo, wi are the weight factors:
p. if Latency(a, h) &iatency
Taatnnyy
p. if Area(h) 1 Tarea
wi 9. otherwise
g. otherwise
When p 0,g -, the reward function imposes hard latency constraint and simply use accuracy as the objective if the measured latency meeting the latency target and only sharply penalize the objective value if the sample violates the latency constraint. When p q -0.0% the reward becomes a soft constraint function. the NAHAS evaluations, we use both rewards for different experiments.
3.5
OPTIMIZING THE JOINT SEARCH SPACE WITHOUT WEIGHT SHARING
PPO (Schulman (et al., (007)) as the controller algorithm to optimize the joint search space from NAS and HAS. The controller samples the search space using a recurrent network, each sample is trained by child program. For the MobileNetV2 search space, we use proxy task that trains each sample on ImageNet for only 5 epochs and it takes 5000 samples for the controller converge. For larger models using EfficientNet search space, we find that training the proxy task for 15 epochs while reducing the total number of samples 2000 improves the results.
Similarly NASNet ZZoph al. 2017) and MNASNet (Tan et al. 2019),
3.6 OPTIMIZING THE JOINT SEARCH SPACE WITH WEIGHT SHARING
To further reduce the search cost, employ efficient search method with weight sharing- Similarly as ProxylessNAS (Cai al 2018) and TuNAS (Bender et al.) (020). we use the controller decisions from the NAS space construct a super-network for optimizing the architecture, meanwhile using the decisions from the HAS space to create a sub-graph for computing the cost. Decision points from both spaces are optimized by RL algorithm within the same graph. For each training step, we train the model weights and the controller decision points in an interleaved way. To achieve better results, we apply the absolute reward function and RL warm-up procedure introduced in TuNAS.
To estimate the latency of the model on a given accelerator configuration, we train cost model with random generated samples using in-house accelerator simulator. We need a cost model because as NAS becomes much faster with oneshot search, the query to the accelerator performance simulator for chip area and inference latency becomes the new bottleneck for NAHAS oneshot search. Given sampled neural architecture configuration and an accelerator configuration, the cost model predicts (h) and model accuracy fi h). We
the accelerator encapsulate the non-linearity in the latency prediction. The area predictor and latency predictor largely share parameters with only separate parameterization in the prediction heads.
MLP network with ReLU to
Loss = MSE fa(h) + AMSE(L,
The cost model was trained with 500k labeled data randomly generated by permuting the neural architecture configurations and accelerator configurations. We use a 3-layer MLP of hidden size 256 and apply a dropout of 0.1 to mitigate overfitting at each layer.
4
EVALUATION
4.1 EXPERIMENT SETUP
Accelerator Performance Simulator: Evaluating the candidate neural and hardware architectures accurately is a key requirement for the NAHAS framework. Simply using number of MACs or parameters of the neural model as proxy for performance can be highly inaccurate their performance highly depends on how the neural network is mapped on the hardware architecture and their unique compute characteristics. To this end, have utilized in-house cycle-accurate performance simulator and an analytical area model based on hardware synthesis. We deployed
Tan et al. 2019) found -0.07 to empirically ensure Pareto-optimal solutions have similar reward under different accuracy-latene trade-offs.
5

---

Under review as a conference paper at ICLR 2021
both of these estimators as service where multiple NAHAS clients send parallel requests. This provides a flexible way to scale-up the performance and area evaluation tasks.
Search Hyperparameters: For the end2end search, choose PPO it is tested by time. We use the average performance of 10 trials as a reward. We use Adam optimizer with the learning rate of 0.0005 update the controller, where the policy gradients are clipped by 1.0. For each trial, we train the sampled candidate by five epochs using RMSProp. For these five epochs, we will first warm up the model by two epochs using the learning rate from 0.66, and then cosine decay from 0.66 to 0 for the rest three epochs. For the oneshot search, we utilize REINFORCE to optimize the controller following TuNAS. We use Adam with a learning rate of 0.0048 to optimize it and use the momentum as 0.95 for baseline. In addition, we use RMSProp to optimize the shared weights following the same learning rate schedule as TuNAS. The latency predictor is pre-trained.
Cost Model Hyperparameters: The cost model described in Section 3.6 was trained with hyperparameters defined Table the oneshot search, we replace the accelerator performance simulator with the trained cost model and search for the best model for five different latency targets (0.3ms, 0.5ms, 0.8ms, I. and 1.3ms). The average error between the latency target and the estimated latency of the best model using the accelerator performance simulator is only 0.4%.
1.4
Latercy target
1.2
Estimated lntency of the best model
Optimizer
Adam
Loss Re-weight
10
Latency (mr)
Batch size
128
0.8
Hidden dimension
256
0.6
Learning rate
0.001
0.4
Training steps
600k
Input feature size
394
0.2
Latency Targat
Figure 2: Cost Model Accuracy.
Table 2: Cost Model Hyperparameters.
4.2 SAMPLE DISTRIBUTIONS
To study sample distribution during search, we first compare our NAHAS search with previous platforn-aware NAS, as shown in Figure 3al and Figure 3b Search is performed two search spaces: EfficientNet-B
(relatively small) and EfficientNet-Bl (relatively large). platform-awan NAS, the target device is fixed to the bascline accelerator design, described in Section 3.3 whereas in NAHAS search, the target chip area is set to the same as the target device of platform- aware search. Latency targets without the flexibility to change the hardware configuration, platform- sub-optimal solutions of either higher latency or lower accuracy. For NAHAS, not all the samples traversed meet the chip area constraint. However, traversing through samples violating the resource constraints (red points in the figure) can help converge to more pareto-optimal samples eventually with both higher accuracy and lower inference latency.
set of 1ms and 2ms for these two search spaces. We observe that
NAS always converges
Figure 3c and Figure 3d also compare two variants of NAHAS: joint search and phase search. In joint NAHAS search, the two search spaces (NAS and HAS) are unified and a sample is taken from the joined search spaces. However in a phase NAHAS search, we fix one search space of the two while searching the other, aiming for similar boost in optimization performance similar to using coordinate descent, Empirically, we observe that with the same number of samples, NAHAS joint search consistently outperforms NAHAS phase search, producing models with higher accuracy and lower latency. It is possible that fixing one search space results in a loss surface with more saddle points such that the optimization algorithms can easily get stuck.
In summary, Figure 3 demonstrates the intuition of the selection of a NAHAS joint search approach.
4.3 NAHAS SEARCH RESULTS
We compare NAHAS with MnasNet, Fixed hardware NAS (NAHAS with default accelerator configuration). and EfficiemtNetBO-B2. by scaling the latency targets from 0.3ms (on the smaller MobileNetV2 search space) to 2.0ms (on the larger EfficientNet search space), and find that NAHAS

---

Under review as a conference paper at ICLR 2021
0.72
0.72
15-epoch Accuracy
0.70
13-egoch Accuracy
0.70
0.65
NAHAS
NAHAS Violates Area
NAHAS Violates Area
Platform amare
Plaffrr--aoe aware
1.25
1.50
1:75
2,00
2.25
2.50
2.0
2.5
3.0
3.5
4.0
(ms)
Latency (ms)
(a) NAHAS and Platform-awes aware NAS Efficienttet-Bi
(b) NAHAS and Platform-awas NAS Efficientnet-Bi
0.72
15-epoch Accuracy
0.70
15-epoch Accuracy
0.70
0.68
0.66
0.68
Search
Natformmemaee
0.64
Platfrm aware
NAHAS Phase Search
NAHAS Plase Search
0.62
1.25
1.50
1.75
2.00
2.25
2.50
2.0
2.5
3.0
3.5
4.0
Latency (ms)
Latency (ms)
NAHAS Joinst Search and NAHAS Phase Search on Efficientnet-B BO
(d) NAHAS Joinst Search and NAHAS
Phase Search on Efficientnet-B
Figure 3: Searched sample distributions comparisons.
Duro
NAHAS MNASNet
Fixed Hardlware
EfficientNet
0.77
Irogetee
0.76
MAHAS-enoZend
NAIIAS oneshot
0.75
Samples
Phase NAHAS 1x Samples
0.74
0.5
3.0
155
2.0
25
05
1.0
15
2.0
Laboncy imst
Licancy (us)
(a) End-to-end NAHAS
(b) NAHAS Variants
Figure ImageNet Top1 Accuracy Inference Latency.
about 0.5% 1% better in accuracy every latency target, shown in Figure 4a we learnt from this experiments include:
The lessons
Changing the hardware accelerator configurations helps identify higher accuracy models that cannot be identified by fixed hardware NAS.
Larger models (e.g. EfficientNetB2) requires higher memory-to-computt ratio in the accelerator design, compared to smaller models (e.g. MobileNetV2 or ProxylessNAS).
Customizing the hardware accelerator for different model sizes and latency targets is beneficial model performance and hardware efficiency.
In addition to the accuracy gains, we also observe latency reduction and chip area reduction when co-optimizing the neural architectures and hardware accelerators, shown in Table
Oneshot search is more effective than end-to-end search for smaller models with lower than Ims latency. NAHAS-oneshol can identify models 2% better in accuracy than MnasNet with lower latency.
NAHAS identify accelerator configurations of much lower area, compared to the baseline default. Apart from improving model accuracy and latency, NAHAS can be applied reduce chip area and power consumption (with different reward).

---

Under review as a conference paper at ICLR 2021
Phase-NAHAS performance depends the initial fixed NAS configuration and often lead to constraint violation. For example, Phase-NAHAS identifies models 1% higher accuracy than EfficientNetBO, however, it violates the latency constraint.
Table Comparison accuracy v.s. latency with previous approaches. Models grouped different regimes and sorted by accuracy. the latency regime, NAHAS-oneshor the best performance, while on the high accuracy regime, NAHAS-endrn end2end achieves the best performance.
Normalized chip area
Model
Top-1 Acc.
Latency (us)
MobileNetV2 Sandler et al.
2018
74.4%
0.3
1.0
Mnasnet-BB B1 ( et al. 2019
74.5%
0,41
1.0
ProxylessNASS (ar 2018
74.8%
0.42
1.0
NAHAS-end2end
74.9%
1.0
NAHAS-onesho
76.5 %
0,35
0.52
Mnasnet-D1 (Tan al. 2019
75.
0.51
1.0
NAHAS-end2Zndhh
75.1% %
0.47
0.5
NAHHAS-end2end-1
75.4%
0.48
1.0
NAHAS-end2end-2
76.1 %
0,66
1.0
Phase-NAHAS--md220n
76.4% %
0.45
1.4
NANAS-oneshot
76.8%
0.49
0.64
EfficientNet-Bi (Tan & Le 2019
76.8%
1,44
1.0
NAHAS-onesho--0
77.2%
1.09
0.78
NAHAS-oneshot-1
77.4%
1.19
0.78
NAHAS-endDent
77.4%
1.0
Phase-AAHAS-cdd2en
77.8%
1.40
1.0
EfficientNet-BI (Tan & Le 2019
78%
2.0
I.0
Phase-NAlIAS-enIZZendd
77.7%
0,93
NAHAS-end2es
78.6%
2.0
0.93
4.4 OPTIMIZATION STRATEGIES
Figure 4b compares variants of NAHAS. More particularly, we compare the end-to-end NAHAS joint search with oneshot search, and phase-basee NAHAS. In a phase-based NAHAS, we start with HAS fixed initial neural architecture in the searc-h space (Mobil-eNetV2. EfficientNet- BO, EfficientNet-Bl, and EfficientNet-B22 with a soft constraint function described in Section [.4 aiming find a accelerator configuration which is pareto-optimal in terms of latency and chip area. Then we apply NAS with hard constraint function described Section 3.4 on the selected best accelerator configuration, aiming identify good neural architectures that strictly meet the hardware latency constrain. We compare phase NAHAS with Ix and 2x total searched samples compared to the NAHAS end-to-end joint search baseline.
Comparing end-to-end with oneshot, we find that oneshot is more effective for smaller models with lower latencies where constructing super network more practical. Oneshot NAHAS reduces the search cost from 32 TPU-days to TPU-day for MobileNetV2 search space. However, oneshot is suitable for large models such as models EfficientNet search spaces.
NAHAS phase search with the same number of searched samples performs much worse than the NAHAS end-to-end joint search. Doubling the total searched samples (total search time) improves the quality of results. However, depending on the search space, latency target, and the initial accelerator configuration, performance varies significantly using NAHAS phase search.
CONCLUSION
We propose NAHAS, software/hardware co-design that jointly optimizes neural architecture search and hardware accelerator search on a industry standard ML accelerator. Jointly optimizing the application and hardware expands the Pareto frontier that enables accurate models for any given latency targets. Moreover, it enables more rapid evolution of hardware along with the software stack.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mlperf. URL https://mlperf.orr
Spec epu 2006. /Rt.ttp:://.........0////////
URL Utt.attpps///oooo.oooooe...oo
Paniti Achararit, Muhammad Abdullah Hanif, Rachmad Vidya Wicaksana Putra, Muhammad Shafique, and Yuko Hara-Azumi. Apnas: Accuncyanl-prfoomnceeawee neural architecture search for neural hardware accelerators. IEEE Access, 8: 16659999-66333, 2020.
Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan- Jeffrey Bosboom, Una-May O'Reilly, and Saman Amarasinghe. Open'Tuer: An extensible framework for program autotuning. In Proceedings of the 23rd international conference on Parallel architectures and compilation, pp. 303-316, 2014.
Prasanna Balaprakash, Ananta Tiwari, Stefan M Wild, Laura Carrington, and Paul D Hovland. AutoMOMML: Automatic multi-objective modeling with machine learning. In International Conference High Performance Computing, pp. 219-239. Springer, 2016.
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le. Can weight sharing outperform random architecture search? an investigation with tunas. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14323-14332, 2020.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. CoRR, abs/1812.00332, 2018. URL http:::/aaaxiio.rgg/bbbb////////
00332
Yu-Hsin Chen, Joel S. Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. In 43rd ACM/EEEE Annual International Symposium Computer Architecture, ISCA 2016, Seoul, South Korea, June 18-22, 2016, pp. 367-379. IEFE Computer Society, 2016. doi: 10.1 2..109/SSA.22016..4 URL
https: http://oi..oo http:///..o.o.//..1111/1
ISCA. 2016.40
Kanghyun Choi, Deokki Hong, Hojae Yoon, Joonsang Yu, Youngsok Kim, and Jinho Lee. Dance: Differentiable accelerator/network co-exploration. arXiv preprint arXiv:2009. 2020.
Jason Cong, Peng Wei, Cody Hao Yu, and Peng Zhang. Automated accelerator generation and optimization with composable, parallel and pipeline architecture. 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2018.
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, Peter Vajda, Matt Uyttendaele, and Niraj K. Jha. Chamnet: Towards efficient network design through platform-wwas aware model adaptation. CVPR, pp. 11398-11400 2019.
Stefan Elfwing, Biji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Suyog Gupta and Berkin Akin. Accelerator-aware neural network design using automl. CoRR, abs/2003.0288 028388 2020.
02838
Md Shahriar Iqbal, Jianhai Su, Lars Kotthoff, and Pooyan Jamshidi. Flexibo: Cost-aware multi- objective optimization of deep neural networks. arXiv preprint arXiv:2001.06:: 2020.
Weiwen Jiang, Lei Yang, Sakyasingha Dasgupta, Jingtong Hu, and Yiyu Shi. Standing the shoulders giants: Hardware and neural architecture co-search with hot start. arXiv preprint arXiv:2007.09087 2020a.
Weiwen Jiang, Lei Yang, Edwin H-M Sha, Qingfeng Zhuge, Shouzhen Gu. Sakyasingha Dasgupta, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. JEEE Transactions Codddder-Aided Design Integrated Circuits and Systems, 2020b.

---

Under review as a conference paper at ICLR 2021
David Koeplinger, Matthew Feldman, Raghu Prabhakar, Yaqi Zhang, Stefan Hadjis, Ruben Fiszel, Tian Zhao. Luigi Nardi, Ardavan Pedram, Christos Kozyrakis, et al. Spatial: A language and compiler for application accelerators. In Proceedings of the 39th ACM SIGPLAN Conference Programming Language Design and Implementation, pp. 296-311, 2018.
Kiseok Kwon, Alon Amid, Amir Gholami, Bichen Wu, Krste Asanovic, and Kurt Keutzer. Co- design of deep neural nets and neural net accelerators for embedded vision applications. CoRR, 005//800..10000 2018. URL http: http://awwii..o.gaaaaaa www.w////rr/b///10///// 10642
Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis and Simulation of Computer and Telecomuunicattion Systems (MASCOTS), pp. 347-358. IEEE, 2019.
Maryam Parsa, John P Mitchell, Catherine D Schuman, Robert M Patton, Thomas E Potok, and
Kaushik Roy. Bayesian multi-objective hyperparameter optimization for accurate, fast, and efficient neural network accelerator design. Frontiers in Neuroscience, 14:667, 2020.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. CoRR, abs/1710.05941, 2017. 2017. URL http: org/ attgg/ab/1711..000
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4510 4520, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs/1700.00347, 2017. URL http: org/ abs/
1707. 06347
Yanan Sun, Bing Xue, Mengjie Zhang, Gary Yen, and Jiancheng Lv. Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Transactions on Cybernetics, 2020.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICMI., 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- aware neural architecture search for mobile. CVPR, 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, June 2019.
Andrew Yang. Deep learning training scale spring crest deep learning accelerator (intel nervana TM nnp-t). In Proceedings of the Hot Chips, August 2019.
Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous asic accelerator designs targeting multiple tasks, 2020.
Yifan Yang, Qijing Huang. Bichen Wu, Tianjun Zhang. Liang Ma, Giulio Gambardella, Michacla Blott, Luciano Lavagno, Kees A. Vissers, John Wawrzynek, and Kurt Keutzer. Synetgy: Algorithm hardwarrr co-design for convnet accelerators embedded fpgas. CoRR, abs/1811.08634 2018. URL URl/http://arx....orr///////
08634
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. CoRR, abs/1707.07012 2017. URL http:://axxiv....
abs 17077771 07012
10


Reviews:
[{"review_id": "zcIcN0z4HuE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes an algorithm to search for hardware designs and neural architectures jointly.\nThe results show that joint optimization improves accuracy and latency, reduces the search samples.\n\n##########################################################################\n\nPros:\n\n- The paper introduces a new dimension, hardware design, to the neural architecture search domain.\n- The results show that joint optimization outperforms phased optimization.\n\n##########################################################################\n\nCons:\n\n- The used techniques are not novel\n- The hypothesis \"joint optimization is better than two-phased optimization\" seems obvious.\n\n##########################################################################\n\nOther comments:\n\n1. Can you describe more of the hardware. Is it a publicly accessable platform? How long does it take to switch between different configurations?\n2. Can you add wall-clock time or used computing resource in table 3?\n3. I also have concerns about the motivation. Is it typical to design an accelerator for just one network? I think an accelerator should be optimized for at least a range of neural networks.\n\ntypo: Section 3.1 \"The objective for NAHAS it to\" -> \"The objective for NAHAS is to\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "qht9SH31hml", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents NAHAS, which is a combination of Neural Architecture Search (NAS) and Hardware Architecture Search (HAS) \nfor software-hardware co-design. It uses PPO with joint search space: model accuracy and hardware constraints.\n\n##########################################################################\n\nReasons for score: \n\nThe paper claims it demonstrate effectiveness of hardware aware NAS for first time, which is dubious.\n\nThe paper should reference the industry-standard parametrized accelerator used. And reference or briefly present the in-house simulator.\n\nIn the beginning of section 3, it is unclear about the entire workflow. Explain the high-level workflow in Figure 1 caption.\n\nTested only on Imagenet, mobilnet and efficientnet. It didn't demonstrate that the method can be generalized across different the vertical stack as claimed in the introduction.\n\nIn 1. introduction, typo: Intel’s Nervana\n\n##########################################################################\n\nPros: \n\n- Demosntrated improved accuracy and latency upon other work\n\n##########################################################################\n\nCons: \n\n\n- The paper needs to improve its clarity by explaining the figure workflow in section 3.\n\n- It also needs to revise the claim that it demonstrate effectiveness of hardware aware NAS for first time. E.g: NSGA-NETmultiobjective\n\n##########################################################################\n\nQuestions during rebuttal period: \n\n \nPlease address and clarify the cons above \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_V8B21k1DkO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes NAHAS for co-designing neural network architecture and hardware architecture. It shows performance improvements on ImageNet compared to previous hardware-aware NAS methods which only optimize the neural network architectures. \n\nPros:\n1. Co-designing neural network architecture and hardware architecture is a promising and important direction. \n2. The proposed method clearly outperforms hardware-aware NAS. \n\t\nCons:\n1. The technical contribution of this paper is limited. The main difference between this paper and previous hardware-aware NAS papers is that this paper has an additional hardware search space beside the neural network architecture search space. The other components, such as the training method, search algorithm, and learning objectives, are borrowed from previous papers with minor modifications. \n\n2. Co-designing neural network architecture and hardware architecture is not new. Similar ideas have been explored in [1]. The authors should discuss the difference between this paper and [1]. \n\nIn summary, co-designing neural network architecture and hardware architecture is not new. The authors do not discuss the difference between this paper and [1]. Besides, I think the technical contribution of the proposed method is limited. Therefore, I recommend rejecting this submission. \n\n[1] Neural-Hardware Architecture Search, NeurIPS 2019 Workshop on Machine Learning for Systems.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sn5Ow78BSsH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe an automated system for co-designing neural architectures and HW accelerators. The system is able to find the best solution under latency and chip-area constraints.  A highly parameterized (commercial) edge accelerator defines the hardware search space.  Results are compared to MnasNet, platform-aware NAS and EfficientNet.\n\nThis is an interesting area and the authors demonstrate clear advantages of their approach. \n\nA claim is made that \"although previous works have optimized either neural architectures given fixed hardware, or hardware given fixed neural architectures, none has considered optimizing them jointly.\". Does the work outlined in the \"co-design\" paragraph of section 2 not attempt to do this?  Some other examples that could have been discussed include: \n\n* “FPGA/DNN co-design: An efficient design methodology for IoT intelligence on the edge”, Hao et. al. DAC’19\n* “Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator”, Abdelfattah et al, DAC’20\n* When Neural Architecture Search Meets Hardware Implementation: from Hardware Awareness to Co-Design\nhttps://ieeexplore.ieee.org/abstract/document/8839421\n\nIt is a little unclear why the papers by Jiang and Yang and the papers above are dismissed? It would be good to understand better how the author's NAS approach improves on these previous works? \n\nLimited information is provided regarding the architecture of the accelerator. What fraction of the HAS search space is unavilable due to restrictions imposed by the compiler? Perhaps these design points are all uninteresting?\n\nIs there anything to learn by a discussion of the good hardware configurations that were found? Are these surprising or unexpected? How do they differ from the baseline configuration?\n\nThe work is interesting but the claimed contributions perhaps need to be clarrified. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of searching over the joint space of hardware and neural architectures to trade-off accuracy and latency. 

Reviewers raised some valid questions about the following aspects:
1. Low technical novelty
2. Prior work on hardware and neural architecture co-design, and closely related work are not addressed
3. Lacking details on hardware platform and discussion on physical constraints to determine invalid hardware designs (addressed somewhat, but the response is not satisfactory)

One additional comment: if we care about latency for a particular hardware platform, it is possible to automatically configure adaptive inference techniques to meet the latency constraints. 

Overall, my assessment is that the paper requires more work before it is ready for publication.
Decision: Reject