Triple-Search: Differentiable Joint-Search of Networks, Precision, and Accelerators
Authors: Yonggan Fu
Yongan Zhang
Haoran You
Yingyan Lin
Venue:  (2021)
Keywords: neural architecture search
network hardware co-design

Abstract: The record-breaking performance and prohibitive complexity of deep neural networks (DNNs) have ignited a substantial need for customized DNN accelerators which have the potential to boost DNN acceleration efficiency by orders-of-magnitude. While it has been recognized that maximizing DNNs' acceleration efficiency requires a joint design/search for three different yet highly coupled aspects, including the networks, adopted precision, and their accelerators, the challenges associated with such a joint search have not yet been fully discussed and addressed. First, to jointly search for a network and its precision via differentiable search, there exists a dilemma of whether to explode the memory consumption or achieve sub-optimal designs. Second, a generic and differentiable joint search of the networks and their accelerators is non-trivial due to (1) the discrete nature of the accelerator space and (2) the difficulty of obtaining operation-wise hardware cost penalties because some accelerator parameters are determined by the whole network. To this end, we propose a Triple-Search (TRIPS) framework to address the aforementioned challenges towards jointly searching for the network structure, precision, and accelerator in a differentiable manner, to efficiently and effectively explore the huge joint search space. Our TRIPS addresses the first challenge above via a heterogeneous sampling strategy to achieve unbiased search with constant memory consumption, and tackles the latter one using a novel co-search pipeline that integrates a generic differentiable accelerator search engine. Extensive experiments and ablation studies validate that both TRIPS generated networks and accelerators consistently outperform state-of-the-art (SOTA) designs (including co-search/exploration techniques, hardware-aware NAS methods, and DNN accelerators), in terms of search time, task accuracy, and accelerator efficiency. All codes will be released upon acceptance.

Under review as a conference paper at ICLR 2021
TRIPLE-SEARCH: DIFFERENTIABLE JOINT-SEARCH JONNTTSSRRREE OF NETWORKS, PRECISION, AND ACCELERATORS
Anonymous authors Paper under double-blind review
ABSTRACT
The record-breaking performance and prohibitive complexity of deep neural net- works (DNNs) have ignited a substantial need for customized DNN accelera- tors which have the potential to boost DNN acceleration efficiency by orders-of- magnitude. While it has been recognized that maximizing DNNs' acceleration efficiency requires joint design/search for three different yet highly coupled aspects, including the networks, adopted precision, and their accelerators, the challenges associated with such joint search have not yet been fully discussed and addressed. First, to jointly search for network and its precision via differentiable search, there exists dilemma of whether explode the memory consumption achieve sub-optimal designs. Second, generic and differentiable joint search the networks and their accelerators non-trivial due (1) the discrete nature of the accelerator space and (2) the difficulty of obtaining operation-wise hardware cost penalties because some accelerator parameters are determined by the whole network. To this end, we propose a Triple-Search (TRIPS) framework to address the aforementioned challenges towards jointly searching for the network structure, precision, and accelerator in a differentiable manner, to efficiently and effectively explore the huge joint search space. Our TRIPS addresses the first challenge above via heterogencous sampling strategy to achieve unbiased scarch with constant memory consumption, and tackles the latter one using a novel co-search pipeline that integrates generic differentiable accelerator search engine. Extensive periments and ablation studies validate that both TRIPS generated networks and accelerators consistently outperform state-of-the-an (SOTA) designs (including co-scarchleexploraiin techniques, hardware-aware NAS methods, and DNN ac- celerators), terms of search time, task accuracy, and accelerator efficiency. All codes will be released upon acceptance.
INTRODUCTION
The powerful performance and prohibitive complexity of deep neural networks (DNNs) have fueled a tremendous demand for efficient DNN accelerators which could boost DNN acceleration efficiency by inders-of-mmanituue (Chen et al. 2016). In response, extensive research efforts have been devoted developing DNN accelerators. Early works decouple the design of efficient DNN algorithms and their accelerators. On the algorithms level, pruning, quantization, or neural architecture search (NAS) adopted trim down the model complexity; On the hardware level, various FPGA-/ASIC-based accelerators have been developed to customize the micro-are siicro-architecturee (e.g., processing elements dimension, memory sizes, and network-on -onchpp design) and soarithmm-tohhardwaar mapping methods (e.g., loop tiling strategies and loop orders) in order to optimize the acceleration efficiency for given DNN. Later, hardware-aware NAS (HA-NAS) has been developed to further improve DNNs acceleration efficiency for different applications (Tan et al. 2019).
More recently, it has been recognized that (1) optimal DNN accelerators require a joint considera- tion/search for all the following different yet coupled aspects, including DNNs' network structure, the adopted precision, and their accelerators' exploring subset of these aspects will lead to sub-optimal designs in terms of hardware efficiency or task accuracy. For example, the optimal accelerators for networks with different structures (e.g.. width, depth, and kernel size) can be very different; while the optimal networks and their bitwidths
aicro-architecture and mapping methods, and (2) merely

---

Under review as a conference paper at ICLR 2021
for different accelerators can differ lot (Wu et al. 2019). However, the direction of jointly designing
or searching for all the three aspects has only been slightly touched on previously. For example, (Chen et al. 2018: Gong et al.) 2019 Wang al. 2020 proposed to jointly search for the structure and precision of DNNs for a fixed target hardware; (Abdelfattah et al. 20200 Yang et al. 2020) Jiang et al., 22020a b) made the first attempt jointly search for the networks and their acceleratonn yet either their network or accelerator choices are limited due to the prohibitive time cost required by
their adopted reinforcement learning (RL) based methods; and EDD (Li et al. 2020) contributed pioneering effort towards this direction by formulating a differentiable joint search framework, which
however only consider one single accelerator parameter (i.e., parallel factor) and more importantly, has not yet fully solved the challenges of such joint search.
Although differentiable search one of the most promising ways in terms of search efficiency to explore the huge joint search space as discussed in Sec. 4.2 plethora of challenges exist to achieve an effective generic joint search for the aforementioned three aspects. First, Challenge 1: to jointly search for a network and its precision via differentiable search, there exists a dilemma whether to activate all the paths during search. On one hand, the required memory consumption can easily explode and thus constrain the search's 's scalability to more complex tasks if all paths are activated; on the other hand, partially activating a subset of the paths lead to sequential training of different precision on the same weights. which might result in inaccurate accuracy ranking among different precision as diseussed in (Jin al. 2020). Second, Challenge the accelerators' parameters are not differentiable, and it is non-trivial to derive the operation- wise hardware-cost penalty in order to perform differentiable search (in considering search efficiency). This is because the optimal accelerator often determined by the whole network instead of one specific operation/layer due to the fact that some accelerator parameters (c.g.. the loop order) need to be optimized for the whole network.
In this paper, we aim to address the aforementioned challenges towards scalable generic joint search for the network, precision, and accelerator. Specifically, we make the following contributions:
We propose Triple-Search (TRIPS) framework to jointly search for the network, precision, and accelerator in differentiable manner to efficiently explore the huge joint search space which cannot be afforded by previous RL-based methods due to their required prohibitive search cost. TRIPS identifies and tackles the aforementioned challenges towards scalable generic joint search of the three for maximizing both the accuracy and acceleration efficiency.
We develop a heterogeneous sampling strategy for simultaneous updating the weights and network structures to (1) avoid the need to sequentially train different precision and (2) achieve unbiased search with constant memory consumption, i.e., solve the above Challenge 1. In addition, we develop a novel co-search pipeline that integrates differentiable hardware search engine to address the above Challenge 2.
Extensive experiments and ablation studies validate the effectiveness of our proposed TRIPS framework in terms of the resulting search time, task accuracy, and accelerator efficiency, when benchmarked over state-of-the-art (SOTA) co-search/explorraion techniques, HA-NAS methods, and DNN accelerators. Furthermore, we visualize the searched accelerators by TRIPS to provide insights towards efficient DNN accelerator design Appendix.
2 RELATED WORKS
Hardware-aware NAS. Hardware-awart NAS has been proposed automate the design of efficient
DNNs. Early works (Tan et al., 2019 Howard et al. 2019 Tan & Le (019) utilize RL-based NAS
that requires a massive search time/cost, while recent works (Wu et al. 2219 Wan et al., 2020 Cai
et al. 2018 Stamoulis et al. 2019) explore the design space in a differentiable way (Liu et al. 2018)
with much improved searching efficiency. Along another direction, one-shot NAS methods (Cai et al.
2019 Guo et al. 2020 Yu et al. 2020) pretrain the supernet and directly evaluate the performances of the sub-networks in a weight sharing manner as a proxy of their independently trained performances at the cost of a longer pretrain time. In addition, NAS has been adopted to search for quantization strategies (Wang et al., 2019; Wu et al., 2018: Cai & Vasconcelos 2020 Elthakeb et al., 2020) for trimming down the complexity of given DNN. However, these works leave unexplored the
2

---

Under review as a conference paper at ICLR 2021
Type
Noth con-ddiferent intereeeeialll thetre Performane
Huge Memory Cost From Precieion Search Optimal Design Pairs
Channel Expansion
Structure
Protititive Jaint Search Space
Karnal Size
Group Num dor group convolution)
TRIPS Framework
Accalerstor Structure
Quantization Choices
Butfer Allocation
MAC liling scheduling
Precision cholces each block l-g.
16-bi))
Pipainn
Allocation
Figure 1: Illustrating our TRIPS framework: the large joint space and tackled challenges.
hardware design space, which is crucial enabler for DNN's acceleration efficiency, thus can lead to sub-optimal solutions.
DNN accelerators. Motivated by customized accelerators' large potential gains, SOTA accelera- tors (Du et al., 2015 Chen et al., 2017) innovate mnittoaarchitectures and aggortthm-tohhardwe mapping methods to optimize the acceleration efficiency, given DNN and the hardware specifi- cations. However, it non-trivial to design an optimal accelerator as it requires cross cross-disciplinary knowledge in algorithm,
arrhttecche,,,eeee and circuit design. SOTA accelerator design relies
either experts' manual design, which is very time consuming or design flow (Chen et al. 2005 2009) Rupnow et al., (011) and DNN accelerator design automation (Wang et al. 2016 Zhang al. 2018a,
Guan et al. 2017 Venkatesan et al. 2019 Wang et al.] [2018a] Gao et 2017). As they merely
explore the accelerator design space, they can result sub-optimal solutions compared SOTA co- oo-search/exploorttin methods and our TRIPS framework.
Co-exploration/search techniques. Pioneering efforts have been made towards jointly searching of DNNs and their accelerators to some extent. For joint searching of DNNs and their precision, (Chen) al. 2018: Gong et al. 2019 Wang et al.] 2020) adopt either differentiable or evolutionary algo- rithms yet without exploring their hardware accelerators. For joint searching of DNNs and their accelerators, (Abdelfattah et al. 2020 Yang et al., 2020; Jiang et al.) 2020a b) conduct RL-based search for the networks and some accelerator parameters/templates where they strictly constrain the search space of the network or accelerator to achieve a practical RL search time, limiting their scalability and achievable efficiency. (Lin et al.) is another pioneering work which co-designs the newtork and accelerator in sequential manner based on the fact that the accelerator's design cycle is longer than the networks. EDD (Li al. 2020) extends differentiable NAS search for layer-wise precision and the accelerators' parallel factor, which is most relevant to our TRIPS. EDD has not yet fully solved the joint search challenges. First, does not discuss or address the potentially explosive memory consumption issue of such joint search; second, EDD's accelerator search space only includes the parallel factor, which can be strictly limited to their accelerator template and cannot generalize to include common accelerator parameters such as the tiling strategies.
Built upon prior art, our TRIPS targets a scalable generic joint search framework to optimally search for the network, its precision, and adopted accelerator in differentiable manner for improving efficiency.
3 THE PROPOSED TECHNIQUES
In this section, we describe our proposed techniques for enabling TRIPS, where Sec. 3.1 provides TRIPS's formulation, Sec. 3.2 and Sec. 3.3 introduce TRIPS's enablers that address the key challenges of scalable generic joint search for networks, precision, and accelerators, and Sec. 3.4 unifies the enablers build comprehensive co-search framework.
3.1 TRIPS: FORMULATION
Fig. I shows overview of TRIPS, which jointly searches for the networks (e.g., kernel size, channel expansion, and group number), precision (c.g., 1-66-88/112//16-bi)) and the accelerators (c.g., PE array type, buffer size, and tiling strategies of each memory hierarchy) in differentiable manner.
3

---

Under review as a conference paper at ICLR 2021
230
Fea dhsoo-e..
0.20
Piid 1244
Hard with active aptions
220
Wrobbabatte
Probubiity
0.25
0.25
Consumeed
2.0
0.24
6-3it
0.26
6:541
220
0.22
0.22
124bi
6-sit
Bisit
6.22
6.sit
B:sit
12.07
20
60
0.21
20
40
80
Num. of precision choices
Epoch
Epoch
(b)
(c)
Figure 2: (a) GPU memory consumption comparison between soft Gumbel Softmax (GS) and hard GS sampling (two active choices) for precision search. Probability of each precision choice during the search process in the 4-th block when searching with: (b) hard GS sampling for updating both weights w and precision choices (result in the lowest 4-bit), and (c) the proposed heterogeneous sampling for updating w and B (result in the highest 12-bit).
TRIPS targets a scalable yet generic joint search framework, which we formulate as bi-level optimization problem:
min Luai(w" prec(()) + XL (hw(7)) >) net(a), pred( 0.3
s.t.
w" = arg min Levaia (w, net( ()))
(2)
s.l. 7 = arg min Leose(hwwl)) prec()))()))
(3)
Where a, 3, and are the continuous variables parameterizing the probability of different choices for the network operators, precision bitwidths, and accelerator parameters as in (Liu et al. 2018), w is the supernet weights, Liroins Leals and Leost are the loss during training and validation and the hardware-cost loss, and nel(a), )), prec(B). and lew(T) denote the network, precision, and accelerator characterized by a, B, Y, respectively.
3.2 TRIPS ENABLER 1: HETEROGENEOUS SAMPLING FOR PRECISION SEARCH
As discussed in Sec I there exists dilemma (i.e., memory consumption explosion or biased search) whether to activate all the paths during precision search, for addressing which we have developed heterogeneous sampling. Next, we first use real experiments to illustrate the joint search dilemma, and then introduce our heterogeneous sampling which effectively address those challenges/issues.
Activating all choices memory explosion and entangled correlation among choices. During precision search, activating all the precision choices as (Wu al. 2018 Gong et al.) (219)) can easily explode the memory consumption especially when the precision is co-searched with the network structures. While composite convolutions (Cai Vasconcelos, 2020) for mixed precision search mitigate this memory explosion issue during search by shrinking the required computation, yet this large memory consumption issue would still exist during training when updating the precision parameters, i.e., / in Eq. (T). For example, shown in Fig. 2 (a), the measured GPU memory consumption of co-searching the network and precision on ImageNet grows linearly with the number of precision choices if activating all precision choices during search. In addition, the entangled correlation (e.g., co-adaptation (Hong et al. 22020), correlation (Li et al.] 2019). and cooperation (Tian) al. 2020g) among different precision choices leads to large gap between the supernet during search and the final derived network, thus failing the joint search.
Activating only a subset of choices Bi- ased search. For addressing the above issues of memory explosion and correlation among choices, one natural choice is to adopt hard Gumbel Softmax by reducing the memory con- sumption, which however can lead to a biased search and thus poor performance. Specifically, activating only subset of the precision choices implies sequential training of different preci- sions that can lead to inaccurate performance
Comparing the resulting accuracy when training a fixed network using different precision schedules, where high2low and low2high denote sequential training from high precision to preci- sion and the inverse case, respectively.
Table 1:
Strategy
4-bir
Acc
16-hit (%)
3248
Independent
63.52
high2low
59.20
low 2high
4.36
26.55
43.58
63.3
63.5
joint
||
63.28
66.98
67.21
67.23
67.36
4

---

Under review as a conference paper at ICLR 2021
ranking. This is because sequential training means different precision choices are applied on top of the same weights and activations. As a result, different precision choices can interfere with each other and different training order would lead to different result. For better understanding, we next show two concrete experiments.
Co-search network and precision using hard Gumbel Softmax: Fig. 2 (b) shows the resulting preci- sion probability evolution when co-searching the network and precision on CIFAR- 100 using hard Gumbel Softmax (activating two precision choices) without imposing any hardware-cost constraints, indicating the desired precision choice would be the highest precision. However, as shown in Fig. 2 (b), the block co-searched using hard Gumbel Softmax collapse to the lowest precision (i.e., the highest probability towards the end of the search is the lowest precision choice 4-bit), indicating an ineffective search direction. Note that the fluctuation in the probability of different precision choices is caused by the intermittent activation of the block due to the hard Gumbel Softmax sampling.
Sequential training of a fixed network with multiple precision choices: As observed in (Jin et al. 2020), when training fixed network with multiple precision choices, either ascending or descending the precision will incur an inferior convergence and thus chaotic accuracy ranking among different precision choices. For example, as shown in Tab. we compare the accuracy of fixed network (all blocks adopt the k3el (kernel size 3 and channel expansion 1) structure in (Wu et al. 22019) under different precision choices, when being trained with different precision schedules, and find that only jointly training all the precision choices can maintain the ranking consistent with that of independently trained ones, while sequential training leads to both inferior accuracy and ranking.
Proposed solution Heterogeneous sampling. To tackle both aspects of the aforementioned
dilemma, we propose heterogeneous sampling strategy as formulated in Eq. (4) where W A' are the composite weights/ activations of the (-th layer as in (Cai & Vasconcelos 2020 which are the weighted sum of weights / activations under different precision choices, e.g., W is the weights quantized to the j-th precision among the total options of the I-th layer. In our heterogencous sampling, for updating the weights in Eq. (2). we jointly update the weights under all the precision choices weighted by their corresponding soft Gumbel Softmax GSS()),, where 3) parameterizes the probability of the j-th precision in the 1-th layer, and the gradients can be estimated by STE (Zhou et al. 2016) as @Lerain 0 T so that no extra intermediate feature maps are needed be stored into the memory during backward. For updating we adopt hard Gumbel Softmax (Jang et al. 2016) with one-hot outputs GShora(33)) to save memory and computation while reducing the correlation among precision choices. In the same co-search setting as Fig. 2 (b), all the blocks searched using our proposed heterogencous sampling converge to the highest precision choice towards the end of the search as shown in Fig. (c).
A 1+1 W A'
Σ j=1
3 W
Σ BJ where 3) J=1
GS(S)) GShard
if updating weight if updating |3
3.3 TRIPS ENABLER 2: DIFFERENTIABLE ACCELERATOR SEARCH ENGINE
Motivation. Although EDD (Li et al. 2020 also co-searches the accelerator with the network, their search space is limited to include only the parallel factor within their template which can be analytically fused into their theoretical computational cost, whereas this is not always applicable to other naturally non-differentiable accelerator design parameters such as tiling strategies. more general and efficient search engine is needed towards generic differentiable accelerator search.
Search algorithm. We propose a differentiable search engine to efficiently search for the optimal accelcrator (including the miiroo-rchiteectures and mapping methods) given a DNN model and its precision based on single-path sampling as discussed in Sec. 3.1 We solve Eq. (3) in a differentiable way:
7" * arg min
net Σ GShardl? (haneehhw((GGSSaaa ()))) (rr(B)) (5)
m=1
where M is the number of accelerator design parameters. Given the network net((O)),)) (w)) and precision prec((B), (), where O',, and Byu are the only operator and precision activated during forward discussed in Sec. 3.4 our search engine utilizes hard Gumbel Softmax G.Spard sampling
5

---

Under review as a conference paper at ICLR 2021
on each design parameter y to build an accelerator hwr (GShard(T""))) and penalize each sampled accelerator parameter with the overall hardware- Lesst through relaxation in a gradient manner.
Hardware template. We adopt unified template for both the FPGA and ASIC accelerators, which is a parameterized chunk-based pipeline micro- aocco-acchitecture inspired by (Shen et al.) 2017). In particular, template comprises multiple sub-accelerators (i.e., chunks) and executes DNNs in a pipeline fashion. Each chunk is assigned with multiple but not necessarily consecutive layers which are executed sequentially within the chunk. Similar to Eyeriss, each chunk consists of levels of buffers/memories (e.g., on-chip buffer and local register files) and processing elements (PEs) facilitate data reuses and parallelism with searchable design knobs such as PE interconnections (i.e., cetwork-on-chip). allocated buffer sizes, MAC operations' scheduling and tiling (i.e., dataflows), and so on (see more details in Appendix B).
the aentwarr/micro-architettte
Discussion about the general applicability. Our search approach is general and can be applicable to different hardware architectures, since we do not hold any prior assumptions about the adopted hardware architecture. Specifically, for any target hardware architecture, including TPU-like or GEMM or other accelerators, our search approach can be directly applied once given (1) a simulator to estimate the hardware cost, and (2) set of user-defined searchable design knobs abstracted from the target hardware architecture.
3.4 TRIPS: THE OVERALL CO- FRAMEWORK
Objective and challenges. TRIPS's iterative search starts from updating both the supernet weights w and accelerator parameters y given the current network net(a) quantized using precision prec (5), and then updates a and 3 based on the derived optimal weights w and accelerator hw(>" resulting from the previous step. Updating w and y have been discussed in Sec. 3n2aadd Sec. 3.3 respectively. The key objective of TRIPS is captured by Eq. (T) which involves all the three major aspects towards efficient DNN accelerators. The key challenges in achieving TRIPS include (1) the prohibitively large joint search space (e.g., 2.3E+21 in this work) which if not addressed will limit the scalability of TRIPS practical yet complex tasks; (2) the entangled co-adaptation (Hong al. 2020), correla- tion (Li et al. 2019), and cooperation (Tian et al. 2020) issues among different network and precision choices can enlarge the gap between the supemee during search and the final derived network, thus failing the joint search; and (3) the non non-trivial triviality of deriving hardware-cost for the layer/block
update during network search, as the hardware-cost determined by the whole network.
Forward: A
Σ
GS GShard 0,(A)) = O'
(6)
Olval da
OGS(at)) o
K oh(aa)
oGS(ak) da
Backward:
Σ
(7)
aGS(aa) da:
d
k=1
Bo
(GGShara(a)) = =1) Loi (hull" net(al), prec(())
da
TRIPS implementation. To tackle the three aforementioned challenges, TRIPS integrates novel co-search pipeline which can be illustrated using the co-search for a as follows and is similarly applicable to co-search for B Note that here we define path to be one of the parallelled candidate operators between the layer input and layer output within one searchable layer, which can be viewed as -gasiee (layer-wise) version of the path definition in (Wang et al. 2018b Qiu et al.) 2019).
For updating both a (see Eq. (6 and 33 during forward, TRIPS adopts hard Gumbel Softmax sampling (Hu et al., 2020a). i.e., only the choice with the highest probability will be activated to narrow the gap between the search and evaluation thanks the single-path property of hard Gumbel Softmax sampling. In Eq. (6). A' and A denote the feature maps of the I-th and (1 + 1)-th layer, respectively, N is the total number of operator choices, O is the i-th operator in the 1-th layer parameterized by a and O',, is the only operator activated during forward.
Single-path forward:
Multi-path backward: For updating both a (see Eq. (7) and |3 during backward, TRIPS activates multiple paths to calculate the gradients of a and 3 through Gumbel Softmax relaxation in order to balance the search efficiency and stability motivated by (Cai et al. 2018 Hu et al. 2020b). For

---

Under review as a conference paper at ICLR 2021
example, a.'s gradients calculated using Eq. 77. where K is the number of activated choices with the top K Gumbel Softmax probability. Similar to Cai et al. 2018 K (1, in TRIPS to control the computational cost.
The network search in Eq. (T) performed in layer/block-wiss manner as in (Liu et al. 2018), thus requiring layer/block-wise hardware-cost penalty which is determined by mapping method and the corresponding layer/block execution cost on the optimal accelerator hww("") ). The optimal mapping method of an accelerator is yet determined by the whole network. To handle this gap. we derive the layer/block-wei the single-path network derived from the current forward would be the final derived network, as this
Hardware-cost penalty:
the wyver//lock-to-ccclleee
hardware-cost assuming that
single-patt path network has a higher not the highest probability to be finally derived. In Eq. (8) an indicator denoting whether og (i.e., the i-th operator in the I-th layer) is activated during forward.
EXPERIMENT RESULTS
4.1 EXPERIMENT SETUP
Software settings. Search space and hyper-params. We adopt the same search space in (Wu al. 2019) for the ImageNet experiments and disable the first two down sampling operations CIFAR- 10/100 experiments. We use [4, 6, 8, 12. 16] as candidate precision choices and one block shares the same precision of weights and activations for more hardware friendly implementation. We activate two paths during backward, i.e., K = 2 in Eq. (7). for search efficiency. For Loose in Eq. (3) we use latency for FPGA as the target metric is Frame- Paaee--er-Seccoee (FPS), and (EDP) for ASIC. Detailed search and training settings are elaborated in Appendix A
Baselines. We mainly benchmark over four kinds of SOTA baselines: (1) the most relevant baseline EDD (Li et al., 2020) which co-searches networks, precision, and accelerators, (2) SOTA methods CO co-exploring networks and accelerators including HS-Co-Op (Jiang et al.] 2020b), NASAIC (Yang
al. 2020). and BSW (Abdeliattah et al. 22200) (3) SOTA methods co-searching the networks and precision including APQ (Wang et al. 2020) and MP-NAS (Gong et al. 2019). and (4) hardware-
aware NAS with uniform precision, including FBNet (Wu al.) 2019 ProxylessNAS (Cai et al.)
2018), Single-Path NAS (Stamoulis et al. 2019), and EfficientNet-BO (Tan & Le 2019).
Hardware settings. To evaluate the generated network and accelerator designs, for FPGA cases, we adopt the standard Vivado HLS (Xilinx Inc.) a) design flow, on the target Xilinx ZC706 de- velopment board (Xilinx Inc., b). which has total 900 DSP48s (Digital Signal Processor) and 1Mb BRAM (Block RAM). For ASIC implementations. we use the SOTA energy estimation tool Timeloop Paarashar al.) 2019) and Accelergy, (Wu et al. 2019), to validate our generated design's performance, with CACT17 (Balasubramonian et al. 2017 and Aladdin (Shao et al.) (014) at 32nm CMOS technology as unit energy and timing cost plugins. Details about the accelerator search space are discussed in Appendix
4.2 BENCHMARK SEARCH EFFICIENCY
To evaluate the superiority TRIPS in terms of search efficiency, we compare the search space size and search time of TRIPS with both RL-based co-search works and one-shot NAS methods using the reported data from the baselines' original papers as shown in Tab. 2 consistently require notably less search time while handling the largest joint search space on all the considered tasks. In particular, compared with the one-shot NAS methods (Guo et al. 2020 Cai
We can see that TRIPS
Table 2: Search efficiency benchmark of TRIPS over co-search works and one-shot NAS methods
Method
Dursset
Netwark Space
Accelerator Space
Precision Space
Joint Space
Search Time (GPU hours)
HS-Co-Opi
[lang etal.
20206
CIFAR-10
1. 1.15E418
1.15E418
103.9
CIFAR-10
2:21E+27
2,10++15
5.30E+63
BSW (Abdelfaitah ei al. 2000
CIFAR-100
1,20E405 9885+420
8.61E+03
3.63E409
5184
TRIPS
CIFAR- 100
2.246+27
2,406+15
5,30E+63
12
HIS-Co-Opt liang 202bb Once-
ImageNet
2.22+188
2
266.8
ImageNet
2.00E+199
1200
APQ Wan S0Ay
IatgeNet
3500
1.00E 10
1.0EE: 45
2400
Single One
2020
ImageNet
7.00E+21
7.00E+211
288
ImageNet
9.554420
2.348427
2.40R+15
5.30R+633
7

---

Under review as a conference paper at ICLR 2021
al. 2019) which can be potentially extended to co-search frameworks while suffering from large pretraining cost, TRIPS achieves 3.6x 30x less search time on ImageNet, while being end-to-end, justifying our choice of differentiable co-search.
4.3 BENCIMARR OVER SOTA METHIOSS
Co-exploration of networks, precision, and accelera- tors. We benchmark our TRIPS framework with SOTA efficient DNN solutions on ImageNet and IPGA-based accelerators under the 512 DSP limits in Fig- 3 follow-
ACC (%)
ing (Abdelfattah et al. 2020). Specifically, we provide four searched results our TRIPS framework; we use the re- ported results for EDD, and search for the optimal accelera- tor in our accelerator space for APQ. MP-NAS, and SOTA hardware- aware NAS methods; for EfficientNet-Bi we ap- ply the SOTA mixed precision strategy searched by (Habi et al. 2020 for fair comparison and the ProxylessNAS: 8bit is reported by APQ (Wang et al. 2020); and the other baselines are all quantized to 8-bit for hardware measure- ment and the accuracies are from the original papers with- considering the quantization effect. We observe from Fig- 3 that (1) the searched networks by our TRIPS framework consistently push forward the frontier accuracy-FPP trade-offs, (2) compared with the most relevant baseline EDD, achieve +1.3% higher accuracy with 1.59x FPS. The effectiveness of TRIPS over various SOTA methods that represent most of the existing co-design directions verifies the necessity and effectiveness of co-searching all the three aspects towards efficient DNN accelerators.
Blickettel
too Net
122% Ano
Since
TAIPS
68
FPS
Figure 3: Accuracy vs. FPS trade-off
of TRIPS against SOTA efficient DNN solutions on ImageNet.
Co-exploration of networks and accelerators. Software- Software-Hardware co-design significant prop- erty of our TRIPS framework, so we further benchmark it with both searched precision and fixed- precision over SOTA networr/accelerrtor co-search works for fair comparison.
2.8x
ACC (%)
+10,01%
+8,01%
ACC (%)
ACC (%)
5540%
4.4x
+5,90%
Turs Mice
Tears Miadd
TRPSS
6.70v
1.40x
6.54x
FPS
FPS
FPS
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet
Figure 4: Benchmark TRIPS w/ and w/o precision search (denoted as TRIPS- Mixed and TRIPS- 16bit,
respectively) with SOTA network/accelerator co ee-elllraation methods (Jiang et al. 20206 Abdelfattah
et al. 2020 on 1FFAR-110100//maeNeet
Co-search FPGA. We benchmark with HS- Co-Opt Jiang et al. 20205 and BSW (Ab-
Table 3: Comparing the accuracy and ASIC effi-
ciency (i.c., EDP and area) of TRIPS and SOTA co-exploration ASIC works Yang et al. 2020).
delfattah al. 2020 on ZC706 under the same DSP limits as the baselines on CIFAR- 10/100/maaeeees Note that all our baselines here adopt 16-bit fixed-point design, so we provide TRIPS with fixed 16-bit in addition to the one with searched precision for a fair com- parison. From Fig. we can see that (1) on both 1IFAR-10/100 dataset, TRIPS with fixed 16-bit consistently achieves better accuracy (up to 10.91 % and 5.15%, respectively) and higher FPS (up to 2.21x and 2.15x, respectively) under the same DSP constraint, and (2) when co-searching the precision, our method can more aggressively push forward the FPS improvement (up to 6.79x and 6.54x, respectively on CIFAR-10/100) 10/100), implying the importance of the co-exploration of the precision dimension in addition to network and accelerator co-explorations. Specifically, TRIPS with searched precision achieves a +5.96% higher accuracy and 4.4x FPS on ImageNet over (Tiang al. 2020b).
Optimization Methods
Accuracy
EDP
Area (um²)
(lolock-cyclk)
NAS ASIC
94.17
3.30E+06
4.83E+09
ASIC HW-NAS
92.53
28114466
3.86F:499
NASAIC
92.62
1.624406
3.34E+09
TRIPS
94.34
4,36E403
5.92E+405
8

---

Under review as a conference paper at ICLR 2021
Co-search on ASIC, We benchmark with NA- SAIC, the first exploration towards network/ ac- celerator co-search targeting ASIC accelerators, with both the co-search results and their reported sequential optimization/hardwarce aware opti- mization results (Yang et al., 2020) on CIFAR- 10 in Tab. 3 We can observe that compared with both co-search, sequential optimization, and hardware-award exploring the ASIC design space, our TRIPS framework consistently achieves notably improved trade-offs between accuracy and energy delay product (EDP), which is energy multiplied with latency. In particular, we achieve a +0.1 12% +1.8 % higher accuracy with 371.56 756.88x reduction in EDP. In the baseline implementations, most of the area is occupied by the support for heterogeneous functionalities, which leads to severe under utilization when executing task, thus contributing to the surprisingly higher area and energy consumption.
Table 4: Benchmark TRIPS over NHAS (Lin et al.) and DANCE (Choi et al. 2020 under the same precision setting.
Co-search
Accuracy
Latency (ms)
Methods
(%)
(mrrr)))
NHAS (Lin et
70.74
1.58
5.87
TKIPS
71.70
1,25
5,50
DANCE Choi et at. 2020
2020
68.70
8.13
2.73
TRIPS
72.20
2.85
2.12
optimization methods for
We further benchmark TRIPS over other two co-search baselines targeting ASIC accelerators, i.e., NHAS (Lin et a and DANCE (Choi et al., (2020). In particular, we fix the precision of TRIPS to be 4-bit and 16-bit to fairly compare with (1) NHAS which adopts 4-bit and (2) DANCE which adopts 16-bit, respectively. As shown in Tab. 4 TRIPS achieves a 0.96% higher accuracy and a 20.9%/64.99 96% reduction in latency together with 6.3%/22.3% reduction in area consumption, as compared with NHAS and DANCE, respectively, verifying the superiority of our TRIPS.
4.4 ABLATION STUDIES ABOUT TRIPS
Scalability under the same DSP. Fig. 5 we show the pareto frontier of our TRIPS frame- work under the same DSP constraint with differ- ent accuracy and FPS trade-off's on CIFAR-I00 to show our TRIPS can handle and scalable with large range of DNN solutions.
78
76
ACC (%)
74
1:11+ PPS
72
Effectiveness of heterogeneous sampling. In addition to the example and analysis in See. 3.2 we further benchmark with the baseline that adopts the same sampling strategy for updating both the weights and precision. We integrate the baseline's sampling strategy into our TRIPS framework (K = 2 for all the experiments), termed as TRIPS w/o h-sampling, and show the trade-offs between the achieved accuracy and FPS in Fig. 5 We find it tends to select lower precision choices which are harmful to the overall accuracy, which is consistently inferior than that of TRIPS with heterogeneous sampling, due to the inaccurate estimation for different precision ranking.
68
TRIPS nilo sampling Sequential Opt
100
150
FPS
Figure 5: Accuracy vs. FPS trade-off of TRIPS, TRIPS w/o heterogeneous sampling, and the se- quential optimization baseline on CIFAR-100.
Comparison with sequential optimization. Due to the great flexibility on both software and hardware side, natural baseline is to search the network and precision based on theoretical efficiency metrics (e.g.. bit operations) and then search for the best matched accelerator given the searched network and precision from the first search. We benchmark over results from such design flow in Fig.5 on CIFAR- 100 and observe that TRIPS consistently outperforms the sequential optimization baseline, e.g., 1.95% higher accuracy with 1.75x FPS, indicating the poor correlation between theoretical efficiency and real-device efficiency measurement.
More ablation studies about the accelerator search engine and visualization of the searched network, precision and accelerator can be found in [ppendix(Clame respectively.
CONCLUSION
We propose a Triple-Search (TRIPS) framework to jointly search for the network structure, precision, and accelerator in a differentiable manner. Our TRIPS framework adopts a heterogeneous sampling strategy and novel co-search pipeline that integrates generic differentiable accelerator search engine to achieve unbiased search with constant memory consumption. Extensive experiments validate the superiority of TRIPS over SOTA designs in terms of accuracy and efficiency.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohamed S Abdelfattah, Łukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane. Best of both worlds: Automl codesign of enn and hardware accelerator. arXiv preprint arXiv:2002.05002, 2020.
Rajeev Balasubramonian, Andrew B. Kahng, Naveen Muralimanohar, Ali Shafice, and Vaishnav Srinivas. Cacti 7: New tools for interconnect exploration in innovative off-chip memories. ACM Trans. Archit. Code Optim., 14(2), June 2017. ISSN 1544-3566.
doi: 10.1 105/0055285572. URL
htttss://do..rgg// 011..///2//0111///855522
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:/812.0033: 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all. Train one network and specialize for efficient deployment. arXiv preprint (108::990.0097 2019.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixeddpreciiion neural net- works. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2349-2358, 2020.
Deming Chen, Jason Cong, Yiping Fan, Guoling Han, Wei Jiang, and Zhiru Zhang. xpilot: platform-bes based behavioral synthesis system. SRC TechCon, 5, 2005.
Deming Chen, Jason Cong, Yiping Fan, and Lu Wan. Lopass: A low-power architectural synthesis system for FPGAs with interconnect estimation and optimization. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 18(4):564-577, 2009.
Y. Chen, T. Krishna, J. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. JSSC 2017, (21))117-1388 2017.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Computer Architecture News, 44(3): 367-379, 2016.
Yukang Chen, Gaofeng Meng, Qian Zhang, Xinbang Zhang, Liangehen Song, Shiming Xiang, and Chunhong Pan. Joint neural architecture search and quantization. arXiv preprint 19X44.181..0942. 2018.
Kanghyun Choi, Deokki Hong, Hojae Yoon, Joonsang Yu, Youngsok Kim, and Jinho Lee. Dance: Differentiable accelerator/network co-exploration. arXiv preprint arXiv:2009.06237 2020.
Z. Du, R. Fasthuber, T. Chen, P. lenne, I. Li, T. Luo, X. Feng, Y. Chen, and O. Temam. Shidiannao: Shifting vision processing closer to the sensor. In ACM SIGARCH Computer Architecture News, volume 43, pp. 92-104. ACM, 2015.
Ahmed Taha Elthakeb, Prannoy Pilligundla, Fatemeh Mireshghallah, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. Releq: A reinforcement learning approach for automatic deep quantization of neural networks. IEEE Micro, 2020.
Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, and Christos Kozyrakis. Tetris:
Scalable and International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 751-764, 2017.
efficient neural network acceleration with 3d memory. In Proceedings of the
Chengyue Gong, Zixuan Jiang, Dilin Wang, Yibo Lin, Qiang Liu, and David Pan. Mixed precision neural architecture search for energy efficient deep learning. In ICCAD, pp. 1-7, 2019.
Yijin Guan, Hao Liang, Ningyi Xu, Wenqiang Wang. Shaoshuai Shi, Xi Chen, Guangyu Sun, Wei Zhang, and Jason Cong. FP-DNN: An automated framework for mapping deep neural networks onto FPGAs with RTL-HLS hybrid templates. In 2017 IEEE 25th Anmual International Symposium rredamProgrammaaee Custom Computing Machines (FCCM), pp. 152-159. IEFE, 2017.
10

---

Under review as a conference paper at ICLR 2021
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision, pp. 544-560. Springer, 2020.
Hai Victor Habi, Roy H Jennings, and Amon Netzer. Hmq: Hardware friendly mixed precision quantization block for cnns. arXiv preprint wwXiv:2007.09952 2020.
Weijun Hong, Guilin Li, Weinan Zhang, Ruiming Tang, Yunhe Wang, Zhenguo Li, and Yong Yu. Dropnas: Grouped operation dropout for differentiable architecture search. In International Joint Conference on Artificial Intelligence, 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1324, 2019.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin.
Dsnas: Direct neural architecture search without parameter retraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12084-1222 12092, 2020a.
Yibo Hu, Xiang Wu, and Ran He. Tf-nas: Rethinking three search freedoms of latency-constrained differentiable neural architecture search. arXiv preprint (rXix:00080051// 2020b.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.00 144, 2016.
Weiwen Jiang, Qiuwen Lou, Zheyu Yan, Lei Yang, Jingtong Hu, X Sharon Hu, and Yiyu Shi. Devicc-ciiruil-acchiteeuuu co-exploration for computing-Bn computing-in-memory in-memory neural accelerators. IEEE Transactions on Computers, 2020a.
Weiwen Jiang, Lei Yang, Edwin H-M Sha, Qingfeng Zhuge, Shouzhen Gu, Sakyasingha Dasgupta, Yiyu Shi, and Jingtong Hu. Hardwarc/softwaar co-exploration of neural architectures. IEEE Transactions Computer-Aides Design of Integrated Circuits and Systems, 2020b.
Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2146-2156, 2020.
Guilin Li, Xing Zhang, Zitong Wang, Zhenguo Li, and Tong Zhang. Stacnas: Towards stable and consistent differentiable neural architecture search. arXiv, pp. arXiv-1909, 2019.
Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. Edd: Efficient differentiable dnn architecture and implementation co-search for embedded ai solutions. arXiv preprint arXiv:2005.02563, 2020.
Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. herwale architecture search.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.009 2018.
A. Parashar, Raina, S. Shao, Y. Chen, V. A. Ying, A. Mukkara, R. Venkatesan, Khailany, S. W. Keckler, and J. Emer. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 304-315, 2019.
Jiantao Qiu, Jie Wang, Song Yao, Kaiyuan Guo, Boxun Li, Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, Sen Song, et al. Going deeper with embedded fpga platform for convolutional neural network. In Proceedings of the 2016 ACM/SIGDA International Symposium on Field- Programmable Gate Arrays, pp. 26-35. ACM, 2016.
Yuxian Qiu, Jingwen Leng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, and Yuhao Zhu. Adversarial defense through network profiling based path extraction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4777-4786, 2019.
11

---

Under review as a conference paper at ICLR 2021
Kyle Rupnow, Yun Liang, Yinan Li, Dongbo Min, Minh Do, and Deming Chen. High level synthesis of stereo matching: Productivity, performance, and software constraints. In 2011 International Conference Field-Progrann Programmable Technology, pp. 1-8. IEEE, 2011.
Y. S. Shao, B. Reagen, G. Wei, and D. Brooks. Aladdin: A pre-rtl, power-performance accelerator simulator enabling large design space exploration of customized architectures. In 2014 ACM/IEEE 41 International Symposium on Computer Architecture (ISCA), pp. 97-108, 2014.
Yongming Shen, Michael Ferdman, and Peter Milder. Maximizing enn accelerator efficiency through resource partitioning. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, pp. 535-547, New York, NY. USA, 2017. Association for Computing Machinery. ISBN 9781450348928. doi: 10. 10.114//309855.3000222 URL https: /tt:///doi..// org/ I0.1145/307988 3080221
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficien convnets in less than hours. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481-497. Springer, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905. 19946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-wware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Yunjie Tian, Chang Liu, Lingxi Xie, Jianbin Jiao, and Qixiang Ye. Discretization architecture search. arXiv preprint arXiv:2007.03154 2020.
Rangharajan Venkatesan, Yakun Sophia Shao, Miaorong Wang, Jason Clemons, Steve Dai, Matthew Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, et al. MAGNet: A Modular Accelerator Generator for Neural Networks. In Proceedings of the International Conference on Computer-Aided Design (ICCAD), 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12965 12974, 2020.
Junsong Wang. Qiuwen Lou, Xiaofan Zhang, Chao Zhu, Yonghua Lin, and Deming Chen. Design flow of accelcrating hybrid extremely low bit-width neural network in embedded FPGA. In 2018 28th International Conference on Field Programmable Logic and Applications (FPL), 2018a.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2078-2087 2020.
Ying Wang, Jie Xu, Yinhe Han, Huawei Li, and Xiaowei Li. Deepburning: Automatic generation of fpga-based learning accelerators for the neural network family. DAC '16, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342360. doi: 10.1145/2897937. 2898003. URL httpt:: http://o..oogg... 10
2898003
Yulong Wang, Hang Su, Bo Zhang, and Xiaolin Hu. Interpret neural networks by identifying critical data routing paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8906-8914 8914, 201 8b.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv: 1ef.18/1..0000 2018.
12

---

Under review as a conference paper at ICLR 2021
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-are aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 10734-10742, 2019.
Y. N. Wu, J. S. Emer, and V. Sze. Accelergy: An architecture-level energy estimation methodology for accelerator designs. In 2019 JEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1-8, 2019.
Qingcheng Xiao, Yun Liang, Liqiang Lu, Shengen Yan, and Yu-Wing Tai. Exploring heterogeneous algorithms for accelerating deep convolutional neural networks on fpgas. In Proceedings of the 54th Annual Design Automation Conference 2017. DAC '17, New York, NY, USA, 2017. Association for Computing Machinery, ISBN 9781450349271. doi: 10.1145/3001699.3022244
URL htt.hhtps///.ooooo www.o..org/100 00....114450000 3062244
Xilinx Inc. Vivado High-Level Synthesis, https://https://ma www. www.w.wwwix...mmmm.. com/
pprouccss/essiqqe 2019-09-16.
html, accessed
Xilinx Inc. Xilinx zynq-7000 soc zc706 evaluation kit. products/boaree
https://wwa xixiiiiinnii com/
html b. (Accessed on 09/30/2020).
Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous asic accelerator designs targeting multiple tasks. arXiv preprint arXiv:2002.041163 2020.
Xuan Yang, Jing Pu, Blaine Burton Rister, Nikhil Bhagdikar, Stephen Richardson, Shahar Kvatinsky, Jonathan Ragan-Kelley, Ardavan Pedram, and Mark Horowitz. A systematic approach to blocking convolutional neural networks, 2016.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architec- ture search with big single-stage models. arXiv preprint arXiv:2003. 111422 2020.
Chen Zhang, Peng Li, Guangyu Sun, Yijin Guan, Bingjun Xiao, and Jason Cong. Optimiz- ing fpga-based accelerator design for deep convolutional neural networks. In Proceedings of the 2015 ACM/SIGDA International Symposium on New York, NY. USA, 2015. Association for Computing Machinery. ISBN 9781450333153, doi: 2..145/2684444.2669000 URL 2684746. 2689060
Gate Arrays, FPGA
'15, pp. 161-170,
Chen Zhang, Guangyu Sun, Zhenman Fang, Peipei Zhou, Peichen Pan, and Jason Cong. Caffeine: Towards uniformed representation and acceleration for deep convolutional neural networks. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018a.
Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. Dnnbuilder: An automated tool for building high-performance dnn hardware accelerators for fpgas. In Proceedings of the International Conference on Computer-Aided Design, ICCAD '18, New York, NY, USA. 2018b. Association for Computing Machinery. ISBN 1781459359504, doi: 10.1145/3240665.3244000 3240801. URL ///oi org 10. 1 32455552446665 001177555344000 3240801
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.066 1ee.11666666600 06000, 2016.
A TRAINING AND SEARCH SETTING
Search settings. For searching on CIFAR- dataset, we use half of the dataset for updating supernet weight and the other half for updating network and precision parameter and B. We search for 90 epochs with an initial gumbel softmax temperature decayed by factor 0.975 every epoch. For searching ImageNet, we randomly sample 100 classes proxy search dataset and
13

---

Under review as a conference paper at ICLR 2021
use 80% data for updating and the other 20% for updating and 3. We pretrain the supernet by 30 epochs without updating network architecture and precision, then search for 90 epochs with an initial temperature 5 decayed by 0.956 every epoch, following (Wu et al.] 2019). For both CIFAR-10/100 and ImageNet, we use initial learning rate 0.1 and an annealing cosine learning rate.
Training settings. For CLFAR-10/100 we train the derived network for 600 epochs with 0.1 initial learning rate and an annealing cosine learning rate on a single NVIDIA RTX-2080Ti gpu following (Liu al. 2018). For ImageNet, we adopt a 0.05 initial learning rate and an annealing cosine learning rate for 150 epochs with four NVIDIA Tesla V100 gpus.
B ACCELERATOR SEARCH SPACE
To flexibly maintain the balance between hardware resource consumption and throughput across different generated networks, we employ the chunk-wise pipeline styled accelerator architecture inspired by (Shen et al. 2017 Zhang al. 2018b). To enable the automatic hardware optimization
and explore as much as the performance frontier, we further free up the hardware configurations
during the co-optimization. Adopted from (Chen al. 2017 Zhang al. 2015 Yang et al., 2016).
these configurations, as illustrated in Fig. [ cover 1) paralee processing elements (PE) settings: number and inter-connections of PEs, 2) buffer management: allocation of lower memory levels between input, weight and output, 3) tiling and scheduling of MAC(Multiply and Accumulate) computations and 4) layer allocation: ways to assign each layer to the corresponding pipeline stage (sub-accelerator). All the above configurations are formatted and maintained through vectors of options to be compatible with the optimization formulation in Sec. Taking AlexNet as an example workload, the total accelerator space size can reach up to 10 for each sub-accelerator and the space can go exponentially larger the number of sub-accelerator (pipeline chunks) increases.
C ABLATION STUDIES ABOUT THE ACCELERATOR SEARCH ENGINE
The proposed accelerator search engine is one key enabler of our TRIPS framework. To evaluate its efficacy, we compare the accelerator efficiency of the TRIPS generated accelerators with SOTA accelerators under the same datasets and models. For FPGA Abbaed accelerators, we consider three representative including (Qiu et al. 2016 Xiao et al.) 2017 Zhang al.) 2018b) on two DNN models (AlexNet and VGG16) on ImageNet. For a fair comparison, when using our own engine to generate optimal accelerators, we adopt the same precision and FPGA resource as the baselines. The results in Tab. show that the TRIPS generated accelerators outperform both SOTA expert-designed and tool-generated accelerators under the same dataset, DNNs, and FPGA resources. For example, the TRIPS generated accelerators achieve up to 2.16x increase in throughput on VGG16. The consistent better performance of our auto generated accelerators validates the effectiveness of our accelerator search engine in navigating the large and discrete design space of DNN accelerators to search for optimal DNN accelerators.
Table 5: TRIPS generated FPGA accelerators vs. SOTA FPGA accelerators built on top of Zynq XC70Z45 with 200 Mhz applied on different networks with fixed 16-bit on ImageNet.
Zhang 20186
Xino 2017
Qiu
2016
TRIPS generated
Zhang all, 201
TRIPS generated
Netwark
VoGtt
VGG16 723/900 DSP
Resource Utilication
680N00 DSP
$24900 DSP
780900 DSP
808/900 DSP
704800 DSP
Parlormance (GOPss
262
250
137
291
217
272
D VISUALIZATION OF SEARCHED NETWORK, PRECISION, AND ACCELERATOR.
Fig. 6 visualizes the searched network. precision, and accelerator achieved a 72.2% top-1 accuracy on ImageNet and 110 FPS on ZC706 FPGA. We conclude the insights below.
Insights for the searched network. We find that wide-shallow networks will better favor real device efficiency ZC706 FPGA while achieve similar accuracy. We conjecture the reason is that wider networks offer more opportunities for feature/channel-vi parallelism when batch size equals to one, leading to higher resource utilization and thus overall higher throughput.
14

---

Under review as a conference paper at ICLR 2021
k5 e1 g2 6bit
k5. 8bit
e1 g2 6bit
ks e 8bit
e3 6bit
k5. 8blt
k5 e. 6bit
k5. 8bit
kS. e6 8blt
K3 e3 6bit
k5. 8blt
k5. 8bit
k5. e3 6bit
k5. e6 6bit
K5. e3 8bit
k5. 8bit
skip
skip
skip
skip
skip
skip
Chunk1
Chunk2
Chunk3
Chunk4
Spatially line acrosa feature height and width
Spabally tiled across feuture height and width
Sputially tild across ripul output channels
Spunialy bied acrows ispui and output channels
Chunk5
Chunk6
DW Chunk1
DW Chunk2
Sputialy bildd Inpul and output channels
Spatially bied across input oudput channels
Spobillly lile across feature Teightt and width
Spetislly tlldd acrosa channets
Figure 6: Visualization of the searched network, precision, and accelerator that achieves a 72.2% top -1 accuracy ImageNet and 110 FPS ZC706 FPGA. The block definition follows (Wu al.
2019
Insights for the searched accelerator of TRIPS. The whole network is partitioned into multiple pipeline chunks to prioritize the resulting throughput, with each color representing Deterministically, two blocks with different quantization schemes will not be put into one chunk due to hardware constraints. Adopted from the design in (Shen et al., 2017), there is dependency of computation among blocks, so the pipeline chunks can take non-consecutive layers and better group the layers with similar dimensions together. Generally, observed in Fig. the chunks which take mostly the early blocks of the network favor spatially tiling the feature map height and width as offers more parallelism, while the later chunks tends choose the architecture tile channels as, after down-sampling parallelism opportunity is more prominent along channel dimensions.
chunk.
15


Reviews:
[{"review_id": "uZYcBOsX7-_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nThe authors present TRIPS, a Triple-Search framework to jointly search for the optimal network, precision and accelerator for a given task with max accuracy and efficiency in a differentiable manner. TRIPS focuses on efficiently exploring the large search space that previous reinforcement learning based solution cannot afford to due to poor scalability. \n\nThe authors propose a heterogeneous sampling approach that enables simultaneous update of weights and precision without biasing the precisions options or exploding the memory consumption. Further, they also formulate a novel co-search pipeline that enables a differentiable search for optimal accelerator.\n\nLastly, the authors present ablation studies and comparison with SOTA solutions that either solve the triple search problem described above or search (network, precision, accelerator) over a subset of its search space.\n\n##########################################################################\nReasons for score: \n \nThe paper presents a thorough description of TRIPS a joint search algorithm that enables scalable search of optimal network, precision and accelerator for maximum accuracy and efficiency. The experiments demonstrate the improvement in accuracy and efficiency compared to a wide range of previous solutions. Further ablation studies show breakdowns of accuracy and efficiency improvements derived from key enablers of TRIPS. With a few clarifications for the questions mentioned in the Cons section this paper should be accepted.\n\n \n##########################################################################\nPros:\n\n1.\tThis work presents a novel attempt to explore the large search space of network, precision and accelerator design, including detailed parameters such as tiling strategies, buffer sizing, processing elements (PE) count and connections etc.\n2.\tSection 2 presents a detailed overview of the past papers that have attempted to optimize all subsets of the search space explored by the proposed solutions.\n3.\tAuthors do a great job of either estimating or collecting data for comparing the proposed solution with the many different approaches discussed in Section 2.\n4.\tFigure 3 demonstrates good data establishing the superiority of TRIPS optimal solution compared with other baselines. It would be helpful if the authors could shed light on how they tune TRIPS to generate a range of different solutions that tradeoff max accuracy with max FPS.\n5.\tFigure 4 presents a useful comparison to previous solutions that do not optimize over precision space. TRIPS shows better accuracy and FPS even with fixed precision.\n6.\tAblation studies in section 4.3, are useful in breaking down the incremental improvement in the accuracy vs FPS tradeoff curve of TRIPS. \n7.\tAppendix sections provide useful details about TRIPS training and accelerator design space explored. Additionally, section C shows another useful contrast of TRIPS with previous solutions based on SOTA expert-designed and tool-generated solutions. Remaining sections provide useful insights on the searched space.\n \n##########################################################################\nCons: \n \n1.\tAuthors should consider citing the following in Section 2, DNN accelerators para, since it explores mapping network to accelerator theme of this section:\nGao, Mingyu, et al. \"Tetris: Scalable and efficient neural network acceleration with 3d memory.\" Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems. 2017.\n2.\tTowards the end of Section 2, authors mention that TRIPS selects network, precision and accelerator that enable better transferability to other tasks/applications. However, there are no results or discussion on transferability of TRIPS models to other applications, consider removing this line.\n3.\tWhile Section 3 is comprehensive in its detailed description of TRIPS implementation and key enablers. It is somewhat difficult to read, consider reordering the implementation discussion and Section 3.2/3.3.   \n4.\tSection 4.2, it’s not clear why the authors selected the hardware limitation of 512DSP units. \n5.\tFurther, Section 4.2, Co-exploration of networks and accelerators part, compares previous solutions that optimize network and accelerator while keeping the precision fixed (searched optimal value). It would be helpful if the authors added the precisions used for the previous work datapoints as well. Are the selected datapoints for previous papers, the max accuracy points? How do authors select the 16bit value they choose for Figure 4 fixed precision TRIPS?\n6.\tAdditionally, the authors do not present TRIPS-16bit datapoint for Figure 4c), please add that.\n7.\tThe analysis presented in Table 2 seems highly skewed, with almost 1000x improvement in Area and EDP. Knowing that the support for heterogenous functionalities is costing baseline ASIC implementations dearly makes the comparison unfair. I recommend that authors attempt to minimize the impact of such features, since they make the baseline ASIC accelerators more general which the proposed solution does not support.\n \n##########################################################################\nQuestions during rebuttal period: \n\nKindly address a few questions in the Cons section. \n\n#########################################################################\nSome typos: \n1.\tSection 1: “to optimize the acceleration efficiency for* a* given* DNN”\n2.\tSection 2: Hardware-aware NAS subsection, “acceleration efficiency, thus can lead* to sub-optimal solutions.”\n3.\tSection 2: Co-exploration/search techniques subsection: “Built upon prior art*, our”\n4.\tSection 4.3: Comparison with sequential optimization subsection: “and hardware side, a natural* design”\n5.\tAppendix D: Insights for the searched network subsection:  “find that while wide-shallow* networks”, reword the whole sentence.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "c1KXsnwtKCT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Not an expert but an interested outsider trying to exercise the best judgement.\n\nThe goal is ambitious and is something that lots of people want to solve, so it's certainly worth a stab at. The nice thing about the paper is that the evaluation is pretty solid, with FPGA implementations and simulations for the ASIC flow (not that those simulators are perfect, but they are at least widely-used in the community so using them is probably OK.)\n\n* Second paragraph on page 2, can you precisely define what you mean by \"path\" there?\n* What exactly is your hardware cost model, L_{cost}?\n* Could you give theoretical analysis as to why you optimization method will converge? It doesn't seem to be using a standard optimization method, which is perfectly fine, but then theoretically analyzing the proposed method would be important. Otherwise, the proposed method looks purely empirical, and is thus perhaps better suited for the CAD community.\n* What basic assumptions about the hardware are you making? Is it a GPU-like SIMD/SIMT architecture, or TPU-like systolic array, or eyeriss-like thing that relies on pretty beefy interconnect? After all, you have a hardware template with several knobs and are searching the parameters of those knobs, so showing the hardware template would make the paper more clearer.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUDwVwmxLJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Triple-Search for jointly optimizing neural network architecture, quantization policy, and hardware architecture. Specifically, it proposes a heterogeneous sampling strategy to tackle the dilemma between exploding memory cost and biased search. Besides, it also integrates a differentiable hardware search engine to support co-search in a differentiable manner. \n\nPros:\n1. Co-searching neural network architecture, quantization policy, and hardware architecture is practically important. \n2. Experiment results show clear improvements over the baselines. \n\t \nCons: \n1. I think this paper is not well motivated. I am not convinced that the challenge (dilemma between exploding memory cost and biased search) that this paper aims to address is a critical problem in practice. This challenge exists because the authors follow the differentiable NAS framework. However, for the chain-like design space studied in this paper, recent state-of-the-art NAS techniques [1,2] have shown that weight training can be decoupled from architecture search, and differentiable search is not a must in practice. In this paper, I do not find any discussions to support why we must need a differentiable AutoML framework for the co-searching task. \n2. The hardware part of the proposed framework is too short and not clearly explained. Please elaborate more on this part. \n3. The same co-searching problem has been studied in [3]. I think the authors should compare the proposed method with [3]. \n\nOverall, I think co-searching neural network architecture, quantization policy, and hardware architecture is an important task. The experiment results also look good. However, I find the motivation of having a differentiable framework is not well supported, the hardware part of the proposed framework is not clearly explained, and some closely related papers are missing. Therefore, I recommend \"Marginally below acceptance threshold\".\n\n[1] Once for all: Train one network and specialize it for efficient deployment. ICLR 2020.\n\n[2] Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420 (2019).\n\n[3] Neural-Hardware Architecture Search. NeurIPS 2019 Workshop on Machine Learning for Systems.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0SAn6hrto4V", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review:\nThis paper proposes Triple-Search (TRIPS), a differentiable framework of jointly searching for network architecture, quantization precision, and accelerator parameters. To address the dilemma between exploding training memory and biased search, the proposed framework leverages heterogeneous sampling where soft Gumbel Softmax is used for weight update and hard Gumbel Softmax is used for probabilities \\beta. To integrate accelerator search, hard Gumbel Softmax is used on hardware design choices and the overall hardware cost is used for penalization. Experiments are conducted on the FPGA platform for CIFAR and ImageNet dataset to show the superiority of TRIPS over NAS-only methods.\n\nPros:\n\n+ The idea of co-searching hardware architecture, neural architecture, and quantization precision is promising. \n\n+ It's nice to see the ablation study on sequential optimization and joint optimization. \n\n \nConcerns: \n\n- The paper is not well organized. Though TRIPS is a co-search framework, the paper mostly elaborates on the quantization searching. The modeling of hardware and the design space of the accelerator search are not well introduced.\n\n- The challenge of exploding training memory comes from the differentiable searching. However, the authors did not explain why the differentiable search is a must. The state-of-the-art NAS algorithms tend to first train a shared weight hyper net and then perform searching algorithms such as the evolutionary algorithm. In this case, the main challenge in this paper does not exist.\n\n- The comparison in Figure 3 may not be fair. The available hardware resources of the FPGA and mobile phones are different. It would be better if TRPIS results can be compared on the same hardware platform.\n\n- The following papers also search quantization precision during NAS. It would better if the paper can show the comparison between TRIPS and the following methods.\n\n[1] Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. Neural-Hardware Architecture Search. NeurIPS Workshop, 2019.\n\n[2] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481–497. Springer, 2019.\n\n- For CIFAR experiments in Figure 4, it would be better to show the improvement over fixed 8bit and fixed 4bit instead of 16bit.\n\n- For ASIC experiments, what is the accelerator design space? The results in Table 2 may not be fair. NASAIC uses MAESTRO as the estimation tool while TRIPS uses Timeloop as the estimation tool. If both NASAIC and TRIPS only search hardware parameters, does NASAIC and TRIPS search on the same hardware architecture (such as Eyeriss/NVDLA)?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a methodology for jointly optimizing neural network architecture, quantization policy, and hardware architecture. There are two key ideas:
- Heterogeneous sampling strategy to tackle the dilemma between exploding memory cost and biased search. 
- Integrates a differentiable hardware search engine to support co-search in a differentiable manner.

The paper tackles an important research problem and experimental results are good.

There are two related issues with this paper:
1. Comparison to one-shot NAS: one-shot NAS methods only need to train the super-net once and then can be applied to multiple use-cases, while the proposed methodology needs to be executed for each use-case.
2. It is not clear whether differentiable search is needed for this joint optimization problem modulo existing tools.

Overall, my assessment is that the paper is somewhat borderline and with some more work will be ready for publication. 
Decision: Reject