LEARNED HARDWARE/SOFTWARE CO-DESIGN OF NEURAL ACCELERATORS
Authors: Zhan Shi
Chirag Sakhuja
Milad Hashemi
Kevin Swersky
Calvin Lin
Venue:  (2021)
Keywords: deep learning accelerator
Bayesian optimization
design space exploration
hardware-software co-design

Abstract: The use of deep learning has grown at an exponential rate, giving rise to numerous specialized hardware and software systems for deep learning. Because the design space of deep learning software stacks and hardware accelerators is diverse and vast, prior work considers software optimizations separately from hardware architectures, effectively reducing the search space.  Unfortunately, this bifurcated approach means that many profitable design points are never explored. This paper instead casts the problem as hardware/software co-design, with the goal of automatically identifying desirable points in the joint design space. The key to our solution is a new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semi-continuous/semi-discrete. We evaluate our optimization framework by applying it to a variety of neural models, improving the energy-delay product by 18% (ResNet) and 40% (DQN) over hand-tuned state-of-the-art systems, as well as demonstrating strong results on other neural network architectures, such as MLPs and Transformers.

Under review as a conference paper at ICLR 2021
LEARNED HARDWARE/SOFTWAR CO-DESIGN OF NEURAL ACCELERATORS
Anonymous authors Paper under double-blind review
ABSTRACT
The use of deep learning has grown at an exponential rate, giving rise to numerous specialized hardware and software systems for deep learning. Because the design space of deep learning software stacks and hardware accelerators is diverse and vast, prior work considers software optimizations separately from hardware architectures, effecttvvly reducing the search space. Unfortunately, this bifurcated approach means that many profitable design points are never explored. This paper instead casts the problem as harfware/software co-design, with the goal of automatically identifying desirable points in the joint design space. The key to our solution is new constrained Bayesian optimization framework that avoids invalid solutions by exploiting the highly constrained features of this design space, which are semi- continuous/semi-diseeee
We evaluate our optimization framework by applying it to variety of neural models, improving the energy-delay product by 18% (ResNet) and 40% (DQN) over hand-tuned state-of-the-an systems, as well as demonstrating strong results on other neural network architectures, such as MLPs and Transformers.
INTRODUCTION
The compute requirements of deep learning are growing at a double exponential rate (Hernandez & Brown 2020), with more powerful models requiring exponentially more compute to train. This growth has been enabled by large systems of hardware accelerators, like GPUs and TPUs (NVIDIA) Jouppi al. 2017). However, the continued scaling of these systems is limited by issues of power density, cooling, and memory, so we need to improve computational efficiency.
Efficiency improvements can be sought at each layer of the deep learning stack, from better learning algorithms (Kingma & Bal 2014), to improved neural network architectures (Tan & Le 2019). to deep learning compilers (Chen et al. 2018), to specialized DNN accelerators that increase hardware efficiency (Chen et al. 2014a| 2016). In this paper, we focus on the low-level software and hardware portions of this stack, with the goal of automatically optimizing the energy X delay product of executing a particular model on hardware accelerator. We consider two components from the deep learning stack: the hardware accelerator and the software compiler that maps model onto that hardware. This area is commonly referred to as hardware/software co-design, and since it requires human expertise from multiple disciplines (software engineers, compiler writers, hardware architects), it is typically driven by manual heuristics or heuristic-based search (Yang et al. 20205).
We propose different approach, recognizing that for given DNN model, this hardware/software co-design can be framed as joint search of the space of all of the valid mappings and hardware architectures that can correctly execute the model. We formally parameterize this space based on prior work (Parashar et al. 2019). and we find that standard optimization techniques, including off-the-shelf Bayesian optimization, perform poorly because the design space gemi-discret, and the vast majority of the points in the space are infeasible. Prior work (Nardi et al. 2019) makes similar observation, noting that (1) complex constraints such as hardware area and energy budget limit the feasible parameter values (i.e. small feasibility set), (2) some constraints are unknown until after a sample point has been evaluated (i.e. unknown feasibility).
Our solution casts the search as bilevel optimization problem, shown in Figure I The outer loop optimizes over hardware architectures, while the inner loop optimizes over software mappings for a given architecture. Both of these are heavily constrained black-box global optimization problems

---

Under review as a conference paper at ICLR 2021
Hardiware Resouuress
Model
Computte
P PC
Stoergg
Bentlrr
Targer optinization
pick sarget sodel [e.g: Restett hardvrre budget
PS
Taanformtrr
for
number of harduere trials
① hardhar aptimizer penerates hardware b.
constrattt)ttt
(rputconstrantt
Target Modie
for 3 = 1:3 number Layars in target
cofteore optinitattinn Ioop
17 get Layer_fit
Harchware Optimizer
Hardware
Layerwise
for nuther software trials
software optimizer genorates masping Aup aspping *is
Softtware Optlimizer
Hardwore
Layen EBe optinizer
Layerulse
Hordware Feeeltock
HNY
Generate a Nappirne
Cont
4 compice Model_ 639 on as sure of best layerwise Epe Σ, min, and feed back to hardvere to optimizer
moael coP
constiito
Code
Figure I: Overview of BO-based nested search for hardware/software co-design.
that require expensive simulations to obtain performance estimates. We therefore propose a nested, constrained Bayesian optimization (BO) formulation that uses Bayesian models of hardware and software performance to guide the search towards promising regions of the design space. Our approach is extensible variety of different neural network architectures, and we make the code publicly available.
We find that when compared against the state-of-the-art manually-designee hardware accelerators that use heuristic software mappings, our BO-based approach provides significant improvements in the speed and energy efficiency of the resulting system, improving the energy-delay product (EDP) by 16.0% to 40.2% on series of neural networks. The key to our solution is our robust BO software optimizer, whose consistent efficiency allows our approach to scale to this huge search space.
This paper makes the following contributions:
We present the first system that automatically co-optimizes both the hardware architecture and software mapping phases of DNN accelerator design using principled and systematic search algorithm.
We present constrained formulation of hardware and software design for BO, a challenging problem given the high ratio (90%) of invalid hardware and software designs.
We present a nested hardware/softwar formulation of BO that extensible to other hardware accelerator designs.
We provide model-specific hardware and state-of-the-ar results multiple models.
2 A FORMAL REPRESENTATION OF SOFTWARE AND HARDWARE
Hardwarc/software co-design typically performed manually, but we believe that this vast design space is best navigated by an intelligent search process. To facilitate this automation, this section formally defines the hardware and software design spaces.
2.1 PARAMETERIZING THE DESIGN SPACE
Software design points can be parameterized by the loop ordering, loop tiling, and computational parallelism of the seven-level loop nest used to compute convolutional layer (see appendix), as has been noted by recent work (Parashar et al. 2019 Yang et al. 2020b). These software parameters are subject to hardware constraints, such as the quantity and layout of processing elements (PEs) and the size of storage elements.
Hardware parameters can be broken down into two broad categories:
Resource configurations represent the physical aspects of hardware, such as buffer sizes, tile sizes, and the cluster size of global buffers, as well as the layouts of the PE array and of the global buffer.
Dataflow configurations represent the usage of the PE array that are implemented in hardware, such as the blocking factors and degree of parallelism at the PE level, which also determines the communication patterns among PEs.
Figure shows two possible design points for ID convolution. Both design points tile and parallelize the channel (C) dimension. To the right of each component in the architecture a set loops that specifies the control logic for the component, which can be broken down into temporal streaming (for loops) and spatial distribution (parallel. for loops). For example, in the architecture on the left, the global buffer distributes across the PEs weight from 4 separate channels (c2), and the
2

---

Under review as a conference paper at ICLR 2021
PEs perform all operations that the weight participates in. In this design, all data reuse is captured within the PE, so the global buffer need not store anything. By contrast, the architecture on the right distributes single output element across the PEs to compute partial sums, which are stored in the global buffer across iterations. Both these design points consist of the same architectural components, but the dataflows vary, imposing different constraints the software. The appendix shows the details of the parameterization of a more practical 2D convolution.
Waighs
Curputt
1D Convolution
fon(ce=0; c9<255; cB1-64)
for (clre; 184256; cB-1128)
DRMM
DRAM
lempora
sperial
for(cte;; clc6d; cl+i4)
for(ci=e; cádi2;; ci-11)
For(n-e; rea; re-1)
for(pue; pc5; pent)
sarallel_for(c--d c204; (2e1))
parsliel_ c2<16; (2r-4)
for(rra; r<3; re-1)
for(p-a; pc5; out(n] In(ereet<e][[4r]]
for(caue; cie;; (le)) outipl anfe@cctachcallpenn M[shcctact4<][]]]
Pt
P2
vi <[cercI-c2](n]
Figure 2: Two architectures computing a 1D convolution.
2.2 CONSTRAINTS IN THE DESIGN SPACE
There are several reasons why the vast space of hardware and software parameters is filled with impractical or invalid design points. First, hardware designs are fundamentally constrained by area (the total amount of compute and storage resources) and factors such as available memory bandwidth. Second, the design cost and latency of additional area grow super-linearly (Shao et al. 2019), which leads to many impractical design points.
Software constraints generally governed by feasibility instead of practicality and predominantly depend on the hardware configuration and the specific neural network workload. For a specific hardware accelerator, there is limited number of available resources, so the software optimization problem can be viewed as a search for the most efficient use of hardware PEs and buffers. For example, the loop blocking optimization factors a neural network across multiple hardware storage buffers-and and the feasible factorizations are constrained by the size of the hardware buffers.
BAYESIAN OPTIMIZATION
3.1 OVERVIEW
Bayesian optimization (Jones al. 1998 Brochu et al., 2010 Shahriari et al. 2015) an effective approach for the optimization of expensive, possibly noisy black-box functions. BO has been used to optimize hyperparameters (Snock et al. 2012), configure algorithms (Hutter et al.] 201T), optimize A/B experiments (Letham et al. 2019). and more. For problem, we have parameterized representation and access to a simulator. Since one of our main concerns is sample efficiency, Bayesian optimization is particularly suitable.
The actual cost of evaluation depends on the experimental infrastructure, but in general, it is much more expensive to evaluate hardware design choice than to evaluate software optimizations, because hardware design can take hours (to produce hardware simulator or an FPGA) to days or even months (to produce an ASIC).
Bayesian optimization has two major components: (1) surrogate model provides a Bayesian posterior probability distribution that predicts potential values of the objective function. (2) an acquisition function uses the model to identify the next point to evaluate.
3.2 GAUSSIAN PROCESSES
common surrogate model is a Gaussian process (GP) (Rasmussen & Williams 2006) due to its simplicity and flcxibility. A GP is prior distribution over the space of functions comprised of a mean function m(x) and a covariance, or kernel function k(x, x'). Suppose we are given a dataset of N input/output pairs over bounded domain Ω with D input dimensions and scalar outputs. For brevity, we write this as (),y), where X E 22Nx and E IR. The posterior predictive distribution over function values of for new input x is given by P(f x, X, = ( (x)), where
n(x)= = Kxx K x (y mx) + m(x),
o² x((xx)
Kxx K x KTx-
3

---

Under review as a conference paper at ICLR 2021
Where X is matrix formed by evaluating the kernel on X, Kxx is the vector of kernel evaluations between x and X, and mx is the vector of mean function evaluations on the input dataset.
A common choice for the kernel is squared exponential. Given two input vectors x, and xj, this is defined as xx), a² exp (-1x)
x "axxxxll"" - a and l are kernel hyperparameters.
Another kernel that we find particularly useful is a linear kernel on top of explicit features. Given a feature mapping d(x) Rd' RK the linear kernel can be written as ((x,, x,) o(x,) "((xxxx))))))))))))>))) When we have strong prior information about the relevant feature interactions that govern the black-box function, this kernel allows to encode these interactions directly and produces more
sample-efficient posterior estimate.
In cases where the observations from the black-box function are noisy, can add a noise kernel Wnnise = - to Xxx, where 72 is hyperparameter. This implies a Gaussian observation likelihood.
Following common practice, we the constant mean m(x) hyperparameters are learned by maximizing the marginal likelihood of the GP on the current dataset.
A All kernel and mean
3.3 ACQUISITION FUNCTIONS
critical component in the BO framework the choice of acquisition function a(-) that assigns each design point a value that represents the utility of testing this point. Two commonly used acquisition functions are expected improvement (EI) and lower confidence bound (LCB).
El computes the amount we expect to improve upon the current best observed objective value y. max by evaluating design point x. Formally, can be written
(x)
max(y, - f.0) |xp((fxx,Xx x)d.f..f.
where f is the latent function from the surrogate model, and y. is the best value observed.
LCB (Srinivas et al. 2009) provides an explicit tradeoff between the predictive mean and variance and is defined as
@LCB (x) = [((x) + (x)).
Where À represents a tradeoff parameter. A small promotes greater exploitation, and a large promotes greater exploration. We found x 1 to work well in our experiments. Beyond these, there are many other possible acquisition functions that could be used in future exploration (Thompson
1933;
Hennig & Schuler 2012 Hemmnddez-Lbaard
al. 2014 Frazier 2009).
3.4 CONSTRAINTS
In our problem, the vast majority of the design space will produce invalid solutions. When the constraints are known function of the input features, we can directly account for them input constraints. Otherwise, we must run the simulation and treat invalid points using an output constraint. Here, we will describe these constraint types, and how they incorporated into BO.
Input constraints are explicit constraints that are used when optimizing the acquisition function. They directly prevent the search from suggesting points that will violate the constraints. As some constraints are non-linear, this optimization is itself very challenging, as it is a global optimization problem with non-convex constraints. the unconstrained case, maximizing the acquisition function often takes a hybrid approach: generating random initial set of points and refining them by gradient ascent. Maintaining feasibility with non-convex constraints is far more challenging, however.
We therefore optimize the acquisition function in simple way by performing rejection sampling on the design space: we randomly sample parameters until we obtain 150 feasible points, and then choose the one the maximizes the acquisition function. On average the sampling takes 22K random samples to get a pool of 150 feasible points. We have found that practically this is a simple yet leave more involved optimization schemes for future work.
effective strategy for our problems,
Output constraints are used when we do not know the form of the constraint a- a-priori and must run the simulator to test feasibility. This is also referred to an "unknown" constraint, and BO has been adapted to incorporate a constraint model in addition to the regression model (Gelbart et al. 2014). These simultaneously learn about the constraint boundaries while modeling the objective.
4

---

Under review as a conference paper at ICLR 2021
Let C(x) denote the event that x satisfies constraint C. Constrained BO uses Bayesian classifier to model ((C(x)). . is relatively straightforward to adapt a GP regressor to classification (Rasmussen & Williams 2006).
Under a Bayesian classifier, the acquisition function a(x) is modified to account for the probability that the constraint is satisfied, with 0 utility if it is not satisfied.
(xx) [[]())][[(x)]]
((C(x))a(xx.
Where [[]( is the indicator function that evaluates to 1 if the constraint is satisfied and O otherwise. We therefore maintain two models: one regression model capture the objective and one classifier to model the constraint in order to avoid evaluations in infeasible regions.
BAYESIAN OPTIMIZATION FOR HARDWARFF/SFFWARR CO-DESIGN
4.1 OVERVIEW OF NESTED HARDWARF/SOFTWARR OPTIMIZATION
Provided the constraints discussed in Section 2 and the BO formulation from Section we propose nested approach for co-optimizing bardwareesoftware parameters. The overall approach is outlined in Figure The goal is find the optimal hardware parameters for a neural model and the optimal set of software parameters for each layer in the neural model. Since software constraints depend on feasible hardware design, we first propose the hardware parameters, then for that hardware co-optimize the software mapping.
Specifically, let XĄ and x, denote the set of hardware and software parameters in the parameter space to be optimized. In the nested search process, we first use the hardware optimizer to generate design of hardware. In particular, we perform the hardware search in the space of possible hardware Sj to optimize all hardware parameters, where the objective is minimize fxx. NN) which we define as the energy-delay product (EDP) of running the neural network (NN) model on the given hardware, assuming the optimal software mapping for each individual layer. This step produces hardware specification and can be formalized as argmin (x,, NN).
For the chosen hardware design, framework performs the software search for each individual neural layer in its constrained software mapping space S,,,, NN; to optimize the mapping parameters, where NN, denotes the jth layer in the neural network model, and the objective becomes f(x, Xh. NN,), which is defined as the EDP of running the layer on the fixed hardware. This step produces design point that represents the best set of software mappings for each layers on the given hardware structure, and can be formalized as argmin seS, Jr. (x, xx).. The layerwise EDPs are then summed up as the EDP of the neural model, which is fed back to the hardware optimizer to generate the next hardware setting.
The iterative search between hardware and software will repeat for a user-defined number of trials. In this work, we set 50 for hardware search and 250 for software search. The combination of hardware and software that achieves the best EDP during the optimization process becomes the final model-specific hardware structure and layer-specific software mappings. A random sample used in the first iteration of both the hardware and software search. In our Bayesian optimization (BO) framework, we use separate BO models to search in the hardware and software space. We now describe their design considerations, particularly the choice of kernel and feature transformation.
4.2 BO FOR OPTIMIZING HARDWARE ARCHITECTURES
Kernel design. The main design choice for BO is the GP kernel to use. For the hardware search, we choose a linear kernel on top of feature transformations that represent the relationship between the different parameters. This feature transformation allows us explicitly encode domain knowledge. For example, by comparing the layout parameters of the 2D PE array and global buffer we can obtain the ratio between these adjacent storage layers, which correlates the maximal degree of parallel buffer accesses in each dimension. The details of the features are given Figure [3] in the appendix. We also add noise kernel deal with noise in the hardware evaluation. This is because the software optimizer is not guaranteed to find the best software mapping for each layer. There is some randomness in the software search, and therefore independent runs of software optimization for a fixed hardware design may yield different results.
Constraints. There are both known and unknown constraints in the hardware search. The known constraints, such as the compute and storage budget, are treated as input constraints that reject invalid samples. The unknown constraints have do with feasibility (if there exist valid software mappings
5

---

Under review as a conference paper at ICLR 2021
of neural layers onto the hardware, and if the valid mappings can be sampled during the software optimization). Following Section[3] these constraints are treated as output constraints and are modeled by GP with squared exponential kernel.
4.3 BO FOR OPTIMIZING SOFTWARE MAPPINGS
Kernel design. Similar to hardware optimization, we use a linear kernel and transform the parame- ters to features that encode relational information. As the hardware is fixed in the search of software mappings, we are able to compute features such as buffer usage, which potentially help make the predictions more accurate. The evaluation of mapping on a given hardware is deterministic in our infrastructure, thus there is no need for a noise kernel in the GPs.
Constraints. As both the hardware and neural model are known during software optimization, all constraints are known and are treated as input constraints that automatically reject invalid samples.
EVALUATION
5.1 METHODOLOGY
Infrastructure. We conduct our evaluation on Timeloop (Parashar et al. 2019), which is an open-source infrastructure for evaluating the hardware design and software optimization of DNN accelerators. Timeloop represents the key architecture attributes of DNN accelerators that realize a broad space of hardware structure and topology, which generate an accurate projection of performance and energy efficiency for DNN workloads. In the evaluation, Timeloop takes two inputs: I) the hardware configuration, which consists of the hardware-relater parameters, and 2) the software mapping, which consists of the software parameters that describe the mapping. As most accelerators designed for neural network inference, we limit the case to inference in this work and leave training for future work.
Workloads. To show that our solution automatically produces efficient hardware for variety of neural networks, we use our BO framework to optimize critical layers from CNNs (ResNet (He et al.) 2016) and DQN (Mnih et al. 2013)), well as an MLP and Transformer (Vaswani al. (017).
Experimental Sctup. We use Eyeriss (Chen et al. 2016), a state-of-the-an art DNN accelerator, as our baseline. All workloads are evaluated on the Eyeriss implementation with 168 PEs (Chen et al. 2016) except for the Transformer model, which runs on the larger version of Eyeriss with 256 PEs (Parashar al. 2019). In the software mapping search, use Eyeriss's hardware specifications and search for the best software mapping for each neural layer. In the hardware search, we perform the search under the same compute and storage resource constraints Eyeriss for each neural model.
Metrics. Hardware accelerators are designed to achieve both speed and energy efficiency, so we adopt the widely used energy-delay product (EDP) the objective. Since EDP values vary by an order of magnitude, we normalize by dividing by the best (minimal) EDP value and take the reciprocal for optimization curves. For the hardware/software co-design, improvements of each neural model, which is averaged across all layers (see Figure [1 and 12 in the appendix). For software mapping optimizations, we report the layer-wise EDP improvements.
report the EDP
Baselines. In hardware search, we compare against constrained random search that repeatedly takes the first random sample in the design space that satisfies the constraints. In software search, we use constrained random search, TVM with XGBoost and TreeGRU (Chen et al.) 2018), and out-of-the-boo BO that optimizes continuous parameter space and rounds to the nearest valid parameters.
Representation of the Search Space. Our representation uses the original values of the parameters (normalized by the downstream GPs), including high-level features such buffer and compute usage. We have tried a few other approaches to defining distance this space, but find that our current representation is the most effective one. For example, applied log transformations the tiling factor to transform the nonlinear constraints to linear, but the GP suffers greatly from the transformed space where the distance is skewed.
'This work focuses model-specific hardware, hut hardware specialization provides larger henefits at finer granularity, i.c. if different layers can execute on customized hardware. We leave this for future work.

---

Under review as a conference paper at ICLR 2021
5.2 SOFTWARE MAPPING OPTIMIZATION
We show the results of software mapping optimization first, as the capability of finding a good mapping is the base of evaluating a hardware design. Figure3Jshows the improvements of BO over our constrained random search formulation. Our BO formulation outperforms random search, both variants of TVM well as standard BO formulation that optimizes discrete parameters using relax-and-roumm approach.
68
3.6
0.6
TV4 (SCBoosc)
TVU (Tinede)))
TV4 (XCBoeuc
TVMl (Theeeii
6.4
0010gmi octsed
0.4
96 (Gam)
(sacelie)
(Raceibe)
901Buadmn
00 (Duadino
200 102 603 ato Number oftrads
1000
200 400
863
20n
430.
kin 1050
230 450
600
1000
Number oftraws
Number oftlns
Number toas
(a) K2
(b) DQN-K2
(c) MLP-K2
(d) Transformer K2
Figure 3: Software mapping optimization layer 2 of ResNet, DQN, MI.P, and Transformer. The y-axis shows the reciprocal of energy-delay product (EDP) (normalized against the best EDP value). Higher is better. Results for other layers can be found the appendix. Best viewed in color.
5.3 HARDWARE CONFIGUR ATION OPTIMIZATION
Figure shows Shows the optimization curves for hardware/software co-design. The comparison of hardware search algorithms shows that BO provides consistently better performance than the constrained random search, and the comparison of software search algorithms shows the importance of mapping optimization in the co-design process. As shown Figure Sa we find that the designs searched by BO achieve significantly better EDP on all neural models compared to the state-of-the-an manually designed accelerator (18.3%, 40.2%, 21.8% and 16.0% for ResNet, DQN, MLP and Transformer respectively).
Narmatzed EDP tecitroal
68
0.8
27
0.6
D0HNN: BOSWI
S0HIM: i BCIENN
200/01 BOSS
107-M1 BCKSH)
Sanderdrin BCASWI
Sandem//Mi ICEEM)
Sandem)/Mi 3015M)
kandom//MM S0150n
Randendren 500
Randon)/M)) Randen35W)
kandom)/n0) 3andon)5W
10
20
30
63
10 20
10
43
10
20
30
43
10 10
45
50
ResNet
(b) DQN
(c) MLP
(d) Transformer
Figure 4: Hardware/software co-optimization. The x-axis shows the number of trials for hardware search, and 250 attempts are made to find the optimal software mapping for each layer the model on the hardware specification. Best viewed in color.
5.4 ABLATION STUDIES
Surrogate models and acquisition functions. There exist popular variants for both the surrogate models and acquisition functions. In Figure[5b] we compare the surrogate models of Gaussian process (GP) with random forest (RF) and the acquisition functions of expected improvement (EI) and lower confidence bound (LCB). As shown, in the transformed feature space, GP generally performs better than RF, and LCB generally outperforms El.
Exploration vs. Exploitation. The LCB acquisition function explicitly balances exploration and exploitation with a hyperparameter A. To further study the impact exploration vs. exploitation in the use of LCB, we test LCB with different À values in Figure We find that LCBs with À values that are greater than 0.5 provide stable performance the optimization curves, while LCB with A 0.1 suffers from insufficient exploration.
5.5 ARCHITECTURAL INSIGHTS
To show that our automated design produce new architectural insights, provide qualitative comparison of Eyeriss with our solution for DQN. Our design predominantly differs in the shape of the PE array, well as in the number of memory buffers used. Eyeriss allocates the majority of its local buffer storage for filter weights, which are poorly utilized. Our design increases buffer utilization by storing multiple inputs and output elements.
7

---

Under review as a conference paper at ICLR 2021
Eyeros Searched
Normalized (reeprtccl
1.0
1.0
0.9
0.0
0.0
0.75
lamndd=00.1
0.50
0.5
GP+1CB
Normal 10
0.4
0.6
lambda=0.55
lambda-1.0
0.4
RF+EI
lambda=1,5
Constrained Random Search
0.2
lambds=2.0
0.00
Reshlet
DON
MLP
Transformer
800
800
1000
Number
Number trials
(a) Comparison with Eyeriss.
(b) BO ablation
(c) LCB ablation
Figure 5: (a): Comparison between the SOTA accelerator (Byeriss) and searched design. Results
EDPs normalized to Eyeriss, and lower is better. (b) Ablation studies on ResNet-K4. Higher results are better. (b) BO with different surrogate models and acquisition functions. (c) LCB acquisition function with different À values.
We can also plug our hardware configuration into the heuristic-based optimizer from prior work (Parashar et al.) 2019) and attempt to find a software mapping. We do this using the 12x 14 PE array from DQN. Timeloop's software optimizers are unable to find the same mapping that we do, with the best result being 52% worse than the baseline. This demonstrates the utility of a learned co-design approach that enables the software optimizer to be robust across different hardware architectures.
6 RELATED WORK
6.1 HARDWARE TO OPTIMIZE DNNS
Accelerators are specialized processors that provide significant performance and efficiency improve- ments by targeting specific workloads; they also typically require significant manual design. For deep learning, these primitives are often basic linear algebra subprograms (BLAS).
Prior work has designed specialized hardware execute BLAS kernels. Google's TPU (Jouppi et al. 2017) uses large hardware structures called systolic arrays (Kung & Leiserson 1979), and NVIDIA's GPUs have tensor cores (NVIDIA) 2017). DianNao (Chen al. 20144a) computes fully connected layers of neural network using multiply-accumulate trees, and its successor, DaDianNao (Chen ct al. 2014b), improves data locality for large neural networks by tiling processing elements (PEs) around on-chip eDRAM banks. A more specialized version, ShiDianNao (Du et al. 2015), focuses on dense convolutional neural networks (CNNs), which exhibit regular computational behavior with high data reuse. Eyeriss (Chen et al. 2016) and Eyeriss v2 (Chen et al.) 2019) also focus CNNs, introducing specific dataflow that exploits a reuse pattern exhibited by 2D convolutions. To improve performance scaling, Eyeriss v2 uses more sophisticated interconnect than its predecessor, and also introduces a variant that targets sparse CNNs. Prior work (Parashar et al., 2017: Zhang et al., 2016) has dealt with sparsity by suppressing zero-valued activations and storing and operating on compressed data. Many other domain specific architectures have been proposed to take ad adnataee local communication patterns (Farabet et al. 20TT). 3D-stacked bandwidth memory (Kim et al.) 2016: Gao et al. 2017). or multi-chip modules (Shao et al., 2019). Recent work (Yang et al. 2002bb) presents heuristics for automatically synthesizing hardware using a domain-specific language.
6.2 SOFTWARE TO OPTIMIZE DNNS
Software compiler optimizations for reducing the compute and storage requirements of neural networks include loop blocking (tiling), loop reordering, and loop unrolling ( (Whaley & Dongarral
1998; Mullapudi et al. 2016 Bondhugula et al.) 2008). Compilers such as TVM (Chen et al.) 2018) have used learned cost models to optimize execution efficiency. Similarly, Timeloop uses a grid or random search to optimize software mappings on user-specified hardware architecture (Parashar
al. 2019). However, all previous software optimizers treat hardware as a black box and ignore interactions between hardware and software.
6.3 NEURAL ARCHITECTURE SEARCH
Neural architecture search (NAS) is research area that seeks to automatically select the optimal neural network topology or architecture to select for each problem, often agnostic to the underlying hardware or software (Zoph & Le 2016). NAS is disjoint our work, lover-level bardware/sottware primitives, but an optimization approach that considers neural network architecture, software mappings, and hardware architecture could be future work.
seek to optimize
8

---

Under review as a conference paper at ICLR 2021
6.4 BI-LEVEL OPTIMIZATION APPROACHES
Recent research has started to optimize pairs of these categories jointly. For example, EDD et al. 2020) co-optimizes NAS and software compiler optimizations on embedded systems, but does not consider hardware architecture. Several prior papers have co-optimized NAS with hardware architecture (Jiang et al. 2020 Yang et al. 2020a; Lin et al. Abdelfattah et al. 2020). These works either fix the software compiler optimizations (Jing et al., 2020; Lin et al. Abdelfattah al,, (020) or tie the compiler optimizations to the hardware search (Yang et 202000) which simplifies the search problem, but the compiler optimizations cannot be customized to different workloads once the hardware is synthesized.
Ours is the first work that systematically co-optimizes the space of both hardware and software compiler optimizations. This larger search space requires principled search method, which motivates our constrained Bayesian optimization framework. Our nested search minimizes expensive hardware generation and allows us optimize the hardware for the entire model, but still optimize software mappings for each layer of the neural network.
CONCLUSION
In this paper, we have cast hardware/software co-design as Bayesian optimization problem. We have shown that standard mechanisms have difficulty navigating the complex, highly constrained design space, so we have presented novel constrained formulation that allows the optimizer to efficiently identify desirable points in this design space. The use of machine learning to automate hardware/software co-design opens many opportunities for future work. For example, transfer learning could dramatically reduce design time across designs and models. The techniques described here are not limited to DNN architectures, which is significant because as enter the golden age of computer architecture (Hennessy & Patterson, 2019). is essential that we develop automatic mechanisms for architectural exploration that quickly produce custom hardware accelerators.
REFERENCES
Mohamed S Abdelfattah, Lukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane. Best of both worlds: Automl codesign of a cnn and its hardware accelerator. arXiv preprint arXiv:2002.05022 2020.
Uday Bondhugula, Albert Hartono, Jagannathan Ramanujam, and Ponnuswamy Sadayappan. practical automatic polyhedral parallelizer and locality optimizer. In Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 101-113, 2008
Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application active user modeling and hierarchical reinforcement learning. 2010.
arXiv preprint
Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Learning to optimize tensor programs. In Advances in Neural Information Processing Systems, pp. 13399-3400 2018.
Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang. Chengyong Wu. Yunji Chen, and Olivier Temam. Diannao: a small-footprint tigh-throughhut accelerator for ubiquitous machine-lcarning In Proceedings of the 19th international conference Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 269-284, 2014a.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks, ACM SIGARCH Computer Architecture News, 44(3): 367-379, 2016.
Yu-Hsin Chen, Tien-Ju Yang. Joel Emer, and Vivienne Sze. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(2):992-308, 2019.
Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang. Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al. Dadiamnao: A machine-learning supercomputer. In 2014 47th Annual IEFF/ACM International Symposium Microarchitecture (MICRO), pp. 609-622 IEBE, 20140 4b.

---

Under review as a conference paper at ICLR 2021
Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo lenne, Ling Li, Tao Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam. Shidiannao: Shifting vision processing closer to the sensor. In Proceedings of the 42nd Annual International Sympostum on Computer Architecture (ISCA), pp. 92-104, 2015.
Clément Farabet, Berin Martini, Benoit Corda, Polina Akselrod, Eugenio Culurciello, and Yann LeCun. Neuflow: A runtime reconfigurable dataflow processor for vision. In Cypr 2011 Workshops, pp. 109-116. IEEE, 2011.
Peter Frazier. Knowledge -rrniggggggddenttt methods for statistical learning. PhD thesis, Citeseer, 2009.
Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, and Christos Kozyrakis. Tetris: Scalable and efficient neural network acceleration with 3d memory. In Proceedings of the Twenty-Secome International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 751-764, 2017.
Michael A Gelbart, Jasper Snoek, and Ryan P Adams. Bayesian optimization with unknown constraints. Uncertainty in Artificial Intelligence, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
John Hennessy and David Patterson. A new golden age for computer architecture. Communica- tions of the ACM, 62(2):48-60, 2019.
Philipp Hennig and Christian J Schuler. Entropy search for information-efficien global optimization. Journal of Machine Learning Research, 13(Jun): 13(Jun):1809-188 1809-1837 2012.
Danny Hernandez and Tom B. Brown. Measuring the algorithmic efficiency of neural networks, 2020.
José Miguel Hernández-Looato Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Advances in neural information processing systems, pp. 918-926, 2014.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, pp. 507-523. Springer, 2011.
Weiwen Jiang, Qiuwen Lou, Zheyu Yan, Lei Yang, Jingtong Hu, X Sharon Hu, and Yiyu Shi. Devicc-cireuit-archteee
co-exploration for computing-in computing-in-memorr in-memory neural accelerators. IEEE Transactions on Computers, 2020.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492 1998.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA), pp. 12, 2017.
Duckhwan Kim, Jaeha Kung, Sek Chai, Sudhakar Yalamanchili, and Saibal Mukhopadhyay. Neu- rocube: A programmable digital neuromorphic architecture with high-density 3d memory. ACM SIGARCH Computer Architecture News, 44(3):380-33) 392. 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412,694, 2014.
HT Kung and Charles E Leiserson. Systolic arrays (for vlsi). In Sparse Matrix Proceedings 1978. volume 1, pp. 256-282. Society for industrial and applied mathematics, 1979.
Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy, et al. Constrained bayesian optimization with noisy experiments. Bayesian Analysis, 14(2)495-519, 2019.
10

---

Under review as a conference paper at ICLR 2021
Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. Edd: Efficient differentiable dnn architecture and implementation co-search for embedded ai solutions. arXiv preprint arXiv:2005.02563, 2020.
Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. search.
architecture
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Ravi Teja Mullapudi, Andrew Adams, Dillon Sharlet, Jonathan
and Kayvon Fatahalian. Automatically scheduling halide image processing pipelines. ACM Transactions Graphics (TOG), 35(4):1-11, 2016.
Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecomnunication Systems (MASCOTS), pp. 347-358. IEEE, 2019.
NVIDIA. NVIDIA Tesla V100 GPU Architecture, The World's Most Advanced Data Center GPU. NVIDIA Corporation, 2017.
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen Keckler, and William J Dally, Scnn: An accelerator for compressed-sparse convolutional neural networks. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA), pp. 27-40, 2017.
Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emer. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 304-315. IEEE, 2019.
Carl Edward Rasmussen and Christopher K.I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matthew Fojtik, Nan Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, and Priyanka Raina. Simba: Scaling deep-learning inference with multi-chip-man module-based architecture. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2019.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3993 2009.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2019.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285 1933.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
R Clinton Whaley and Jack Dongarra. Automatically tuned linear algebra software. In SC'98: Proceedings of the 1998 ACM/EEE conference Supercomputing, pp. 38-38. IEEE, 1998.
11

---

Under review as a conference paper at ICLR 2021
Lei Yang. Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration neural architectures and heterogeneous asic accelerator designs targeting multiple tasks. arXiv preprint arXiv:2002.04116, 2020a.
Xuan Yang, Mingyu Gao, Qiaoyi Liu, Jeff Setter, Jing Pu, Ankita Nayak. Steven Bell, Kaidi Cao, Heonjae Ha, Priyanka Raina, et al. Interstellar: Using halide's scheduling language to analyze dnn accelerators. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 369-383, 2020b.
Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li, Qi Guo, Tianshi Chen, and Yunji Chen. Cambricon-x: An accelerator for sparse neural networks. In 2016 49th Annual IEEE/ACM International Symposium Microarchitecture (MICRO), pp. 1-12. IEEE, 2016.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1rXiv:6611.0157, 2016.
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
PARAMETERS AND CONSTRAINTS
Type
Index
Hardware Parameters
Valid Range
Mearing
H1
PE mesh-X
Factors of PEs
Decide arrangement of the 2-D PE array.
PF
H2
PE mesh-Y
Factors of R PES
Input Local
local buffer entries
H3
buffer
Decide the partit on of local buffer, partition leads sub buffers with inflexible sires. This useful as the latency to access each smaller sub-buffer decreases.
Local
weights entries in Local buffer
O to # local buffer
H4
buffer
entries
outputs Local
local buffer entries
HS
buffer
H6
Global buffer instances
Factors of APEs
Determine the arrangeme nt global buffec, and connection between global buffer and PE's local buffer (Local buffer of PEs along the X-axis shares the global buffer along X-axis].
Global buffer
FF-mmss--X
HB
Global buffer mesh-Y
Factors PE-mesh-Y
buffer
Determines the width of global buffer entry
H9
Global buffer block
Factors 16
Determines of the number of wider structures where multiple entries are ganged into
H10
Global buffer cluster size
Factors of 16
Datallow option of filter width
Options determine filter width PE's local buffer
H11
1,
Cataflow
Detallow option of filter height
Options that determine the size of filter he igtt local buffer
H12
Figure 6: Hardware parameters.
Type
Hardware Constraints
PE
mesh-X (H1) mesh-Y [H2] PEs
Local buffer
The sum of local sub: buffers (H3, He, HS) does not exceed buffer size
Global mesh-X (H7)
(H8)
instances (9)
Local buffer & global buffer (unimawn)
valid software mapping exists depending ma nyy on local bufeer partition (H3, H4, and glahal buffer arrangement
HR)
Figure Hardware constraints.
Type
Index
Software Parameters
Valid Range
Meaning
of
Determines
52
Blocking factors of
Factors of
(paralleliam) of each type of data (irputs, weights and outputs) each storage layer (except those that the hardware dataflow).
Loog blacking and engres parailelism
Blocking
Factors
54
Blocking lactors of Q
Factors
55
Blocking factors of C
Factors
56
Blocking factors of K
Factors
S/
Loop order local buffer
Permutations of factors
Affects the of type of data (inputs, wrights outputs) each layer.
Permutations of Permutations of non-1 factors
Laop rearder
S8
Laop order global bufler
Loop order
Figure 8: Software parameters.
B HYPERPARAMTERS FOR BO
In Figure T0
report the hyperparamters for BO.
13

---

Under review as a conference paper at ICLR 2021
Type
Software Constraints
Product all blocking factors of equals Rof the noural layer
Product all blocking factors of (92) equals of the target noural layer
Loop blocking dogrez of parallelism
Product all blocking factors of (S3) equals P of the target neural layer
Produutt blocking factors 154) equals Q the target neural layer
Product of all blocking factors of (S5) equals of the target neurel layer
blacking
equals of target neural layer
Buffer capacity (local)
inp.iss/weightt/outpt. sites (51-56) cannot exceed comesponding local sub-buffer capacity
Buffer capacity (glcbal)
(S1 does exceed global buffer capacity Product of blocking factors in global buffer X-ax (51-56) cannot exceed PFs in X axiss
Size all
Product blocking factors goobal buffer (51-58) cennot exceed total PEs
Figure 9: Software constraints.
number of independent trials
(HW), 10 (SW)
number random points
50 (HW), 150 (SW)
number of warmup data points (HW), 30 (SW)
number of samples for
1000
lambda for LCB
1.0
Figure 10: Hyperparamters for BO.
C NEURAL MODEL SPECIFICATIONS.
In Figure and Figure 12 we report the specifications neural models benchmarked in this paper.
D PARAMTERIZATION OF 2D CONVOLUTION
Listing 14 gives the seven-level nested loop that comprises 2D convolution.
Figure 17 shows design point for the CONV4 layer of ResNet. The architecture components are again the same as in the ID example, but since the memory footprint is significantly larger, the PE can no longer capture all data reuse, so the Global Buffer must store large portions of the inputs and outputs.
E EXAMPLE PARAMETER VECTOR
Below example vectors of hardware and software parameters BO optimizes.
F ADDITIONAL RESULTS
SOFTWARE OPTIMIZATION
In Figure 18 we show more examples of the software optimization over multiple layers of the different architectures. Our Bayesian optimization formulation consistently outperforms the baselines (Chen
et al. 2018)
F.2 ABLATIONS
In Figure 19 compare different surrogate models and acquisition functions for Bayesian optimiza- tion of the software mapping. We found Gaussian processes with LCB to consistently outperform other alternatives.
14

---

Under review as a conference paper at ICLR 2021
Model
Layers
Specifications
Filter size: 3x3
Output size: 56x
ResNet-K1
# input channel: 64
# output channel: 64
Stride: 2
Filter size;
Output size: 28x28
ResNet-K2
input channel: 128
# output channel: 128
Stride:
ResNet
Filter size: 3x3
Output size: x4x14
ResNet- K3
input channel: 256
output channel: 256
Stride:
Filter size: 3x3
Output size: 7x7
ResNet-K4 K4
# input channel: 512
output channel: 512
Stride:
Filter size: 8x8
Output size: 20x20
DQN-K1
input channel:
# output channel: 16
Stride:
DQN
Filter size: 4x4
Output size: 9x9
DQN-K2
# input channel: 16
# output channel: 32
Stride:
Figure 11: Specifications of ResNet (ResNet-18) (He et al 2016) and DQN (Mnih et al. 2013)
In Figure 20 we investigate the robustness of I.CB software optimization using different values of X. We found that À = 0.1 tends to be too greedy, but that above À 0.5, LCB tends to be fairly robust.
15

---

Under review as a conference paper at ICLR 2021
Model
Layers
Specifications
d Cous: 512
MLP-K1
MLP
d dour: 1024
MLP-K2
modee 512
d,= 32
Transformer-Ke K1
d, = 32
16
madel 512
d, 64
Transformer-Ko K2
=
h 8
Transformer
dade 512
d, 128
Transformer-K
d 128
h= 4
dood 512
d, 512
Transformer-Ke
d 512 h=1
Figure 12: Specifications of MLP and Transformer Vaswani et al. 2017
Model
Feature
Description
mesh r_taoo
The ratio of PE and global builer x-axis The ratio of PF arryy and global buffer along y y-ixss
Hardware
messh_y_ratio
input_bufie btfffe__uaageeeeee
input (local) buffer weight data size inputt [local] buffer size output data site / input (local) buffer size global buffer
input
weight buffer_ usage
usagge
outout_ haapt__fffe_reeeeeeeeeeee _sseer
Soltware
Latal_bbfffer_usss_ ussage
parailelism, ratio_x
used paralleliami sm available parallelism in the w-aiss of global butfer paralleli parallelism the y glabal buffer
parailelism_raate _rtio_y
Figure 13: Extra features used by the hardware and software BO optimizers.
for n in (O:N)
for k in [O:K)
for r in [O:R)
for in [0:S)
for in (0:P)
for q in (0:Q)
for in (0:C)
outputs (n][k]lll]]
+= weights [ s][r]
inputs [ ][p+r]
Figure 14: Computing a 2D convolution with a seven-level nested loop.
16

---

Under review as a conference paper at ICLR 2021
Convolution
For(ctuts (BCS6;; (Bed128)
forbeet; kG25; ted))
Rr2
fartcied; peida; p-i))
Rosss
forted: gud;;
forppet pdta and))
fortnee; me)) for(chel) cich (het)
Tempprs
spatall
Figure 15: An architecture computing the CONV4 layer of ResNet.
Index
Type
Range of Values
int
Factors of 256
Factors of 256
3
int
0-220 (total local buffer size)
0-220 (total local buffer size]
5
int
0-220 (total local buffer size)
Factors of 168
int Factors of H1
Factors
int
Factors of 16
10
int
Factors 16
11
categorical
0,
12
categorical
0, 1
Figure 16: An example vector of hardware parameters. Please refer to Figure 6 for more detailed descriptions.
Index
Type
Range of Values
1-2
int
Factors of 3
3-4
int
Factors of 3
5-6
Factors 28
7-9
int
Factors of 28
10-12
int Factors 128
13-17
int Factors of 128
18
categorical
0-1
19
categorical
0-5
20
categorical
0-1
21
categorical
0-1
22
categorical
0-23
Figure 17: for more detailed descriptions. In this example, parameters 1-17 correspond row-wise to S1-S6 respectively, parameters 18-20 correspond to S7, and parameters 21 and 22 correspond to S8 and S9 respectively.
An example vector of software parameters (with ResNet-K2). Please refer to Figure
17

---

Under review as a conference paper at ICLR 2021
Nomalized (nciproca)
Normaizze EDP (neciproca)
Normalized EDV (rec mncceroca
0.8
0.8
0.6
0.6
TVN (XGBoost)
IXGBBesti
(XEBcost)
TVW (TreeGRUl
TVM (TreeSRUI
TVM (TreeGRU)
80 [Baselina]
80 (Baseline)
BD (Baseline
Constrained Random Search
02
Conntrained Bandom Search
Coral rained Bendom Search
400
Number trall
Numbor
(a) ResNet-K1
(b) ResNet-K2
(c) ResNet-K3
(raciprec30)
Normalzze EDP (reciproca))
Normell fzed EDP (reciorocal)
0.8
0.8
(Durs]
0.6
8D (Ours]
0.4
TVN (TreeGRUt
TVW DXGBoost)
0.4
TVM (TrecGRU)
TVM IXGBoest!
TVM (XGBeost)
TVM (TreeGRU)
Romm
[Baselinel
(Bosolinc)
(Baseline)
02
0.2
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Number trials
Number trall
Number of tnais
(d) ResNet-K4
(e) DQN-KI
(f) DQN-K2
1.0
Normalized EDP (recipraca)
1.0
Normalizen liee EDP (recipraca)
1.0
0.8
0.8
0.8
0.6
0.6
0.4
0.4
(XGBoadt)
(TreeGiU)
(TreaGRU)
0.2
(Baseline
0.2
8D (Buselin))
0.2
(flese
Constroincd Random Search
Constrolnoo Randem Search
Corisrainas nad Rondoms Search
200
400
600
800
1000
200
400
600
800
1000
6
200
460
600
1000
Number trials
Number of tnais
Number of trials
(g) MLP-KI
(h) MLP-K2
Transformer- K1
Normalizee (recorozc))
2.0
Normalized EDP (recipoocal)
1.0
Normelizzed EDP (recignocel)
1.0
0.6
BD 10ust
80 (0uns)
80 (Oum)
0.4
TVN D0G8aostt
0.4
TVM (XGBosst)
0.4
TVW DCGBoost)
TVN (TreERUU)
TV0 (TreeGRU)
TVM (TreeGRU)
BD (Bapelin))
80 (Baseline)
0.2
90 (Base line
Constrained Random Search
Constrainos Random Search
Consira ned Random Search
200
600
800
Number
Number
Number trials
(i) Transformer-K2
Transformer-K:
Transformer-K4
Figure 18: Software mapping optimization ResNet, DQN, MLP, and Transformer. The Y-axis
shows the reciprocal energy-delay product (EDP) (normalized against the best EDP value). Higher better.
1.0
Normalized EDP red corocall al
1.0
1.0
0.9
0.9
0.8
0.8
0.8
GF+LCB
Normalized
0.6
Normalices
0.5
RF+8
RF+Ei
0.4
RF+EI
0.4
Constrained Random Search
Constrotned Random Search
0.3
Condtrained Randem Search
Number
Number
(a) ResNet-K2
(b) ResNet-K3
(c) ResNet- K4
Figure 19: GP with different surrogate models and acquisition functions.
18

---

Under review as a conference paper at ICLR 2021
Normalized Hasmalzzed EDP (eedpacci)
1.0
0.8
0.8
0.7
lombda-0.1
1mbdo=00.1
lambds-0.1
hormalnoe
0.6
0.5
lambda-0.
ambda=0
0.5
lambda-0.3 0.5
lambdam1.
1.5
ambdam 1.5
lambda= 1.5
03
1am
0.2
2.0
0.3
lambda-2.0
200
400
600
800
1000
200
400
600
800
1000
200 400
600
800
1000
Number trials
Number of trail
Number of brats
(a) ResNet-K2
ResNet-K3
(c) ResNettKK4
Figure 20: LCB acquisition function with different lambda values.
19


Reviews:
[{"review_id": "OXk69LIQ5lA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents a method for hardware and software co-design using Bayesian optimization method. It is based on previous work of Timeloop\n\nThis paper seems a just an application of Bayesian optimization on the Timeloop framework. It searches using set parameters that are tunable by using some optimization method. Nevertheless, the paper shows improvement upon state-of-the-art neural networks such as transformers.\n\n##########################################################################\n\nReasons for score: \n\nThe paper shows improvement upon state-of-the-art neural networks such as transformers.\n\nThe paper being first system that co-optimizes both the hardware and software for DNN, which is dubious given the pletora of NAS and HAS papers.\n\n##########################################################################\n\nPros: \n\n- Demontrated improved energy delay design on different workloads\n\n##########################################################################\n\nCons: \n\n- The paper needs to revise its claim of being the  first system that co-optimizes both the hardware and software for DNN\n\n- Evaluate using different machine learning optimization methods such as RL and Evolutionary algorithms\n\n##########################################################################\n\nQuestions during rebuttal period: \n\n \nPlease address and clarify the cons above \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GJ1y-WjYemv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to optimize the co-design of code mapping and hardware configuration for neural accelerators. The method is based on a two-level Bayesian optimization (BO), with each BO optimizing the code mapping and hardware configuration, respectively. The design space of both code mapping and hardware configuration contains many complex constraints, some of which remains unknown until running expensive simulation. The obvious constraints are handled using rejection sampling and the unknown constraints are modeled by a Baysian classifier. This method is validated by experiments to optimize several network layers (ResNet, DQN, MLP, transformer) and achieves better performance than baselines.\n\nStrength of the paper: this paper aims at solving a meaningful problem: SW/HW co-design of code mapping and neural accelerators. It adopts a simple optimization method (BO), and according to the experiments, it achieves good performance gain compared with manual design and other baselines methods. \n\nWhat is unclear to me: I am not familiar with the literature around code-mapping and hardware configuration co-design, so I don't have enough context to judge if the proposed method is novel, compared with previous methods.\n\nWeaknesses of the paper: \n1/ this paper spends a lot of texts explaining the basics of BO, but what's missing to me is the representation of software design-points and hardware configurations. The usage of BO is based on the assumption that we have a good representation of the design space where the distance of each design point is clearly defined. For example, for two nested for-loop with different orders, how they are represented and what's the distance between the two representation? Is the representation canonical? What's the meaning of distance in the representation space? Without understanding these questions, it is difficult for me to see whether BO is the right optimization method for this. \n2/ In the experiment section, the method is only used to optimize one layer of a network, however, it is not clear how general this is as people usually care about the overall performance of a network or even a wide variety of networks. The experiments did not show if the proposed method is effective or general enough to optimize the overall performance. \n\nI hope authors can address the concerns. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DEPnOZ2IuAZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: \nThis paper proposes a Bayesian optimization framework for exploring the hardware configuration and compute mappings on generated hardware. Given a target NN model, it first generates hardware configuration candidates. Then for each hardware candidate, it generates corresponding mapping candidates of each NN layer. These candidates are generated with explicit constraints, i.e., it will randomly sample hardware/mapping until it obtains the desired number of feasible candidates. The evaluated EDP is fed into the Bayesian optimizer of mapping and hardware separately in sequential. The main contribution of this paper is a principled and systematic pipeline for co-searching the hardware parameters and mappings. Results are shown on the task of ResNet, DQN, MLP, and Transformer on Eyeriss architecture. It improves EDP by 16% to 40%.\n\n \nPros: \n\n+ The solution framework for co-searching hardware parameters and software mappings is neat. \n\n+ The results section is well structured.  It's nice to see the comparison between the proposed method and TVM; and the ablation study on whether to co-search hardware and software. \n\n\nConcerns: \n\n- The key concern about the paper is the lack of novelty. There are increasing works on hardware search for neural networks (see below), while this paper doesn't mention or compare to any of those.\n\n- The related work lacks work on hardware search, such as\n[1] Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. EDD: Efficient differentiable DNN architecture and implementation co-search for embedded AI solutions. arXiv preprint arXiv:2005.02563, 2020.\n[2] Weiwen Jiang, Qiuwen Lou, Zheyu Yan, Lei Yang, Jingtong Hu, X Sharon Hu, and Yiyu Shi. Device-circuit-architecture co-exploration for computing-in-memory neural accelerators. IEEE Transactions on Computers, 2020a.\n[3] Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous ASIC accelerator designs targeting multiple tasks. arXiv preprint arXiv:2002.04116, 2020.\n[4] Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. Neural-Hardware Architecture Search. NeurIPS Workshop, 2019.\n[5] Mohamed S Abdelfattah, Łukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane. Best of both worlds: Automl codesign of a CNN and its hardware accelerator. arXiv preprint arXiv:2002.05022, 2020.\n\n-  The results are not conclusively in favor of the proposed method. Why do authors adopt Bayesian optimization instead of evolution or reinforcement learning? How is Bayesian optimization reasonable for searching loop reordering? There is no discussion of the choice of optimization algorithm from the results. \n\n- On the same note, the results would have been more complete if experiments can be done on multiple hardware architectures such as NVDLA, ShiDianNao, and so on.\n\n- It would be better if the paper can elaborate on Section 2 in more detail. How does the proposed framework represent hardware and mapping? What is the range of each dimension? How large is the design space? What is the sample efficiency of the proposed method?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BzmjL_12UtP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to co-optimize the software and hardware for DNN executions to maximize the energy-delay product. There are three main contributions: 1) propose a formal representation of software and hardware that facilitate the search process; 2) propose a Bayesian optimization framework; 3) propose to search the hardware and software space separately and the search process is optimized by leveraging the Gaussian process model. Below are some questions and concerns:\n\n1:  the representation of software basically considers loop ordering, loop tiling, and computational parallelism, while the hardware representation is focused on available hardware resource and dataflow configuration. However, the biggest bottleneck of DNN processing actually comes from the main memory or large buffers. Compared with computing itself, data access normally consumes 2 orders of magnitude more energy and latency. Could the authors explain if they consider data access during their search process and how they famulated in the representation?\n\n2: the search goal of this work is to minimize EDP. I wonder is it flexible for this framework to change this goal to fit different designs? For instance, is it possible to search a software/hardware co-designed model for extremely low-power applications? And how can the proposed framework adopt for different search goals?\n\n3: the evaluation is only performed with previous work Eyeriss. Given the abundant accelerator with various design goals, more evaluations against more counterparts are needed. \n\n4: for the comparison results against Eyeriss, the authors show that their results outperform Eyeriss in EDP. But I am also curious about the detailed energy and performance comparison. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of hardware and software co-design for neural accelerators. Specifically, it looks at hardware and the software compiler that maps DNN to hardware. It employs Bayesian Optimization (BO) to perform joint search over hardware and software design parameters in an alternating manner. To handle black-box constraints that cannot be evaluated without performing simulations, the method uses constrained BO algorithms. 

The paper talks about two technical challenges:
1) Black-box constraints. There is a lot of literature on constrained BO.
2) Semi-discrete design variables. The paper didn't propose any generic solution. There are some recent papers to handle mixed variables that may be useful.
https://arxiv.org/abs/1907.01329
https://arxiv.org/abs/1906.08878

BO methodology is justified. There is recent work on hardware and software co-design for neural accelerators and should be taken into account for both qualitative and quantitative comparison. 

Overall, my assessment is that the paper in its current form lacks technical novelty for acceptance.
Decision: Reject