Data-Driven Offline Optimization for Architecting Hardware Accelerators
Authors: Aviral Kumar
Amir Yazdanbakhsh
Milad Hashemi
Kevin Swersky
Sergey Levine
Venue: ICLR 2022 Poster (2022)
Keywords: computer architecture and systems
machine learning
data-driven optimization

Abstract: To attain higher efficiency, the industry has gradually reformed towards application-specific hardware accelerators. While such a paradigm shift is already starting to show promising results, designers need to spend considerable manual effort and perform large number of time-consuming simulations to find accelerators that can accelerate multiple target applications while obeying design constraints. Moreover, such a simulation-driven approach must be re-run from scratch every time the set of target applications or design constraints change. An alternative paradigm is to use a data-driven, offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations. Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes. In this paper, we develop such a data-driven offline optimization method for designing hardware accelerators, dubbed PRIME, that enjoys all of these properties. Our approach learns a conservative, robust estimate of the desired cost function, utilizes infeasible points and optimizes the design against this estimate without any additional simulator queries during optimization. PRIME architects accelerators---tailored towards both single- and multi-applications---improving performance upon stat-of-the-art simulation-driven methods by about 1.54x and 1.20x, while considerably reducing the required total simulation time by 93% and 99%, respectively. In addition, PRIME also architects effective accelerators for unseen applications in a zero-shot setting, outperforming simulation-based methods by 1.26x.

Published as a conference paper at ICLR 2022
DATA-DRIVEN OFFLINE OPTIMIZATION FOR ARCHITECTING HARDWARE ACCELERATORS
Aviral Kumar Amir Yazdanbakhshh
Milad Hashemi Kevin Swersky
Sergey Levine*
Google Research * UC Berkeley
( E'uual Contribution)
Barrakkkbekeeee ey. ayazdan@googles com
ABSTRACT
To attain higher efficiency, the industry has gradually reformed towards application- specific hardware accelerators. While such a paradigm shift already starting show promising results, designers need to spend considerable manual effort and perform large number of time-consuming simulations to find accelerators that can accelerate multiple target applications while obeying design constraints. Moreover, such a "simulation-drive" of target applications design constraints change. An alternative paradigm is to use a "data-driven", offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations. Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes. In this paper, we develop such data-driven offline optimization method for designing hardware accelerators, dubbed PRIME, that enjoys all of these properties. Our approach learns a conservative, robust estimate of the desired cost function, utilizes infeasible points and optimizes the design against this estimate without any additional simulator queries during optimization. PRIME architects accelerators-tal tailored towards performance upon star-of-the-
approach must be re-run from scratch every time the set
both single- and intii-pppiiaaion-iimproree art iililatioondrivee methods by about 1.54x and 1.20x, while considerably reducing the required total simulation time by 93% and 99%, respectively. addition, PRIME also architects effective accelerators for unseen applications in zero-shot setting, outperforming simulation-base methods by 1.26x.
INTRODUCTION
The death of Moore's 's Law [11] and its spiraling effect on the semiconductor industry have driven the growth of specialized hardware accelerators. These specialized accelerators tailored to specific applications [64, 47, 41, 53). To design specialized accelerators, designers first spend considerable amounts of time developing simulators that closely model the real accelerator performance, and then optimize the accelerator using the simulator. While such simulators can automate accelerator design, this requires a large number of simulator queries for each new design, both in terms of simulation time and compute requirements, and this cost increases with the size of the design space [65, 53, 25]. Moreover, most the accelerators in the design space are typically infeasible [25, 64] because of build errors in silicon or compilation/maappng failures. When the target applications change or new application added, the complete simulation-driven procedure generally repeated. To make such approaches efficient and practically viable, designers typically "bake-in" constraints otherwise narrow the search space, but such constraints can leave out high- nigh-performing solutions [9, 44, 7].
alternate approach, proposed in this work, devise a data-driven optimization method that only utilizes database of previously tested accelerator designs, annotated with measured performance metrics, to produce new optimized designs without additional active queries to an explicit silicon or cycle-accurate simulator. Such a data-driven approach provides three key benefits: (1) it significantly shortens the recurring cost of running large-scale simulation sweeps, (2) alleviates the need to explicitly bake in domain knowledge search space pruning, and (3) it enables data re-use by empowering the designer to optimize accelerators for new unseen applications, by the virtue of effective generalization. While data-driven approaches have shown promising results in

---

Published as a conference paper at ICLR 2022
Optimizer,
Optimizer,
One- Time
Conservative Leanned Model;
Optimized
Collected Dataset
Optimizers
Hardware Accelerators
(TPU, NVDLA, ShiDianNao)
Figure 1: Overview of PRIME. We one-time collected datasct of prior accelerator designs, including TPU-style [65], NVDLA-style [42], and ShiDianNao-style [10] accelerators to train conservative surrogate model, which is used to design accelerators to meet desired goals and constraints.
biology [14, 5, 57], using offline optimization methods design accelerators has been challenging primarily due to the abundance of infeasible design points [64, 25].
The key contribution of this paper a data-driven approach, PRIME. to automatically architect high- performing application-specinn accelerators by using only previously collected offline data. PRIME learns a robust surrogate model of the task objective function from an existing offline dataset, and finds high-performing application-speccfic accelerators by optimizing the architectural parameters against this learned surrogate function, as shown in Figure 1. While naively learned surrogate functions usually produces poor-performing. oul-of-distribution designs that appear quite optimistic under the learned surrogate [35. 5, 57]. The robust surrogate in PRIME is explicitly trained to prevent overestimation "adversarial" designs that would be found during optimization. Furthermore, in contrast to prior works that discard infeasible points [25, 57], our proposed method instead incorporates infeasible points when learning the conservative surrogate by treating them additional negative samples. During evaluation, PRIME optimizes the learned conservative surrogate.
Our results show that PRIME architects hardware accelerators that improve over the best design in the training dataset, on average, by 2.46x (up to 6.7x) when specializing for a single application. In this case, PRIME also improves over the best conventional simulator-dri driven optimization methods by 1.54x (up to 6.6x). These performance improvements are obtained while reducing the total simulation time to merely 7% and 1% of that of the simulator-driven methods for single-task and multi-task optimization, respectively. More importantly, a contextual version of PRIME can design accelerators that are jointly optimal for a set of nine applications without requiring any additional domain information. In this challenging setting, PRIME improves over simulator-driven methods, which tend to scale poorly as more applications are added, by 1.38x. Finally, we show that the surrogates trained with PRIME accelerators for unseen target applications, without any retraining on the new application. Even in this zero-shot optimization scenario, PRIME outperforms simulator-based methods that require re-training and active simulation queries by up to 1.67 x. In summary, PRIME allows us to effectively address the shortcomings of simulation-driven approaches, (1) significantly reduces the simulation time, (2) enables data reuse and enjoys generalization properties, and (3) does not require domain-specific engineering or search space pruning.
set of training applications can be readily used obtain
BACKGROUND ON HARDWARE ACCELERATORS
The goal of specialized hardware accelerators Google TPUs [29, 23], Nvidia GPUs [43], Graph- Core 121]-is to improve the performance of specific applications, such as machine learning models. To design such accelerators, architects typically create a parameterized design and sweep over parameters using simulation.
Target hardware accelerators. Our primary evaluation uses an industry-grade and highly param- eterized template-bast based accelerator following prior work [65]. This template enables architects to determine the organization of various components, such as compute units, memory cells, memory, etc., by searching for these configurations in discrete design space. Some MI applicaiions may have large memory requirements (e.g., large language models [6]) demanding sufficient on-chip memory resources, while others may benefit from more compute blocks. The hardware design workflow directly selects the values of these parameters. In addition to this accelerator and to further show the generality of our method to other accelerator design problems, evaluate two distinct dataflow accelerators with different search spaces, namely NVDLA-style [42] and ObiDianNao-sty/l [10] from Kao al. [30] (See Section 6 and Appendix C for a detailed discussion; See Table 6 for results).
2

---

Published as a conference paper at ICLR 2022
Table 1: The accelerator design space parameters for the primary accelerator search space targeted in this work. The maximum possible number accelerator designs (including feasible and infeasible designs) is 452,760,000. PRIME only uses a small randomly sampled subset of the search space.
Accelerator Parameter
# Discrete Values
|| Accelerator Parameter
Discrete Values
PEs-X
# of PEs-Y
10
PE: Memory
7
# of Cores
Core Memory
of Compute Lanes
Instruction Mcmory
4
Parameter Mcmory
Activation Memory
DRAM Bandwidth
How does an accelerator work? We briefly explain the computation flow on template-based accelerators (Fig- ure 2) and refer the readers to Appendix C for details on other accelerators. This template-based accelerator is a 2D array of processing elements (PEs). Each PE is capable of performing matrix multiplications in a single instruc- tion multiple data (SIMD) paradigm [20]. A controller orchestrates the data transfer (both activations and model parameters) between off-chip DRAM memory and the on-chip buffers and also reads in and manages the instructions (e.g. convolution, pooling, etc.) for execution. The computation stages on such accelcrators start by sending set of activations to the compute lanes, executing them in SIMD manner, and either storing the partial computation results or offloading them back into off-chip memory. Compared to prior works [25, 10, 30], this parameterization is unique it includes multiple compute lanes per each PE and enables SIMD execution model within each compute lane and yields distinct accelerator search space accompanied by an end-to-end simulation framework. More details in Appendix C.
Figure 2: An industry-level machine learn ing accelerator [65].
PROBLEM STATEMENT, TRAINING DATA AND EVALUATION PROTOCOL.
Our template-based parameterization maps the accelerator, denoted as x, a discrete design space. x = [x1, X2, ... xk], and each x, is discrete-valued variable representing one component of the microarchitectural template, as shown in Table (See Appendix C for the description of other accelerator search spaces studied in our work). A design maybe be infeasible due to various reasons, such as compilation failure or the limitations of physical implementation, and we denote the set of all such feasibility criterion as Feasible(x). The feasibility criterion depends on both the target software and the underlying hardware, and it not easy to identify if given x infeasible without explicit simulation. We will require optimization procedure not only learn the value of the objective function but also to learn to navigate through a sea of infcasible solutions to segh-performing feasible solutions x" satisfying Feasible(x")) = 1.
Our training dataset D consists of modest set of accelerators x; that are randomly sampled from the design space and evaluated by the hardware simulator. We partition the dataset D into two subsets, Dreasible and Dintensible- Let fx) denote the desired objective (e.g., latency, power, etc.) we intend to optimize over the space of accelerators x. We do not possess functional access to f(x), and the optimizer can only access fxx values for accelerators x in the feasible partition of the data, Dieusible- For all infeasible accelerators, the simulator does not provide any value of f(x)). In addition to satisfying feasibility, the optimizer must handle explicit constraints on parameters such as area and power [13]. In our applications, we impose an explicit area constraint, Area(x) S ag, though additional explicit constraints are also possible. To account for different constraints, we formulate this task as a constrained optimization problem. Formally:
min fxx s.t. Area(x)= S ag. Feasible(x)
on D Diessible U Dinfeasible = ((xI, (xN, (N), (x),
While Equation may appear similar to other standard black-box optimization problems, solving it over the space of accelerator designs is challenging due to the large number of infeasible points, the need to handle explicit design constraints, and the difficulty in navigating the non-smooth landscape (See Figure 3 and Figure 11 in the Appendix) of the objective function.
What makes optimization over accelerators challenging? Compared to other domains where model-based optimization methods have been applied [5. 57], optimizing accelerators introduces
3

---

Published as a conference paper at ICLR 2022
4.0 x10*
3.5
In Infeasible False
400.0
3.0
True
300.0
2.5
2.0
1.5
1.0
100.0
0.5
0.0
2000 o
4000.
6000.0
8000.0
10000
a
12000.0
14000.0
16000.0
0.0
1000
1500
2000.0
o
2500.0
3000
3500
4000.0
Runtime
Runtime
Figure Left: histogram of infeasible (right orange bar with large values) and feasible (left cluster of bars) data points for ModieeNeEEdgeTPU: Right: zoomed-in histogram (different number of bins) focused on feasible points highlighting the variable latencies.
Table 2: The description of the applications, their domains, number of (convolutions, depth-wis wise convolutions,
XLA ops, model parameter size, instruction sizes in bytes, number of compute operations.
Name
Domain
of XI.A Ops (Cony, D/W, FF)
Model Param
Instr: Size
# Compute Ops.
MobileNetEdgeIPU
Image Class.
(45, 13, 1)
3.87 MB
476,736
1,989,811,
MobileNetV2
Image
(35, 1)
3.31 MB
416,032
609,353,376
MobileNetV3
Image Class.
(32, 15, 17)
5.20 MB
1,331,360
449,219, 6000
M4
Object Det.
(32, 1)
6.23 MB
317,600
3,471,920. 128
M5
Object Det.
(47,27.0) 0)
2.16 MB
328,672
939.752,960
M6
Object Det.
(53,
0.41 MB
369,952
228,146,848
U-Net
Image Seg.
(35,0,0)
3,69 MB5
224,992
13,707.114,848
T-NNN Dec
Speech Rec.
0, 19)
19 MB
915,008
40.116.221
TRNN Enc
Speech Rec.
(0, 0, 18)
21.62 MB
909,696
45,621,248
number of practical challenges. First, accelerator design spaces typically feature narrow manifold of feasible accelerators within sea infeasible points [41, 53, 17], as visualized in Figure 3 and Appendix (Figure 12). While some of these infeasible points can be identified via simple rules (e.g. estimating chip area usage), most infeasible points correspond to failures during compilation or hardware simulation. These infeasible points are generally not straightforward to formulate into the optimization problem and requires simulation [53, 44, 64].
Second, the optimization objective can exhibit high sensitivity to small variations in some architecture parameters (Figure lbb) in some regions of the design space, but remain relatively insensitive in other parts, resulting in complex optimization landscape. This suggests that optimization algorithms based on local parameter updates (c.g., gradient ascent, evolutionary schemes, etc.) may have a challenging task traversing the nearly flat landscape of the objective, which lead to poor performance.
Training dataset. We used offline dataset D of (accelerator parameters, latency) via random sampling from the space of 452M possible accelerator configurations. Our method is only provided with relatively modest set of feasible points (S 8000 points) for training, and these points are the worst-performing feasible points across the pool of randomly sampled data. This dataset is meant to dataset of accelerators that could have been
reflect an easily obtainable and an application-agnoee generated once and stored to disk, or might come from real physical experiments. We emphasize that no assumptions or domain knowledge about the application use case was made during dataset collection. Table 2 depicts the list of target applications, evaluated in this work, includes three variations of MobileNet [23, 50, 27], three in ho-uu industry-Level models for object detection (M4, M5, M6; names redacted to prevent anonymity violation), U-net model [48], and two RNN-based encoder-decoder language models [22, 24, 49, 38). These applications span the gamut from small models, such M6, with only 0.4 MB model parameters that demands less on-chip memory, the medium-sized models (2 5 MB), such as MobileNetV3 and M4 models, and large models (2 19 MB), such t-RNSs, hence requiring larger on-chip memory.
Evaluation protocol. To compare oate-of-theeat simulator-drive methods and our data-driven method, we limit the number of feasible points (costly to evaluate) that can be used by any algorithm to equal amounts. We still provide infeasible points to any method and leave it up to the optimization method to use it or not. This ensures our comparisons are fair in terms of the amount of data available to each method. However, it is worthwhile to note that in contrast to our method where worse -uurstty data points from small offline dataset used, the ieuulltoo-driverr methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance. Following prior work [5, 57, 58), evaluate each run of
4

---

Published as a conference paper at ICLR 2022
method by first sampling the top 256 design candidates according to the algorithm's predictions, evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design. The final reported results is the median of ground truth objective values across five independent runs.
4 PRIME: ARCHITECTING ACCELERATORS VIA CONSERVATIVE SURROGATES
As shown in Figure 4, our method first learns a conservative surrogate model of the optimization objective using the offline dataset. Then, it optimizes the learned surrogate using a discrete optimizer. The optimization process does not require access to a simulator, nor to real-world experiments beyond the initial dataset, except when evaluating the final top-performing n 256 designs (Section 3).
Learning conservative surrogates using logged offline data. Our goal is to utilize logged dataset of feasible accelerator designs labeled with the de- sired performance metric (e.g.. latency), D Decasibee and infeasible designs, Diffesible to learn a map- ping fo X - R. that maps the accelerator con- figuration x to its corresponding metric y. This learned surrogate can then be optimized by the optimizer. While straightforward approach for learning such a mapping is to train via supervised regression. by minimizing the mean-squared error Ex..u--l( [f (x;) - ". prior work 36, 57] has shown that such predictive models can arbitrar- ily overestimate the value of unseen input x;- This can cause the optimizer find solution x* that performs poorly in the simulator but looks promising under the learned model. We empirically validate this overestimation hypothesis and find it to confound the optimizer in on our problem domain as well (See Figure 13 in Appendix). To prevent overestimated values at unseen inputs from confounding the optimizer, we build on COMs [57] and train fx) with an additional term that explicitly maximizes the function value fx) at unseen x values. Such unseen designs x, where the learned function fe (x) is likely to be overestimated, are "negative mined" by running a few iterations of an approximate stochastic optimization procedure that aims to maximize So in the inner loop. This procedure is analogous to adversarial training [19]. Equation formalizes this objective:
Aud
A [MK) xf
Max)
(AC
OFf
Overview of PRIME which trains a con- servative surrogate fx(x)) using Equation 3. Our neural net model for fo(x) utilizes two transformer layers [59], and multi-headed architecture which is pooled via soft-attention layer.
Figure
arg min ((0)= == Ex,.yy [ff(x;)) yt) - aFEx,, ~Ope(fo)
[fo( (2)
B
x; denotes the negative samples produced from an optimizer Opt(-) that attempts to maximize the current learned model, fa. We will discuss our choice of Opt in the Appendix Section B.
Incorporating design constraints via infeasible points. While prior work [57] simply optimizes Equation 2 to learn surrogate, this is not enough when optimizing over accelerators, as we will also show empirically (Appendix A. This is because explicit negative mining does not provide any information about accelerator design constraints. Fortunately, this information is provided by infeasible points, Dinfcasible- The training procedure in Equation 2 provides simple way to do incorporate such infeasible points: we simply incorporate x² ~ iiteusible as additional negative samples and maximize the prediction at these points. This gives rise to our final objective:
min C ci" == [(6) - BEx
[x(]
(3)
Multi-model optimization and zero-shot generalization. One of the central benefits of data- driven approach is that it enables learning powerful surrogates that generalize over the space of applications. potentially being effective for new unseen application domains. In our experiments, we evaluate PRIME on designing accelerators for multiple applications denoted as k = 1, K, jointly or for a novel unseen application. In this case, we utilized a dataset D (DI,". each Dr consists of a set of accelerator designs, annotated with the latency value and the feasibility criterion for a given application k. While there are few overlapping designs in different parts of the dataset annotated for different applications, most of the designs only appear in one part. To train a single conservative surrogate fx(x) for multiple applications, we extend the training procedure in Equation 3 to incorporate context vectors C₁ Rd for various applications driven by a list of application properties in Table 2. The learned function in this setting is now conditioned on the context fo(x, ). We train so via the objective in Equation 3, but in expectation over all the contexts
DR). ). where
5

---

Published as a conference paper at ICLR 2022
and their corresponding datasets: ming [R] (0)]- Once such contextual surrogate is learned, we can either optimize the average surrogate across set of contexts {c;, c Cn) to obtain an accelerator that is optimal for multiple applications simultaneously on an average "fmulti-model"" optimization), optimize this contextual surrogate for novel context vector, corresponding to an unseen application ("zero-shot" generalization). In this case, PRIME is not allowed to train on any data corresponding to this new unseen application. While such zero-shot generalization might appear surprising at first, note that the context vectors are not simply one-hot vectors, but consist of parameters with semantic information, which the surrogate can generalize over.
Learned conservative surrogate optimization. Prior work [64] has shown that the most effective optimizers. We therefore choose to optimizers for accelerator design are meta- seattauurisic/evoluttiaaaa utilize, firefly [62. 63, 39] to optimize our conservative surrogate. This algorithm maintains a set of optimization candidates (a.k.a. "fireflies") and jointly update them towards regions of low objective value, while adjusting their relative distances appropriately to ensure multiple nigh-performing. but diverse solutions. We discuss additional details in Appendix B.I.
Cross validation: which model and checkpoint should we evaluate? Similarly to supervised learning, models trained via Equation 3 can overfit, leading to poor solutions. Thus, we require a procedure select which hyperparameters and checkpoints should actually be used for the design. This is crucial, because we cannot arbitrarily evaluate as many models as we want against the simulator. While effective methods for model selection have been hard to develop in offline optimization [57, 58], we devised a simple scheme using validation set for choosing the values of a and 3 (Equation 3), as well as which checkpoint to utilize for generating the design. For each training run, we hold out the best 20% of the points out of the training set and use them only for cross- validation as follows. Typical voss-valiontion strategies in supervised learning involve tracking validation error (or risk), but since our model is trained conservatively, its predictions may not match the ground truth, making such validation risk values unsuitable for our use case. Instead, we track Kendall's ranking correlation between the predictions of the learned model fx(x,) and the ground truth values yi (Appendix B) for the held-out points for each run. We pick values of a, 3 and the checkpoint that attain the highest validation ranking correlation. We present the pseudo-code for PRIME (Algorithm 1) and implementation details in Appendix B.1.
5 RELATED WORK
Optimizing hardware accelerators has become more important recently. Prior works [45, 28, 56, 41, 8. 34. 4, 3, 25, 60, 61] mainly rely on expensive-to-query hardware simulators to navigate the search space and/or target singlc-application accelerators. For example, HyperMapper [41] targets compiler optimization for FPGAs by continuously interacting with the simulator in design space with relatively few infeasible points. Mind Mappings [25], optimizes software mappings to a fixed hardware provided access to millions of feasible points and throws away infeasible points during learning. MAGNet [60] uses a combination of pruning heuristics and online Bayesian optimization to generate accelerators for image classification models in a single-application setting. AutoDNNChip [61] uses two-level online optimization to generate customized accelerators for ASIC and FPAG platforms. In contrast, PRIME, does not only learn a surrogate using offline data but can also leverage information from infeasible points and can work with just a few thousand feasible points. In addition, we devise a contextual version of PRIME that is effective in designing accelerators that are jointly optimized for multiple applications, different from prior work. Finally, to our knowledge, our work, is the first to demonstrate generalization to unseen applications for accelerator design, outperforming state-of-the-art online methods.
A popular approach for solving black-box optimization problems is model-based optimization (MBO) [55, 51, 54]. Most of these methods fail to scale high-dimensions, and have been extended with neural networks [55, 54, 31, 16. 15, 2, 1, 40J. While these methods work well in the active setting, they are susceptible to -istrr--iitiiittion inputs [58] in the offline, data-driven setting. To prevent this, offline MBO methods that constrain the optimizer to the manifold of valid, in-distribution inputs have been developed Brookes et al. [5], Fannjiang & Listgarten [12]. Kumar & Levine [35]. However, modeling the manifold of valid inputs can be challenging for accelerators. PRIME dispenses with the need for generative modeling, while still avoiding out-of df-tiiiiibttion inputs. PRIME builds on "conservative" offline RL and offline MBO methods that train robust surrogates [36, 57]. However, unlike these approaches, PRIME can handle constraints by learning from infeasible data and utilizes a better optimizer (See Appendix Table for comparison). In addition, while prior works area mostly

---

Published as a conference paper at ICLR 2022
Table 3: Optimized objective values (i.c., latency in milliseconds) obtained by various methods for the task of learning accelerators specialized a given application. Lower latency is better. From left right: our method, online Bayesian optimization ( ("Bayes Opt"), online evolutionary algorithm ("Evolutionary"), and the best design in the training dataset. On average (last row), PRIME improves over the best in the dataset by 2.46x (up to 6.69 in I-RNN Dec) and outperforms best online optimization methods by 1.54x (up to 6.62x in t-RNN Enc). The best accelerator configurations identified highlighted in bold.
Online Optimization
Application
PRIME
Bayes Opt
Evolutionary
MBO
(Best in Training)
MedieeNe
298.50
319.00
320.28
332.97
354.13
MobileNerV2
207.43
240.56
238.58
244.98
410.83
MobileNetV3
454.30
534.15
501
535,34
938.41
M4
370.45
396.36
383.58
405.60
779.98
M5
208 21
201.59
198.86
219.53
449.38
M6
131.46
12T.83
120.49
119.56
369.85
U-Net
740.27
872.23
791.64
888.10
1333. 18
IRNNN Dec
132.88
771.11
770.93
771.70
890.22
R-NNN Enc
130.67
865.07
865.07
866.28
584.70
Geomean of PRIME'S ' Improvement
1.0%
1. 58
1.54X
1. 61x
2. 2.46x
restricted to a single application, we show that PRIME is effective in multi-task optimization and zero-shot generalization.
EXPERIMENTAL EVALUATION
Our evaluations aim to answer the following questions: Q(1) Can PRIME design accelerators tailored for given application that are better than the best observed configuration in the training dataset, and comparable to or better than state-of-the-ar simulation-drive
methods under a given simulator-query budget? Q(2) Does PRIME reduce the total simulation time compared to other methods? Q(3) Can PRIME produce hardware accelerators for a family of different applications? Q(4) Can PRIME trained for family of applications extrapolate to designing a high- perforrrffrming accelerator for new, unseen application, thereby enabling data reuse? Additionally, we ablate various properties PRIME (Appendix A.6) and evaluate its efficacy in designing accelerators with distinct dataflow architectures, with a larger search space (up to 2.5 x 10114 possible candidates).
Baselines and comparisons. We compare PRIME against three online optimization methods that actively query the simulator: (1) evolutionary search with the firefly optimizer [64] ("Evolu- tionary"). ") which is the shown to outperform other online meth- ods for accelerator design; (2) Bayesian Optimization ("Bayes Opt") [18], (3) MBO [1]. In all the experiments, we grant all the methods the same number of feasible points. Note that our method do not get to select these points, and use the same exact offline points across all the runs, while the online methods can actively select which points to query, and therefore require new queries for every run. "D(Best in Training)" denotes the best latency value in the training dataset used in PRIME. We also present ablation results with different components of our method removed in Appendix A.6, where we observe that utiliz- ing both infeasible points and negative sampling are generally important for attaining good results. Appendix A.1 presents additional comparisons to COMs Trabucco et al. [57]-whide which only obtains negative samples via gradient ascent the learned surrogate and does not utilize infeasible points-and and P3BO Angermueller et al. [2]-an state-of-the-art online method in biology.
PRINT
Online (Evolutionory)
Total Simuiaaiioon
1000 1030 2000 1006 1023 0003 1000
6030
Optimization Iterations
Figure 5: Comparing the total simu- lation time PRIME (for PRIME this is the total time for a forward- through the trained surrogate and evolutionary method on MobileNet- Edge PRIME only requires about 7 7% of the total simulation time of the online method.
CPU)
Architecting application-specéfic accelerators. We first evaluate PRIME in designing specialized accelerators for each of the applications in Table 2. We train a conservative surrogate using the method in Section 4 on the logged dataset for each application separately. The area constraint (Equation 1) is set to a = 29 mm? a realistic budget for accelerators [64]. Table 3 summarizes the results. On average, the best accelerators designed by PRIME outperforms the best accelerator configuration in the training dataset (last row Table 3). by 2.46x. PRIME also outperforms the accelerators in the best online method by 1.54x (up 5.80 and 6.62x in I-RNN Dec and t-RNN
7

---

Published as a conference paper at ICLR 2022
Table 4: Optimized average latency (the lower, the better) across multiple applications (up ten applications) from diverse domains by PRIME and best online algorithms (Evolutionary and MBO) under different area constraints. Each row show the (Best, Median) of average latency across five runs. The geometric mean of PRIME'S s improvement over other methods (last row) indicates that PRIME is at Icast 21 % better.
Applications
Area
PRIME (Ours)
Evolutionary (Ouline)
MBO (Online)
MobileNon (EdgeTPU, V2, V3)
29 mm?
(310.21, 354 70)
(315.72. 325.69)
(342.02, 351 92)
MobileNet (V2. V5, M5,
29 mm
(26847, 271
(288 285 681
(295.2, 307 09)
MobileNet (Eddge V2, V3S, MEG, MS, Mo
29 mm
(311.39, 313.76)
(314.31, 316.00)
(321.48, 339,27)
MobileNet (EU, V2, V3,, M6. M5, M6. U-NeL L-NN Enc
29 nm
(308.77 310.99
(400.05, 404 59)
(404.06 412.90)
MobileNet FU, V2, VaX, ME.
ERIC
mus
(286.45, 287.98)
(401 404 29)
(404166 404.94)
MobileNes (EAgeTPU. V2, V3J, M4, MS, M6, ERNN (ite, Enc)
29 mm"
(426.65, 426.65)
(586.55, 586.55)
(62662, 692.6)
MobileNer (E. V2, V3),
U.Net ERNN (Dec,
aus"
(383.57, 385.56)
519.37)
(526.37, 530.9)
Geomesn of PRIME'S Y Improvement
(1.0%, 1.0%)
(1.21x, 1. 1.20%)
(1.24%, 1.27%)
Enc, respectively). Moreover, perhaps surprisingly, PRIME generates accelerators that are better than all the online optimization methods in 7/9 domains, and performs on par in several other scenarios (on average only 6.8% slowdown compared to the best accelerator with online methods in M5 and M6). These results indicates that offline optimization of accelerators using PRIME can be more data-efficient compared to online methods with active simulation queries. To answer Q(2), we compare the total simulation time of PRIME and the best evolutionary approach from Table 3 on the MobileNetEdgeTPU domain. On average, not only that PRIME outperforms the best online method that we evaluate, but also considerably reduces the total simulation time by 93%, shown in Figure 5. Even the total simulation time to the first occurrence of the final design that is eventually returned by the online methods is about 11x what PRIME requires to fine better design. This indicates that data-driven PRIME is much more preferred in terms of the performance-tine trade-off The fact that our offiine approach PRIME outperforms the online evolutionary method (and also other offtc-of-the-art online MBO methods; see Table 8) is surprising, and we suspect this is because online methods get stuck carly-on during optimization, while utilizing offline data allows us PRIME to find better solutions via generalization (see Appendix B.1.1).
Architecting accelerators for multiple applications. To an- swer Q(3), we evaluate the efficacy of the contextual version of PRIME in designing an accelerator that attains the lowest la- tency averaged over a set of application domains. As discussed previously, the training data used does not label given accel- erator with latency values corresponding to each application, and thus, PRIME must extrapolate accurately to estimate the latency of an accelerator for a context it is not paired with in the training dataset. This also means that PRIME cannot simply return the accelerator with the best average latency and must run non-trivial optimization. We evaluate our method in seven different scenarios (Table 4), comprising various combinations of models from Table 2 and under different area constraints, where the smallest set consists of the three MobileNet variants and the largest set consists of nine models from image clas- sification, object detection, image segmentation, and speech recognition. This scenario is also especially challenging for online methods since the number of jointly feasible designs expected drop significantly as more applications are added. For instance, for the case of the MobileNet variants, the training dataset only consists of a few (20-30) accelerator configurations that are jointly feasible and terhpeeroorning (Appendix B.g-Figure 9).
Total Symulatton Time (tec)
Ontine (evelutonary)
Oniine (M600)
Ovline (Bayes Opt1
wid
Optimizaionn Iterations
Figure 6: Comparing the total simu- lation time needed by PRIME and on- line methods on seven models (Area S 100mm²). PRIME only requires about 1 1%, 6%, and 0.9% of the to- tal simulation time of Evolutionary. MBO, and Bayes Opt, respectively, al- though PRIME outperforms the best on- line method by 41%.
Table 4 shows that, on average, PRIME finds accelerators that outperform the best online method by 1.2x (up to 41%). While PRIME performs similar to online methods in the smallest three-model scenario (first row). it outperforms online methods as the number of applications increases and the set of applications become more diverse. addition, comparing with the best jointly feasible design point across the target applications, PRIME finds significantly better accelerators (3.95 x) Finally, as the number of model increases the total simulation time difference between online methods and PRIME further widens (Figure 6). These results indicate that PRIME is effective in designing accelerators jointly optimized across multiple applications while reusing the same dataset as for the single-task, and scales more favorably than its simulation-driven counterparts. Appendix A.4 expounds the details of the designed accelerators for nine applications, comparing our method and the best online method.
8

---

Published as a conference paper at ICLR 2022
Table S: Optimized objective values (i.e.. latency in milliseconds) under zero-shot setting. Lower latency better. From left to right: the applications used to train the surrogate model in PRIME the target applications for which the accelerator is optimized for, the area constraint of the accelerator, PRIME'S (best, median) latency, and best online method's (best, median) latency. PRIME does not use any additional data from the target applications. On average (last row), PRIME yields optimized accelerator for target applications (with zero query to the target applications' dataset) with 1.26x (up to 1.66x) lower latency over the best online method. The best accelerator configurations identified is highlighted in bold.
Train Applications
Test Applications
Area
PRIMEE (Ours)
Evolutionary (Online)
MobileNNer (EdgeTPU, V33
MobileNetV2
mm3
(311.39, 313.76)
(314.31, 31665)
MobileNer (V2, V3), M5, M6
MobdleNerFdgee M4
29 mm
1357.05, 364.92)
(384.59, 357.29)
MobileNer (EdgeTPU, V2, V3), M5, M6, L-RNN Lnc
U-Net, L-NN Dec
29 mm
(745.87, 745.91)
(1075.91, 1127.64)
MobileNot (EdgeTPU. V2, V3)M4, M5, M6, E-RNN Ene
U-Net, E-NNN Dee
100 mm²
(517.76, 517.89)
(859.76, 861.69)
Geomean of PRIME'S ' Improvement
(1.0%, 1,0%)
(1.24x, 162)))
Table 6: Optimized objective values (i.e. total number of cycles) for two different dataflow architectures, NVDLA-style [42] and ShiDianNao-style [10], across three classes of applications. The maximum search space for the studied accelerators are == 2.5 10t14, PRIME generalizes to other classes of accelerators with larger search space and outperforms the best online method by 1.06 and the best data seen in training by 3.75x (last column). The best accelerator configurations is highlighted in bold.
Applications
Dataflow
PRIME
Evolutionary (Online)
D (Best in Training)
MobileNetV2
NVDLA
2.5
2.70x 10"
1.32x 10
MobileNetV2
ShiDianNao
2.65x10"
2.84x 10"
1.27x10
ResNet50
NVDLA
2.83x10'
3.13x 10
1.63 10
ResNet50
ShiDianNao
3,44x10'
3.74x 10
2.05x 10"
Transformer
NVDLA
7,8x10'
7,8 10
1.3x 10"
Transfomer
ShiDianNao
7.8x10*
7.8x 10
1.5x 10"
Geomean of PRIME's 's Improvement
1.0%
1.06x
75x
Accelerating previously unseen applications ("zero-shot"" optimization). Finally, we answer Q(4) by demonstrating that our data-driven offline method, PRIME enables effective data reuse by using logged accelerator data from set of applications design accelerator for an unseen new application, without requiring any training on data from the new unseen application(s). We train a contextual version of PRIME using a set of "training applications" and then optimize an accelerator using the learned surrogate with different contexts corresponding to "test applications," without any additional query the test application dataset. Table 5 shows, on average, PRIME outperforms woteeeoorpeernnt
the best online method by 1.26x (up to 66%) and only 2% slowdown in 1/4 cases. Note that the difference performance increases as the number of training applications increases. These results show the effectiveness of PRIME in the zero-shot setting (more results in Appendix A.5).
Applying PRIME other accelerator architectures and dataflows. Finally, to assess the the generalizability of PRIME to other accelerator architectures Kao et al. [30], we evaluate PRIME to optimize latency of two style of dataflow accelerators NVDLA-sttle and ShiDianNao-style- across three applications (Appendix C details the methodology). As shown in Table 6, PRIME outperforms the online evolutionary method by 6% and improves over the best point in the training dataset by x.75x. This demonstrates the efficacy of PRIME with different dataflows and large design spaces.
7 DISCUSSION
In this work, we present data-driven offline optimization method, PRIME automatically architect hardware accelerators. Our method learns a conservative surrogate of the objective function by leveraging infeasible data points better model the desired objective function of the accelerator using a one-time collected dataset of accelerators, thereby alleviating the need for time-consuming simulation. Our results show that, on average, our method outperforms the best designs observed in the logged data by 2.46x and improves over the best simulator-driven approach by about 1.54x. In the more challenging setting of designing accelerators jointly optimal for multiple applications or for new, unseen applications, zero-shot, PRIME outperforms simulator-drive methods by 1.2x, while reducing the total simulation time by 99%. The efficacy of PRIME highlights the potential for utilizing the logged offline data in an accelerator design pipeline. While PRIME outperforms the online methods we utilize, in principle, a strong online method can be devised by running PRIME in the inner loop. Our goal is to not advocate that offline methods must replace online methods, but that training a strong offline optimization algorithm on offline datasets of low wor-performing designs can be highly effective ingredient in hardware accelerator design.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank the "Learn to Design Accelerators" team at Google Research and the Google EdgeTPU team for their invaluable feedback and suggestions. In addition, we extend our gratitude to the Vizier team, Christof Angermueller, Sheng- Chng Kao, Samira Khan, Stella Aslibekyan, and Xinyang Geng for their help with experiment setups and insightful comments.
REFERENCES
[I] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based Reinforcement Learning for Biological Sequence Design. In ICLR, 2019.
[2] Christof Angermueller, David Belanger, Andreea Gane, Zelda Mariet, David Dohan, Kevin Murphy, Lucy Colwell, and Sculley. Population-Bas Black-Bor -ox Optimization for Biological Sequence Design. In ICML, 2020.
[3] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May O' O'Reilly, and Saman Amarasinghe. OpenTuner: An Extensible Framework for Program Autotuning. In PACT, 2014.
[4] Prasanna Balaprakash, Ananta Tiwari, Stefan M Wild, Laura Carrington, and Paul D Hovland. AutoMOMML: Automatic Multi- culti-Obbecctiv Modeling with Machine Learning. HiPC, 2016.
[5] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by Adaptive Sampling for Robust Design. In ICML, 2019.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models Few-Shot Learners. arXiv preprint arXiv:2005. 2020.
[7] Prasanth Chatarasi, Hyoukjun Kwon, Natesh Raina, Saurabh Malik, Vaisakh Haridas, Tushar Krishna, and Vivek Sarkar. MARVEL: A Decoupled Model-driven Approach for Efficiently Mapping Convolutions Spatial DNN Accelerators. arXiv preprint arXiv:2002.07752, 2020.
[8] Jason Cong, Peng Wei, Cody Hao Yu, and Peng Zhang. Automated Accelerator Generation and Optimization with Composable, Parallel and Pipeline Architecture. In DAC, 2018.
[9] Shail Dave, Youngbin Kim, Sasikanth Avancha, Kyoungwoo Lee, and Aviral Shrivastava. DMazeRunner: Executing Perfectly Nested Loops on Dataflow Accelerators. TECS, 2019.
[10] Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo Ienne, Ling Li, Tao Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam. ShiDianNao: Shifting Vision Processing Closer to the Sensor. In ISCA, 2015.
[11] Hadi Esmaeilzadeh, Emily Blem, Renee St Amant, Karthikeyan Sankaralingam, and Doug Burger. Dark Silicon and the End of Multicore Scaling. In ISCA, 2011.
[12] Clara Fannjiang and Jennifer Listgarten. Autofocused Oracles for Model-Based Design. arXiv preprint arXiv:2006.00 20t.:00000..0,,, 2020.
[13] Michael Flynn and Wayne Luk. Computer System Design: System-on-Chip. John Wiley Sons, 2011.
[14] Justin Fu and Sergey Levine. Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation. In ICLR, 2021.
[15] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In ICML, 2018.
[16] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural Processes. arXiv preprint arXiv: 1807.01280.12 2018.
10

---

Published as a conference paper at ICLR 2022
[17] Michael A Gelbart, Jasper Snock, and Ryan P Adams. Bayesian Optimization with Unknown Constraints. arXiv preprint arXiv:1403.5607, 2014.
[18] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski. John Karro, and Sculley. Google Vizier: A Service for Black-box Optimization. SIGKDD, 2017.
[19] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adver- sarial Examples. ICLR, 2015.
[20] Robert Gove, Keith Balmer, Nicholas K Ing-Simmons, and Karl M Guttag. Multi-Processor Reconfigurable in Single Instruction Multiple Data (SIMD) and Multiple Instruction Multiple Data (MIMD) Modes and Method of Operation, 1993. US Patent 5,212,777.
[21] GraphCore. GraphCore.
ai/, ai/, 2021. Accessed: 2021-05-16.
[22] Alex Graves. Sequence Transduction with Recurrent Neural Networks. arXiv preprint
11X11:1211.3711, 2012.
[23] Suyog Gupta and Berkin Akin. Accelerator-Aware Neural Network Design using AutoML. arXiv preprint arXiv:2003.02888 2020.
[24] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming End-to-End Speech Recognition for Mobile Devices. In ICASSP, 2019.
[25] Kartik Hegde, Po-An Tsai, Sitao Huang, Vikas Chandra, Angshuman Parashar, and Christo- pher W Fletcher. Mind Mappings: Enabling Efficient Algorithm-AAcceeratoo Mapping Space Search. ASPLOS, 2021.
[26] Andrew Howard and Suyog Gupta. Introducing the Next Generation of On-Device Vision Mod- els: MobileNetV3 and MobileNetEdgeTPU. https: //ai http://aaa.goggoll////o/o www.lebloooooooooooooooooooooooo ebllg.llo...oo com/ 2019/ 11
html, 2020.
[27] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In CVPR, 2019.
(28) Md Shahriar Iqbal, Jianhai Su, Lars Kotthoff, and Pooyan Jamshidi. FlexiBO: Cost-Aware Multi-Ohjective Optimization of Deep Neural Networks. arXiv preprint arXiv:2001.0658 2020.
[29] Norman P Jouppi. Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-Datacenter Performance Analysis of Tensor Processing Unit. In ISCA, 2017.
[30] Sheng-Chun Kao, Geonhwa Jeong, and Tushar Krishna. ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning. In MICRO, 2020.
[31] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In ICLR, 2019.
[32] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
[33] Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv preprint arXiv: 13/2.61 14, 2013.
[34] David Koeplinger, Matthew Feldman, Raghu Prabhakar, Yaqi Zhang, Stefan Hadjis, Ruben Fiszel, Tian Zhao, Luigi Nardi, Ardavan Pedram, Christos Kozyrakis, et al. Spatial: A Language and Compiler for Application Accelerators. In PLDI, 2018.
[35] Aviral Kumar and Sergey Levine. Model Inversion Networks for Model-Basee Optimization. NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2022
[36] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS, 2020.
[37] Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, and Tushar Krishna. Understanding Reuse, Performance, and Hardware Cost of DNN Dataflow:
A Data-Centric Approach. In MICRO, 2019.
[38] Bo Li, Anmol Gulati, Jiahui Yu, Tara N Sainath, Chung-Chee Cheng Chiu, Arun Narayanan, Shuo- Yin Chang, Ruoming Pang. Yanzhang He, James Qin, et al. A Better and Faster end-to-end Model for Streaming ASR. In ICASSP, 2021.
[39] Changnian Liu, Yafei Tian, Qiang Zhang, Jie Yuan, and Binbin Xue. Adaptive Firefly Optimiza- tion Algorithm Based on Stochastic Inertia Weight. In ISCID, 2013.
[40] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A Graph Placement Methodology for Fast Chip Design. Nature, 2021.
[41] Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical Design Space Exploration. In MASCOTS, 2019.
[42] Nvidia. NVDLA deep learning accelerator. 10-01.
2021. Accessed: 2021-
[43] Nvidia. Nvidia.
2021. Accessed: 2021-05-16.
[44] Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emer. Timeloop: A Systematic Approach to DNN Accelerator Evaluation. In ISPASS. IEEE, 2019.
[45] Maryam Parsa, John P Mitchell, Catherine D Schuman, Robert Patton, Thomas E Potok, and Kaushik Roy. Bayesian Multi-objective Hyperparameter Optimization for Accurate, Fast, and Efficient Neural Network Accelerator Design. Frontiers Neuroscience, 14:667, 2020.
[46] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing Posterior Collapse with Delta-VAEs. In ICLR, 2018.
[47] Brandon Reagen, José Miguel Hermández-Lobato, Robert Adolf, Michael Gelbart, Paul What- mough, Gu- Ycon Wei, and David Brooks. A Case for Efficient Accelerator Design Space Exploration via Bayesian Optimization. In ISLPED, 2017.
[48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI. Springer, 2015.
[49] Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo- yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, et al. A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency. In ICASSP, 2020.
[50] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- -hiehh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR, 2018.
[51] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 2016.
[52] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv preprint arXiv: 063386 2017.
[53] Zhan Shi, Chirag Sakhuja, Milad Hashemi, Kevin Swersky, and Calvin Lin. Learned Hard- ware/Software Co-Design of Neural Accelerators. arXiv preprint arXiv:2010.02075, 2020.
[54] Jasper Snock, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In NIPS, 2012.
12

---

Published as a conference paper at ICLR 2022
[55] Jasper Snoek. Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian Optimization Using Deep Neural Networks. In ICML, 2015.
[56] Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Jiancheng Lv. Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification. IEEE Transactions on Cybernetics, 2020.
[57] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative Objective Models for Effective Offline Model-Based Optimization. In ICML, 2021.
[58] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Design-Bench- Benchmarks for Data-Driven Offline Model-Based Optimization, 2021. URL https: /oopenreviww net forrmmid=c002f26aaa
/
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. arXiv preprint arXiv:1706.037722 2017.
[60] Rangharajan Venkatesan, Yakun Sophia Shao, Miaorong Wang, Jason Clemons, Steve Dai, Matthew Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, al. MAGNet: Modular Accelerator Generator for Neural Networks. ICCAD, 2019.
[61] Pengfei Xu, Xiaofan Zhang, Cong Hao, Yang Zhao, Yongan Zhang, Yue Wang, Chaojian Li, Zetong Guan, Deming Chen, and Yingyan Lin. AutoDNNchip: An Automated DNN Chip Predictor and Builder for both FPGAs and ASICs. In FPGA, 2020.
[62] Xin-She Yang. Nature-Inspired Metaheuristic Algorithms. Luniver press, 2010.
[63] Xin-She Yang and Suash Deb. Eagle Strategy Using Lévy Walk and Firefly Algorithms for Stochastic Optimization. NICSO. Springer, 2010.
[64] Amir Yazdanbakhsh, Christof Angermueller, Berkin Akin, Yanqi Zhou, Albin Jones, Milad Hashemi, Kevin Swersky, Satrajit Chatterjee, Ravi Narayanaswami, and James Laudon. Apollo: Transferable Architecture Exploration. arXiv preprint arXiv:2102.01723. 2021.
[65] Amir Yazdanbakhsh, Kiran Seshadri, Berkin Akin, James Laudon, and Ravi Narayanaswami An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks. arXiv preprint arXiv:2102. 10423, 2021.
13

---

Published as a conference paper at ICLR 2022
Appendices
A ADDITIONAL EXPERIMENTS
In this section, present additional experiments compared the method of Trabucco al. [57]), present some additional results obtained by jointly optimizing multiple applications (Appendix A.2), provide analysis of the designed accelerators (Appendix A.4) and finally, discuss how trained conservative surrogate can be used with a different evaluation time constraint (Appendix A.3).
A.1 COMPARISON TO OTHER BASELINE METHOSS
Comparison to COMs. In this section, we perform a comparative evaluation of PRIME to the COMs method Trabucco et al. [57]. Like several offline reinforcement learning algorithms [36], our method, PRIME and COMs are based on the key idea of learning conservative surrogate of the desired objective function, such that it does not overestimate the value of unseen data points, which prevents the optimizer from finding accelerators that appear promising under the learned model but not actually promising under the actual objective. The key differences between COMs are: (1) PRIME uses an evolutionary optimizer (Opt(-)) for negative sampling compared to gradient ascent of COMs, which be vastly beneficial in discrete design spaces our results show empirically, (2) PRIME can explicitly learn from infeasible data points provided to the algorithm, while COMs does not have mechanism to incorporate the infeasible points into the learning of surrogate. To further assess the importance of these differences in practice, we run COMs on three tasks from Table 3, and present comparison our method, COMs, and Standard method in Table The "Standard" method represents a surrogate model without utilizing any infeasible points. On average, PRIME outperforms COMs by 1.17x (up to 1.24x in M6).
method and
Table 7: Optimized objective values (i.e.. latency in milliseconds) obtained by PRIME and COMs [57] when optimizing over single applications (MobilenetV2. MobilenetV3 and M6), extending Table 3. Note that PRIME outperforms COMs. However, COMs improves over baseline "Standard" method (last column).
Application
PRIME (Ours)
||
COMs
Standard
MobileNetV2
207.43
251.58
374.52
MobileNetV3
454.30
485.66
575.75
M6
131.46
163.94
180.24
Geomean of PRIME'S Improvement
1.0x
1.17x
1. 46x
Comparison to generative offline MBO methods. We provide comparison between PRIME and prior offline MBO methods based on generative models [35]. We evaluate model inversion
networks (MINs) [35] on our accelerator data. However, we were unable to train a discrete objective-conditione GAN model to 0.5 discriminator accuracy on offline dataset, and often observed collapse of the discriminator. As result, we trained a 5 VAE [46], conditioned on the objective function (i.e., latency). A standard VAE [33] suffered from posterior collapse and thus informed our choice of utilizing a VAE.. The la- tent space of a trained objective-conditionedd 8- VAE corresponding to accelerators on a held-out validation dataset (not used for training) visualized in the t-SNE plot in the figure on the right. This is a 2D t-SNE of the accelerators configurations (aablle 1). The color of a point denotes the latency value of the corresponding acceler- ator configuration, partitioned into three bins. Observe that while we would expect these objective conditioned models to disentangle accelerators with different objective values in the latent space, the models we trained did not exhibit such structure, which will hamper optimization. While our method PRIME could also benefit from generative optimizer (i.e., by using generative optimizer in place of Opt(-) with a conservative surrogate). we leave it for future work to design effective generative optimizers the accelerator manifold.
14

---

Published as a conference paper at ICLR 2022
1e6
Evolutionary
0.8
Total Simulation Time
0.-
1000 1500
2000
3000
2500
4000
Optimization iterations
Figure Comparing the total simulation time needed by the P3BO and Evolutionary method on MobileNet- EP Note that, not only the total simulation time of P3BO around 3.1 higher than the Evolutionary method, but also the latency of final optimized accelerator is around 18% for MobileNetEdge'TPP The total simulation time of our method is around 7% of the Evolutionary method (See Figure 5).
Comparison to P3BO. We perform a comparison against P3BO method, the state-of-the-art online method in biology [2]. On average, PRIME outperforms the P3BO method by 2.5x (up 8.7x in U-Net), In addition, we present the comparison between the total simulation runtime of the P3BO and Evolutionary methods in Figure Note that, not only the total simulation time of P3BO is around 3.1x higher than the Evolutionary method, but also the latency of final optimized accelerator around 18% for MobileNetEdge the task of accelerator design for MobileNetEdgeTPP is lower than both methods (only 7% of the Evolutionary method as shown Figure 5).
the other hand, total simulation time PRIME for
Table Optimized objective values (i.e.. latency milliseconds) obtained by PRIME P3BO [2] when optimizing over single applications (MobileNetEddeePPU,, M4, t-RNN Dec, t-RNN Enc, and U-Net), On average, PRIME outperforms P3BO by 2.5x.
Application
PRIME (Ours)
P3BO
Mo
298.50
376.02
M4
370.45
483.39
U-Net
740.27
771.70
RRNN Dec
132.88
865.12
I-NNN Enc
130.67
1139.48
Geomean of PRIME'S Improvement
1.0x
2.5x
A.2 LEARNED SURROGATE MODEL REUSE FOR ACCELERATOR DESIGN
Extending our results in Table 4, present another variant of optimizing accelerators jointly for multiple applications. In that scenario, the learned surrogate model reused to architect an accelerator for subset of applications used for training. We train contextual conservative surrogate on the variants of MobileNet (Table 2) as discussed in Section 4, but generated optimized designs by only optimizing the average surrogate only two variants of MobileNet (MobileNetEdgee and MobileNerV2). This tests the ability of our approach PRIME to provide general contextual conservative surrogates, that be trained only once and optimized multiple times with respect different subsets of applications. Observe in Table 9, PRIME architects high-performing accelerator configurations (better than the best point in the dataset by 3.29 last column) while outperforming the online optimization methods by 7%.
A.3 LEARNED SURROGATE MODEL REUSE UNDER DIFFERENT DESIGN CONSTRAINT
We also test the robustness of our approach in handling variable constraints at test-time such as different chip area budget. We evaluate the learned conservative surrogate trained via PRIME under a reduced value of the area threshold, a, in Equation I. To do so, we utilize a variant of rejection sampling- take the learned model trained for default area constraint 29 mm? and then reject
15

---

Published as a conference paper at ICLR 2022
Table 9: Optimized objective values (i.e., latency in milliseconds) obtained by our PRIME when using the joindy optimized model on three variants MobileNets and use for Mobe and MobileNetV2 for different dataset configurations. PRIME outperforms the best online method by 7 % and finds an accelerator that x.29 x better than the best accelerator in the training dataset (last row). The best accelcrator configuration is highlighted in bold.
PRIMK
Online Optimization
Applications
All
-Opt
-Ineeasible
Standard
Bayes Opt
Evolutionayy
D (Best Training)
(MbbikNeeEdgeCPP. Moobiiea
Mob)
253.85
297.36
264.85
341.12
275.21
271.71
834.68
Table 10: Optimized objective values (i.c., latency in milliscconds) obtained by various methods for the task of learning accelerators specialized to MobileNertEgeePPP under chip arca budget constraint 18 mm² reusing the already learned model by our method for MobileNe(EdgeTPPU (shown in Table 3). Lower latency/runtime better. From left to right: our method, our method without negative sampling ("PRIME-Opl") and without utilizing infeasible points (" ("PRIME-Infeasille) standard surrogate ("Standard"), online Bayesian optimization (" ("Bayes Opt"), online evolutionary algorithm ("Evolutionary") and the best design in the training datasct. Note that PRIME improves the best in the dataset by 12%, outperforms the best online optimization method by 4.4%. The best accelerator configuration is highlighted in bold.
PRIME
Online Optimization
Applications
-Opt
-Infcasible
Standard
Bayes Opt Evolutionary
Training)
MohileNedEdgeePU. Area 18
315.15
433.81
351.22
470.09
329.13
354.13
Table 11: Per application latency for the best accelerator design suggested by PRIME and the Evolutionary method according to Table for multi-task accelerator design (nine applications and area constraint 100 mm²). PRIME outperforms the Evolutionary method by 1.35x.
Lafency (ms)
Applications
PRIME
Evolutionary (Online)
Improvement of PRIME Evolutionary
MobileNetEdgeTPU
288.38
319.98
1.10x
MobileNerV2
216.27
255.95
1.18x
MobileNetV3
487.46
573.57
1.17x
M4
400.88
406.28
1.01
M5
248.
239.18
0.96x
M6
164,9%
U-Net
1268. 73
908.86
0.71
ERNN Dec
191.
862.14 14
5.13x
I-NN Ence
185.41
Average (Latency ms)
383.
1.35x
all optimized accelerator configurations which do not satisfy a reduces area constraint: Area(x) < oo 18 mm3 Table 10 summarizes the results for this scenario for the Mode [23] application under the new area constraint (a 18 mm²). A method that produces diverse designs which are both hirf--meno and are spread across diverse values of the area constraint are expected to perform better. As shown in Table 10, PRIME provides better accelerator than the best online optimization from scratch with the new constraint value by 4.4%, even when PRIME does not train its conservative surrogate with this unseen test-time design constraint. Note that, when the design constraint changes, online methods generally need to restart the optimization process from scratch and undergo costly queries to the simulator. This would impose additional overhead in terms of total simulation time ($ Figure and Figure 6). However, the results in Table 10 shows that learned surrogate model can be reused under different test-time design constraint eliminating additional queries to the simulator.
A.4 ANALYSIS OF DESIGNED ACCELERATORS
In this section, we overview the best accelerator configurations that PRIME and the Evolutionary method identified for multi-task accelerator design (See Table 4), when the number of target applica- tions are nine and the area constraint is set to 100 mm? The average latencies of the best accelerators found by PRIME and the Evolutionary method across nine target applications are 383,57 ms and 518.58 ms, respectively. In this setting, our method outperforms the best online method by 1.35x.
16

---

Published as a conference paper at ICLR 2022
Table 12: Optimized accelerator configurations (See Table 1) found by PRIME and the Evolutionary method for multi-task accelerator design (nine applications and area constraint 100 mm²). Last row shows the accel- erator area in mm3 PRIME reduces the overall chip area usage by x.97x. The difference in the accelerator configurations are shaded in gray.
Parameter Value
Accelerator Parameter
PRIME
Evolutionary (Online)
# of PEs-X
4
4
# of PEs-Y
6
8
# Cores
128
" of Compute Lanes
4
PE Memory
2,097,152
1,048,576
Core Memory
131,072
131,072
Instruction Memory
32,768
8,192
Parameter Memory
4.096
4,096
Activation Memory
512
2,048
DRAM Bandwidth (Gbps)
30
30
Chip Area (mm")
78
92.
Table 11 shows per application latencies for the accelerator suggested by our method and the Evolu- tionary method. The last column shows the latency improvement of PRIME over the Evolutionary method. Interestingly. while the latency of the accelerator found by our method for MobileNetEd- geTPU, MobileNetV2, MobileNetV3, M4, t-RNN Dec, and t-RNN Enc are better, the accelerator identified by the online method yields lower latency in M5, M6, and U-Net.
To better understand the trade-off in design of each accelerator designed by our method and the Evolutionary method, we present all the accelerator parameters (See Table 1) in Table 12. The accelcrator parameters that are different between each of the designed accelcrator are shaded in gray (e.g. # of PEs-Y Y, of Cores, of Compute Lanes, PE Memory, Instruction Memory, and Activation Memory). Last row of Table 12 depicts the overall chip area usage in mm? PRIME not only outperforms the Evolutionary algorithm in reducing the average latency across the set of target applications, but also reduces the overall chip area usage by 1.97x. Studying the identified accelerator configuration, observe that PRIME trade-offs compute 64 cores vs. 128 cores) for larger PE
memory size 2,,97,1522 vs. 1,048,576 ). These results show that PRIME favors PE memory size to accommodate for the larger memory requirements in t-RNN Dec and t-RNN Enc (See Table Model Parameters) where large gains lic. Favoring larger on-chip memory comes the expense of lower compute power in the accelerator. This reduction in the accelerator's compute power leads to higher latency for the models with large number of compute operations, namely M5, M6, and U-Net (See last row in Table 2). M4 is an interesting case where both compute power and on-chip memory is favored by the model (6.23 MB model parameters and 3,471,920, 28 number of compute operations). This is the reason that the latency of this model on both accelerators, designed by our method and the Evolutionary method, are comparable (400.88 ms in PRIME vs. 406.28 ms in the online method).
A.5 COMPARISON WITH ONLINE METHOOS IN ZERO-SHOT SETTING
We evaluated the Evolutionary (online) method under two protocols for the last two rows of Table 5: first, we picked the best designs (top-performing 256 designs similar to the PRIME setting in Section 4) found by the evolutionary algorithm on the training set of applications and evaluated them on the target applications and second, we let the evolutionary algorithm continue simulator-driven optimization on the target applications. The latter is unfair, that the online approach is allowed access to querying more designs in the simulator. Nevertheless, we found that in either configuration, the evolutionary approach performed worse than PRIME which does not access training data from the target application domain. For the area constraint 29 mm² and 100 mm? the Evolutionary algorithm reduces the latency from 1127.64 + 820. 11 and 861.69 - 552.64, respectively, although still worse than PRIME. In the second experiment in which unfairly allow the evolutionary algorithm to continue optimizing on the target application, the Evolutionary algorithm suggests worse designs than Table (e.g. 29 mm?: 1 1127.64 1181.66 and 100 mm?: 861.69 861.66).
17

---

Published as a conference paper at ICLR 2022
Table 13: Optimized objective values (i.e., latency milliseconds) ohtained by various methods for the task of learning accelerators specialized to given application. Lower latency/runtime is better. From left right: method, method without negative sampling ("PRIME-Opt") and without utilizing infeasible points ("PRIME- Infeasible"), standard surrogate ("Standard")), online Bayesian optimization ("Bayes Opt"), online evolutionary algorithm ("Evolutionary") and the best design in the training dataset, Note that, in all the applications PRIME improves over the best in the dataset, outperforms online optimization methods in 7/9 applications and the complete version of PRIME generally performs best. The best accelerator designs are in bold.
PRIME
Online Optimization
Application
All
-Opt
Infeasible
Standard
Bayes Opt
Evolutionary
D (Best in Training)
MobileNcrEdge
298.50
435.40
322.20
411.12
319.00
320.28
354.13
MobileNetV2
207.43
281.01
214.71
374.52
238.58
MohileNerV3
454.30
489.45
483.96
575.75
534.15
501.27
938.41
M4
370.45
478.32
432.78
1139.76
396.36
383.58
779.98
M5
208.21
319.61
246.80
307.37
201.39
198.86
449.38
M6
131.46
197.70
162.12
180.24
121.83
120.49
369.85
740.27
740.27
765.59
763.10
791.64
1333.
I-RNN Dee
132.88
172.06
135.47
136.20
771.11
770.93
890.22
I-RNN Ein:
130.67
134.84
137.28
150.21
865.07
865.07
584.70
Table 14: The comparison between the accelerator designs suggested by PRIME and EPg [65, 23] for single model specialization. On average (last row), with single- model specialization our method reduces latency by 2.69x while minimizes the chip area usage by 1.50x.
Latency (milliseconds)
Chip Area (mm²)
Application
PRIME
EdgeTPU
Improvement
PRIME
Edg
Improvement
MoUT
294.34
523.48
1.78x
1.50x
MobileNetV2
208.72
408.24
1.96x
17.11
1.58x
MobileNetV3
459.59
831.80
1.81
11.86
27
2.28x
M4
370.45
675.53
1.82
19.12
27
1.41
M5
208.49 42
377.32
1.81
22.84
27
1.18x
M6
132.98
234.88
1.77x
16.93
1.59
U-Net
1465.70
2409.73
1.64x
25.27
1.07x
t-NN Dec
132.43
1384.44
10.45
14.82
27
1.82x
[-NN Enc
130.45
1545.07
11.84x
19.87
27
1.36x
Average Improvement
2.69
1.50
A.6 PRIME ABLATION STUDY
Here ablate over variants of our method: (1) Opt was not used for negative sampling ("PRIME- Opt" Table 13) (2) infeasible points were not used ("PRIME-IIfeesibbe" in Table 13). shown in Table 13, the variants of our method generally performs worse compared to the case when both negative sampling and infeasible data points are utilized training the surrogate model.
A.7 COMPARISON WITH JIMMAN-ENGINEERED ACCELERATORS
In this section, we compare the optimized accelerator design found by PRIME that is targeted towards single applications the manually optimized EdgeTPU design [65, 23). EPgeTPU accelerators are primarily optimized towards running applications in image classification, particularly, MobileNetV2. MobileNetV3 and MobileNetEdgeTPU. The goal of this comparison to present the potential benefit of PRIMEfor dedicated application when compared human designs. For this comparison, utilize an area constraint of 27 mm? and a DRAM bandwidth of 25 Gbps, to match the specifications of the TPg accelerator.
Table 14 shows the summary of results in two sections. namely "Latency" and "Chip Area". The first and second under each section show the results for PRIME and EdgeTPU, respectively. The final column for each section shows the improvement of the design suggested by PRIME over EdgeTPU. On average (as shown in the last row), PRIME finds accelerator designs that are 2.69x (up to 11.84x in t-RNN Enc) better than Edge in terms of latency. Our method achieves this improvement while, on average, reducing the chip area usage by 1.50x (up to 2.28 x MobileNetV3), Even on the MobileNet image-clasifficttinn domains, attain average improvement of 1.85x.
18

---

Published as a conference paper at ICLR 2022
Table 15: Optimized objective values (i.e., latency in milliseconds) under zero-shot setting when the test applications include all the nine evaluated models (e.g. MobileNet (EdgeTPU, V2. V3), M4, M5, M6. t-RNN Dec, t-RNN Enc, U-Net). Lower latency is better. From left to right: the applications used to train the surrogate model in PRIME the target applications for which the accelerator is optimized for, the area constraint of the accelerator, PRIME'S (best, median) latency, and best online method's (best, median) latency. The best accelerator configurations identified highlighted in bold.
Train Applications
Area
PRIME
Evolutionary (Online)
MobileNer TP, M4, M5, M6, I-RNN Enc
29 mm3 100 mm²
(426.65, 427.94)
(586.55, 586.55)
MobileNet (E.,eTPU.V2.V33) M4, M5, M6, I-NNN Enc
?
(365.95, 366.64)
(518.58, 519.37)
Geomean of PRIME'S Improvement
(1.0x, L0x)
(1. (1.40x, 1. 39x)
A.8 ZERO-SHOT RESULTS ON ALL APPLICATIONS
In this section, present the results of zero-shot optimization from Table 5 all the nine appli- cations we study in the paper (i.e., test applications = all nine models: MobileNet (EdgeTPU, V2, V3), M6, M5, M4, I-RNN (Enc and Dec), and U-Net). We investigate this for two sets of training applications and two different area budgets. As shown in Table 15, we find that PRIME does perform well compared the online evolutionary method.
A.9 DIFFERENT TRAIN AND VALIDATION SPLITS
In the main paper, we used the worst 80% of the feasible points in the training dataset for training and used the remaining 20% of the points for cross-validation using our strategy based on Kendall's s rank correlation. this section, we explore some alternative caining-validation split strategies to see how they impact the results. To do so, we consider two alternative strategies: (1) training on 95% of the worst designs, validation on top 5% of the designs, and (2) training on the top 80% of the designs and validation on the worst 20% of the designs. We apply these strategies to MobileNe(Edge M6 and I-RNN Enc models from Table 3, and present comparative evaluation in Table below.
Results. As shown in Table 16, we find that cross cross- validating using the best 5% of the points in the dataset led to a reduced latency (298.50 + 273.30) on MobileNetEdgeTPU, and retained the M6. However, it increased the latency on t-RNN Enc (130.67
same performance This indicates at the possibility that while top 5% of the datapoints can provide a better signal for cross-validation in some cases, this might also hurt performance if the size of the 5% dataset becomes extremely small (as in the case of t-RNN Enc, the total dataset size is much smaller than either MobileNetEEdge or M6).
137.45).
The strategy of cross-validating using the worst 20% of the points hurt performance on M6 and t-RNN Enc, which perhaps as expected, since the worst 20% of the points may not be indicative of the best points found during optimization. However, while it improves performance on the MobileNerEdge application compared to the split used in the main paper but is still worse than using the top 5% of the points for validation.
Table 16: Performance of PRIME (as measured by median lateney of the accelerator found across five runs) under various train-test splits on three applications studied in Table 3.
Applications
Best 5% Validation
Best 20 % Validation (Table 3)
Worst 20 % Validation
MobileNetEddgeIPU
273.30
298.50
286.53
M6
131. .6
131.46
142.68
I-NNN Enc
137.45
130.67
135.71
B
DETAILS OF PRIME
In this section, we provide training details of our method PRIME including hyperparameters and compute requirements and details of different tasks.
B.1 HYPERPARAMETER AND TRAINING DETAILS
Algorithm outlines our overall system for accelerator design. PRIME parameterizes the function fx(x) as deep neural network as shown in Figure 4. The architecture of fo(x) first embeds the
19

---

Published as a conference paper at ICLR 2022
discrete-valuees accelerator configuration x into a continuous- -40-dimensional embedding via two layers of a self-attention transformer [59]. Rather than directly converting this 640-dimensional network, which we found bit unstable to train with Equation 3, possibly due to the presence of competing objectives for a comparison), we pass the 640-dimensional embedding into M different networks that map it to M different scalar predictions (x)()) Finally, akin to attention [59] and mixture of experts [52], we train an additional head to predict weights (wo)) 2 of a linear combination of the predictions at different heads that would be equal the final prediction: fo(x) = Σi= (x). Such architecture allows the model to use different predictions (x),, depending upon the input, which allows for more stable training. To train [x)x,, we utilize the Adam [32] optimizer. Equation utilizes a procedure Opt that maximizes the learned function approximately. We utilize the same technique as Section 4 ("optimizing the learned surrogate") ") to obtain these negative samples. We periodically refresh Opt, once in every 20K gradient steps on fo(x) over training.
embedding into a scalar output via simple
Algorithm Training the conservative surrogate in PRIME
1: Initialize a neural model fe, (x) and a set M = 23 of negative particles to be updated by the firefly optimizer [x] (0), ... x (0), x (0)} to random configurations from the design space. for iteration i 0, 2, 2. until convergence do for firefly update step = O, ......4 4
2:
3:
> Inner loop do
4: Update the M fireflies according to the firefly update rule in Equation 5,
5: towards maximizing fo, (x) according to:
> Negative mining
6:
(ti + 1) = (ti) + ((x xy (ti)) ne
7:
end for
8:
Find the best firefly found in these steps to be used as the negative sample:
9:
x; = arg min ffe, (xi (ti)).... fo, (x (ti))}
> Find negative sample
10:
Run one gradient step on 0 using Equation 3 with x; as the negative sample
11:
if 2%p == 0, (p 20000), then: Reinitialize firefly particles {xī (0). ,.,,,,.))))). x (0), M (0)) to random designs.
> Periodically reinitialize the optimizer
12:
13:
end for
14: Return the final model fa- (x)
The hyperparameters for training the conservative surrogate in Equations 3 and its contextual version are as follows:
Architecture of fo(x). As indicated in Figure 4, our architecture takes in list of categor- ical (one-hot) values of different accelerator parameters (listed in Table 1). converts each parameter into 64-dimensional embedding, thus obtaining 10 x 64 sized matrix for each accelerator, and then runs two layers of self-attention [59] on it. The resulting 10 x 64 output is flattened to vector in R640 and fed into M different prediction networks that give rise to f (x)..... Jad (x), and an additional attention 2-layer feed-forward network (layer sizes [256, 256) that determines weights wi," ... WM, such that w; 2 and ΣM w; = 1. Finally the output is simply fx(x = E; (x).
Optinizer/learning rate for training (x).. Adam, le - 4, default 31 = 0.9, B2 = 0.999.
Validation set split. Top 20% high scoring points in the training dataset are used to provide a validation set for deciding coefficients a. | and the checkpoint to evaluate.
Ranges of o, B. We trained several fo(x) models with E [0.0, 0.01, 0.1, 0.1,,0.1,000 0.5, 1.0, 5.0] and 3 E [0.0, 0.01, 5.0, 0.1, 1.0]. Then we selected the best values of a and / based on the highest Kendall's 's ranking correlation on the validation set. Kendall's ranking correlation between two sets of objective values: S = {2/, U2. } corresponding to ground truth latency values on the validation set and S' {8/ -....... u corresponding to the predicted latency values on the validation set is given by J equal to:
NNN
N.N I(( 15) v/1) > 0J - ΣNj
N.N [[[
n, N;)(us v1))= S 0]
(4)
N (N -N1
20

---

Published as a conference paper at ICLR 2022
Table 17: Dataset sizes for various applications that study in this paper. Observe that all of the datasets smaller than 8000.
Application
Dataset size
MobilcNetEdgeTPU
7697
MobileNetV2
7620
MobileNetV3
5687
M4
3763
M5
5735
M6
7529
U-Net
557
R-RNN Dec
1211
t-RNN Enc
1240
Clipping fo(x) during training. Equation increases the value of the learned function fo(x) at Xo Dinfeusible and x Opt(fo)- We found that with the small dataset, these linear objectives can run into numerical instability, and produce +oo predictions. To avoid this, we clip the predicted function value both above and below by 10000.00 where the valid range of ground-truth values is (1000).
Negative sampling with Opt(-). As discussed in Section 4, we utilize the firefly optimizer for both the negative sampling step and the final optimization of the learned conservative surrogate. When used during negative sampling, we refresh (i.e., reinitialize) the firefly parameters after every p 20000 gradient steps of training the conservative surrogate, and rum t steps firefly optimization per gradient step taken on the conservative surrogate.
The initial population of fireflies depends the number of accelerator configurations (C) following the formula 10 + [ CC x 0.5]. In our setting with ten accelerator parameters (See Table 1), the initial population of fireflies is 23. We the = 1.0, 30 = 1.0, for the optimizer in all the experiments and never modify it. The update a particular optimization particle (i.e., firefly) x,, at the t-th step of optimization is given by:
Details of firefly:
same hyperparameters:
x,(t + 1) = (t) + ((x x, (t)) + i.i.d. Gaussian noise,
(5)
where x, (t), is different firefly that achieves better objective value compared to and the function 3 given by: 3(r) B
Training set details: The training dataset sizes for the studied applications are shown in Table 17. To recap, to generate the dataset, we first randomly sampled accelerators from the deign space, and evaluated them for the target application, and constituted the training set from the worst-performing feasible accelerators for the given application. Since different applications admit different feasibility criteria (differences in compilation, hardware realization, and etc.), the dataset sizes for each application are different, as the number of feasible points is different. Note however that mentioned in the main text, these datasets all contain 8000 feasible points.
Discussion on data quality: the cases of I-RNN Dec, I-RNN Enc, and U-Net, find that the number of feasible points is much smaller compared to other applications, and we suspect this because our random sampling procedure does not find enough feasible points. This is limitation of our data collection strategy and we intentionally chose this naïve strategy to keep data collection simple. Other techniques for improving data collection and making sure that the data does not consist of only infeasible points includes strategies such as utilizing logged data from past runs of online evolutionary methods, mixed with some data collected via random sampling to improve coverage of the design space.
21

---

Published as a conference paper at ICLR 2022
Table 18: Comparing the latency of the accelerators designed by the evolutionary approach for variable number of simulator access budgets (8k and 32k). Even with 4x as much allowed simulator interaction, online methods are unable to perform that well in our case.
Evolutionary (Online)
Application
Area
PRIME
8k data points 32k data points
ModileNetEdgeTPUU
29 mm²
298.50
320.28
311.35
1-RNN Dee
29 mm3
132.88
770.93
770.63
1-RNN Enc
29 mm3
130.67
865.07
865.07
Geomean of PRIME'S Improvement
(1.0x. L0x)
3.45x
3.42x
B.1.1
DETAILS OF FIREFLY USED FOR OUR ONLINE EVOLUTIONARY METHOD
In this section, we discuss some details for firefly optimization used in the online evolutionary method.
Stopping criterion: We stopped the firefly optimization when the latency of the best design found did not improve over the previous 1000 iterations, but also made sure for at least 8000 iterations, to make sure that both the online and offline methods match in terms of the data budget. We also provide the convergence curves for firefly optimization on various single-appe applliaappii problems from Table 3 in Figure 8.
firefly optimization
What happens if run firefly optimization for longer? We also experimented with running the evolutionary methods for longer (i.e., 32k simulator accesses compared to 8k), to check if this improves the performance of the evolutionary approach. As shown in Table 18, we find that while this procedure does improve performance in some cases, the performance does not improve much beyond 8k steps. This indicates that there is possibility that online methods can perform better than PRIME if they are run for many more optimization iterations against the simulator, but they may not be as data-efficient as PRIME.
Hyperparameter tuning for firefly: Since the online optimization algorithms we run have access to querying the simulator the course of training, we can simply utilize the value of the latest proposed design as a way to perform early stopping and hyperparameter tuning. A naïve way to perform hyperparameter tuning for such evolutionary methods is run the algorithm for multiple rounds with multiple hyperparameters, however this is compute and time intensive. Therefore, we adopted dynamic hyperparameter tuning strategy. Our implementation of the firefly optimizer tunes hyperparameters by scoring a set of hyperparameters based on its best performance over a sliding window of T data points. This allows us to adapt to the best hyperparameters on the fly, within the course of optimization, effectively balancing the number of runs that need to be run in the simulator and hyperparameter tuning. This dynamic hyperparameter tuning strategy requires some initial coverage of the hyperparameter space before hyperparameter tuning begins, and therefore, this tuning begins only after 750 datapoints. After this initial phase, every T = 50 iterations, the parameters y and Bo updated via evolutionary scoring strategy towards their best value.
Discussion of t-RNN Enc and t-RNN Dec. Finally, we discuss the results of the evolutionary approach on the t-RNN Enc and t-RNN Dec tasks, for which the convergence plots are shown in Figures 8h and 8i. Observe that the best solution found by this optimization procedure converges quite quickly in this case (with about 1000 iterations) and the evolutionary method, despite the dynamic hyperparameter tuning is unable to find a better solution. We hypothesize that this is because the performance of local optimization method may suffer heavily due to the poor landscape of the objective function, and it may get stuck if it continuously observes only infeasible points over the course of optimization.
1.1.2 EXACT HYPERPARAMETER FOUND BY OUR CROSS- VALIDATION STRATEGY
In this section, present the exact hyperparameters found by our cross-validation strategy discussed in Section 4. To recap. our offline cross- validation strategy finds the early stopping checkpoint and held-out
selects the values of and 3 in Equation 3 that attain the highest rank correlation validation set consisting of top 20% of the dataset feasible samples. The values of a, 3 and checkpoint selected for the experiments in Table 3, Table 4, Table and are shown in Table 19.
22

---

Published as a conference paper at ICLR 2022
900
2020
1730
1820
1310
1820
L200
1.50
1.400
730
500
850
520
300
250
of Data Po
of Data Points
of Data Poincs
(a) Tobi
(b) MobileNetV2
(c) MobileNetV3
2000
125
200
275
503
250
1200
Latevey
225
2000
400
200
of Data Points
of Data Prints
of Data Points
(d) M4
(e) M5
(f) M6
1000
2200
2350
1000
18000
.400
1400
1750
2200
1.000
1002
1200
1350
1000
060
200
4020
5002
4002
2002
2300
4000
3500
Points
(g) U-Net
(h) I-RNN Dec
(i) I-RNN Enc
Figure 8: Optimization behavior of Firefly optimizer (Online Evolutionary). Observe that the optimization procedure converges and plateaus very quickly (at least 1000 iterations advance) and hence we stop at 8000 iterations. In the case of (-RNN Enc and I-RNN Dec, we find that the evolutionary algorithm performs poorly and we suspect this is because it saturates quite quickly to a suboptimal solution and is unable to escape. This is also evident from Figures 8h and 8i, where observe that online optimization platcaus the fastest for these RNN applications.
B.2 DETAILS OF ARCHITECTING ACCELERATORS FOR MULTIPLE APPLICATIONS SIMULTANEOUSLY
Now we will provide details of the tasks from Table 4 where the goal is to architect an accelerator which is jointly optimized for multiple application models. For such tasks, augment data-pot data-points for each model with the context vector Cj from Table 2 that summarizes certain parameters for each application. For entries in this context vector that have extremely high magnitudes (e.g., model parameters and number of compute operations), we normalize the values by the sum of values across the applications considered only encode the relative scale, and not the absolute value which is not required. To better visualize the number of feasible accelerators for joint optimization. Figure 9 show the tSNE plot (raw architecture configure configurations are used as input) of high-performing accelerator configurations. The blue-colored dots are the jointly feasible accelerators in the combined dataset, and note that these data points are no more than 20-30 in total. The highlighted red star presents the best design suggested by PRIME with average latency of 334.70 (Table 4). This indicates that this contextual, multi- aapti-application problem poses a challenge for data-driven methods: these methods need to produce optimized designs even though very few accelerators are jointly feasible in the combined dataset. Despite this limitation, PRIME successfully finds more efficient accelerator configurations that attain low latency values each of the applications jointly, as shown in Table 4.
23

---

Published as a conference paper at ICLR 2022
Table 19: Hyperparameters a, 3 and checkpoint index (measured in terms of gradient steps on the learned conservative model) for PRIME found by our offfine cross-validation strategy discussed in Section 4, that is based the Kendall's rank correlation tune hyperparameters) In the case of the multi-tast and zero-shot scenarios, when training on more than one application, the batch size used for training PRIME increases to N -flold, where N is the number of applications in the training set, therefore we likely find that even few gradient steps are good enough.
the validation set (note that simulator queries were used to
Table
Application
Checkpoint Index
Table 3
MobeleNetEdgeTPU
0.01
5.0
80000
Table 3
MobileNerV2
5.0
5.0
1200000
Table 3
MobileNeiV3
5.0
0.01
800000
Tablo
M5
80000
Table 3
M6
600000
Table 3
Ne
0.0
1.0
1000000
Table 3
ERNN Dec
1.0
0.0
60000
Table
I-NNN Eice
60000
Tablo 4
MobileNet (EdgeTPU. V2, V3)
5.0
0.01
60000
Tablo
(Vb V3), M5, M6
30000
Table 4
MobileNei (PU. V2, V3), M4, M5, M6
0.5
0.0
1000000
Tabli 4
MobileNet (EdgeTPU, V2, V3), M4, MS, M6, E-RNN Eoc (Area 29.0)
0.0
1.0
20000
Tablo 4
MobileNet (EdgeTIPU, V2, V3), M4, MS,, M6, I-RNN Eoc (Area 100.0)
0.0
0.0
20000
Table 4
MobileNet (Edge V2, V3), M4, M.S, M6, U-Net, I-NN (Enc, Dec) (Area 29.0)
0.01
0.01
10000
4
MobileNet (EdgeTTPU V2, V3), M4, M.S, M6, U.Net, I-NNN (Enc, Dec) (Area 100.0)
0.01
20000
Table 5
Train (7ero-Shot): Mor (EdgeTPUU V3)
0.01
60000
Tabls
Train (Zero-Shot): MobileNer (V2, M5, MGG
30000
Tablc 5
Train (Zero-Sbett; MobileNer (Edge IPU, V2, V3), M4, M5. M6, I-RNN Enc (Area 29.0)
0.0
1.0
20000
Table 5
Train (Sem-Shott; MobileNer (PU) V2, V3), M4, MS, M6, N-RNN Finc (Area 100.0)
0.1
5.0
20000
Table 6
MobileNeiV2 (NVDLAA)
0.0
1.0
40000
Tablo 6
MobileNetV2 (ShanDDinnNa))
0.0
0.0
40000
Table 6
ResNet 50 (NVDLA)
0.01
0.0
40000
Table 6
ResNet Nam)nD)a
75000
Table 6
Transformer (NVDLA)
0.01
200000
Table 6
Transformer Noo))
0.0
0.1
100000
15.0
10.0
5.0
1SNE-
0.0
5.0
-10.0
11.00
-15.0
-10.0
-5.0
0.0
5.0
10.0
15.0
ISNE x
Figure (SNE plot of the joint dataset and randomly sampled infeasible data points. The blue points show the accelerator configurations that are jointly feasible for all the applications. The highlighted point with red star shows the best design proposed by PRIMEThe rest of the points show the infeasible points.
B.3 DATASET SENSITIVITY TO ACCELERATOR PARAMETERS
We visualize the sensitivity of the objective function (e.g. latency) with respect to the changes in certain accelerator parameters, such as memory size (Table 1), in Figure 1lb, illustrating this sensitivity. As shown in the Figure, the latency objective that we seek to optimize can exhibit high sensitivity to small variations in the architecture parameters, making the optimization landscape particularly ill-behaved. Thus, small change in one of the discrete parameters, can induce large change in the optimization objective. This characteristic of the dataset further makes the optimization task challenging.
24

---

Published as a conference paper at ICLR 2022
Decoder
NBin
PE
PE
...
PE
Buffer Contralle
PE
NBout
PE
PE
ALU
Figure 10: Overview of NaiDianNao dataflow accelerator. This dataflow accelerators exhibits output- stationary dataflow where keeps the partial results stationary within each processing elements (PEs),
Table 20: The evaluated applications, their model parameter size, number of compute operations, and normalized compute-to-memory ratio.
Name
Model Param
of Compute Ops.
Normalized Compute-fo-Memory Ratio
MobileNetEdgeePPL
3.87 MB
1,989,81 1,168
1.38e-
MobileNelV2
3.31
609,353,376
4.96e-2
MobilcNet3
5.20
449,2199,000
2.33c-2
M4
6.23 MB
3,471.920,128
1.5c-1
2.16
939,752.960
1.17
M6
0.41 MB
228, 146,848
1.5c-1
U-eet
3.69 MB
13,707,214,844
1.0
ERNN
ERNN Enc
21.62 MB
45,621,248
5,68e-4
C OVERVIEW OF ACCELERATORS AND SEARCH SPACE
This section briefly discuss the additional accelerators (similar to [30]) that we evaluate in this work, namely NVDLA [43] and ShiDianNao [10], and their corresponding search spaces.
NVDLA:
Nvidia Deep Learning Accelerator NVDLA [42] is an open architecture inference accel- erator designed and maintained by Nvidia. In compared other inference accelerators, NVDLA is weight stationary accelerator. That is, it retains the model parameters on each processing elements and parallelizes the computations across input and output channels. NVDLA-style dataflow accelerators generally yield better performance for the computations of layers at the later processing stages. This because these layers generally have larger model parameters that could benefit from less data movement associated to the model parameters.
Vision Accelerator Figure 10 shows the high-level schematic of ShiDianNao acceler- ator [10]. ShiDianNao-style dataflow accelerator an output-stationary accelerator. That is, keeps the partial results inside each PE and instead move the model parameters and input channel data. As such, in compared to NVDLA-style accelerators, ShiDianNao provides better performance for the computations of the layers with large output channels (generally first few layers of model).
ShiDianNao:
Search space of dataflow accelerators. We follow similar methodology as [30] to evaluate additional hardware accelerators, discussed in the previous paragraphs. We use MAESTRO [37]. an analytical cost model, that supports the performance modeling of various dataflow accelerators. In this joint accelerator design and dataflow optimization problem, the total number of parameters to be optimized is up to 106-the tuple of (# of PEs, Buffers) per per model layer-wile with each parameter taking one of 12 discrete values. This makes the hardware search space consist of 2.5 x 10 accelerator configurations. We also note that while the method proposed in [30] treats the accelerator design problem as sequential decision making problem, and reinforcement learning techniques, PRIME simply designs the whole accelerator in a single step, treating as a model-based optimization problem.
D SUBSET OF APPLICATIONS FOR GOOD ZERO-SHOT PERFORMANCE
In this section, we present the results of an ablation study with the goal to identify subset of applications such that training data from only these applications yields good zero-shot performance
25

---

Published as a conference paper at ICLR 2022
x10
4.0
infeasible False True
3.5
700.0
3.0
Count
2.5
Runtime
650.0
2.0
1.5
600.0
1.0
0.5
550.0
0.0
0.0
2000.0
4000.0
6000.0
8000.0
10000 .
12000.0
14000.
16000.0
500.0
0
4 x10
Runtime
Core Memory Size
Figure 11: The (a) histogram of infeasible (orange bar with large score vflussbee (blue bars) data points and (b) the sensitivity of runtime to the size of core memory for the MobileNe(EdgeTPPU [26] dataset.
30
10
-10
-20
-30
-20
-10
1.0
x1
Figure 12: tSNE plot of the infeasible and feasible hardware accelerator designs. Note that feasible designs (shown in blue) embedded sea of infeasible designs (shown red), which makes this a challenging domain optimization methods.
across all the nine applications studied in this work. Since cannot train PRIME for every subset of applications because the space of subsets of all applications is exponentially large, we utilized some heuristics in devising the subset of applications would train on, with the goal to make interesting observations that allow us to devise rough guidelines for performing application selection.
Our heuristic for devising subsets applications: Building on the intuition that applications with very different compute to memory ratios (shown in Table 20) may require different accelerator designs for example, if our goal is to run a compute-intensive intensive application, we likely need an accelerator design with more compute units MobileNerV3,, M6, M5, and (2) MobileNetV2, MobilenetV3, M5, t-RNN Enc. Note that, these two combinations only differ in whether some RNN application Table 20, the I-RNN applications admit very diffeernnt compute to memory ratio, for instance, while this ratio is 5.68e 4 for t-RNN Enc and I-RNN Dec, much different 0.01 0.2 for other models MobileNetEdge t-RNN Enc and Dec will require different kinds of accelerators for good performance compared to the other applications.
study two subsets of training applications: (1) MobileNetV2,
used in training not. As shown in
MobileNetV2 MobileNetV3, M5, and M6. This means that likely
Results: We present the performance of zero-shot evaluating the designed accelerator obtained by training on combinations (1) and (2), and also the accelerator obtained by training (3) seven applications from Table in Table 21 as reference. We make some key takeaways from the results:
The performance of both configuration (1) and training with seven applications ((3,, last of Table 21) are similar.
26

---

Published as a conference paper at ICLR 2022
1000
y=x
Accelerator configs
800
Predicted Latency
600
400
200
200
600 Actual Latency
400
800
1000
Figure 13: To verify if the overestimation hypothesis-that optimizing accelerator against a naive standard surrogate model is likely to find optimizers that appear promising under the learned model, but do not actually attain low-latency values-in our domain, we plot a calibration plot optimizing a naively trained standard surrogate model. In the scatter plot, we represent each accelerator as point with its x-coordinate equal to the actual latency obtained by running this accelerator design in the simulator and the y-coordinate equal to the predicted latency under the learned surrogate. Note that for large chunk of designs, the predicted latency is much smaller than their actual latency (i.c., these designs lie beneath the y = line in the plot above). This means that optimizing designs under naive surrogate model is prone to finding designs that appear overly promising (i.e., attain lower predicted latency values), but are not actually promising. This confirms the presence of the overestimation hypothesis on our problem domain.
top accelerator designs found by
PRIME does not overestimate
PRIME (zoomed in)
16000
16000
14000
14000
12000
y=x
Accelerator configs
5000
4000
4003
y=X Accelerator configs
2003
2000 4000 6050 8050 10000 12000 14030 16000
200
300
400
500
600
700
800
300
1000
Actual Latency
Actual Latency
Figure showing the calibration plot of predicted (y-axis) actual latencies (x-axis) of accelerators found by PRIME. Compared Figure 13, observe that all the acclerator configurations above y x. meaning that PRIME predicts a higher latency (y-axis) compared to the actual latency. This means that PRIME does not think that accelerators that attain high-latency values under the simulator, actually good. We also provide zoomed-in version of the plot the right, which shows that there accelerators do have meaningfully distinct latency predictions under PRIME. Observe in the zoomed-in plot latencies also perform relatively better under the actual latency compared to the designs that attain larger predicted latency of ~ 14000- 1600 under the PRIME surrogate. Optimizing against PRIME is still effective because optimization just needs relative correctness of values, not absolutely correct latency predictions.
designs that attain small predicted
In case (2), when the training applications consist of four applications in which one appli- cation is t-RNN Enc, with drastically different compute to memory ratio (Table 20), the performance on an average across all applications becomes slightly worse (compare the performance in (2) (3)).
Conclusion and guidance selecting good applications: The above results indicate that only few applications (e.g., four applications in case (1)) can be enough for good performance on nine applications. While this set may not be not minimal, it is certainly much smaller set compared to the nine applications applicaaiin considered. Adding an RNN application in case (2) increases latency in compared to case (1). because t-RNN Enc likely admits a very different optimal accelerator compared to the other applications due to a very different compute/memory ratio, which in turn skews the generalization of the surrogate learned by PRIME when trained only this limited set of four applications. However,
27

---

Published as a conference paper at ICLR 2022
Additional ablation study under zero-shot setting when the test applications include all the nine evaluated models (e.g. MobileNet (EdgeTPU, V2. V3), M4, M5, M6, I-RNN Dec, t-RNN Enc, U-Net). Lower latency is better. From left to right: the applications used to train the surrogate model in PRIME, the area constraint of the accelerator, PRIME'S (hest, median) latency.
Table 21:
Train Applications
PRIME (best, median)
(1) MobileNet (V2, V3), M5, M6
29 mm3
(426.65, 427.35)
(2) MobileNet (V2, V3), M5, I-RNN Enc
29 mm3
(461.79, 464.87)
(3) MobileNet (EdgeTPU, V2, V3), M4, M5, M6, I-RNN Enc
29 mm
(426.65, 427.94)
when seven applications are provided in case (3), even when the set of training applications includes I-RNN, its contribution on the PRIME surrogate is reduced since many other compute intensive applications are also provided in the training set and the resulting accelerator performs well.
Practitioner guidance: The primary practitioner guidance we can conclude here is that the models used for training must be representative of the overall distribution of the target models that we want to zero-shot generalize to. Since a number of our applications are compute intensive, we were able to simply utilize set (1) to attain good performance on all the applications. On the other hand, in case (2), when the t-RNN Enc application was over-represented while seven of nine applications we considered were primarily compute intensive, one out of four applications we used for training in case (2) were memory intensive this hurt performance on the overall set. Therefore, we believe that ensuring that the training subset of applications is adequately aligned with the overall set of applications in terms of computc/memory ratio statistic is imperative for favorable zero-shot performance.
For a practitioner deciding between zero-shot generalization and additional data collection, it may make sense to test if the target application admits a similar value of the compute to memory ratio as an already existing application. If it does, then the practitioner might be able to utilize the zero-shot generalization, as is indicated with the good performance of case (1), whereas the compute/memory ratio is heavily different from any seen application, zero-shot generalization the target application may be worse. Finally, making sure that the training applications adequately reflect the compute/memory ratio statistic for the overall target set important.
28


Reviews:
[{"review_id": "pT--Yyww9DE", "reviewer": "Reviewer_7gpC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This work attacks a very commonly seen problem from prior work: the over-optimism and the uselessness of the infeasible design points in hardware performance modeling. Problems are well-defined and solutions are targeted and justified. The technical presentation of the methods proposed is solid. \n- Experiments are extensive and have compared offline and online methods very thoroughly. The paper also shows the results of applying PRIME on other accelerator designs to examine the applicability. \n- Writing is clear and the paper is easy to follow.\n\nWeakness:\n- The challenge of simulation time in online methods is now shifted to dataset construction. How is the cost of constructing the training datasets, especially when a new dataset is required for a new context (application)? \n- Figure 12 shows the challenge of overoptimism in performance modeling. However, there is no experiment showing the difference of predicted and measured latency/energy when using the proposed method. It is not clear whether the results of PRIME are produced by the performance model or by the simulation/measurement using the optimization results.\n- The results in Table 5 are not enough to show the capability of PRIME under the zero-shot setting. One of the benefits of the offline method is to reuse the logged training data. It would be better if authors could show the results on all benchmarks (instead of 1 or 2 networks) when using different training application subsets. Meanwhile, given a new application, when will PRIME choose to build a new training dataset and when will PRIME choose to use a zero-shot setting? How should PRIME choose the training applications? What is the minimal training application subset to get the best results on all benchmarks?\n", "summary_of_the_paper": "This paper mainly focuses on the architectural sizing of the neural network accelerators. Optimizing accelerator parameters is expensive due to the time-consuming simulation and numerous infeasible design candidates. This work presents a data-driven offline optimization framework, PRIME, that reuses the previous simulation results to produce new designs without additional active queries to an explicit silicon or simulator. The key idea is to train a conservative surrogate model, that predicts the cost (e.g. latency) of an architecture candidate, by minimizing the prediction error of feasible designs and penalizing the infeasible designs for avoiding optimism. Experiments show that PRIME offers 1.20x - 1.54x speedup on average over manual design and 1.26x speedup for unseen applications.", "summary_of_the_review": "This work presents an effective framework PRIME to tackle the challenges of automating hardware design optimization. Though it seems to be an incremental medication of previously published efforts, the proposed method is clear and useful. The paper could be stronger with a better analysis as stated in the Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "isV66KqhsFb", "reviewer": "Reviewer_cuvZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Novelty and Significance\n\nThe paper addresses an important emerging problem, designing custom hardware architectures to accelerate machine learning algorithms; however, the presented algorithm is general to any configuration tuning / design space exploration approach. None of the techniques within the paper are deeply novel (using adversarial training in the loss function [1], adding feasibility as a term in the loss function [2], employing context vectors to reuse design space exploration for multiple applications [3], and the actual optimizer used to optimize against Prime [4]), the combination of approaches and application to the specific problem under study is sufficiently novel to be worth publishing. Regardless, I would appreciate a more strong statement of precisely what components of the paper should be considered novel.\n\n## Correctness and Clarity\n\nAt a high level, the approach and evaluation in the paper seems mostly correct, and the paper is very easy to read. However, I do have the following questions and concerns:\n- Many of the details of the evaluation are omitted. Specifically:\n  - (*) what is the size of the training set?\n  - (*) What is the stopping criteria for the various techniques (specifically, how many iterations is fireflies ran for against the surrogate)?\n  - What is the total amount of compute for Prime? Figures 5 and 6 show the simulation time (which seems to be increasing for Prime, which I also don't understand, since I had thought that Prime was entirely offline), but does not include the cost of training and evaluating the surrogate.\n- I am not entirely convinced that all of the experimental results are correct. Specifically, the following results are surprising enough that I would like some clarification from the authors:\n  - (*) Bayes Opt can result in performance worse than D(Best in training) on t-RNN Enc -- though it depends on exactly how the final point is selected, this should likely not be possible.\n  - \"These results indicates that offline optimization of accelerators using Prime can be more data-efficient compared to online methods with active simulation queries\" (page 7). This is a very surprising result to me. Online approaches should lead to at least as good accuracy or data efficiency than offline approaches given the  (since any given offline approach can be viewed as an instance of an online approach, whereas the converse is not true). Specifically, I would expect the online version of the fireflies approach to perform better than the offline version (unless there is some characteristic of the surface induced by Prime that is a better fit for optimizing with fireflies). This finding then indicates that either there is significant room for improvement in online approaches for this task, or that the chosen baselines or evaluation metrics are not correct. The paper should include more discussion of this point, or more narrowly scope it to the specific online approaches evaluated in the paper.\n- \"Top 20% high scoring points in the training dataset are used to provide a validation set...\" This is a fascinating point, and I think should be highlighted significantly more in the paper. It seems plausible that this is a key element of the methodology. The conventional wisdom is that DNNs do not extrapolate outside of their training distribution. This approach explicitly requires extrapolation (finding accurate dips between sampled data); because the validation points are not IID and there is a significant amount of hyperparameter search performed, it is possible that selecting for the model that has the best prediction on this set of points selects for a model that is able to generalize to dips in latency space a bit more. I have several questions for the authors based on this point:\n  - (*) Is the surrogate model that you ultimately select the exact model resulting from hyperparameter search / validation, or is the model re-trained from scratch with the discovered hyperparameters?\n  - Have the authors experimented with other splits between the training and validation set (e.g., training on the best points and validating on the worst)?\n  - (*) Is the reported D(Best in training) including the points in this validation set?\n  - Are the hyperparameters for the other approaches also tuned using this validation set?\n  - (*) What are the actual discovered hyperparameters after this hyperparameter search?\n- What fraction of the space is infeasible? How do you do your initial sampling to avoid these infeasible points? What happens if all your final points are infeasible? This is a generally hard problem in high dimensional space. This is a challenge for all optimization algorithms here, the paper doesn't at all discuss the tractability of this sampling or what to do if the final points are all infeasible.\n- How sensitive is the optimization to the number of particles, and the parameters of the negative mining optimization?\n- As a minor point, alpha is overloaded within the approach: it is used as both the area constraint (Equation 1) and the COM hyperparameter (Equation 2)\n\n[1] Brandon Trabucco, Aviral Kumar, Xinyang Geng, Sergey Levine. Conservative Objective Models for Effective Offline Model-Based Optimization. ICML, 2021.\n\n[2] https://en.wikipedia.org/wiki/Barrier_function\n\n[3] https://en.wikipedia.org/wiki/Multi-armed_bandit#Contextual_bandit\n\n[4] Xin-She Yang. Nature-Inspired Metaheuristic Algorithms. Luniver Press, 2010.\n\n\n# Response to author response\n\nThanks to the authors for the very detailed response. All of the responses to the (*) questions are satisfactory, allowing for sufficient understanding of the experimental setting to understand and reproduce results. I'm also pleased to see the additional clarifications and experimental results (especially the clarifications around the hyperparameter search process and the additional validation set experiments) -- these results have convinced me that Prime is doing something beyond just searching for the best point in the validation set. Based on the response, I plan to raise my score to an accept.\n", "summary_of_the_paper": "The paper proposes an approach to optimizing parameters of hardware architectures to design architectures that are more efficiently able to execute neural networks. To accomplish this, the paper proposes Prime, an offline optimization algorithm that constructs a surrogate of the design space then optimizes against that design space. Specifically, Prime is trained to be robust to both unseen points in the design space (rather than underestimating their cost) and to infeasible points. For each of these goals, Prima adds a term to the loss function, penalizing adversarial examples and infeasible training points. The authors evaluate Prime by showing that given the same number of evaluations of the ground-truth data, Prime results in lower cost results than online optimization.", "summary_of_the_review": "The paper proposes an interesting approach to offline design space exploration, and demonstrates that it performs better than online approaches. Though I am not totally satisfied with the evaluation, it is a relatively low lift to bring it up to par with existing literature in the space. A more thorough evaluation would make more a much more compelling paper, but in my view this is not necessary for acceptance.\n\nWeak accept, conditioned on the authors providing more details about the evaluation (specifically, all of the questions marked with (*) above). I would be willing to raise my score further if the authors address some of my other questions (evaluating the sensitivity of the approach to the validation set, discussion the tractability of optimizing with infeasible points, evaluating the amount of compute required to train and evaluate the surrogate, etc).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XMbjHwHkxKv", "reviewer": "Reviewer_aghq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors take a clearly defined problem (design a neural net hardware accelerator), motivate it well (simulations are expensive; surrogates struggle with these specific problems in this domain), introduce a solution targeted at overcoming those specific issues, and then demonstrate that it generally works better than other accepted approaches.\n\nStrengths:\n\n- The evaluation results are compelling. While the absolute design point gains are modest, they must be understood in the context of dramatically lower computational time, which is a huge selling point by itself. I appreciated the various evaluation axes the authors have explored: single accelerator, shared (multi-target) accelerator, zero-shot, and applicability to multiple accelerator design paradigms.\n- The organization is logical, and the writing is for the most part clear. I appreciate that writing papers on hardware accelerators in the ML community is sometimes difficult due to wildly varying levels of background in the audience. I feel the authors have struck a solid balance here: detail enough for those with a background, but helpfully reducing the discussion to just the important pieces.\n\nWeaknesses:\n\n- The paper presents PRIME as its solution, however the actual novel contributions are not particularly well analyzed. The authors highlight infeasible design points and a rough optimization landscape as the key problem characteristics, but the evaluation results and analysis are almost strictly black-box tests. This is a bit unfortunate because it makes it harder to understand how the proposed modifications are contributing to actually solving the described problems in a quantitative way. Some of this is discussed in appendices (e.g., the ablation study in A.6), but it is brief (and relegated to an appendix).\n\n- The paper is squished. It seems clear the authors had a lot of material to incorporate, but quality has been sacrificed somewhat. Margins and labels on figures and tables in the paper are hard to read and run into text.\n\n\n\nDisclosure: I have previously anonymously reviewed a version of this work. Comments below reflect the progress and are mostly for the authors' benefit.\n\nI applaud the authors for the changes they have incorporated into this paper based on past reviews. Prior concerns about generalizability on both application and hardware are largely gone, and the paper has mostly retained its clarity. Some of the intuitive explanations have been omitted (probably due to space concerns), but it still generally holds up. This is a better paper.\n\nI did notice that the discussion of prediction behavior as it scales with dataset sample size is gone. This is a bit unfortunate since it was one of the pieces that helped shed light on how the method worked (and where it breaks down). I don't believe this fundamentally harms the takeaway of the paper (since the existing evaluations are pretty clear that the method can be made to work in a variety of situations), but it might be nice if that analysis made its way back into an appendix.", "summary_of_the_paper": "The authors present an offline performance model for optimizing neural net hardware accelerators. They introduce a mechanism for dealing with non-viable hardware design points and for specializing towards specific applications from a single surrogate. They present results versus online methods on a variety of axes.", "summary_of_the_review": "The paper is clear, describes its problem and solution well, clearly showcases its results across a variety of tests, and generally succeeds at convincing the reader that the approach works as advertised. There is room for improving the 'why', but the paper is solid nonetheless.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1TgebXBl1dy", "reviewer": "Reviewer_TEzV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are as follows: \n+ This paper provides a new perspective for tackling the overfitting of hardware efficiency predictors by leveraging adversarial training and infeasible design samples;\n+ The developed tool can be useful as an early-stage development tool, and the released accelerator dataset can be useful to the community;\n+ This paper is the first to consider unseen applications in a zero-shot setting when it comes to accelerator optimization. \n\nThe weaknesses of the paper are below: \n- This paper does not review, discuss, and evaluate against another group of related methods for designing hardware accelerators, which considers automated accelerator designs, e.g., [MAGNet, ICCAD 2019] and AutoDNNchip, FPGA 2020];\n- This work does not evaluate the resulting accelerators with any SOTA dedicated accelerators. I think it is not convincing for a accelerator tool to be practical without evaluating where the resulting design with the dedicated expert design for the same application/task.", "summary_of_the_paper": "The authors first introduce two practices for application-specific hardware accelerators: 1) Designers need to spend considerable manual effort and perform large number of time-consuming simulations to find accelerators that can accelerate multiple target applications while obeying design constraints. Moreover, such a simulation-driven approach must be re-run from scratch every time the set of target applications or design constraints change. 2) An alternative paradigm is to use a data-driven, offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations. Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes.\n\nAs such, they develop such a data-driven offline optimization method for designing hardware accelerators, dubbed PRIME, which learns a conservative estimate of the desired cost function, utilizes infeasible points, and optimizes the design against this estimate without any additional simulator queries during optimization. The authors evaluate PRIME architects accelerators---tailored towards both single- and multi-applications as well as unseen applications in a zero-shot setting.", "summary_of_the_review": "This paper develops a data-driven offline optimization for identifying good hardware accelerator designs, and conducts sufficient ablation studies to validate their methods, while the evaluation against SOTA methods can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors give an effective framework PRIME to tackle the challenges of automating hardware design optimization.  This problem is of importance to the community.  Overall, the reviewers thought the paper gave a nice clean approach to the problem and that the community would be interested with these results.
Decision: Accept (Poster)