Data-Driven Offline Optimization for Architecting Hardware Accelerators

1 INTRODUCTION . The death of Moore ’ s Law ( Esmaeilzadeh et al. , 2011 ) and its spiraling effect on the semiconductor industry have driven the growth of specialized hardware accelerators . These specialized accelerators are tailored to specific applications ( Yazdanbakhsh et al. , 2021a ; Reagen et al. , 2017 ; Nardi et al. , 2019 ; Shi et al. , 2020 ) . To design specialized accelerators , designers first spend considerable amounts of time developing simulators that closely model the real accelerator performance , and then optimize the accelerator using the simulator . While such simulators can automate accelerator design , this requires a large number of simulator queries for each new design , both in terms of simulation time and compute requirements , and this cost increases with the size of the design space ( Yazdanbakhsh et al. , 2021b ; Shi et al. , 2020 ; Hegde et al. , 2021 ) . Moreover , most of the accelerators in the design space are typically infeasible ( Hegde et al. , 2021 ; Yazdanbakhsh et al. , 2021a ) because of build errors in silicon or compilation/mapping failures . When the target applications change or a new application is added , the complete simulation-driven procedure is generally repeated . To make such approaches efficient and practically viable , designers typically “ bake-in ” constraints or otherwise narrow the search space , but such constraints can leave out high-performing solutions ( Dave et al. , 2019 ; Parashar et al. , 2019 ; Chatarasi et al. , 2020 ) . An alternate approach , proposed in this work , is to devise a data-driven optimization method that only utilizes a database of previously tested accelerator designs , annotated with measured performance metrics , to produce new optimized designs without additional active queries to an explicit silicon or a cycle-accurate simulator . Such a data-driven approach provides three key benefits : ( 1 ) it significantly shortens the recurring cost of running large-scale simulation sweeps , ( 2 ) it alleviates the need to explicitly bake in domain knowledge or search space pruning , and ( 3 ) it enables data re-use by empowering the designer to optimize accelerators for new unseen applications , by the virtue of effective generalization . While data-driven approaches have shown promising results in biology ( Fu & Levine , 2021 ; Brookes et al. , 2019 ; Trabucco et al. , 2021a ) , using offline optimization methods to design accelerators has been challenging primarily due to the abundance of infeasible design points ( Yazdanbakhsh et al. , 2021a ; Hegde et al. , 2021 ) ( See Figure 3 and Appendix Figure 13 ) . The key contribution of this paper is a data-driven approach , PRIME , to automatically architect high-performing application-specific accelerators by using only previously collected offline data . PRIME learns a robust surrogate model of the task objective function from an existing offline dataset , and finds high-performing application-specific accelerators by optimizing the architectural parameters against this learned surrogate function , as shown in Figure 1 . While naïvely learned surrogate functions usually produces poor-performing , outof-distribution designs that appear quite optimistic under the learned surrogate ( Kumar & Levine , 2019 ; Brookes et al. , 2019 ; Trabucco et al. , 2021a ) . The robust surrogate in PRIME is explicitly trained to prevent overestimation on “ adversarial ” designs that would be found during optimization . Furthermore , in contrast to prior works that discard infeasible points ( Hegde et al. , 2021 ; Trabucco et al. , 2021a ) , our proposed method instead incorporates infeasible points when learning the conservative surrogate by treating them as additional negative samples . During evaluation , PRIME optimizes the learned conservative surrogate using a discrete optimizer . Our results show that PRIME architects hardware accelerators that improve over the best design in the training dataset , on average , by 2.46× ( up to 6.7× ) when specializing for a single application . In this case , PRIME also improves over the best conventional simulator-driven optimization methods by 1.54× ( up to 6.6× ) . These performance improvements are obtained while reducing the total simulation time to merely 7 % and 1 % of that of the simulator-driven methods for single-task and multi-task optimization , respectively . More importantly , a contextual version of PRIME can design accelerators that are jointly optimal for a set of nine applications without requiring any additional domain information . In this challenging setting , PRIME improves over simulator-driven methods , which tend to scale poorly as more applications are added , by 1.38× . Finally , we show that the surrogates trained with PRIME on a set of training applications can be readily used to obtain accelerators for unseen target applications , without any retraining on the new application . Even in this zero-shot optimization scenario , PRIME outperforms simulator-based methods that require re-training and active simulation queries by up to 1.67× . In summary , PRIME allows us to effectively address the shortcomings of simulation-driven approaches , ( 1 ) significantly reduces the simulation time , ( 2 ) enables data reuse and enjoys generalization properties , and ( 3 ) does not require domain-specific engineering or search space pruning . To facilitate further research in leveraging machine learning in architecting hardware accelerators , we also release the dataset in our experiments ( Appendix B.3 ) , consisting of many accelerator design points ( both infeasible and feasible ) . 2 BACKGROUND ON HARDWARE ACCELERATORS . The goal of specialized hardware accelerators—Google TPUs ( Jouppi et al. , 2017 ; Gupta & Akin , 2020 ) , Nvidia GPUs ( Nvidia , 2021b ) , GraphCore ( GraphCore , 2021 ) —is to improve the performance of specific applications , such as machine learning models . To design such accelerators , architects typically create a parameterized design and sweep over parameters using simulation . Target hardware accelerators . Our primary evaluation uses an industry-grade and highly parameterized template-based accelerator following prior work ( Yazdanbakhsh et al. , 2021b ) . This template enables architects to determine the organization of various components , such as compute units , memory cells , memory , etc. , by searching for these configurations in a discrete design space . Some ML applications may have large memory requirements ( e.g. , large language models ( Brown et al. , 2020 ) ) demanding sufficient on-chip memory resources , while others may benefit from more compute blocks . The hardware design workflow directly selects the values of these parameters . In addition to this accelerator and to further show the generality of our method to other accelerator design problems , we evaluate two distinct dataflow accelerators with different search spaces , namely NVDLA-style ( Nvidia , 2021a ) and ShiDianNao-style ( Du et al. , 2015 ) from Kao et al . ( 2020 ) ( See Section 6 and Appendix C for a detailed discussion ; See Table 6 for results ) . Table 1 : The accelerator design space parameters for the primary accelerator search space targeted in this work . The maximum possible number of accelerator designs ( including feasible and infeasible designs ) is 452,760,000 . PRIME only uses a small randomly sampled subset of the search space . Accelerator Parameter # discrete values Accelerator Parameter # discrete values # of PEs-X 10 # of PEs-Y 10 PE Memory 7 # of Cores 7 Core Memory 11 # of Compute Lanes 10 Instruction Memory 4 Parameter Memory 5 Activation Memory 7 DRAM Bandwidth 6 How does an accelerator work ? We briefly explain the computation flow on our template-based accelerators ( Figure 2 ) and refer the readers to Appendix C for details on other accelerators . This template-based accelerator is a 2D array of processing elements ( PEs ) . Each PE is capable of performing matrix multiplications in a single instruction multiple data ( SIMD ) paradigm ( Gove et al. , 1993 ) . A controller orchestrates the data transfer ( both activations and model parameters ) between off-chip DRAM memory and the on-chip buffers and also reads in and manages the instructions ( e.g . convolution , pooling , etc . ) for execution . The computation stages on such accelerators start by sending a set of activations to the compute lanes , executing them in SIMD manner , and either storing the partial computation results or offloading them back into off-chip memory . Compared to prior works ( Hegde et al. , 2021 ; Du et al. , 2015 ; Kao et al. , 2020 ) , this parameterization is unique—it includes multiple compute lanes per each PE and enables SIMD execution model within each compute lane—and yields a distinct accelerator search space accompanied by an end-to-end simulation framework . More details in Appendix C . 3 PROBLEM STATEMENT , TRAINING DATA AND EVALUATION PROTOCOL . Our template-based parameterization maps the accelerator , denoted as x , to a discrete design space , x = [ x1 , x2 , · · · , xK ] , and each xi is a discrete-valued variable representing one component of the microarchitectural template , as shown in Table 1 ( See Appendix C for the description of other accelerator search spaces studied in our work ) . A design maybe be infeasible due to various reasons , such as a compilation failure or the limitations of physical implementation , and we denote the set of all such feasibility criterion as Feasible ( x ) . The feasibility criterion depends on both the target software and the underlying hardware , and it is not easy to identify if a given x is infeasible without explicit simulation . We will require our optimization procedure to not only learn the value of the objective function but also to learn to navigate through a sea of infeasible solutions to high-performing feasible solutions x∗ satisfying Feasible ( x∗ ) = 1 . Our training dataset D consists of a modest set of accelerators xi that are randomly sampled from the design space and evaluated by the hardware simulator . We partition the dataset D into two subsets , Dfeasible and Dinfeasible . Let f ( x ) denote the desired objective ( e.g. , latency , power , etc . ) we intend to optimize over the space of accelerators x . We do not possess functional access to f ( x ) , and the optimizer can only access f ( x ) values for accelerators x in the feasible partition of the data , Dfeasible . For all infeasible accelerators , the simulator does not provide any value of f ( x ) . In addition to satisfying feasibility , the optimizer must handle explicit constraints on parameters such as area and power ( Flynn & Luk , 2011 ) . In our applications , we impose an explicit area constraint , Area ( x ) ≤ α0 , though additional explicit constraints are also possible . To account for different constraints , we formulate this task as a constrained optimization problem . Formally : min x f ( x ) s.t . Area ( x ) ≤ α0 , Feasible ( x ) = 1 on D = Dfeasible ∪ Dinfeasible = { ( x1 , y1 ) , · · · , ( xN , yN ) } ∪ { x′1 , · · · , x′N ′ } ( 1 ) While Equation 1 may appear similar to other standard black-box optimization problems , solving it over the space of accelerator designs is challenging due to the large number of infeasible points , the need to handle explicit design constraints , and the difficulty in navigating the non-smooth landscape ( See Figure 3 and Figure 12 in the Appendix ) of the objective function . What makes optimization over accelerators challenging ? Compared to other domains where model-based optimization methods have been applied ( Brookes et al. , 2019 ; Trabucco et al. , 2021a ) , optimizing accelerators introduces a number of practical challenges . First , accelerator design spaces typically feature a narrow manifold of feasible accelerators within a sea of infeasible points ( Nardi et al. , 2019 ; Shi et al. , 2020 ; Gelbart et al. , 2014 ) , as visualized in Figure 3 and Appendix ( Figure 13 ) . While some of these infeasible points can be identified via simple rules ( e.g . estimating chip area usage ) , most infeasible points correspond to failures during compilation or hardware simulation . These infeasible points are generally not straightforward to formulate into the optimization problem and requires simulation ( Shi et al. , 2020 ; Parashar et al. , 2019 ; Yazdanbakhsh et al. , 2021a ) . Second , the optimization objective can exhibit high sensitivity to small variations in some architecture parameters ( Figure 12b ) in some regions of the design space , but remain relatively insensitive in other parts , resulting in a complex optimization landscape . This suggests that optimization algorithms based on local parameter updates ( e.g. , gradient ascent , evolutionary schemes , etc . ) may have a challenging task traversing the nearly flat landscape of the objective , which can lead to poor performance . Training dataset . We used an offline dataset D of ( accelerator parameters , latency ) via random sampling from the space of 452M possible accelerator configurations . Our method is only provided with a relatively modest set of feasible points ( ≤ 8000 points ) for training , and these points are the worst-performing feasible points across the pool of randomly sampled data . This dataset is meant to reflect an easily obtainable and an application-agnostic dataset of accelerators that could have been generated once and stored to disk , or might come from real physical experiments . We emphasize that no assumptions or domain knowledge about the application use case was made during dataset collection . Table 2 depicts the list of target applications , evaluated in this work , includes three variations of MobileNet ( Gupta & Akin , 2020 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) , three in-house industry-level models for object detection ( M4 , M5 , M6 ; names redacted to prevent anonymity violation ) , a U-net model ( Ronneberger et al. , 2015 ) , and two RNN-based encoderdecoder language models ( Graves , 2012 ; He et al. , 2019 ; Sainath et al. , 2020 ; Li et al. , 2021 ) . These applications span the gamut from small models , such as M6 , with only 0.4 MB model parameters that demands less on-chip memory , to the medium-sized models ( ≥ 5 MB ) , such as MobileNetV3 and M4 models , and large models ( ≥ 19 MB ) , such as t-RNNs , hence requiring larger on-chip memory . Evaluation protocol . To compare state-of-the-art simulator-driven methods and our data-driven method , we limit the number of feasible points ( costly to evaluate ) that can be used by any algorithm to equal amounts . We still provide infeasible points to any method and leave it up to the optimization method to use it or not . This ensures our comparisons are fair in terms of the amount of data available to each method . However , it is worthwhile to note that in contrast to our method where worse-quality data points from small offline dataset are used , the simulator-driven methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance . Following prior work ( Brookes et al. , 2019 ; Trabucco et al. , 2021a ; b ) , we evaluate each run of a method by first sampling the top n = 256 design candidates according to the algorithm ’ s predictions , evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design . The final reported results is the median of ground truth objective values across five independent runs .

Summary: The authors first introduce two practices for application-specific hardware accelerators: 1) Designers need to spend considerable manual effort and perform large number of time-consuming simulations to find accelerators that can accelerate multiple target applications while obeying design constraints. Moreover, such a simulation-driven approach must be re-run from scratch every time the set of target applications or design constraints change. 2) An alternative paradigm is to use a data-driven, offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations. Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes.  As such, they develop such a data-driven offline optimization method for designing hardware accelerators, dubbed PRIME, which learns a conservative estimate of the desired cost function, utilizes infeasible points, and optimizes the design against this estimate without any additional simulator queries during optimization. The authors evaluate PRIME architects accelerators---tailored towards both single- and multi-applications as well as unseen applications in a zero-shot setting.