Data-Driven Offline Optimization for Architecting Hardware Accelerators

1 INTRODUCTION . The death of Moore ’ s Law ( Esmaeilzadeh et al. , 2011 ) and its spiraling effect on the semiconductor industry have driven the growth of specialized hardware accelerators . These specialized accelerators are tailored to specific applications ( Yazdanbakhsh et al. , 2021a ; Reagen et al. , 2017 ; Nardi et al. , 2019 ; Shi et al. , 2020 ) . To design specialized accelerators , designers first spend considerable amounts of time developing simulators that closely model the real accelerator performance , and then optimize the accelerator using the simulator . While such simulators can automate accelerator design , this requires a large number of simulator queries for each new design , both in terms of simulation time and compute requirements , and this cost increases with the size of the design space ( Yazdanbakhsh et al. , 2021b ; Shi et al. , 2020 ; Hegde et al. , 2021 ) . Moreover , most of the accelerators in the design space are typically infeasible ( Hegde et al. , 2021 ; Yazdanbakhsh et al. , 2021a ) because of build errors in silicon or compilation/mapping failures . When the target applications change or a new application is added , the complete simulation-driven procedure is generally repeated . To make such approaches efficient and practically viable , designers typically “ bake-in ” constraints or otherwise narrow the search space , but such constraints can leave out high-performing solutions ( Dave et al. , 2019 ; Parashar et al. , 2019 ; Chatarasi et al. , 2020 ) . An alternate approach , proposed in this work , is to devise a data-driven optimization method that only utilizes a database of previously tested accelerator designs , annotated with measured performance metrics , to produce new optimized designs without additional active queries to an explicit silicon or a cycle-accurate simulator . Such a data-driven approach provides three key benefits : ( 1 ) it significantly shortens the recurring cost of running large-scale simulation sweeps , ( 2 ) it alleviates the need to explicitly bake in domain knowledge or search space pruning , and ( 3 ) it enables data re-use by empowering the designer to optimize accelerators for new unseen applications , by the virtue of effective generalization . While data-driven approaches have shown promising results in biology ( Fu & Levine , 2021 ; Brookes et al. , 2019 ; Trabucco et al. , 2021a ) , using offline optimization methods to design accelerators has been challenging primarily due to the abundance of infeasible design points ( Yazdanbakhsh et al. , 2021a ; Hegde et al. , 2021 ) ( See Figure 3 and Appendix Figure 13 ) . The key contribution of this paper is a data-driven approach , PRIME , to automatically architect high-performing application-specific accelerators by using only previously collected offline data . PRIME learns a robust surrogate model of the task objective function from an existing offline dataset , and finds high-performing application-specific accelerators by optimizing the architectural parameters against this learned surrogate function , as shown in Figure 1 . While naïvely learned surrogate functions usually produces poor-performing , outof-distribution designs that appear quite optimistic under the learned surrogate ( Kumar & Levine , 2019 ; Brookes et al. , 2019 ; Trabucco et al. , 2021a ) . The robust surrogate in PRIME is explicitly trained to prevent overestimation on “ adversarial ” designs that would be found during optimization . Furthermore , in contrast to prior works that discard infeasible points ( Hegde et al. , 2021 ; Trabucco et al. , 2021a ) , our proposed method instead incorporates infeasible points when learning the conservative surrogate by treating them as additional negative samples . During evaluation , PRIME optimizes the learned conservative surrogate using a discrete optimizer . Our results show that PRIME architects hardware accelerators that improve over the best design in the training dataset , on average , by 2.46× ( up to 6.7× ) when specializing for a single application . In this case , PRIME also improves over the best conventional simulator-driven optimization methods by 1.54× ( up to 6.6× ) . These performance improvements are obtained while reducing the total simulation time to merely 7 % and 1 % of that of the simulator-driven methods for single-task and multi-task optimization , respectively . More importantly , a contextual version of PRIME can design accelerators that are jointly optimal for a set of nine applications without requiring any additional domain information . In this challenging setting , PRIME improves over simulator-driven methods , which tend to scale poorly as more applications are added , by 1.38× . Finally , we show that the surrogates trained with PRIME on a set of training applications can be readily used to obtain accelerators for unseen target applications , without any retraining on the new application . Even in this zero-shot optimization scenario , PRIME outperforms simulator-based methods that require re-training and active simulation queries by up to 1.67× . In summary , PRIME allows us to effectively address the shortcomings of simulation-driven approaches , ( 1 ) significantly reduces the simulation time , ( 2 ) enables data reuse and enjoys generalization properties , and ( 3 ) does not require domain-specific engineering or search space pruning . To facilitate further research in leveraging machine learning in architecting hardware accelerators , we also release the dataset in our experiments ( Appendix B.3 ) , consisting of many accelerator design points ( both infeasible and feasible ) . 2 BACKGROUND ON HARDWARE ACCELERATORS . The goal of specialized hardware accelerators—Google TPUs ( Jouppi et al. , 2017 ; Gupta & Akin , 2020 ) , Nvidia GPUs ( Nvidia , 2021b ) , GraphCore ( GraphCore , 2021 ) —is to improve the performance of specific applications , such as machine learning models . To design such accelerators , architects typically create a parameterized design and sweep over parameters using simulation . Target hardware accelerators . Our primary evaluation uses an industry-grade and highly parameterized template-based accelerator following prior work ( Yazdanbakhsh et al. , 2021b ) . This template enables architects to determine the organization of various components , such as compute units , memory cells , memory , etc. , by searching for these configurations in a discrete design space . Some ML applications may have large memory requirements ( e.g. , large language models ( Brown et al. , 2020 ) ) demanding sufficient on-chip memory resources , while others may benefit from more compute blocks . The hardware design workflow directly selects the values of these parameters . In addition to this accelerator and to further show the generality of our method to other accelerator design problems , we evaluate two distinct dataflow accelerators with different search spaces , namely NVDLA-style ( Nvidia , 2021a ) and ShiDianNao-style ( Du et al. , 2015 ) from Kao et al . ( 2020 ) ( See Section 6 and Appendix C for a detailed discussion ; See Table 6 for results ) . Table 1 : The accelerator design space parameters for the primary accelerator search space targeted in this work . The maximum possible number of accelerator designs ( including feasible and infeasible designs ) is 452,760,000 . PRIME only uses a small randomly sampled subset of the search space . Accelerator Parameter # discrete values Accelerator Parameter # discrete values # of PEs-X 10 # of PEs-Y 10 PE Memory 7 # of Cores 7 Core Memory 11 # of Compute Lanes 10 Instruction Memory 4 Parameter Memory 5 Activation Memory 7 DRAM Bandwidth 6 How does an accelerator work ? We briefly explain the computation flow on our template-based accelerators ( Figure 2 ) and refer the readers to Appendix C for details on other accelerators . This template-based accelerator is a 2D array of processing elements ( PEs ) . Each PE is capable of performing matrix multiplications in a single instruction multiple data ( SIMD ) paradigm ( Gove et al. , 1993 ) . A controller orchestrates the data transfer ( both activations and model parameters ) between off-chip DRAM memory and the on-chip buffers and also reads in and manages the instructions ( e.g . convolution , pooling , etc . ) for execution . The computation stages on such accelerators start by sending a set of activations to the compute lanes , executing them in SIMD manner , and either storing the partial computation results or offloading them back into off-chip memory . Compared to prior works ( Hegde et al. , 2021 ; Du et al. , 2015 ; Kao et al. , 2020 ) , this parameterization is unique—it includes multiple compute lanes per each PE and enables SIMD execution model within each compute lane—and yields a distinct accelerator search space accompanied by an end-to-end simulation framework . More details in Appendix C . 3 PROBLEM STATEMENT , TRAINING DATA AND EVALUATION PROTOCOL . Our template-based parameterization maps the accelerator , denoted as x , to a discrete design space , x = [ x1 , x2 , · · · , xK ] , and each xi is a discrete-valued variable representing one component of the microarchitectural template , as shown in Table 1 ( See Appendix C for the description of other accelerator search spaces studied in our work ) . A design maybe be infeasible due to various reasons , such as a compilation failure or the limitations of physical implementation , and we denote the set of all such feasibility criterion as Feasible ( x ) . The feasibility criterion depends on both the target software and the underlying hardware , and it is not easy to identify if a given x is infeasible without explicit simulation . We will require our optimization procedure to not only learn the value of the objective function but also to learn to navigate through a sea of infeasible solutions to high-performing feasible solutions x∗ satisfying Feasible ( x∗ ) = 1 . Our training dataset D consists of a modest set of accelerators xi that are randomly sampled from the design space and evaluated by the hardware simulator . We partition the dataset D into two subsets , Dfeasible and Dinfeasible . Let f ( x ) denote the desired objective ( e.g. , latency , power , etc . ) we intend to optimize over the space of accelerators x . We do not possess functional access to f ( x ) , and the optimizer can only access f ( x ) values for accelerators x in the feasible partition of the data , Dfeasible . For all infeasible accelerators , the simulator does not provide any value of f ( x ) . In addition to satisfying feasibility , the optimizer must handle explicit constraints on parameters such as area and power ( Flynn & Luk , 2011 ) . In our applications , we impose an explicit area constraint , Area ( x ) ≤ α0 , though additional explicit constraints are also possible . To account for different constraints , we formulate this task as a constrained optimization problem . Formally : min x f ( x ) s.t . Area ( x ) ≤ α0 , Feasible ( x ) = 1 on D = Dfeasible ∪ Dinfeasible = { ( x1 , y1 ) , · · · , ( xN , yN ) } ∪ { x′1 , · · · , x′N ′ } ( 1 ) While Equation 1 may appear similar to other standard black-box optimization problems , solving it over the space of accelerator designs is challenging due to the large number of infeasible points , the need to handle explicit design constraints , and the difficulty in navigating the non-smooth landscape ( See Figure 3 and Figure 12 in the Appendix ) of the objective function . What makes optimization over accelerators challenging ? Compared to other domains where model-based optimization methods have been applied ( Brookes et al. , 2019 ; Trabucco et al. , 2021a ) , optimizing accelerators introduces a number of practical challenges . First , accelerator design spaces typically feature a narrow manifold of feasible accelerators within a sea of infeasible points ( Nardi et al. , 2019 ; Shi et al. , 2020 ; Gelbart et al. , 2014 ) , as visualized in Figure 3 and Appendix ( Figure 13 ) . While some of these infeasible points can be identified via simple rules ( e.g . estimating chip area usage ) , most infeasible points correspond to failures during compilation or hardware simulation . These infeasible points are generally not straightforward to formulate into the optimization problem and requires simulation ( Shi et al. , 2020 ; Parashar et al. , 2019 ; Yazdanbakhsh et al. , 2021a ) . Second , the optimization objective can exhibit high sensitivity to small variations in some architecture parameters ( Figure 12b ) in some regions of the design space , but remain relatively insensitive in other parts , resulting in a complex optimization landscape . This suggests that optimization algorithms based on local parameter updates ( e.g. , gradient ascent , evolutionary schemes , etc . ) may have a challenging task traversing the nearly flat landscape of the objective , which can lead to poor performance . Training dataset . We used an offline dataset D of ( accelerator parameters , latency ) via random sampling from the space of 452M possible accelerator configurations . Our method is only provided with a relatively modest set of feasible points ( ≤ 8000 points ) for training , and these points are the worst-performing feasible points across the pool of randomly sampled data . This dataset is meant to reflect an easily obtainable and an application-agnostic dataset of accelerators that could have been generated once and stored to disk , or might come from real physical experiments . We emphasize that no assumptions or domain knowledge about the application use case was made during dataset collection . Table 2 depicts the list of target applications , evaluated in this work , includes three variations of MobileNet ( Gupta & Akin , 2020 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) , three in-house industry-level models for object detection ( M4 , M5 , M6 ; names redacted to prevent anonymity violation ) , a U-net model ( Ronneberger et al. , 2015 ) , and two RNN-based encoderdecoder language models ( Graves , 2012 ; He et al. , 2019 ; Sainath et al. , 2020 ; Li et al. , 2021 ) . These applications span the gamut from small models , such as M6 , with only 0.4 MB model parameters that demands less on-chip memory , to the medium-sized models ( ≥ 5 MB ) , such as MobileNetV3 and M4 models , and large models ( ≥ 19 MB ) , such as t-RNNs , hence requiring larger on-chip memory . Evaluation protocol . To compare state-of-the-art simulator-driven methods and our data-driven method , we limit the number of feasible points ( costly to evaluate ) that can be used by any algorithm to equal amounts . We still provide infeasible points to any method and leave it up to the optimization method to use it or not . This ensures our comparisons are fair in terms of the amount of data available to each method . However , it is worthwhile to note that in contrast to our method where worse-quality data points from small offline dataset are used , the simulator-driven methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance . Following prior work ( Brookes et al. , 2019 ; Trabucco et al. , 2021a ; b ) , we evaluate each run of a method by first sampling the top n = 256 design candidates according to the algorithm ’ s predictions , evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design . The final reported results is the median of ground truth objective values across five independent runs .

Summary: This paper mainly focuses on the architectural sizing of the neural network accelerators. Optimizing accelerator parameters is expensive due to the time-consuming simulation and numerous infeasible design candidates. This work presents a data-driven offline optimization framework, PRIME, that reuses the previous simulation results to produce new designs without additional active queries to an explicit silicon or simulator. The key idea is to train a conservative surrogate model, that predicts the cost (e.g. latency) of an architecture candidate, by minimizing the prediction error of feasible designs and penalizing the infeasible designs for avoiding optimism. Experiments show that PRIME offers 1.20x - 1.54x speedup on average over manual design and 1.26x speedup for unseen applications.