Question: Whether the results of PRIME are produced by the performance model or by the simulation/measurement using the optimization results or not?
Evidence: Our template-based parameterization maps the accelerator, denoted as x, to a discrete design space,
and each x i is a discrete-valued variable representing one component of the microarchitectural template, as shown in Table (See Appendix C for the description of other accelerator search spaces studied in our work). A design maybe be infeasible due to various reasons, such as a compilation failure or the limitations of physical implementation, and we denote the set of all such feasibility criterion as Feasible(x). The feasibility criterion depends on both the target software and the underlying hardware, and it is not easy to identify if a given x is infeasible without explicit simulation. We will require our optimization procedure to not only learn the value of the objective function but also to learn to navigate through a sea of infeasible solutions to high-performing feasible solutions x * satisfying Feasible(x * ) = 1.
Our training dataset D consists of a modest set of accelerators x i that are randomly sampled from the design space and evaluated by the hardware simulator. We partition the dataset D into two subsets, D feasible and D infeasible . Let f (x) denote the desired objective (e.g., latency, power, etc.) we intend to optimize over the space of accelerators x. We do not possess functional access to f (x), and the optimizer can only access f (x) values for accelerators x in the feasible partition of the data, D feasible . For all infeasible accelerators, the simulator does not provide any value of f (x). In addition to satisfying feasibility, the optimizer must handle explicit constraints on parameters such as area and power. In our applications, we impose an explicit area constraint, Area(x) ≤ α 0 , though additional explicit constraints are also possible. To account for different constraints, we formulate this task as a constrained optimization problem. Formally:
While Equation 1 may appear similar to other standard black-box optimization problems, solving it over the space of accelerator designs is challenging due to the large number of infeasible points, the need to handle explicit design constraints, and the difficulty in navigating the non-smooth landscape (See Figure and Figure in the Appendix) of the objective function.
What makes optimization over accelerators challenging? Compared to other domains where model-based optimization methods have been applied, optimizing accelerators introduces a number of practical challenges. First, accelerator design spaces typically feature a narrow manifold of feasible accelerators within a sea of infeasible points, as visualized in Figure and Appendix (Figure). While some of these infeasible points can be identified via simple rules (e.g. estimating chip area usage), most infeasible points correspond to failures during compilation or hardware simulation. These infeasible points are generally not straightforward to formulate into the optimization problem and requires simulation.
Second, the optimization objective can exhibit high sensitivity to small variations in some architecture parameters (Figure) in some regions of the design space, but remain relatively insensitive in other parts, resulting in a complex optimization landscape. This suggests that optimization algorithms based on local parameter updates (e.g., gradient ascent, evolutionary schemes, etc.) may have a challenging task traversing the nearly flat landscape of the objective, which can lead to poor performance. Training dataset. We used an offline dataset D of (accelerator parameters, latency) via random sampling from the space of 452M possible accelerator configurations. Our method is only provided with a relatively modest set of feasible points (≤ 8000 points) for training, and these points are the worst-performing feasible points across the pool of randomly sampled data. This dataset is meant to reflect an easily obtainable and an application-agnostic dataset of accelerators that could have been generated once and stored to disk, or might come from real physical experiments. We emphasize that no assumptions or domain knowledge about the application use case was made during dataset collection. Table depicts the list of target applications, evaluated in this work, includes three variations of MobileNet, three in-house industry-level models for object detection (M4, M5, M6; names redacted to prevent anonymity violation), a U-net model, and two RNN-based encoderdecoder language models. These applications span the gamut from small models, such as M6, with only 0.4 MB model parameters that demands less on-chip memory, to the medium-sized models (≥ 5 MB), such as MobileNetV3 and M4 models, and large models (≥ 19 MB), such as t-RNNs, hence requiring larger on-chip memory.
Evaluation protocol. To compare state-of-the-art simulator-driven methods and our data-driven method, we limit the number of feasible points (costly to evaluate) that can be used by any algorithm to equal amounts. We still provide infeasible points to any method and leave it up to the optimization method to use it or not. This ensures our comparisons are fair in terms of the amount of data available to each method. However, it is worthwhile to note that in contrast to our method where worse-quality data points from small offline dataset are used, the simulator-driven methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance. Following prior work, we evaluate each run of a method by first sampling the top n = 256 design candidates according to the algorithm's predictions, evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design. The final reported results is the median of ground truth objective values across five independent runs.
Evaluation protocol. To compare state-of-the-art simulator-driven methods and our data-driven method, we limit the number of feasible points (costly to evaluate) that can be used by any algorithm to equal amounts. We still provide infeasible points to any method and leave it up to the optimization method to use it or not. This ensures our comparisons are fair in terms of the amount of data available to each method. However, it is worthwhile to note that in contrast to our method where worse-quality data points from small offline dataset are used, the simulator-driven methods have an inherent advantage because they can steer the query process towards the points that are more likely to be better in terms of performance. Following prior work (Brookes et al., 2019; Trabucco et al., 2021a;b), we evaluate each run of a method by first sampling the top n = 256 design candidates according to the algorithm’s predictions, evaluating all of these under the ground truth objective function and recording the performance of the best accelerator design. The final reported results is the median of ground truth objective values across five independent runs.
Answer: Per our interpretation of the question, we would like to clarify that the results we report in Tables 3, 4, 5 and 6 in the main paper correspond to the actual latency of the optimized designs found by PRIME in a purely offline fashion, which are then evaluated under the simulator to produce actual latencies. More details of our evaluation protocol can be found in the paragraph titled "Evaluation Protocol" in Section 3. We are happy to clarify any details if something is unclear.