Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors first introduce two practices for application-specific hardware accelerators: 1) Designers need to spend considerable manual effort and perform large number of time-consuming simulations to find accelerators that can accelerate multiple target applications while obeying design constraints. Moreover, such a simulation-driven approach must be re-run from scratch every time the set of target applications or design constraints change. 2) An alternative paradigm is to use a data-driven, offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations. Such an approach not only alleviates the need to run time-consuming simulation, but also enables data reuse and applies even when set of target applications changes.

As such, they develop such a data-driven offline optimization method for designing hardware accelerators, dubbed PRIME, which learns a conservative estimate of the desired cost function, utilizes infeasible points, and optimizes the design against this estimate without any additional simulator queries during optimization. The authors evaluate PRIME architects accelerators---tailored towards both single- and multi-applications as well as unseen applications in a zero-shot setting.
main_review: The strengths of the paper are as follows: 
+ This paper provides a new perspective for tackling the overfitting of hardware efficiency predictors by leveraging adversarial training and infeasible design samples;
+ The developed tool can be useful as an early-stage development tool, and the released accelerator dataset can be useful to the community;
+ This paper is the first to consider unseen applications in a zero-shot setting when it comes to accelerator optimization. 

The weaknesses of the paper are below: 
- This paper does not review, discuss, and evaluate against another group of related methods for designing hardware accelerators, which considers automated accelerator designs, e.g., [MAGNet, ICCAD 2019] and AutoDNNchip, FPGA 2020];
- This work does not evaluate the resulting accelerators with any SOTA dedicated accelerators. I think it is not convincing for a accelerator tool to be practical without evaluating where the resulting design with the dedicated expert design for the same application/task.
summary_of_the_review: This paper develops a data-driven offline optimization for identifying good hardware accelerator designs, and conducts sufficient ablation studies to validate their methods, while the evaluation against SOTA methods can be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper mainly focuses on the architectural sizing of the neural network accelerators. Optimizing accelerator parameters is expensive due to the time-consuming simulation and numerous infeasible design candidates. This work presents a data-driven offline optimization framework, PRIME, that reuses the previous simulation results to produce new designs without additional active queries to an explicit silicon or simulator. The key idea is to train a conservative surrogate model, that predicts the cost (e.g. latency) of an architecture candidate, by minimizing the prediction error of feasible designs and penalizing the infeasible designs for avoiding optimism. Experiments show that PRIME offers 1.20x - 1.54x speedup on average over manual design and 1.26x speedup for unseen applications.
main_review: Strength:
- This work attacks a very commonly seen problem from prior work: the over-optimism and the uselessness of the infeasible design points in hardware performance modeling. Problems are well-defined and solutions are targeted and justified. The technical presentation of the methods proposed is solid. 
- Experiments are extensive and have compared offline and online methods very thoroughly. The paper also shows the results of applying PRIME on other accelerator designs to examine the applicability. 
- Writing is clear and the paper is easy to follow.

Weakness:
- The challenge of simulation time in online methods is now shifted to dataset construction. How is the cost of constructing the training datasets, especially when a new dataset is required for a new context (application)? 
- Figure 12 shows the challenge of overoptimism in performance modeling. However, there is no experiment showing the difference of predicted and measured latency/energy when using the proposed method. It is not clear whether the results of PRIME are produced by the performance model or by the simulation/measurement using the optimization results.
- The results in Table 5 are not enough to show the capability of PRIME under the zero-shot setting. One of the benefits of the offline method is to reuse the logged training data. It would be better if authors could show the results on all benchmarks (instead of 1 or 2 networks) when using different training application subsets. Meanwhile, given a new application, when will PRIME choose to build a new training dataset and when will PRIME choose to use a zero-shot setting? How should PRIME choose the training applications? What is the minimal training application subset to get the best results on all benchmarks?

summary_of_the_review: This work presents an effective framework PRIME to tackle the challenges of automating hardware design optimization. Though it seems to be an incremental medication of previously published efforts, the proposed method is clear and useful. The paper could be stronger with a better analysis as stated in the Main Review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach to optimizing parameters of hardware architectures to design architectures that are more efficiently able to execute neural networks. To accomplish this, the paper proposes Prime, an offline optimization algorithm that constructs a surrogate of the design space then optimizes against that design space. Specifically, Prime is trained to be robust to both unseen points in the design space (rather than underestimating their cost) and to infeasible points. For each of these goals, Prima adds a term to the loss function, penalizing adversarial examples and infeasible training points. The authors evaluate Prime by showing that given the same number of evaluations of the ground-truth data, Prime results in lower cost results than online optimization.
main_review: ## Novelty and Significance

The paper addresses an important emerging problem, designing custom hardware architectures to accelerate machine learning algorithms; however, the presented algorithm is general to any configuration tuning / design space exploration approach. None of the techniques within the paper are deeply novel (using adversarial training in the loss function [1], adding feasibility as a term in the loss function [2], employing context vectors to reuse design space exploration for multiple applications [3], and the actual optimizer used to optimize against Prime [4]), the combination of approaches and application to the specific problem under study is sufficiently novel to be worth publishing. Regardless, I would appreciate a more strong statement of precisely what components of the paper should be considered novel.

## Correctness and Clarity

At a high level, the approach and evaluation in the paper seems mostly correct, and the paper is very easy to read. However, I do have the following questions and concerns:
- Many of the details of the evaluation are omitted. Specifically:
  - (*) what is the size of the training set?
  - (*) What is the stopping criteria for the various techniques (specifically, how many iterations is fireflies ran for against the surrogate)?
  - What is the total amount of compute for Prime? Figures 5 and 6 show the simulation time (which seems to be increasing for Prime, which I also don't understand, since I had thought that Prime was entirely offline), but does not include the cost of training and evaluating the surrogate.
- I am not entirely convinced that all of the experimental results are correct. Specifically, the following results are surprising enough that I would like some clarification from the authors:
  - (*) Bayes Opt can result in performance worse than D(Best in training) on t-RNN Enc -- though it depends on exactly how the final point is selected, this should likely not be possible.
  - "These results indicates that offline optimization of accelerators using Prime can be more data-efficient compared to online methods with active simulation queries" (page 7). This is a very surprising result to me. Online approaches should lead to at least as good accuracy or data efficiency than offline approaches given the  (since any given offline approach can be viewed as an instance of an online approach, whereas the converse is not true). Specifically, I would expect the online version of the fireflies approach to perform better than the offline version (unless there is some characteristic of the surface induced by Prime that is a better fit for optimizing with fireflies). This finding then indicates that either there is significant room for improvement in online approaches for this task, or that the chosen baselines or evaluation metrics are not correct. The paper should include more discussion of this point, or more narrowly scope it to the specific online approaches evaluated in the paper.
- "Top 20% high scoring points in the training dataset are used to provide a validation set..." This is a fascinating point, and I think should be highlighted significantly more in the paper. It seems plausible that this is a key element of the methodology. The conventional wisdom is that DNNs do not extrapolate outside of their training distribution. This approach explicitly requires extrapolation (finding accurate dips between sampled data); because the validation points are not IID and there is a significant amount of hyperparameter search performed, it is possible that selecting for the model that has the best prediction on this set of points selects for a model that is able to generalize to dips in latency space a bit more. I have several questions for the authors based on this point:
  - (*) Is the surrogate model that you ultimately select the exact model resulting from hyperparameter search / validation, or is the model re-trained from scratch with the discovered hyperparameters?
  - Have the authors experimented with other splits between the training and validation set (e.g., training on the best points and validating on the worst)?
  - (*) Is the reported D(Best in training) including the points in this validation set?
  - Are the hyperparameters for the other approaches also tuned using this validation set?
  - (*) What are the actual discovered hyperparameters after this hyperparameter search?
- What fraction of the space is infeasible? How do you do your initial sampling to avoid these infeasible points? What happens if all your final points are infeasible? This is a generally hard problem in high dimensional space. This is a challenge for all optimization algorithms here, the paper doesn't at all discuss the tractability of this sampling or what to do if the final points are all infeasible.
- How sensitive is the optimization to the number of particles, and the parameters of the negative mining optimization?
- As a minor point, alpha is overloaded within the approach: it is used as both the area constraint (Equation 1) and the COM hyperparameter (Equation 2)

[1] Brandon Trabucco, Aviral Kumar, Xinyang Geng, Sergey Levine. Conservative Objective Models for Effective Offline Model-Based Optimization. ICML, 2021.

[2] https://en.wikipedia.org/wiki/Barrier_function

[3] https://en.wikipedia.org/wiki/Multi-armed_bandit#Contextual_bandit

[4] Xin-She Yang. Nature-Inspired Metaheuristic Algorithms. Luniver Press, 2010.


# Response to author response

Thanks to the authors for the very detailed response. All of the responses to the (*) questions are satisfactory, allowing for sufficient understanding of the experimental setting to understand and reproduce results. I'm also pleased to see the additional clarifications and experimental results (especially the clarifications around the hyperparameter search process and the additional validation set experiments) -- these results have convinced me that Prime is doing something beyond just searching for the best point in the validation set. Based on the response, I plan to raise my score to an accept.

summary_of_the_review: The paper proposes an interesting approach to offline design space exploration, and demonstrates that it performs better than online approaches. Though I am not totally satisfied with the evaluation, it is a relatively low lift to bring it up to par with existing literature in the space. A more thorough evaluation would make more a much more compelling paper, but in my view this is not necessary for acceptance.

Weak accept, conditioned on the authors providing more details about the evaluation (specifically, all of the questions marked with (*) above). I would be willing to raise my score further if the authors address some of my other questions (evaluating the sensitivity of the approach to the validation set, discussion the tractability of optimizing with infeasible points, evaluating the amount of compute required to train and evaluate the surrogate, etc).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present an offline performance model for optimizing neural net hardware accelerators. They introduce a mechanism for dealing with non-viable hardware design points and for specializing towards specific applications from a single surrogate. They present results versus online methods on a variety of axes.
main_review: The authors take a clearly defined problem (design a neural net hardware accelerator), motivate it well (simulations are expensive; surrogates struggle with these specific problems in this domain), introduce a solution targeted at overcoming those specific issues, and then demonstrate that it generally works better than other accepted approaches.

Strengths:

- The evaluation results are compelling. While the absolute design point gains are modest, they must be understood in the context of dramatically lower computational time, which is a huge selling point by itself. I appreciated the various evaluation axes the authors have explored: single accelerator, shared (multi-target) accelerator, zero-shot, and applicability to multiple accelerator design paradigms.
- The organization is logical, and the writing is for the most part clear. I appreciate that writing papers on hardware accelerators in the ML community is sometimes difficult due to wildly varying levels of background in the audience. I feel the authors have struck a solid balance here: detail enough for those with a background, but helpfully reducing the discussion to just the important pieces.

Weaknesses:

- The paper presents PRIME as its solution, however the actual novel contributions are not particularly well analyzed. The authors highlight infeasible design points and a rough optimization landscape as the key problem characteristics, but the evaluation results and analysis are almost strictly black-box tests. This is a bit unfortunate because it makes it harder to understand how the proposed modifications are contributing to actually solving the described problems in a quantitative way. Some of this is discussed in appendices (e.g., the ablation study in A.6), but it is brief (and relegated to an appendix).

- The paper is squished. It seems clear the authors had a lot of material to incorporate, but quality has been sacrificed somewhat. Margins and labels on figures and tables in the paper are hard to read and run into text.



Disclosure: I have previously anonymously reviewed a version of this work. Comments below reflect the progress and are mostly for the authors' benefit.

I applaud the authors for the changes they have incorporated into this paper based on past reviews. Prior concerns about generalizability on both application and hardware are largely gone, and the paper has mostly retained its clarity. Some of the intuitive explanations have been omitted (probably due to space concerns), but it still generally holds up. This is a better paper.

I did notice that the discussion of prediction behavior as it scales with dataset sample size is gone. This is a bit unfortunate since it was one of the pieces that helped shed light on how the method worked (and where it breaks down). I don't believe this fundamentally harms the takeaway of the paper (since the existing evaluations are pretty clear that the method can be made to work in a variety of situations), but it might be nice if that analysis made its way back into an appendix.
summary_of_the_review: The paper is clear, describes its problem and solution well, clearly showcases its results across a variety of tests, and generally succeeds at convincing the reader that the approach works as advertised. There is room for improving the 'why', but the paper is solid nonetheless.
review_title: null

Meta-review: The authors give an effective framework PRIME to tackle the challenges of automating hardware design optimization.  This problem is of importance to the community.  Overall, the reviewers thought the paper gave a nice clean approach to the problem and that the community would be interested with these results.
Decision: Accept (Poster)