Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors propose to co-optimize the software and hardware for DNN executions to maximize the energy-delay product. There are three main contributions: 1) propose a formal representation of software and hardware that facilitate the search process; 2) propose a Bayesian optimization framework; 3) propose to search the hardware and software space separately and the search process is optimized by leveraging the Gaussian process model. Below are some questions and concerns:

1:  the representation of software basically considers loop ordering, loop tiling, and computational parallelism, while the hardware representation is focused on available hardware resource and dataflow configuration. However, the biggest bottleneck of DNN processing actually comes from the main memory or large buffers. Compared with computing itself, data access normally consumes 2 orders of magnitude more energy and latency. Could the authors explain if they consider data access during their search process and how they famulated in the representation?

2: the search goal of this work is to minimize EDP. I wonder is it flexible for this framework to change this goal to fit different designs? For instance, is it possible to search a software/hardware co-designed model for extremely low-power applications? And how can the proposed framework adopt for different search goals?

3: the evaluation is only performed with previous work Eyeriss. Given the abundant accelerator with various design goals, more evaluations against more counterparts are needed. 

4: for the comparison results against Eyeriss, the authors show that their results outperform Eyeriss in EDP. But I am also curious about the detailed energy and performance comparison. 

review_title: This manuscript is interesting, and the work is promising. However, the formal representation of hardware and software lack important parameters. Moreover, the effectiveness of the proposed approached is evaluated against one previous work on a single metric.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method to optimize the co-design of code mapping and hardware configuration for neural accelerators. The method is based on a two-level Bayesian optimization (BO), with each BO optimizing the code mapping and hardware configuration, respectively. The design space of both code mapping and hardware configuration contains many complex constraints, some of which remains unknown until running expensive simulation. The obvious constraints are handled using rejection sampling and the unknown constraints are modeled by a Baysian classifier. This method is validated by experiments to optimize several network layers (ResNet, DQN, MLP, transformer) and achieves better performance than baselines.

Strength of the paper: this paper aims at solving a meaningful problem: SW/HW co-design of code mapping and neural accelerators. It adopts a simple optimization method (BO), and according to the experiments, it achieves good performance gain compared with manual design and other baselines methods. 

What is unclear to me: I am not familiar with the literature around code-mapping and hardware configuration co-design, so I don't have enough context to judge if the proposed method is novel, compared with previous methods.

Weaknesses of the paper: 
1/ this paper spends a lot of texts explaining the basics of BO, but what's missing to me is the representation of software design-points and hardware configurations. The usage of BO is based on the assumption that we have a good representation of the design space where the distance of each design point is clearly defined. For example, for two nested for-loop with different orders, how they are represented and what's the distance between the two representation? Is the representation canonical? What's the meaning of distance in the representation space? Without understanding these questions, it is difficult for me to see whether BO is the right optimization method for this. 
2/ In the experiment section, the method is only used to optimize one layer of a network, however, it is not clear how general this is as people usually care about the overall performance of a network or even a wide variety of networks. The experiments did not show if the proposed method is effective or general enough to optimize the overall performance. 

I hope authors can address the concerns. 
review_title: Interesting paper but needs more clarification

final_score: rating: 4: Ok but not good enough - rejection
review_content: Review: 
This paper proposes a Bayesian optimization framework for exploring the hardware configuration and compute mappings on generated hardware. Given a target NN model, it first generates hardware configuration candidates. Then for each hardware candidate, it generates corresponding mapping candidates of each NN layer. These candidates are generated with explicit constraints, i.e., it will randomly sample hardware/mapping until it obtains the desired number of feasible candidates. The evaluated EDP is fed into the Bayesian optimizer of mapping and hardware separately in sequential. The main contribution of this paper is a principled and systematic pipeline for co-searching the hardware parameters and mappings. Results are shown on the task of ResNet, DQN, MLP, and Transformer on Eyeriss architecture. It improves EDP by 16% to 40%.

 
Pros: 

+ The solution framework for co-searching hardware parameters and software mappings is neat. 

+ The results section is well structured.  It's nice to see the comparison between the proposed method and TVM; and the ablation study on whether to co-search hardware and software. 


Concerns: 

- The key concern about the paper is the lack of novelty. There are increasing works on hardware search for neural networks (see below), while this paper doesn't mention or compare to any of those.

- The related work lacks work on hardware search, such as
[1] Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. EDD: Efficient differentiable DNN architecture and implementation co-search for embedded AI solutions. arXiv preprint arXiv:2005.02563, 2020.
[2] Weiwen Jiang, Qiuwen Lou, Zheyu Yan, Lei Yang, Jingtong Hu, X Sharon Hu, and Yiyu Shi. Device-circuit-architecture co-exploration for computing-in-memory neural accelerators. IEEE Transactions on Computers, 2020a.
[3] Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous ASIC accelerator designs targeting multiple tasks. arXiv preprint arXiv:2002.04116, 2020.
[4] Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. Neural-Hardware Architecture Search. NeurIPS Workshop, 2019.
[5] Mohamed S Abdelfattah, Łukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane. Best of both worlds: Automl codesign of a CNN and its hardware accelerator. arXiv preprint arXiv:2002.05022, 2020.

-  The results are not conclusively in favor of the proposed method. Why do authors adopt Bayesian optimization instead of evolution or reinforcement learning? How is Bayesian optimization reasonable for searching loop reordering? There is no discussion of the choice of optimization algorithm from the results. 

- On the same note, the results would have been more complete if experiments can be done on multiple hardware architectures such as NVDLA, ShiDianNao, and so on.

- It would be better if the paper can elaborate on Section 2 in more detail. How does the proposed framework represent hardware and mapping? What is the range of each dimension? How large is the design space? What is the sample efficiency of the proposed method?

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper presents a method for hardware and software co-design using Bayesian optimization method. It is based on previous work of Timeloop

This paper seems a just an application of Bayesian optimization on the Timeloop framework. It searches using set parameters that are tunable by using some optimization method. Nevertheless, the paper shows improvement upon state-of-the-art neural networks such as transformers.

##########################################################################

Reasons for score: 

The paper shows improvement upon state-of-the-art neural networks such as transformers.

The paper being first system that co-optimizes both the hardware and software for DNN, which is dubious given the pletora of NAS and HAS papers.

##########################################################################

Pros: 

- Demontrated improved energy delay design on different workloads

##########################################################################

Cons: 

- The paper needs to revise its claim of being the  first system that co-optimizes both the hardware and software for DNN

- Evaluate using different machine learning optimization methods such as RL and Evolutionary algorithms

##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 


review_title: The paper needs to revise its claims

Meta-review: This paper considers the problem of hardware and software co-design for neural accelerators. Specifically, it looks at hardware and the software compiler that maps DNN to hardware. It employs Bayesian Optimization (BO) to perform joint search over hardware and software design parameters in an alternating manner. To handle black-box constraints that cannot be evaluated without performing simulations, the method uses constrained BO algorithms. 

The paper talks about two technical challenges:
1) Black-box constraints. There is a lot of literature on constrained BO.
2) Semi-discrete design variables. The paper didn't propose any generic solution. There are some recent papers to handle mixed variables that may be useful.
https://arxiv.org/abs/1907.01329
https://arxiv.org/abs/1906.08878

BO methodology is justified. There is recent work on hardware and software co-design for neural accelerators and should be taken into account for both qualitative and quantitative comparison. 

Overall, my assessment is that the paper in its current form lacks technical novelty for acceptance.
Decision: Reject