Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:

The paper proposes an algorithm to search for hardware designs and neural architectures jointly.
The results show that joint optimization improves accuracy and latency, reduces the search samples.

##########################################################################

Pros:

- The paper introduces a new dimension, hardware design, to the neural architecture search domain.
- The results show that joint optimization outperforms phased optimization.

##########################################################################

Cons:

- The used techniques are not novel
- The hypothesis "joint optimization is better than two-phased optimization" seems obvious.

##########################################################################

Other comments:

1. Can you describe more of the hardware. Is it a publicly accessable platform? How long does it take to switch between different configurations?
2. Can you add wall-clock time or used computing resource in table 3?
3. I also have concerns about the motivation. Is it typical to design an accelerator for just one network? I think an accelerator should be optimized for at least a range of neural networks.

typo: Section 3.1 "The objective for NAHAS it to" -> "The objective for NAHAS is to"
review_title: Good exploration with weak technical contributions.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors describe an automated system for co-designing neural architectures and HW accelerators. The system is able to find the best solution under latency and chip-area constraints.  A highly parameterized (commercial) edge accelerator defines the hardware search space.  Results are compared to MnasNet, platform-aware NAS and EfficientNet.

This is an interesting area and the authors demonstrate clear advantages of their approach. 

A claim is made that "although previous works have optimized either neural architectures given fixed hardware, or hardware given fixed neural architectures, none has considered optimizing them jointly.". Does the work outlined in the "co-design" paragraph of section 2 not attempt to do this?  Some other examples that could have been discussed include: 

* “FPGA/DNN co-design: An efficient design methodology for IoT intelligence on the edge”, Hao et. al. DAC’19
* “Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator”, Abdelfattah et al, DAC’20
* When Neural Architecture Search Meets Hardware Implementation: from Hardware Awareness to Co-Design
https://ieeexplore.ieee.org/abstract/document/8839421

It is a little unclear why the papers by Jiang and Yang and the papers above are dismissed? It would be good to understand better how the author's NAS approach improves on these previous works? 

Limited information is provided regarding the architecture of the accelerator. What fraction of the HAS search space is unavilable due to restrictions imposed by the compiler? Perhaps these design points are all uninteresting?

Is there anything to learn by a discussion of the good hardware configurations that were found? Are these surprising or unexpected? How do they differ from the baseline configuration?

The work is interesting but the claimed contributions perhaps need to be clarrified. 




review_title: NAHAS review 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes NAHAS for co-designing neural network architecture and hardware architecture. It shows performance improvements on ImageNet compared to previous hardware-aware NAS methods which only optimize the neural network architectures. 

Pros:
1. Co-designing neural network architecture and hardware architecture is a promising and important direction. 
2. The proposed method clearly outperforms hardware-aware NAS. 
	
Cons:
1. The technical contribution of this paper is limited. The main difference between this paper and previous hardware-aware NAS papers is that this paper has an additional hardware search space beside the neural network architecture search space. The other components, such as the training method, search algorithm, and learning objectives, are borrowed from previous papers with minor modifications. 

2. Co-designing neural network architecture and hardware architecture is not new. Similar ideas have been explored in [1]. The authors should discuss the difference between this paper and [1]. 

In summary, co-designing neural network architecture and hardware architecture is not new. The authors do not discuss the difference between this paper and [1]. Besides, I think the technical contribution of the proposed method is limited. Therefore, I recommend rejecting this submission. 

[1] Neural-Hardware Architecture Search, NeurIPS 2019 Workshop on Machine Learning for Systems.
review_title: Official Blind Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper presents NAHAS, which is a combination of Neural Architecture Search (NAS) and Hardware Architecture Search (HAS) 
for software-hardware co-design. It uses PPO with joint search space: model accuracy and hardware constraints.

##########################################################################

Reasons for score: 

The paper claims it demonstrate effectiveness of hardware aware NAS for first time, which is dubious.

The paper should reference the industry-standard parametrized accelerator used. And reference or briefly present the in-house simulator.

In the beginning of section 3, it is unclear about the entire workflow. Explain the high-level workflow in Figure 1 caption.

Tested only on Imagenet, mobilnet and efficientnet. It didn't demonstrate that the method can be generalized across different the vertical stack as claimed in the introduction.

In 1. introduction, typo: Intel’s Nervana

##########################################################################

Pros: 

- Demosntrated improved accuracy and latency upon other work

##########################################################################

Cons: 


- The paper needs to improve its clarity by explaining the figure workflow in section 3.

- It also needs to revise the claim that it demonstrate effectiveness of hardware aware NAS for first time. E.g: NSGA-NETmultiobjective

##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 


review_title: The paper needs to improve its clarity

Meta-review: This paper considers the problem of searching over the joint space of hardware and neural architectures to trade-off accuracy and latency. 

Reviewers raised some valid questions about the following aspects:
1. Low technical novelty
2. Prior work on hardware and neural architecture co-design, and closely related work are not addressed
3. Lacking details on hardware platform and discussion on physical constraints to determine invalid hardware designs (addressed somewhat, but the response is not satisfactory)

One additional comment: if we care about latency for a particular hardware platform, it is possible to automatically configure adaptive inference techniques to meet the latency constraints. 

Overall, my assessment is that the paper requires more work before it is ready for publication.
Decision: Reject