Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Not an expert but an interested outsider trying to exercise the best judgement.

The goal is ambitious and is something that lots of people want to solve, so it's certainly worth a stab at. The nice thing about the paper is that the evaluation is pretty solid, with FPGA implementations and simulations for the ASIC flow (not that those simulators are perfect, but they are at least widely-used in the community so using them is probably OK.)

* Second paragraph on page 2, can you precisely define what you mean by "path" there?
* What exactly is your hardware cost model, L_{cost}?
* Could you give theoretical analysis as to why you optimization method will converge? It doesn't seem to be using a standard optimization method, which is perfectly fine, but then theoretically analyzing the proposed method would be important. Otherwise, the proposed method looks purely empirical, and is thus perhaps better suited for the CAD community.
* What basic assumptions about the hardware are you making? Is it a GPU-like SIMD/SIMT architecture, or TPU-like systolic array, or eyeriss-like thing that relies on pretty beefy interconnect? After all, you have a hardware template with several knobs and are searching the parameters of those knobs, so showing the hardware template would make the paper more clearer.
review_title: Ambitious problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces Triple-Search for jointly optimizing neural network architecture, quantization policy, and hardware architecture. Specifically, it proposes a heterogeneous sampling strategy to tackle the dilemma between exploding memory cost and biased search. Besides, it also integrates a differentiable hardware search engine to support co-search in a differentiable manner. 

Pros:
1. Co-searching neural network architecture, quantization policy, and hardware architecture is practically important. 
2. Experiment results show clear improvements over the baselines. 
	 
Cons: 
1. I think this paper is not well motivated. I am not convinced that the challenge (dilemma between exploding memory cost and biased search) that this paper aims to address is a critical problem in practice. This challenge exists because the authors follow the differentiable NAS framework. However, for the chain-like design space studied in this paper, recent state-of-the-art NAS techniques [1,2] have shown that weight training can be decoupled from architecture search, and differentiable search is not a must in practice. In this paper, I do not find any discussions to support why we must need a differentiable AutoML framework for the co-searching task. 
2. The hardware part of the proposed framework is too short and not clearly explained. Please elaborate more on this part. 
3. The same co-searching problem has been studied in [3]. I think the authors should compare the proposed method with [3]. 

Overall, I think co-searching neural network architecture, quantization policy, and hardware architecture is an important task. The experiment results also look good. However, I find the motivation of having a differentiable framework is not well supported, the hardware part of the proposed framework is not clearly explained, and some closely related papers are missing. Therefore, I recommend "Marginally below acceptance threshold".

[1] Once for all: Train one network and specialize it for efficient deployment. ICLR 2020.

[2] Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420 (2019).

[3] Neural-Hardware Architecture Search. NeurIPS 2019 Workshop on Machine Learning for Systems.

review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: Review:
This paper proposes Triple-Search (TRIPS), a differentiable framework of jointly searching for network architecture, quantization precision, and accelerator parameters. To address the dilemma between exploding training memory and biased search, the proposed framework leverages heterogeneous sampling where soft Gumbel Softmax is used for weight update and hard Gumbel Softmax is used for probabilities \beta. To integrate accelerator search, hard Gumbel Softmax is used on hardware design choices and the overall hardware cost is used for penalization. Experiments are conducted on the FPGA platform for CIFAR and ImageNet dataset to show the superiority of TRIPS over NAS-only methods.

Pros:

+ The idea of co-searching hardware architecture, neural architecture, and quantization precision is promising. 

+ It's nice to see the ablation study on sequential optimization and joint optimization. 

 
Concerns: 

- The paper is not well organized. Though TRIPS is a co-search framework, the paper mostly elaborates on the quantization searching. The modeling of hardware and the design space of the accelerator search are not well introduced.

- The challenge of exploding training memory comes from the differentiable searching. However, the authors did not explain why the differentiable search is a must. The state-of-the-art NAS algorithms tend to first train a shared weight hyper net and then perform searching algorithms such as the evolutionary algorithm. In this case, the main challenge in this paper does not exist.

- The comparison in Figure 3 may not be fair. The available hardware resources of the FPGA and mobile phones are different. It would be better if TRPIS results can be compared on the same hardware platform.

- The following papers also search quantization precision during NAS. It would better if the paper can show the comparison between TRIPS and the following methods.

[1] Yujun Lin, Driss Hafdi, Kuan Wang, Zhijian Liu, and Song Han. Neural-Hardware Architecture Search. NeurIPS Workshop, 2019.

[2] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481–497. Springer, 2019.

- For CIFAR experiments in Figure 4, it would be better to show the improvement over fixed 8bit and fixed 4bit instead of 16bit.

- For ASIC experiments, what is the accelerator design space? The results in Table 2 may not be fair. NASAIC uses MAESTRO as the estimation tool while TRIPS uses Timeloop as the estimation tool. If both NASAIC and TRIPS only search hardware parameters, does NASAIC and TRIPS search on the same hardware architecture (such as Eyeriss/NVDLA)?

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################
Summary:
 
The authors present TRIPS, a Triple-Search framework to jointly search for the optimal network, precision and accelerator for a given task with max accuracy and efficiency in a differentiable manner. TRIPS focuses on efficiently exploring the large search space that previous reinforcement learning based solution cannot afford to due to poor scalability. 

The authors propose a heterogeneous sampling approach that enables simultaneous update of weights and precision without biasing the precisions options or exploding the memory consumption. Further, they also formulate a novel co-search pipeline that enables a differentiable search for optimal accelerator.

Lastly, the authors present ablation studies and comparison with SOTA solutions that either solve the triple search problem described above or search (network, precision, accelerator) over a subset of its search space.

##########################################################################
Reasons for score: 
 
The paper presents a thorough description of TRIPS a joint search algorithm that enables scalable search of optimal network, precision and accelerator for maximum accuracy and efficiency. The experiments demonstrate the improvement in accuracy and efficiency compared to a wide range of previous solutions. Further ablation studies show breakdowns of accuracy and efficiency improvements derived from key enablers of TRIPS. With a few clarifications for the questions mentioned in the Cons section this paper should be accepted.

 
##########################################################################
Pros:

1.	This work presents a novel attempt to explore the large search space of network, precision and accelerator design, including detailed parameters such as tiling strategies, buffer sizing, processing elements (PE) count and connections etc.
2.	Section 2 presents a detailed overview of the past papers that have attempted to optimize all subsets of the search space explored by the proposed solutions.
3.	Authors do a great job of either estimating or collecting data for comparing the proposed solution with the many different approaches discussed in Section 2.
4.	Figure 3 demonstrates good data establishing the superiority of TRIPS optimal solution compared with other baselines. It would be helpful if the authors could shed light on how they tune TRIPS to generate a range of different solutions that tradeoff max accuracy with max FPS.
5.	Figure 4 presents a useful comparison to previous solutions that do not optimize over precision space. TRIPS shows better accuracy and FPS even with fixed precision.
6.	Ablation studies in section 4.3, are useful in breaking down the incremental improvement in the accuracy vs FPS tradeoff curve of TRIPS. 
7.	Appendix sections provide useful details about TRIPS training and accelerator design space explored. Additionally, section C shows another useful contrast of TRIPS with previous solutions based on SOTA expert-designed and tool-generated solutions. Remaining sections provide useful insights on the searched space.
 
##########################################################################
Cons: 
 
1.	Authors should consider citing the following in Section 2, DNN accelerators para, since it explores mapping network to accelerator theme of this section:
Gao, Mingyu, et al. "Tetris: Scalable and efficient neural network acceleration with 3d memory." Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems. 2017.
2.	Towards the end of Section 2, authors mention that TRIPS selects network, precision and accelerator that enable better transferability to other tasks/applications. However, there are no results or discussion on transferability of TRIPS models to other applications, consider removing this line.
3.	While Section 3 is comprehensive in its detailed description of TRIPS implementation and key enablers. It is somewhat difficult to read, consider reordering the implementation discussion and Section 3.2/3.3.   
4.	Section 4.2, it’s not clear why the authors selected the hardware limitation of 512DSP units. 
5.	Further, Section 4.2, Co-exploration of networks and accelerators part, compares previous solutions that optimize network and accelerator while keeping the precision fixed (searched optimal value). It would be helpful if the authors added the precisions used for the previous work datapoints as well. Are the selected datapoints for previous papers, the max accuracy points? How do authors select the 16bit value they choose for Figure 4 fixed precision TRIPS?
6.	Additionally, the authors do not present TRIPS-16bit datapoint for Figure 4c), please add that.
7.	The analysis presented in Table 2 seems highly skewed, with almost 1000x improvement in Area and EDP. Knowing that the support for heterogenous functionalities is costing baseline ASIC implementations dearly makes the comparison unfair. I recommend that authors attempt to minimize the impact of such features, since they make the baseline ASIC accelerators more general which the proposed solution does not support.
 
##########################################################################
Questions during rebuttal period: 

Kindly address a few questions in the Cons section. 

#########################################################################
Some typos: 
1.	Section 1: “to optimize the acceleration efficiency for* a* given* DNN”
2.	Section 2: Hardware-aware NAS subsection, “acceleration efficiency, thus can lead* to sub-optimal solutions.”
3.	Section 2: Co-exploration/search techniques subsection: “Built upon prior art*, our”
4.	Section 4.3: Comparison with sequential optimization subsection: “and hardware side, a natural* design”
5.	Appendix D: Insights for the searched network subsection:  “find that while wide-shallow* networks”, reword the whole sentence.

review_title: Review for Submission 2589

Meta-review: This paper introduces a methodology for jointly optimizing neural network architecture, quantization policy, and hardware architecture. There are two key ideas:
- Heterogeneous sampling strategy to tackle the dilemma between exploding memory cost and biased search. 
- Integrates a differentiable hardware search engine to support co-search in a differentiable manner.

The paper tackles an important research problem and experimental results are good.

There are two related issues with this paper:
1. Comparison to one-shot NAS: one-shot NAS methods only need to train the super-net once and then can be applied to multiple use-cases, while the proposed methodology needs to be executed for each use-case.
2. It is not clear whether differentiable search is needed for this joint optimization problem modulo existing tools.

Overall, my assessment is that the paper is somewhat borderline and with some more work will be ready for publication. 
Decision: Reject