Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper designs a hybrid search space that includes multiplication-based operators and multiplication-free operations to find good trading points between accuracy-efficiency. Further, this work defines the problem when training weight-sharing supernet on the hybrid search space and proposes the heterogenous weight sharing algorithm to address the problem. The proposed method is validated on both NLP and CV tasks, outperforming several competitive baseline methods in terms of accuracy and saving efficiency on latency or energy. 
main_review: **Strengths**
- The paper is well-motivated, well-written, good quality.
- To my knowledge, it is the first paper to design a hardware-inspired hybrid search space between multiplication-based operators and multiplication-free operators for NAS.
- Clear contribution: this work tackles the problem caused when applying weight-sharing-based or supernet-based search algorithms on the hybrid search space by proposing heterogenous weight sharing.
- Solid empirical experiments including SOTA baseline methods.

**Weaknesses**
- Search algorithm is not new. 
- If I read correctly, ShiftAddNet which is one of the most relevant baselines of the proposed work is missing in the experiment section. The reviewer thinks that adding ShiftAddNet in the main Table 4 (or additional analysis section) is needed to show the performance improvement with increasing latency/energy compared with ShiftAddNet. 
- The reviewer thinks the cost to train supernet and search cost should be reported. 
- Generality issue on multiple hardware devices and energy & latency information are missing in Table 4. Even if this work handles 'hardware-inspired NAS', the target device in the main experiments is only eyeriss. The reviewer thinks that this work should validate the generality of this method in more cases by applying this method on multiple hardwares (at least GPU/CPU/Mobile) and reporting latency/energy saving.
- [Minor] in Table 4, the number of parameters and MACs of the model obtained by this method is not SOTA, yet it is bolded. It would be better to be modified.
summary_of_the_review: The reviewer thinks that the proposed method is clear enough and practical to obtain neural architecture used for resource-constrained edge devices, yet, some experiments or information that are mentioned above should be added to validate the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a NAS method for both multiplication-based and adder-based networks. The contributions mainly lie on hybrid search space and new weight sharing strategy. The experimental results shows the method can obtain energy-efficient networks with high performance.
main_review: Strengths:
+ The specially designed search space that incorporates both multiplication-based and multiplication-free operators is new.
+ The analysis on distributions of convolution and add operators is reasonable.
+ The proposed weight sharing strategy can enable effective weight sharing among different operators that follow heterogeneous distributions.

Weaknesses:
- The weight sharing strategy only considers convolution and add operations. What's the distribution of Shift operation? Should shift op be considered together?
- In the experiment, lacking comparison with ShiftAddNet[1].

[1] ShiftAddNet: A Hardware-Inspired Deep Network. NeurIPS 2020.
summary_of_the_review: The main contribution of the paper is a NAS framework for multiplication-based and multiplication-free operators. The weaknesses do not affect the main contribution. I recommend accptance for it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propose a hybrid search space including both multiplication-based and multiplication-free operators. The authors also give a novel weight sharing strategy to enable effective weight sharing among different operators. 
main_review: Pros: 
1. This paper is innovative. The search space contains both multiplication-free and multiplication-based operators.
2. The results are the state-of-the-arts on both NLP and CV tasks.

Cons:
1. What is the meaning of Naïve WS in Table 5? It means weight sharing between operations without transformation or no weight sharing between operations? Generally, we only share weights of the same operator between different architectures, why sharing weights between operators here? In most cases, the more weights are shared, the lower the Kendall-tau is.
2. A learnable transformation kernel is used to translate the convolution to shift or adder operation. What if we just apply this kernel to the whole convolution neural network? Can we get an energy efficient adder or shift neural network without much accuracy loss with this kernel? There should be more analyses and experiments to show the effectiveness of this kernel.
3. In Table 4, the number of Add in ShiftAddNAS is always the same as MACs. In my opinion, this means no adder operator is used, since one MAC actually means two Add for adder operation (just like the AdderNet in Table 4). Why there is no adder operator at all? There should be figures to show your final searched models.

summary_of_the_review: The idea of this paper is novel and the results are satisfying, but there are more details should be provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces ShiftAddNAS that performs neural architecture search on a hybrid design space that contains conv operations, attention operations, and hardware-efficient shift/add operations. A heterogeneous weight sharing strategy is proposed to share weights between different types of blocks. 
main_review: Strengths:
1. I think it is quite interesting to integrate hardware-efficient shift/add operations into the search space of NAS, which can potentially improve the efficiency of neural networks on hardware.  
2. The motivation of heterogeneous weight sharing is clear and reasonable. 

Weaknesses:
1. Insufficient technical contributions. The idea of heterogeneous weight sharing is not new. As far as I know, when training super-nets with flexible quantization bits (e.g., [1]), a similar strategy has already been used: maintain a shared weight pool, fp32 directly uses the weight, int8/4/2 use the quantized weights. Compared to the previous strategy, this work has additional regularization terms (Gaussian for Conv ops and Laplacian for Add ops). To convince me that this modification provides sufficient technical contributions, I would expect to see solid experimental studies showing clear and significant improvements from these regularization terms. In addition, as the search space only has three different types of operations (according to Figure 4), maintaining the weight separately for different types of blocks and having three activate paths seems to be doable. I would expect to see comparisons with this approach as well.
2. Weak results. With a more hardware-efficient design space and the proposed heterogeneous weight sharing, I expect to see clear and significant improvements compared to previous SOTA methods. However, the improvements seem to be marginal. For example, in Table 3, ShiftAddNAS achieves the same BLEU score on WMT'14 En-De as HAT while providing only 1.05x latency reduction. In Table 4, the improvements of ShiftAddNAS also look marginal compared to strong baselines (e.g., EfficientNet, FBNetV3, etc). In addition, it would be helpful to provide the results of ShiftAddNAS with smaller MACs in Table 4 (e.g., <600M, <300M, etc). 

[1] Bai, Haoping, et al. "BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer." arXiv preprint arXiv:2105.08952 (2021).
summary_of_the_review: Overall, I think this paper explores an interesting topic and potentially can benefit real-world efficient deep learning applications. However, I find the technical contributions and empirical results presented in this version are not strong enough. Thus, I recommend rejection.
review_title: null

Meta-review: This paper develops a hybrid search space consisting of both multiplication-based and multiplication-free operators. It also presents a weight-sharing mechanism for searching in the introduced search space.

Pros:
* A hybrid search space is developed.
* Strong empirical results are reported for both CV and NLP tasks.
* The paper is well written and is easy to follow.

Cons:  
* Incremental technical novelty.
* Missing baselines and competing methods.
* Missing information on the search cost.
* Lack of insights into the discovered architectures

The rebuttal has provided most missing information and comparisons, and it has provided additional insights into the searched architectures. However, the reviewers still rate this paper at borderline primarily due to the limited technical novelties. Unfortunately, given these concerns, this submission does not meet the bar for acceptance at ICLR.
Decision: Reject