ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks
Authors: Haoran You
Baopu Li
Huihong Shi
Yingyan Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Architecture Search
Bit-wise Shift and Add
Hardware Acceleration
Multiplication-Reduced Networks

Abstract: Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are powerful yet power hungry, impeding their more extensive deployment into resource-constrained edge devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks in general under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the effectiveness of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score as compared to state-of-the-art expert-designed and neural architecture searched NNs, while leading to up to 93% or 69% energy and latency savings, respectively. All the codes will be released upon acceptance.

Under review as a conference paper at ICLR 2022
SHIFTADDNAS: HARDWARE-INSPIREH SEARCH FOR MORE ACCURATE AND EFFICIENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are powerful yet power hungry, impeding their more extensive de- ployment into resource-comee constrained edge devices. As such, multiplication-fret networks, which follow common practice in energy-efficient hardware imple- mentation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free net- works in general under-perform their vanilla counterparts in terms of the achieved accuracy, To this end, this work advocates hybrid NNs that consist of both power- ful yet costly multiplications and efficient yet less powerful operators for marry- ing the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that in- corporates both multiplication-basel and multiplication-fiee free operators for facili- tating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different op- erators that follow heterogeneous distributions (c.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to largely reduced super- net size and much better searched networks. Extensive experiments and ablation studies various models, datasets, and tasks consistently validate the effective- ness of ShiftAddNAS, e.g., achieving up to a +7.7% % higher accuracy or a +4.9 better BLEU score as compared chitecture searched NNs, while leading to up to 93% or 69% energy and latency savings, respectively. All the codes will be released upon acceptance.
ofate-of-the-ar expert-designed and neural ar-
INTRODUCTION
The unprecedented performance achieved by neural networks (NNs), e.g., convolutional neural net- works (CNNs) and Transformers, requires intensive multiplications and thus prohibitive training and inference costs, contradicting the explosive demand for embedding various intelligent function- alities into pervasive snourcceconssrrinne edge devices. In response, multiplication-fiee -free networks have been proposed to alleviate the prohibitive resource requirements by replacing the costly mul- tiplications with lower-cost operators for boosting hardware efficiency. For example, AdderNet (Chen et al., 2020) utilizes mere additions to design NNs; and ShiftAddNet (You et al., 2020a) follows commonly used hardware practice to re-parameterize NNs with both bitwise shifts and additions. Despite their promising performance in hardware efficiency, multiplication-fre NNs in general under-perform their CNN and Transformer counterparts in terms of task accuracy for both computer vision (CV) and natural language processing (NLP) applications.
To marry the best of both worlds, we advocate hybrid multiplication-redaee reduced network architectures that integrate both multiplication-baut based operators (c.g., vanilla convolution (Krizhevsky et al., 2012) and attention (Vaswani et al., 2017)) and multiplication-fee operators (e.g., shift and add (You et al., 2020a)) to simultaneously boost task accuracy and efficiency. Thanks to the amazing success of neural architecture search (NAS) in automating the process of designing state-of-the-art NNs, it is natural to consider NAS as the design engine of the aforementioned hybrid NNs for various applications and tasks, each often requiring different performance eef.eeieccfccyyyyccceee trade-off. However, there still exist few challenges in leveraging NAS to design the hybrid NNs. First, existing NAS methods mostly consider the search for either efficient CNNs (Wan et al., 2020). Transformers (Chen al., 2021b), or hybrid ONN-Transformert (Ding al., 2021; Li et al., 2021), and there still is

---

Under review as a conference paper at ICLR 2022
lack of a seminal work that searches for hardware- -inppirre- networks that incorporate both bitwise shifts and additions. Second, a hybrid search space could make it more challenging to achieve effective NAS and further aggravate the search burden, due to the enlarged search space imposed by the newly introduced multiplication- free operators. It is worth noting that existing weight sharing strategies of NAS do not directly apply to the target hybrid search space, because weights of different operators follow heterogeneons distributions, leading to a dilemma of either inefficient search or inconsistent architecture ranking. Specifically, weights in convolutional and adder layers follow Gaussian and Laplacian distributions, respectively, as also highlighted by (Chen et al., 2020; Xu al., 2020). As such, forcing weight sharing among heterogeneous operators could hurt the capacity and thus the achieved accuracy of the resulting NNs, while treating them separately could explode the search space and make it more difficult to achieve effective NAS, i.e., the dilemma mentioned above.
hybrid networks, especially for the
To tackle the aforementioned challenges towards more accurate and efficient NNs, this work makes the following contributions:
1. We propose a generic NAS framework dubbed ShiftAddNAS, which for the first time can automatically search for efficient hybrid NNs with both superior accuracy and efficiency. Our ShiftAddNAS integrates hybrid hardware-inspired search space that incorporates both multiplication-base operators (e.g., convolution and attention) and multiplication- free operators (e.g.. shift and add), and can serve play-and module to be applied on top of SOTA NAS works for further boosting their achievable accuracy and efficiency.
2.
We develop a new weight sharing strategy for effective search with hybrid search spaces, which only incurs a negligible overhead when searching for hybrid operators with het- erogeneous (e.g., Gaussian vs. Laplacian) weight distributions as compared to the vanilla NAS with merely multiplication-basee operators, alleviating the dilemma mentioned above regarding either inefficient search or inconsistent architecture ranking.
3. We conduct extensive experiments and ablation studies to validate the effectiveness of Shif- tAddNAS against ofate-of-the-art (SOTA) works. Results on multiple benchmarks demon- strate the superior accuracy and hardware efficiency of its searched NNs as compared to both (1) manually designed multiplication-Ff free networks, CNNs, Transformers, and hybrid CNN-Transformmrs, and (2) SOTA NAS works, on both CV and NLP tasks.
2 RELATED WORKS
Multiplication-free NNs. Many efficient NNs aim to reduce their intensive multiplications that dominate the time/energy costs. One important trend is to replace the multiplications with lower- cost operators: BNNs (Courbariaux et al., 2016; Juefei-Xu et al., 2017) binarize both the weights and activations and reduce multiplications to merely sign flips at non-negligible accuracy drops; AdderNets (Chen et al., 2020; Xu et al., 2020; Wang al., 2021b) fully replace the multiplications with lower-cost additions and further develop an effective backpropagation scheme for efficient AdderNet training; Shift-based NNs leverage either spatial shift (Wu et al., 2018) or bit-wise shift operations, e.g., DeepShift (Elhoushi et al., 2021). to reduce the amount of multiplications; and ShiftAddNet (You et al., 2020a) draws inspirations from efficient hardware designs to reparamatize NNs with mere bitwise shifts and additions. While mfltiplication-free NNs under-perform their vanilla NN counterparts in terms of achieved accuracy, ShiftAddNAS aims to automatically search for multiplication-rcdueed NNs that incorporate both multiplication-base based and multiplication-Frr operators for marrying the best of both worlds, i.e., boosted accuracy and efficiency.
Neural architecture search. NAS has achieved an amazing success in automating the design of efficient NN architectures. For searching for CNNs, early works (Tan & Le, 2019; Tan et al., 2019; Howard et al., 2019) adopt reinforcement learning based methods that require prohibitive search time and computing resources, while recent works (Liu et al., 2018; Wu et al., 2019a; Wan et al., 2020; Yang et al., 2021) utilize differentiable search to greatly improve the search efficiency. More recently, SOTA works adopt one-shot NAS (Guo et al., 2020; Cai et al., 2019; Yu et al., 2020; Wang et al., 2021a) to decouple the architecture search from supernet training and then evaluates the performance of sub-networks whose weights are directly inherited from the pretrained supernet. For searching for Transformers, recently emerging works (Wang et al., 2020a; Su et al., 2021: Chen et al., 20212;222b adopt one- NAS and evolutionary algorithm to search for optimal Transformer architectures for both NLP and CV tasks. Additionally, BossNAS (Li et al., 2021) and HR-NAS (Ding et al., 2021) further search for hybrid CNN-Transformer architectures.
2

---

Under review as a conference paper at ICLR 2022
Nevertheless, little effort has been made exploring NAS methods especially their search strategies for multipliiation-redueee NNs that incorporate both multiplication-ban and multiplication -feee operations. Furthermore, it is not clear whether existing efficient NAS methods applicable to searching for such multipiicaionn-reduedd NNs. Specifically, prior weight sharing strategies may not work since weights and activations in CNNS and AdderNets follow a different distribution (Chen et al., 2020). As such, it is highly desirable to develop NAS methods, e.g., ShiftAddNAS dedicated for hardware-inspires multiplictionn-rcduee NNs to increase achievable accuracy and efficiency.
Transformers. Transformers (Vaswani et al.. 2017) were first proposed for NLP tasks, which has inspired many interesting works. Some advance Transformer design by improving the attention mechanism (Chen et al., 2018). training deeper Transformers (Wang et al., 2019), and replacing the attention with convolutional modules (Wu et al., 2019b); and others strive to reduce Transformers' computational complexity by adopting sparse attention mechanisms (Zaheer et al., 2020), low-rank approximation (Wang et al., 2020b). or compression techniques (Wu et al., 2020). Recently, there has been growing interest in developing Transformers for CV tasks: Vision Transformer (VIT) (Dosovitskiy et al., 2021) for the first time successfully applies pure Transformers to image classi- fication and achieves SOTA task accuracy, which yet relies on pretraining on giant datasets (Hinton et al., 2015); following works including DeiT (Touvron et al., 2021) T2T- (Yuan et al., 2021) de- velop new training recipes and tokenization schemes, for achieving comparable accuracy without the necessity of costly pretraining; and another trend is to incorporate CNN modules into Transformer architectures for better accuracy and efficiency tradeoff's (Wu al., 2021; Xiao al., 2021: Graham et al., 2021). In contrast, we advocate hybrid multiplication reduucdd NNs and develop an automated search framework that can automatically search for such hardware inspired hybrid models.
3 THE PROPOSED SHIFTADDNAS FRAMEWORK
In this section, we first introduce the hybrid search space from both algorithmic and hardware costs perspectives, providing high-level background and justification for motivating ShiftAddNAS: Sec. 3.2 elaborates the one-shot search method of ShiftAddNAS by first analyzing the dilemma of either inefficient search inconsistent architecture ranking and then introducing the proposed novel heterogeneous weight sharing strategy tackling the aforementioned dilemma.
3.1 SHIFTADDNAS: HYBRID SEARCH SPACE
Candidate blocks. The first step of developing ShiftAddNAS is to construct a hybrid search space incorporating suitable building blocks that exhibit various performance-efficiinee trade-offs. Specif- ically, we hypothesize that integrating both multppication-basee and multiplication-fiee blocks into the search space could lead to both boosted accuracy and efficiency, and consider blocks from two trends of designing NNs: (1) capable NNs. e.g., vanilla CNNs and Transformers, leverage either convolutions (Conv) or multi-head self-attentions (Attn) that comprise of intensive multiplica- tions to capture local or global correlations, achieving SOTA accuracy in both CV and NI.P tasks; NNs, c.g. ShiftAddNet, draw inspirations from hardware design
and (2) efficient multiplication- practices to incorporate two efficient and complementary blocks, i.c., coarse-gras goarse-rrained Shift and fine-grained Add, for favoring hardware efficiency, while maintaining a decent accuracy. While our constructed general hybrid search space for both NLP and CV tasks shown in Fig. 2, we next analyze the building blocks from both algorithmic and hardware costs perspectives.
Attn is a core component of Transformers (Vaswani et al.. 2017), which consists of a num- ber of heads H with each capturing different global-context information by measuring pairwise correlations among tokens as defined below:
0 Attn = Con ,H,)) WO, where H, =S=foftmax QW (KWA))) VWY, (1) ddl
where h denotes the number of heads, Q, K, V E Ruxd are the query, key, and value embeddings of hidden dimension obtained by linearly projecting the input sequence of length n. For each head, W.O W.K W.V. E gdxds are learned projection weight matrices where de = d/h h is the embedding dimension of each head. In this way, the Attn block first computes dot-products between key-query pairs, scales to stabilize the training. uses tmax to normalize the resulting attention scores, and then computes weighted sum of the value embeddings corresponding to different inputs. Finally, the results from all heads are concatenated and further projected with a weight matrix o Rdxd to generate the outputs.
3

---

Under review as a conference paper at ICLR 2022
Conv is a key operator of CNNs, which models the local-context information of high- dimensional inputs such as images through sliding kernel weights W on top of imputs X to mea- sure their similarity (Gu et al., 2018), as defined in Eq. (2). Its translation invariant and weight sharing ability leads to various SOTA CNNs (He et al., 2016) or hybrid
mod- els (Xiao al., 2021). However, the computational complexities of CNNs can be prohibitive due to their required intensive multiplications. For example, one forward pass of ResNet-50 (He et al., 2016) requires 4G floating point multiplications.
Cony =
Σ x * W: Oshin
Σ x" (S.2)); Oada =
Σ | - WIl,, (2)
Shift is a well-known efficient hardware primitive, motivating the recent development of shift- based efficient NNs. For example, DeepShift (Elhoushi et al., 2021) parametrizes NNs with bit- wise shifts and sign flips, as formulated the middle of Eq. (2), with W = S 2" denoting weights in the shift blocks, where S E (-1,00 0, } are sign flip operators and the power-of-rwo parameter for P represents the bitwise shifts. However, NNs built with shift blocks and quan- tized weights are observed to be inferior to multiplication based NNs in terms of expressiveness (accuracy) as validated in (You et al., 2020a).
Add is another efficient hardware primitive which motivates recent works (Chen et al., 2020; Wang et al., bb; Song et al., 2021) to design efficient NNs using merely additions to measure the similarity between kernel weights W and inputs X. shown in the right part of Eq. (2). Such add-based NNs (Chen et al., 2020: Xu et al., 2020) in general have a better expressive capacity than their shift-based counterparts. For example, AdderNets (Chen et al., 2020) achieve a 1.37% higher accuracy than DeepShift under similar or even lower FLOPss on ResNet- 18 with the ImageNet dataset. However, add-based operators (i.e., repeated additions) are not parameter- efficient compared to bitwise shift operations (You et al., 2020a). While NNs combining shfit and add achieve a boosted accuracy, efficiency, and robustness than NNs using merely either of them, their accuracy still compares unfavorably as compared with vanilla CNNs or Transformers.
Based on the above introduction, the search space in ShiftAddNAS incorporates all the four different types of blocks (i.c., Attn, Conv, Shift, and Add), aiming to push forward both NNs' accuracy and efficiency. Note that we refer to all operators as blocks, and adopt block based search space because it has been evidenced and proven that block based ones can reduce the search space size and lead to more accurate architecture ranking/rating (Li et al., 2020b;a).
Hardware cost analysis. As mentioned, multiplication- based operators (e.g., Attn and Conv) favor a superior ac- curacy yet is not hardware efficient, while multiplication free operators (e.g., Shi ft and Add) favors a better hardware ef- ficiency yet hurt the achievable accuracy. Specifically, as shown in Fig. 1, bitwise shifts can save as high as 196x and 24x energy costs over multiplications, when implemented in a 45mm CMOS technology and SOTA FPGA (Xilinx Inc.), re- spectively; with a 16-bit precision, bitwise shifts may achieve at least 9.7 x and 1.45x average power and area savings than multipliers (Elhoushi can save up to 196x and 31x x energy costs over multipli- cations in 32-bit fixed-point (FIX32) formats, and 47x and 4.1x energy costs in bi- floating-point (FP32) formats. when implemented in a 45mm CMOS technology and SOTA FPGA (Xilinx Inc.), respectively, while aggressively leading to 1.84x, 25.5x, and 7.83 x area savings than multiplications in a 45mm CMOS technology with FP32, FIX32, and FIX8 formats, respectively (Chen et al., 2021c).
ASIC (45mm) Measurement
Unit Energy (p-)
0.1
0.01
FP32
FIX52
FPGA (ZC706) Measurement
Liit Energy (pa)
100
10
196
al., 2021); and similarly, additions
0.01
Multiplication
Add
Shift
Figure 1: Unit energy comparisons.
Supernet for NLP tasks. Based the above search space, we construct supernet for the con- venience of search following SOTA one-shot NAS methods (Cai et al., 2018; Guo et al., 2020) by estimating the performance of each candidate hybrid model (i.e., subnet) without fully training it. As shown in Fig. 2 (a), each macro-block in the supernet includes all the aforementioned four candidate blocks and three multi-branch combinations (e.g., Attn+Conv) along the channel dimension for cap- turing both global and local context information following (Wu et al., 2020), where the candidate blocks in the same layer are isolated with each followed by two-layer MLPs and enabling elastic embedding dimension, head numbers, and MLP hidden dimension for fine- ine-grained search for effi- cient NNs as (Wang et al., 2020a). Overall, our supernet for NLP tasks contains about 10 subnet
4

---

Under review as a conference paper at ICLR 2022
Block 1-1
Update
Ingut
Disabled
Block
Input
Arti
Attn
Conv
Altoy
Add
Altn
Stift
Comv SenfuAdd
Attn
1BF
Block
Shift
Mask (0: deactivated; activated)
Block 1+1
Add
1xT Conv
Output
Input Scale
Oulput
Position
Enooding
(a) The SuperNet for NLP Tasks
(b) The SuperNet for CV Tasks
Figure 2: Supernets for NLP and CV tasks: (a) For NLP, we adopt a multi-branch structure for each block of the supernet, where Attn+Conv represents the channel-wise concatenation of these two blocks, and (b) for CV tasks, we consider a multi-resolution pipeline for each block of the supernet.
candidates, and the searchable choices are listed in Tab. 1. During training, all possi-
Table 1: The search space for NLP tasks.
[Atn, Attn-CConv. AtiffShift
Encoder block types
ble subnets are uniformly sampled and only one
[Aun+Add, Cony, Shift, Add]
path is activated for each layer run-time con-
Decoder block types
[Attn, Attn+Conv] |Attn+Shift, Atmn+Add]
sidering the practical concern on memory con-
Num. of decoder blucks
[6, S, 4, 3,3, 2, 1] [1024, 768, 512]
sumption for supernet training. For ease of
Elastic embed. dim.
evaluation, we incorporate common treatments of NAS in our suppenet design. First, for the elastic dimensions mentioned above, all sub- nets share the front portion of weights or heads of the largest dimension. Second, all decoder blocks can take the last one, two, or three encoder blocks as inputs for abstracting both high-level and low- level information (Wang et al., 2020a). Note that the number of decoder blocks are also searchable and the conv, shift and add operators are disabled for decoder blocks, as they are observed to be sensitive and activating those paths might hurt the accuracy (You et al., 2020a; Wu et al., 2019b).
Elastic head num.
[16, 8, 4]
Elastic MLP dim.
14096 3072, 2048, 1024]
Arbitrary Alin
[3, 2, 1]
Supernet for CV tasks. Different from the com- monly used elastic hidden dimension design for NLP tasks, various spatial resolutions scales are essential for CV tasks. As such, to ensure more capable feature description of the searched NNs, we adopt a multi-resolution supernet design. As shown in Fig. (b), the supernet incorporates flexible downsampling options, where the spatial resolution for each layer can either stay unchanged or be reduced to half of its previous layer's scale until reaching the smallest resolution. In this way, the four candidate blocks can work collabora- tively to deliver the multiscale features required by most CV tasks. Overall, our supernet contains about 10 subnets, for which the detailed searchable choices are summarized in Tab. 2. Note that the Attn block is followed by two-layer MLPs and we also include residual connection for each block as inspired by (Srinivas et al., 2021). During training, the supernet performs uniform sampling and only activates one path of the chosen resolution and block type for each layer as for the NLP tasks.
Table 2: The scarch space for CV tasks.
Block types
[Attn. Cony. Shift, Add]
Num, of 56'x x 128 blocks
[1, 3, 4]
Num. of 28 x26 blocks
[,,2,3.4]
Num. of 14' blocks
[3.4, 5,
Num. of 1024 blocks
5, 7, 8, 9]
3.2 SHIFTADNNA: SEARCH METHOO
3.2.1 BACKGROUND AND FORMULATION OF ONE-SHOT NAS
We adopt one-shot NAS for improved search efficiency, i.e., assuming that the subnet candidates can directly inherit their weights from the supernet, following SOTA NAS works. Such a strategy is commonly referred as weight sharing. Specifically, the supernet ' with parameters W is trained to obtain the weights for all subnets within the search space S. Since the supernet training and archi- tecture search are decoupled in one-shot NAS, it usually requires two-level optimization: supernet training and architecture evaluation as defined below:
W s = arg min Levain ( (N(S,W))
(3)
a" arg max ACCmat(N(a, Ws(a))).
(4)
RES
where (SS, W) represents all possible candidate subnets within the search space. We first train the supernet by uniformly sampling different subnets a from as formulated in Eq. (3), after which all subnet candidates a directly inherit their corresponding weights Ws(a) from the supernet Ws- Finally, we evaluate the accuracy ACCrat(.) of each path on the validation set and search for the best subnet with the highest accuracy formulated in Eq. (4).
5

---

Under review as a conference paper at ICLR 2022
Input
TO
Shared Weight pool
Shift Weights
Quant.
Conv
7(-)
Add
Weights
Weights
0g
Output
(a) Heterogenous Weight Sharing Strategy
(b) Transformation Kernel
Figure 4: (a) Illustration of the proposed heterogeneous weight sharing strategy, where weights of shift blocks are quantized to powers of two; (b) visualization the adopted learnable transformation kernel T-)) for mapping the shared weights of Gaussian distribution a Laplacian distribution.
3.2.2 PROPOSED HETEROGENEOUS WEIGHT SHARING STRATEGY
Dilemma of vanilla ShiftAddNAS. The target hybrid search space of ShiftAddNAS inevitably enlarges the supemet due to the newly considered operators. As such, activating all block choices without weight sharing (Gong et al., 2019; Cai et al., 2018) can easily explode the mem- ory consumption of NAS. On the other hand, di- rectly sharing weights among different operators as (Chen et al., 202Ib) will lead biased search, especially for our hardware- pendwwrr-inspire hybrid search space where weights and activations of different operators follow heterogeneous distributions, e.g.. weights of the Conv and Add blocks follow Gaussian and Laplacian distribution, respectively, as shown in Fig. 3 and also highlighted in (Chen et al., 2020). Specifically, if we follow the existing weight sharing strategy to enforce a homo- geneous weight distribution among different operators during training the supernet, the resulting weights will not match the heterogeneous weight distributions of independently trained optimal hy- brid subnets. That is to say, for NAS with the target hybrid search space, there exists optimization gap between the goals of weight sharing optimization and individual subnet optimization, where the former is approximated while the latter is accurate (Xie et al., 2020). Hence, naively adopting the homogeneous weight sharing strategy can lead to inconsistent architecture ranking, which is major issue associated with one-shot NAS pointed out by (Chu al., 2019; You al., 2020b).
Gaussian
Laplacian
0.5
0.5
1.0
(b) Weights Add Figure 3: Heterogeneous weight distributions.
(a) Weights in Conv
heterogeneous weight sharing. To tackle the aforementioned dilemma, we propose heterogeneous weight sharing strategy that can simultancously reduce the supernet size corresponding to the target hybrid search space and allow weights of different blocks to follow heterogeneous distributions. Specifically, the learning objective for the supernet includes both the traditional cross-entropy loss and KL-divergence loss that is to regularize weight distributions to be close to either a standard Gaussian distribution (O, 7) or Laplacian distribution (0, A), (Xie et al., 2020; Chen et al., 2020), where / is the identity matrix and À 1, dedicated for the Conv and Add blocks, respectively, to reduce the aforementioned optimization gap as formulated in Eq. (5);
Proposed solution:
Ls -CC PKL - (( (og(((()));))
(5)
+ (PL((Po (Ws) II (V( 1)) DEL (Paea (T(Ws)))))) || (,,)
where (xi Wi are training data, ÿ denotes the output prediction, and (xl(p) =
- p(2) p(s) dz measures the KL-divergence between two distributions. During training, main- tain a shared weight pool for each layer share weights across all the Conv, Add, and Shift blocks, as illustrated in Fig. 4 (a). Meanwhile, weights of the Conv blocks directly leverage the corresponding ones in the shared weight pool for both forward and backpropagation, while being encouraged to follow a Gaussian distribution by the objective function; weights of the Shift blocks quantize the shared weights of Gaussian distribution to powers of two before multiplying with the input features; and for the Add blocks, we make use of a learnable transformation kernel T(-) to map the shared weights of Gaussian distribution to a Laplacian distribution. For the learnable trans- formation kernel as captured by Eq. (6), the core idea is apply piece-wise linear transformation after flattening and sorting the weights in a descending order, and then to reshape and rearrange the transformed weights back to their positions before sorting.
o(2)

---

Under review as a conference paper at ICLR 2022
T(W) Σ a4 x W (++1)x
(6)
where {a denote the learnable parameters in T(-), (W;). 1i=1 represent the sorted weights (a total of n) in the pool, = n/d denotes an interval within which the transformation is linear, as illustrated Fig. 4 (b). As validated in our experiment (e.g., Fig. 3), such a transformation kernel can successfully transform the shared weights of Gaussian to the desired Laplacian distribution, which is consistent with previous observations about kernel learning via linear transformation (Jain et al., 2012). In our design, each layer has its own Iearnable kernel T(-) with dimension d of 200 throughout all the experiments as we observed that such a dimension is adequate to learn the transformation across all the models and datasets, leading to over 40% supernet size reduction while only incurring a negligible (< 0.01% of the supernet size and computational costs) search overhead. After the supernet well trained, evolution search is applied to find the optimal subnets.
EXPERIMENT RESULTS
In this section, we first describe experiment setups, and then benchmark ShiftAddNAS over SOTA CNNs, Transformers, and previous NAS frameworks on both NLP and CV tasks. After that, conduct ablation studies regarding ShiftAddNAS's 's heterogeneous weight sharing strategy.
4.1 EXPERIMENT SETUPS
Datasets, baselines, and evaluation metries. For NLP tasks, we consider two machine translation datasets, WMT' 14 English to French (En-Fr) and English to German (En-De), which consist of 36.3M and 4.5M pairs of training sentences, respectively. The train/val/test splits follow the tra- dition as in (Wang et al., 2020a; Gehring et al., 2017). We consider five baselines: Transformer (Vaswani et al., 2017). Lightweight Conv (Wu et al., 2019b), Lite Transformer (Wu et al., 2020), and two previous NAS works including Evolved Transformer (So et al., 2019) and HAT (Wang et al., 2020a). We evaluate in terms of five evaluation metrics: the number of parameters/FLOPs. achieved BLEU, and hardware energy and latency measured on a SOTA accelcrator Eyeriss (Chen et al., 2016) clocked at 250MHz, where the BLEU is calculated with case-sensitive tokenization following (Wang et al., 2020a). For CV tasks, we consider the ImageNet dataset and four kinds of SOTA base- lines: four multiplication--fre CNNs (Chen et al., 2020; Xu al., 2020: Courbariaux et al., 2016; Elhoushi et al.. 2021), two CNNs (He et al., 2016; Hu et al., 2018), five Transformers (Dosovitskiy al., 2021; Touvron et al., 2021; Yuan al., 2021; Han al., 2021: Srinivas et al., 2021), and four NAS works (i.e., HR-NAS (Ding et al., 2021). BossNAS (Li et al., 2021), AutoFormer (Chen et al., 2021b), and VITAS (Su al., 2021)). Similar to those for the NLP tasks, we adopt five evaluation metrics: the number of parameters/MACs, achieved accuracy, and hardware energy and latency.
Search and training settings. For NLP tasks, after training the supernet for 40K steps, we adopt an evolutionary algorithm (Wang et al., 2020a) to search for subnets with various latency and FLOPs constraints ranging from 1.5G to 4.5G for 30 steps with a population of 125, a crossover population of 50, and a mutation population of 50 with a probability of 0.3. During search, measuring latency for each chosen subnet can be tine-consuming Instead, we estimate the latency using a three- layer NN trained with encoding architecture parameters as features and measured latency labels following (Wang et al., 2020a). The latency predictor is accurate with an average prediction error of < 5%. The searched subnets then retrained from seratch for another 40K steps with an Adam optimizer and a cosine learning rate (LR) scheduler, where the LR is linearly warmed up from 10-7 to 10-3 and then annealed (same for training supernets). For CV tasks, we conduct an evolutionary search with FLOPs constraints for 20 steps with a population of 50, a crossover population of 25, and a mutation population of 25 with a probability of 0.2 following (Chen et al., 2021b). We train both the supernet and searched subnets using the same recipe and hyperparameters as DeiT (Touvron et al., 2021). Note that the position encoding in the attention blocks is replaced with depthwise convolution following (Li et al., 2021) for reducing the computational complexity.
4.2 SHIFTADDNAS vs. SOTA ON NLP TASKS
We compare ShiftAddNAS with SOTA language models on two NLP tasks to evaluate its effective- ness. Fig. 5 shows that ShiftAddNAS consistently outperforms all the baselines in terms of BLEU scores and FLOPs. Specifically, ShiftAddNAS with full precision achieves 11.8% 73.6% % FLOPs reductions while offering a comparable or better BLEU score (-0.3 ~ +1.1), over all the full preci- sion baselines. To benchmark with Lite Transformer (8-bit) which is dedicated for mobile devices,
7

---

Under review as a conference paper at ICLR 2022
WMT'14 En-Fr
WMT'14 14 En-De
42
40-
BLEU Scores
39-
Transformer
26
Transformer
Evolved Transformer
23
HAT
HAT
Lighteeight Conv
ShiRAAdNNS
24
ShiftAddNAS
Lite Transfarnferrrr (8 (8-bit)
Lite Transformer (8-bit)
36-
ShiRAddNAS (8-bit)
ShiftAddNAS (8-bit)
0.0
0.2
0.4
0.6
0.8
1.0
0.0
6.2
0.4
0.6
0.8
10
FLOPS
lc10
FLOPS
1c10
Figure 5: BLEU scores vs.
FLOPs of ShiftAddNAS over SOTA baselines on NLP tasks.
Table 3: ShiftAddNAS vs. SOTA baselines in terms of accuracy and efficiency on NLP tasks.
WMT Fin
WMT" 14 Fa-De
Params
HLOPS
BLEU
Latency
Energy
Params
FLOPs
BL.U
Latency
Energy
Transformer
176M
10.6G
41.2
130ms
214ml
176M
10.6G
28.4
130ms
214mJ
Evolved Truns.
175M
10.8G
41.3
47M
2.9G
28.2
HAT
48M
3.4G
41.4
49ms
81mJ
44M
2.7G
28.2
42ms
69ml
ShiftAddNAS
46M
3.0G
41.8
43ms
71mJ
43M
2.7G
28.2
40ms
66mJ
HAT
46M
2.9G
41.1
42ms
69mJ
36M
2.2G
27.6
34ms
56mJ
ShifAddNAS
41M
2.7G
41.6
39ms
64m,]
33M
2.1G
27.8
31ms
52m,J
HAT
30M
1.8G
39.1
29ms
48mJ
25M
L5G
25.8
24ms
40mJ
ShiftAddNAS
29M
1.8G
40,2
16ms
45mJ
25M
1.6G
26.7
24ms
40mJ
Lite Trans. (8-bit)
17M
IG
39.6
19ms
31mJ
17M
IG
26.5
19ms
31ml
ShiftAddNAS (8-bit)
11M
0.2G
41.5
11ms
16mJ
17M
0.3G
28.3
16ms
24mJ
Lite Trans. (8-bit)
12M
0.7G
39.1
14ms
24mJ
12M
0.7G
25.6
14ms
24mJ
ShiftAddNAS (8-bit)
10M
0.2G
41.1
10ms
15mJ
12M
0.2G
26.8
9.2ms
14mJ
we refer to SOTA quantization technique (Banner al., 2018) for quantizing ShiftAddNAS 8-bit
fixed point: ShiftAddNAS (8-bit) achieves +1.8 ~ +4.9 BLEU scores improvements over Lite Trans- former (8- (B-, while offering 5.0% 82.7% FLOPs reductions, and aggressively reduces 91.6%
98.4% FLOPs compared to all the full- precisieiinn baselines with comparable BLEU scores (-0.1 ~ +0.3). Note that for quantized models, we follow (Zhou al., 2016) to use FLOPs x (Bit/32)² for calculating the effective FLOPs which is proportional to the number of bit-operations. We fur- ther compare various aspects of ShiftAddNAS with all baselines in Tab. 3. As illustrated in this Table, ShiftAddNAS consistently outperforms the baselines, c.g., achieves up to +2 BLEU scores improvement when comparing ShiftAddNAS (8-bit) with Lite Transformer on WMT 14 En-Fr and 69.1% % and 69.2% energy and latency savings when comparing ShiftAddNAS with Transformer WMT" 14 En-De, with a comparable or even fewer model parameters and FLOPs.
4.3 SHIFTADDNAS vs. SOTA ON CV TASKS
We further compare ShiftAddNAS over SOTA base- lines on ImageNet to evaluate effectiveness on the image classification task. As shown in Tab. 4. ShiftAddNAS outperforms a wide range of baselines. Here refer MACs as Multi- ply-accumulate or Shift-accumulate operations. For example, ShiftAddNet-TTO (searched with 4.5G
ImageNet
AdderNet (Mult.ffree)
Top-1 Acc. (%)
82
ResNet
VIT-8/16 (Transformer)
80
-6444%
BOSsNAS (CNN Trans.)
ShiftAddNAS (Hybrid)
Acc.!
93.0% Energy
MACs constraint) with 3.7G MACs achieves an im-
76
proved top-1 accuracy of (1) +5.3% +26.3%
over SOTA multiplication-free CNNs, (2) +0.7%
1000
1500
2000
+6.0% over SOTA CNNs, (3) +0.4% +7.6%
Energy (mj)
over SOTA Transformers, (4) +1.3% % ~ +4.8% % over SOTA CNN-Transformers, and (5) +1.3%, +4.7% %,
Figure 6: Accuracy vs. energy costs of Shif-
tAddNAS over baselines.
and +0.4% over previous SOTA NAS baselines BossNAS, VITAS, and Autoformer, respectively, under a comparable or even less MACs. More- over, considering looser MACs constraints, we follow BossNAS to remove the downsampling in the last stage, resulting in ShiftAddNAS-TT with a accuracy of 82.7 % and 6.4G MACs that surpasses T2T-ViT and BoTNet-S1-55 by +0.8 % and +1.0% at even less MACs. By directly testing larger input resolutions without finetuning, ShiftAddNAS-THe (w/ 256 input resolution) offers an accu- racy of 83.0%, surpassing BossNAS-T1 and Autoformer-B by +0.8 % and +0.6 % with comparable
8

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison with SOTA baselines ImageNet classification task.
Model
Top-1 Acc.
Top-5 Acc.
Params
Res.
MACs
#Mult.
#Add
#Shift
Model Type
BNN
55.8%
78.4%
26M
224'
3,9G
0.1G
3.9G
3.8G
Mult.-free
AdderNet
74.9%
91.7%
26M
224"
3,9G
0.1G
7.6G
0
Mrlt.-free
AdderNet-PKKD
76.8%
93.3%
26M
224
3.9G
0.1G
7.6G
0
Mult.-free
DeepShift-Q
70.7%
90.2%
26M
224"
3.9G
0.1G
3.9G
3.8G
Mult.-free
DeepShift-PS
71.9%
90.2%
52M
224
3,9G
0.1G
3.9G
3.8G
Mfle.-free
ShiftAddNer
72.3%
64M
224
10G
0.1G
16G
3,9G
Mult.-fre
ResNet-50
76.1%
92.99
26M
224
3,9G
3.9G
3.9G
CNN
ResNet-101
77.4%
94.2%
45M
2242
7.6G
7.6G
7.6G
CNN
SENet-50
79.4%
94.6%
26M
224"
3.9G
3.9G
3.9G
CNN
SENct-101
81.4%
95.7%
45M
224"
7.6G
7.6G
7.6G
CNN
VIT-B/16
77.9%
86M
3844
18G
I8G
17G
Transformer
VIT-L/16
76.5%
304M
384
64G
64G
63G
Transformer
DeiT-T
74.5%
6M
224
1.3G
1.3G
1.3G
Transformer
DeiT-S
81.2%
22M
224
4.6G
4.6G
4.6G
Transformer
VITAS
77.4%
93.8%
2244
2.7G
2.7G
2.7G
Transformer
Autoforee-SS
81.7%
95.7%
23M
224
5.IG
5.1G
5.1G
Transformer
BoT-50
78.3%
94.2%
21M
224
4.0G
4.0G
4.0G
CNN + Trans.
BoT-50 SE
79.6%
94.6%
21M
224"
4.0G
4.0G
4.0G
CNN + Trans.
HR-NAS
77.3%
6.4M
224
0,4G
0.4G
0.4G
CNN + Trans.
BossNAS-TO
80.5%
95.0%
38M
224
3.5G
3.5G
3.5G
CNN + Trans.
BossNAS--TO SE
80.8%
95.2%
38M
224
3.5G
3.5G
3.5G
0
CNN Tians.
ShitAAdNAS-TO
82.1%
95.8%
30M
224
3.7G
2.7G
3.8G
1.0G
Hybrid
ShitAAdNAS-TTO
82.6%
96.2%
30M
256
4.9G
3.6G
4.9G
1.4G
Hybrid
T2T-VF-19
81.9%
224"
8.9G
8.9G
8.9G
Transformer
TNT-S
81.3%
95.6%
24M
224
5.2G
5.2G
5.2G
Transformer
Autofomer-B
82.4%
95.7%
54M
224
11G
11G
11G
Transformer
BoTNet-SS159
81.7%
95.8%
28M
224'
7.3G
7.3G
7.3G
CNN + Trans.
BossNAS-TI
82.2%
95.8%
38M
224
8.0G
8.0G
8.0G
CNN + Trans.
ShiltAddNA--TI
82.7%
96.1%
30M
224"
6.4G
5.4G
6.4G
LOG
Hybrid
ShittAddNA--TT
83.0%
96.4%
30M
256
8.5G
7.1G
8.5G
1.4G
Hyhrid
Table 5: Ablation study of ShiftAddNAS w/ (1) naive and (2) heterogeneous weight sharing.
ShiftAddNAS
Kendall T Pearson
Spearman p
Top-1 Acc.
Params
Energy
Latency
w/ Naive WS
0.49
0.67
0.69
81.3%
30M
440mJ
387ms
w/ HWS
0.54
0.75
0.74
82.7%
30M
413mJ
252ms
or even less MACs, respectively. Finally, we compare ShiftAddNAS with representative baselines of various model types terms of accuracy and energy cost Fig. where each line represents a series searched/designed models with various FLOPs constraints. We can see that ShiftAddNAS consistently outperforms all the baselines, on average offering a +0.8% +7.7% higher accuracy and 24% 93% energy savings. Specifically, our ShiftAddNAS on average achieves +0.8% higher accuracy and 30 % energy savings against the competitive NAS baseline BossNAS
4.4 ABLATION STUDIES OF SHIFTADDNAS
We conduct ablation studies on ShiftAddNAS's heterogeneous weight sharing (HWS) strategy, as shown in Tab. 5. First, for searching on ImageNet, we use three ranking correlation metries: Kendall Pearson R. and Spearman p. measure the ranking correlation between ShiftAddNAS w/ and w/o HWS and find that the former leads to a higher ranking correlation than the naive WS. Second, the proposed HWS leads to more accurate searched subnets. Specifically, the searched subnet achieves a +1.4% % higher accuracy than that of naive WS, at comparable or even smaller energy and latency costs. Also, HWS effectively reduces the supernet size from 615M (w/o WS) to 364M (41% sav- ings). This set of ablation studies validate the effectiveness of our proposed HWS strategy.
CONCLUSION
We propose ShiftAddNAS for searching for multiplication-redueed NNs incorporating both pow- erful yet costly multiplications and efficient yet less powerful shift and add operators for marrying the best of both worlds. ShiftAddNAS is made possible by integrating: (1) the first hybrid search space that incorporates both multiplication-basel and multiplication-fres operators for searching for more accurate and efficient NNs; and (2) a novel heterogeneous weight sharing strategy that allows different operators to follow heterogeneous distributions for alleviating the dilemma of either inef- ficient search or inconsistent architecture ranking when applying NAS for hybrid NNs. Extensive experiments both NLP and CV tasks demonstrate the superior accuracy and energy efficiency of ShiftAddNAS's searched NNs over various SOTA baselines. Our ShiftAddNAS has opened up a new perspective in searching for more accuracy and efficient NNs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. arXiv preprint arXiv: 1046, 2018.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:18/2.00332, 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint (90,:0990.999 2019.
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. volume 005/2107.02960, 2021a. URL https:t/t///t. http//arivvvo//////
org/abs / 2107 02960.
Hanting Chen, Yunhe Wang. Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addemnet Do we really need multiplications in deep learning? The IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani. Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 76- 86. Association for Computational Linguistics, July 2018. doi: 10.18653/v 1/1533/1/P18--1 1008. URL https: httpss:///caaathllogggeo
org/p18- -0088.
Minghao Chen, Houwen Peng. Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Peng Chen, Jing Liu, Bohan Zhuang, Mingkui Tan, and Chunhua Shen. Aqd: Towards accurate quantized object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pustern Recognition, pp. 104-113, 2021c.
Yu-Hsin Chen, Tushar Krishna, Joel Emer, and Vivienne Sze. Eyeriss: An energy-efficient recon- figurable accelerator for deep convolutional neural networks. j.uEnalunal of solid-state circuits, 12(1):127-138, 2016.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint 197.00190..0/845, 2019.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 2016
or-1. arXiv preprint 1r02v:1602.000
Mingyu Ding, Xiaochen Lian, Linjie Yang, Peng Wang. Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / / /popnreviiew...
Mostafa Elhoushi, Zihao Chen, Farhan Shafiq. Ye Henry Tian, and Joey Yiwei Li. Deepshift: To- wards multiplication-ler neural networks. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops, pp. 2359-2368, June 2021.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In Proceedings the 34th International Conference Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1243-1252, PMLR, 06- 11 Aug 2017. URL https: / prwpeeooceeioooooooooooo ngs ml wwt.ggs.mm....s.....mmmmmmmmmm.e httpsss/vv//////// 7 html.
10

---

Under review as a conference paper at ICLR 2022
Chengyue Gong, Zixuan Jiang, Dilin Wang. Yibo Lin, Qiang Liu, and David Z Pan. Mixed precision neural architecture search for energy efficient deep learning. In 2019 IEEE/ACM International Conference Computer-Aide -ided Design (ICCAD), pp. 1-7. IEEE, 2019.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin. Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2 136, 2021.
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang. Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 77:354 377, 2018.
Zichao Guo, Xiangyu Zhang. Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision, pp. 544-560 Springer, 2020.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. URL https: torch.
hwtp///aaabbooiieeeeeeo resnet.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.080 02531, 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Prateek Jain, Brian Kulis, Jason V Davis, and Inderjit S Dhillon. Metric and kernel learning using linear transformation. The Journal of Machine Learning Research, 13(1):519 547, 2012.
Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local binary convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 19-28, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Changlin Li, Jiefeng Peng. Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xi- aojun Chang. Blockwisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1989-19998 2020a.
Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang, Xiaodan Liang, and Xiaojun Chang. BossNAS: Exploring hybrid CNN-transfommers with block-wisely self-supervised neural architecture search. In ICCV, 2021.
Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13836-13845, 2020b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differenniable architecture search. arXiv preprint arXiv: 100.00005000 2018.
David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pp. 5877-5886. PMLR, 2019.

---

Under review as a conference paper at ICLR 2022
Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15648-15657, 2021.
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16519-16529, 2021.
Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. arXiv preprint arXiv:2106.1370 13700, 2021.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, 14. PMLR, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 10347-10357. PMLR, 18-24 Jul 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Diffcrentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12965-12974, 2020.
Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernet with alpha-divergence. arXiv preprint arXiv:2102.07954 2021a.
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005. 14187, 2020.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810-1822. Association for Com- putational Linguistics, July 2019. doi: 11118653/1//11-1111
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766 2020b.
Yunhe Wang, Mingqiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arx: 100. 2021b.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial con- volutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9127-9135, 2018
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742 2019a.
12

---

Under review as a conference paper at ICLR 2022
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019b. URL
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.m 1580, 2021.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In International Conference on Learning Representations, 2020. URL https:
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2 2021.
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Zhengsu Chen, Lanfei Wang, An Xiao, Jianlong Chang. Xiaopeng Zhang, al. Weight-sharing neural architecture search: battle to shrink the optimization gap. arXiv preprint arXiv:2008. 01475,11555 2020.
Xilinx Inc. Xilinx zynq-7000 soc zc706 evaluation kit. https:/ // /wwww//wwwww///////// xi xilinxx......... com/ products/boa and-ki 22-p0/006 html.
Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing XU, and Yunhe Wang. Kernel based progressive distillation for adder neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 11222-13333 Curran Associates, Inc., 2020.
Yibo Yang, Shan You, Hongyang Li, Fei Wang, Chen Qian, and Zhouchen Lin. Towards improving the consistency, efficiency. and flexibility of differentiable neural architecture search. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6667-6676, June 2021.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inse wew.are--nsppeedddeee deep network. In Advances in Neural Infor- mation Processing Systems, volume 33, pp. 2771-2783. Curran Associates, Inc., 2020a.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020b.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang. Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In European Conference on Computer Vision pp. 702-717. Springer, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11988. 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pp. 17283-17297. Curran Associates, Inc., 2020.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXiv:1606.06116/ 2016.
13

---

Under review as a conference paper at ICLR 2022
A EVALUATE THE SEARCH COST
We further supply the total search cost of ShiftAddNAS on both NLP tasks and CV tasks to Table 6 and 7, respectively. For NLP tasks, with one Nvidia V100 GPU, ShiftAddNAS uses on average
9.3 GPU days (Gds) for searching which is comparable to HAT (Wang et al., 2020a) and 9,821 less than the Evolved Transformer (So et al., 2019); For CV tasks, ShiftAddNAS uses on average 8.9 Gds for searching which is 1 11% and 82% less than DARTS (Liu et al., 2018) and BossNAS (Li al., 2021), respectively. In addition, we provide a concrete breakdown analysis of ShiftAddNAS search cost in Table 8. For NLP tasks, ShiftAddNAS uses on average 8.5 Gds for supernet training and 0.8 Gds for architecture searching; For CV tasks, ShiftAddNAS uses on average 7.7 Gds for supernet training and 1.2 Gds for architecture searching.
Table 6: Search cost on NLP tasks.
Table 7: Search cost CV tasks.
Table 8: Breakdown analysis of the search cost of ShiftAddNAS.
Methods
Search Cost
Methods
Search Cost
Evolved Trans.
91,334 Gds
DARTS
50 Gds
NLP
CV
HAT
9.3 Gds
BossNAS
10 Gds
Supernet Train
8.5 Gds
7.7 Gds
ShifAAdNNA
9.3 Gds
ShiftAddNAS
8.9 Gds
Arch. Search
0.8 Gds
1.2 Gds
B VISUALIZATION OF THE HETEROGENEOUS WEIGHT DISTRIBUTIONS
For better understanding the proposed heterogeneous weight sharing strategy, we further supply the visualization of the heterogeneous weight distributions in Conv/Add/Shte layers, respec- tively, as shown in the Fig. 7.
Gaussian
Laplacian
Discrete
(a) Weights in Conv
(b) Weights in Add
(c) Weights Shift
Figure 7: Visualization of the heterogeneous weight distributions in Conv/Add/Sha Shift layers.
C VISUALIZATION OF THE SEARCHED ARCHITECTURE
For better understanding of the searched architecture, supply the visualization of the searched architecture by ShiftAddNAS in Fig. 8. The searched architecture contains two adder blocks and one shift block, and achieves 82.8% top-1 test accuracy ImageNet with 8.4G MACs (Mult: 7.4G; Add: 9.1G; Shift: 0.5G). Also, the searched architecture prefers Conv as early blocks while consider Attn later blocks, which is also consistent with the previous empirical observation that early convolutions help the overall performance (Xiao et al., 2021).
5
L-2
L-1
1/4
1/8
1/16
1/32
Attn
Conv
Add
Shift
Figure& 8: Visualization of the searched architecture with 82.8% 1 test accuracy on ImageNet.
14


Reviews:
[{"review_id": "RXWjJ4IQog", "reviewer": "Reviewer_evhx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. This paper is innovative. The search space contains both multiplication-free and multiplication-based operators.\n2. The results are the state-of-the-arts on both NLP and CV tasks.\n\nCons:\n1. What is the meaning of Naïve WS in Table 5? It means weight sharing between operations without transformation or no weight sharing between operations? Generally, we only share weights of the same operator between different architectures, why sharing weights between operators here? In most cases, the more weights are shared, the lower the Kendall-tau is.\n2. A learnable transformation kernel is used to translate the convolution to shift or adder operation. What if we just apply this kernel to the whole convolution neural network? Can we get an energy efficient adder or shift neural network without much accuracy loss with this kernel? There should be more analyses and experiments to show the effectiveness of this kernel.\n3. In Table 4, the number of Add in ShiftAddNAS is always the same as MACs. In my opinion, this means no adder operator is used, since one MAC actually means two Add for adder operation (just like the AdderNet in Table 4). Why there is no adder operator at all? There should be figures to show your final searched models.\n", "summary_of_the_paper": "This paper propose a hybrid search space including both multiplication-based and multiplication-free operators. The authors also give a novel weight sharing strategy to enable effective weight sharing among different operators. ", "summary_of_the_review": "The idea of this paper is novel and the results are satisfying, but there are more details should be provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PbFWOjv4yYY", "reviewer": "Reviewer_1Due", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. I think it is quite interesting to integrate hardware-efficient shift/add operations into the search space of NAS, which can potentially improve the efficiency of neural networks on hardware.  \n2. The motivation of heterogeneous weight sharing is clear and reasonable. \n\nWeaknesses:\n1. Insufficient technical contributions. The idea of heterogeneous weight sharing is not new. As far as I know, when training super-nets with flexible quantization bits (e.g., [1]), a similar strategy has already been used: maintain a shared weight pool, fp32 directly uses the weight, int8/4/2 use the quantized weights. Compared to the previous strategy, this work has additional regularization terms (Gaussian for Conv ops and Laplacian for Add ops). To convince me that this modification provides sufficient technical contributions, I would expect to see solid experimental studies showing clear and significant improvements from these regularization terms. In addition, as the search space only has three different types of operations (according to Figure 4), maintaining the weight separately for different types of blocks and having three activate paths seems to be doable. I would expect to see comparisons with this approach as well.\n2. Weak results. With a more hardware-efficient design space and the proposed heterogeneous weight sharing, I expect to see clear and significant improvements compared to previous SOTA methods. However, the improvements seem to be marginal. For example, in Table 3, ShiftAddNAS achieves the same BLEU score on WMT'14 En-De as HAT while providing only 1.05x latency reduction. In Table 4, the improvements of ShiftAddNAS also look marginal compared to strong baselines (e.g., EfficientNet, FBNetV3, etc). In addition, it would be helpful to provide the results of ShiftAddNAS with smaller MACs in Table 4 (e.g., <600M, <300M, etc). \n\n[1] Bai, Haoping, et al. \"BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer.\" arXiv preprint arXiv:2105.08952 (2021).", "summary_of_the_paper": "This paper introduces ShiftAddNAS that performs neural architecture search on a hybrid design space that contains conv operations, attention operations, and hardware-efficient shift/add operations. A heterogeneous weight sharing strategy is proposed to share weights between different types of blocks. ", "summary_of_the_review": "Overall, I think this paper explores an interesting topic and potentially can benefit real-world efficient deep learning applications. However, I find the technical contributions and empirical results presented in this version are not strong enough. Thus, I recommend rejection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MLjKaJ1Dy4D", "reviewer": "Reviewer_KyCd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The specially designed search space that incorporates both multiplication-based and multiplication-free operators is new.\n+ The analysis on distributions of convolution and add operators is reasonable.\n+ The proposed weight sharing strategy can enable effective weight sharing among different operators that follow heterogeneous distributions.\n\nWeaknesses:\n- The weight sharing strategy only considers convolution and add operations. What's the distribution of Shift operation? Should shift op be considered together?\n- In the experiment, lacking comparison with ShiftAddNet[1].\n\n[1] ShiftAddNet: A Hardware-Inspired Deep Network. NeurIPS 2020.", "summary_of_the_paper": "This paper proposes a NAS method for both multiplication-based and adder-based networks. The contributions mainly lie on hybrid search space and new weight sharing strategy. The experimental results shows the method can obtain energy-efficient networks with high performance.", "summary_of_the_review": "The main contribution of the paper is a NAS framework for multiplication-based and multiplication-free operators. The weaknesses do not affect the main contribution. I recommend accptance for it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "445EHrpoMZR", "reviewer": "Reviewer_PKFY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is well-motivated, well-written, good quality.\n- To my knowledge, it is the first paper to design a hardware-inspired hybrid search space between multiplication-based operators and multiplication-free operators for NAS.\n- Clear contribution: this work tackles the problem caused when applying weight-sharing-based or supernet-based search algorithms on the hybrid search space by proposing heterogenous weight sharing.\n- Solid empirical experiments including SOTA baseline methods.\n\n**Weaknesses**\n- Search algorithm is not new. \n- If I read correctly, ShiftAddNet which is one of the most relevant baselines of the proposed work is missing in the experiment section. The reviewer thinks that adding ShiftAddNet in the main Table 4 (or additional analysis section) is needed to show the performance improvement with increasing latency/energy compared with ShiftAddNet. \n- The reviewer thinks the cost to train supernet and search cost should be reported. \n- Generality issue on multiple hardware devices and energy & latency information are missing in Table 4. Even if this work handles 'hardware-inspired NAS', the target device in the main experiments is only eyeriss. The reviewer thinks that this work should validate the generality of this method in more cases by applying this method on multiple hardwares (at least GPU/CPU/Mobile) and reporting latency/energy saving.\n- [Minor] in Table 4, the number of parameters and MACs of the model obtained by this method is not SOTA, yet it is bolded. It would be better to be modified.", "summary_of_the_paper": "This paper designs a hybrid search space that includes multiplication-based operators and multiplication-free operations to find good trading points between accuracy-efficiency. Further, this work defines the problem when training weight-sharing supernet on the hybrid search space and proposes the heterogenous weight sharing algorithm to address the problem. The proposed method is validated on both NLP and CV tasks, outperforming several competitive baseline methods in terms of accuracy and saving efficiency on latency or energy. ", "summary_of_the_review": "The reviewer thinks that the proposed method is clear enough and practical to obtain neural architecture used for resource-constrained edge devices, yet, some experiments or information that are mentioned above should be added to validate the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a hybrid search space consisting of both multiplication-based and multiplication-free operators. It also presents a weight-sharing mechanism for searching in the introduced search space.

Pros:
* A hybrid search space is developed.
* Strong empirical results are reported for both CV and NLP tasks.
* The paper is well written and is easy to follow.

Cons:  
* Incremental technical novelty.
* Missing baselines and competing methods.
* Missing information on the search cost.
* Lack of insights into the discovered architectures

The rebuttal has provided most missing information and comparisons, and it has provided additional insights into the searched architectures. However, the reviewers still rate this paper at borderline primarily due to the limited technical novelties. Unfortunately, given these concerns, this submission does not meet the bar for acceptance at ICLR.
Decision: Reject