ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks

1 INTRODUCTION . The unprecedented performance achieved by neural networks ( NNs ) , e.g. , convolutional neural networks ( CNNs ) and Transformers , requires intensive multiplications and thus prohibitive training and inference costs , contradicting the explosive demand for embedding various intelligent functionalities into pervasive resource-constrained edge devices . In response , multiplication-free networks have been proposed to alleviate the prohibitive resource requirements by replacing the costly multiplications with lower-cost operators for boosting hardware efficiency . For example , AdderNet ( Chen et al. , 2020 ) utilizes mere additions to design NNs ; and ShiftAddNet ( You et al. , 2020a ) follows a commonly used hardware practice to re-parameterize NNs with both bitwise shifts and additions . Despite their promising performance in hardware efficiency , multiplication-free NNs in general under-perform their CNN and Transformer counterparts in terms of task accuracy for both computer vision ( CV ) and natural language processing ( NLP ) applications . To marry the best of both worlds , we advocate hybrid multiplication-reduced network architectures that integrate both multiplication-based operators ( e.g. , vanilla convolution ( Krizhevsky et al. , 2012 ) and attention ( Vaswani et al. , 2017 ) ) and multiplication-free operators ( e.g. , shift and add ( You et al. , 2020a ) ) to simultaneously boost task accuracy and efficiency . Thanks to the amazing success of neural architecture search ( NAS ) in automating the process of designing state-of-the-art NNs , it is natural to consider NAS as the design engine of the aforementioned hybrid NNs for various applications and tasks , each often requiring a different performance-efficiency trade-off . However , there still exist a few challenges in leveraging NAS to design the hybrid NNs . First , existing NAS methods mostly consider the search for either efficient CNNs ( Wan et al. , 2020 ) , Transformers ( Chen et al. , 2021b ) , or hybrid CNN-Transformers ( Ding et al. , 2021 ; Li et al. , 2021 ) , and there still is a lack of a seminal work that searches for multiplication-reduced hybrid networks , especially for the hardware-inspired networks that incorporate both bitwise shifts and additions . Second , a hybrid search space could make it more challenging to achieve effective NAS and further aggravate the search burden , due to the enlarged search space imposed by the newly introduced multiplicationfree operators . It is worth noting that existing weight sharing strategies of NAS do not directly apply to the target hybrid search space , because weights of different operators follow heterogeneous distributions , leading to a dilemma of either inefficient search or inconsistent architecture ranking . Specifically , weights in convolutional and adder layers follow Gaussian and Laplacian distributions , respectively , as also highlighted by ( Chen et al. , 2020 ; Xu et al. , 2020 ) . As such , forcing weight sharing among heterogeneous operators could hurt the capacity and thus the achieved accuracy of the resulting NNs , while treating them separately could explode the search space and make it more difficult to achieve effective NAS , i.e. , the dilemma mentioned above . To tackle the aforementioned challenges towards more accurate and efficient NNs , this work makes the following contributions : 1 . We propose a generic NAS framework dubbed ShiftAddNAS , which for the first time can automatically search for efficient hybrid NNs with both superior accuracy and efficiency . Our ShiftAddNAS integrates a hybrid hardware-inspired search space that incorporates both multiplication-based operators ( e.g. , convolution and attention ) and multiplicationfree operators ( e.g. , shift and add ) , and can serve as a play-and-plug module to be applied on top of SOTA NAS works for further boosting their achievable accuracy and efficiency . 2 . We develop a new weight sharing strategy for effective search with hybrid search spaces , which only incurs a negligible overhead when searching for hybrid operators with heterogeneous ( e.g. , Gaussian vs. Laplacian ) weight distributions as compared to the vanilla NAS with merely multiplication-based operators , alleviating the dilemma mentioned above regarding either inefficient search or inconsistent architecture ranking . 3 . We conduct extensive experiments and ablation studies to validate the effectiveness of ShiftAddNAS against state-of-the-art ( SOTA ) works . Results on multiple benchmarks demonstrate the superior accuracy and hardware efficiency of its searched NNs as compared to both ( 1 ) manually designed multiplication-free networks , CNNs , Transformers , and hybrid CNN-Transformers , and ( 2 ) SOTA NAS works , on both CV and NLP tasks . 2 RELATED WORKS . Multiplication-free NNs . Many efficient NNs aim to reduce their intensive multiplications that dominate the time/energy costs . One important trend is to replace the multiplications with lowercost operators : BNNs ( Courbariaux et al. , 2016 ; Juefei-Xu et al. , 2017 ) binarize both the weights and activations and reduce multiplications to merely sign flips at non-negligible accuracy drops ; AdderNets ( Chen et al. , 2020 ; Xu et al. , 2020 ; Wang et al. , 2021b ) fully replace the multiplications with lower-cost additions and further develop an effective backpropagation scheme for efficient AdderNet training ; Shift-based NNs leverage either spatial shift ( Wu et al. , 2018 ) or bit-wise shift operations , e.g. , DeepShift ( Elhoushi et al. , 2021 ) , to reduce the amount of multiplications ; and ShiftAddNet ( You et al. , 2020a ) draws inspirations from efficient hardware designs to reparamatize NNs with mere bitwise shifts and additions . While multiplication-free NNs under-perform their vanilla NN counterparts in terms of achieved accuracy , ShiftAddNAS aims to automatically search for multiplication-reduced NNs that incorporate both multiplication-based and multiplication-free operators for marrying the best of both worlds , i.e. , boosted accuracy and efficiency . Neural architecture search . NAS has achieved an amazing success in automating the design of efficient NN architectures . For searching for CNNs , early works ( Tan & Le , 2019 ; Tan et al. , 2019 ; Howard et al. , 2019 ) adopt reinforcement learning based methods that require a prohibitive search time and computing resources , while recent works ( Liu et al. , 2018 ; Wu et al. , 2019a ; Wan et al. , 2020 ; Yang et al. , 2021 ) utilize differentiable search to greatly improve the search efficiency . More recently , SOTA works adopt one-shot NAS ( Guo et al. , 2020 ; Cai et al. , 2019 ; Yu et al. , 2020 ; Wang et al. , 2021a ) to decouple the architecture search from supernet training and then evaluates the performance of sub-networks whose weights are directly inherited from the pretrained supernet . For searching for Transformers , recently emerging works ( Wang et al. , 2020a ; Su et al. , 2021 ; Chen et al. , 2021b ; a ) adopt one-shot NAS and an evolutionary algorithm to search for optimal Transformer architectures for both NLP and CV tasks . Additionally , BossNAS ( Li et al. , 2021 ) and HR-NAS ( Ding et al. , 2021 ) further search for hybrid CNN-Transformer architectures . Nevertheless , little effort has been made to exploring NAS methods especially their search strategies for multiplication-reduced NNs that incorporate both multiplication-based and multiplication-free operations . Furthermore , it is not clear whether existing efficient NAS methods are applicable to searching for such multiplication-reduced NNs . Specifically , prior weight sharing strategies may not work since weights and activations in CNNs and AdderNets follow a different distribution ( Chen et al. , 2020 ) . As such , it is highly desirable to develop NAS methods , e.g. , ShiftAddNAS , dedicated for hardware-inspired multiplication-reduced NNs to increase achievable accuracy and efficiency . Transformers . Transformers ( Vaswani et al. , 2017 ) were first proposed for NLP tasks , which has inspired many interesting works . Some advance Transformer design by improving the attention mechanism ( Chen et al. , 2018 ) , training deeper Transformers ( Wang et al. , 2019 ) , and replacing the attention with convolutional modules ( Wu et al. , 2019b ) ; and others strive to reduce Transformers ’ computational complexity by adopting sparse attention mechanisms ( Zaheer et al. , 2020 ) , low-rank approximation ( Wang et al. , 2020b ) , or compression techniques ( Wu et al. , 2020 ) . Recently , there has been a growing interest in developing Transformers for CV tasks : Vision Transformer ( ViT ) ( Dosovitskiy et al. , 2021 ) for the first time successfully applies pure Transformers to image classification and achieves SOTA task accuracy , which yet relies on pretraining on giant datasets ( Hinton et al. , 2015 ) ; following works including DeiT ( Touvron et al. , 2021 ) T2T-ViT ( Yuan et al. , 2021 ) develop new training recipes and tokenization schemes , for achieving comparable accuracy without the necessity of costly pretraining ; and another trend is to incorporate CNN modules into Transformer architectures for better accuracy and efficiency tradeoffs ( Wu et al. , 2021 ; Xiao et al. , 2021 ; Graham et al. , 2021 ) . In contrast , we advocate hybrid multiplication-reduced NNs and develop an automated search framework that can automatically search for such hardware inspired hybrid models . 3 THE PROPOSED SHIFTADDNAS FRAMEWORK . In this section , we first introduce the hybrid search space from both algorithmic and hardware costs perspectives , providing high-level background and justification for motivating ShiftAddNAS ; Sec . 3.2 elaborates the one-shot search method of ShiftAddNAS by first analyzing the dilemma of either inefficient search or inconsistent architecture ranking and then introducing the proposed novel heterogeneous weight sharing strategy tackling the aforementioned dilemma . 3.1 SHIFTADDNAS : HYBRID SEARCH SPACE . Candidate blocks . The first step of developing ShiftAddNAS is to construct a hybrid search space incorporating suitable building blocks that exhibit various performance-efficiency trade-offs . Specifically , we hypothesize that integrating both multiplication-based and multiplication-free blocks into the search space could lead to both boosted accuracy and efficiency , and consider blocks from two trends of designing NNs : ( 1 ) capable NNs , e.g. , vanilla CNNs and Transformers , leverage either convolutions ( Conv ) or multi-head self-attentions ( Attn ) that comprise of intensive multiplications to capture local or global correlations , achieving a SOTA accuracy in both CV and NLP tasks ; and ( 2 ) efficient multiplication-free NNs , e.g , ShiftAddNet , draw inspirations from hardware design practices to incorporate two efficient and complementary blocks , i.e. , coarse-grained Shift and fine-grained Add , for favoring hardware efficiency , while maintaining a decent accuracy . While our constructed general hybrid search space for both NLP and CV tasks are shown in Fig . 2 , we next analyze the building blocks from both algorithmic and hardware costs perspectives . • Attn is a core component of Transformers ( Vaswani et al. , 2017 ) , which consists of a number of heads H with each capturing different global-context information by measuring pairwise correlations among tokens as defined below : OAttn=Concat ( H1 , · · · , Hh ) ·WO , where Hi=Softmax ( QWQi · ( KWKi ) T√ dk ) · VWVi , ( 1 ) where h denotes the number of heads , Q , K , V ∈ Rn×d are the query , key , and value embeddings of hidden dimension d obtained by linearly projecting the input sequence of length n. For each head , WQi , W K i , W V i ∈ Rd×dk are learned projection weight matrices where dk = d/h is the embedding dimension of each head . In this way , the Attn block first computes dot-products between key-query pairs , scales to stabilize the training , uses Softmax to normalize the resulting attention scores , and then computes a weighted sum of the value embeddings corresponding to different inputs . Finally , the results from all heads are concatenated and further projected with a weight matrix WO ∈ Rd×d to generate the outputs . • Conv is a key operator of CNNs , which models the local-context information of highdimensional inputs such as images through sliding kernel weights W on top of inputs X to measure their similarity ( Gu et al. , 2018 ) , as defined in Eq . ( 2 ) . Its translation invariant and weight sharing ability leads to various SOTA CNNs ( He et al. , 2016 ) or hybrid CNN-Transformer models ( Xiao et al. , 2021 ) . However , the computational complexities of CNNs can be prohibitive due to their required intensive multiplications . For example , one forward pass of ResNet-50 ( He et al. , 2016 ) requires 4G floating point multiplications . OConv = ∑ XT ∗W ; OShift = ∑ XT ∗ ( S · 2P ) ; OAdd = − ∑ ‖X −W‖1 , ( 2 ) • Shift is a well-known efficient hardware primitive , motivating the recent development of shiftbased efficient NNs . For example , DeepShift ( Elhoushi et al. , 2021 ) parametrizes NNs with bitwise shifts and sign flips , as formulated in the middle of Eq . ( 2 ) , with W = S · 2P denoting weights in the shift blocks , where S ∈ { −1 , 0 , 1 } are sign flip operators and the power-of-two parameter for P represents the bitwise shifts . However , NNs built with shift blocks and quantized weights are observed to be inferior to multiplication-based NNs in terms of expressiveness ( accuracy ) as validated in ( You et al. , 2020a ) . • Add is another efficient hardware primitive which motivates recent works ( Chen et al. , 2020 ; Wang et al. , 2021b ; Song et al. , 2021 ) to design efficient NNs using merely additions to measure the similarity between kernel weights W and inputs X , as shown in the right part of Eq . ( 2 ) . Such add-based NNs ( Chen et al. , 2020 ; Xu et al. , 2020 ) in general have a better expressive capacity than their shift-based counterparts . For example , AdderNets ( Chen et al. , 2020 ) achieve a 1.37 % higher accuracy than DeepShift under similar or even lower FLOPs on ResNet-18 with the ImageNet dataset . However , add-based operators ( i.e. , repeated additions ) are not parameterefficient as compared to bitwise shift operations ( You et al. , 2020a ) . While NNs combining shfit and add achieve a boosted accuracy , efficiency , and robustness than NNs using merely either of them , their accuracy still compares unfavorably as compared with vanilla CNNs or Transformers . Based on the above introduction , the search space in ShiftAddNAS incorporates all the four different types of blocks ( i.e. , Attn , Conv , Shift , and Add ) , aiming to push forward both NNs ’ accuracy and efficiency . Note that we refer to all operators as blocks , and adopt block based search space because it has been evidenced and proven that block based ones can reduce the search space size and lead to more accurate architecture ranking/rating ( Li et al. , 2020b ; a ) . Hardware cost analysis . As mentioned , multiplicationbased operators ( e.g. , Attn and Conv ) favor a superior accuracy yet is not hardware efficient , while multiplication-free operators ( e.g. , Shift and Add ) favors a better hardware efficiency yet can hurt the achievable accuracy . Specifically , as shown in Fig . 1 , bitwise shifts can save as high as 196× and 24× energy costs over multiplications , when implemented in a 45nm CMOS technology and SOTA FPGA ( Xilinx Inc. ) , respectively ; with a 16-bit precision , bitwise shifts may achieve at least 9.7× and 1.45× average power and area savings than multipliers ( Elhoushi et al. , 2021 ) ; and similarly , additions can save up to 196× and 31× energy costs over multiplications in 32-bit fixed-point ( FIX32 ) formats , and 47× and 4.1× energy costs in 32-bit floating-point ( FP32 ) formats , when implemented in a 45nm CMOS technology and SOTA FPGA ( Xilinx Inc. ) , respectively , while aggressively leading to 1.84× , 25.5× , and 7.83× area savings than multiplications in a 45nm CMOS technology with FP32 , FIX32 , and FIX8 formats , respectively ( Chen et al. , 2021c ) . Supernet for NLP tasks . Based on the above search space , we construct a supernet for the convenience of search following SOTA one-shot NAS methods ( Cai et al. , 2018 ; Guo et al. , 2020 ) by estimating the performance of each candidate hybrid model ( i.e. , subnet ) without fully training it . As shown in Fig . 2 ( a ) , each macro-block in the supernet includes all the aforementioned four candidate blocks and three multi-branch combinations ( e.g. , Attn+Conv ) along the channel dimension for capturing both global and local context information following ( Wu et al. , 2020 ) , where the candidate blocks in the same layer are isolated with each followed by two-layer MLPs and enabling elastic embedding dimension , head numbers , and MLP hidden dimension for fine-grained search for efficient NNs as ( Wang et al. , 2020a ) . Overall , our supernet for NLP tasks contains about 1014 subnet candidates , and the searchable choices are listed in Tab . 1 . During training , all possible subnets are uniformly sampled and only one path is activated for each layer at run-time considering the practical concern on memory consumption for supernet training . For ease of evaluation , we incorporate common treatments of NAS in our suppenet design . First , for the elastic dimensions mentioned above , all sub- nets share the front portion of weights or heads of the largest dimension . Second , all decoder blocks can take the last one , two , or three encoder blocks as inputs for abstracting both high-level and lowlevel information ( Wang et al. , 2020a ) . Note that the number of decoder blocks are also searchable and the conv , shift and add operators are disabled for decoder blocks , as they are observed to be sensitive and activating those paths might hurt the accuracy ( You et al. , 2020a ; Wu et al. , 2019b ) . Supernet for CV tasks . Different from the commonly used elastic hidden dimension design for NLP tasks , various spatial resolutions or scales are essential for CV tasks . As such , to ensure more capable feature description of the searched NNs , we adopt a multi-resolution supernet design . As shown in Fig . 2 ( b ) , the supernet incorporates flexible downsampling options , where the spatial resolution for each layer can either stay unchanged or be reduced to half of its previous layer ’ s scale until reaching the smallest resolution . In this way , the four candidate blocks can work collaboratively to deliver the multiscale features required by most CV tasks . Overall , our supernet contains about 109 subnets , for which the detailed searchable choices are summarized in Tab . 2 . Note that the Attn block is followed by two-layer MLPs and we also include a residual connection for each block as inspired by ( Srinivas et al. , 2021 ) . During training , the supernet performs uniform sampling and only activates one path of the chosen resolution and block type for each layer as for the NLP tasks .

Summary: This paper designs a hybrid search space that includes multiplication-based operators and multiplication-free operations to find good trading points between accuracy-efficiency. Further, this work defines the problem when training weight-sharing supernet on the hybrid search space and proposes the heterogenous weight sharing algorithm to address the problem. The proposed method is validated on both NLP and CV tasks, outperforming several competitive baseline methods in terms of accuracy and saving efficiency on latency or energy.