Once Quantized for All: Progressively Searching for Quantized Compact Models

Automatic search of Quantized Neural Networks ( QNN ) has attracted a lot of attention . However , the existing quantization-aware Neural Architecture Search ( NAS ) approaches inherit a two-stage search-retrain schema , which is not only time-consuming but also adversely affected by the unreliable ranking of architectures during the search . To avoid the undesirable effect of the search-retrain schema , we present Once Quantized for All ( OQA ) , a novel framework that searches for quantized compact models and deploys their quantized weights at the same time without additional post-process . While supporting a huge architecture search space , our OQA can produce a series of quantized compact models under ultra-low bit-widths ( e.g . 4/3/2 bit ) . A progressive bit inheritance procedure is introduced to support ultra-low bit-width . Our searched model family , OQANets , achieves a new state-of-the-art ( SOTA ) on quantized compact models compared with various quantization methods and bit-widths . In particular , OQA2bit-L achieves 64.0 % ImageNet Top-1 accuracy , outperforming its 2 bit counterpart EfficientNet-B0 @ QKD by a large margin of 14 % using 30 % less computation cost . 1 INTRODUCTION . Compact architecture design ( Sandler et al. , 2018 ; Ma et al. , 2018 ) and network quantization methods ( Choi et al. , 2018 ; Kim et al. , 2019 ; Esser et al. , 2019 ) are two promising research directions to deploy deep neural networks on mobile devices . Network quantization aims at reducing the number of bits for representing network parameters and features . On the other hand , Neural Architecture Search ( NAS ) ( Howard et al. , 2019 ; Cai et al. , 2019 ; Yu et al. , 2020 ) is proposed to automatically search for compact architectures , which avoids expert efforts and design trials . In this work , we explore the ability of NAS in finding quantized compact models and thus enjoy merits from two sides . Traditional combination of NAS and quantization methods could either be classified to NASthen-Quantize or Quantization-aware NAS as shown in Figure 1 . Conventional quantization methods merely compress the off-the-shelf networks , regardless of whether it is searched ( EfficientNet ( Tan & Le , 2019 ) ) or handcrafted ( MobileNetV2 ( Sandler et al. , 2018 ) ) . These methods correspond to NAS-then-Quantize approach as shown in Figure 1 ( a ) . However , it is not optimal because the accuracy rank among the searched floating-point models would change after they are quantized . Thus , this traditional routine may fail to get a good quantized model . Directly search with quantized models ’ performance seems to be a solution . Existing quantization-aware NAS methods ( Wang et al. , 2019 ; Shen et al. , 2019 ; Bulat et al. , 2020 ; Guo et al. , 2019 ; Wang et al. , 2020 ) utilize a two-stage search-retrain schema as shown in Figure 1 ( b ) . Specifically , they first search for one architecture under one bit-width setting1 , and then retrain the model under the given bit-width . This two-stage procedure undesirably increases the search and retrain cost if we have multiple deployment constraints and hardware bit-widths . Furthermore , due to the instability brought by quantization-aware training , simply combining quantization and NAS results in unreliable ranking ( Li et al. , 2019a ; Guo et al. , 2019 ) and sub-optimal 1One bit-width setting refers to a specific bit-width for each layer , where different layers could have different bit-widths . quantized models ( Bulat et al. , 2020 ) . Moreover , when the quantization bit-width is lower than 3 , the traditional training process is highly unstable and introduces very large accuracy degradation . To alleviate the aforementioned problems , we present Once Quantized for All ( OQA ) , a novel framework that : 1 ) searches for quantized network architectures and deploys their quantized weights immediately without retraining , 2 ) progressively produces a series of quantized models under ultra-low bits ( e.g . 4/3/2 bit ) . Our approach leverages the recent NAS approaches which do not require retraining ( Yu & Huang , 2019 ; Cai et al. , 2019 ; Yu et al. , 2020 ) . We adopt the search for kernel size , depth , width , and resolution in our search space . To provide a better initialization and transfer the knowledge of higher bit-width QNN to the lower bit-width QNN , we propose bit inheritance mechanism , which reduces the bit-width progressively to enable searching for QNN under different quantization bit-widths . Benefiting from the no retraining property and large search space under different bit-widths , we can evaluate the effect of network factors . Extensive experiments show the effectiveness of our approach . Our searched quantized model family , OQANets , achieves state-of-the-art ( SOTA ) results on the ImageNet dataset under 4/3/2 bitwidths . In particular , our OQA2bit-L far exceeds the accuracy of 2 bit Efficient-B0 @ QKD ( Kim et al. , 2019 ) by a large 14 % margin using 30 % less computation budget . Compared with the quantization-aware NAS method APQ ( Wang et al. , 2020 ) , our OQA4bit-L-MBV2 uses 43.7 % less computation cost while maintaining the same accuracy as APQ-B . To summarize , the contributions of our paper are three-fold : • Our OQA is the first quantization-aware NAS framework to search for the architecture of quantized compact models and deploy their quantized weights without retraining . • We present the bit inheritance mechanism to reduce the bit-width progressively so that the higher bit-width models can guide the search and training of lower bit-width models . • We provide some insights into quantization-friendly architecture design . Our systematical analysis reveals that shallow-fat models are more likely to be quantization-friendly than deep-slim models under low bit-widths . 2 RELATED WORK . Network Architecture Search without retraining Slimmable neural networks ( Yu et al. , 2018 ) first propose to train a model to support multiple width multipliers ( e.g . 4 different global width multipliers for MobileNetV2 ) . OFA ( Cai et al. , 2019 ) and BigNAS ( Yu et al. , 2020 ) push the envelope forward in network architecture search ( NAS ) by introducing diverse architecture space ( stage depth , channel width , kernel size , and input resolution ) . These methods propose to train a single over-parameterized supernet from which we can directly sample or slice different candidate architectures for instant inference and deployment . However , all of the aforementioned methods are tailored towards searching for floating-point compact models . Converting the best floating-point architecture to quantization tends to result in sub-optimum quantized models . In the quantization area , recent papers ( Jin et al. , 2019a ; Yu et al. , 2019 ) propose to train a single model that can support different bit-widths . But they only quantize manually design networks ( e.g . ResNet , MobileNetV2 ) under relatively high bit-widths ( e.g . 4 bit for MobileNetV2 ) , our OQA can search for architectures and produce quantized compact models with lower bit-width ( e.g . 2 bit ) . Quantization-aware Network Architecture Search Recent studies combine network quantization and NAS to automatically search for layer bit-width with given architecture or search for operations with given bit-width . HAQ ( Wang et al. , 2019 ) focuses on searching for different numbers of bits for different layers in a given network structure and shows that some layers , which can be quantized to low bits , are more robust for quantization than others . AutoBNN ( Shen et al. , 2019 ) utilizes the genetic algorithm to search for network channels and BMobi ( Phan et al. , 2020 ) searches for the group number of different convolution layer under a certain 1 bit . SPOS ( Guo et al. , 2019 ) trains a quantized one-shot supernet to search for bit-width and network channels for heavy ResNet ( He et al. , 2016 ) . BATS ( Bulat et al. , 2020 ) devises a binary search space and incorporates it within the DARTS framework ( Liu et al. , 2018a ) . The aforementioned methods concentrate on the quantization of heavy networks , like ResNet ( He et al. , 2016 ) , or replace the depthwise convolution with group convolution . Moreover , they inherit a two-stage search-retrain schema : once the best-quantized architectures have been identified , they need to be retrained for deployment . This procedure significantly increases the computational cost for the search if we have different deployment constraints and hardware bit-widths . Compared with all these methods , our OQA can search for quantized compact models and learn their quantized weights at the same time without additional retraining . Without our bit inheritance mechanism , these approaches also suffer from a significant drop of accuracy when a network is quantized to ultra-low bit-widths like 2 . 3 METHOD . 3.1 OVERVIEW . Our OQA aims to obtain compact quantized models that can be directly sampled from quantized supernet without retraining . As shown in Figure 1 ( c ) , the overall procedure of OQA is as follows : Step 1 , Quantized Supernet Training ( Section 3.3 ) : Train a K-bit supernet by learning the weight parameters and quantization parameters jointly . Step 2 : given a constraint on computational complexity , search for the architecture with the highest quantization performance on the validation dataset . If K = N , the whole process is finished . Step 3 , Bit Inheritance ( Section 3.4 ) : Use the weight and quantization parameters of the K bit supernet to initialize the weight and quantization parameters of the K − 1 bit supernet . Step 4 : K ← K − 1 and Go to step 1 . The starting bit-width K and the ending bit-width N of the bit-inheritance procedure can be arbitrary . In this paper , we focus on quantized compact models under one fixed low bit-width quantization strategy , thus the starting bit-width and ending bit-width is 4 and 2 . 3.2 PRELIMINARIES . Neural Architecture Search without Retraining . Recently , several NAS methods ( Yu et al. , 2018 ; Cai et al. , 2019 ; Yu et al. , 2020 ) are proposed to directly deploy subnets from a well-trained supernet without retraining . Specifically , a supernet with the largest possible depth ( number of blocks ) , width ( number of channels ) , kernel size , and input resolution is trained . Then the subnet with top accuracy is selected as the searched network among the set of subnets satisfying a given computational complexity requirement . A subnet is obtained from parts of the supernet with depth , width , and kernel size smaller than the supernet . The subnet uses the well-trained parameters of the supernet for direct deployment without further retraining . Quantization Neural Network Learning . To enable the training of quantized supernets , we choose a learnable quantization function following the recent state-of-the-art quantization method LSQ ( Esser et al. , 2019 ) . In the forward pass , the quantization function turns the floating-point weights and activation into integers under the given bit-width . Given the bit-width K , the activation is quantized into unsigned integers in the range of [ 0 , 2K − 1 ] and weights are quantized into signed integers in the range of [ −2K−1 , 2K−1 − 1 ] . Given floating-point weights or activation v , and learnable scale s , the quantization function Q and its corresponding approximate gradient using the straight-through estimator ( Bengio et al. , 2013 ) is defined as follows : Quantization function : vq = Q ( v , s ) = bclip ( v |s| , Qmin , Qmax ) e × |s| , Approximate gradient : ∂Q ( v ) ∂v ≈ I ( v |s| , Qmin , Qmax ) , ( 1 ) where all operations for v are element-wise operations , clip ( z , r1 , r2 ) returns z with values below r1 set to r1 and values above r2 set to r2 , bze rounds z to the nearest integer , Qmin and Qmax are , respectively minimum and maximum integers for the given bit-width K , I ( v|s| , Qmin , Qmax ) means the gradient of v in the range of ( Qmin × |s| , Qmax × |s| ) is approximated by 1 , otherwise 0 . |s| returns the absolute value of s , which ensures that the semantics of scale s is only interval , without inverting the signs of weights or activation . The scale s is learned by back-propagation and initialized as 2√ Qmax × ¯|v| , where ¯|v| denotes the mean of |v| .

Summary: This paper presents a new method to search for quantized neural networks. This method is different from others that it results in quantized weights which can be deployed without post-process such as fine-tuning. Proposed method first trains a 4-bit quantized supernet, and search for the best performance sub-net using the validation dataset. Then, the method initialize the 3-bit supernet using the 4-bit supernet, and trains 3-bit supernet using the knowledge distilation method. Proposed method iterates the initialization, training, and search process until the goal bit resolution is achieved.