Once Quantized for All: Progressively Searching for Quantized Compact Models
Authors: Mingzhu Shen
Feng Liang
Chuming Li
Chen Lin
Ming Sun
Junjie Yan
Wanli Ouyang
Venue:  (2021)
Keywords: quantized neural networks
network architecture search
compact models

Abstract: Automatic search of Quantized Neural Networks (QNN) has attracted a lot of attention. However, the existing quantization-aware Neural Architecture Search (NAS) approaches inherit a two-stage search-retrain schema, which is not only time-consuming but also adversely affected by the unreliable ranking of architectures during the search. To avoid the undesirable effect of the search-retrain schema, we present Once Quantized for All (OQA), a novel framework that searches for quantized compact models and deploys their quantized weights at the same time without additional post-process. While supporting a huge architecture search space, our OQA can produce a series of quantized compact models under ultra-low bit-widths(e.g. 4/3/2 bit). A progressive bit inheritance procedure is introduced to support ultra-low bit-width. Our searched model family, OQANets, achieves a new state-of-the-art (SOTA) on quantized compact models compared with various quantization methods and bit-widths. In particular, OQA2bit-L achieves 64.0\% ImageNet Top-1 accuracy, outperforming its 2 bit counterpart EfficientNet-B0@QKD by a large margin of 14\%  using 30\% less computation cost. 

Under review as a conference paper at ICLR 2021
ONCE QUANTIZED FOR ALL: PROGRESSIVELY SEARCHING FOR QUANTIZED COMPACT MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Automatic search of Quantized Neural Networks (QNN) has attracted lot of attention. However, the existing quantization-awan Neural Architecture Search (NAS) approaches inherit two-stage search-retrain schema, which not only time-consuming but also adversely affected by the unreliable ranking of archi- tectures during the search. To avoid the undesirable effect of the search-retrain schema, we present Once Quantized for All (OQA), a novel framework that searches for quantized compact models and deploys their quantized weights at the same time without additional post-process. While supporting a huge architec- ture search space, our OQA can produce a series quantized compact models under ultra-low bit-widths(e.g. 4/3/2 bit). A progressive bit inheritance proce- dure is introduced to support ultra-low bit-width. Our searched model family, state-of-the-art (SOTA) quantized compact mod- els compared with various quantization methods and bit-widths. In particular, OQA2bit-L achieves 64.0% ImageNet Top-1 accuracy, outperforming its 2 bit counterpart EfficientNet-BOO @KD by a large margin of 14% using 30% less computation cost.
OQANets, achieves
INTRODUCTION
Compact architecture design (Sandler et al. 2018 Ma et al. 2018 and network quantization meth- ods (Choi et al. 2018 Kim et al.J 2019 Esser et al. 2019) are two promising research directions to deploy deep neural networks mobile devices. Network quantization aims at reducing the number of bits for representing network parameters and features. On the other hand, Neural Architecture Search(NAS) (Howard et al., 2019: Cai al. 2019 Yu et al., 2020) is proposed automatically search for compact architectures, which avoids expert efforts and design trials. In this work, we explore the ability of NAS in finding quantized compact models and thus enjoy merits from two sides. Traditional combination of NAS and quantization methods could either be classified to NAS- then- Quantizae or Quantization-awee aware NAS shown in Figure
Conventional quantization methods merely compress the off-the-shelf networks. regardless of whether is searched (EfficientNet (Tan Le (019)) or handcrafted (MobileNetV2 (Sandler al.) 2018)). These methods correspond to NAS-then- -hS-then-Quantize approach as shown in Figure 1 a).. How- ever, not optimal because the accuracy rank among the searched floating-pon point models would change after they quantized. Thus, this traditional routine may fail to get good quantized model. Directly search with quantized models' performance seems to be a solution.
Existing quantization-awwe Guo et al. 2019: Waan et al. (020) utilize two-stage search-retrain schema as shown Fig- ure (b. Specifically, they first search for one architecture under one bit-width setting and then retrain the model under the given bit-width. This two-stage procedure undesirably increases the search and retrain cost if we have multiple deployment constraints and hardware bit-widths. Fur- thermore, due to the instability brought by quantization-awar training, simply combining quanti- zation and NAS results in unreliable ranking (Li al. 2019a Guo al.] 2019) and sub-optimal
NAS methods (Wang et al. 2019: Shen et al. 2019 Bulat et al. 2020
'One wit-w setting refers to specific bit-width for each layer, where different layers could have different bit-widths.

---

Under review as a conference paper at ICLR 2021
Legend
Floaling Point
Floating Paint Search Space
K Bits
Search
NAS Method
Retrain
Retrain model into K, bits
Search Space
Retrain
No Retrain
Deploy
Floating
Search
Kg Retrain
Kg Bits
Search
Many Archs Directyy Deploy
Point
Interilance
(a). NAS-then-uannize
Kg Bits
Search
K Bits
Search
Kg Retrain
One Arch Deploy
Bit Inherilance
K, Bits
Search
K₂ Bits
Search
K₂ Retrain
(b). Traditional Quantization-awar NAS
(c). Once Quantized for All(Ours)
The overall frameworks of existing works on combining quantization and NAS and our method. (a) denotes directly converting the best searched foating-poinn architecture to quantization. (b) first adopts quantization aware search algorithm find single architecture, then retrain the quantizaed weights and activation. Our OQA (c) can search for many quantized compact models under various bit-widths and deploy their quantized weights directly.
Figure 1:
quantized models (Bulat et al. 2020). Moreover, when the quantization bit-width is lower than 3, the traditional training process highly unstable and introduces very large accuracy degradation.
To alleviate the aforementioned problems, we present Once Quantized for All (OQA), novel frame- work that: 1) searches for quantized network architectures and deploys their quantized weights im- mediately without retraining, 2) progressively produces series of quantized models under ultra-low bits(e.g. 4/3/2 bit). Our approach leverages the recent NAS approaches which do not require retrain- ing (Yu & Huang 2019 Cai et al. 2019 Yu et al. 2020). We adopt the search for kernel size, depth, width, and resolution in our search space. To provide a better initialization and transfer the knowl- edge of higher bit-width QNN to the lower bit-width QNN, propose bit inheritance mechanism, which reduces the bit-width progressively to enable searching for QNN under different quantiza- tion bittw.. Benefiting from the retraining property and large search space under different bit-widths, we can evaluate the effect of network factors.
Extensive experiments show the effectiveness of approach. Our searched quantized model fam- ily. OQANets, achieves state-of-the-art (SOTA) results on the ImageNet dataset under 4/3/2 bit- widths. In particular, et al.) 2019)) by large 14% margin using 30% less computation budget. Compared with the quantization-awwe aware NAS method APQ (Wang al. 2020), our DQA4bit-LMMBV2 uses 43.7% less computation cost while maintaining the same accuracy as APQ-B.
OQA2bit-L. far exceeds the accuracy of bit Efficient-BO@ QKD (Kim)
To summarize, the contributions of our paper are three-fold:
Our OQA is the first quantization-awar NAS framework search for the architecture of quantized compact models and deploy their quantized weights without retraining.
We present the bit inheritance mechanism to reduce the bit-width progressively so that higher bit-width models can guide the search and training of lower bit-width models.
We provide some insights into quantization-friendd architecture design. Our systematical analysis reveals that shallow-fat models more likely to be quantization-frinddy than deep-slim models under low bit-widths.
RELATED WORK
Network Architecture Search without retraining Slimmable neural networks YYu et al. 2018) first propose to train model to support multiple width multipliers(e.g. 4 different global width
2

---

Under review as a conference paper at ICLR 2021
multipliers for MobileNetV2). OFA (Cai al. 2019) and BigNAS (Yu et al. 2020) push the enve- lope forward in network architecture search (NAS) by introducing diverse architecture space (stage depth, channel width, kernel size, and input resolution). These methods propose to train a single over-parameterized supernet from which we can directly sample or slice different candidate archi- tectures for instant inference and deployment. However, all of the aforementioned methods tailored towards searching for floating-point compact models. Converting the best floating-point architecture to quantization tends to result in sub-optimum quantized models. In the quantization area, recent papers (Jin et al. [019a] Yu et al. (2019) propose to train single model that can support different bit-widths. But they only quantize manually design networks(e.g. ResNet, MobileNetV2) under relatively high bit-widths (e.g. 4 bit for MobileNetV2), our OQA can search for architectures and produce quantized compact models with lower bit-width 2 bit).
Quantization-aware Network Architecture Search Recent studies combine network quantiza- tion and NAS to automatically search for layer bit-width with given architecture or search for oper- ations with given bit-width. HAQ [Wang et al.) 2019) focuses on searching for different numbers of bits for different layers in given network structure and shows that some layers, which can be quan- tized to low bits, are more robust for quantization than others. AutoBNN (Shen et al.) (019) utilizes the genetic algorithm to search for network channels and BMobi (Phan et al. 2020) searches for the group number of different convolution layer under a certain bit. SPOS (Guo et al. (019) trains a quantized one-shot supernet to search for bit-width and network channels for heavy ResNet (He) et al. 20. BATS (Bulat et al.] (020)) devises binary search space and incorporates it within the DARTS framework (Liu et al. 2018a). The aforementioned methods concentrate on the quantization of heavy networks, like ResNet (He et al. 2016), or replace the depthwise convolution with group convolution. Moreover, they inherit a two-stage search-retrain schema: once the best-quantized architectures have been identified, they need to be retrained for deployment. This procedure signif- icantly increases the computational cost for the search if we have different deployment constraints and hardware bit-widths. Compared with all these methods, our OQA can search for quantized compact models and learn their quantized weights at the same time without additional retraining. Without our bit inheritance mechanism, these approaches also suffer from a significant drop of ac- curacy when network is quantized to ultra-low bit- widths like 2.
METHOD
3.1 OVERVIEW
Our OQA aims to obtain compact quantized models that can be directly sampled from quantized supernet without retraining. As shown in Figure [ c). the overall procedure of OQA is as fol- lows: Step 1, Quantized Supernet Training (Section 3.3 Train a K'bit supernet by learning the weight parameters and quantization parameters jointly. Step 2: given a constraint computational complexity, search for the architecture with the highest quantization performance on the validation dataset. If K .N, the whole process is finished. Step 3, Bit Inheritance (Section 3.4): Use the weight and quantization parameters of the K bit supernet to initialize the weight and quantization parameters of the K bit supernet. Step K + K and Go to step 1.
The starting bit-width K' and the ending bit-width N of the bit-inheritance procedure can be arbi- trary. In this paper, we focus on quantized compact models under one fixed low bit-width quantiza- tion strategy, thus the starting bit-width and ending bit-width is 4 and 2.
3.2 PRELIMINARIES
Neural Architecture Search without Retraining. Recently, several NAS methods (Yu et al, 2018 Cai et al.] 2019: Yu et al. 2020) are proposed to directly deploy subnets from a well-trained supernet without retraining. Specifically, a supernet with the largest possible depth (number of blocks), width (number of channels), kernel size, and input resolution is trained. Then the subnet with top accuracy is selected as the searched network among the set of subnets satisfying a given computational complexity requirement. A subnet is obtained from parts of the supernet with depth, width, and kernel size smaller than the supernet. The subnet uses the well-trained parameters of the supernet for direct deployment without further retraining.
3

---

Under review as a conference paper at ICLR 2021
Quantization Neural Network Learning. To enable the training of quantized supernets, choose a learnable quantization function following the recent state-of-the-art quantization method LSQ (Esser et al.) 2019). In the forward pass, the quantization function turns the floating-pone weights and activation into integers under the given bit-width. Given the bit-width K, the activa- tion is quantized into unsigned integers in the range of [0,2k 1] and weights are quantized into 1]. Given floating-pont poontt weights or activation v, and learnable scale 8, the quantization function Q and its corresponding approximate gradient using the duraight-through estimator (Bengio et al. 2013) is defined as follows:
signed integers in the range of [-2R-1, 2
Quantization function: v= = -[clip( V [s]
Qmins Qmax)) x |8],
aQ(v)
Approximate gradient:
R II Qmin Qmax),
dv
where all operations for are element-wise operations, clip(z, rl, r2) returns with values below rl set to r1 and values above r2 set to r2, [=] rounds z to the nearest integer. Qmin and Qmaz are, respectively minimum and maximum integers for the given bit-width K, I( [1] Quine Qmaz) means the gradient of the range of (Qmin x |s], Qmaz [s)) is approximated by 1, otherwise 0. |s] returns the absolute value of s, which ensures that the semantics of scale is only interval, without inverting the signs of weights or activation. The scale is learned by back- pack-propagation and [v], where |v| denotes the mean of [v].
initialized as Viqaz
3.3 PRANTIZATION-WAARE NAS WITHOUT RETRAINING
Existing problems of quantization-aware NAS. Existing weight-sharing based quantization- aware NAS methods suffer from more unreliable order preserving (Guo et al. 2019] Bulat et al. 2020), as the quantization function introduces more instability on the learned weights. In our per- spective, combining non-retrain NAS methods with quantization to avoid this problem can be natural solution.
Search space and quantized supernet. Our search space is based on MobileNetV2 (Sandler et al,, 2018) and MobileNerV3 (Howard et al. 2019), which has the flexible input resolution, filter kernel size, depth (number of blocks in each stage), and width (number of channels). Our search space consists of multiple stages. Each stage stacks several inverted residual blocks. Further details about search space can be found in Appendix A.. A.6
Unlike the floating-pont -point supernet training (Cai al. 2019 Yu et al. 2020), utilize the quan- tization function Eq. [ to discretize the weights and activation values for the quantized supernet training. Meanwhile, the Roating-point weights need to be retained for reducing quantization loss the training stage of each bit-width. For a convolution layer with input activation a and weight w, define the corresponding learnable scales of activation and weight as Sa and 8w. The forward pass for a quantized convolution layer is defined as follows:
w Q(w, sw);
a Q(a, 8a),
y w * 2?,
where Q(--) is the learnable quantization function defined in Eq. is the convolution operation and y is the output of this layer.
Subnet sampling. During the supernet training, different subnets are sampled and trained in each iteration. non-retrain NAS methods, a subnet has a smaller scale than the supernet in resolution, width, depth, and kernel size, which can be obtained by cropping the corresponding part from the supernet. our settings, a stage with blocks a subnet inherits the weights from the first blocks in the same stage of the supernet. A depthwise convolution layer in subnet with width and kemnl
4

---

Under review as a conference paper at ICLR 2021
size & are cropped from the central is * ls region of the first kernlss in the supernet's corresponding convolution layer. The imput image of each subnet is resized to its resolution r.
Our subnet sampling strategy combines the superee training pipeline proposed in (Cai al. 2019 Yu et al. 2020), and it has two stages. First, we only sample the biggest quantized subnet until converges in (Cai et al. 2019). Afterwards, sample subnets which means one iteration, the biggest subnet and the smallest subnet, and tWo random sampled subnets are sampled. Further details can be found in the Appendix AA.2
the sandwich rules (Yu & Huang 2019 to
Architecture search of quantized supernet. We directly evaluate the sampled subnets from the supernet without further retraining. It's worth mentioning that we use the predictive accuracy on 10K validation images sampled from trainset to measure the subnets in the search procedure. Fur- thermore, we exploit a coarse-to-fine architecture selection procedure, similar to Yu et al. (0200) We first randomly sample 10K candidate architectures from the supernet with the FLOPs of the corresponding floating-pos point models ranging from 50M to 300M (2K in every 50M interval). Af- ter obtaining the good skeletons (input resolution, depth, width) in the pareto front of the first 10K models, we randomly perturb the kernel sizes to further search for better architectures.
3.4 QUANTIZATION-AWANT NAS WITH BIT INHERITANCE
The problem of quantization-aware NAS with ultra-low bit-widths. When the quantization bit lower than 3 bit, the traditional quantization-ane aware training (QAT) (Kim etal 2019 Bhalgat al. 2020) process is highly unstable and introduces very large accuracy degradation for the challenging case of the 2 bit model. Using the approach introduced in Section 3.3 we obtain the quantized supernet with the highest K = 4 bit. To further obtain the quantized supernets of lower bit- widths (e.g. K - 1, K- - 2), we can use the QAT to directly train quantized supernets for each bit-width. As shown in Table [ the biggest architecture in our search space suffers a 19.1 % accuracy drop between 4 bit and 2 bit when using the QAT. Besides, QAT requires much more computational cost.
Inheritance from the high bit to low bit. We pro- pose bit inheritance procedure to compensate for both the disadvantages. First, we use the K bit su- pernet to provide good initialization for the weights of K 1 bit-width supernet. In the initialization for K K-1 bit supernet, we first inherit the weight param- eters and scale parameters from K bit supernet. Then the scale parameters are doubled because they map the floating-poit values to the integer range of K bit, which half the range of K bit. Finally, to compen- sate for the statistics shift of each layer's output caused by quantization error, we forward the model to recal- culate the mean and variance of the BatchNorm lay-
Table I: The accuracy of the biggest model in quantization-awas training(QATT) and progressive bit inheritance from high bit to low bit. Start denotes the accuracy with one epoch training, and end denotes the ac- curacy at the end of training.
4/4
3/3
2/2
QAT Start
48.1%
23.2%
0.8%
QAT-End Bitlnheritance-Sin
75.1%
72.1%
56%
71.7%
49.9%
BitLnheritance-Bn End
72.7%
63.9%
ers (Yu & Huang 2019) with randomly sampled 4096 training images. During training, use the K and K 1 bit supernets as teacher and student, and train them in a knowledge distillation way (Hinton et al. 2015) to further reduce the quantization error between the K and K bit parameters.
The benefit of bit inheritance. In the Appendix A. we show that the loss of weights of the K bit network is bounded (close to that of the K network) bit inheritance is used, where the parameters of the K 1 bit network inherit from the parameters of the K bit network. Therefore, bit inheritance help to guarantee the 2 bit network to be close to the bit network in training loss. In comparison, existing methods like QAT start from a floating-poin model which is far away from the 2 bit model and cause unstable training. The experimental results in Table [ also validate the ef- fectiveness of our design. The row named Bitlnheritance-SS Start means with only one epoch training, the initial accuracy is good enough in 3 bit. After finetuning with fewer epochs, the bit iit-inberitance achieves higher accuracy performance, especially for 2 bit.
5

---

Under review as a conference paper at ICLR 2021
75
63
62-
Accuracy
60
Topl. Accuracy
58
60
*
0QA4bi1@25
OFA@25
52
50
Pareto 2bit
OF Ag25 3bit
Pareto@FF
70
71
73
74
75
76
77
FLOPS(M)
Top1 Accuracy
(a)
(b)
Comparison of the parato models of Pas-then-Quantiie and OQA. (a) OFA FP supernet is used for NAS and LSQ is used as the quantization method. @25 denotes finetuning for 25 epoch. (b) The accuracy of Pareto 2bit/FP is obtained in the correponding 2bir/FP supernet.
Figure 2:
4 EXPERIMENTAL ANALYSIS
4.1 EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
We evaluate our method on the ImageNet dataset (Deng et al. 2009). If not specified, we follow standard practice for quantized models (Kim et al. 2019 Gong et al. 2019) on quantizing the weights and activation for all convolution layers except the first convolution layer, last linear layer, and the convolution layers in the SE modules(Hu et al. 2018). To fairly compare with quantized compact models, the FLOPs are used and defined follows. Denote the FLOPs of the FP layer by a, following (/hou et al. 2016 Li et al. 2019b Phan et al. 2020 Bulat et al. 2020), the FLOPS of the FP layer
and the FLOPs of bit weight and bit activation quantized layer is BitOPs mn x a following (Wu et al. 2018). Unless otherwise noted, all results are sampled from MBV3 search space denoted OQA, OQA-MBV2 represents the MBV2 search space. Further training details can be found in Appendix A.2
a,
4.2 HAS-THEN--UANTUE
OQA
We denote pareto models as those models on the pareto front of the cost/accuracy trade-off curve. In Figure aa), we quantize the pareto models of the OFA floating-point supernet, which corresponds to the T'AStthen-uuntii procedure in Figure a)). We compare it with the pareto models of our OQA that are directly obtained from the quantized supernet. In the comparison under 3 bit, the pareto curve of our OQA is far above that of the NAS-then-Quantize
In Figure b), we sample 10k subnets from the search space, and we validate these architectures from the FP supernet and 2 bit supernet. The pareto front of the subnets denoted Parato @ FP are selected with the FP accuracy and Pareto 2bit are selected with the 2 bit accuracy. With the accuracy of the floating-point models, the accuracy of the model from the 2 bit pareto is higher than the model from the FP pareto. If our target is to search architectures for the quantized models, 2igure shows that searching from the quantized supernet as our OQA did is better than searching from FP supernet and then quantize. The advantage is more evident for lower bit-widths. Further details can found in Appendix A.3
4.3 EXISTING QUANTIZATION- NAS OR OQA
In Table 2 we compare our OQANet model family with several quantization-awan NAS meth- ods, named SPOS (Guo al. 2019) BMobi (Phan al. 2020), BATS (Bulat al. 2020 and APQ (Wang et al. 2020).
With the non-retraining quantized supernets, OQANet model family shows great advantages over traditional weight-sharing methods corresponding to the paradigm of Figure [bb. While SPOS (Guo et al. 2019) focuses the search of network channels and bit-width of heavy

---

Under review as a conference paper at ICLR 2021
Table Quantization-awee NAS performance under different bit-widths on ImageNet dataset, Bir
( (W/A) denotes the bit-width for both weights and activation. The number of bit for different layers
different for SPOS (Guo et al. 2019) with bit-width in the range of {1, , 3. 4] and APQ (Wang
et al.
2020) with bit-width in the range of (4. 6, 8). BMobi (Phan et al. 2020), BATS (Bulat et al.
2020
and OQA use the same bit-width for different layers.
Methods
Models
Bit (W AA)
FLOPs (M)
BiOOss (G)
Top-I Acc.(%)
SPOS
ResNet-34
(1, 3, 4]
337
13.11
71.5
SPOS
ResNet- 18
(1. 3,
221
6.21
66.4
BATS
2x
155
9.92
66.1
BATS
1x
98.5
6.30
60.4
BMobi
M2
62
3.97
59.3
BMobi
M3
33
2.11
51.1
OQA
OQA3biL-I
48
3.07
71.3
OQA
QQA3bit-M
30
1.92
68.3
OQA
OQA2bit-M
19
1,21
61.7
APQ
APQ-B
14, 6,
258
16.5
74.1
APQ
APQ-A
6,
206
13.2
72.1
OQA
0QA4bit--MMBV
145
9.28
74.1
OQA
OQA4bi-M-MBY.
4
107
6.85
72.4
Er1BO
EffBO
62
72
MobV2
68
Topl Accuracy
MobV3Mobv2
MobVi
MobV3
Mc5V2
MobV2
MoBV2
Emrpo
*- OQW@g2
60
3 bit quant
2 bit quant.
MobV3 MobV2
64
* 00A@25
46
DQA@25
LSc*
OQA
OQA
QKD
LSQ"
42
LS0*
OKD
OKD
60
52
38
25
50
125 150 (LOPs(M)
100
50 FLOPs(M)
100
FLOPs(M)
Figure 3: Comparison with the state-of-the-ee quantization methods (LSQ, QKD, SAT) in various netwrt((MobilNeee//V3. EfficientNet-BO on the ImageNet dataset.
ResNet (He et al. 2016). we focus on the search of compact models with fixed bit width and achieve better results with fewer FLOPs. BMobi (Phan et al.) 2020) and BATS (Bulat et al.) (2020) did not provide the results for bit, 3 bit or 4 bit. Therefore, we would like not to directly compare our ap- proach with BMobi and BATS, because the results obtained from different bit-widths. However, only the PLOPs-accuracy trade-off is concerned, our OQA with a higher bit-width can better solution. APQ corresponds to the NAS-then- -then-Quantize paradigm Figure Ia). It first searches for floating-poin network architecture in the FP supernet, then trains quantization aware predictor to predict the searched quantized architecture. The transfer learning from FP predictor quantized predictor brings the proxy problem and it also needs retraining. Our OQA4bit-L-MBB uses 43.7% less computation cost while maintaining the same accuracy as APQ-B.
4.4 FURTHER COMPARISON WITH TAS-THEN-QUANNE METHODS
We compare with several strong quantization methods including LSQ Esser et al. 2019 I.SQ+ (Bhalgat et al. 2020). APOT (L. et al., 2019b), QKD(Kim et al. 2019). SAT (Jin et al.) 2019b), and LSQ which is the LSQ implemented by baselines. The result of 4 bit ResNet- 18@ LSQ* validates that our implementation is comparable.
different models to construct strong
Our OQA benefits from joint quantization and network architecture search, well as the bit in- heritance for lower bits. As shown in Table 3, our OQANets outperforms multiple quantization methods models like MobileNetV2 (Sandler et al. 20188), EfficientNet-BD (Tam & Le 2019) and MbV3 (Howard et al., 2019) under all bit-widths we implements. bit: OQA4bit-L has 1% ac- curacy gain higher than Efficient-BOl QKD. OQA4bit-M outperforms ResNer- 18 @1LS00 with 10%

---

Under review as a conference paper at ICLR 2021
Table ImageNet performance under 4, 3, bit. OQA4bit-M and OQA4bit-L denote medium and large model size in the 4 bit OQANets family respectively. @25 means we take weights from the supernet and finetune for 25 epochs. W/A denotes the bit-width for both weights and activation.
Models
Method QKD
Bit (W AA)
FLOPs (M)
Top-1 Acc.(%)
Efficient-BD BO
106
73.1
4QA4bit-L.
@QA@25
73
74.1
ResiNef-13 18
1.3 /
542 /542
71.7 T.
MobileNerV2
LSQ SAT
85
71.3/71.1
MbV3-L (1.0x)
LSQ*
71.7
0QA4bit-M
@QA@2
47
72.3
ResNet-118
LSQ/ APOT
357 / 298
70.6769.9
Efficient-B0
QKD
65
69.2
OQA3bit-L
OQA@2
71.3
MobileNetV2
LSQ*7 QKD
53//5
68.2762.6
MbV3-L 1.0x
LSQ*
38
67.5
OQA3bit-M
OQA@25
30
68.3
Efficient-BO
QKD / LSQ+
36
50.0 / 49.1
MobileNetV2
LSQ" QKD
30
55.7 / 45.7
OQAZbit-L
OQA@25
25
64.0
OQA2bit-S
OQA@25
14
57.7
-2
-14
Topl Accuracy Drop
Top1 Accuracy Drop
-4
-16
-6
-18
-8-
bit quant.
-20
bit quant. conrtt-shatiow-aat
shallow-fat pareto-deep-Sim shallow-fat deep-slim
-10
-22
shallow-fat
pareto-deep-simm
deep-Slm
-24
20
30
40 FLOPs(M)
50
60
10
15
20
FLOPs(M)
Figure 4: The quantization accuracy drop of shallow-fat and deep-slim subnets which are sampled from FP 4/2 bit supernets the ImageNet dataset.
of its FLOPs. bit: Our OQA3bit-L also match the accuracy of bit ResNet-18@@Ly with 13% FLOPs and 3 bit Efficient-BO@OO QKD with 74% FLOPs. bit: Our OQA2bit-L requires less FLOPS but achieves significantly higher Top-1 accuracy (64.0%) when compared with EfficientNet- @ QKD (50.0%) ) and MobileNetV2@Ls. LSQQ (55.7%). The results verify that the joint design of quantization and NAS results in more quantization-friett friendll compact models. We also show more searched models in the Figure and our OQANets significantly outperform other quantization meth- ods.
4.5 SHALLOW-FAT OR DEEP-SLIM
With the different quantized supernet obtained by the bit inheritance process, we analyze the fol- lowing factors network: depth(D), kernel size(K), expand width(E) and bits(B). According the intuitive shape, we divide the models into two groups: 1) shallow-fat: D, K and E. are sampled from (2,3), (5. 7) and (4, 6} respectively, 2) deep-slim: D, K, and F. are sampled from (3, 4), (3,5) and {3, 4} respectively.
Following the rules above, we randomly generate 1.5K architectures and obtain the accuracy of the floating-point and quantized models from the corresponding supernet. The input resolution is also randomly sampled. For a certain model, we use the accuracy drop from floating-poin models to quantized models measure the tolerence of quantization. As shown in Figure 4 the shallow-fat models are more quantization-friene
than deep-slim models. When the bit and 2 situations compared, the trend becomes more obvious when the quantization bits lower.
8

---

Under review as a conference paper at ICLR 2021
Table 4: Top-1 Accuracy on ImageNet dataset under different settings. FLOPs calculated with the corresponding floating-pon models. Models named Nas-then-Quantee is chosen from the pareto front with the floating-por point performance while the other is chosen with 2 bit performance. QAT@ @ 150 means the existing routine of quantization-awar training with 150 epochs.
Top-1 Acc.(%)
Top-1 Acc.(%)
Top-1 Acc.
Top-1 Acc.(%) QAT@500
Models
FLOPs
FP Supernet
BI 2 bit Supernet
QAT 150
Pes-then--uunting Joint Design
144
73.6%
54.4%
43.5% 28.1%
142
73.4%
56.1%
33.2%
47.2%
Table 5: The comparison of the search cost and retrain cost with existing quantiztion-awant awarre nas methods. N denotes the number of models to be deployed. The total cost is calculated with N 40.
Methods
SPOS
BMobi
BATS
APQ
OQA
scarch cost (GPU hours)
288
24NN
29/V
6NN
2400
0.5.V
1200
0.5N
retrain cost (GPU hours)
240N
256N
75N
30NV
total cost (GPU hours)
10.8k
11.4k
3.2k
3.6k
1.2k
4.6 ABLATION STUDY
The effectiveness of joint design. In Table the top-1 accuracy on ImageNet under different settings is listed. NAS-then-Quantive means the model is selected from the pareto front with the mooaiing-piin accuracy, and Joint Design is with the 2 bit accuracy. With similar FLOPs, similar floating-poin accuracy, Joint Design surpass the accuracy of Nas-then- (raining(QAT) with 150 epochs and the joint
with 1.7% in 2 bit
accuracy. We also perform the quantization-awan design results in over 5% accuracy improvement, which verifies the effectiveness of Joint Design in finding quantization -fiendlyy architectures.
The effectiveness of bit inheritance. In Table performs quantization-awee tra)ning(QAT)) with 150 epochs and 500 epochs. In QAT, pretrained quantized floating-pon models are used to provide better initialization point, while our bit inher- itance proposes to initialize models with one bit-width higher. Although 2 bit models benefit from QAT with more epochs, the bit inheritance (BI) supernet still improves the two models' accuracy by a large margin, even compared to training with pretrained floating-poii weights by 500 epochs. The huge accuracy improvement verifies that bit inheritance a better practice compared with the exist- ing routine of quantization-awae aware training because it alleviates the problem that quantized compact models with low bit- are highly unstable to train.
further compare these two models which
The efficiency of OQA framework. In Table 5 we compare the search cost and the retrain cost of OQA with existing methods. The search cost is defined the time cost of the supernet training and the search process to get the final searched models under latency targets. The retrain cost is defined as the training cost get the final accuracy of the searched architecture. When N is larger than 5, the retrain cost of SPOS (Guo et al. 2019) and BMobi (Phan et al. 2020) already surpasses the total cost of OQA. And We reduce at least half of the total cost compared with APQ (Wang et al., (020). APQ needs to train a once-for-all doating-poim supernet and sample thousands of FP subnets to perform quantization-awar training, and thus requires the transfer learning from the floating-point predictor to quantization predictor. Our OQA only needs to train one once-for-all quantized supernets and support huge search space with over 1019 subnets that can be directly sampled from supernet without retraining, Thus, the average computational cost is relatively low.
CONCLUSION
In this paper, we present Once Quantized for All (OQA), a novel framework that deploys the searched quantized models without additional retraining and solves the problem of large accuracy degradation with bit inheritance mechanism under ultra-low bit-widths. With our proposed meth- ods, search for the OQANet model family which far exceeds the existing quantization-awe NAS and quantization methods. Our results reveal the potential of quantized compact models under ultra-low bit-width.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Yash Bhalgat, Jinwon Lee. Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Im- proving low-bit quantization through learnable offsets and better initialization. arXiv preprint arXiv:2004.08 099520004.977,, 09576, 2020.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks. arXiv preprint arXiv:1909.
2019.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos, Bats: Binary architecture search. arXiv preprint arXiv:2003.0177 2020.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: pew.::998.099999 2019.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: /805006885 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint 100iv:190.008/555 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 4852 4861, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. 7echun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint 1rXiv:1944.00420 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1003. 2015.
02531v1
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference Computer Vision, pp. 1314-1324, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitttion networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. arXiv preprint arXiv: 19/2.09666, 2019a.
Qing Jin, Linjie Yang, and Zhenyu Liao. Towards efficient training for neural network quantization. arXiv preprint arXiv: 1912. 1922.10222220.. 2019b.
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. arXiv preprint 1rXiv:1911.122 124,, 2019.
Xiang Li, Chen Lin, Chuming Li. Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. arXiv preprint 4rXiv:1910.02543, 2019a.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-twe quantization: A non-uniform dis- cretization for neural networks. arXiv preprint 1909. 2019b.
10

---

Under review as a conference paper at ICLR 2021
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 2018a.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp. 722-737. 2018b.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference Computer Vision (ECCV), pp. 116-131, 2018.
Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Swang-Ting Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution-based searching. arXiv preprint arXiv:2005.06305, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 45200, 2018.
Mingzhu Shen, Kai Han, Chunjing Xu, and Yunhe Wang. Searching for accurate binary neural ar- chitectures. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2078-2087. 2020.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00099. 2018
Haichao Yu, Haoxiang Li, Honghui Shi, Thomas S Huang, and Gang Hua. Any-precision deep neural networks. arXiv preprint 1rXiv:1911.07344 2019.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.
arXiv preprint arXiv:1812.0899 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. arXiv preprint arXiv: 142, 2020.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1666:0/60606666 06100, 2016.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 EFFECTIVENESS OF BIT INHERITANCE
Given convolution layer in the K bit supernet, we denote sk as the scale parameters of this layer, and SK-1 = 2sk as the doubled scale for the K - bit superntt. We use w and Nw to denote the weights of this layer which inherited from K to K - 1. Next, we show that the L1 distance 1) is bounded by Nw Sx. It means the initialized Q(w, SK-1) 1) has
of Q(w.SH sx) and Q(w, bounded distance with the well-trained Q(w, 8x).
For each Wi, we have:
(wI, [clip(
- 1k-1, - 1)] x |sk|
8k
((wi))K-1) = |clip(
W;
2 - 2,22 1)] x |sK-1l (3)
SK-1
= c
W;
22 10-2,,2-3 2k
1)] |sx]
[2s
Based on this expression, we further get:
[Q(w;, sK) Q(wi; SK sx-1)]=
[[clpp(
Wi
2k-1,2K-11 1)1 c|clip( [2
W:
-2 22 -1)] x |sk].
(4)
For any wi and sk. we have:
|lcip 8k
-2 2k 2--12-1_ -1]1 1-) - 22cclip(
2k 2.-222K-3 22 1)]1 S
(5)
[28
Thus,
[Q(wi,sk) Q(w; sx-1) |
and
(6)
[[Q(w, - Q(w, - N |sx|-
A.2 TRAINING DETAILS
Dataset config: We evaluate our method on the ImageNet dataset. The training dataset is made up of 1.28 million images with resolution 224 x 224 belonging to 1000 classes and the validation set has 50k images. For ImageNet training, we use the typical random resized crop, randomly horizontal flipping and color jitter of (32/255, 0.5, 0) for data augmentation. During evaluation, first determine the active image size 8, and resize the image into [s/0.875] x [s/0.875] and center crop image.
Quantization aware training: We reimplement LSQ (Esser et al. 2019) as our base quantization method. We start from a floating-point model and finetune the model for I50 epochs. The optimizer is SGD with Nesterov momentum 0.9 and weight decay 3e-5, and the label smoothing ratio is 0.1. The initial learning rate 0.04 under the batch of 1024, with the cosine annealing schedule. The dropout rate is O. Except for learning rate and training epochs, we follow this protocol in the OQA procedure.
OQA procedure: Combining the advantages of OFA (Cai al. 2019) and BigNas (Yu et al. 2020). the overall OQA procedure is divided into four steps as follow:
StepO: train the 4 bit biggest models in the search space. It follows the typical quantization- aware training, we use the pooating-poin pre-trained model as initialization and finetuning for 150 epochs. The learning rate is 0.04 with batch-size of 1024.
Stepl: in the supernet training phase, the biggest model obtained in step 0 is used as initialization. The input resolution, kernel size, width, and depth are randomly sampled. This whole process takes
12

---

Under review as a conference paper at ICLR 2021
200 epochs. In one iteration, four models are sampled with the sandwich rule Yu & Huang (2019) which is the biggest subnet and the smallest subnet, and two random sampled subnets. The learning rate is 0.02 with batch size of 1024.
With bit inheritance, the training of the 3/2 bit supernet is simplified. And the training time is reduced.
Step2: In the bit supernet, we use the bit supernet obtained in Architecture shrinking Step1 part2 initialization. We directly random sample input resolution, kernel, width, and depth. four models are sampled, which is the biggest subnet and the smallest subnet, and two random sampled subnets for update. We only 25 epochs and the learning rate is 0.0016.
Step3: In the bit supernet, we use the 3 bit supernet obtained in Step2 as initialization. We also directly random sample resolution, kernel, width, and depth. Four models are sampled, which the biggest subnet and the smallest subnet, and two random sampled subnets for one update. We only use 120 epochs and the learning rate is 0.0256.
OQA subnet finetuning: Our OQA performance can be further improved by finetuning the subnet weights sliced from the OQA supernet as suggested by OFA (Cai et al. 2019). The accuracy of the subnet is already higher than training from scratch. In default, the subnets are finetuned for 25epochs. The initial learning rate is 0.0016 with a batch size 1024, with the cosine annealing schedule.
The knowledge distillation(KD) used in our experiment is the traditional loss(KD) proposed in (Hinton et al.) 2013). The student's 's logits and teacher's 's logits are used to calculating the cross-entropy loss. The temperate is 1 and the kd loss weight is 1.
Knowledge distillation:
13

---

Under review as a conference paper at ICLR 2021
63
62-
62
3bit Top1 Accuracy
2bit Top1 Accuracy
2bit Top1 Accuracy
60-
66
56
58
56
52-
Partto@abit
50
Paretog 2bit
Pareto@
Pareto@FF
Par
Par etogs 3bit
70
71
72
73 74
75
76
77
40
71
72
73
74
75
76
77
63
64
65
66
67
68
69
70
PP Topl Accuracy
Topl Accuracy
3bit
Topl Accuracy
Figure 5: Comparison of the Top-1 validation accuracy ImageNet dataset between the FP pareto, 3 bit pareto, and bit pareto. The pareto is selected from the corresponding supernet and the aceu- racy is also obtained from the supernet.
A.3 NAS-THEN-QUANTIAE OR OQA
In Figure we sample 10k subnets from the search space, and we validate these subnets from the FP supernet, 3bit, and 2 bit supernet. The pareto front of the subnets de- noted as Parato @ FP is selected with the accuracy in the floating-point supernet and Pareto @ 3bit/Pareto @ 2bit is selected with the accuracy in the 3 bit/2 bit supernet. In
Paretog22bit
30-
3brttod3bbit
Pair Pareto@FF
Probability(
20
Figure 5 the first two figure reveals that as the bit de-
creases, the accuracy gain increases with searching di-
rectly in the quantized supernet. We further compare the
10
Pareto @ 3bit and Pareto 2bit, and it shows that with the
same bit accuracy, the accuracy of the model from the 2 bit pareto is higher than the model from the 3 bit pareto, but the accuracy gain is less compared with the FP pareto. To search for bit quantized models, it is best search directly in the 2 bit supernet, and it is better to search in the 3 bit supernet than searching in the FP supernet.
10 13
17
Total Depth
Figure 6: The probability distribution of total depth.
Observed the difference in the accuracy of the pareto ar- chitectures different supernets at the same 2 bit, we are curious whether the accuracy performance is attributed to the structure difference in the pareto ar- chitecture. In Figure we plot the distribution of the total depth the pareto subnets from different supernet. In the depth dimension, the distribution reveals that Pareto @P FP favors deeper models while Pareto 3bit/2bit favors shallow models. And the distribution of depth is closer between Pareto @ 3bit and @areto@2bit.
14

---

Bireal (Liu et al. Under review as a conference paper at ICLR 2021
A.4
THE DETAILS OF QUANTIZATION AWARE NAS.
Table 6: The details of quantization-awwen nas named SPOS (uo et al. 2019 BMobi (Phan al.
2020), BATS (Bulat et al. 2020), APQ (Wang et al. 2020) are given, inc Including network architecture.
search space, bit-width and quantization algorithm PACT (Choi et al. 2018),
2018b), Xnor-net++ (Bulat & Tzimiropoulos, 2019), HAQ (Wang et al. 2019).
LSQ (Esser et al, 2019). Group MobileNet denotes the MobileNet with group convolution in place of depthwise convoluton.
SPOS
BMobi
BATS
APQ
OQA
Quantization
PACT
Bireal
Xnor-nct++
HAQ
LSQ
Algorithm
Network
MobileNetV2.
ResNet
Group MobileNet
Group Darts
MobileNetV2
Architecture
MobileNetV3
Bit Width
(12.3344
[1]
(4.
(2. 3.4) width,
width.
width, bit-width
operation, connection
depth.
depth.
Search Space
group number
kernel size, kernel size,
bit-width
resolution
Retrain
A.5 MOBILENETV2 SEARCH SPACE
Table 7: MBConv refers to inverted residual block which has '1 pointwise k x depthwise- 1 x pointwise' structure without SE module (Hu et al.] 2018). MBConv-SE: is the MBCony block with SE module. Channels means the number of output channels in this stage. Depth means the number of blocks or layers in this stage. Expand ratio refers the expand ratio of input channels which controls the width of the depthwise convolution. Convolution layers in the first and last has no expand ratio. Kernel size refers to the kernel size k of the depthwise convolution.
Stage
Operator
Resolution
Channels
Depth
Expand ratio
Kernel size
Conv
128 x 128 224 x 224
32
3
MBConv
64 112
112
16
2
MBConv
64 x 64 112 x 112
24
2, 3, 4
3, 4, 6
3, 5,
3
MBConv
32 x 32 56 x 56
40
2. 3, 4
3, 4. 6
3, 5.
MBConv
16 x 16 28
80
3,
3, 6
3,
5
MBConv
8 8 8
14
96
2, 3, 4
3, 3,4,6 6
3,
6
MBConv
8x8- 14
192
2, 4
3,4, 6
3,
MBConv
4
320
3,4,6
3,5,7
Conv
4 4-
1280
15

---

Under review as a conference paper at ICLR 2021
A.6 MOBILENETV3 SEARCH SPACE
Table MBConv refers to inverted residual block which has 1 x pointwise' structure without SE module (Hu al. 2018). MBConv-SE is the MBConv block with SE module. Channels means the number of output channels in this stage. Depth means the number of blocks in this stage. Expand ratio refers the expand ratio of input channels which controls the width of the depthwise convolution. Convolution layers in the first and last has no expand ratio. Kernel size refers to the kernel size of the depthwise convolution.
pointwise k x depthwise-
Stage
Operator
Resolution
Channels
Depth
Expand ratio
Kernel size
Conv
128 x 128 224 x 224
MBCCnv
64 x 64 112 x 112
16
3
2
MBConv
x 64 112 112
24
2, 3,4 4
6
3, 5, 7
3
MBConv-SE
32 x 56
4
4, 6
5,
4
MBConv
16 16 28 x 28
80
2, 4
3,4. 6
3, 7
5
MBConv-SE SE
8 14 14
112
2, 4
3, 3,4. 6
3, 7
MBConv-SE
8 x 14
2, 3, 4
6
3. 5,
Conv
4 4- 7 X
960
Conv
1
1280
A.7
ARCHITECTURE VISUALIZATION
Stage1
Stage2
Stage3
Stage4
Stage5
Stage6
220'220
Conv 3x3
MB 1893K
MBE3K3
МВЕ4К7
MB B8
MB 33 K3
MB E8 K5
MBE4KS
MB K5
MBE4KS
MB 18E4
MB ES
MB EB K5
MBE6K
Conv 1x1
Conv
OFA@F
Conv 3x3
MB K3
MB K3
MB E4 K7
MB E4 K3
MB 64 K6
MB E4
MB E4
MB E6
MB E4
MB E4
Cony 1x1
Conv 1x1
224'224
OQA@4b
MB
Conv 3x3
E3
MB E6
MB K6
MB K3
MB K4
MB K3
MB E3
MB E4
MB KS
MB E6K5
Conv 1x1
Conv 1x1
OQA@2b
MB
MB
Figure 7: Architecture visualization of OFANet and our searched OQANets. 'MB K3' indicates "mobile block with expansion ratio 3, kernel size 3x3'. From top to bottom, there are FP OFANet. 4-bit OQANet and 2-bit OQANet. There are under similar computation cost, around 220M FLOPs.
16


Reviews:
[{"review_id": "sHhcDgCMAfh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method to search for quantized neural networks. This method is different from others that it results in quantized weights which can be deployed without post-process such as fine-tuning. Proposed method first trains a 4-bit quantized supernet, and search for the best performance sub-net using the validation dataset. Then, the method initialize the 3-bit supernet using the 4-bit supernet, and trains 3-bit supernet using the knowledge distilation method. Proposed method iterates the initialization, training, and search process until the goal bit resolution is achieved.\n\nI find that the idea of constructing quantized super-nets using Bit Inheritance is interesting and the paper is well written. However, I think more precise description of the training method and additional analysis is required to improve the paper.\n\n1. In section 3.4, you described the K to K-1 supernet inheritance process as \"During training, we use the K and K − 1 bit-width supernets as teacher and student, and train them in a knowledge distillation way to further reduce the quantization error between the K −1 and K bit-width parameters.\". However, the knowledge distillation method is not specified after the statement. Is it similar to QKD? Or is it more of a traditional knowledge distillation approach? More precise description will be helpful.\n\n2. In the introduction, you described that \"... This two-stage procedure will undesirably increase the number of models to be retrained if we have multiple deployment constraints and hardware bit-widths...\". However, there is no analysis on the search cost under such scenarios. Since the proposed method induces more supernet training process compared to NAS-then-Quantize approaches such as APQ, it is unclear whether your method will acheive lower search cost or not. I believe that additional analysis on the benefit of deploying the quantized weights without retraining in the mean of search cost must be given in the paper as one of the main contribution of the paper is that the proposed method allows the deployment without retraining.\n\n3. Minor: In section 3.4, you described that \"... where the parameters of the K − 1 bit network inherit from the parameters of the K − 1 bit network.\". I think the later K - 1 must be K.", "extra_scores": {}, "extra_text": {}}, {"review_id": "rJM2EOtWCJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method to train quantized supernets which can be directly deployed without retraining. The motivation is to have a supernet with a given quantization bit-width which only train once and can be deployed with different architectures (under different FLOPs budget). This paper made a bunch of experiments showing that the proposed once quantized for all method can find DNN architectures which have SOTA performance with low bit-width. The paper also shows that when training lower-bits supernet, it is helpful to use the weights from the trained higher-bits supernet.\n\nPros:\n\n1. This paper targets a very practical problem that quantization is actually required for most resource-constrained devices. Recently, supernets (e.g. OFA, BigNAS) validate that it is possible to directly obtain DNNs with different FLOPs budget from a single big supernet without retraining. This saves lots of training time in the cases that one want to have DNNs with different FLOPs. Combining quantization-aware training and supernet is an effective approach to save training time when we want search different DNNs (with different size/FLOPs) which are quantized with certain bit-width.\n\n2. The authors also show that it is very important to use the pretrained weights (from a trained higher-bits supernet) as initialization when training the quantized supernet. This observation is also meaningful for the cases that low-bits quantization-aware training is hard or unstable.\n\n3. Using the proposed once-quantized-for-all method, the authors get several quantized DNNs which have SOTA results on ImageNet. The authors compared both \"SOTA architectures + quantization aware training\" and recent \"quantization-aware NAS method\". The accuracy / flops of the searched DNNs is better than the compared methods.\n\nCons:\n\n1. It's natural to apply quantization-aware training on supernets when we want a supernet to be quantization-aware. Both quantization-aware training and supernets are ready-to-use techniques and the combination is straightforward. The benefit of bit-inheritance is a good observation, while using pretrained weights as initialization is kind of a common practice in quantization or model compression. At this point, the contribution in terms of the novelty is limited.\n\n2. The proposed method can also outperform methods that can also search layer-wise bit-width (e.g., Table 2), although the method in this paper only uses the same bitwidth for all the layers. It's not clear what is the main factor in this comparison. Are all the methods using the same experiment setup? E.g., quantization algorithm (LSQ or min-max), architecture search spaces. It will be better to have an ablation study to understand which part of the proposed algorithm plays the key role to the better performance.\n\n\nIn general, I think this paper did a great job on the experiments of quantization-aware supernet, but the novelty contribution is slightly under the criteria of ICLR. So my rating is borderline. I hope the authors can give some response to the cons listed above, and I'd like to consider changing my rating if I missed something important.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QeHKiPg6_mD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\tThis paper performs a joint optimisation for DNN models, making the NAS scheme is aware of both the quantisation and architectural search spaces. The paper presented a large range of comparisons to different quantisation strategies and ran a lot of experiments to support their claims. However, the writing quality of this paper is worrying. Also, I am a little worried about the novelty of this paper.\n\nStrength:\n1. There are a lot of experiments with the proposed method, showing a great empirical value for researchers in this field. I consider results shown in Figure 2 and Table 2 very supportive evidence of the effectiveness of the proposed method.\n2. It is nice to see a large scale study (1.5K architectures) on some common properties of network architectures and their interactions with quantisation.\n3. To my knowledge, this paper does present a state-of-the-art number for low-precision ImageNet classification.\n\nWeakness:\n1. The writing quality of this paper is worrying. This is not simply to do with the use of language, but also on the clarity of some matters. I strongly recommend the authors to have a serious polish of their paper, since they do present valuable results and STOA numbers.\n2. To me, the novelty of this paper is limited, it seems like an extension to Once-for-all, and the authors also cited this work. The teacher-student technique is also a published idea. The authors claim this is the first piece of work of NAS without re-training. However, they are iteratively reducing the bit-width K, which implies a large training cost and is somehow equivalent to re-training. The method in the paper looks like a combination of a number of well-known techniques, which might limit the novelty claim in this paper. However, I have to say I am not very troubled with combining a bunch of existing techniques if it show new STOA that is outperforming by a significant margin. This weakness is only minor to me.\n\nMy suggestions & confusions:\n1. It seems like you can boost the performance of quantised networks from a) jointly search for architectures and quantisation and b) teacher-student alike quantisation training with inherited weights. Could you test these two parts in isolation and quantify the contributions of each technique?\n2. Why you quantise activations to unsigned numbers (Page 4)? Don’t you consider activations like leakyrelu in your activation search space? Or you do not search activations at all?\n‘... NAS methods suffer from more unreliable order preserving’, Who are you comparing to in this case? Is it more unreliable compared to RL based NAS?\n3. What is your Flops reported in Table 2? Flops means floating point operations, do you mean bitops? or you somehow scaled flops with respect to bitwidths?\n4. ‘we focus on the efficient models under one fixed low bit-width quantization strategy’ Do you mean the network is uni-precision? So no layer-wise mixed-precision is allowed?\n5. I spotted a number of misused languages, and will strongly recommend authors to check mistakes like:\na) Ambiguity: \n i) ‘with high floating-point performance’: do you mean floating-point models? Or you mean customised floating-point models? Describe floating-point as high is very misleading.\nii) ‘quantize the network with retraining’: I guess I understand what you mean, but you might say “retrain the quantised models” to be less ambiguous. \n\nb) Grammar:\ni). ‘different bit-width’ -> ‘different bit-widths’\nii) ‘quantization supernet’ -> ‘quantized supernet’ and so on.\n\nc) Do not assume readers have prior knowledge:\ni) ‘we use sandwich rules’ -> ‘we use the sandwich rule’ and maybe you should consider explain what it is.\n\nI cannot present all the mistakes here, these are just examples, I would iterate again that I would strongly recommend you to polish the paper since I do like the results you are presenting and think if the code is open-sourced, they will benefit the community.\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9pVunn-T8a-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents One Quantize for all framework. The framework claims to search for the network and the quantization without the need for retraining. \n\nResults are promising although it is not clear to me if the comparisons are fair (different bit size). \nWould also be great to see the resulting architecture and how it is different from other NAS approaches. It is not clear to me why the number of FLOPS is so low compared to related methods. \n\nIn the experiments, I missed the computational cost for training and how it compares to the other approaches (as this paper suggests there is no need for retraining or fine-tuning). \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a method to train quantized supernets which can be directly deployed without retraining. A main concern is that there is limited novelty. The proposed method looks like a combination of well-known techniques. Experimental results are promising. However, it is not clear if the comparisons are fair and if all the methods are using the same setup. It is desirable to have additional analysis and ablation studies. The writing can also be improved.
Decision: Reject