BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
Authors: Haoping Bai
Meng Cao
Ping Huang
Jiulong Shan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Joint Neural Architecture Search and Quantization
Neural Architecture Search
Mixed-Precision Quantization
Model Compression

Abstract: As the applications of deep learning models on edge devices increase at an accelerating pace, fast adaptation to various scenarios with varying resource constraints has become a crucial aspect of model deployment. As a result, model optimization strategies with adaptive configuration are becoming increasingly popular. While single-shot quantized neural architecture search enjoys flexibility in both model architecture and quantization policy, the combined search space comes with many challenges, including instability when training the weight-sharing supernet and difficulty in navigating the exponentially growing search space. Existing methods tend to either limit the architecture search space to a small set of options or limit the quantization policy search space to fixed precision policies. To this end, we propose BatchQuant, a robust quantizer formulation that allows fast and stable training of a compact, single-shot, mixed-precision, weight-sharing supernet. We employ BatchQuant to train a compact supernet (offering over $10^{76}$ quantized subnets) within substantially fewer GPU hours than previous methods. Our approach, Quantized-for-all (QFA), is the first to seamlessly extend one-shot weight-sharing NAS supernet to support subnets with arbitrary ultra-low bitwidth mixed-precision quantization policies without retraining. QFA opens up new possibilities in joint hardware-aware neural architecture search and quantization. We demonstrate the effectiveness of our method on ImageNet and achieve SOTA Top-1 accuracy under a low complexity constraint (<20 MFLOPs).

BatchQuant: Quantized-for-al A Search with Robust Quantizer
Haoping Bai"
Meng Cao
Ping Huang
Jiulong Shan
Apple
hhopingg bai
mengcao,
huang- -ping
jiulong_shan wwwpssannnnpppppecco com
Abstract
As the applications deep learning models on edge devices increase at an acceler- ating pace, fast adaptation to various scenarios with varying resource constraints has become crucial aspect of model deployment. As a result, model optimization strategies with adaptive configuration single-shot quantized neural architecture search enjoys flexibility in both model architecture and quantization policy, the combined search space comes with many challenges, including instability when training the weight-sharing supernet and difficulty in navigating the exponentially growing search space. Existing methods tend to either limit the architecture search space to small set of options or limit the quantization policy search space to fixed precision policies. To this end, we propose BatchQuant, a robust quantizer formulation that allows fast and stable training of a compact, single-shot, pixeei-recissio, weight-sharing supernet. We employ BatchQuant to train a compact supernet (offering over 1076 quantized sub- nets) within substantially fewer GPU hours than previous methods. Our approach, Quantized-fom forlll (QFA), is the first to seamlessly extend one-shot weight-sharing NAS supernet to support subnets with arbitrary ultra-low bitwidth mixed-precision quantization policies without retraining. QFA opens up new possibilities in joint hardware-aware neural architecture search and quantization. We demonstrate the effectiveness of our method ImageNet and achieve SOTA Top-1 accuracy under a low complexity constraint (< 20 MFLOPs). The code and models will made publicly available at httttts://ggitiii cww con/bhpfelix/(FA
becoming increasingly popular. While
Introduction
In order to deploy deep learning models on essource-constraing edge devices, careful model opti- mization, including pruning and quantization is required. While existing works have demonstrated the effectiveness of model optimization techniques in speeding up model inference [
model optimization increases human labor by introducing extra hyperparameters. Consequently. automated
3
4
methods such as neural architecture search (NAS) 5
6
9
10 I
22
13
14
15
[6 and
automated quantization policy search 717 18 19 have emerged to alleviate the human bandwidth required for obtaining compact models with good performance.
In this paper, we focus on finding the best of both worlds- best combination of architecture and mixed-precisioe precision quantization policy. However, combining two complex search spaces is inherently challenging. not to mention that quantization usually requires a lengthy quantization-awwi -waree training (QAT) procedure to recover performance. Thus, previous methods tend to employ proxies to estimate the performance of an architecture and quantization policy combination. For example, APQ [20] performs QAT on each of 5000 architecture and quantization policy combinations for 0.2 GPU hours
"Corresponding author: Haoping Bai http://g__aa@pppee...omm
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Comparisons of quantized architecture search approaches: DNAS 19 SPOS [2], HAQ [7]. APQ 20). OQA (23] and BQ (Ours). "Single-shot mixed-precisiog QAT means the supernet is directly trained with QAT on arbitrary mixed-precision quantization policies. "No training during search" means there is no need for re-training the sampled network candidate during the search phase, and this is accomplished by single-shot supernet training similar to 122]. "No evaluation during search" means that we do not have to evaluate sampled network candidates the validation dataset during the search phase, and this is achieved by training an accuracy predictor similar to means that we do not have to finetune any searched architectures as
22.. "No retraining/finetunnn weights inherited from supernet already allow inference for the given architecture under the specified quantization policy. "Weight-sharing full-precision weights so that we can obtain mixed-precistee weights by fusing the corresponding quantizers with the same set of full precision weights. "Compact MobileNet search space" means that supernet is based on the Mobilenet search space, which offers better accuracy v.s. complexity trade-off but is more sensitive to quantization than heavy search space such ResNet.
means that quantization policies share the same underlying
DNAS SPOS
HAQ
APQ
OQA
QFA
Single-shot mixed-precision QAT
No training during search
No evaluation during search
No retraining / finetuning
Mixed-precision quantization
V
Weight-sharing
Compact MobileNet search space
and uses the sampled combinations to train an accuracy predictor. BATS [21] searches for cell structures that are repeatedly stacked to form the target architecture. Proxy- methods necessitate both careful treatments ensure reliable ranking of quantized architectures and time-consuming retraining procedure when the target quantized architecture is identified, rendering proxy-based approaches impractical for frequently changing deployment scenarios.
To avoid the lengthy retraining process, NAS methods that train a single-shot weight-sharing su- pernet 112 22 are ideal. However, many previous works have shown evidence that QAT of mixed- precision supernets can easily become highly unstable 23 21] As a result, existing single-shot quantized architecture search methods usually limit the size of the combined search space. For example, SPOS [12 sacrifices architecture search space size to only allow channel search and re- quires retraining to recover performance. OQA (33 limits its quantization policy search space to fixed-precision quantization policies and trains a separate set of weights for each bitwidth.
To successfully train the mixed-preese precision supernet, we propose BatchQuant (BQ). Analogous to batch normalization, BQ leverages batch statistics to adapt to the shifting activation distribution as a result of quantized subnet selection, offering better robustness to outliers than quantizer with vanilla running min/max based scale estimation, and better flexibility than learnable quantizer that only learns fixed set of parameters. Without limiting the architecture search space, our joint architecture and quantization policy search space contains over 1076 possible quantized subnets, providing much more flexibility than previous search spaces (e.g. The OQA search space has 1020 possible quantized subnets). While our approach and OQA both follow the supernet training strategy introduced in [22. our weight-sharing supernet takes only 190 epochs to train despite the complex search space, significantly less than the 495 epochs required by OQA. We further leverage the NSGA-II algorithm to produce Pareto set of quantized architectures that densely covers varying complexity constraints, eliminating the marginal cost of adapting architecture to new deployment scenarios.
The contributions of the paper are
To the best of our knowledge, we present the first result to train one-shot to support subnets with arbitrary
supernet quantization policy without retraining.
We propose BatchQuant, an activation quantizer formulation for stable mixed-precision supernet training. The general formulation of BatchQuant allows easy adaptation of new scale estimators.
Compared with existing methods, our method, QFA, takes shorter time to train a significantly more complex supernet with over 7076 possible quantized subnets and discovers quantized subnets at SOTA efficiency with no marginal search cost for new deployment scenarios.

---

Table 2: Design runtime comparison with state-of-the-ar art quantized architecture search methods. Here we use (-) to denote the runtime of an algorithm. Note we separate the accuracy predictor training from the search procedure because it a one-time cost amortized across deployment scenarios. We define marginal cost as the cost for searching in new deployment scenario, and we use N to denote the number of deployment scenarios. Our method eliminates the marginal search cost completely.
Method
Design Runtime
SPOS
T(Supernet Training) T(Search + Finetune) x
APQ
T(Supemet Training Accuracy Predictor Training) T(Search + Finctune) x N T(Supernet Training Accuracy Predictor Training) T(Search + Finetune) x N
OQA
Ours
T(Supernet Training + Accuracy Predictor Training + Search)
2
Related Work
2.1 Mixed Precision Quantization
Different layers of network have different redundancy and representation power, each layer will react to quantization differently and may achieve varying levels of efficiency gain on hardware. In fact, many fixed-bitwidth quantization methods are inherently mixed-precision by making design decisions to leave the first convolution layer, batch-norm layers [24), squeeze and excitation layers [25], and the last fully connected layer at full precision int8 precision
As result, mixed- precision quantization methods [26] 18] 27 17 emerge in the place of fixed precision quantization and are receiving increasing attention from hardware manufacturers [17]. In contrast to srixed-precision quantization methods that optimize for a single quantization policy on single architecture, QFA allows the discovery of the most suitable quantization policy for arbitrary subnet architectures.
3 2
19 12 23
Adaptive Quantization also closely related to our work. Methods include Adabits [88 Any- Precision DNN 29 , Gradient l regularization or quantization robustness [30], and KURE [31
train model that can adaptively switch to different bitwidth configuration during inference. Most
methods 28 29 30 only address the case of fixed precision quantization where all weights share the same bitwidth and activations share another. Our quantization policy search space offers much more flexibility in terms of layerwise mixed-precision quantization.
2.2 Joint Mixed-Precision Quantization and Neural Architecture Search
There exist many exciting works in the intersection of mixed precision quantization and NAS. JASQ [32] employs population-base based training and evolutionary search to produce a quantized architecture under combined accuracy and model size objective. DNAS 19] is a differentiable NAS method that optimizes for a weighted combination accuracy and resource constraints. SPOS [2] trains quantized one-shot supernet to search for bit-width and network channels for heavy ResNet search space. APQ (0) builds upon a full precision one-shot NAS supernet and trains 5000 quantized subnets to build a proxy accuracy predictor. The best quantized architecture proposed by the predictor is then retrained. Most existing methods fall into two categories. The first optimizes for single weighted objective of accuracy and complexity and produces only single few quantized architectures [32 19]. which is difficult to scale to multiple deployment scenarios. The second category is capable of estimating the performance for many quantized architectures by using proxies such as weight-sharing supernet and partially trained models [12] (2).. Due to the accuracy degradation in weight-sharing supernet SPOS and the use of proxy in APQ, both methods require retraining when the best quantized architecture is discovered. On the contrary, our approach does not require retraining. Training the weight-sharing supernet with BQ reduces accuracy degradation, allowing quantized subnets to reach competitive performance without retraining. As a result, QFA enjoys no marginal cost for deploying quantized subnet to new scenarios. Table [ details the difference of our approach from other architecture search approaches. Table cobbeere the algorithmic complexity of our method with other mixed-precision quantized architecture search methods.
3

---

Stabilizing Mixed Precision Supernet Training with BatchQuant
We adopt the single-path weight-sharing MobileNetV3 3
supernet formulated in [2 that supports adaptive input resolution, network kernel size, depth, and width. In addition, we assign simulated quantization operations (quantizers) (34], one for each bitwidth, to each weight and activation tensor within the supernet to support quantization-awal aware training (QAT). We allow bitwidths {2, 3, 4] for both weight quantizers and activation quantizers.
The compact weight-sharing supernet formulation allows us to share a single set of quantizers across all the subnets. However, such compactness also leads to instability. Specifically, supernet training becomes highly unstable when activations from different subnets are quantized by a shared quantizer. In the following sections, we will review the fundamentals of quantization and address why shared activation quantizer can cause unstable supernet training. Then, we introduce BatchQuant as solution to stabilize supernet training.
3.1 Quantization Preliminaries
To help address the difficulty in training the mixed-precision supernet, we start by introducing common notations for quantization. WLOG, consider the case of uniform affine (asymmetrie) quantization. Let x: = { N be a floating point vector/tensor with range (Tmin: mmaz) that needs to be quantized to b-bitwidth precision. The integer coding Eg will have range [n. p] = [0, 2 1]. Then we derive two parameters: Scale (A) and Zero-point(2) which map the floating point values to integers (See []] The scale specifies the step size of the quantizer and floating point zero maps to zero-point
34], an integer which ensures that zero is quantized with error. The procedure is follows:
I mea
Imin
(-1
Fmmé
clamp
.T.P
Zg
clamp
2, n.P)
② Dgg 2
where |-] indicates the round function and the clamp(-) function clamps all values to fall between and p. The quantized tensor is then used for efficient computation by matrix multiplication libraries that handle A efficiently (See (355). Note that during training, the range (Tmin, Tmmaa) of activations are not known beforehand. Therefore, standard practice
keep track of an exponential moving
average (EMA) of past extreme values (min&max) to generate the scale estimator Å. As both the calculation of and 2, relies good A estimate, we will later provide insights into why using EMA estimator Å for activation quantization the supernet could lead unstable training.
3.2 Weight Quantization with LSQ
Since the value distribution of a weight tensor is relatively stable across updates and is empirically observed to be symmetrical around zero, many methods treat A learnable parameter and discard the zero point = [36] 4). We leverage the symmetric LSQ quantizer [36] to quantize weight tensors in our supernet as follows:
Zg = clamp .n.P)
n.p))
& xr,
where [n, = 1-22-1 20-1-111 11. Note the rounding operation [-1 has 0 derivative almost everywhere, QAT applies the straight through estimator (STE) 337 to allow gradients to backpropagate through.
3.3 Challenges of Activation Quantization in One-shot Supernet
While weight quantization for weight-sharing supernet is similar to that of normal networks, activation quantization requires careful treatment enable stable supernet training. According the insights in 31 as well shown in Equation ① unstable A estimate could impact the quantized tensor à and negatively impact QAT performance. Given the significant delay in the EMA estimator Å when

---

the ranges shift rapidly, practical approaches such as [4] completely disables activation quantization for the first 50 thousand 2 million steps. However, we now show that one-shot supernet will always have rapidly shifting activation ranges due to subnet sampling.
EMA Estimator Á is problematic. Let N = x C' x H x W denote the number of elements within an activation tensor with batch size B. channel number C. height H, and width W. Due to adaptive input resolution and network width (number of channels). the activation map at a given layer will vary size (H, W) and channel number C' depending on the activated subnet. For example, at a batch size of 64 the size of incoming activations to the second mobile inverted residual block in our search space can vary from [64, 72, 64, 64] to [64, 144, 112, 112].
We now consider the case where elements within x are i.i.d random variables with cumulative distribution F, and let MN max(2; maximum extreme value distribution with the following cumulative distribution function:
, .IN) denote the maximum. Then My follows the
P(MN $) = P(11 tt EN S t) = P( t) P(Nx S St) - F(t)
(3)
Following the assumptions in [38. we model activations within neural networks as tensor-vales valued Laplace random variables. WLOG, when considering the maximum value distribution, we can model each element as an exponential random variable, because the Laplace distribution can be thought of as two exponential distributions spliced together back-to-back With oy ~ Exp(A), i ... N..
[[MN]
(1 F(t)()) / 0
So
(1 (1- e1- -M
(Iet 1 - -x)
(4)
11 W du
1 k
(og(NV)
k=1
We can see that [] slowly diverges to oo as N increases. For a symmetric activation distribution where the minimum and maximum value distri- butions are symmetric about O, can consider [[Å]- 2 x (og(IV) for the EMA estima- tor. As a result, the EMA estimator Á will never become stable because it is sensitive to the activa- tion tensor size and could change significantly for each update depending the example mobile inverted residual block given above, the largest difference in [[]]]] is propor- tional to log(151142)) * batch_ sizc). which can lead to substantial instability in the EMA estima- tion. Furthermore, due to adaptive layer skipping, layers that are only present in deep subnets will activate less frequently and have substantially more lag in EMA update than layers shared among shallow and deep subnets, exacerbating the instability in EMA and the quantized activations. Figure | shows the maximum extreme value distribution for the respective activations collected from 1000 batches of training data.
0.36
Activation Tanpoz $ize
0.35
6--602204 12x111111
Density
0.34
0.33
0.01
the sampled subnet. For
60
80
100
1.0
1.0
Maxiimum Activation Value
Figure 1: Shifting Max Value Distribution of activation tensors during training.
Note such shifting extreme value distribution problem also happen to weight tensors in weight- sharing supernets because different subnets can have different weight tensor sizes. Conveniently, our supernet construction ensures that at each layer, the weight tensor for smaller subnets is nested within the weight tensor for the larger subnet. Therefore we simply perform quantization on the entire weight tensor before indexing out the weights for the given subnet to stabilize weight quantization.
Learnable scale does not help. Methods that learn a fixed A is also suboptimal the case of supernet training. According to (38) the optimal quantization level depends on the variance in activation distribution (e.g. Laplace parameter b for Laplace distribution or the variance o² of normal distribution). However, as observed in (29). quantizing weights and input to varying precision leads to activations with different mean and variance. While we can casily resolve this issue fixed- -red-sirciiion quantization by learning different set of A for each bitwidth setting, it is impossible to assign set of A parameters for each mixed- -reedd-preiiion quantization policy on each subnet.
Empirically, during the training of our compact MobileNet search space, EMA estimator Å leads to gradient explosion, and a fixed set of learnable A leads to diverging training loss well.
5

---

3.4 Batch Quantization for Robust Activation Quantization
To alleviate the unstable scale estimation issue, we propose the following general definition of batch quantizer (BatchQuant)
Â xmaa Emin
(-l
Brinn
2 = clamp
n
p p-n
Bq caamp
+ + + B, n.
(5)
D7
(
B)
Analogous to batch normalization (24), we leverage batch statistics to help standardize the activation distribution across different sampled subnets. Instead of using EMA, we estimate the extreme values min and Rmar only from the current batch to negate the effect of changing extreme value distribution due to subnet sampling. In addition, we learn multiplicative residual on Á and additive residual 33 on & to facilitate learning of the optimal clipping range.
Note, similar to LSQ+ 4 zero in activation may not be exactly quantizable based on such formulation. However, we share emprical observation as LSQ+, that the learned often small and aids in empirical performance by reducing the quantization eπror for H-swish activation [33]. Because we use symmetric quantization for weights, asymmetric quantization of activations has no
additional cost during inference as compared to symmetric quantization since the bias term can be precomputed and fused with the bias in the succeeding layer:
bias
①. = (w, (x, - 1 3) We Mu
(m B
(6)
BatchQuant (BQ) is generally applicable. Note our formulation of BatchQuant is general. We do not explicitly define the batch extreme value estimators Aii and Amaz. because it is straightforward to plug in different definitions the usage fit. We will provide three simple example formulations of Emin and &maw below. BatchQuant could potentially he a good drop-in replacement for conventional QAT with EMA scale estimator when adjusting training batch size or performing tasks that have varying activation size such as semantic segmentation. Due to the adaptive formulation. BatchQuan offers much simpler initialization strategy than the optimization-basee strategy in LSQ+ 4. With proper choice of the batch extreme value estimator (fmin Imaa), we can simply initialize the residual terms with y and 0. As training is stabilized from the beginning. BQ can also avoid the delayed activation quantization strategy introduced in [34]. which requires choosing a starting schedule for activation quantization.
Extreme Value Estimators Due to presence of learnable residuals t and 3, the extreme value estimators (fmmin: Fmazz) does not need to capture the exact value of the scale long as the calculated scale Å captures the range variation in activation. Consider 4D activations with vector index i (in, ic, in, iw) that indexes batch, channel, height, and width dimension respectively, we present three example definitions to use our experiment:
Enin min Zi,
imaz
max xY
imin
min
Ci,
Eman
C
Σ max C;
(8)
ig io
cc
Emin
30,
Emaz
+ 30
where a is the censor-wise mean and o is the wise variance. Note that Equation 7 provides unbiased estimation of extreme values and is analogous to EMA A estimator but only based on batch statistics. Equation 9 is a biased estimation with the least variance and is equivalent to the scale parameter initialization strategy of LSQ+ 4 Plugging Equation 9 into Equation 5 is thus analogous of using a learned scale quantizer. Equation g is a biased estimation with a bias- variance trade-off between Equation 7 and 9
We show empirically that Equation leads to stable supernet training.
Batch QuachQuant Calibration during test time Similar to the treatment of batchnorm in 22 to obtain a set of extreme value estimations (fmmnn &moz) specific to quantized subnet for inference, we simply perform BatchQuant calibration by forwarding few batches of data and accumulate running mean of the estimations.

---

4 Mixed-Precision Quantized Architecture Search
Equipped with BatchQuant for stable mixed-precision supernet training, we now introduce the search procedure of our mixed-precision quantized architecture search method, QFA.
Supernet Search Space Definition Following the convention 3 19 12 23 of only searching and quantizing the residual blocks in the Mobilenet V3 search space, we keep the input and weights of the first convolutional layer at full precision and quantize the incoming activation and weights of the last fully-connected layer to 8-bit precision. Specifically, our Mobilenet V3 search space contains 5 stages each with 4 mobile inverted residual blocks. Each stage can optionally skip or 2 of its last blocks. Each block has kernel size options {3, 5, 7} and expansion ratio choices {3, 4, 6). Within each block, there are 3 convolution layers. Each convolution layer can independently choose an activation bitwidth and a weight bitwidth. Thus, each block has (3 x 3 x 33x2) = 6561 configurations, each stage has 6561 x (1 + 6561 x (1 6561)) & 1.85 x 10t5 configurations. Without considering the elastic input resolution choices, our complete search space contains over (1.85 x 101535 2.19 1076 different quantized architectures.
Supernet Training and Elastic Quantization. Fol- lowing the common practice of starting QAT from trained full precision network [I] we start by pretrain- ing the supernet without quantizers on ImageNet [39].
Table The accuracy of the biggest fixed- precision model after single-stage and two- stage elastic quantization. Single-stage means that we directly train supernet with bitwidth choices {2, 3, 4} for 190 epochs, and two-stage means that first train su- pernet with bitwidth choices {2, 4, 32] for 65 epochs, then continue training the supernet with bitwidth choices {2, 3, 4} for 125 epochs. W/A denotes the bitwidth for weights and activation respectively.
We follow the training protocol introduced in [2] to train the supernet. Then, we perform QAT starting from the full precision supernet. Unlike [22 23 that
adopts a progressive shrinking strategy which gradually opens up smaller subnet choices as training progresses, we directly allow all possible quantized subnet options. Our clastic quantization training consists of two stages. In the first stage, we train the supernet for 65 epochs with bitwidth choices (2, 3, 4, 32). where a bitwidth of 32 means floating-poin precision. Empirically, found that mixing in the well-trained floating- point lay- helps the low precision layers learn and leads to more stable training. Then, we proceed to the second stage and train the supernet for 125 epochs with only low bitwidth choices (2, 4). Table shows that our two-staged training strategy outperforms training a supernet with bitwidth choices 32,3, 4)
W/A
4/4
3/3
2/2
Single Stage
74.8%
73.7%
66.3%
Two Stage
75.6%
74.3%
68.4%
for the same number of epochs. Our elastic quantization training requires only 190 epochs of training. In comparison, the staged training of fixed precision supernet at each bitwidth in [23 takes 495 epochs in total.
16 for each supernet training update, we sample and accumulate gradients from multiple subnets. Specifically, adopt the sandwich rule proposed in [16] by sampling 4 quantized subnets accompanied by the smallest architecture and the largest architecture within the supernet with the entire architecture set to random bitwidth.
Subnet Sampling. Similar to
Multi-ohjective Evolutionary Search To balance multiple competing objectives, we leverage multi- objective evolutionary search to produce desirable subnets. Unlike the aging evolution in [22] that only produce one output architecture under a given set of constraints at a time, we adopt the NSGA-II algorithm [40]. which outputs Pareto population at once. As result, we can quickly access the Pareto front of a trained supernet. At each iteration, the NSGA-II algorithm selects the best-fit population with non-dmmintted sorting. Then, a crowding distance is calculated to ensure the selected individuals cover the Pareto front evenly without concentrating at a single place. To speed up the search process, we follow 2 and train accuracy predictor that predicts the accuracy of given quantized architecture configuration. We use one-hot encoding to encode quantized architecture configurations into binary vector.
5 Experimental Analysis and Results
Experiment Settings and Implementation Details We base our codebase on the open-source implementation of (22] under the MIT License, and we follow the exact training procedure on
7

---

Comparison with
Compazison with F Fixed-Precision Methods
Mixed-Precisior: Me thods
Top 1 Accuracy ($) (8)
70
APQ
APOT
SPOS
60
00A
50
Ouza
ouz
BMoble
+180+
100
200
300
10
20
50
100
200
500
MFLOFs
MFLOPs (Log Scale)
Figure 2: Comparison with state-of-the-art quantization methods on the ImageNet dataset. The left subplot compares our Pareto architectures with those existing mixed-precision quantized architecture search methods (APQ. SPOS). The right subplot compares our Pareto architectures with existing fixed peed-precision quantization methods (OQA, BATS, BMobi, LSQ. LSQ+, APOT, QKD). For the case of viewing, we denote each discovered architecture with marker only on the left subplot.
ImageNet [39] to obtain the full precision supernet. For both stages of the clastic quantization procedure, we follow the common hyperparameter choice of [22] and use an initial learning rate of 0.08. For all experiments, we clip the global norm of the gradient at 500. We train with a batch size of 2048 across 32 V100 GPUs our internal cluster. Unless otherwise mentioned,
keep all other settings the same as [22]. After training is complete, we randomly sample 16k quantized subnets and evaluate on 10k validation images sampled from the training set train the accuracy predictor. During the evolutionary search, we keep a population size of 500 for 1000 generations. For each generation, once we identify the Pareto population based nondominated sorting and crowding distance, we breed new genotypes through crossover and mutation with a crossover probability of 0.007 and a mutation probability of 0.02. We determine both crossover and mutation probability through grid search. To demonstrate the effecctivneess of QFA, we conduct experiments that cover the accuracy and complexity trade-off of searched subnets. For a fair comparison with previous methods, we adopt FLOPs for full precision layers and BitOPs for quantized layers our complexity measure. While there is no direct conversion from BitOPs FLOPs, we follow the convention where given full precision layer with FLOPs a, its quantized counterpart with m bit weight and bit activation will have a BitOPs of mm a and a FLOPs of (mn x a) a//44. To be concise, we report our complexity as FLOPs of FP layers BitOPs of quantized layers / 64 = Total FLOPs.
[2] 23),
Stable Mixed-precision Supernet Training with BQ To test the effectiveness of BQ in stabilizing supernet training, we attempt to train a bascline supernet with LSQ quantized weights and LSQ+ quantized activations. To test the robustness of low bitwidth training, we only allow bitwidth choices (2,3, 4). We follow the practice of LSQ+ and initialize the scale and offsst parameter layer-by- layer through an optimization procedure that minimizes the MSE between quantized output and full precision output at the given layer. However, the training loss diverged at the start of the second learning rate warm-up epoch. Replacing LSQ+ activation quantizer with BQ, we obviate the aptimization-basees initialization procedure, and the divergent training loss problem no longer occurs.
Effect of Residual Terms ? and /3 To test the effect of y and / in BQ, we removed and from all BQ operations when training the supernet. The training loss diverged within a few update steps.
Comparing Extreme Value Estimator Choices To investigate the stability of different extreme value estimators, we train BQ supernet with bitwidth choices {2,3, 3, 4} and plug-in Equations 7
as extreme value estimator. Training loss for both Equations 7 and diverged after the first lew updates, and only Equation g led to stable training.
Comparison with SOTA Mixed-Precision Quantized Architecture Search We compare with existing quantization-awar NAS methods including APQ (0) and SPOS [12] as shown in the left subplot of Figure 2 Our results outperform both APQ and SPOS by a large margin. Thus, in terms of existing mixed-precision quantized architecture search methods, we are able achieve the best accuracy v.s. FLOPs trade-off.

---

Table Design cost comparison with state-of of-thh--rtt-r-r- quantized architecture search methods. Our method eliminates the marginal search time. Thus the marginal CO2 emission (lbs) [1 is negligible for search in a new scenario. Here marginal cost means the cost for searching new deployment scenario, we use N to denote the number of up-coming deployment scenarios.
Design cost (GPU hours)
CO2e (lbs) (marginal)
Method
SPOS
288 + 24N
6.81
APQ
2400 0.5N
0.14
OQA
2400 + 0.5N
0.14
Ours
1805 + O.N
0.
Comparison with SOTA fixed-precision quantized models We further compare with several strong
fixed-precision guantization methods including OQA [23]. BATS [2I]. BMobi [42], LSQ (36), LSQ+ 44, APOT 43 and QKD[44) as shown in Figure 2 Our results demonstrate the competitive performance quantized subnets discovered by QFA against quantized models produced by state-of- the-art quantization and quantization-awee aware NAS methods. For models 20 MFLOPs,
able to
achieve comparable performance as OQA with much less training cost- OQA supernets require total of 495 epochs to train, while our mixed- precision supernet took only 190 epochs to train. We mixed-precision achieve SOTA ImageNet top-1 accuracy on models under 20 MFLOPs. Our advantage over OQA < 20 MFLOPs could be mainly due the flexibility of our search space smoothly interpolate between the Pareto frontiers of fixed-precision architectures by mixing layers with varying precision.
Design Cost Comparison with Quantized Architecture Search Methods As shown in Table 4 approach demonstrates better efficiency in terms of GPU hours and marginal carbon emission. We report the total GPU hours for our approach, including 1200 GPU hours of full- full-precision supernet training, 565 GPU hours of mixed- data collection and training. Running NSGA-II with the trained accuracy predictor does not make use of GPU and finishes within hour. For a fair comparison, we added the GPU hours to train the full precision supemee to the quantized architecture search time reported by the authors of OQA.
supernet training, and 40 hours of accuracy predictor
6 Discussion
While we demonstrate the superior performance of BatchQuant in combination with weight-sharing supernet, there remain many other directions for future investigation. QFA shares some limitations common to many NAS methods with weight-sharing supernets. Since we only visit small fraction of subnets during training, some subnets may receive insufficient training. Employing targeting this limitation may prove helpful for our especially complex search space. Finally, thorough theoretical explanation of the effectiveness of BatchQuant on the training dynamics of weight-sharing mixed- mixed-precision supernet is still an open problem, and we leave the investigation for future exploration.
techniques 45 46
Conclusion
In this paper, we present Quantized-for-All (QFA), novel mixed-precision quantized architecture search method that can jointly search for architecture and mixed-precisse quantization policy combination and deploy quantized subnets SOTA efficiency without retraining finetuning. We provide analysis on the challenge of activation quantization in mixed mrixei-precision supernet. To allow stable training of QFA supernet, proposed BatchQuant (BQ), a general, plug-and-play quantizer formulation that stabilizes the mixed-precision supernet training substantially. We demonstrate the robust performance of QFA in combination with BQ under ultra-low bitwidth settings (2/3/4). By leveraging NSGA-II, we produce a family of Pareto architectures at once. Our discovered model family achieves competitive accuracy and computational complexity trade-off in comparison to existing state-of-the-art quantization methods.

---

References
[1] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepa- per, 2018.
[2] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks, 2018.
[3] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, 0tns/1606.06160 06160, 2016.
[4] Yash Bhalgat, Jinwon Lec, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offscts and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 696-697, 2020.
[5] Barret Zoph and Quac Lc. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, volume abs/161 1.01578, 2017.
[6] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. 2018 JEEEJCVF Conference on Computer Vision and Pattern Recognition, Jun 2018.
[7] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
[8] Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search on object detection. arXiv preprint arXiv:1903. 2019.
[9] Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang. Alan Yuille, and Jianchao Yang. Atomnas: Fine- Emee-grinned end-to- and-to-end neural architecture search. In International Conference on Learning Representa- rions, 2020.
[10] Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik-Mand Xnas: Neural architecture search with expert advice. NeurlPS, ths//196600031,, 2019.
[11] Han Cai, Ligeng Zhu, and Song Han. ProxylesNNAS: Direct neural architecture search target task and hardware. In International Conference on Learning Representations, 2019.
[12] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv: /904.00420, 2019.
[13] Mingxing Tan, Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[14] Golnaz Ghiasi, Tsung Yi Lin, and Quoe V. Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR). June 2019.
[15] Liam Li and Amect Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2019.
[16] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. arXiv preprint ar 2003:22003.11 / 142, 2020.
[17] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[18] Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed- precisson 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.
[19] Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search, 2019.
[20] Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy, 2020.
10

---

[21] Adrian Bulat, Brais Martinez. and Georgios Tzimiropoulos. Bats: Binary architecture search. arXiv preprint arXiv:2003.01711 2020.
[22] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations 2020.
[23] Mingzhu Shen, Feng Liang, Chuming Li, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Once quantized for all: Progressively searching for quantized efficient models, 2020.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448 456, Lille, France, 07-09 Jul 2015, PMLR.
[25] Hu, Li Shen, and Gang Sun. Squeeze-am and and---ndd-iiieee excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[26] Manuele Rusci, Alessandro Capotondi, and Luca Benini. Memory- driven mixed low precision quantization for enabling deep network inference on microcontrollers. arXiv preprint arXiv: 0e..//905..... 13082, 2019.
[27] Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5741-5752. Curran Associates, Inc., 2020.
[28] Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit-widths. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2146-2156. 2020.
[29] Haichao Yu, Haoxiang Li, Honghui Shi, Thomas S Huang, and Gang Hua. Any-precision deep neural networks. arXiv preprint arXiv:191..07346. 2019.
[30] Milad Alizadeh, Arash Behboodi, Mart van Baalen, Christos Louizos. Tijmen Blankevoort, and Max Welling, Gradient & regularization for quantization robustness. In International Conference Learning Representations, 2020.
[31] Moran Shkolnik, Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, and Uri Weiser Robust quantization: One model rule them all. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5308-5317. Curran Associates, Inc., 2020.
[32] Yukang Chen, Gaofeng Meng, Qian Zhang, Xinhang Zhang, Liangchen Song, Shiming Xiang, and Chunhong Pan. Joint neural architecture search and quantization. CoRR, 011009961,9 2018.
[33] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pages 1314-1324, 2019.
[34] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam. and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference, 2017.
[35] Benoit Jacob and Pete Warden. gemmlowp: small self-contained low-precision gemm library.
[36] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.08153, 2019.
[37] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
[38] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid- depdodeeployment. Conference on Neural Information Processing Systems (NeuriPS), 2019,
[39] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255 2009.
[40] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions Evolutionary Computation, 6(2):182- 197, 2002.
11

---

[41] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. ACL, 2019.
[42] Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Xwang-Ting Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution- based searching. arXiv preprini arXhv:2005.06303 2020.
[43] Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-wo quantization: A non-uniform discretization for neural networks. arXiv preprint arXiv:1909.13144 2019.
[44] Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-awwre knowledge distillation. arXiv preprint 1911v1111..12411, 2019.
[45] Zhichao Lu, Gautam Sreekumar, Erik D. Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. Neural architecture transfer. CoRR, abs/2005.05859, 2020.
[46] Houwen Peng, Hao Du, Hongyuan Yu, Qi Li, Jing Liao, and Jianlong Fu, Cream of the crop: Distilling prioritized paths for one-shot neural architecture search. Advances in Neural Information Processing Systems, 33, 2020.
12


Reviews:
[{"review_id": "aURKgEj5o0m", "reviewer": "Reviewer_Q9hH", "summary": "This paper proposed to conduct quantization-aware NAS: in the meanwhile of neural architecture search (basically it is searching subnet within a supernet), it searches for bitwidth allocation for different layer. To stablize the training for quantized activation, it proposed BatchQuant, following similar method of batch normalization.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The protocol of NAS follows previous work, which is less novel. The proposed BatchQuant is interesting: Basically, quantization-aware NAS is adding quantization bit search into NAS. However, BatchQuant is clamined to be necessary for quantization-aware NAS. \n2. Can BatchQuant be applied on normal network quantization training ? If possible, are there any experiments?\n3. MobileNetV3 is used as spernet, following the convention used in previous work. Is other network possible for supernet? \n4. In Line 297-298, accuracy predictor is trained using quantized subnets, how are subnets processed to be inputs for predictor?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N.A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SgZMCCYmLMO", "reviewer": "Reviewer_JFEL", "summary": "This paper proposes a combination of neural architecture search and per-batch quantization statistics calibration in order to quantize neural networks to low bit-width using mixed precision. The proposed NAS method is applied to ImageNet and pareto-optimal networks in the accuracy vs complexity space are found.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall an interesting idea and a generally well written paper. Some questions/comments that I have:\n\nI wish to point out that in eq. (4) $\\frac{1-u^N}{1-u} = \\sum_{k=0}^{N-1}u^k$ for $u\\in[0,1]$. Therefore, we get $E[M_N] = \\int_0^1 \\frac{1}{\\lambda} \\sum_{k=0}^{N-1}u^k du = \\frac{1}{\\lambda} \\sum_{k=0}^{N-1}\\int_0^1u^k du = \\frac{1}{\\lambda}\\left(1+ \\sum_{k=1}^{N-1}\\frac{1}{k}\\right)$. The conclusion that follows eq. (4) doesn't change much but the correction should be made nonetheless. I also checked reference [38] and did not find a justification for modeling activations as being Laplacian distributed. Can we get some comments on the validity of this assumption. Otherwise, section 3.3 is a good one?\n\nRegarding BatchQuant itself (section 3.4), I am having a hard time understanding how it differs from standard QAT approaches that observe the max value in the instantaneous tensor (such as DoReFaNet). This seems to be what BatchQuant is doing based on the description in lines 202-203. In BatchNorm, there is a moving window that averages mean and variance of the batches temporally and then uses the converged values post-training for inference. That doesn't seem to be what is going on here. Instead, it appears that simple per-tensor scaling is employed. Can we please get some clarification?\n\nRegarding the use of NAS, is there a specific contribution by which mixed-precision is made possible? The introduction and abstract seem to suggest so, however, the description in Section 4 indicates that this is done simply by using prior arts. Either of the two description needs to be corrected for exactness and consistency.\n\nPost-rebuttal: I thank the reviewers for their responses. I am willing to keep my score of a weak accept. But I wish to ask that the authors double check all the math (and crucially its exposition) as well as the details of BatchNorm vs BatchQuant. Ideally, there should be no room for a doubt.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, there is a nice discussion section on that on Page 9.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3 hrs"}, "extra_text": {}}, {"review_id": "HahZcm8M-lR", "reviewer": "Reviewer_552g", "summary": "The paper presents the use of batch statics to help stabilize the NAS+Quantization process. It is based on the one-shot NAS idea, but applies the batch idea to assist the quantization policy search during the Neural Architecture Search. The results are compared to some alternative methods and show a faster convergence.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n* The problem tackled in the paper, NAS+Quantization combination, is challenging. \n* The idea seems to help speed up the convergence.\n\nWeaknesses:\n* Incremental novelty.\n* No theoretical analysis on the effects of the introduced technique.\n* The evaluation is limited.\n\nSolving the two problems (NAS+Quantization) in a combined manner is challenging. The area has however been explored by prior studies. The proposed idea of using batch statistics to help stabilize the process appears to be a small change rather than anything fundamental. The paper gives no theoretical analysis of the effects of the technique. So it is hard to tell whether the technique is always going to help. A strong comprehensive evaluation could have mitigate the concern. But the evaluation in the paper is quite lightweight, leaving the practical impact of the work unclear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The reviewer didn't find discussions of the limitations of the work or societal impact in the paper. The limitations would be good to add, while the societal impact isn't that important to have for the nature of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "DYA8fI4zZ9R", "reviewer": "Reviewer_HzjN", "summary": "This paper summarizes a technique to allow quantizing a weight-sharing superset to with flexible quantization policies to allow choosing different network architecture and quantization policies for desired efficiency-accuracy trade-off. The key observation of this method is that quantizers needs to estimate the maximum and minimum values of a activations. However, the extreme values of activations are dependent on the activation tensor's size. During training, the activation size changes as we sample different architectures, so the extreme value do not converge when using EMA estimator, and the quantizer become unstable during training.  To fix this, this paper proposes a simple method to estimate the extreme values (therefore quantizer parameters) based on current batch statistics. This leads to better stability and performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper proposes a method that effectively combines superset-based architecture search with mixed precision quantization in one-shot, allowing more flexible network configuration for different accuracy-efficiency trade-off.\n+ This paper is established a convincing loop from observation (input size difference -> unstable quantizer parameters) to techniques (use batch statistics) and results (improved performance). The story is very clear. \n\nWeaknesses:\n- The mathematical derivation in Equation (3, 4) needs improving. E.g., what is the \"z\" variable in Equation (3)? It is not clearly defined. Also, according to Equation (4), since E(M_n) ~ log (n), seems that the expectation will saturates if n (number of elements in activation) is large enough? Does this imply that if we only sample layers with large activations, we can somehow bypass this problem? \n- Any explanation why Equation (8) is the only working estimator? \n- MFLOPs calculation questionable. In L308, this paper states that the FLOP operation reported by this paper is computed by mn/a/64. I understand that it is convenient to use one single metric to evaluate the efficiency, but this should not be called as FLOP, which exactly mean floating-point operations. \n- Configurations of the optimal networks? It will be valuable to share the configurations of Pareto-optimal network configurations, which is not available in this paper. \n\nOverall this is a neat paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No need for discussion. ", "review_text": "", "extra_scores": {"ethical_concerns": "None. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "ARSvH-qvGbq", "reviewer": "Reviewer_JJnk", "summary": "Authors propose a robust quantizer formulation BatchQuant, which utilizes the batch statistics to provide a more stable quantizer for different quantization levels. With benefit of BatchQuant, they train a compact, single-shot, mix-precision, weight-sharing supernet for NAS. They evaluate BatchQuant on ImageNet.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written,\n\nIn neural architecture search, it is very challenging to include the mixed precision quantization as a part of policy, since it enlarges the search spaces and increases the quantization instability during the training. BatchQuant helps to alleviate the  instability introduced by quantization and enables to train a large supernet considering different quantization levels of each layer.\n\nBatchQuant estimates the max and min of activations using the current batch statistics and also learns two scalars gamma and beta for quantization. The method is solid and simple.\n\nIn experiment, they evaluate on ImageNet and compare with SPOS, APQ, OQA methods. BatchQuant achieves better performance in terms of FLOPS and Top 1 acc. \n\nFor rebuttal,\n1. Is there any other dataset or other task to evaluate the method on? It is good that when BatchQuant evaluates on other datasets as well\n2. Authors choose [2, 3, 4, 32] as bitwidth choices. Can authors elaborate the reason for choosing these bitwidths? It seems there is a huge gap between 32 bits and 4 bits.\n3. Even the authors claim that the re-training is not required. But if we re-train the networks, will the performance still get improved?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have mentioned and discussed the limitation in Sec 6", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper combines the idea of mixed precision quantization and single shot NAS method, and proposed a single-shot weight sharing quantization method that requires no training during search. 3 out of 4 reviewers acknowledges that the idea is solid and simple. Concerns remain about the novelty. This paper is recommended for acceptance given the effectiveness and satisfying result. 
Decision: Accept (Poster)