BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
Authors: Huanrui Yang
Lin Duan
Yiran Chen
Hai Li
Venue: ICLR 2021 Poster (2021)
Keywords: Mixed-precision quantization
bit-level sparsity
DNN compression

Abstract: Mixed-precision quantization can potentially achieve the optimal tradeoff between performance and compression rate of deep neural networks, and thus, have been widely investigated. However, it lacks a systematic method to determine the exact quantization scheme. Previous methods either examine only a small manually-designed search space or utilize a cumbersome neural architecture search to explore the vast search space. These approaches cannot lead to an optimal quantization scheme efficiently. This work proposes bit-level sparsity quantization (BSQ) to tackle the mixed-precision quantization from a new angle of inducing bit-level sparsity. We consider each bit of quantized weights as an independent trainable variable and introduce a differentiable bit-sparsity regularizer. BSQ can induce all-zero bits across a group of weight elements and realize the dynamic precision reduction, leading to a mixed-precision quantization scheme of the original model. Our method enables the exploration of the full mixed-precision space with a single gradient-based optimization process, with only one hyperparameter to tradeoff the performance and compression. BSQ achieves both higher accuracy and higher bit reduction on various model architectures on the CIFAR-10 and ImageNet datasets comparing to previous methods.

Published as a conference paper at ICLR 2021
BSQ: EXPLORING BIT-LEVEL SPARSITY FOR MIXED- PRECISION NEURAL NETWORK QUANTIZATION
Huanrui Yang, Lin Duan, Yiran Chen & Hai
Department of Electrical and Computer Engineering
Duke University
Durham, NC 27708, USA
(huanrui yang, 1in. duan, yiran.chen, hai @dukee edu
ABSTRACT
Mixed-precision quantization can potentially achieve the optimal tradeoff between performance and compression rate of deep neural networks, and thus, have been widely investigated. However, lacks a systematic method to determine the exact quantization scheme. Previous methods either examine only small manually- designed search space or utilize a cumbersome neural architecture search to explore the vast search space. These approaches cannot lead to optimal quantization scheme efficiently. This work proposes bit-level sparsity quantization (BSQ) to tackle the mixed-precision sparsity. We consider each bit of quantized weights as an independent trainable variable and introduce a differentiable bit-sparsity regularizer. BSQ can induce all-zero bits across a group of weight elements and realize the dynamic precision reduction, leading to a mixed-precision quantization scheme of the original model. Our method enables the exploration of the full mixed mixed-precision space with single gradient- based optimization process, with only one hyperparameter to tradeoff the performance and compression. BSQ achieves both higher accuracy and higher bit reduction on various model architectures on the CIFAR- 10 and ImageNet datasets comparing to previous methods.
quantization from new angle of inducing bit-level
INTRODUCTION
Numerous deep neural network (DNN) models have been designed to tackle real-world problems and achieved heyond-human performance. DNN models commonly demand extremely high computation cost and large memory consumption, making the deployment and real-time processing on embedded and edge devices difficult (Han et al. 2015b; Wen et al. 2016). To address this challenge, model compression techniques, such as pruning (Han et al. 2015b Wen et al., 2016 Yang et al., 2020). factorization (Jaderberg et al. 2014 Zhang al. 2015) and fixed-poin quantization (Zhou et al.) 2016 Wu et al.] 2019; Dong et al. 2019), have been extensively studied. Among them, fixed fixed-point quantization works directly the data representation by converting weight parameters originally in the 32-bit floating-poin form to low-precision values in a fixed-point format. For DNN model, its quantized version requires much less memory for weight storage. Moreover, it can better utilize fixed-point processing units in mobile and edge devices to run much faster and more efficiently.
Typically, model compression techniques aim to reduce DNN model size while maintaining its performance. The two optimization objectives in this tradeoff, however, have contrary nature: the performance can be formulated as a differentiable loss function L(W) w.r.t. the model's weights W: yet the model size, typically measured by the number of non-zero parameters or operations, is discrete function determined mainly by the model architecture. To co-optimize the performance and model size, some previous pruning and factorization methods relax the representation of model size as a differentiable regularization term R(W). For example, group Lasso (Wen et al., 2016) and DeepHoyer (Yang et al. 2020) induce weight sparsity for pruning, and the attractive force regularizer (Wen et al. 2017) ) and nuclear norm (Xu et al. 2018) are utilized to induce low rank. The combined objective L(W) R(W) can be directly minimized with gradient- based optimizer for optimizing the performance and model size simultaneously. Here, the hyperparameter a controls the strength of the regularization and governs the performance-size tradeoff of the compressed model.

---

Published as a conference paper at ICLR 2021
Unlike for pruning and factorization, there lacks a well-defined differentiable regularization term that can effectively induce quantization schemes. Early works in quantization mitigate the tradeoff exploration complexity by applying the same precision to the entire model. This line of research focuses on improving the accuracy of ultra low low-precision DNN models, e.g.. quantizing all the weights to or less bits (Zhou et al. 2016 ZZhang et al.) 2018), even to 1-bit (Rastegari al. 2016). These models commonly incur significant accuracy loss, even after integrating emerging training techniques like straight-through estimator (Bengio al. 2hou al.
2016 dynamic range scaling (Polino
et al. 2018) and non-linear trainable quantizers (Zhang et al. 2018) As different layers of a DNN model present different sensitivities with performance, a mixed precisire quantization scheme would be ideal for the performance-sine tradeoff (Dong et al. (211)). There have also been accelerator designs to support the efficient inference of mixed-precision DNN models (Sharma et al. 2018). However, to achieve the optimal layer-wise precision configuration, it needs to ex exhaustively explore the aforementioned discrete search space, the size of which grows exponentially with the number of layers. Moreover, the dynamic change of cach layer's ' precision cannot be formulated into differentiable objective, which hinders the efficiency of the design space exploration. Prior studies (Wu et al., 2019; Wang et al., 2019) utilize neural architecture search (NAS), which suffers from extremely high searching cost due to the large space of mixed-precision quantization scheme. Recently, Dong al. (219) propose to rank each layer based on the corresponding Hessian information and then determine the relative precision order of layers based on their ranking. The method, however, still requires to manually select the precision level for each layer.
Here, we propose to revisit the fixed-point quantization process from new angle of bit-level sparsity: decreasing the precision of fixed-point number can be taken as foreing one or few bits, most likely the least significant bit (LSB), to be zero; and reducing the precision of a layer is equivalent to zeroing out a specific bit of all the weight parameters of the layer. In other words, the precision reduction can be viewed as increasing the layer-wise bit-level sparsity. By considering the bits of fixed-point DNN parameters as continuous trainable variables during DNN training, we can utilize a sparsity-inducing regularizer to explore the bit-level sparsity with gradient- bbsed optimization, dynamically reduce the layer precision and lead to a series of mixed -reediiiiiisin quantization schemes. More specific, we propose Bit-level Sparsity Quantization (BSQ) method with the following contributions:
We propose a gradient based training algorithm for bit-level quantized DNN models. The algorithm considers each bit of quantized weights as an independent trainable variable and enables the gradient-based optimization with traight-through estimator (STE).
We propose a bit-level group Lasso regularizer to dynamically reduce the weight precision of every layer and therefore induce mixed-ppecisse quantization schemes.
BSQ uses only one hyperparameter, the strength of the regularizer, to trade-off the model performance and size, making the exploration more efficient.
This work exclusively focuses on layer-wise -eecci-preisoon quantization, which is the granularity considered in most previous works. However, the flexibility of BSQ enables it to explore mixed- precision quantization of any granularity with the same cost regardless of the search space size.
2 RELATED WORKS ON DNN QUANTIZATION
Quantization techniques convert floating-point weight parameters to low-precision fixed-point repre- sentations. Directly quantizing a pre-trained model inevitably introduces significant accuracy loss. So many of early research focus on how to finetune quamtized models in low-precision configurations. As the quantized weights adopt discrete values, conventional gradient-based methods that are designed for continuous space cannot be directly used for training quantized models. To mitigate this problem, algorithms like DoReFa-Net utilize a trruight-througe estimator (STE) to approximate the quantized model training with trainable floating-poin tooinn parameters (Zhou et al. 2016). As shown in Equation (I. a floating-poit weight element w is kept throughout the entire training process. Along the forward pass, the STE will quantize to n-bit fixed-pont point representation Wg. which will be used to compute the model output and loss L. During the backward pass, the STE directly pass the gradient w.r.t. Wy onto w, which enables w to be updated with the standard gradient-based optimizer.
Forward: Wy = 2" - Round (2 1)w); Backward: OL OC
1
(1)
Ow Bwy
2

---

Published as a conference paper at ICLR 2021
Early studies revealed that weights of different layers have different dynamic ranges. is important to keep the dynamic range of each layer for maintaining the model performance, especially for quamtized models. [e et al. (2016b) and Polino al. (2018) propose explicitly keep track of the dynamic range of each layer by scaling all the weight elements a layer to the range of [0,1] ] at every training step, before applying the quantization STE. Other techniques, such learnalee nonlinear quantifier function (Zhagg et al., 2018) and incremental quantization (Zhou et al., (017), are also useful in improving the performance of quantized models. However, is still very difficult to quantize the entire DNN model to unified ultra-low precision without incurring significant accuracy loss.
Recent research shows that different layers DNN model contribute to the overall performance in varying extents, Therefore mixed-precision quantization scheme that assigns different precision to layers (Wu et al., 2019; Dong et al. 2019) presents better eccuracy-compression tradeoff. The challenge lies in how to determine the quantization scheme, i.e., the precision of each layer, as it needs to explore large and discrete search space. Some works design quantization criteria based on concepts like "noise gain" (Sakr & Shanbhag 2018 2019) to constraint the relationship between each layer's precision and thus largely reduce the search space, yet those criteria are often heuristic, preventing these methods to reach ultra-low precision and find the optimal tradeoff point between model size and accuracy. Other works utilize neural architecture search (NAS). For example, Wang al. 2019) consider the precision assignment of each layer design policy via reinforcement learning. Wu et al. (2019) combine all possible design choices into a "stochastic super net" and approximate the optimal scheme via sampling. However, the cost of NAS methods scales up quickly as the quantization search space grows exponentially with the number of layers. Common practices of constraining the search cost include limiting the precision choices or designing the quantization scheme in a coarse granularity. A recent line of research work attempts to rank layers based on their importance measured by the sensitivity or Hessian information. Higher precision will then be assigned to more important layers (Dong et al. (019). The exact precision of each layer, however, needs to be manually selected. So these methods cannot adequately explore the whole search space for the optimal quantization scheme.
action and seek for the optimal
THE BSQ METHOD
BSQ aims obtain optimal mixed-precision quantization scheme through single-pass training process of a quantized model. In this section, we first introduce how to convert a DNN model to the bit representation and propose gradient-basee algorithm for training the resulted bit-level model. A bit-level group Lasso regularizer is then proposed to induce precision reduction. In the end, we elaborate the overall training objective of BSQ and the dynamical precision adjustment procedure.
3.1 TRAINING THE BIT REPRESENTATION OF DNN
As illustrated in Figure I a). we convert a floating-point weight matrix W of pretrained network to its bit representation through pipeline of scaling, quantization and binary conversion. Similar to the practice in (He et al., 2016b; Polino et al., 2018), we retain the dynamic range of W by scaling all the elements to the range of [0, I] before applying quantization. However, these prior works always to fully utilize all the quantized bins every training step, which scale the largest element
makes the dynamic precision reduction impossible. Instead, our method conducts the scaling only
Scaling
1.3 0.8] [0.2
R(0.2 x 1.3 0.8
[2.01
2.0
w/ (20)
STE forward
1.2
[1.0]
[
1 0
DNN FP E ((WW))
0.2
W,
0.6
(2.00))
STE backward
W
0.1
[0.0
0.7]
[0.4
0.2 0.1]
0.7*
dw,
Quantization
W
Binary conversion
Updated and trimmed w. (2.0)
dw, (2:0
DNN BP
(a)
(b)
Figure An example of DNN training under the bit representation with precision n = 3. (a) Pipeline of converting from the floating -oont weight W' to the bit representation; (b) Training the bit-level model weight with STE.
3

---

Published as a conference paper at ICLR 2021
once, which is right before the bit representation training. Formally, before converting W to its bit representation, we first extract its dynamic range as W = . Wa, where = max is the scaling factor and W. is the scaled weight matrix. The absolute value of any element w, in W, is within the range of [0, 1]. Now we apply an n-bit uniform quantization to the absolute value of w, such as wly Round w.. x(2" 1)) 1). Then Wy can be exactly represented by n-bit binary number as wa = 12 ws 2"1 - 1), where wa denotes the 5t bit in the binary representation.
Till this point, W in the floating- point form is replaced with
1
W = WW) @ sW, = sign () o
2
2nd
b=0
where @ denotes the element-wise Hadamard product. We consider the bit representation W!!) where b E [0, 1] and the scaling factor s as independent trainable variables in the training process.
Note that w is composed of binary values by definition and sign(W) is a discrete function. Neither of them can be directly trained with gradient descent. To mitigate the binary constraint of
W: we adopt the STE proposed by |Bngio et al. (2013) during the training process. As shown in Equation (T). STE enables quantized model to be trained with continuous floating- ppinn weights. Specifically, the STE for the bit representation training is defined as:
OC
26
dL
Forward: W = 2 - Round
Backward:
(3)
aw
2 OW, g
STE relaxes the binary constraint and allows gradient updates for the elements in w! As illustrated in Figure [b), during the forward pass, 8. W will be used to reconstruct the model weight W and compute the loss, which demonstrates the performance of the current model after quantization. The gradient w.r.t. W from the back procppgaaaatiion will be passed through the rounding function and updated on the continuous values of (( The proposed bit representation can therefore be trained with any gradient- based optimizer.
The proposed bit representation training only leads to minimal computational and run-time memory overhead comparing to the normal back propagation procedure. From the memory consumption perspective, the bit representation training treats each bit as separated floating-point trainable variables, so a N-bit model in bit representation will have N times more parameters and gradients to be stored comparing to that of the baseline training. Though for actual run-time memory consumption, the hidden feature between each layer consumes significantly larger memory than weights and gradients. As the bit representation does not affect the hidden features, the increase in trainable variables does not lead to significant increase in run-time memory consumption. From the perspective of computation cost, note that the gradient w.r.t. each w: can be computed as the gradient w.r.t. the corresponding Wg scaled by a power of 2. So under -ibit scheme there will only be N additional scaling for each parameter comparing to the normal training. These additional computations are very cheap comparing to the floating-point operations involved in back propagation. So the proposed bit representation training only leads to minimal computational overhead comparing to normal back propagation.
We restrict the value of (b))) within [0, 2] throughout the training, so that the corresponding W, has the chance to increase or decrease its precision in the "precision adjustment" step. which will be
discussed in Section 3.3 This is enforced by trimming (((b) to O or 2 if exceeds the range after training step.
To enable the dynamic update of sign(W) during training, we separate the positive and negative elements in W. as W, = (W, W,,) before quantization. Here W, = W, & 1(W, 2 0) contains all the positive elements and W,. = WW, @ 1 (W, & < 0) includes the absolute value of all the negative weight elements. Wp and W,, will be respectively converted to w/(h) and Wi(*) by following the process in Equation (2) so that w-(0) w/ w Note that the replacement of ((()) with w:(") ( does not introduce any non-differentiable function. Therefore all elements in W and w take continuous values between [0, 2] and be trained with the bit representation STE in Equation (3.. As such, the original weight matrix W is converted into trainable variables W (() and throughout the BSQ training process.
4

---

Published as a conference paper at ICLR 2021
3.2 BIT-LEVEL GROUP LASSO
To induce the mixed-precision quantization scheme of a DNN model during training, we propose bit-level group Lasso (Ber. regularizer based on the group Lasso (Hastie al. 2015) and apply it to wi and wil that are converted from group of weights W The regularizer is defined as:
(c
Σ W P
where ( and w!(h) are bit representations converted from W and [:: denotes the concatenation of matrices. BGL could make certain bit b of all elements in both W/)) and W! zero simultane- ously. The bit can thus be safely removed for the precision reduction. Note that the granularity of the quantization scheme induced by BeL is determined by how W is grouped. Our experiments organize W in a layer-wise fashion. So all elements in layer have the same precision, which common setting in previous mixed-precision quantization work. W can also be arranged as any group of weight elements, such as block-wise, filter-wise or even element-wise if needed. Accordingly, the formulation of the regularizer need to revised to assist the exploration of the mixed preccisooeiiionn quantization at the given granularity. The cost for evaluating and optimizing the regularizer will remain the same under different granularity settings.
3.3 OVERALL TRAINING PROCESS
The overall training process starts with converting each layer of pretrained floating-point model to the bit representation with relatively high initial precision (e.g.. 8-bit fixed-point). BSQ training then preformed on the achieved bit representation with bit-level group Lasso integrated into the training objective. Re-quantization steps are conducted periodically to identify the bit-level sparsity induced by the regularizer and allow dynamic precision adjustment. As the mixed-precision quantization scheme is finalized, the achieved model is further finetuned for higher accuracy.
Objective of BSQ training. For higher memory efficiency it is desired find mixed-precistee quantization scheme that minimizes the total number of bits in the model. Thus, in BSQ training we propose to penalize more on the layers with more bits by performing a memory consumption-avesd reweighing to BeL across layers. Specifically, the overall objective of training a L-layer DNN model with BSQ is formulated as:
L = (= +a # #Para(W") x #Bi(WW")
#Para(W(aLL) BGL(W!).
Here LcE (w is the original cross entropy loss evaluated with the quantized weight Wg acquired from the STE in Equation 3.. is hyperparameter controlling the regularization strength, and #Para(W') and # Bi((W" respectively denote the parameter number and precision of layer I. The loss function in Equation (5) enables layer-wise adjustment in the regularization strength by applying a stronger regularization on layer with higher memory usage.
Re-quantization and precision adjustment. As BSQ trains the bit representation of the model with floating-poit variables, we perform re-quantization to convert wrSh) and wi(h) to exact bi- nary values and identify the all-zero bits that can be removed for precision reduction. The re- quantization step reconstructs the quantized scaled weight W from Wr(C) and w(()) as: W
Round Exz (0) 2b Σn= (0)) As we allow the values of w-(h) and w-(b) to be within [0, 2]. the reconstructed W N has maximum absolute value of 2 In this way, W is converted to a (n + -bit binary number, where each bit is denoted by w>(b) After the re-quantization, we will adjust the precision of each layer. Specifically, we first check ((()) from the MSB down the LSB and remove the bits with all zero elements until the first non-zero bit. The scaling factor of the layer remains unchanged during this process. similar check will then be conducted from the LSB up to the MSB. needs to be doubled when bit from the LSB side is removed, as all elements in W' are shifted right for one bit. Assume that the precision adjustment makes the precision of a layer change from n', the scaling factor will be updated s' 2d
In this way, the bit representations 2
5

---

Published as a conference paper at ICLR 2021
40,000
16,000
Bit Precision
50,000
25,000
PPrrame
10,000
10,000
0
layer cocol1
cond
layer nxcomn
liver L.Laani
layer' Lansd
layert comi
layert mer..2.. zcnuz
layer 1ayr2...oovi
leyer 2.co2.0.ood
Syyer Loawwi
layer Lyye7...aod Laann
laye2 2.ood
laver S d.onn cand
layer 3.0 A.concc
layers hyer3.1.. Leanoii
layer Leaww layera 2.2oll laver lnver3.2.omd
laver2
With reweighing: Comp 14.24x, Acc 92. 32%
Without reweighing: Comp 10. 86,, Acc 91.87% .e.-PParam
Figure 2: Quantization schemes achieved with or without layer-wise regularization reweigh-
in. The compression rate and the accuracy after finetuning are listed in the legend.
of W before and after the precision adjustment are equivalent, indicated in Equation ( The precision a n-bit layer may change between and (n + 1)t after the precision adjustment.
W
W 2
w
(6)
2
2
6=
b=0
As formulated in Equation (5). the regularization strength assigned each layer will change with the quantization scheme of the model. The re-quantization and precision adjustment step will be performed periodically during the training process, with an interval of several training epochs. After each precision adjustment, we separate the positive elements and negative elements in W to form
the new w and W respectively. The training can then resume with the newly adjusted W/()) and ( and scaling factor s'. is worth mentioning that sW, from the forward pass STE remains unchanged before and after the re-quantization and precision adjustment, so the model performance and the gradient from the loss LCE will not be affected. The interval between re-quantizations needs to be carefully chosen: it shall promptly and properly adjust the regularization strength for stable convergence. The ablation study on re-quantization interval selection is presented Appendix B.1
Activation quantization. Since BSQ modifies only the precision of weights but not affecting the precision of activations, we predetermine the activation precision and fix throughout the BSQ training process. The activations For training stability, we use ReLU-6 activation function for layers with 4-bit or above activations, and use PACT (Choi et al.) 2018) for layers with lower activation precision.
quantized in the same way proposed by PPlino et al. (2018).
Post-training finetuning. At the end of the BSQ training, we perform a final re-quantization and precision adjustment to get the final mixed-quantization scheme. The achieved model can be further finetuned under the obtained precision for improving the overall accuracy. As the quantization scheme is fixed, we adopt the quantization-awe training method proposed by Polino et al. (2018) for finetuning in our experiment.
4 ABLATION STUDY
We perform the ablation studies on key design choices of the BSQ algorithm. This section presents the effectiveness of layer-wise memory consumption-awwan regularization reweighing and the model size-accuracy tradeoff under different regularization strengths. All experiments are conducted with ResNet-20 models (He al. 2016a) with 4-bit activation on the CIFAR-I0 dataset (Krizhevsky & Hinton 2009.. Detailed experiment setup and hyperparameter choices can be found in Appendix A
4.1 EFFECT OF MAYER-WISE REGULARIZATION REWEIGHING
As stated in Equation (5). we propose to apply layer-wise memory consumption-aware reweighing on the Ber. regularizer to penalize more larger layers during the BSQ training. Figure compares comparess the quantization scheme and the model performance achieved when performing the BSQ training with or without such reweighing term. Here we set the regularization strength to 5e-3 when training with the reweighing, and to 2e-3 when training without the reweighing to achieve comparable compression rates. All the other hyperparameters kept the same. As shown in the figure, training without

---

Published as a conference paper at ICLR 2021
Bit Precision
3e-3
5
5e-3
7e-3
1e-2
layer accome cond
lyer 1.com
layer Lccovri layar Leamd
layeri layeri.2.oo L.Z.awvl
layerd Z.oon
layera dccomel
layer2 accond
layer layer2. 1.yer2...coml Lconci Lconci
layer ZaarZii coom
layer? Z.Z.oml
layera dacovt
layera accoma
layers 3 I.aoed
layer3 Leomz
3ayer3 2.coni
laye Z.coma
2e-2
layarz
Figure Layer-wise precision comparison of the quantization schemes achieved under different regularization strengths.
Accuracy-#Bil tradeoff under different regularization strengths. "FT"" stands for finetuning. The last row is achieved by training with quantization schemes achieved by BSQ from scratch.
Table I:
Strength
3e-3
5e-3
7e-3
le-2
2e-2
#Bits per Para Comp (x)
3.02 / 10.60
2.25 / 14.24
1.66/
19.24
1.37 1.37/3..4 23.44
0.87 36.63
BSQ acc before after FT (%)
91.30 / 92.60
90.98
92.32
90.42 / 91.48
90.35
/91.6
85.77
89.49
Train from scratch acc (%)
91.72
91.45
91.12
89.57
89.14
the reweighing term will lead wer-penalization earlier layers with fewer parameters, while later layers with more parameters are not compressed enough. Therefore the achieved quantized model will have less accuracy even with smaller compression with layer-wise regularization reweighing. As we only show one pair of comparison here, the difference between BSQ training with or without the reweighing term is consistent when varying the regularization strength a. Additional results with other a values are shown in Appendix B.2
comparing to the model achieved
4.2 ACCURACY-#BITS TRADEOFF UNDER DIFFERENT REGULARIZATION STRENGTHS
We fix all the other hyperparameters while varying only the regularization strength a from 3e- 3 2e-2, to control the tradeoff between the model size and accuracy achieved by BSQ. The quantization schemes achieved by running BSQ with different a's are shown in Figure 3 and the detailed comparison
the compression rate comparing the 32-bit floating point model (denoted as "Comp") and the validation accuracy (denoted as "Acc") is summarized in Table As shown in Figure 3 the relative ranking of the precision assignment mostly consistent under different a's which consistent with the previous observation that more important layers should be assigned with higher precision. This effect further illustrated in Appendix B.3] where we compare the quantization scheme achieved by BSQ with the layer importance measured in HAWQ (Dong et al. 2019). Furthermore, as a increases, the overall bit reduction increases with cost of small performance loss. This tradeoff is also observed on models trained with 2-bit or 3-bit activation show their guantization schemes and performances in Appendix B.4 Note that some layers achieve O-bit precision under large regularization strength, indicating all the weights become zero and the layer can be skipped. This is possible as the shortcut connection existing in the ResNet architecture enables the pass of information even if the weights all zero in some layers. We also note that BSQ not only finds the desired mixed-precision quantization scheme, but also provides model with higher performance under the same quantization scheme. As shown Table [ with the same guantization scheme as achieved by BSQ using the DoReFa-Ner Net algorithm (Zhou et al. 2016) from scratch, the resulted accuracy always lower than the BSQ model after finetuning.
when training a model
EXPERIMENTAL RESULTS
In this section we compare BSQ with previous state-of-the art methods. Here, ResNet-20 models are used for the comparison on the CIFAR-10 dataset, and ResNet-50 and Inception- V3 models (Szegedy) jet al., (016) are utilized for the experiments on the ImageNet dataset (Russakovsky et al.) (2015). The hyperparameters used for BSQ training and finetuning listed in Appendix|A All the compression
7

---

Published as a conference paper at ICLR 2021
Table Quantization results of ResNet-20 models the CIFAR-10 dataset. BSQ compared with DoReFa-Net (Zhou et al. 2016), PACT (Choi et al. 2018), LQ-Net (Zhang et al. 2018). DNAS (wu
a.. 2019 and HAWQ Dong et al.
2019). "MP" denotes mixed-precision quantization.
Benchmarks
BSQ
Act. Prec.
Method
Weight Prec.
Comp (x)
Acc (%)
Comp (x)
Ace (%)
32-bit
Baseline
32
1.00
92.62
LQ-Nets
3
10.67
92.00
5e-3
14.24
92.77
DNAS
MP
11.60
92.72
7c-3
19.24
91.87
LQ-Nets
2
16.00
91.80
4-bit
HAWQ
MP
13.11
92.22
5c-3
14.24
92.32
3-bit
LQ-Nets
3
10.67
91.60
2e-3
11.04
92.16
PACT
3
10.67
91.10
5e-3
16.37
91.72
DoReFa
3
10.67
89.90
2-bit
LQ-Nets
16.00
90.20
PACT
2
16.00
89.70
Se-3
18.85
90.19
DoReFa
2
16.00
88.20
Table Quantization results ResNet-50 and Inception- V3 models on the ImageNet dataset. BSQ
compared with DoReFa-Net (Zhou al.) 2016), PACT (Choi al. 2018), LSQ (Esser et al.,
2019
LQ-Net (Zhang et al. 2018), Deep Compression (DC) (Han et al. 2015a), Integer (Jacob et al. 2018
RVQ (Park et al. 2018). HAQ (Wang al. 2019 and HAWQ Dong al. 2019).
ResNet-50
Inception-V V3
Method
Prec.
Comp (x)
Top1 (%)
Method
Prec.
Comp (x)
Top1 (%)
Baseline
32
1.00
76.13
Baseline
32
1.00
77.21
DoReFa
10.67
69.90
Integer
4.00
75.40
PACT
3
10.67
75.30
Integer
7
4.57
75.00
LQ-Nets
3
10.67
74.20
RVQ
MP
10.67
74.14
DC
3
10.41
75.10
HAWQ
MP
12.04
75.52
HAQ
MP
10.57
75.30
LSQ
3
10.67
75.80
BSQ Se-3
MP
11.90
75.29
BSQ le-2
MP
11.38
76.60
BSQ 7e-3
MP
13.90
75.16
BSQ 2e-2
MP
12.89
75.90
rates reported Table and Table 3 are compared to the 32-bit floating point model, and all the accuracy reported is the testing accuracy evaluated on models after finetuning.
Table reports the quantization results of ResNet-20 models on the CIFAR- 10 dataset. Here we the activation of the first convolutional layer and the final FC layer bits while all the other activations to 4, 3 or bits respectively to match the settings of previous methods. The reported 32-bit activation model performance is achieved by finetuning the 4-bit activation model under full precision activation. The exact BSQ quantization schemes of the 4-bit activation models are listed in Figure 3 while those of the 2-bit and 3-bit activation models be found Appendix B.4 Comparing to previous mixed- precision quantization methods, the model obtained by BSQ with 4-bit activation and
5e-3 has slightly higher accuracy as the model achieved by HAWQ (Dong et al. 2019), but higher compression rate (14.24x vs. 13.11x). The same model with 32-bit activation obtains 23% more compression rate with the same accuracy as the model found by DNAS (Wu et al., 2019), with much less training cost
method does not involve the costly neural architecture search. The advantage of BSQ is even larger comparing to single-precision quantization methods (Zhou et al.) 2016, Choi al. 2018 Zhang al. 2018). BSQ achieves both higher compression rate and higher accuracy comparing to all methods with the same activation precision.
The results of BSQ and previous quantization methods the ImageNet dataset summarized in T The exact BSQ quantization schemes can found in Appendix C For ResNet models, the activation of the first and the final layer are set to 8 bits while all the other activations set
8

---

Published as a conference paper at ICLR 2021
bits. For Inception- V3 models the activation of all the layers set to 6 bits. On ResNet-50 models, BSQ with a = 5e-3 achieves the same top-1 accuracy as PACT (Choi et al. 2018) and 0.5% less top-1 accuracy the best available method LSQ (Esser et al. 2019) with a higher compression rate (11.90x vs. 10.67 x ), showing competitive accuracy-compressee
tradeoff. BSQ can further increase the compression rate of ResNet-50 to 13.90 with a 7e-3, with only 0.13% top-1 accuracy loss over the "Se-3" model. On Inception-V3 V3 models, BSQ with = 2e-2 achieves both higher accuracy (75.90% vs. 75.52%) and higher compression rate (12.89 12.04x) comparing the best previous method HAWQ (Dong et al., 2019). Adopting smaller a = le-2 makes BSQ to achieve 0.7% accuracy improvement trading off 100% less compression rate comparing to the "2e-2" model.
6 CONCLUSIONS
In this work, we propose BSQ, which fully explores the accuracy-modes size tradeoff of DNN's mixed-precision quantization schemes with differentiable training algorithm using DNN's bit representation as trainable variables. A bit-level group Lasso regularizer with memory consumption- aware layer-wise reweighing is applied to induce bit-level sparsity, which leads the dynamic adjustment of each layer's precision and finally a mixee-precisioon quantization scheme through a single-pass gradient-based training process. This enables BSQ dynamically produce series of quantization schemes trading off accuracy and model size and provides models with higher accuracy comparing to training from scratch under the same quantization scheme. We apply BSQ in training ResNet-20 models on the CIFAR-10 dataset and training ResNet-50 and Inception- V3 models on the ImageNet dataset. all the experiments, BSQ demonstrates the ability to reach both a better accuracy and higher compression rate comparing previous quantization methods. Our results prove that BSQ can successfully fill in the gap of inducing a mixed- -eexediireiiion quantization scheme with a differentiable regularizer, to effectively explore the tradeoff between accuracy and compression rate for finding DNN models with both higher accuracy and fewer bits.
ACKNOWLEDGMENTS
This work is supported in part by NSF CCF-1910299 and NSF CNS-1822085.
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1308.3333 2013.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce 1-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 8605.0685, 2018.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer, Hawq: Hessian aware quantization of neural networks with mixed-precision In Proceedings of the IEEE International Conference on Computer Vision, pp. 293-302, 2019.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint 190.0119,,,, 2019.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arxiv: 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135 1143, 2015b.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity: the lasso and gemeralizations. CRC press, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016a.
Qinyao He, He Wen, Shuchang Zhou, Yuxin Wu, Cong Yao, Xinyu Zhou, and Yuheng 7ou. Effective quantization methods for recurrent neural networks. arXiv preprint arXiv: 161 1. 10176, 2016h.

---

Published as a conference paper at ICLR 2021
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmen only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2704 2713, 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3888 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Eunhyeok Park, Sungjoo Yoo, and Peter Vajda. Value-aware quantization for training and inference of neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 580-595, 2018.
A. Polino, R. Pascanu, and D. Alistarh. Model compression via distillation and quantization. arXiv preprint 0rXiv:1802.05000 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer. 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2 2015. doi: 10.1007/s1 263-015-0816-1
Charbel Sakr and Naresh Shanbhag. An analytical method to determine minimum per-layer precision of deep neural networks. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 1090- 1094. IEEE, 2018
Charbel Sakr and Naresh Shanbhag. Per-tensor fixed-point quantization of the back-propagation algorithm. In International Conference on Learning Representations, 2019. URL https://openreview n Forum? didnkiidaaaN AA9ml A9Ym
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmacilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 764-775. IEFE, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton, On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1439-1147, 2013.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-2668,, 2016.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612 8620, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 658-666. 2017.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-an aware aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10722, 2019.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. arXiv preprint arXiv: 1812.02222 2018.
Huanrui Yang. Wei Wen. and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale- invariant sparsity measures. In International Conference on Learning Representations, 2020. URI. httpp 7/apenreve net met//oormmiddrye 3K44DD
10

---

Published as a conference paper at ICLR 2021
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 065-382,2 2018.
Xiangyu Zhang, Jianhua Zou, Kaiming He. and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10): 1943-1955, 2015.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cmns with low-precision weights, arXiv preprint arXiv:1702.0300 03044, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou, Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1606 2016.

---

Published as a conference paper at ICLR 2021
A HYPERPARAMETER CHOICES IN THE EXPERIMENTS
A.1 CIFAR-10 10 EXPERIMENTS
We use Re models the CIFAR-10 10 dataset (Krizhevsky & Hinton 2009) to do all of our ablation studies and evaluate the performance of BSQ. The CIFAR- 10 dataset can be directly accessed through the dataset API provided in the "torchvision" python package. We do not change the splitting between the training and the test set. Standard preprocessing procedures, including random crop with padding of 4, random horizontal flip and normalization, are used the training set to train the model. The validation set is normalized with the same mean and variance as the training set. We implemented ResNet-20 models following the description in (He et al. 2016a), and pretrain the model for 350 epochs. The learning rate is set to 0.1 initially, and decayed by 0.1 at epoch 150, 250 and 325. The weights of all the layers except the batch normalization are then quantized to 8-bit before the BSQ training. The batch normalization layers are kept in the foating-poin format throughout the training process. Similar to previous quantization works, we also apply the activation quantization during the training. For 4-bit or above activation precision we replace all the ReI.U activation function in the model with the RcLU6 activation function. For lower activation precision we use the trainable PACT activation (Choi et al. 2018) with weight decay 0.0001. These changes will help achieving higher accuracy and better training stability when the activation is quantized as it eliminates extremely large activation values. As BSQ does not consider activation quantization an objective, we fix the activation precision throughout the BSQ training and the finetuning process.
We start the BSQ training with the 8-bit quantized pretrained model following the process described in Section 3.3 The BSQ training is done for 350 epochs, with the first 250 epochs using learning rate 0.1 and the rest using learning rate 0.01. Unless otherwise specified, the re-quantization and precision adjustment is done every 100 epochs, as well as after the BSQ training is finished to adjust and finalize the quantization scheme. Different regularization strengths a are tried explore the tradeoff between accuracy and compression rate. The exact a used for each set of experiment is reported alongside the results in the main article. For comparing with previous methods, we further finetune the achieved mixxd-ppecision model with the DoReFaa-Ntt algorithm (Zhou et al.] (016)) while fixing the quantization scheme. The finetuning is performed for 300 epochs with an initial learning rate 0.01 and the learning rate decay by 0.1 at epoch 150 and 250. The "train from scratch" accuracy reported in Table is achieved by first quantizing a pretrained Mooting-poin model to the mixed precision quantization scheme achieved by BSQ. then performing DoReFa- Net quantization aware training on the model. The training is done for 350 epochs, with an initial learning rate 0.1 and the learning rate decay by 0.1 at epoch 150, 250 and 325. All the training tasks are optimized with the SGD optimizer (Sutskever et al. 2013) with momentum 0.9 and weight decay 0,0001, and the batch size is set 128. All the training processes are done on single TITAN XP GPU.
A.2 IMAGENET EXPERIMENTS
The ImageNet dataset is used to further compare BSQ with previous methods in Table 3 The ImageNet datasct is a large-scale color-image dataset containing 1.2 million images of 1,000 categories (Russakovsky et al. 2015). which has long been utilized as important bench- mark on image classification problems. In this paper, we use the "ILSVRC2012" version of the dataset, which can be found at http::////// / www. image-net..... org/ cha 1 Lengee LSSRCC downloade We use all the data in the provided training set to train our model, and use the provided validation set to evaluate our model and report the testing accuracy. We follow the data reading and preprocessing pipeline suggested by the official PyTorch ImageNet example. For training images, we first perform the random sized crop on the training images with the desired input size, then apply random horizontal flipping and finally normalize them before feeding them into the network. We use an input size of 224 224 for experiments on the ResNet-50, and use an size of 299 x 299 for the Inception- V3 experiments. Validation images are resized then center cropped to the desired input size and normalized before used for testing. For both the ResNet-50 and the Inception-V3 model, the model architecture and the pretrained model provided in the "torchvision" package are directly utilized. The first convolutional layer of the Re model and the first 5 conventional layers of the Inception-V3 V3 model quantized to 8-bit, while all the other layers are quantized to 6-bits before the BSQ training. Similar to the CIFAR- 10 experiments, the batch normalization layers are kept as floating-point.
12

---

Published as a conference paper at ICLR 2021
* Int 20
Test accuracy (%6)
02
* Int 50
01.5
Int 100
* No requant)
80.5
#Bits reduction rate (x)
Figure 4: Range of testing accuracy and bit reduction rate achieved from 5 repeated runs with different random seeds. Solid line links the average performance, error bar marks the maximal and minimal performance achieved with each set of hyperparameters.
We start the BSQ training with the quantized pretrained model. For both ResNet-50 and Inception- -3 models, the BSQ training is done for 90 epochs, with the first 30 epochs using learning rate 0.01 and the rest using learning rate 0.001. The re-quantization interval is set to 10 epochs for all the ImageNet experiments. The regularization strength a used is reported alongside the results in Table 3 The model after BSQ training is further finetuned with DoReFa-Ner for 90 epochs, with the initial learning rate 0.001 and a learning rate decay by 0.1 after 30 epochs. All the models are optimized with the SGD optimizer with momentum 0.9 and weight decay 0.0001, and the batch size set as 256 for all the experiments. Two TITAN RTX GPUs are used in parallel for the BSQ training and finetuning of both ResNet-50 and Inception- V3 models.
B ADDITIONAL ABLATION STUDY RESULTS
B.1 CHOICE OF RE-QUANTIZATION INTERVAL
We propose the myer-wise regularization reweighing in Section 3.3 and show its importance in Section 4.1 This reweighing can be more effective if we adjust the precision of each layer reg- ularly throughout the BSQ training routine. The precision adjustment done through periodic re-quantization. From the one hand, a smaller re-quantization interval would help the precision to be adjusted in n-timme. From the other hand, may cause the training unstable due to the frequent change in bit representation and regularizer values. So here we gradually increase the re-quantization interval to find the best choice that can reach high and stable performance. Figure F demonstrates the stability and performance under re- re-quantization intervals 20, 50, 100 and compare them with the performance achieved without re- re-quantization during the training. Each point in the figure corresponds to the averaged compression rate and accuracy after 5-time repeated BSQ training with fixed regularization strength a but with different random seeds. The observation in the figure supports our analysis that as re-quantization is important to reach better accuracy-# # bits tradeoff, applying too frequent will make the training unstable and hinders the overall performance. Comparing to not performing the re-quantization and applying every 20 epochs, re-quantizing every 50 or 100 epochs yields similarly better tradeoff between accuracy and compression rate. Re-quantization interval 100 leads to a higher accuracy in a wider range of compression rate comparing to the Int 50 model, and the performance is more stable throughout the repeated trails. Therefore in all the other CIFAR-10 experiments we set the re-quantization interval to 100 epochs.
B.2 ADDITIONAL RESULTS ON REGULARIZATION REWEIGHING
Figure 5 and Figure 6 compares the quantization scheme and the model performance achieved when performing the BSQ training with or without the memory consumption-awe reweighing of the bit-level group Lasso regularizer under additional choices of regularization strength a. The a used for each set of experiment reweighing. The a used are listed in the caption of the figures. All the other hyperparameters are kept the same. From both figures observe consistent trend that training without the reweighing
chosen that comparable compression rates are achieved with without
13

---

Published as a conference paper at ICLR 2021
40,000
Bit Precision
30,000
#Parameter
20,000
10,000
lyye1. comii
conrs
liver c0.02
liyer Loaml
laper comi
laver cont1
come
layer w. L2coodd
lever 2.2.amr
lyyer c.am..
layer Loomn
layer Loonz
layer 2come
layer 2.mm222.owv 2oow2
tayer comdi
laye layer layer coom
commz comdi
comit
laver &. 2.cooei
lever 2.am2
With reweighing: Comp 15x Acc 91.90%
Without reweighing: Comp 15.02x, Acc 90.82% Paarm
Figure 5: Quantization schemes achieved with or without layer-wis regularization reweigh-
ing. The compression rate and the accuracy after finetuning are listed in the legend.
a =6e-3 with reweighing and a =3e-3 without reweighing.
40,000
Bit Precision
30,000
#Parameter
20m
10,m
lyye1...com cocovii
conet
leve comm2
loyer cooml
camel
layer comi
laver1 com't
layer LZoom2
lever 2.mmm Donmv
layerZ0. camva
layer coone
kaye2 convz
layer2.2com 2 dcomel
tayer c.00m Zcom2
layer com2
layer comdt
laye comt
laver coowi Zonmi
laver.22.omm 2.0m2
layer
With rehngghing: Comp 27. 27.06x, Acc 90.45%
Without reweighing: Comp 25,39x, Acc 88.47% Paarm
Figure 6: Quantization schemes achieved with or without layer-wise regularization reweigh- ing. The compression rate and the accuracy after finetuning are listed in the legend. a 0.015 with reweighing and =5e-3 without reweighing.
term will lead to less precision assigned to earlier layers with fewer parameters, while later layers with more parameters are not compressed enough. Therefore the achieved quantized model will have less accuracy and smaller compression rate comparing to the model achieved with layer-wise regularization reweighing. This observation is consistent with the results shown in Section 4.1 All these results show that the memory consumption-awe awarre reweighing proposed in BSQ training is crucial for generating models with both higher compression rate and higher accuracy.
B.3 QUANTIZATION SCHEME COMPARISON WITH HAWQ
As discussed in Section and shown in Figure 3 for the same model architecture the relative ranking of the precision assignment by BSQ is mostly consistent under different a's. Here compare quantization schemes achieved by BSQ with the "layer importance ranking" measured in HAWQ (Dong et al. 2019) further analyze this consistency. HAWQ proposes rank all the layers in the model with an importance score S, = >//n,, where X; denotes the top eigenvalue of the Hessian matrix of layer and n; represents the number of parameters layer i. A layer with higher S; will be assigned with higher precision in the mixed-precist quantization scheme. The quantization schemes achieved by BSQ and HAWQ are compared Figure where the black dotted
3e-3
Bit Precision
5
Se-3
7
1e-2
Ze-2
layer Laper...ccov accoml
convi
laeek lccomuz
lyeri Azoovi
laeer L.ccavii Loar2
layer2 layer 1ayer12.comd L.cooel
dccondl
layec? 1.ooo 1.com2
Jayer Hayer.. Lccond
Jayer Z.covi
laver2.2.om 2.cov222comz 2.conZ
layer3 layer acconel
layer Lccovt
layer3 Loomd
layor3 Z.convi
layer 2.comzz
HAWQ
layer2
layer
Figure Layer-wise precision comparison between the quantization schemes achieved with BSQ and the scheme achieved with HAWQ (Dong et al. 2019) on the ResNet-20 model.
14

---

Published as a conference paper at ICLR 2021
Bit Precision
5
1e-3
2e-3
3e-3
5e-3
layer accome cond
byer 1.com
layer L.coari layor Leamd
layert layeri.2.o L.Z.awv
layerd Z.oon
layerz dccomet
layer2 accont
layer.. 1.yer2...coml Lconri Lconci
layer Z.aaZii coom
layer3 layer? layer2.2.oo 2.Z.oml
Gxcovt
layar3 facomzz
layerS Icanel
layar3 Lecmzz
layer3 2.coork
laye'3 Z.coma
layerz
layer
Figure Layer-wise precision comparison of the quantization schemes achieved under different regularization strengths with 2-bit activation.
Table 4: Accuracy-# #Bits tradeoff with 2-bit activation. "FT" stands for finetuning.
Strength
le-3
2e-3
3e-3
5e-3
#Bits per Para Comp (x) BSQ acc before / after FT (%)
3.77 / 8.48
2.86 11.20
2.26 14.13
1.70 / 18.85
91.03 / 91.21
90.19 /90.70
89.54 / 90.39
88.13 / 90.
line shows the HAWQ scheme while the color solid lines demonstrate the schemes achieved by BSQ under different a. It can be observed that the relative ranking of BSQ's precision is consistent with the ranking of precision in HAWQ, which some extent shows that BSQ can dynamically identify the important layers during the training and assign higher precision to them. Note that HAWQ can only come up with the precision ranking each layer, while the exact precision is designed manually. BSQ on the other hand is able to explicitly assign the precision to each layer during single training process, and can dynamically tradeoff model size and accuracy by only changing a. Thus BSQ can easily find better tradeoff points with both higher accuracy and higher compression rate comparing to HAWQ and other quantization methods, as discussed in Section
B.4 MODELS ACIIEVED UNDER DIFFERENT ACTIVATION PRECISION
As we discuss the BSQ quantization schemes and model performances under 4-bit activation in Figure 3 and Table here we show the tradeoff between model size and accuracy under different regularization strength a with 2-bit activation Figure g and Table as well as those with 3-bit activation in Figure 9 and Table In both cases observe that the relative ranking of the precision assignment is mostly consistent under different s. As a increases, less bits are assigned to each layer, leading increasing overall bit reduction with the cost of small performance loss. This tradeoff consistent with our previous observations on the 4-bit activation models.
C DETAILED QUANTIZATION SCHEMES FOR IMAGENET EXPERIMENTS
The quantization schemes the reported ResNet-50 and Inception- V3 models can be found in Table Table 7 respectively.
Bit Precision
2e-3
Se-3
8e-3
1e-2
loyer come
comd
lover com2
layer 1.1 comrl
layor I.I. cnm2
layer 1.2 conrt
layer2 laver2 1.yer1.2 cnee
l cunut
Denm2
layer2. Lonri
layor2.1.om Lemd
layer 2.ccmri
layert.1 averd
layer3 covoir
layer3 crout
layer' conei
layor3. 1.1. Lenzz
1.ver3.7 1.2 coori
layer3. coma Loma
Figure 9: Layer-wise precision comparison of the quantization schemes achieved under different regularization strengths with 3-bit activation.
15

---

Published as a conference paper at ICLR 2021
Table 5: Accuracy #Bits tradeoff with 3-bit activation. "FT" stands for finetuning.
Strength &
2c-3
5e-3
8e-3
le-2
#Bits per Para Comp (x) BSQ acc before after FT (%)
2.90 11.04
1.95 16.37
1.39
23.04
/ 25.06
90.45 / 92.16
90.44 99.722
89.01 / 90.93
88.41 / 990.1
Table 6: Quantization schemes of ResNer-50 models on ImageNet dataset achieved by BSQ in Table 3 The scheme on the left is achieved with 5e-3 and the one on the right is achieved with a 7e-3. Except the first row for the leading convolutional layer and the last row for the FC layer, each row in the table reports the precision assigned the 3 layers a residual block, with layer listed from left to right.
BSQ 5e-3
BSQ 7
Conv
Black 1-0
7
6
6
Block
Block 1-2
6
6
6
5
Block
3
3
Block 2-1
4
4
4
3
Block 2-2
4
4
4
4
3
4
Block 2-3
4
3
3
Block 3-0
4
3
4
3
Block 3-1
3
3
3
Block
3
3
Block 3-3
3
3
3
2
Block 3-4
3
3
3
3
3
Block 3-5
3
3
Block 4-0
3
2
3
2
Block 4-1
2
2
2
2
Block 4-2
FC
Table Quantization schemes of Inception- V3 model on ImageNet dataset achieved by BSQ Ta- 33 The scheme on the left is achieved with 1e-2 and the one on the right is achieved with 2c-2. Except for the first convolutional layers and the final FC layer, each row the table reports the precision assigned to the layers within the inception block. The order from left to right follows the parameter definition order provided in the torchvision package implementation /
BSQ le-2
BSQ 2e-2
Cony la
Conv 2a
Conv 2b
Conv
Cony 4a
Mixed
4
3
3
3
Mixed 5c
4
3
3
4
3
3
3
Mixed 5d
4
4
4
4
3
4
4
3
4
3
3
4
Mixed 6a
4
4
3
4
3
Mixed 6b
4
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
Mixed 6c
4
3
4
3
3
3
3
3
3
3
2
Mixed
3
3
3
2
Mixed 6e
4
3
3
3
3
3
4
3
2
3
3
3
3
3
3
Mixed
4
2
Mixed 7b
3
3
3
2
2
3
3
2
3
2
1
2
2
Mixed 7c
2
3
3
3
2
3
2
2
3
2
2
3
2
PC
16


Reviews:
[{"review_id": "vNIUUt6xMP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to implement each layer with different precision (mixed-precision quantization). The method employed is referred to as bit-level sparsity quantization whereby each bit of the parameter set is treated as a trainable parameter. A differential bit sparsity regularizer enables a smooth trade-off between accuracy and complexity/compression. \n\n1) results are a slight improvement over SOTA methods. This is to be expected given the maturity of this topic.\n2) typo in Table 2 (\"wight precision\")\n\n3) It will be good to relate this work to [1] that also studies mixed-precision quantization using a pre-trained floating point network.\n\n[1] Sakr et al., An analytical method to determine minimum per-layer precision of deep neural networks.\n\nOverall a well-written paper with a solid reasoning behind the work. The results improve marginally over SOTA methods. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "rGrCw41wlM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quantization of weights in DNNs is a very effective way to reduce the computational and storage costs which can enable deployment of deep learning at the edge. However, determining suitable layer-wise bit-widths while training is a difficult task due to the discrete nature of the optimization problem. This paper proposes to utilize bit-level sparsity as a proxy for bit-width and employ regularization techniques to formulate the problem so that precision can be reduced while training the model.\n\nThe method proposed by the authors is sound. It leverages insights that have been employed in a neighboring area (pruning via regularization) and re-purposes those to the problem of quantization. The empirical evaluation is robust as well. \n\nOne issue I have with the proposed method is that the parameter space is expanded by a large amount. Since for every scalar weight, we end up with a collection of binary weights. Doesn't this make training more difficult? It would be nice to discuss this issue. And more importantly how does the extra effort compare to other approaches (such as Dorefanet and others).\n\nRegarding the proposed regularization technique. Lasso (least absolute shrinkage and selection operator) is, as far as I am aware, an optimizer that regularizes the L_1 norm of the parameters. Why is the regularizer in eq. (4) using the L_2 norm? Maybe I am missing something and/or there is an inconsistency is the notation/wording.  \n\nThe authors do a good job comparing with related works. However, one of the main early claims is that all works trying to find per-layer precision do so manually. This is not true, there have been some works that have done exactly that. One example is [1] which analytically determines precisions at all layers using a noise gain concept. It would be nice to contrast with such works as well.\n\nMinor issue:\n'comp x' is used in the results (tables) without being defined. It appears to indicate 'compression ratio'. This has to be explicitly stated at least once (maybe in the captions).\n\nreferences:\n[1] Sakr, Charbel, and Naresh R. Shanbhag. \"Per-tensor fixed-point quantization of the back-propagation algorithm.\" ICLR 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VROGuIPtvNR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper basically proposed to learn the quantization bits (precision) in each layer. Specially, weights are constructed with binary representation as $W_s = \\[W_s^1,...,W_s^b\\]$. During training, $W_s^i$ is relaxed to $ \\in \\[0, 2\\]$. And a group sparsity is imposed to all $W_s^i$ for all weights in a layer, leading to certain $W_s^i \\to 0$, thus cancelling the bit allocation in $i$-th. Experimental results is promising.\n\nPros:\n1. It is interesting to see that weights are represented in binary format, while each bit is trained in a full-precision scheme. \n\nCons:\n1. Training process is intricated: one has to tune the penalty in group sparsity. Also, training is separated in several steps: training and post-training finetuning.\n\nQuestions:\n1. After determining the quantization bit in (\"fake\") quantization training (although $W_q$ is quantized but $W_s^i$ is not exactly binary, which is the exactly weight we want) using Eq.5. Author mention in \"Re-quantization and precision adjustment\" that $W_q^{'}$ is converted to binary value. But how to deal with the precision loss here? i.e. from $W_s^i \\in \\[0,2\\]$ to $\\{0, 1\\}$\n2. Author mentioned that DoReFa-Net is adopted to finetune the trained model. Since DoReFa-Net use tanh to constrain value to $[0,1]$. it seems there is no connection to the proposed quantization scheme (Eq.6). How to exactly finetune ?\n3. Why is necessary for $W_s$ to be separated into postive and negative part ($W_p$, $W_n$) in processing ?\n4. Since $W_s^i$ is float and trainable, is it necessary to incorporate a trainable $s$ ?", "extra_scores": {}, "extra_text": {}}, {"review_id": "3IwNLMTP-DB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method to quantize neural networks in a differentiable manner. Proposed method applies the group lasso on the bit-planes of the weight parameters to let certain LSBs in each layer to be zero-ed out. STE is used to train the binary representation of each bit-plane and the sign of weights during the training. Results demonstrate that the proposed method can achieve higher accuracy and compression ratio compared to previous studies.\n\nI think that the idea of introducing group lasso to prune the entire bit-plane is very interesting and the paper is well written, but some additional analysis will be helpful.\n\n1. I think the result must be compared with more recent papers, such as LSQ (Esser, Steven K., et al., 2020). For example, LSQ demonstrates that it acheives 75.8% top-1 and 92.7% top-5 accuracy with 3/3-bit model (weight/activation) on ResNet-50. However, according to the appendix C, proposed method seems to achieve only 92.16% when the activation is quantized to 3-bit.\n2. This is more of a question than suggestion: after the requantization, would the batch-normalization layers function correctly? It seems like the parameters of batch normalization layers are kept the same after the requantization, while the requantization will impact the distribution of activations. Analysis on the difference between the distribution of the activation before the batch normalization layer before and afther the requantization will be helpful to see if the distribution of the activations really do not differ that much, or the batch-norm layer will just adapt to the occasion.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores a solution for mixed precision quantization. The authors view the weights in their binary format, and suggest to prune the bits in a structured way. Namely, all weights in the same layer should have the same precision, and the bits should be pruned from the least significant to most significant. This point of view allows the authors to exploit techniques used for weight pruning, such as L1 and group lasso regularization.

Although the field of quantization and model compression/acceleration is quite mature by now and has a large body of works, this paper is novel in its approach. Although the improvements provided over SoTA results are not very large, I believe that the novelty of the approach would make this paper a welcome addition to ICLR.

There are a few issues to be dealt with pointed out by the reviewers such as confusing terminology or required clarifications, but these are minor revisions that I trust the authors will be able to add to their paper.

Decision: Accept (Poster)