ABS: Automatic Bit Sharing for Model Compression
Authors: Jing Liu
Bohan Zhuang
Peng Chen
Yong Guo
Chunhua Shen
Jianfei Cai
Mingkui Tan
Venue:  (2021)
Keywords: Quantization
Pruning
Model Compression
AutoML

Abstract: We present Automatic Bit Sharing (ABS) to automatically search for optimal model compression configurations (e.g., pruning ratio and bitwidth). Unlike previous works that consider model pruning and quantization separately, we seek to optimize them jointly. To deal with the resultant large designing space, we propose a novel super-bit model, a single-path method, to encode all candidate compression configurations, rather than maintaining separate paths for each configuration. Specifically, we first propose a novel decomposition of quantization that encapsulates all the candidate bitwidths in the search space. Starting from a low bitwidth, we sequentially consider higher bitwidths by recursively adding re-assignment offsets. We then introduce learnable binary gates to encode the choice of bitwidth, including filter-wise 0-bit for pruning. By jointly training the binary gates in conjunction with network parameters, the compression configurations of each layer can be automatically determined. Our ABS brings two benefits for model compression: 1) It avoids the combinatorially large design space, with a reduced number of trainable parameters and search costs. 2) It also averts directly fitting an extremely low bit quantizer to the data, hence greatly reducing the optimization difficulty due to the non-differentiable quantization. Experiments on CIFAR-100 and ImageNet show that our methods achieve significant computational cost reduction while preserving promising performance. 

Under review as a conference paper at ICLR 2021
ABS: AUTOMATIC BIT SHARING FOR MODEL COM- PRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
We present Automatic Bit Sharing (ABS) to automatically search for optimal model compression configurations (e.g., pruning ratio and bitwidth). Unlike pre- vious works that consider model pruning and quantization separately, we seek to optimize them jointly. To deal with the resultant large designing space, we propose a novel super-bit model, single-path method, to encode all candidate compression configurations, rather than maintaining separate paths for each con- figuration. Specifically, we first propose novel decomposition of quantization that encapsulates all the candidate bitwidths the search space. Starting from a low bitwidth, we sequentially consider higher bitwidths by recursively adding re- assignment offsets. We then introduce learnable binary gates to encode the choice of bitwidth, including filter-wise 0-bit for pruning. By jointly training the binary gates in conjunction with network parameters, the compression configurations of each layer can be automatically determined. Our ABS brings two benefits for model compression: 1) It avoids the combinatorially large design space, with a reduced number of trainable parameters and search costs. 2) It also averts directly fitting an extremely low bit quantizer to the data, hence greatly reducing the op- timization difficulty due to the non-differentiable quantization. Experiments on CIFAR-100 and ImageNet show that our methods achieve significant computa- tional cost reduction while preserving promising performance.
INTRODUCTION
Deep neural networks (DNNs) have achieved great success in many challenging computer vision tasks, including image classification (Krizhevsky et al. 2012 He et al. 2016) and object detec- tion (Lin et al. 2017a b). However, a deep model usually has large number of parameters and consumes huge amounts of computational resources, which remains great obstacles for many appli- cations, especially on essurce-limitte devices with limited memory and computational resources, such smartphones. To reduce the number of parameters and computational overhead, many meth- ods (He et al., 2019; Zhou et al.] 2016) have been proposed to conduct model compression by removing the redundancy while maintaining the performance.
In the last decades, we have witnessed a lot of model compression methods, such as network prun- ing (He al. 2017 2019) and quantization (Zhou al. 2016 Hubara al., (0166). Specifically, network pruning reduces the model size and computational costs by removing redundant modules while network quantization maps the full-precision values to low-precision ones. It has been shown that sequentially perform network pruning and quantization is able to get compressed network with small model size and lower computational overhead (Han et al. 2016). However, performing pruning and quantization in a separate step may lead to sub-optimal results. For example, the best quantization strategy for the uncompressed network is not necessarily the optimal one after network pruning. Therefore, need to consider performing pruning and quantization simultancously.
Recently, many attempts have been made to automatically determine the compression configura- tions of each layer (i.e., pruning ratios, and/or bitwidths). either based on reinforcement learning (RL) (Wang et al. 2019)). evolutionary search (ES) (Wang et al. 2020), Bayesian optimization (BO) Tung & Mori 2018) differentiable methods (Wu et al. 20188 Dong & Yang 2019). In particular, previous differentiable methods formulate model compression as a differentiable searching problem to explore the search space using gradient-based optimization. As shown in Figure T(a). each candi-

---

Under review as a conference paper at ICLR 2021
232
32-bit
732
2 Input
72
Output
Super-bit
4-bit
2
74
2 Output
Imput:
64
E16
612
24 22 1
2-bit
22
24
22
=022 22 + + 232232
2 =2 + 9 (E+ + ... + (116 + 982
(a) Multi-path scheme (Wu et al. 2018
(b) Single-path scheme (Ours)
Figure I: Multi-path v.s. single-path compression scheme. (a) Multi-path search scheme (Wu al. 2018): represents each candidate configuration as a separate path and formulates the compression problem as path selection problem, which gives rise to huge numbers of trainable parameters and high computational overhead when the search space becomes combinatorially large. Here, 24 is the k-bit quantized version of and ag the architecture parameters corresponding to the path of k-bit quantization. (b) Single-path search scheme (Ours): represents each candidate configuration as a subset of "super-bit" and formulates the compression problem as a subset selection problem, which greatly reduces the computational costs and optimization difficulty from the discontinuity of quantization. Here, the super-bit denotes the highest bitwidth in the search space, ge is a binary gate that controls the decision of bitwidth, and c is the re-assignment offset (quantized residual error).
date operation is maintained as a separate path, which leads huge number of trainable parameters and high computational overhead when the search space becomes combinatorially large. Moreover, due to the non-differentiable quantizer and pruning process, the optimization of heavily compressed candidate networks can be more challenging than that in the conventional search problem.
In this paper, we propose a simple yet effective model compression method named Automatic Bit Sharing (ABS) to reduce the search cost and ease the optimization for the compressed candidates. Inspired by recent single-path neural architecture search (NAS) methods (Stamoulis et al., 2019] Guo et al. 2020). the proposed ABS introduces novel single-path super-bit to encode all effective bitwidths in the search space instead of formulating each candidate operation as separate path, as shown in Figure [(bb] Specifically, we build upon the observation that the quantized values of high bitwidth can share the ones low bitwidths under some conditions. Therefore, we are able to decompose the quantized representation into the sum of the lowest bit quantization and series of re-assignment offsets. We then introduce learnable binary gates to encode the choice of bitwidth, including filter-wise O-bit for pruning. By jointly training the binary gates and network parameters, the compression ratio of each layer can be automatically determined. The proposed scheme has several advantages. First, we only need to solve the search problem as finding which subset of the super-bit bit to use for each layer's weights and activations rather than selecting from different paths. Second, we enforce the candidate bitwidths to share the common quantized values. Hence, we are able to optimize them jointly instead of separately, which greatly reduces the optimization difficulty from the discontinuity of discretization.
Our main contributions are summarized as follows:
We devise a novel super-bit scheme that encapsulates multiple compression configurations in a unified single-path framework. Relying on the super-bit scheme, we further introduce learnable binary gates to determine the optimal bitwidths (including filter-wise O-bit for pruning). The proposed ABS casts the search problem as subset selection problem, hence significantly reducing the search cost.
We formulate the quantized representation as a gated combination of the lowest bitwidth quantization and a series of re-assignment offsets, in which we explicitly share the quan- tized values between different bitwidths. In this way, enable the candidate operations to learn jointly rather than separately, hence greatly easing the optimization, especially in the non-differentiable quantization scenario.
2

---

Under review as a conference paper at ICLR 2021
We evaluate our ABS CIF and ImageNet over various network architectures. Extensive experiments show that the proposed method achieves the state-of-the-art perfor- mance. For example, on ImageNet, our ABS compressed MobileNetV2 achieves x Bit-Operation (BOP) reduction with only 0.2% performance drop on the Top-1 accuracy.
RELATED WORK
Network quantization. Network quantization represents the weights, activations and even gradients in low-precision to yield compact DNNs. With low-precision integers power-of-two representa- tions, the heavy matrix multiplications can be replaced by efficient bitwise operations, leading to much faster test-time inference and lower power consumption. To improve the quantization perfor- mance, current methods either focus on designing accurate quantizers by fitting the quantizer to the data (Jung al.) 2019 Zhang et al. 2018 Choi et al. 2018 Cai et al. (0177) or seek approximate the gradients due to the non-differentiable discretization (Ding et al. 2019: Louizos et al. 2019]
Zhuang al. 2020). Moreover, most previous works assign the same bitwidth for all layers (Zhou et al. 2016) Zhuang et al. 2018a 2019, Jung et al. (019 Jin et al. 2019 Li et al. 2020 Esser et al., 2020 Though attractive for simplicity, setting a uniform precision places no guarantee on optimiz- ing network performance since different layers have different redundancy and arithmetic intensity. Therefore, several studies proposed mixed-precision quantization (Wang et al., 2019] Dong et al.] 2019 Wu et al.) 2018)] [hlich et al. (2200) to set different bitwidths according to the redundancy of each layer. this paper, based on the proposed quantization decomposition, we devise an approach that can effectively learn appropriate bitwidths for each layer through gradient-based optimization.
J01g.Jung
NAS and pruning. Neural architecture search (NAS) aims to automatically design efficient archi- tectures with low model size and computational costs, either based on reinforcement learning (Pham ctt al. 2018 Guo et al. 2019). evolutionary search (Real et al. 2019) or gradient-busee based methods (Liu ct al. 2019a). In particular, gradient-based NAS has gained increased popularity, where the search space can be divided into the multi-path design (Liu et al. 2019a Cai et al. 2019) and single-patl formulation (Stamoulis et al. 2019 Guo et al. (2020), depending
whether adding each operation as a separate path or not. While prevailing NAS methods optimize the network topology, the focus of this paper is search optimal compression ratios for a given architecture. Moreover, network pruning can be treated as fine-graines NAS, which aims at removing redundant modules to acceler- ate the run-time inference speed, giving rise to methods based on unstructured weight pruning (Han) et al. 2016 Guo et al. 2016) or structured channel pruning (He et al. 2017 ZZhaan et al. 2018b] Luo et al. (017). Based on channel pruning, our paper further takes quantization into consideration to generate more compact networks.
Auto.L for model compression. Recently, much effort has been put into automatically determin- ing either the optimal pruning rate (Tung & Mori 2018 Dong & Yang 2019; He et al. 2018), or the bitwidth (Lou et al., 2019 Cai & Vasconcelos 2020) of each layer via hyper-parameter search, without relying heuristics. In particular, HAQ (Wang et al. 2019) employs reinforcement learn- ing to search bitwidth strategies with the hardware accelerator's feedback. Meta-pruning (Liu et al.) 2019h) uses meta-learning to generate the weight parameters of the pruned networks and then adopts an evolutionary search algorithm to find the layer-wise sparsity for channel pruning. More recently, several studies (Wu al., 20118] Cai & Vasconcelos, 2020) have focused on using differentiable schemes via gradient-based optimization.
Closely related methods. To further improve the compression ratio, several methods propose to jointly optimize pruning and quantization strategies. In particular, some works only support weight quantization (Tung & Mori 20188) Ye al. 2019) or use fine-graid fine-grained pruning (Yang et al. 2020). However, the resultant networks cannot be implemented efficiently on edge devices. Recently, sev- eral methods (Wu et al. 2018.) Wang et al. 2020 Ying et al.) 2020) have been proposed to consider filter pruning, weight quantization, and activation quantization jointly. In contrast to these methods, we carefully design the compression search space by sharing the quantized values between differ- ent candidate configurations, which significantly reduces the search cost and eases the optimization. Compared with those methods that share the similarities of using quantized residual errors (Chen) et al. 2010 Gong et al., 2014 Li et al. 2017b van Baalen et al. 2020), our proposed method recur- sively uses quantized residual errors to decompose a quantized representation as a set of candidate bitwidths and parameterize the selection of optimal bitwidth via binary gates.
3

---

Under review as a conference paper at ICLR 2021
Our proposed ABS and Bayesian Bits (van Baalen et al. 2020) are developed coneurrently that share a similar idea of quantization decomposition. Critically, our ABS differs from Bayesian Bits in several aspects: 1) The quantization decomposition in methods can be extended to non-power- of-two bit widths (i.e., bj can be set to arbitrary appropriate integer values), which is a general case of the one in Bayesian Bits. 2) The optimization problems are different. Specifically, formulate model compression as a single-path subset selection problem while Bayesian Bits casts the optimization of the binary gates variational inference problem that requires more relaxations and hyperparameters. 3) Our compressed models with less or comparable BOPs outperform those of Bayesian Bits by large margin on ImageNet (See T
PROPOSED METHOD
3.1 PRELIMINARY: NORMALIZATION AND QUANTIZATION FUNCTION
Without loss of generality, given a convolutional layer, let x and w be the activations of the last layer and its weight parameters, respectively. First, for convenience, following (Choi et al. 2018 Bai
et al. 2019), we can normalize x and w into scale [O, 1] by Ta and Tw. respectively:
2x = (x) = clip
- _001
I 0.1))
(1)
Uz
1
w
1. +1 +1),
2u = (w) = 2
clip
Vw 11.11,1
(2)
Ulow), tup) clips any number into the range [Flow: dup]. and U/ and Uw are trainable quantization intervals which indicate the range of weights and activations to be quantized. Then, can apply the following function quantize the normal- ized activations and parameters, namely 2n E [0. 1] and Zu E [0, to discretized ones:
where the function clip (v, Vlow Pup)
D(2, 8) round (
where round(.) returns the nearest integer of given value and s denotes the normalized step size. Typically, for k-bit quantization, the normalized step size can be computed by
(4)
2
After doing the k-bit quantization, we shall have 2 quantized values. Specifically, we obtain the quantization Qww) and (ax)) by
(ww)= T ( 5)) = -w (2 DZw 8) 1), (3z, (xx) T (D(2, 8)) = 11= D(2x, 8),
(5)
(6)
where T=1 and T-1 denote the inverse functions of Tw and Tr. respectively.
3.2 BIT SHARING DECOMPOSITION
Previous methods consider different compression configurations as different paths and reformulate model compression as a path selection problem, which gives rise to a huge number of trainable parameters and high computational costs. In this paper, we seek to conduct filter pruning and quan- tization simultaneously by solving the following problem:
min L L(W, o², a.)) W W,0P
where L(-) denotes some losses, and W is the parameters of the network. a and a are the pruning and quantization configurations, respectively. As shown in Eq. (7), we propose to encode all compression configurations in a single-path super-bit model (See Figure ((b)). In the following, we first introduce the bit sharing decomposition and then describe how to learn for compression.
To illustrate the bit sharing decomposition, we begin with an example of 2-bit quantization for z E (āez ). Specifically,, we consider using the following equation to quantize to 2-bit:
22 D(2, 92);
82
(8)
22
4

---

Under review as a conference paper at ICLR 2021
where 22 and $2 are the quantized value and the step size of 2-bit quantization, respectively. Due to the large step size, the residual eπrоr 2 - 22 E [- 82/2| may be big and result a significant performance drop. To reduce the residual error, an intuitive way is to indicates that we quantize= 2 to higher bitwidth. Since the step size 84 = 1/(2* - 1) in 4-bit quanti- zation is divisor of the step size 82 in 2-bit quantization, the quantized values of 2-bit quantization are among the ones of 4-bit quantization. In fact, based on 2-bit quantization. the 4-bit counterpart introduces additional unshared quantized values. In particular, if z2 has zero residual error, then 4-bit quantization maps to the shared quantized values (i.e., 22). In contrast, if 22 is with non-zero residual error, 4-bit quantization likely to map 2 to the unshared quantized values. In this case, 4-bit quantization can be regarded as performing quantized value re-assignment based 22. Such a re-assignment process can be formulated as follows:
smaller step size, which
(9) where 24 is the 4-bit quantized value and eạ is the re-assignment offset based on 22. To ensure that the results of re-assignment fall into the unshared quantized values, the re-assignment offset εд must be an integer multiplying of the 4-bit step size 84. Formally, ca can be computed by performing 4-bit quantization on the residual error of 22:
24 2/ +
1
E1 D(2 22, 81), $4 2
(10)
+
21
21
Therefore, according to Eq. 9 4-bit quantized value can be decomposed into the 2-bit represen- tation and its re-assignment offset. Similarly, an 8-bit quantized value can also be decomposed into the 4-bit representation and its corresponding re-assignment offset. In this way, we can generalize the idea of decomposition to arbitrary effective bitwidths as follows.
Definition (Quantization decomposition) Let z E [0. 1] be a normalized full-precision input, (bi...., bk} J be a sequence of candidate bitwidths, and bj be. < bk-1 bx. We use the following quantized ă to approximate 2:
K
Sby 8b,-1
2 = 2b,
+
Cbyy
where Cb2 = =D (00-1,,b,), 8by
2 +1
2 -1
(1I)
j=2
In other words, the quantized approximation 2 can be decomposed into the sum of the lowest bit quantization and series of recursive re-assignment offsets. In Definition (I), to enable quantized value re-assignment, we need to constrain that 8b, is divisible by 8b,, which requires the bitwidths (j > 1) to satisfy the following relation:
(12) In fact, the bitwidth b can be set to arbitrary appropriate integer values (e.g., 1, 2, 3, etc.). To get fordware-friendle compressed network] we set dn to 2, which ensures that all the decomposi- tion bitwidths are ofwer-of-two. Moreover, since 8-bit quantization achieves lossless performance compared with the full-precision counterpart (Zhou et al. 2016), we only consider those candidate bitwidths that are not greater than 8-bit. In other words, we constrain the value of j to [1,3].
bj 2 bi-
The proposed bit sharing decomposition has several advantages. First, the proposed method only needs to maintain a small number of trainable parameters, which greatly reduces the computational costs during search. Second, we are able directly extract a low-precision repre- sentation from its higher precision, which allows optimizing different bitwidths jointly and ease the discontinuous optimization due to quantization.
Remark 1
3.3 LEARNING FOR COMPRESSION
Note that different layers have different levels of redundancy, which indicates that different layers may choose different subsets of the quantized values. To learn the quantized approximation for each layer, we introduce a layer-wiss wise binary quantization gate L E {0, 1} on each of the ee-assignment offsets in Eq. to encode the choice of the guantization bitwidth, which can be formulated as
98,
[1 2b,,, - -0,, >0):
(13)
2b + 0h (cbz + +... + Dix (fox-1 + Sox <bx) )),
Mooe details can be found in Appendix A
5

---

Under review as a conference paper at ICLR 2021
where 1()) is the indicator function and aj, is layer-wise threshold that controls the choice of bitwidth. Specifically, if the quantization error || - -b,-1ll is greater than the threshold az we activate the corresponding quantization gate to increase the bitwidth so that the residual error can be reduced, and vice versa.
Note that from Eq. (3)) we can consider the filter pruning as 0-bit filter-wise quantization. To avoid the prohibitively large filter-wise search space, we propose to divide the filters into groups based on indexes and consider the group-wise sparsity instead. To be specific, we introduce a binary gate oe for each group to encode the choice of pruning, which can be formulated follows:
ge = 11[[[[1wW)) a² >100 0),
(14)
Ee = (2c,b, + 98, (cc,b2 + ..... 9bx_, (Ce,bx- + g Cc,bx (c,bx))))
where E, is the e-th group of quantized filters and er.b, is the corresponding re-assignment offset by quantizing the residual error 2e Zc,by-1' Here, o² is a layer-wise threshold for filter pruning. Fol- lowing PFEC (Li al. 2017a), use nn-rrmm evaluate the importance of the filter. Specifically, if a group of filters is important, the corresponding pruning gate will be activated and vice versa.
Note that both quantization and pruning have their corresponding thresholds. Instead of manually setting the thresholds, we propose to learn them via gradient descent. However, the indicator func- diffefferent...ee. To address this, we use thiighttthrough estimator (STE) (Ben- gio et al. 2013 Zhou et al. (2016) to approximate the gradient of the indicator function 1(-) using the gradient of the sigmoid function (.)) which can be formulated as:
tion in Eq. (13
0y 01 (A-c) a) da
do (A-a) -a (A (1-
(15)
da
da
where g is the output of a binary gate, a E {a?, a.) } is the corresponding threshold and A denotes some specific metrics (i.e., fi-ormm of the filter or the quantization error). By jointly training the binary gates and the network parameters, the pruning ratio and bitwidth of each layer can be auto- matically determined. However, the gradient approximation of the binary gate inevitably introduces noisy signals, which can be even more severe when we quantize both weights and activations. Thus, we propose to train the binary gates of weights and activations in an alternative manner. Specifically, when training the binary gates of weights, we fix the binary gates of activations, and vice versa.
Search Space for Model Compression. Given an uncompressed network with 7. layers, we use C to denote the number of filters the I-th layer. To obtain the compressed model, we first divide the filters of each layer into groups and then search for the optimal bitwidths for the considered layer. Let B be the number of filters in a group. For any layer 1, there would be C. groups in total. Since we quantize both weights and activations, given K candidate bitwidths, there are K2 different quantization configurations for each layer. Thus, for the whole network with T. layers, the size of the search space Q can be computed by
B
)
C2 B
|2 =
II t=1
K2
(16)
Eq. (66) indicates that the search space is large enough to cover the potentially good configurations.
Training Objective. To design hardware-efficient network, the objective function in Eq. 7 should reflect both the accuracy of the compressed network and its computational costs. Following (Cai
al. 2019, we train the network and architecture by minimizing following loss function:
L(W, a², a²) = (W,, o², a.) + X log R R(W, a aa))
(17)
where Loe() is the cross-entropy loss, R(-) is the computational costs of the network and À is a balancing hyper-parameter. Following single-path NAS (Stamoulis et al. 2019), we use similar formulation of computational costs to preserve the differentiability of the objective function. The details of the differentiable computational loss can be found in Appendix Once the training is fin- ished, we can obtain the compressed network by selecting those filters and bitwidths with activated binary gates. Then, we fine-tune the compressed network to compensate the accuracy loss.

---

Under review as a conference paper at ICLR 2021
Table 1: Comparisons of different methods CIFAR-100. "W" and "A" represent the average quantization bitwidth of the weights and activations, respectively.
Network
Method
BOPS (M)
comp. raho
W
Top-1 Acc. (8)
Top-5 Acc. (98
Full precision
4T79K.6
1.0
32.07 22.732.00
67.5
90.8
4-bil precision
674.6
62.0
4.0/4.0
67.810.3
9044002
DQ
1180.0
35,4
5.3/6.1
67.7=0.6
90.4=0.5
ResNet-20
HAQ
653.4
64.0
3.7/4.2
67.720.1
90.40..3
DNAS
660.0
62.9
4.6/3.8
67,8+0,3
90.4=0.2
ABS-P (Ours)
32.0/32.0
67.9+0.1
90.7-0022
ABS-Q (Ours)
649.5
64.4
4.4/4.2
68.1=0,1
90.5=0.00
(Ours)
68.1+0.33
90.6-0.2
Tull-precision
128771.7
1.0
32.0732.0
71.7
92.2
precision
70.9+0.3
91.2+004
DQ
2222.9
57.9
3.8/4.6
70.7=002
9144=0.4
ResNet-56
HAQ
3,3149
71,2+0.1
91.1=-02
DNAS
2035.7
65.3
5.3/3.2
71.2--0.2
91.3+-0.3
(Ours)
32.0/32.0
71.5+0.1
91.8-022
ABS-Q (Ours)
1970.7
65.3
4.1/40
71.5+0.2
91.5+-0.2
(Ours)
71.5
71.5
Top-1Ace (%
Top-1Acc. (%
71.0
71.0
70.5
Fived-precision DNAS
70.5
Fixed-precision DNAS
70.0
A ABB-Q
HAQ
70.0
ABS-Q ABS
HAQ
-BABS
69.5
69.5
1000
1500 BO (M)
2000
400
500
600
700
M emory Footprints (KB)
(a) BOPs vs. Top-1 accuracy
(b) Memory footprints vs. Top-1 Accu- racy
Figure 2: Results of different compressed networks with different BOPs and memory footprints. We use different methods to compress ResNet-56 and report the results on CIFAR-100.
4 EXPERIMENTS
Compared methods. To investigate the effectiveness of the proposed method, we consider the following methods for comparisons: ABS: our proposed method with joint pruning and quantiza- tion; ABS-Q: ABS with quantization only; ABS-P: ABS with pruning only; and several state-of-
the-art model compression methods including HAQ Waagg et al. 2019). DQ (Uhlich et al.) 2020).
DJPQ (Ying al. 2020), Bayesian Bits (van Baalen et al. (020) and DNAS (Wu et al. 2018). We measure the performance of different methods in terms of the Top-1 and Top-3 accuracy. Follow- ing (Guo et al. 2020, Ying al. 2020). measure the computational costs by the Bit-Operation (BOP) count. The BOP compression ratio is defined as the ratio between the total BOPs of the uncompressed and compressed models. We also measure the computational costs with the total weights and activations memory footprints following DQ (Uhlich et al., 2020). Moreover, follow- ing (Stamoulis al. [2019] Liu al. 20119), we use the search cost measure the time of finding an optimal compressed model.
Implementation details. Following HAQ (Wang al. 2019). quantize all the layers, in which the first and the last layers are quantized to 8-bit. Following ThiNer (Luo et al. 2017). we only conduct filter pruning for the first layer in the residual block. For ResNet-20 and ResNet-56 on CIFAR- 100 (Krizhevsky et al. (2099), we set B to 4. For ResNet-18 18 and MobileNetV2 on Ima- geNet (Russakovsky et al.) 2015). B is set to 16 and respectively. We first train the full-precision models and then use the pretrained weights to initialize the compressed models. Following (Li cet al. 2020 [sser et al., 2020). we introduce weight normalization during training. We use SGD with nesterov (Nesterov 1983) for optimization, with a momentum of 0.9. For CIFAR-100, the same data augmentation as (He et al. 2016). including translation and horizontal flipping. For ImageNet, images resized to 256 256, and then 224 x 224 patch is randomly cropped from an image or its horizontal flip for training. For testing, 224 x 224 center cropped is chosen. We first train the uncompressed network for 30 epochs CIFAR- 100 and 10 epochs ImageNet.
use
7

---

Under review as a conference paper at ICLR 2021
Table 2: Comparisons ImageNet. "**h denotes that get the results from the figures in (van
Baalen et al., 2020) and "_" denotes that the results are not reported. Moreover, "W" and "A" represent the average quantization bitwidth of the weights and activations, respectively.
Network
Metbod
BOPs (Ú)
BOP comp, ratio
W7A
Top-1 Acc. (98)
Top-5 Acc. (%)
Fall precision
1857.6
1.0
3207
70.7
89.8
4-bit precision
34.7
53.5
4,0/4.0
71.0
89.8
DQ
40.7
40.6
-/-
68.5
ResNet-18
DIPQ
HAQ
34.7
35.5
52.3
53.5
4.0/4.0
-/-
70.2
69.1
89.5
Bayesian Bits"
35.9
51.7
-/-
69.5
ABS-Q (Ours)
33.1
56.1
4.8
70.9
89.7
ABS (Ours)
32.3
$7.5
44,
70.8
89.6
Full-precision
308.0
1.0
32.0732.0
71.9
90.3
6-bit precision
11.2
27.5
6.066.0
71.8
90.3
DQ HAQ
19.6
1,9
6.0
70.4
89.7
MobileNetV2
10.8
28.5
5.61 /6227
71.2
90.0
Rayesian ABS Q (Ours)
10.8
10.9
28.3
6.8 6.88
71.8
90.4
(Ours)
10.8
6.1/7.1
The learning rate is set to 0.001. We then fine-tune the searched compressed network to recover the performance drop. On CIFAR-100, we train the searched network for 200 epochs with a mini-batch size of 128. The learning rate is initialized to 0.1 and is divided by 10 at 80-th and 120-th epochs. Experiments on CIFAR- 100 are repeated for 5 times and we report the mean and standard deviation. For ResNet-18 18 on ImageNet, we finetune the searched network for 90 epochs with mini-bater size of 256. For MobileNetV2 on ImageNet, we fine-tune for 150 epochs. For all models on ImageNet, the learning rate starts at 0.01 and decays with cosine annealing (Loshchilov & Hutter 2017).
4.1 MAIN RESULTS
We apply the proposed methods to compress ResNet-20, ResNet-56 on CIFAR-100 and ResNet- 18, MobileNetV2 on ImageNet. We compare the performance of different methods in Table T and Table We also show the results of the compressed ResNet-56 with different BOPs and memory footprints in Figure From the results, we can see that 4-bit quantized networks achieve lossless performance. Also, 6-bit MobileNetV2 only leads to a 0.1 % performance drop the Top-1 Ac- curacy. Compared with fixed-precsiion quantization, mixed-precision methods are able to reduce the BOPs while preserving the performance. Critically, our proposed ABS-Q outperforms the state- of-the-arts baselines with less computational costs. Specifically, ABS-Q compressed ResNet- outperforms the one compressed by HAQ with more BOPs reduction. More critically, our proposed ABS achieves significant improvement in terms of BOPs and memory footprints. For example, in Figure 2(a), our ABS compressed ResNet-56 model yields much fewer BOPs (395.25 vs. 536.24) but achieves comparable performance compared with the fixed-precision counterpart. Morcover, by combing pruning and quantization, ABS achieves nearly lossless performance while further re- ducing the computational costs of ABS-Q. For example, ABS compressed ResNet- 18 reduces the BOPs by 57.5 x while still outperforming the full fuel-precision network by 0.1% in terms of the Top-1 accuracy on ImageNet.
4.2 FURTHER STUDIES
Effect of the bit-sharing scheme. To investigate the effect of the bit-sharing scheme, we apply our methods quantize ResNet-20 and ResNet-56 with and without the bit sharing scheme on CIFAR- 100. We report the testing accuracy and BOPs in Table 3 We also present the search costs and consumed GPU memory measured on a GPU device (NVIDIA TITAN Xp). It can be seen from the results that the method with the bit sharing scheme consistently outperforms the ones without the bit sharing scheme while significantly reducing the search costs and GPU memory.
Effect of the one-stage compression. To investigate the effect of the one-stage compression scheme (perform pruning and quantization jointly), we extend ABS to two-stage optimization, where we se- quentially do filter pruning and quantization, denoted as in Table Compared with the two-stage counterpart, ABS achieves better performance with less computational costs, which shows the superiority of the one-stage optimization. For example, ABS compressed ResNet-56 outperforms the counterpart by 0.4% on the Top- accuracy with less com- putational overhead.
The results are shown
8

---

Under review as a conference paper at ICLR 2021
Table 3: Effect of the bit-sharing scheme. We report the testing accuracy, BOPs, and search costs CIFAR-100. The search costs are measured on a GPU device (NVIDIA TITAN Xp).
Network
Method
Top-1 Acc.
Top-5 Acc.
BOPs (M)
Search Cost (GPU hours)
GPU Memory (GB)
w/o sharing
67.810.11
90.5100.2
664.2
2.8
4.4
ResNet-20
w/ bit sharing
68.110.11
90.5:100
649.5
0.8
1.5
w/o sharing
71.3:0..3
91,410.4
2001.1
8.7
10.9
ResNet-56
w/ bit sharing
71,5=0.2
91.5+0.22
1970.7
1,9
3.0
Table Effect of the one-stage compression. We report the results of ResNet-56 on CIFAR-100.
Network
Method
Top-I Acc.
Top-5 Acc.
BOPs (M)
ABS-P + ABS-Q
10.4 70,8 0.4
90.8
1077.7
ResNet-56
ABS
91.2.10.1
1042.5
Effect of the alternative training scheme. To investigate the effect of the alternative training scheme introduced Section 3.33 we apply our method to compress ResNet-56 using a joint training scheme and an alternative training scheme CIFAR-100 100. Here, the joint training scheme denotes that we train the binary gates weights and activations jointly. From the results of Table [5) the model trained with the alternative scheme achieves better performance than the joint one, which demonstrates the effectiveness of the alternative training scheme.
Table 5: Effect of the alternative training scheme. We report the results of ResNet-56 on CIFAR-100.
Network
Method
Top-1 Acc.
Top-5 Acc.
BOPs (M)
Joint
71.3=002
91.6+0.3
1942.4
ResNet-56
Alternatvee
71.6+0.1
91.8+0.4
1918.8
Resource-constrained compression. To demonstrate the effectiveness of our ABS on hardware devices, further apply our methods to compress MobileNetV2 under the resource constraints on the BitFusion architecture (Sharma et al. 2018). Instead of using BOPs, we use the latency and energy on simulator of the BitFusion to measure the computational costs. We report the results in Table 6 Compared with fixed-precision quantization, ABS achieves better performance with lower latency and energy. Specifically, ABS compressed MobileNetV2 with much lower latency and energy even outperforms 6-bit MobileNetV2 by 0.2% in the Top-1 1 accuracy.
compression on BitFusion. We evaluate the proposed ABS under the latency- and energy constraied and report the Top-1 and Top-5 accuracy ImageNet.
Table
Network
Method
Latency- consty--o
Energy-constraing
Acc.-I (%)
Acc.- (%)
Latency (ms)
Acc.-I (%)
Acc.-5 (%)
Energy (mJ)
6-bit precision
71.8
90.3
24.9
71.8
90.3
32.8
MobileNetV2
ABS (Ours)
72.0
90.4
17.2
72.0
90.3
26.3
CONCLUSION AND FUTURE WORK
In this paper, we have proposed a novel model compression method called Automatically Bit Shar- ing (ABS). Specifically, our ABS based on the observation that quantized values of a high bitwidth share the ones of lower bitwidths under some constraints. We therefore have proposed the decompo- sition of quantization that encapsulates all candidate bitwidths. Starting from a low bitwidth in the search space, we sequentially increase the effective bitwidth by recursively adding re-assignment offsets. Based on this, we have further introduced learnable binary gates to encode the choice of different compression policies. By training the binary gates, the optimal compression ratio of each layer can automatically determined. Experiments on CIFAR- 1-00 and ImageNet have shown that methods able achieve significant cost reduction while preserving the performance. In the future, we plan to work on a joint search for architecture, pruning, and quantization to find a compact model with better performance.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. In Proc. Int. Conf. Learn. Repren., 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients 2013.
through stochastic neurons for conditional computation. arXiv preprint
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In Proc. Int. Conf. Learn. Repren., 2019.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed seeedd-peeision neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
Yongjian Chen, Tao Guan, and Cheng Wang. Approximate nearest neighbor search by residual vector quantization. Sensors, 10(12):1125--11277 2010.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint /80.006088 2018.
Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
Xuanyi Dong and Yi Yang. Network praning via transformable architecture search. In Proc. Adv. Neural Inf. Process. Syst., 2019.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proc. IEEE Ini. Conf. Comp. Vis., 2019.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S. Modha. Learned step size quantization. In Proc. Ini. Conf. Learn. Repren., 2020.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev, Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv: 1xXv::442.6611. 2014.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, In Proc. Adv. Neural Inf. Process. Syst., 2016.
Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. Nat: Neural architecture transformer for accurate and compact architectures. In Proc. Adv. Neural Inf. Process. 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Proc. Eur. Conf. Comp. Vis., 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proc. Int. Conf. Learn. Repren., 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conf. Comp. Vis. Patr. Recogn,, 2016.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
10

---

Under review as a conference paper at ICLR 2021
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proc. IEEE Int. Conf. Comp. Vis., 2017.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proc. Eur. Conf. Comp. Vis., 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. Proc. Adv. Neural Inf. Process. Syst., 2016.
Qing Jin, Linjie Yang. and Zhenyu Liao. Towards efficient training for neural network quantization. arXiv preprint 1rXiv: 2019.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Proc. Adv. Neural Inf. Process. Syst., 2012.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. Proc. Int. Conf. Learn. Repren., 2017a.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-
uniform discretization for neural networks. Proc. Int. Conf. Learn. Repren., 2020.
Zefan Li, Bingbing Ni, Wenjun Zhang, Xiaokang Yang, and Wen Gao. Performance guaranteed network acceleration via high-order residual quantization. In Proc. IEEE Int. Conf. Comp. Vis., 2017b.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proc. IEEE Int. Conf. Comp. Vis., 2017b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proc. Int. Conf. Learn. Repren., 2019a.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proc. IEEE Int. Conf. Comp. Vis., 2019b.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Proc. Int. Conf. Learn. Repren., 2017.
Qian Lou, Lantao Liu, Minje Kim, and Lei Jiang. Autoqb: Automl for network quantization and binarization on mobile devices. arXiv preprint 19Xiv:1902.05660 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. In Proc. Int. Conf. Learn. Repren., 2019.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proc. IEEE Int. Conf. Comp. Vis., 2017.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o
(1/k" 2). In Proceedings of the USSR Academy of Sciences, volume 269, pp. 543-547, 1983.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proc. Inf. Conf. Mach. Learn., 2018.

---

Under review as a conference paper at ICLR 2021
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. Proc. AAAI Conf. on Arti. Intel., 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Int. J. Comp. Vis., 115(3):211-2355) 2015.
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In International Symposium on Computer Architecture, pp. 764-775. IEEF, 2018.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481-497, 2019.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in- in-parallel pruning- quantization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 7873-7882, 2018.
Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is good parametrization. In Proc. Int. Conf. Learn. Repren., 2020.
Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. In Proc. Adv. Neural Inf. Process. Syst. 2020.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00090 2018.
Haichuan Yang. Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-base approach. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Shaokai Ye, Tianyun Zhang, Kaiqi Zhang, Jiayu Li, Jiaming Xie, Yun Liang. Sijia Liu, Xue Lin, and Yanzhi Wang. A unified framework of dnn weight pruning and weight clustering/quantiaation using admm. In Proc. AAAI Conf. on Arti Intel., 2019.
Wang Ying, Lu Yadong, and Blankevoort Tijmen. Differentiable joint pruning and quantization for hardware efficiency. In Proc. Eur. Conf. Comp. Vis., 2020.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proc. Eur. Conf. Comp. Vis., 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06110 2016.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 201 Sa.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
12

---

Under review as a conference paper at ICLR 2021
Bohan Zhuang, Lingqiao Liu, Mingkui Tan, Chunhua Shen, and Ian Reid. Training quantized neural networks with a full-pree precision auxiliary module. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrinination-awanl channel pruning for deep neural networks. In Proc, Ady. Neural Inf. Process. Syst., 2018b.
13

---

Under review as a conference paper at ICLR 2021
Appendix for ABS: Automatic Bit Sharing for Model Compression
A HARDWARE-FRIENDAA DECOMPOSITION
As mentioned in Sec. 3.2 bj can be set to arbitrary appropriate integer values (e.g., 1, 2, etc.). By default, we set bj for better hardware utilization. On general purpose computing devices (e.g., CPU, GPU), byte (8 bits) is the lowest data type for operations. Other data types and ALU registers are all composed with multiple bytes in width. By setting b1 = 2. 2-bit/ 4-bit/ 8-bit quantization values can be packed into byte (or short, int, long) data type without bit wasting. Otherwise, if by = or bj = 3. it is inevitable to have wasted bits when packing mixed-precision quantized tensors on general purpose devices. For example, one 32-bit int data type can be used to store ten 3-bit quantized values with bits wasted. One might argue that these 2 bits can be leveraged with the next group of 3-bit data, but it will result in irregular memory access patterns, which will degrade the hardware utilization more seriously. Moreover, 8-bit quantization has been demonstrated to own similar performance with the full precision counterparts for many networks. Therefore, there is no need to consider a bitwidth larger than 8.
B FORMULATION OF DIFFERENTIABLE COMPUTATIONAL LOss
In this section, we introduce the differentiable computational loss mentioned in Section 3.3 Unlike the cross-entropy loss in Eq. (17). the computational costs R?W,, a", a²) is non-differentiable. To solve this issue, we model the computational costs as function of binary gates as:
R(W, a², a4)
ge
Rovwn + 9bs RRxe.vg - Re + + gox (Re. Rec.x-1)
(18)
where Ro is the computational cost for the e-th group of filters with by- quantization and G is the number of groups in total.
C QUANTIZATION CONFIGURATIONS
All the methods in Tables and 2 layer-wise and symmetric quantization schemes and the compared methods strictly follow the quantization configurations in their original papers. Specif- ically, for DQ (Uhlich al., 2020). we parameterize the fixed-point quantizer using case U3 with 0 = [d, 9max] We initialize the weights using pre-trained model. The initial step size is set to d 2 [ (max([W))/((2 1-1)] for weights and 2-3 for activations. The remaining quantization pa- rameters are set such that the initial bitwidth is 4-bit. For HAQ (Wang et al., 2019). we first truncate the weights and activations into the range of [- [yc] and [0, Uz). respectively. We then perform linear quantization for both weights and activations. To find more proper Vw and Ug. we minimize the Kl.-divergence between the original weight distribution W and the quantized weight distribution Q(W). For DNAS (Wu et al. 2018), we follow DoReFa-Net (Zhou et al. 2016) to quantize weights and follow PACT (Choi et al.
2018) to quantize activations. We initialize the learnable upper bound
to 1. For DJPQ (Ying et al.
2020) and Bayesian Bits (van Baalen et al. 2020), we directly get the results from original papers. For other methods in Tables I and we use the quantization function
introduced in Section 3.1 The trainable quantization intervals U2 and Var are initialized to 1.
D SEARCH COST COMPARISONS
To evaluate the efficiency of the proposed ABS, we compare the search cost of different methods and report the results in Table 7 From the results, the search costs of the proposed ABS is much smaller than the otate-of-the-art methods. Moreover, compared with ABS-Q, Q. ABS only introduces a small amount of computational overhead, which demonstrates the efficiency of the proposed methods.
14

---

Under review as a conference paper at ICLR 2021
Table 7: Comparisons of the search costs CIFAR-100. The search costs measured GPU
device (NVIDIA TITAN Xp).
Search Cost
Network
Method
(GPU hours)
HAQ
5.8
DQ
3.0
DNAS
2.8
ResNet-20
ABS-Q Q (Ours)
0.8
ABS-P (Ours)
0.2
ABS (Ours)
1.0
Table 8:
Comparisons of different methods w.r.t. memory footprints. We compress ResNet-56
using different methods and report the results on CIFAR-I00. 100.
Method full-preciision 4-hit precision
Memory footprints
M.f. comp.
Top-1 Acc.
Top-5
5653.4
1.0
71.7
92.2
711.7
7.9
70.9+0.3
91.2+0044
DNAS
708.9
8.0
71.5++0.
91.3++0.1
HAQ
700.0
71.3+0.1
91.1=0.1
ABS-Q (Ours)
674.5
8.4
71.5++0.2
91.6++022
ABS (Ours)
657.3
8.6
71.6+
91.8++0.4
Table 9: Comparisons of different methods with MobileNetV3 on CIFAR-100
BOPs (M)
comp.
Top-1 Acc. (%)
Top-5 Acc. (%)
Full-precision
68170.1
1.0
76.1
93.9
6-bit precision
2412.6
28.3
76.11:0.0
93.7::0.0
DQ
2136.3
75.9++0.1
93.74-0.1
HAQ
2191.7
31.1
76.11:0.1
93.5+10.0
DNAS
2051.9
33.2
76.1100.1
93.71:0.1
ABS-P (Ours)
59465.8
76.0:10.0
93.5+00.0
ABS (Ours)
2021.9
33.7
76.1+00.1
93.7+-0.1
ABS (Ours)
2006.6
34.0
76.1:10.1
93.7.10.1
E MORE RESULTS ON MEMORY FOOTPRINTS
To further demonstrate the effectiveness of the proposed ABS, we replace BOPs with total weights and activations memory footprints (Uhlich et al., (020). We apply different methods to compress ResNet-56 and report the results in Table 8 From the results, ABS compressed ResNet-56 outper- forms other methods with fewer memory footprints. These results show the effectiveness of our proposed ABS in terms of memory footprints reduction.
F DETAILED STRUCTURE THE COMPRESSED NETWORK
We illustrate the pruning rate and bitwidth of each layer's weights and activations of the compressed ResNet- 18 and MobileNetV2 in Figure 3 and Figure $ respectively. From the results, we observe that ABS assigns more bitwidths to the weights in the downsampling convolutional layer in ResNet- 18 and depthwise convolutional layer in MobileNerV2. Intuitively, this is because the num- ber of parameters of these layers is much smaller than other layers. Moreover, ABS inclines to prune more filters in the shallower layers, which can significantly reduce the number of parameters and computational overhead. Finally, we also observe that the correlation between the bitwidth and pruning rate is as follows. If a layer is set to a high pruning rate, our ABS tends to select higher bitwidth compensate for the performance drop. contrast, if layer is with low pruning rate, our ABS tends to select lower bitwidth to reduce the model size and computational costs.
G MORE RESULTS ON MOBILENETV3
To evaluate the proposed ABS on the lightweight model, we apply our methods to MobileNetV3 on CIFAR-100. 100. Following LSQ+ (Bhalgat et 2020), we introduce learnable offset to handle the negative activations in hard-swish. We show the results in Table From the results of MobileNetV3, our proposed ABS still outperforms the compared methods, which demonstrates its effectiveness.
15

---

Under review as a conference paper at ICLR 2021
Downsampling layer: more bits
8
4
2
#bits
0
2
4
8
1
2
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Layer
#weight bit
#activation bit
(a) Bitwidth configuration of the compressed ResNet-18.
0.25
0.20
Pruning rate (%)
0.15
0.10
0.05
0.00
2
4
5
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Layer
(b) Pruning configuration of the compressed ResNet-18. 18. The pruning rate is defined the ratio between #pruned weights compressed models #weights of the uncompressed models.
Figure 3: Detailed configurations of the compressed ResNet-18.
16

---

Under review as a conference paper at ICLR 2021
Aweight bit ddpthoisse coev)
factivation bit (dopthwiso conv) lactivaaton hit (poinnwise conv) Reclivation bit (convit))
Aveiight bit (pointwise ornv) Auscight bit (come'l))
#bils
Layer
(a) Bitwidth configuration of the compressed MobileNetV2.
0.05
0.04
Pruting rato (%)
0.03
0.02
0.01
0.00
Layer
(b) Pruning rate configuration of the compressed MobileNetV2. The pruning rate is defined as the ratio between #pruned weights of the compressed models and #weights of the uncompressed models.
Figure 4: Detailed configurations of the compressed MobileNetV2.
17


Reviews:
[{"review_id": "zmSsNI1hbK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes the method to determine optimal quantization bit-width and pruning configuration for the neural network compression. Different from other approaches, the proposed method integrates multiple bit configurations (including pruning) into a single architecture, which is named “Super-bit”. The architecture uses binary gates to automatically select bit resolution. In addition, the super-bit model is differentiable and jointly trainable with parameters.\n\nThe idea of decomposing bit configuration into 2, 4, 8 bits, and using binary gates does not seems to be novel [van Baalen, 2020]. This paper also adopts the trainable gate parameter and unifies the pruning scheme.\n\nCompared to [van Baalen, 2020], the proposed super-bit use different gate function (forward step, backward sigmoid) and different loss function, so there is some novelty. However, there is no direct comparison of the algorithm. To be fair, [van Baalen, 2020] is quite recent work and both ideas may be developed concurrently. So, I think inserting a single paragraph of comparison would be enough.\n\nUsing cost function R is a good idea to directly reduce the computation cost. Experiments, especially ABS-P and ABS-Q are well designed and suitable.\n\nSome minor questions/suggestions:\n1)\tDoes the cost R consider both weight and activation bits? If yes, the input resolution change should affect BOPs and will change the optimal bit-width. It would be meaningful to show the correlation between optimal bit-width and the number of filters (but not necessary).\n2)\tHow is the ‘search cost’ calculated? As mentioned, the key point of the algorithm is to simultaneously train weights and configurations. Is ‘search time’ means the fine-tuning process after training the uncompressed network? Is it equal to other works? (Table 5)\n3)\tIn Figure 3 & 4, there is no layer assigned to 2-bit. Can you provide average bit-width for ABS networks?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U_K4jyhHpK7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Inspired by gradient-based NAS of single-path formulation, the authors propose a super-bit model, a single-path method, to decide the optimal number of quantization bits and pruning of a group of filters. While it can be a time-consuming process to study the impact of quantization of certain filters (or layers) on model accuracy, the proposed scheme finds a particular compression configuration in a trainable manner. The experimental results show that the proposed method presents higher model accuracy or lower computational cost (measured as the bit-operation count).\n\nWhile the idea of presenting trainable binary quantization gate signals to automatically assign the number of quantization bits to layers is interesting, this reviewer has the following concerns:\n\n- For Table 1 and 2, are all methods assuming the same quantization configurations (e.g., layer-wise vs channel-wise and symmetric vs asymmatric, etc..)? Detailed quantization configuration should be included for fair comparisons.\n\n- For Table 1 and 2, why top-1 accuracy numbers are very high for all methods? How 6-bit precision of MobileNetV2 can be 71.8% (almost no accuracy degradation)? Moreover, the accuracy of the 4-bit-quantized ResNet-18 on ImageNet is even higher than that of full-precision ResNet-18. Even so, why not decrease BOPs while lowering top-1 accuracy for 'ABS (ours)' such that the gain of ABS is maximized?\n\n- BOPs look quite similar for many configurations (including the second row of a fixed number of quantization bits). Even though this reviewer does not believe BOPs can show performance benefits in hardware well, this reviewer cannot see distinguished advantages of ABS in Table 1 and Table 2. The gain on BOPs of ABS seems to be marginal.\n\n- The results do not describe memory footprint savings. Can ABS provide a reduced memory footprint compared to previous ones for a target model accuracy? In Figure 2, if BOPs can be replaced with memory footprints, it would be useful to estimate the compression capability of the proposed method. In Figure 3 and 4, comparisons are missing.\n\n- How can we optimize alpha_p and alpha_q? Is it sensitive to model accuracy? If the goal of this work is a fast exploration of compression configurations, optimizing alpha_p and alpha_q should be easy, but there is no relevant information in the manuscript (hence, Table 5 is not very reliable).\n\nOverall, unfortunately, this reviewer cannot easily find the advantages of the proposed methods in the experiments. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-hrzH16qpf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\tThe paper presents a technique called ABS (Automatic Bit Sharing), that combines group-wise filter pruning with residual-aware quantisation in a NAS framework. The presented results on a range of image classification tasks show great performance when evaluating using an Accuracy/BOps metric. In general, I think the paper shipped interesting ideas in terms of automating residual quantisation, but it could be better if the network architecture space is jointly optimised.\n\n\nStrength:\n1. The paper is well-written (although it has some minor flaws), but in general easy to follow.\n2. I like the formulation of using thresholds to automatically decide numerical representations, this looks neat and novel to me.\n3. It is the first time for me to see residual quantisation in a NAS framework and the formulation of the indicator function looks interesting.\n4. The composed search space gives a reduction in search time.\n\nWeakness:\n1. Both channel pruning and residual quantisation are existing ideas, this might reduce the novelty aspect of this paper.\n2. This work focuses on searching for compression on pretrained architectures, such as ResNets and MobileNets. However, these models are not state-of-the-art on the considered datasets, and one might argue that these models are inherently more redundant than models like MobileNetV3. Also, the authors do not consider jointly optimise the network architectures, which could naturally be an extension of this work and might help to further boost the accuracy. \n\nMy suggestions & confusions:\n1. I stumbled with the number representation until finishing reading section 3.1. It is probably worth mentioning that you are applying fixed-point quantisation to residual terms at the very beginning of the paper, especially in figure 1.\n2. The argument that you’ve introduced 0-bit quantisation for pruning is very misleading. Apparently your pruning is channel-wise but quantisation is layer-wise, it is naturally to consider you used 0-bit quantisation on the entire layer, but this is not the case.\n3. In Page 6, you mentioned you train the binary gates for weights and activations alternatively and this provides better performance. Could you quantify the performance gap? Also, if one simply looks at eq(17), this interleaved optimisation is not captured in this over-simplified optimisation target. \n4. Third line of page 6, I guess you mean group-wise sparsity?\n5. Page 6: Specifically, if a group of filters are ... -> if a group of filters is …?\n6. You mentioned you are using group-wise sparsity, do you show the number of groups anywhere? What is your choice of B in Eq(16) for all the experiments? Or you simply mean you are doing channel-wise sparisity? I am a little confused here.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to integrate multiple bit configurations (including pruning) into a single architecture, and then automatically select bit resolution through binary gates. The overall approach can be differentiable and optimized with parameters. However, as pointed out by the reviewers, the novelty of this paper can be the big question. Also, it seems to be hard or unpractical to employ different number of bits for layers, given the standard GPU and CPU hardware. 
Decision: Reject