Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bitwise Regularization
Authors: Jung Hyun Lee
Jihun Yun
Sung Ju Hwang
Eunho Yang
Venue:  (2021)
Keywords: Quantization
Compression
Efficient Inference
Deep Learning

Abstract: Network quantization, which aims to reduce the bit-lengths of the network weights and activations, has emerged as one of the key ingredients to reduce the size of neural networks for their deployments to resource-limited devices. In order to overcome the nature of transforming continuous activations and weights to discrete ones, recent study called Relaxed Quantization (RQ) [Louizos et al. 2019] successfully employ the popular Gumbel-Softmax that allows this transformation with efficient gradient-based optimization. However, RQ with this Gumbel-Softmax relaxation still suffers from bias-variance trade-off depending on the temperature parameter of Gumbel-Softmax. To resolve the issue, we propose a novel method, Semi-Relaxed Quantization (SRQ) that uses multi-class straight-through estimator to effectively reduce the bias and variance, along with a new regularization technique, DropBits that replaces dropout regularization to randomly drop the bits instead of neurons to further reduce the bias of the multi-class straight-through estimator in SRQ. As a natural extension of DropBits, we further introduce the way of learning heterogeneous quantization levels to find proper bit-length for each layer using DropBits. We experimentally validate our method on various benchmark datasets and network architectures, and also support the quantized lottery ticket hypothesis: learning heterogeneous quantization levels outperforms the case using the same but fixed quantization levels from scratch.

Under review as a conference paper at ICLR 2021
SEMI-RELAXED QUANTIZATION WITH DROPBITS: TRAINING LOW-BIT NEURAL NETWORKS VIA BIT- WISE REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Network quantization, which aims to reduce the bit-lengths of the network weights and activations, has emerged one of the key ingredients to reduce the size neural networks for their deployments to resource-limited devices. order to overcome the nature of transforming continuous activations and weights to discrete ones, recent study called Relaxed Quantization (RQ) [Louizos al. 2019] successfully employ the popular Gumbel-Softuax that allows this transformation with efficient gradient-based optimization. However, RQ with this Gumbel-Softmax relaxation still suffers from large quantization error due the high temperature for low variance of gradients, hence showing suboptimal performance. To resolve the issue, we propose a novel method, Remi-Relaxed Quantization (SRQ) that uses multi-class straight-thoough estimator to effectively reduce the quantization error, along with new regularization technique, DropBits that replaces dropout regularization to randomly drop the bits instead of neurons to reduce the distribution bias the multi-class straight-through estimator SRQ. As a natural extension of DropBits, we further introduce the way of learning heterogeneous quantization levels find proper bit-length for each layer using DropBits. We experimentally validate our method various benchmark datasets and network architectures, and also support new hypothesis for quantization: learning heterogeneous quantization levels outperforms the case using the same but fixed quantization levels from scratch.
INTRODUCTION
Deep neural networks have achieved great success in various computer vision applications such as image classification, object detection/segmeetation pose estimation, action recognition, and so on. However, state-of-the-art neural network architectures require too much computation and memory to be deployed to resource- limuree devices. Therefore, researchers have been exploring various approaches to compress deep neural networks to reduce their memory usage and computation cost.
In this paper, we focus on neural network quantizaaion, which aims to reduce the bit- width of a neural network while maintaining competitive performance with a full-precision network. It is typically divided into two groups, uniform and heterogeneous quantization. In uniform quantization, the simplest methods is to round the full-precision weights and activations the nearest grid points: f a - / where controls the grid interval size. However, this naïve approach incurs severe performance degradation on large datasets. Recent network quantization methods tackle this problem from different perspectives. In particular, Relaxed Quantization (RQ) (Louizos et al.) 2019) employs Gumbel-Softmax (Jang et al.) 2017; Maddison et al. 2017) to force weights and activations to be located near quantization grids with high density. Louizos et al. (2019) notice the importance of keeping the gradient variance small, which leads them to use high Gumbel-Softme
of
temperatures in RQ. However, such high temperatures may cause large quantization error, thus preventing quantized networks from achieving comparable performance full-precision networks.

---

Under review as a conference paper at ICLR 2021
To resolve this issue, first propose Semi-Relaxed Quantization (SRQ) that uses the mode of the original categorical distribution in the forward pass, which induces small quantization error. It is clearly distinguished from distribution. To cluster weights cohesively around quantization grid points, we devise a multi-class straight-through estimator (STE) that allows for efficient gradient- baadee optimization as well. As this STE biased like a traditional one (Bengio et al, 2013) for the binary case, we present a novel technique, DropBits reduce the distribution bias of the multi-calss STE in SRQ. Motivated from Dropout (Srivastava al. 2014), DropBits drops bits rather than neurons/filters train low-bit neural networks under SRQ framework.
choosing argmaz among the samples from the concrete
In addition to uniform quantization, DropBits allows for heterogeneous quantization, which learn different bit-width per parameter/channellayye by dropping redundant bits. DropBits with learnable bit-drop rates adaptively finds out the optimal bit-width for each group of parameters, possibly further reducing the overall bits. In contrast to recent studies (Wang et al. 2019 Uhlich et al. 2020) in heterogencous quantization that exhibit almost all layers possess at least 4 bits, up to 10-bit, our method yields much more resource-efficien low-bit neural networks with at most 4 bits for all layers.
With trainable bit-widths, we also articulate new hypothesis for quantization where one can find the learned bit-width network (termed 'quantized sub-network') which can perform better than the network with the same but fixed bit-widths from scratch.
Our contribution is threefold:
We propose new quantization method, Semi- Remi-Relaxed Quantization (SRQ) that introduces the multi-class straight- through estimator to reduce the quantization error of Relaxed Quantization for transforming continuous activations and weights to discrete ones. We further present a novel technique, DropBits to reduce the distribution bias of the multi-class straight-through estimator in SRQ.
Extending DropBits technique, we propose a more resource-efficien heterogeneous quantization algorithm to curtail redundant bit-widths across groups of weights and/or activations (c.g. across layers) and verify that our method able to find out 'quantized sub-networks'.
We conduct extensive experiments on several benchmark datasets to demonstrate the effec- tiveness of our method. We accomplish new state-of-the-art results for ResNet-18 and Mo- bileNetV2 on the ImageNet dataset when all layers are uniformly quantized.
RELATED WORK
While our goal in this work is to obtain an extremely low-bit neural network both for weights and activations, here we broadly discuss existing quantization techniques with various goals and settings. BinaryConnect (Courbariaux et al. 2013) first attempted to binarize weights to +1 by employing deterministic or stochastic operation. To obtain better performance, various studies (Rastegari et al.) 2016, Li et al. 2016 Achterhold et al. 2018 Shayer et al. 2018) have been conducted in binarization and ternarization. To reduce hardware cost for inference, Geng et al.) (2019) proposed softmax approximation via a look-up table. Although these works effectively decrease the model size and raise the accuracy, they are limited to quantizing weights with activations remained in full-precision. To take full ad advantage of quantization at run-time, it is necessary to quantize activations well.
Researchers have recently focused more on simultaneously quantizing both weights and activations (Zhou et al. 2016 Yin et al. 2018 Choi et al. 2018 Zhang et al. 2018 Gong et al., 2019: Jung et al. 2019] Esser et al. 2020). XNOR-Net (Rastegari et al. 2016), the beginning of this line work, exploits the efficiency of XNOR and bit-counting operations. QIL (Jung et al., 2019) also quantizes weights and activations by introducing parametrized learnable quantizers that can be trained jointly with weight parameters. Esser et al.) (2020) recently presented simple technique to approximate the gradients with respect to the grid interval size to improve QIL. Nevertheless, these methods do not quantize the first or last layer, which leaves room to improve power-efficiency.
2

---

Under review as a conference paper at ICLR 2021
For ease of deployment in practice, it is inevitable quantize weights and activations of all layers, which is the most challenging. Louizos (oouzoos et al. (219)) proposed to cluster weights by using Gumbel-
Softmax, but it shows drawbacks as we will discuss in Section 3.2 Jain et al. (2019) presented
efficient fixed-point implementations by formulating the grid interval size to the power of two, but
they quantized the first and last layer to at least 8-bit. ZZhao et al. (020 proposed to quantize the grid
interval size and network parameters in batch normalization for the deployment of quantized models
on low-bit integer hardware, but requires a specific accelerator only for this approach.
As another line of work, Fromm al. (2018) proposed heterogeneous binarization given pre-defined bit-distribution. HAWQ (Dong al. 2019) determines the bit-width for each block heuristically based the top eigenvalue of Hessian. Unfortunately, both of them do not learn optimal bit-widths for heterogeneity. Toward this, Wang al. (2019) and Uhlich et al.) (2020) proposed layer-wise heterogeneous quantization by exploiting reinforcement learning and learning dynamic range of quantizers, respectively. However, their results exhibit that almost all layers have up to 10-bit (at least 4-bit), which would be suboptimal. [ou et al. (2020) presented a channel-wist heterogeneous quantization by exploiting hierarchical reinforcement learning, but channel-wist precision limits the structure of accelerators, thereby restricting the applicability of the model.
3 METHOD
In this section, we briefly review Relaxed Quantization (RQ) (Louizos al.. 2019) and propose Semi-Relaxed Quantization, which selects the nearest grid point in the forward pass to decrease the quantization error. To make it learnable and to cluster compressed parameters cohesively, SRQ expresses the nearest grid selection of the forward pass as the equivalent form, the combination of logistic distribution and argmax, and performs the backward pass on it. Then, we present DropBits technique to reduce the distribution bias of SRQ and its extension to heterogeneous quantization.
3.1 PRELIMINARY: RELAXED QUANTIZATION
Relaxed Quantization (RQ) considers the following quantization grids for weights: & a[-25-1 ..... 0, .,......................... 2h 1] =: [9o..... ........9''-111. where b is the bit-width and a learnable parameter for each layer controls grid interval. When quantizing activations, the grid points in & start from zero since the output of ReLU is always non-negative. Then, æ (a weight or an activation) is which enables gradient-based optimization for nonddifferentiable
perturbed by noise rounding. The noise e follows distribution p(c) Logistic(O,, o) so that p(T) is governed by Logistic( σ) where represents the standard deviation. Under p(7), we can easily compute the unnormalized probability of 7 being quantized to each grid point 9: in closed form as below:
F; = p( gel ar, o) = Sigmoid ((g: + aa/2 - /0) Sigmoid ((y) - //////)/)) (1)
where è denotes quantized realization of ž. Note that the cumulative distribution function of the logistic distribution is just a sigmoid function. Finally, given unnormalized categorical probability
m - for grid points ς = {g. 122 RQ discretizes x r.G by sampling r (r=)2"0
from the concrete distribution (Jang al. 2017 Maddison et 2017) with temperature
cxp (llog n, uz)/T)
2h
04 ~ Gumbel(O, 1), Ti =
f
Σ 1=0
righ.
(2)
Σ/= exp ((log ,, + u, uz)/T)
The algorithm of RQ is described in Appendix in detail.
3.2 SEMI-RELAXED QUANTIZATION FIXING PITFALLS OF RQ
Although RQ achieves competitive performance with both weights and activations of neural networks quantized, the quantization probability modeling of RQ may still incur large quantization error, thereby yielding suboptimal performance. To be specific, Louizos et al. (2019) recommend high temperatures for the concrete distribution (e.g. 1.0 or 2.0) in ② since exploiting low tempera- tures hinders networks from converging due to high variance of gradients. However, it turns out that
3

---

Under review as a conference paper at ICLR 2021
RQ baseline
baseline
0.6
0.6
0.6
Probbblity
0.4
Probabilly
0.4
Prebabikty
0.4
0.3
0.3
0.3
0.1
0.1
2a
-1
-2a
-0
-2
(b)
(c)
0.6
RO bascline
0.6
RD baseline
0.4
Gumbel10.1)
0.6
SRO (Durs)
Protablitty Digra butbo
(a)
0.5
0.5
0.3
0.5
0.4
0.2
0.1
0.
0.2
0.2
0.1
0.1
0.1
24
-2
24
22 -0
(d)
(e)
(I)
(g)
Figure 1: Probability of being quantized each grid point by RQ (r 1.0) and SRQ respectively. (a e): RQ according to different Gumbel samples, (f): Gumbel samples used in (a e), and (g): SRQ. the same as the original categorical distribution. The z-axis denotes weight value (except (f)), denotes the original value a/2. computed by
For RQ (a~-e), represents 3 for input
the concrete distribution with such a high temperature almost similar to the uniform distribution. As concrete example, we consider 2-bit quantization with G a 11, 0, for fixed scale parameter 0, a a/3, and we set 7 to 1.0 in Louizos al. (2019). Now, suppose that the original weight value is az ax/2. As in Figure (b,d,e) RQ as the original categorical distribution has support for and -2a. It may be okay average, but RQ computes only one sample in each forward pass due to computational burden, which can accidentally lead to very large quantization error for these particular sample.
be sporadically quantized to below zero by
To avoid the counterintuitiv--aappl with large quantization error as seen in Figure I (b.d.c), we propose "Semi-Relaxed Quantization' (SRQ) which rather directly considers the original categorical distribution in Figure D (g). To be concrete, for weight or an activation æ, the probability x being quantized each grid is r; = n. " y= Ty for E {0.... 2 1 with b-bit precision, where Ti computed as [1. In such manner, selecting grid point for
be thought of as sampling from the categorical distribution with categories = {9: - and the corresponding probabilities 2--1 as illustrated in Figure (g). Then, the grid point 9i.... with 'max argmax, r; would be the most reasonable speculation due the highest probability. SRQ therefore chooses the mode of the original categorical distribution, Dim.. and assign it T, entirely discriminated from Gumbel-Softmax which selects the argmax among samples from the concrete distribution. As result, SRQ does not suffer from counterintutive-sane
)
problem that RQ encounters at all.
The last essential part for SRQ is to handle the non con-differentiable argmax operator in computing imax- Toward this, we propose a multi-class straight-through estimator (STE) that allows for backpropagat- ing through a mon-differentiaale categorical sample by approximating àC/Or.... to OL/O 2//0..... and 0L/Or, to zero for * 'max. where the cross entropy between true label and the prediction made by a quantized neural network as delineated in the previous paragraph and 1/imax is the imax-th entry of the one-hot vector y. The forward and backward passes of SRQ are summarized follows.
ØL
OL and dc Ōr:
Forward: y = one hothargma//arri.. r:), Backward:
for i imax (3)
Dr.... dy:....
Such formulation brings two important advantages in network quantization. First of all, 3 makes
the variance of gradient estimator become zero. Since SRQ always chooses the mode of the original categorical distribution (i.e., there is no randomness in the forward pass SRQ), and the gradient of loss function C with respect to the individual categorical probabilities is defined as zero everywhere except for the coordinate corresponding to the mode, the variance of gradients SRQ indeed zero.
The other advantage is that the backward pass rameters cohesively. Under the assumption that
(3) can cluster network weight pa- 8C mi.
8z is proportional to
4

---

Under review as a conference paper at ICLR 2021
2000
Layer
15000:
Layer
8000
Layer 2
Layer
1500
40000
1000
4000
500
5000
2000
20000
0.1
0.0
0.44
-02
0.0
0.2
0.4
502
-0.1
0.0
0.1
0.2
10.4
-4.2
0.0
0.2
0.4
(a) RQ
(h) SRQ
Figure 2: Weight distributions for 3-bit quantized LeNet-S by (a) RQ and (b) SRQ. The æ-axis and y-axis represent the weight values and their frequencies, respectively. The vertical dashed lines denote grid points.
OTra 1 (Sigmoid( Prne Sanntaa/200 Sigmoid( 9rm ta/2==)/=)) Sigmoid 2 P5m--0/2-1 Sigmoid de
a )
Pie -) When is close to Pimss iT i /0z is nearly zero, so is 0C/Oz. With an appropri- ate learning rate, x converges Pimes which leads SRQ cluster weights better than RQ as shown in ac is almost zero, a is still trained. After a is updated, there is gap between Figure ] Although öi
and a so that x can be trained. Hence, the network will continue to train until it finds the optimal a.
3.3 DROPBITS
Although our multi-class STE enjoys low variance of gradients, it is biased to the mode as the binary one in inngnnoo et al. (0013). To reduce the bias of a STE, Chung et al. (2016) propose the slope anneal- ing trick, but this strategy is only applicable to the binary case. To address this limitation, we propose novel method, DropBits, to decrease the distribution bias of multi-class STE. Inspired by dropping neurons in Dropout (Srivastava al. 2014). we drop an arbitrary number of grid points at random every iteration, where in effect the probability of being quantized to dropped grid points becomes zero. However, the design policy that each grid point has its own binary mask would make the num- ber of masks increase exponentially with bit- width. Taking into account appropriate noise levels with a less aggressive design, the follow- ing two examples are available: (a) endpoints in the grids share the same binary mask, and (b) the grid points in the same bit-level share the same binary mask (see Figure 3 Hereafter, consider (b) bitwise-sharine masks for groups of grid points, unless otherwise specified.
E
(a) audppints-sharing mask (b) Bitwise-sharing mask
Figure 3: Two mask designs in 3-bit
Now, we introduce how to formulate binary masks. Unlike practical Dropout implementation through dividing activations by 1 - (here, p is a dropout probability), we employ an explicit binary mask Z whose probability II can be optimized jointly with model parameters. The Bernoulli random variable being non-differentiable, we relax binary mask via the hard concrete distribution (Louizos et al. 2018). While the binary concrete distribution (Maddison et al., 2017) has its support (0, 1), the hard concrete distribution stretches it slightly at both ends, thus concentrating more mass on exact 0 and 1. Assuming disjoint masks, describe the construction of binary mask Zμ. for the k-th bit-level using the hard concrete distribution as
Ug Uniform(0, 1), S₃ = Sigmoid (log log (1 Ux) + log I1 log(1 H+))/T) (4)
S, = Sx(s - + and Z/ = min (max( (xx 0),
where 7' is temperature for the hard concrete distributions with 7 0 and Ç 0 reflecting stretching level. For i 2 - 1, 2 and 2 + 1, we do not sample from the above procedure but fix Z = 1 so to prohibit all the binary masks from becoming zero (see 'No Mask' in Figure B
With the value of each mask generated from the above procedure, the probability of being quantized to any grid point is re-calculated by multiplying म; 's by their corresponding binary masks Z₁ (e.g. To- = Zn mo) and then normalizing them (to sum to 1). As seen in Figure the sampling distribution
Sinne imaz does change, the assumption is unreasonable.
5

---

Under review as a conference paper at ICLR 2021
10
2.8
Probubbley
Probabilty
Proboabilyy
Viaiuee
Vo
Val lans
(a) Categorical distribution
(b) SRQ
(c) SRQ DropBits
Figure 4: The illustration of the effect of DropBits on SRQ. For a certain weight, (a) the categorical distribution
indicates r; = F o for each grid (i = O, .7), (b) the distribution of SRQ is sampling distribution
after taking the argmax of r. ns o0 and (c) the distribution of SRQ DropBits is a sampling distribution after taking the argmax r, n. 2 -- R... Here, 11. are initialized to 0.7 for clear understanding.
For each entry th weight W:
Compute the srobabiity fellingg on point
For
Bockprop sia Striigh-Throsgh
Choose the grid point
quantied
14h liyer
probul
highest prebiiiyyy Yu with anginax
with
weight
Figure Illustration Semi-Relaxed Quantization (SRQ) framework with DropBits technique.
of SRQ is biased the mode, -3a. With DropBits adjusting मं to T.' based on Zy 's, the sampling distribution of SRQ + DropBits more resemble the original categorical distribution than that of SRQ. which means that DropBits could effectively reduce the distribution bias of the multi-class straight- through estimator in SRQ. Not only that, DropBits does not require any hand-crafted scheduling at all due to the learnable characteristic of Ih. whereas such scheduling is vital for Humbel-Softman
(Jang et al. 2017 Maddison et al., 2017) and slope annealing trick (Chung et al. 2016).
Although quantization grids for weights are symmetric with respect to zero, those for activations start from zero, which makes it difficult to exploit symmetrically-designe
DropBits for activations.
Therefore, DropBits applied only for weights in our experiments. Assuming that Zu are shared The overall algorithm
across all weights of each layer. the overall procedure is described in Figure [S] including the test phase deferred to Appendix due space limit.
3.4 EMPIRICAL ANALYSIS OF QUANTIZATION ERROR, DISTRIBUTION BIAS, VARIANCE
In this section, we empirically compare (i) the expectation of quantization error, (ii) the f--nrrm of distribution bias, and (iii) the gradient variance of RQ, SRQ, and SRQ + DropBits. The expectation of quantization error indicates the expected value of difference between an input value and its quantized value, E| ] where स is a quantized value of æ by each algorithm. Here, the expectation is taken over the randomness of T.
As described in Appendix is not possible to compute the bias of gradient with respect to π,, so we instead compute the bias gradient with respect as proxy. However, it is not straightforward to say that SRQ + Drop is better than RQ and vice versa in terms of the bias with respect (see Figure (b) in Appendix B). As additional indirect metric, we suggest distribution bias as the difference between the original categorical distribution and the distribution approximated by each
algorithm, [p] Torigin where a vector Torigin == (T./ -2" TT))"" is the original categorical distribution and a vector p == (p.)²"0² is the approximated one (i.e., (p,))²² = (r:))"²' in 2 for RQ, and (p: (==) (yx)² )² in 3 for SRQ (+ DropBits)). As a distribution bias is also a vector, compare the fo-m of a distribution bias of each algorithm. Note that our new notion of distribution bias is very loosely coupled with the gradient in the sense that both converge to zero if there is no approximation for the non-differentiable parts. However, it can be used as an indirect indicator to measure how much biased distribution is used in computing the gradient estimator.
For the variance, we use the floeddffrm of the gradient estimator Ør of each algorithm: for * j. dn (a) RQ: ar; nur with randomness on u; and uj, and (b) SRQ + DropBits: θн, Øx, rxy
2, with randomness binary masks 7; and Zy. The case for i j can be derived similarly.
âr. = r.ty

---

Under review as a conference paper at ICLR 2021
Disrrbutin Bloe
6.15
Varfunce of
6.15
0.10
6.2
335
S5 35
23p
230
-350
-230
159
2rp0
Figure Comparison of RQ, SRQ, and SRQ DropBits quantization error/distribution bias/variance.
Armed with these terms, we conduct a case study 3-bit quantization with grid points Č al-4, 3] via Monte Carlo simulation when a is simply set to 1.0. Here, let z be the midpoint of -3.5a. 2.5a. For gradient variance, since there are many pairs
consecutive grid points, i... of dri for different i and j, we representatively choose as the index of the grid point closest to z, 3ny and j as the indices of two neighboring grid points of x (e.g. if x E 1g 9il, then j = i- - and i).
In Figure 6 SRQ shows smaller quantization error than RQ for all æ - -3.5a, ... 2.5a. This because SRQ deliberately performs biased estimation on the underlying categorical distribution to prevent large quantization error from even occurring in the forward pass while sharing this underlying distribution with RQ in the backward pass. Instead, distribution bias of SRQ. which is indeed the case can be seen in Figure Interestingly, SRQ + DropBits can also achieve smaller distribution bias than RQ for all x = -3.50, ... 2.5a.
devised DropBits reduce the incurred
3.5 LEARNING BIT-WIDTH TOWARDS RESOURCE- RESOURCEE-EFFRCIENNY
recent studies on heterogeneous quantization use at least 4-bit in almost all layers, up to 10-bit, which leaves much room for the saving of energy and memory. Towards more resource-efficient one, we introduce an additional regularization on DropBits to drop redundant bits.
As noted in Section D and 2
Since the mask design in Figure (b) reflects the actual bit-level and the probability of each binary mask in DropBits is learnable, we can penalize the case where we use higher bit-levels via a sparsity encouraging regularizer like & Louioozss et al. (2018) proposed relaxed éu regularization using the hard concrete binary mask, we adopt this continuous version of lo as a sparsity inducing regularizer. Following (4), we define the smoothed fn-norm as R(Z; II) = Sigmoid I-TT - T log -7) One caveat here is that we do not have to regularize masks for low bit-level a higher bit-level is still alive (in this case such a high bit-level is still necessary for quantization). We thus design a regularization in such a specific way as only to permit the probability of binary mask at the current highest bit-level to approach zero. More concretely, for bit-level binary masks { [ 1 as in Figure (b) and the corresponding probabilities {TT,) k= our regularization term to learn the wit-width b-1
II
b-1
R(ZzH (IL - I(Z > o)(
II I(2,
I(2,
- 0)
Note that {Zx | f-1 is assigned to each group (e.g. all weights activations in layer or channel for instance). Hence, every weight in group shares the same sparsity pattern (and bit-width result), and learned bit-widths across groups are allowed to be heterogeneous.
1=1
k+1
Assuming the 1-th layer shares binary masks Z = {z/)==) associated with probabilities IT' = (TT ( b-1 our final objective function for a L-layer neural network becomes L(O, a, IIII)
ΣΣ -- R(Z II'), where a = (ar)t= and o = {ot) represent the layer-wise grid interval parameters and standard deviations of logistic distributions, Z = = |/ II (II') (II') and is a regularization parameter. inference phase, we just drop unnecessary bits based on the values of II.
3.6 NEW HYPOTHESIS FOR QUANTIZATION
Frankle & Carbin 2019) articulated the 'lottery ticket hypothesis', stating that one can find some sparse sub-networks, 'winning tickets', from randomly-initialzzed, dense neural networks that are easier to train than sparse networks resulting from pruning. In this section, we define a new hypothesis for quantization with slightly different (opposite some sense) perspective from the original one. Notation. a bit b and -itt. b denote that has strictly higher bit-width than b for at least one of all groups (c.g. channels or layers), and has the same bit bre-preci as b across all groups, respectively. Definition. For a network [x; 0) with randomly-initiaizeed parameters 0, we consider quantized network (x:; 0') from (x; 0) such that 0 >is 0'. If the accuracy of f [a;; 0*) is higher than that
7

---

Under review as a conference paper at ICLR 2021
Table I: Test error (%) for LeNet-5 MNIST and VGG-7 on CIFAR-I0. "Ann." stands for annealing the temperature 7 in RQ-
Learning Curve
100
90-
Training Accuracy (8)
80-
Dataset
# Bits
W.A.A
RQ
RQ+ Ann. 2
SRQ
SRQ + DropBits
70-
100
98
4/4
0.58
0.62
0.59
0.53
270
250
290
300
MNIST
3/3
0.69
0.74
0.67
0.58
40-
SRQ
2/2
0.76
0.72
0.63
30
20
RQ w/ Anncoling
4/4
8.43
8.47
7.15
6.85
50
100
150
200
250
300
CIFAR-10
3/3
9.56
10.78
7.08
6.94
Epochs
Figure 7: Learning curves of VGG-7 quantized by
2/2
11.75
7.68
7.51
RQ. RQ with annealing т, and SRQ in 3-bit.
Table 2: 1op-1/Top-5 error (%) with ResNet-18 and MobileNetV2 on ImageNet.
Bits
ResNet-18 1op-1/Top-5
MobileNetV2 1opp1/
Method
W.A.
Full precision
32/32
30.24/10.92
28.12/9.71
RQ Louizos
2019
4/4
38.45/16.01
-/-
RQ
ST
Louizos et al
2019
4/4
37.54/ 15.22
-1-
QIL Jung
2019
4/4
31.05/11.23
32.77/12.51
L.FQQF Than et al.
2020
4/4
30.60/11.28
32.63/12.01
3/3
33.33/12.58
-/-
TQT
Jain cl.
2019
Ublich et al. 2020
4/4
30.19/-
32.21/
SRQ DropBits
4/4
30.37/10.96
30.83/11.0
3/3
32. 32.79/12.57
36.96/
15.
of f fa; (x;;")) [z; 0') is referred to as a quantized sub-network of f(;; 0).
0*) where fx; 0") is trained from scratch with fixed bit-widths such that 0' -bit 0", then
This hypothesis implies that learning bit-width would be superior to pre-defined bit-width. To the best of our knowledge, our study the first attempt to delve into this hypothesis.
4 EXPERIMENTS
Since popular deep learning libraries such TensorFlow (Abadi al. 2016) and PyTorch from v1.3 (Paszke et al., 2019) already provide their own 8-bit quantization functionalities, focus lower bit-width regimes (i.e. 4-bit). In contrast to some other quantization papers, our method uniformly quantizes the weights and activations of all layers containing both the first and last layers. We first show that SRQ and DropBits have its own contribution, none of which is negligible. Then, we evaluate our method, SRQ + DropBits on totally large-scale dataset with deep networks. Finally, we demonstrate that our heterogeneous quantization method yields promising results even all layers have at most 4-bit and validate new hypothesis for quantization in Section 3.6
4.1 ABLATION STUDIES
To validate the efficacy of SRQ and DropBits, we successively apply each piece of our method to RQ for LeNet-5 (LeCun al., 1998) on MNIST and VGG-7 (Simonyan & Zisserman, (014) on CIFAR-10. Table shows that SRQ outperforms RQ in most cases. One might wonder that the issue of RQ introduced in Section 3.2 can be addressed by an annealing schedule of the temperature 7 in RQ. It could be possible, but RQ with an annealing schedule suffers from high variance of gradients due to low temperatures at the end of training as shown in Figure 7 As a result, annealing 7 gives rise to worse performance than RQ as shown in Table I However, SRQ does not suffer from both problems at all, thus displaying the best learning curve in Figure 7 Finally, it can be clearly identified that Brop consistently improves SRQ by decreasing the distribution bias our multi-class STE.
We cannot reproduce the results of RQ in 2-bit, so we experiment only on 3-bit and 4-bit RQ Our implementation with all layers quantized by using pretrained models available from PyTorch
8

---

Under review as a conference paper at ICLR 2021
Table 3: Test (%) for quantized sub-networks using LeNet-S MNIST, VGG-7 CIFAR-IO and
ResNet-18 on ImageNet. Here, an underline means the learned bit-width and "T" stands for ternary precision.
Initial Bits
Trained W. Bits
Test Eroor (Fixed)
Test
Test Error
Model
Ermr
W/A
per layer
(Reg.)
4/4
0.53
4/4/3/4
0,55
0.52
LcNet-5
3/3
0.58
3/2/3/3
0.65
0.55
2/2
0.63
2/2/2/T
0.68
0.59
4/4
6.77
4/4/4/4/4/3/3/4
6.74
6.65
VGG-7
3/3
6.82
3/3/3/3///2////
6.81
6.77
2/2
7.49
2/2/2/2/2/2/2/22
7.43
7.36
4/4
33.20
4/3/3/3/
34.58
34.30
R8sNet-18
3/3
37.80
3/3/
41.01
40.30
4.2 RESNET- AND MOBILENETV2 ON IMAGENET
To verify the effectiveness of our algorithm the ImageNet dataset, quantize the ResNet-18 (He et al. 2016) and MobileNetV2 (Sandler et al. 2018) architectures initialized with each pre-trained full-precision network available from the official PyTorch repository. In Table 2 our method is only compared to the state-of-ther -frt---r-t algorithms that quantize both weights and activations of all layers for fair comparisons. The extensive comparison against recent works that remain the first or last layer in the full-precision is given in Appendix.
Table 2 illustrates how much better our model performs than the latest quantization methods as well as our baseline, RQ. In ResNet-18, SRQ + DropBits outdoes RQ. QIL, LLSQF, and TQT. even achieving the top-1 and top-5 errors 4-bit nearly close those of the full-precision network. In MobileNetV2, SRQ DropBits with 4-bit surpasses all existing studies by more than percentage point. Moreover, quantize MobileNetV2 to 3-bit, obtaining competitive performance, which is remarkable due to the fact that none of previous works successfully quantizes MobileNetV2 to 3-bit.
4.3 FINDING QUANTIZED SUB-NETWORKS
In this experiment, we validate a new hypothesis for quantization by training the probabilities of binary masks using the regularizer Section 3.. Sc.5ion3..5to learn the bit-width of each layer. For brevity, only weights are heterogeneously quantized, and the bit-width for activations is fixed the initial one.
In Table 3 the fourth column represents the bit-width per layer learned by regularizer, and the fifth and last columns indicate the test error when fixing the bit-width of each layer same trained bit-widths (fourth column) from scratch and when using our regularization approach, respectively. 3 shows that learned structure by our heterogeneous quantization method (last column) is superior to the fixed structure with learned bit-widths from scratch (fifth column) for all cases. might be doubtful whether our regularizer is able to recognize which layer is really redundant or not. This may indirectly substantiated by the observation that the fixed structure with trained bit-widths from scratch (fifth column) outperforms the uniform quantization (third column) on CIFAR-10. More experiments different values of the regularization parameter À are deferred Appendix.
CONCLUSION
Quantization (SRQ), which effectively clusters the weights in low bit- width regimes, along with DropBits that reduces the distribution bias of SRQ. We empirically showed that both SRQ and DropBits possess its own value, thereby leading SRQ + DropBits to achieve the state-of-the-art performance for ResNet-18 and MobileNetV2 on ImageNet. Furthermore, we took one step forward consider heterogeneous quantization by simply penalizing binary masks in Drsp which enables find quantized sub-networks. As future work, we plan extend heterogeneous quantization method activations and its application other quantizers.
We proposed

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow; A system for large-scale machine learning. In USENIX Symposium on Operating Systems Design and Implementation, pp.
265-283, 2016.
Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantization. In International Conference Learning Representations, 2018. URL https
net / TW-Wb WAb
Yoshua Bengio, Nicholas Leonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 3xXiv:330..3433 2013.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural
networks. CoRR, htss/1805.06085 2018. /tL/tttp///axx....rr/////////// 06085
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations, 2016. UtL/ssttss///
net / forum?
Matthieu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 3123-3131. Curran Associates, Inc., 2015.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed seexed-preciion.. In The IEEE International Conference on Computer Vision (ICCV), 2019.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar-
mendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. URL https: httttp:://operrevier .nw. feee:/oooummmd==00055555
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019. URL https
//openreview.oo net. wor.m?id-rrdrrr po---mmid--r1--3Re77
Joshua Fromm, Shwetak Patel, and Matthai Philipose. Heterogeneous bitwidth binarization in convolutional neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 4006 4015. Curran Associates, Inc., 2018
Xue Geng, Jie Lin, Bin Zhao, Anmin Kong, Mohamed M. Sabry Aly, and Vijay Chandrasekhar, Hardware-aware softmax approximation for deep neural networks. In C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler (eds.), Computer Vision - ACCV 2018, pp. 107-122. Springer International Publishing, 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In The IEEE International Conference on Computer Vision (ICCV), 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Sambhav R Jain, Albert Gural, Michael Wu, and Chris Dick. Trained quantization thresholds for ac- curate and efficient fixed-point inference of deep neural networks. arXiy preprint 1rXiv:1903.0800 2019.
10

---

Under review as a conference paper at ICLR 2021
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. In International Conference on Learning Representations, 2017. URL htttp:://ppeereeeeee
Net//orrm?id=rkeee
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 13350-355, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
Fengfu Li, Bo Zhang, and Bin Liu, Temary weight networks. NIPS Workshop on EMDNN, 2016.
URL https::/arxiv. http:///wwwa....oo///////...... 04711, pdf
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http://ss///peeeeveeee.eeeeee/oe
Bkg6RiCqY7
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang. Autoq: Automated kernel-wise neural network quantization. In International Conference on Learning Representations, 2020. URL
https://cpenreview net io
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018. URL https
net /pdf //openreview.ner
d= Y8hhgobb
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019.
Chris Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous
relaxation of discrete random variables. In International Conference on Learning Representations, 2017. URL /tt.apts////operreeee..e////rrea wttp://wwww...eeeeooooooo.e.eeee
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, seshhperform deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 525-542, Cham, 2016. Springer International Publishing.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local repa- rameterization trick. In International Conference on Learning Representations, 2018. URL
net
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
11

---

Under review as a conference paper at ICLR 2021
Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is good parametrization. In International Conference on Learning Representations, 2020. URL
htttps::/gpenreview..u net www.:oorouuiiioo
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8612-8620, 2019.
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Blended coarse gradient descent for full quantization of deep neural networks. arXiv preprint 1808.0880000544 2018.
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference on Computer Vision (ECCV), 2018.
Xiandong Zhao, Ying Wang, Xuyi Cai, Cheng Liu, and Lei Zhang. Linear symmetric quantization of neural networks for low low-precision integer hardware. In International Conference on Learning Representutions, 2020. URL / htwp:/ewwwwwweeeeeeeeooeeeeeeeee
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/ 0611/066666666666 60, 2016. URL http://://aaxxi org. abs /1../6666.000 11tpe/166666.6.00
12

---

Under review as a conference paper at ICLR 2021
A ALGORITHM OF SEMI-RELAXED QUANTIZATION WITH DROPBITS
Algorithm Semi-Relaxed Quantization (SRQ) + DropBits
I: Input: Training data
2: Initialize: Bit-width b, network parameters (Wi, bi layer-wise grid interval parameters and the
standard deviations of a logistic distribution in the 1-th layer {an, oth Initialize layer-wise grid G
arl-20-1, ,... 20-1 1----1 (0t.0 "1.1,"'' -91,2-11 : for / f
3: procedure TRAINING
1. ,L do
5: 4- Fach entry of Wi or by
J Gi a/2
Shift the grid by -a//2 Compute CDFs
7:
F Sigmoid 4-=
8:
T. = Fli +
F[] for 0,...."
9:
if DropBits then
10:
Sample mask Ze for ls = 0,..., b--1 from
11:
== ( 7
12:
/ Σド= 7
Figure
13:
else
14:
Ti / Σラ= T
15:
end
16:
y one hot (argmax r3]
Multi-class Straight-Throur Estimator Quantization
17:
F = y C C:
18:
Wi + Each entry of W; quantized to T
19:
by Each entry of by quantized to 3
20:
Forward pass with quantized W; and
21:
Activation can be quantized in the way, with DropBits being False
22:
end for
23:
end procedure
24:
25: procedure DEPLOYMENT
26:
for | 1,. L do
27:
W: = min(maax(ar (o.nddWW//))..90))).......
28:
b min(max(a; Round(b:/an (a)))./..))))) -1
29:
end for 30: end procedure
13

---

Under review as a conference paper at ICLR 2021
B COMPARISON OF BIAS BETWEEN RQ, SRQ, AND SRQ + DROPBITS
general, when the loss involves discrete random variables, the true gradient, gradient of the expectation the loss with respect the parameter of discrete random variables, can be obtained by using existing stochastic gradient estimation techniques such as the score function estimator (see Equation (4) and (5) Maddison al. (2077). However, the reason why did explicitly compare the bias of gradient is that distribution parameters (i.e. function of æ in Given that an inverse of function л does not exist because is not one-to-one with respect to 2 each i, not possible to directly apply existing techniques such score function to compute unbiased estimators. Instead, we compute the bias of gradient with respect to as proxy of that of with respect to n.
{ { in our setting are independent of network parameters In fact, is
100
100
SRQ
DropBits
RQ
6 x 10
6 x 10
Bias
4 x 10-1
VM W M
Bias
4 x 10
10-1
3 10
2 x 10
2 x 10-1
SRQ + DropBits SRQ
3a -2
2a
3a
-3 -2a
0
2a
3a
The location of x
The location of x
(a) SRQ SRQ DropBits
SRQ DropBits RQ
Figure 8: The comparison bias (a) between SRQ and SRQ + DropBits, and (b) between SRQ + DropBits and RQ, when training LeNet-5 on MNIST in 3-bit. In this experiment, only weights are quantized. The a-axis denotes the value a weight in the last fully connected layer.
Although the bias of SRQ DropBits is smaller than that of SRQ displayed in Figure 8 (a), SRQ DropBits RQ exhibit comparable levels bias respect to as shown Figure 3 (b).
14

---

Under review as a conference paper at ICLR 2021
C EXTENSIVE COMPARISON FOR RESNET-18 AND MOBILENETV2 ON IMAGENET
Our method, SRQ DropBits surpasses quantization methods remaining the first or layer the full precision well the latest algorithms that quantize both the weights and activations of all layers including the first and last layers, except QIL (Jung et al. 2019)) and LSQ (Esser et al. 2020) both which utilize the full- cull-precisio first and last layer well as employ their own ResNet-18 18 pretrained model performing much higher than available from the official PyTorch repository.
Table Top- 5/Tp5/ error (%) with ResNet-18 and MobileNetV2 on ImageNet using 4-bit. denotes the use of the full-pree pell-preciion first or last layer, and $ indicates our own implementation with all layers quantized by using pretrained models available from the official PyTorch repository.
ResNet- 18
MobileNetV2
Methods
Top-1
Top-5
Top-1
Top-5
Full-precision
30.24
10.92
28.12
9.71
DoReFa
Zhou et al.
2016)
31.9
BOGD
Yin al.
2018)
32.64
12.24
LQ-Nets
Zhang et al. 2018
30.7
11.2
PACT1
Choi el
2018
Wang et al.
2019
30.8
38.56
17.30
RQ
(Louizos et al.
2019
38.48
16.01
RQ ST
Louizos
al 2019
37.54
15.22
DSQ
Gong et al.
2019)
30.44
35.20
QIL
Jung et al.
2019
31.05
11.23
32.77
12.51
QIL
Jung et al.
2019
29.90
TQT
Jain et al.
2019
Uhlich et al.
2020
30.49
32.21
LLSQF
(Zhao et al
2020
30.60
11.28
32.63
12.01
LSQ
Esser et al.
2020
28.90
10.0
SRQ DropBits (Ours)
30.37
10.96
30.83
11.26
15

---

Under review as a conference paper at ICLR 2021
D MORE EXPERIMENTS ON HETEROGENEOUS QUANTIZATION
As we can see 5 our heterogeneous quantization method capable of finding quantized sub-networks broad range of regularization parameter À.
Table 5: Test emror (%) for quantized sub-networks using LeNet-S on MNIST. VGG-7 on CIFAR- a0d
ResNet-18 ImageNet. Here, underline means the learned bit-width and "T" stands for ternary precision.
Initial
Trained
Test
Test
Test
Model
Bits
W. Bits
Error (Fixed)
Error
Error
W/A
per layer
(Reg.)
4/4
0.53
4/4/3/4
0.55
0.52
4/3/3/4
0.59
0.58
LeNet-5
3/3
0.58
3/2/3/3
0.65
0.55
3/T/3/2
0.85
0.60
2/2
0.63
2/2/2/T
0.68
0.59
T/2/2/T
0.70
0.64
1/4
6.77
4/4/4/4/4/3/3/4
6.74
6.65
4/3/4/4/4/3/3//
6.87
6.80
VGG-7
3/3
6.82
3/3/3/3/3/2/3/3
6.81
6.77
3/2/2/2/2/T/2/3
7.13
7.04
2/2
7.49
2/2/2/2/2/2/2/T
7.43
7.36
T/1/1T//1/T/2//
9.62
7.55
4/4
33.20
4/3/3/3
34.58
34.30
ResNet-18
3/3 3
36.46
34.94
3/3
37,80
3/3/2/3 2
41.01
40.30
//3/2/2 3/ /2/
43.41
42.13
16

---

Under review as a conference paper at ICLR 2021
E COMPARISON OF SRQ + DROPBITS WITH GUMBEL-SOFTMAX + MULTI-CLASS STE
As described Section our SRQ DropBits shows smaller quantization error, variance of gradients, and distribution bias than RQ, while maintaining stochasticity. For this reason, we employ the deterministic scheme the first place and then encourage stochasticity via DropBits.
To show the effectiveness of SRQ DropBits further, empirically compare it with algorithm using the Humbel-Softman STE the forward pass instead DropBits and our multi- STE the backwared pass. Let such algorithm be called "Gumbel-Softmax multi-class STE".
Table 6: Test error (%) for LeNet-5 on MNIST and VGG-7 on CIFAR-10.
Bits
Gumbel-Softmax
SRQ + DropBits
Dataset
W./..
multi-class STE
4/4
0.57
0.53
MNIST
3/3
0.65
0.58
2/2
0.74
0.63
4/4
8.25
6.85
CIFAR-10
3/3
8.44
6.94
2/2
10.41
7.51
Although employing our multi-class STE in the backward pass, Gumbel-Softmax + multi-class STE performs worse than SRQ DropBits, This is primarily due to the fact that Gumbol-Softmax STE still incurs a large quantization error like RQ.
17

---

Under review as a conference paper at ICLR 2021
F IMPLEMENTATION DETAILS
The weights and activations of all layers including the first and last layers (denoted by and A) are assumed to be perturbed as W W + and Á = A + respectively, under L(0, o) we describe in Section 2.
Concerning DropBits regularization in 3.3 we initialize the probability each binary mask with II (0.9, 0.01 3) (i.c. corresponding to low dropout probability). The conerete distribution of binary mask is stretched to ( <= 1.1 and 7 = 0.1 as recommended in Louizos et al. (2018). and r' is initialized to 0.2 to make discretized.
binary mask
For MNIST experiments, we train LeNet-5 with 32C5 MP2 64C5 MP2 512FC Softmax architecture for 100 epochs irrespective of the bit-width. In addition, learning rate is to Se-4 regardless of the bit-width and exponentially decayed with decay factor 0.8 for the last 50 epochs. The input is normalized into [-1, 1] range without any data augmentation.
For CIFAR-10 10 experiments, following the convention that the location of max- max-pooling layer is changed, which originates from Rastegari et al. (2016), a max-pooling layer is located after convolutional layer, hut before batch normalization and an activation function. We train VGG-7 with 2x(128C3)- MP2 2x(256C3) MP2 2x(512C3) MP2 1024FC Softmax architecture for 300 epochs, and a learnnng rate initially set to le-4 regardless of the bit-width. The learning rate is multiplied by 0.1 at 50% of the total epochs and decay exponentially with the decay factor 0.9 during the last 50 epochs. The input images are preprocessed by substracting its mean and dividing by its standard deviation. The training set is augmented as follows: (i) random 32 x 32 crop is sampled from padded image with pixels on each side, (ii) images are randomly flipped horizontally. The test set evaluated without any padding cropping. Note that a batch normalization layer is put after every convolutional layer in VGG-7, but not in LeNet-5.
In Section 4.1 RQ with annealing schedule of the temperature in RQ is implemented by following Jang : r is annealed every 1000 iterations by the schedule 7 = max exp ( (-t/100000)) in 3-bit and max(0.5, 2 exp (-t/1000) (-t/10000)) in 1-bit in order make the decreasing rate small as possible. Here, t is the global training iteration.
et al. (017))
For ImageNet experiments in Section 4.2 the weight parameters of both ResNet-18 and MobileNetv2 initialized with the pre-trained full precision model available from the official Py repository. When quantizing ResNet- 18 3-bit, fine-tuning is implemented for 80 epochs with hatch size of 256: learning rate is initialized to 2c-5 and divided by two at 50, 60, and 68 epochs. When quantizing ResNet-18 to 4-bit, fine-tune tin-ttuingg& carried out 150 epochs with a batch size 128: for the first 125 epochs. learning rate set to 5e-6, but le-6 for the last 25 epochs. When quantizing MobileNetV2 to 3-bit and 4-bit, fine-tuning is performed for 25 epochs with a batch size 48 and an initial learning rate of 2e-5: the learning rate divided by two at 15 and 20 epochs for 3-bit and at 10, 12, 18, and 20 epochs for 4-bit. We employ AdamW in Decoupled Weight Decay Regularization (Loshchilov & Hutter 2019) with a weight decay factor of 0.01.
In Section 4.3 and D if the probability of binary mask is less than 0.5, then drop the corresponding bits. For 1eNet-5 on MNIST and VGG-7 on CIFAR-10, our regularization term Section 3.5 is activated only for the first 50% of the total epochs. With the remained bit-width for each layer, fine- tuning process is conducted for the last 50% of the total epochs. For ResNet-18 on ImageNet, we initialize the weights of ResNet- 18 with the pre-trained full precision model and train it for ten epochs for simplicity. During training, our regularization term in Section 3.5 is activated only for the first 9 epochs, and fine-tuning process is done for the last epoch with the remained bit-width of each layer fixed. All experiments in Tandee]andd5 were conducted by the of Adam W: the weight decay value is set to 0.01 for LcNet-5, 0.02 for VGG-7, and 0.01 for ResNet-18. 18. We consider the regularization parameter À E [5 x 10-5 10 2 to encourage layer-wise heterogeneity.
18

---

Under review as a conference paper at ICLR 2021
G ALGORITHM OF RQ
We provide the algorithmic details of RQ follows. For quantizing weights, & [o- al-2"~1, ,,..,,,- .,,,, - 1]; however, when quantizing activations, the grid points start from zero since the
outputs of ReLU activations are always non-negative, that is, 9 = [9:10= ato.... 2" 1]. The objective function RQ cross-entropy loss function of class labels class probabilities predicted by quantized weights, biases, and activations, which is the same the loss function L in method.
Algorithm 2 Relaxed Quantization (RQ) Louizos
Algorithm Relaxed Quantization (RQ) Louizos et (019) for inference
al. 2019) for training
Input:
weight an activation)
1: Input: (a weight or activation)
2: Initialize: scale a, standard deviation o,
Require: scale a, grids G [9u. 92 1
grids G [g.]="0 L90. .... 922--1
3: Require: bit-width temperature
3: ã round
4: T 90
2 92 - +
4: Output: min(max(E (onax(E, 92)
2
2.
I
5: F = Sigmoid
> Compute CDF
6: Fli+1 F(i] for io = 0, ... 2 -
7: > Unnormalized Prob. for each grid point
8: Sampling from the concrete distribution
9: 04 Gumbel 1) for 0,..., 2b exp((1ogr गं (4)/T)
10: r;
E, eepp((ogg)) + uz)/T)
Output: 3 Ei-0' rigi
19


Reviews:
[{"review_id": "uIy2QxyAxif", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel network quantization method to reduce the bit-lengths of the network weights\nand activations, which is one of the most important problem for resource-limited devices.\nThe presentation of this paper is well written and organized.\nThe problem definition is clear, i.e., relaxed quantization with the Gumbel-Softmax relaxation suffers from bias-variance trade-off depending on the temperature parameter of Gumbel-Softmax.\nThe proposed method to solve this problem is reasonable.\nThe experimental results are also convincing.\n\nMinor comments:\n1. In Table 3, \"MNIST\" should be LeNet-5.\n2. Section 3.4 is a little dense.\n3. I understand that there is little space, but embedding formulas in sentences is difficult  to read.\n4. I recommend you to show the result only using SQR for  ImageNet  in Table 2 as in MNIST and CIFER10 in Table 1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "m8ZrJJS6Bm-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with network quantization. It proposes Semi-Relaxed Quantization (SRQ) that uses a multi-class straight-through estimator to effectively reduce the bias and variance, along with a new regularization technique, DropBits that replaces dropout regularization to randomly drop the bits.  Extensive experiments are conducted to validate our method on various benchmark datasets and network architectures.\n\nPros:\n- Reducing variance of gradient estimator in Gumbel of RQ using multi-class STE.\n- A novel dropbits technique to reduce bias in gradient estimator of SRQ, as well as supporting the mixed-precision scheme.\n- A \"quantized winning tickets\" is introduced to train probs of binary masks so that learning proper bitwise for each layer. \n\nCons:\n- The SRQ process seems not to be novel enough. Actually here pi is just a multi-class sigmoid-output, and y is derived directly by argmax(prob_pi) in forward process (and calculate gradients using STE.). This\nis quite similar to existing quantization methods using softmax + STE. [1]\n- Only 3/4 bit results of Res18/MobileNetV2 showed in ImageNet. I'd appreciate it if authors could offer more quantitive analysis on more architectures and tasks.\n- More comparisons on latency/energy/flops during the training and evaluation process should be provided to validate the SRQ + DropBits.\n\n[1] Hardware-aware Softmax Approximation for Deep Neural Networks. Xue Geng et al.\n\n***After rebuttal and discussion\n\n This paper proposes a new network quantization framework. In particular, the proposed DropBits is somewhat novel. However, it lacks sufficient and accurate analysis of SRQ+DropBits.  For example,  why SRQ can reduce quantization error has not been well motivated and explained.  The definition of distribution bias is still unclear.  I think that an accurate description of terminology is crucial and required for scientific research. Hence, the paper still needs minor polishing for publishing. I would like to decrease my rating to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1DFqPsD2riD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis work presents 1) Semi-Relaxed Quantization (SRQ), a method that targets learning low-bit neural networks, 2) DropBits, a method that performs dropout-like regularization on the bit width of the quantizers with an option to also automatically optimise the bit-width per layer according to the data, and 3) quantised lottery ticket hypothesis. SRQ is an extension of Relaxed Quantization (RQ), which is prior work, in two ways; firstly the authors replace the sampling from the concrete relaxation during training to deterministically selecting the mode (which is non-differentiable) and, secondly, they propose a specific straight-through gradient estimator (STE) than only propagates the gradient backwards for the elements that were selected in the forward pass. DropBits is motivated from the perspective of reducing the bias of the STE gradient estimator by randomly dropping grid points associated with a specific bit-width and then renormalising the SRQ distribution over the grid. This essentially induces stochasticity in the sampling distribution for the quantised value (which was removed before by selecting the mode in SRQ). The authors further extend DropBits in a way that allows for learning the drop probabilities for each bit-width, thus allowing for learning mixed-precision networks. Finally the authors postulate the quantised lottery ticket hypothesis, which refers to that “one can find the learned bit-width network which can perform better than the network with the same but fixed bit-widths from scratch”.\n\n### Pros\n- This work provides a set of additions that improves upon prior work\n- The DropBits method is novel and allows for learning the bit-width in a straightforward manner\n- The results improve upon recent works that learn quantised neural networks\n\n### Cons\n- Some claims from the authors are misleading while others are not precise\n- The computational complexity of the method is not discussed \n- Some experimental settings might not be consistent with some of the baselines\n\n### Detailed feedback\nThis work tackles the problem of learning quantized neural networks and the authors show empirically that their proposed method achieves good results. The DropBits extension is particularly interesting in that it allows for learning the appropriate bit-width of a given tensor via traditional pruning approaches. I also like the fact that the authors explain illustratively their proposed approach via several figures, which provide a nice boost to clarity. Nevertheless, I believe that there are still some important aspects that need to be addressed before I recommend acceptance for this work.\n\nFirst of all, the comparison against the prior work on figure 1 is misleading; the authors compare the *entire categorical distribution* (i.e., the one obtained after discretising the logistic onto the quantization grid) of SRQ at Fig 1(b) with a *single sample* from the concrete relaxation of the same distribution at Fig. 1(a) right for RQ. In fact, the underlying categorical distribution will be the same for both SRQ and RQ in the specific example of figure 1. Furthermore, it is worthwhile to notice that the underlying categorical distribution (i.e., pre relaxation) does have support for the value of -a (as the p(g_i = -a) is nonzero at Fig.1 (b)), thus it is not unreasonable that there are specific samples which lead to the quantised value being -a, thus incurring larger quantization loss.\n\nFurthermore, I believe that the discussion about SRQ misses some important points that would improve the clarity of the work if they are addressed. Selecting the most probable point in the categorical distribution for the forward pass is equivalent to rounding to the nearest grid point, which can be done much more efficiently than computing the entire categorical over the grid and then taking the argmax. In addition, this is also the same as taking sigma -> 0 for the logistic distribution that is to be discretized in the forward pass. Finally the authors argue that their novel multi-class STE reduces the variance of the gradient estimator but no formal justification is given apart from some hand-wavy arguments. Why is the variance lower if the gradient only flows through to r_imax? Furthermore, for the second point with respect to the benefits of the multi-class STE; while it does seem desirable that it aggressively clusters the weights and activations around the grid-points, I wonder how much can that hinder convergence. Do you ever observe that the weights can be prematurely “stuck” (and thus lead to a bad local minimum) and do the weights ever move further away than just the closest grid point? DropBits could potentially help with the latter part, but it would be interesting to see what happens without it.\n\nDropBits in my opinion is the main novel idea of this work, and it is an interesting way to learn the bit-precision of each tensor in the network. I have two main points for this section in general and DropBits in particular. The main motivation behind DropBits seems to be converting the sampling distribution of SRQ (which is deterministic) to a stochastic one. If this is the case, then why have it be deterministic in the first place? You could just sample from the original categorical distribution (by, e.g., using the Gumbel-Softmax STE which gives samples exactly on the grid) in the forward pass and use your multi-class STE approximation in the backward pass.  It would be interesting to see how that fares with SRQ + DropBits and would highlight whether the main benefit of DropBits was the regularization aspect (and not that of improving the sampling distribution). As for DropBits in particular; it seems that you drop bits independently with each of the gates z_1, z_2, … (i.e., figure 3). If this is the case, then you could end up with a non-uniform grid that cannot be exactly represented as a fixed point tensor (e.g., on figure 3b you could have z_1 = 0 and z_2 = 1). If this is the case, then comparison against other approaches that use uniform quantization is not apples-to-apples.\n\nFinally, a couple of other things that I believe should be addressed; the authors don’t make any discussions about the computational and memory complexity of the resulting algorithm. It seems that for every individual weight and activation they first have to construct the categorical distribution over the entire grid (which can quickly become very large, e.g., for 8 bits there are 256 categories), in order to take the weighted sum. This doesn’t seem to scale very well. How expensive is something like this in practice and how long do experiments take on, e.g., Imagenet? Furthermore, the quantised lottery ticket hypothesis (QLTH) is a bit peculiar. The original lottery ticket hypothesis (LTH) was about finding sparse networks at initialisation that can be trained from scratch and achieve the same accuracy as the original dense equivalents. This is different than what the authors articulate here, specifically that it is about finding sparse networks that are easier to train compared to sparse networks obtained from pruning. As a result, their QLTH seems to state the opposite than what the original LTH was about; it states that a QLT is obtained when you manage to find a network X that a.) has smaller bit width than the original network and b.) has better performance than a network initialised to the bit-width of X and trained to convergence. Following the arguments of the LTH, I would expect a QLT to be obtained when you can quantise a neural network to a specific bit-width at initialisation and when you train from scratch that particular quantised network, you obtain the same performance as the full precision equivalent. I would thus encourage the authors to clarify this point and better align with the original LTH. \n\nBased on the aforementioned points, I cannot at the moment recommend acceptance for this work. Nevertheless, as I believe DropBits is an interesting idea, I would encourage the authors to put in the effort and rework the paper by addressing these points over the rebuttal.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Summary
The paper was initially well received by reviewers, remarking the new gradient estimator, a new dropbits technique and an interesting observations of better performance when the bitwidth is learned. The experimental results also look promising: showing improved training performance and test performance (including on ImageNet with ResNet-18), properties to reduce quantization error of learned weights, possibility to learn number of bits via learning stochastic bit-dropping masks.

A deeper verification of the specific methods proposed however showed principal issues:
- The methods proposed in the paper are not sufficiently justified by verifiable formal arguments. The proposed intuitive explanations are entangled and actually lead to wrong conclusions. In particular a main claim of the paper that the proposed estimator reduces bias and variance of Gumbel-Softmax estimator was shown wrong and was removed in the revision. The remaining claim that the estimator reduces quantization error is also wrong (see below). With these issues, the gradient part of the paper is largely incorrect, which is in a strong discrepancy with good experimental results.

- Other parts of the paper, comprising the remaining technical contributions are not properly positioned with respect to the SOTA and thus are not necessary novel / improving.

The main technical issues were discussed with all reviewers and were either supported or not objected. Therefore, I am confident that the submission has critical problems and must be rejected. I recommend the authors to thoroughly investigate all the raised issues (by all reviewers) before resubmitting to other venues. 

# Details

## Gradient

The overclaim of reducing bias and variance / resolving bias/variance tradeoff has been removed in the revision, but the new gradient estimator remains a central innovation proposed. It is however not justified and cannot indeed be regarded as a good estimator:

* The justification argues about the bias of the Gumbel-Softmax sampling distribution, but the proposed estimator does not use a sampling distribution in the forward pass, and thus by design cannot address this problem.

* The backward pass to use gradient in i_max only (Eq. 3) is not based on any justification at all. 

* The remaining claimed good property: "to reduce the quantization error" is, according to the definition in sect. 3.4, not a property of a gradient estimator, but of the stochastic relaxation alone. There is an experimental evidence Fig.2 that the estimator _leads_ to lowering the quantization error. This is however in a contradiction with a direct verification of the proposed estimator that was conducted:
The verification inspects gradient in a single variable $x$ and a linear loss function of the quantized variable $\hat x$.  It shows that the gradient is zero at grid points and discontinuously reverses the direction at half-grid points. Because of such zigzagging, *it does not correspond to minimizing the loss function*, i.e. not a reasonable estimator. The grid points, where the gradient vanishes, may correspond to either local minima or to local maxima of the estimator. Which of the two cases occurs depends exclusively on the sign of the incoming gradient from the loss function. For $L(\hat x) = \hat x$ we observe that the negative gradient points towards nearest grid point, but for $L(\hat x) = -\hat x$ it points away from the nearest grid point, i.e. a step would *increase the quantization error*. The implementation of this verification is attached anonymously:
https://colab.research.google.com/drive/1PibzRMXQ-NVZMUdfgTIK0Q5FxUKyxfqI?usp=sharing

* Alternative existing estimators are not sufficiently discussed: e.g. common deterministic STE, as used in quantization papers: to just treat the quantization operation as identity on the backward pass. Estimator used by Shayer et al. (2018),  Ajanthan et al. 2019 “Mirror descent view for neural network quantization”, Unbiased estimators (e.g. Yin et al. 2019 “ARSM: Augment-REINFORCE-Swap-Merge Estimator for Gradient Backpropagation Through Categorical Variables”). While unbiased estimators may still have too high variance and or be too computationally demanding for deep networks, they can be used for verification purposes. 

* The claim that it is not possible to apply unbiased estimators, in particular score function estimator, because of dependency on x is incorrect. See e.g. Schulman et al. 2015 “Gradient Estimation Using Stochastic Computation Graphs”. Many works on advanced unbiased estimators also demonstrate experiments with 2 or more layers of hidden discrete stochastic variables. From this and technical discussion with authors, it is seen that the experimental study is Sec 3.4 is very limited and erroneous. 

* The rule by which the probability mass of the dropped bits is uniformly spread over the remaining bits is not justified and appears methodologically incorrect. In Fig.4 it is not clear what bits were dropped and why the mass at $-2\alpha$ has decreased.

## Gradient and Other Techniques relative to SOTA

* The bias problem of GS estimator, detailed in Fig.1. is not novel to me, it is in fact known that the mean under the concrete distribution (of linear or non-linear objective) differs from the mean under the categorical distribution, see e.g.

Lorberbom et al. (2018) Direct Optimization through argmax for Discrete Variational Auto-Encoder (Fig.1)

Andriyash et al. (2018) Improved Gradient-Based Optimization Over Discrete Distributions

Thus analysis of individual samples in Fig.1 appears unnecessary detailed. The issue that the relaxed distribution of Gumbel-Softmax may cause a large estimation error for gradients downstream is already discussed by Louizos et al. (2018) and other works, e.g. 

Choi 2017, "Unsupervised Learning of Task-Specific Tree Structures with Tree-LSTMs" Sec 3.2

and Andriyash (2018). This later problem was previously addressed in many cases by the ST Gumbel-Softmax heuristic. This heuristic indeed performs better in CIFAR-10 experiments in the submission / Louizos (2018), which is likely to be a better tuned and more controlled experiment than ImageNet.
* More methods should be discussed that reduce the quantization error during learning. E.g.

Cong et al. (2018): “Extremely low bit neural network: Squeeze the last bit out with ADMM”, 

who include terms explicitly minimizing the 
quantization error. In fact most works quantizing network weights primarily focus on reducing the quantization error, e.g.

Nagel et al. (2019)" Data-Free Quantization Through Weight Equalization and Bias Correction

* The prior works on learning bit width should be more extensively discussed / compared to, especially if this part becomes central to the submission. E.g.

Baalen et al. (2020) “Bayesian Bits: Unifying Quantization and Pruning” (or references therein if this is considered contemporaneous).  

Courbariaux & David (2015): Training deep neural networks with low precision multiplications

* The new hypothesis for quantization is in fact similar to the effect observed elsewhere that quantizing neural networks progressively leads to better results.  E.g. 

Zhou et al. (2017) Incremental Network Quantization- Towards Lossless CNNs with Low-Precision Weights.

It is questionable whether the link to the lottery ticket hypothesis is justified, since the latter shows quite the opposite, as reviewers have pointed.
Decision: Reject