Fighting Quantization Bias With Bias
Alexander Finkelstein∗

Uri Almog∗
Hailo Technologies

Mark Grobman

arXiv:1906.03193v1 [cs.LG] 7 Jun 2019

alex.finkelstein,uri.almog,mark@hailotech.com

Abstract

environment. In this paper we follow the 8-bit integer quantization scheme used in [12] which is widely employed
across both cloud- and edge-devices due to its easy andefficient implementation on hardware. For most networkarchitectures, 8-bit quantization carries a minimal degradation penalty [7] leading recent research to double-down on
more aggressive schemes, using 4 bits or less for activations, weights or both [14, 1, 4, 13]. Nonetheless, some architectures, such as Mobilenet [10, 20], Inception-V1 [23]and Densenet [11] still exhibit significant degradation whenquantized to 8 bits. Mobilenets are of particular interest as
they were designed specifically for embedded applications
and as such are often deployed in their quantized form. In
order to successfully deploy these networks a quantizationaware training phase [22, 7, 12] is usually introduced toregain the network accuracy. This phase is costly in timeand requires access to the original dataset on which the network was trained. Moreover, while effective, the training
offers little insight into why there was significant degradation in the first place. Recently, there has been more focus
on methods that do not rely on an additional training phase[15, 1, 27, 16] but an exhaustive 8-bit quantization schemethat excludes training remains an open challenge.
In this paper, we begin by showing that some networks
exhibit a significant shift in the mean activation value following quantization, which we henceforth denote as MAS(mean activation shift). We note that in order for such a shift
to have an impact on the performance of the network it must
be of significant magnitude with respect to the mean activation itself. The shift is the result of quantization rounding
errors being highly unbalanced - a ’small numbers effect’that is statistically implausible for layers with many parameters, but can become the main error source for layers witha small amount of parameters (e.g. depthwise convolution
with 9 parameters per channel). Note that the frugal use ofparameters is especially prevalent in network architectures
aimed at embedded applications [10, 20, 26] making themmore susceptible to quantization induced shifts. We thenshow that the shift introduced by the quantization can becanceled by using the bias parameter of the channels. Since
the bias term is additive, any constant added to it will shift

Low-precision representation of deep neural networks(DNNs) is critical for efficient deployment of deep learning application on embedded platforms, however, converting the network to low precision degrades its performance.
Crucially, networks that are designed for embedded applications usually suffer from increased degradation since they
have less redundancy. This is most evident for the ubiquitous MobileNet architecture [10, 20] which requires acostly quantization-aware training cycle to achieve acceptable performance when quantized to 8-bits. In this paper,
we trace the source of the degradation in MobileNets to ashift in the mean activation value. This shift is caused by aninherent bias in the quantization process which builds up
across layers, shifting all network statistics away from thelearned distribution. We show that this phenomenon happens in other architectures as well. We propose a simple
remedy - compensating for the quantization induced shift byadding a constant to the additive bias term of each channel.
We develop two simple methods for estimating the correction constants - one using iterative evaluation of the quantized network and one where the constants are set using ashort training phase. Both methods are fast and require onlya small amount of unlabeled data, making them appealing
for rapid deployment of neural networks. Using the above
methods we are able to match the performance of trainingbased quantization of MobileNets at a fraction of the cost.

1. Introduction
In the last years, an increasing amount of effort is invested into executing DNN inference in low-precision arithmetic. While very effective in cutting down on memory,compute and power usage, this comes at a price of degradednetwork performance caused by weight- and activationrounding errors. Quantization, the conversion of a net
to its low-precision version, is performed according to a”scheme”, a conceptual model of the hardware execution∗ Equal contribution

1

the activation distribution of the channel. Once the task isdefined as fixing the shifts, the problem then becomes how
to estimate them. To this end, we develop two procedures one based on direct estimation over a set of test images andthe other performing a fine-tuning phase involving only thebias terms of the network. The main contributions of thispaper are:• Mean activation shift (MAS): We explore both thestatistical origins and the impact of MAS on layerlevel quantization errors. We show that MAS can arise
and be responsible for a large component of the error
when quantizing network architectures relying on layers with a very small number of parameters (e.g. MobileNet’s depthwise layers).
• Shift compensation using bias terms: We show MAS
can be compensated by using the bias terms of the
network layers, and this alone can drastically reduce
degradation, further establishing the previous claim.
We propose two algorithms to that effect - Iterative
Bias Correction (IBC) and Bias Fine Tune (BFT) - andwe analyze their performance on a variety of Imagenet
trained classifiers.
While our experiments use convolutional neural networks(CNNs) for classification, we expect both the techniques
and the analysis to extend into other tasks (e.g. ObjectDetection) building on these classifiers as their feature extractors.

2. Previous Work2.1. Quantization Procedures
A slew of ideas on DNN quantization were publishedin recent years, with widely and subtly diverging assumptions about the capabilities of the underlying deployment
hardware . For instance, support for non-uniform quantization [21, 2, 8], fine-grained mixed precision [18] or perchannel (”channelwise”) quantization [7, 1, 5, 6]. Leaving
the batch-normalization layer unfolded [14] is another nuance since, similarly to channelwise quantization, it enables
per-channel scaling at the cost of additional computations.
In this work, we restrict ourselves to the simplest setting
folded-batch normalization, layerwise, UINT8-activations,
INT8-weights that was introduced by [12]. The attractiveness of this simple setting is that it is supported efficiently
by most existing hardware on which deep-learning applications are deployed. The literature can also be dissectedby the cost and complexity of the quantization procedures,
falling into two main categories:
Post-training quantization: these methods work directly
on a pre-trained network without an additional training

phase to compensate for the effects of quantization. Generally lean, these methods have minimal requirements interms of data, expertise and effort (of both man and machine). Most works are focused on how to best estimate
the dynamic range of a layer given its activation distribution. These range from simple heuristics for outlier removal [7, 14], to more advanced methods employing statistical analysis [16, 19, 4, 1, 5, 6]. In this work we use
naive min/max statistics (without clipping) to determine
dynamic range in order to keep the comparative experiments on IBC/BFT as clean as possible, however, it can be
combined with more advanced methods. Other approaches
try to lower the dynamic range of the layer by reducing
the intra-layer channel range. In [27] it was proposed to
improve layerwise quantization by splitting channels withoutliers, improving quantization at the cost of additional
computations. Finally, in [15] it was proposed to employ
inversely-proportional factorization to equalize the dynamic
ranges of channels within a layer.
Quantization-aware training involves optimizing thequantized weights using a large training set, with netspecific hyperparameters [14] usually taking the pre-trained
net as a starting point. The optimization target may use theground-truth labels [12] as in the normal training, and/or
an adaptation of knowledge distillation [9] loss, namely using the full-precision net as the target (”Teacher”), with
the quantized (”Student”) net punished for distance of its
output to the Teacher’s [19, 17]. Rendering the training”quantization-aware” is non-trivial since the rounding andclipping operations are non-differentiable. The simplest andstandard approach is that of a Straight-Through-Estimator(STE) [3, 19] which essentially consists of using the quantized version of the net on forward-pass while using thefull-precision network on the backward-pass. One of themethods developed in this work (BFT) employs knowledgedistillation and a STE within a very short training procedure (micro-training) restricted to the network biases only.
Concurrently to our work, in [5] a micro-training of scaling factors (multiplicative rather than additive as in ours)
was proposed. Like us, they categorize this as post-training
quantization since very little data and training time is used.

2.2. MobileNets Quantization
Mobilenets [10, 20] are CNNs based on ”separable”
convolutions - intermittent depthwise and pointwise convolution layers. Their compact and low-redundancy design makes quantization challenging as they have less resilience to noise. The basic layerwise 8-bit quantization of
Mobilenet-v1 leads to complete loss of accuracy [22, 7]. A
major source of the degradation is due to issues caused by
the folding of batch-normalization to degenerate channels(i.e. channel with constant zero output) [22]. Zeroing out

these channels reduces degradation to 10% and we employ
this technique as well. Quantization-aware training methods [12, 7] are able to reduce degradation to 1% for both
Mobilenet-v1 and v2. Current post-training quantization
methods can achieve similarly good results (i.e. < 1%)
only by using channelwise quantization [7] which is notsupported on all hardware. Authors of [22] achieve 2.5%
degradation in a layerwise post-training quantization setting but resort to remodeling the network architecture into a”quantization-friendly” version by changing the activation
function and modifying the separable-convolution building
block. In [15] a 3% degradation is achieved without either
remodelling or retraining by using inversely-proportional
factorization to equalize the dynamic range of channels; thisfurther reduces to 0.6% when the BFT method described inthis paper is applied on top, setting the new state-of-the-art
for Mobilenet-v1 and v2 ”basic-gemmlowp” quantization
while being easy and fast to run.

(q)

Where el,ch , xl,ch , xl,ch are vectors (for brevity we sometimes drop the subscripts where they are understood fromcontext). Some works make a distinction between rounding ”noise” and clipping ”distortion”; in this work both aretreated indivisibly and we use the terms ”noise” and ”error”
interchangeably. For every channel we define the signal energy, E(x2 ), as:E(x2 ) =A general sentiment underlying quantization of neuralnetworks is that [24]:”...as long as the errors generally cancel eachother out, they’ll just appear as the kind of random noise that the network is trained to cope withand so not destroy the overall accuracy by introducing a bias...”Contrary to the above assumption, in this work we show thatwhen comparing the activations of a feature map in a fullprecision and quantized net (QNN), a significant ”DC” shift
between the distributions can be observed (Fig. 1). This
conceptual gap could arise from thinking on the level of asingle tensor, whose rounding error distribution is assumed
to be uniform and symmetric; then assuming this property
to propagate through the net. However, the law of large
numbers doesn’t always hold, and small asymmetries might
get amplified and accumulate across layers.
In the rest of this section, we first (3.1) define the quantization error and it’s ”DC” component (denoted as MAS).
Then in 3.2 we quantify in precise terms the MAS contribution to the error, empirically demonstrating its significance.
Finally, in 3.3 we explore further the statistical mechanismof MAS generation and its dependence on layer structure.

3.1. Quantization Error Measures
For a given layer l and feature ch we define the quantization error as the element-wise difference between activations in the original DNN (x) and those in the quantized
DNN (QNN) (x(q) ) :(q)el,ch = (xl,ch − xl,ch ).

(1)

(2)

The expectations E() here and below are to be understoodas taken across all pixels of the testing set. We use themean rather than sum (as in standard L2 norm definition)
so that our analysis will be invariant to the number of elements within a layer. Similarly, we define E(e2 ) as theenergy of the quantization error. Next, we define the Mean
Activation Shift (MAS) as:(q)

3. Problem Statement and Analysis

1 X 2
xl,ch
N

(q)

∆l,ch = E(xl,ch )−E(xl,ch ) = E(xl,ch −xl,ch ) = E(el,ch )(3)
For convenience of exposition we also define we also define the following two quantites: inverse root quantization
to noise ratio (rQNSR) and Mean activation Shift to Signal
Ratio (MSSR) as:s
E(e2 )(4)
rQN SRl,ch =E(x2 )E(el,ch )M SSRl,ch = q
E(x2l,ch )(5)

MSSR measures how significant the MAS is compared tothe average activation. The rQSNR is a measure of the overall noise level in a channel. Note that MSSR and rQNSR
scale similarly, allowing for easy comparison.
All definitions above being random variables’ expectations, in practice we use estimates generated by sampling
the activations of the network across inferences on a batchof images.

3.2. The Effect of Mean Activation Shifts
One view of MAS as defined in eq. (3) is as the activations’ distribution shift - between the original and the quantized net (see fig. 1). Another analysis avenue relates it to
the MSE decomposition:
M SE = E(error2 ) = M ean2 (error) + V ar(error)(6)
In this framework, the question of the MAS’s significancecan be formulated as gauging the relative contribution of
the error mean to error magnitude, which in our notation
E 2 (e)can be written as: E(e2l,ch) . To answer this question we esl,chtimate these quantities on data samples obtained by runningrection can be done relatively precisely. The natural mitigation taken together with the significance of the MAS contribution strikes a great cost-benefit balance, compared toe.g. quantization-aware training methods (and actually performing on-par with those in some cases as we shall see insection 5).
Note that for a deep networks the situation is more complex since MAS for different channels in the first layers are
mixed in deeper layers, causing input errors that may becomparable with (or larger than) the MAS of the deeper layers.
We will see how these issues are tackled by our optimization methods in section 4.
Figure 1: Mean activation shift in two different layers ofMobilenet 1 1.0 224: Left: weight rounding errors (vertical axis) vs. full precision (FP) weights (horizontal axis).
An ideal transformation would place all the dots on the nullhorizontal line. Black indicates a weight whose quantizedvalue is greater than the FP, and magenta indicates weights
whose quantized value is smaller or equal to the FP value.
Right: 32-image activation histograms for the channel corresponding to the weights on the left. Top row is for the
conv1/channel1 kernel slice, with 27 weight elements (1.a)
and a small (0.029) relative mean kernel shift, producing
a small MAS (1.b). Bottom row is for the are for depthwise1 layer (channel12), with 9 weight elements (1.c). The
relative mean kernel error is large, (0.239), resulting in alarge MAS. (1.d). The vertical blue and red lines mark themean activation values for FP and quantized networks, witha clear shift in 1.d and an indiscernible shift in 1.b

the full precision and the quantized nets over batches of few
tens of images each to obtain signal and error vectors; seefigs. 2, 3 (curves for different choices of batch are very
similar; we drop all but one from plot for visual brevity).
We can see that for many layers the contribution is significant, sometimes dominating the error energy. Fortunately,
the degradation-complicit ∆l,ch lends itself to a convenient
correction, one that doesn’t entail changing the NN’s computational graph but only adjusting its parameters - namely,
subtracting MAS from the (per-channel) biases Bl,ch to be
added pre-activation 1 . For a single linear (Fact = I) layer,
this bias-fixing operation can indeed be proven as the optimal error reduction by changing biases alone - effectively
removing the Mean contribution in eq. (6). Note also thatin contrast to the weights (and their potential adjustments),
the bias is quantized at high precision in most implementations (the cost for this being rather low), so the above cor1 Referring to a generic convolutional layer with kernel of size K:P
ch0 ,dx,dy
KXl,ch = Fact
+ Bl,chch0 ,dx,dy Xl−1,ch0 Wl,chFigure 2: 32-image
estimatesof (a) E(el,ch ) (the MAS)qq

and (b),(c) E(x2l,ch ), E(e2l,ch ) (L2 norms of activation signal, and its quantization error), per channel, for typical layers of two nets strongly differing in quantizationfriendliness. When quantization error is large, the meanshift tends to be larger even in relative terms, becoming the
major contribution to q-error.

3.3. MAS Generation Mechanism in MobileNets
We now wish to connect the number of weight elements
in a layer and the typical magnitude of MSSR in that layerand show that small kernels tend to increase MAS. Let us
consider layer l in which the calculation of an output channel activation involves k kernel elements and k input activations. We define a weight rounding error as δWi where idenotes the weight elements in the set of size k. The weightrounding errors are i.i.d and follow the distribution:f (δWi ) = U [−max(|W |) max(|W |)
,
]
2N −1
2N −1

(7)

The mean and standard-deviation of the sum of rounding
errors is given by:kXE( (δWi )) = 0(8)kernel values to this grid and so the expectation in the above
equations is taken with respect to possible kernel values.
The activation shift for an input image, defined in 1 canbe expressed as:
el,ch =

kX(xiin + δxi in ) × (wi + δwi ) −

i=1

k
X

xiin × wi (10)

i=1

or,
el,ch =kX

δxi in × wi +

i=1

k
X

xiin × δwi

(11)

i=1

In the first layer of the network, the first term in the righthand side is negligible since when the input rounding errors
are small such as in the first layer. We consider only thesecond term in the following analysis. Averaging over input
data set and the spatial dimensions of layer l to achieve thenumerator in 5, we get:E(el,ch ) = E(kX

xiin × δwi ) = E(xin ) ×

i=1

k
X

δwi (12)i=1

where we made the assumption that the input data xi are
i.i.d. The separation of terms in the mean is possible sincefor a given channel the δwi are constant and independent of
the data.
Let us rewrite M SSRl,ch as:

Figure 3: 32-image estimates of the MSSR, QNSR, andtheir ratio ((a)/(b), (c)/(b), (a)/(c) ratios in terms of fig. (2))
L2-aggregated (RMS) over channels, for all layers of same2 nets. Comparing top and center we can see again, now
at the whole net level, that for mobilenet the overall error isgreater and it’s driven in large part by the MAS. Some typical behaviors can be spotted - e.g. average-pooling layer
reduces QNSR but the mean-shift contribution increases.
Comparing center and bottom plots, the great error reduction following the BFT procedure is evident (IBC gives asimilar result); being driven by reducing MAS, it’s not surprising that the relative contribution of MAS (black line)
goes down as well

k
X
p
σ(
δWi ) = C (k/12)(9)

with C depending only on max(|W |) and N wheremax(|W |) is the maximum element magnitude in theweight kernel and N is the number of bits used for quantized representation. It is worth noting that the assumptionof a given max(|W |) means that quantization grid is constant. The rounding error is the results of decimating thekXE(xin )×δwi
M SSRl,ch = p
E(x2out ) i=1(13)

We regard the first term of the product as the ’data term’.
If we treat the convolution as a matched filter it can be approximated as:
E(x )
p in
≈ 1/k
(14)
E(x2out )Using the above approximation we express the mean andvariance of M SSRl,ch with regards to possible kernel values in layer l as:
E(M SSRl,ch ) = C1 × 1/k × E(kXδwi ) = 0

(15)

i=1

k
X
σ(M SSRl,ch ) = C2 1 × 1/k × σ(
δw i )

≈

√

i=1

√

(16)

k/k = 1/ k

Equation 16 tells us that layers with small kernels willtend to have larger MAS energy to activation energy ratio. This conclusion is in agreement with our observationthat Mobilenet architecture exhibits strong mean activation

shifts, since it incorporates many depthwise (dw) convolutions, involving small (9 elements) kernel slices. In contrast,
conventional 2D convolutions typically involve hundreds ofelements for the calculation of each output channel.
This is only one possible source of MAS and the occurrence of significant mean activation shift in Inception v1 despite the absence of depthwise layers suggests other possible sources for MAS. One such source can be the neglected
term in 11 which becomes significant once the input of thelayer exhibits large noise.

4. Optimization Methods
We now present two methods that compensate for the
QNN MAS (sec. 3) by updating the bias terms. While thefirst (IBC) is very fast and requires a very small dataset,the second (BFT) is more general and utilizes a stronger
optimizer.

4.1. Iterative Bias Correction (IBC)
Estimating the MAS of all the layers and channels andupdating the bias terms of the network in one pass cannot
be done, since correcting the bias terms of layer l changes
the input to layer l + 1, which, in turn, gives a different
MAS than the one calculated before. Therefore, the correction process has to be performed iteratively according toquantalgorithm 1, where actorig
are the activationl,ch and actl,chvalues of layer l, channel ch in the original and quantizednetworks, respectively. The inner ’for’ loop in the algorithm
can be vectorized and done on all of the channels of a layerl simultaneously. The set of input images used for the Iterative Bias Correction (short-named IBC-batch), has a typicalsize of 8-64 images, and can be drawn from the same set
used for the QNN calibration. We find that the Iterative Bias
Correction effectiveness can vary with IBC-batch size, andlarger batches do not necessarily produce better results. The
strength of IBC lies in its simplicity and high speed. Withas little as 8 unlabeled images (that can be just the quantization batch) and just a few minutes computation we wereable to achieve the results in table 1.

4.2. Bias Fine Tuning (BFT)
The observations discussed in section 3.2 and in fig. 2
suggest that modifications of the QNN’s biases, reducing
or otherwise modulating the MAS, have the potential to reduce noise energy, impact downstream QSNR and eventually the final layer’s output accuracy. This line of thoughtsuggests attempting a generic optimization of the biases
alone, with the eventual objective of reducing the overall
QNN’s loss and accuracy degradation. To this end, we use
a quantization-aware training procedure, which starts fromthe pre-trained weights and biases, but restricts the trainable variables to be the biases only. This restriction enables
fast training on a minuscule part of the training set, as small

Result: Corrected QNN checkpoint
evaluate actorigl,ch for all l, ch in original net;l ← 0;
while not end of original net do
evaluate actquantl,ch ;
for ch ∈ l do
quant∆l,ch ←< actorig>;l,ch > − < actl,ch
quant
quant
biasl,ch ← biasl,ch + ∆l,ch ;endl ← l + 1;
end
Algorithm 1: Iterative Bias Correction (IBC). The inputs
to the IBC algorithm are the original and quantized networks, a topologically sorted list of layers, where 0 is the
input layer and the last layer is the output layer. In each
iteration the output of the current layer for both the original and quantized networks is evaluated and averaged overn images the spatial dimensions to give a single value per
output channel ch. The MAS is computed and added to
the layer bias.

as 1K images, without significant overfit, since the number of trainable variables is several orders of magnitude lessthan with a full training. Our procedure uses global optmization in contrast to the local estimation procedure usedin IBC. Drawing inspiration from [17, 19], we use pureteacher-student distillation loss, namely the cross-entropy
between the logits of the full-precision and the quantizednet. This enables a label-free training. To make the training
quantization-aware, we use quantized weights for the trainable net, and add F akeQuant operations of the TensorFlow
framework on the activations’ path, implementing the standard Straight-Through Estimator (STE) approach [3, 25].
Since the weights are fixed for the procedure, our method bealso viewed as a miniature variant of Incremental Training[28, 29], in the sense that biases are optimized to compensate for the error generated by quantization of the weights.
Finalizing our procedure, the fine-tuned full precision biases are re-quantized to 16 bits which is precise enough forthis final step to have no accuracy impact.

5. Experiments
For our tests we used the following setup and procedure(all experiments were done with the TensorFlow framework):
1. Publicly available pre-trained full-precision models
from tf-slim collection is ingested; any BatchNorm ops
are folded onto preceding layer’s weights.
2. For mobilenets: dead channels (channels with constant
output / zero variance) are dropped.

Table 1: Top-1 accuracy of original nets and accuracy
degradation with basic 8-bit quantization, Iterative Bias
Correction (IBC), and Bias Fine Tune (BFT). All numbers
are in units of absolute percentage (%). Best IBC results forwere achieved with IBC batch size of 8.
Network
Name
Mobilenet-v1-1.0
Mobilenet-v2-1.0
Mobilenet-v2-1.4
Inception-v1
Resnet-50-v1

Original
Top-1
accuracy
71.02
71.8
74.95
69.76
75.2

Basic
Quant.

IBC

BFT

7.90
16.44
6.42
2.26
0.27

0.92
1.42
1.1
0.44
> 10

1.03
1.2
0.85
0.47
0.30

3. Weights are quantized onto a uniform, symmetric 8-bit(INT8), per-layer grid. Biases are quantized to 16-bit
representation.
4. Activations are quantized (at run-time) onto a fixed,
uniform, asymmetric 8-bit (UINT8), per-layer grid
over a range defined by simple min, max statistics on
a 64-image calibration batch.
5. QNN accuracy is evaluated on the standard 50K ImageNet validation set.
Table 1 summarizes the results of simple-case tests over
several neural nets. The second column gives the Top-1 accuracy of the original, floating-point representation net. The
third column gives the top-1 accuracy degradation using thebasic quantization described in [12]. The last 2 columns
give the top-1 accuracy degradation of the quantized nets
after applying the IBC and BFT optimization tools.

5.1. Experiments With IBC
Testing IBC with different sizes of IBC batches between
8 to 64 images shows a variance of a few 0.1% in results.
Out of the cases we tested, the best results were achieved
using an IBC batch of 8 images from the calibration batch.
While MobileNet and Inception architectures benefit from
employing IBC, Resnets show large degradation by approximately 10% when utilizing IBC. We were unable to understand the source of this degradation.
The IBC algorithm discussed throughout this paper compares post-activation means in the original and quantizednets. In Mobilenet v1 the activation function is Relu6, clipping the output at 0 and 6, which results in a loss of information regarding the convolution output distribution. It
would be reasonable to assume that updating the bias terms
according to pre-activation distribution would utilize thefull information available and produce even better results
than those shown in table 1. Indeed, testing a variant ofalgorithm 1 in which ∆l,ch is calculated pre-activation on

Mobilenet v1, yields a degradation of 0.76% - 0.16% better
than the post-activation degradation given in table 1).

5.2. Experiments With BFT
We trained using the standard Adam optimizer, witha learning rate schedule scanning a wide range as recommended in [14]: in our default schedule we use
10−3 , 10−4 , 10−5 , 10−6 rates for 16 mini-epochs each, fora total of 64 mini-epochs using the same 1K images. This
schedule is used across all nets presented here (see table 1), proving a high degree of robustness. That stands
in contrast to the normally high appetite of training-based
methods for expertise and data. Given the above, and especially the relaxed input requirement of only 1K labelless images, (similar in size to calibration sets typicallyused [16, 7]), we claim that the procedure should beseen as a post-training quantization method despite sharingthe ”backprop+STE+SGD” approach with of quantizationaware training methods.
The results (table 1) are quite similar to IBC (suggesting that both utilize well a common ”resource” discussed
above) - with the exception of ResNet which isn’t degradedwith BFT. On both Mobilenet-v1 and v2, we perform on
par with the state-of-the-art of 1% degradation reached withfull quantization-aware training [7]. The restriction to biases apparently enables harvesting the low-hanging fruitsof quantization-aware training - which happen to include
the lion’s share of the degradation to be reduced - using afraction of the cost. When combining ChannelEqualization [15] with BFT on the Mobilenet-v2 nets, we achieve
state-of-the-art quantized net accuracy of 71.1% (v2-1.0)
and 74.3% (v2-1.4).

6. Discussion
Mean activation shift (MAS) occurs in several neuralnet architectures after quantization. One progenitor for thisshift was demonstrated to be a non-symmetrical distributionof the rounding error of weights, more pronounced when
the weight kernels involved are small, such as in depthwise
convolutions. The phenomenon was observed in Mobilenetv1 and v2 which incorporate depthwise convolutions. It wasalso observed in Inception v1 which incorporates 2D convolutions and concatenation layers, and no depthwise layers,
leading to the conclusion that there are more sources forMAS.
We presented two methods that compensate for mean activation shift of nets by modifying biases - Iterative Bias
Correction (IBC), and Bias Fine Tuning (BFT). Employing either one significantly improves the accuracy of Mobilenet v1, v2 and Inception v1, bringing their degradation down by an order of magnitudes relative to the baseline quantization. For Mobilenet-v1/v2, post-BFT degradation is on par with the 1% state-of-the-art [7] achieved byresource-intensive quantization-aware training, tested withthe same 8-bit quantization scheme. Both tools require very
little in terms of tuning effort, run-time and data. BFT performs a robust micro-training procedure, incorporating astrong optimizer (gradient descent) and requiring ˜1000 unlabeled images; typically taking ˜20min on a single GPU.
IBC uses a simpler direct-estimation, runs even faster (23min is typical) and requires as little as 8 unlabeled images.
We expect these methods to readily extend to mobilenetbased nets used as a base for other tasks (detection, segmentation, etc.) and at other quantization schemes (e.g. 4-bit
weights). Both may permit further improvement by using
more data and tuning their parameters in a scheme- and netspecific way; however, we refrain from it here, emphasizing instead the ”without-bells-and-whistles” effectivenessof the methods, which highlights our basic insight (sec. 3.2)
about the underlying issue. The default setting seems to beenough for sub-1% degradation on most presented nets (table 1). On the other hand, in yet more challenging cases(e.g. quantization to n < 8 bits) the prospect for the methods described is to be used as a part of a wider post-training
quantization toolset. We leave that to future work.

References[1] R. Banner, Y. Nahshan, E. Hoffer, and D. Soudry. Post
training 4-bit quantization of convolution networks for rapiddeployment. CoRR, abs/1810.05723, 2018. 1, 2
[2] C. Baskin, E. Schwartz, E. Zheltonozhskii, N. Liss,
R. Giryes, A. M. Bronstein, and A. Mendelson. Uniq: uniform noise injection for the quantization of neural networks.
arXiv preprint arXiv:1804.10969, 2018. 2[3] Y. Bengio, N. Léonard, and A. C. Courville. Estimating or
propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. 2, 6
[4] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint
arXiv:1805.06085, 2018. 1, 2
[5] Y. Choukroun, E. Kravchik, and P. Kisilev. Low-bit quantization of neural networks for efficient inference. arXiv preprint
arXiv:1902.06822, 2019. 2[6] A. Goncharenko, A. Denisov, S. Alyamkin, and E. Terentev.
Fast adjustable threshold for uniform neural network quantization. arXiv preprint arXiv:1812.07872, 2018. 2[7] R. K. (Google). Quantizing deep convolutional networks for
efficient inference: A whitepaper. CoRR, abs/1806.08342,
2018. 1, 2, 3, 7
[8] S. Han, H. Mao, and W. J. Dally.
Deep compression: Compressing deep neural networks with pruning,trained quantization and huffman coding. arXiv preprint
arXiv:1510.00149, 2015. 2[9] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge
in a neural network. arXiv preprint arXiv:1503.02531, 2015.
2[10] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision
Applications. arXiv e-prints, page arXiv:1704.04861, Apr
2017. 1, 2
[11] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connectedconvolutional networks. CoRR, abs/1608.06993, 2016. 1[12] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard,
H. Adam, and D. Kalenichenko. Quantization and Trainingof Neural Networks for Efficient Integer-Arithmetic-Only Inference. 2017. 1, 2, 3, 7
[13] S. Jung, C. Son, S. Lee, J. Son, Y. Kwak, J. Han, and C. Choi.
Joint training of low-precision neural network with quantization interval parameters. CoRR, abs/1808.05779, 2018. 1[14] J. L. McKinstry, S. K. Esser, R. Appuswamy, D. Bablani,
J. V. Arthur, I. B. Yildiz, and D. S. Modha. Discovering lowprecision networks close to full-precision networks for efficient embedded inference. arXiv preprint arXiv:1809.04191,
2018. 1, 2, 7
[15] E. Meller, A. Finkelstein, U. Almog, and M. Grobman.
Same, same but different - recovering neural network
quantization error through weight factorization. CoRR,
abs/1902.01917, 2019. 1, 2, 3, 7
[16] S. Migacz. 8-bit Inference with TensorRT. 2017. 1, 2, 7
[17] A. K. Mishra and D. Marr. Apprentice: Using knowledge
distillation techniques to improve low-precision network accuracy. CoRR, abs/1711.05852, 2017. 2, 6
[18] E. Park, S. Yoo, and P. Vajda. Value-aware quantization for
training and inference of neural networks. In Proceedingsof the European Conference on Computer Vision (ECCV),
pages 580–595, 2018. 2[19] A. Polino, R. Pascanu, and D. Alistarh. Model compression
via distillation and quantization. CoRR, abs/1802.05668,
2018. 2, 6
[20] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C.
Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. arXiv e-prints, page arXiv:1801.04381, Jan 2018. 1,
2[21] S. O. Settle, M. Bollavaram, P. D’Alberto, E. Delaye, O. Fernandez, N. Fraser, A. Ng, A. Sirasao, andM. Wu. Quantizing convolutional neural networks for lowpower high-throughput inference engines. arXiv preprint
arXiv:1805.07941, 2018. 2[22] T. Sheng, C. Feng, S. Zhuo, X. Zhang, L. Shen,and M. Alf2018arXiv180104381Seksic. A quantizationfriendly separable convolution for mobilenets.
CoRR,
abs/1803.08607, 2018. 1, 2, 3
[23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR, abs/1409.4842. 1[24] P. Warder. What Ive learned about neural network quantization. petewarden.com, 2008. 3[25] P. Yin, J. Lyu, S. Zhang, S. J. Osher, Y. Qi, and J. Xin. Understanding straight-through estimator in training activation
quantized neural nets. In International Conference on Learning Representations, 2019. 6

[26] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An
extremely efficient convolutional neural network for mobile
devices. CoRR, abs/1707.01083, 2017. 1[27] R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. Improving
neural network quantization using outlier channel splitting.
arXiv preprint arXiv:1901.09504, 2019. 1, 2
[28] A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen. Incremental network quantization: Towards lossless cnns with lowprecision weights. arXiv preprint arXiv:1702.03044, 2017.
6[29] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid. Towards effective low-bitwidth convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 7920–7928, 2018. 6