Same, Same But Different - Recovering Neural Network Quantization Error
Through Weight Factorization

Eldad Meller Alexander Finkelstein Uri Almog Mark Grobman
Hailo Technologies

arXiv:1902.01917v1 [cs.LG] 5 Feb 2019

Abstract
Quantization of neural networks has become common practice, driven by the need for efficient implementations of deep neural networks on embedded devices. In this paper, we exploit anoft-overlooked degree of freedom in most networks - for a given layer, individual output channels can be scaled by any factor provided thatthe corresponding weights of the next layer are
inversely scaled. Therefore, a given network has
many factorizations which change the weights of
the network without changing its function. We
present a conceptually simple and easy to implement method that uses this property and showthat proper factorizations significantly decrease
the degradation caused by quantization. We show
improvement on a wide variety of networks andachieve state-of-the-art degradation results for
MobileNets. While our focus is on quantization,
this type of factorization is applicable to otherdomains such as network-pruning, neural nets
regularization and network interpretability.

1. Introduction
Early efforts in the field of deep learning have focused
mostly on the training aspect of neural networks. The success of these efforts has led to widespread deployment oftrained neural networks in data-centers (Park et al., 2018;
Jouppi et al., 2017) and on embedded devices (Ignatov et al.,2018) where they are used for inference which in turn emphasized the need to make the inference phase more efficient.
Quantization, which means conversion of the arithmetic
used within the net from high-precision floating-points to
low-precision integers, is an essential step for efficient deployment, however, quantization degrades network performance. Here, we follow the commonly used quantization
scheme described in Jacob et al. (2017) but note that otherschemes exist (Vanhoucke et al., 2011; Gupta et al., 2015;
Courbariaux et al., 2014) to which our results apply as well.
Briefly, integer quantization consists of approximating realvalues with intergers according to xQ = x/scale where

scale = (max(x) − min(x))/2N and N is the number ofbits used in the approximation. Each layer’s weights andactivations are given a different scale according to theirextremum values. The noise introduced by this limited precision approximation encapsulates a fundamental dynamic
range-precision trade-off.
Existing approaches to decreasing induced degradation arequantization-aware training (Jacob et al., 2017; Banner et al.,2018a; Zhou et al., 2017; 2018; McKinstry et al., 2018) andreducing the dynamic range of activations by clipping outliers (Migacz, 2017; Choi et al., 2018; Banner et al., 2018b).
Training is a powerful method but it is time-consuming, hard
to implement, and requires access to the original training
dataset which might not always be available (e.g. when theuser wishes to use an off-the-shelf pre-trained model). Clipping has limited effect since it only addresses noise from
activation quantization.
In this paper, we propose a different approach. Insteadof focusing on improving the quantization process itself,
we explore equivalent weight arrangement that make thenet less sensitive to quantization. An equivalent weightarrangements is a factorization that changes the weightsof the networks without changing its function - i.e. for agiven input, the network output remains the same. During
quantization, the range of each layer is set by the channel
with the largest absolute activation which we term the dominant channel. This single channel determines the noisein all other channels, many of which have smaller values.
Therefore, amplifying these channels to match the dominantchannel while compensating the change at the next layer
can reduce the effect of the noise.
We begin by analyzing the noise introduced by quantizationof weights and activations in terms of signal-to-quantizationnoise ratio (SQNR). We inspect the effect of channel-scalingon SQNR and introduce an equalization procedure which,under some constraints, tries to scale each output channelsuch that its range matches that of the dominant channel.
We show that equalization reduces the layer SQNR andthen apply equalization iteratively layer-by-layer and empirically show that the overall post-quantization degradationof the network decreases. Since our approach can be a pre-Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

processing step prior to quantization, it is fully compatible
with other approaches that improve the quantization process.
Nevertheless, an appealing aspect of our scheme is that formost nets it reduces the quantization induced degradation
enough as to make quantization-aware training unnecessary
and thus facilitates rapid deployment of quantized models.
The main contributions of this paper are:• Inversely proportional factorization: we show theutility of weight factorization for the task of quantization. Future work can benefit by exploiting these factorizations in other settings. To the best of our knowledge
this work is the first to both highlight and show theusefulness of inversely proportional factorizations.
• Equalization: we show that layers which have channels with similar ranges are less affected by quantization and we show how to transform a network closer
to this ideal. We also perform a quantitative analysisof the effect of equalization on quantization noise andquantization induced degradation for a wide range of
network architectures.

2. Previous Work
Equalization. Having channels with similar dynamic
ranges motivated (Jacob et al., 2017) to use of Relu6 activations which were subsequently used in MobileNets (Howard
et al., 2017). However, in practice, many channels remain
un-clamped and the dynamic range strongly varies withina layer (Sheng et al., 2018). It was also observed (Krishnamoorthi, 2018) that having a scale for each channel ofa layer greatly improves quantization performance. Whileeffective for networks where most of the degradation stems
from the quantization of weights, it doesn’t improve performance of networks that are degraded by the quantization ofactivation such as DenseNets (Huang et al., 2016).
Quantization Noise Analysis. The properties of noise induced by the quantization of both activations and weights
were analyzed in Lin et al. (2015) focusing on the optimal
bit width assignment to each layer across the network. We
follow a similar analysis but focus on the dynamic ranges ofindividual channels within a layer. Sakr et al. (2017) gives
an upper bound on the relationship between SQNR andnetwork accuracy. An empirical disambiguation of the contributions of activation and weight noise to total degradation
was given in Krishnamoorthi (2018) for several networks.
The consensus of previous works seems to be that weight
quantization is responsible for the bulk of degradation but
we show the opposite for some common networks.

3. Theoretical Foundation
For a given network architecture there exist many weightassignments that result in networks that realize the same
mapping from input to output. Thus, we are afforded with animportant degree of freedom enabling us to choose assignments that have desirable properties for the task at hand. We
show, that for a family of networks it is possible to graduallyswitch between equivalent assignments through the use of
inversely proportional factorizations. These factorizations
enable us to scale individual channels within a layer by any
positive factor. We then analyze the source of quantizationnoise and show that by scaling channels we can improve
the SQNR withing a layer. Our analysis is done for Convolutional Neural Networks (CNNs) but the same principles
can apply to other types of nets as well. Since our focus
is on trained networks we assume that batch-normalization(Ioffe & Szegedy, 2015) layers are always folded back to
the preceding layer and we ignore them.
3.1. Channel scaling through inversely-proportional
factorization
Consider a convolutional layer with kernel W, bias B, input
X, and output Y. For notional simplicity we eschew convolutions and consider matrix multiplication. To this end wei,ji,jdenote X1×(K, Y1×F, the channel vectors whenout
x ·Ky ·Fin )the kernel is centered on spatial position i,j within X,Y. We
can then write the kernel as a matrix W(Kx ·Ky ·Fin )×Fout
and the bias as a vector B1×Fout . The following two factorizations hold:Y i,j = (X i,j C1 )(C1−1 W ) + B(1)

Y i,j C2 = X i,j (W C2 ) + (BC2 )(2)

where for both cases Ci is a diagonal matrix with positive
entries and Ci−1 is its the inverse diagonal matrix. The first
factorization scales the channels of the layer’s input andthe second factorization scale the channels of layer’s output.
We now consider a simple setting where L1 and L2 are twoconsecutive convolutionalal layers in a network. We assume
that the activation function of L1, A(·), is homogeneouswith degree 1 for positive numbers. That is, it satisfies
equation 3.
A(α · x) = α · A(x) ∀α > 0(3)With this assumption, if Y1 is the output of L1 and X2 is the
input to L2, the scaling of Y1 results in a corresponding scaling of X2 . Combining all of the above we arrive to our main
result - the post-activation output channels of L1 can bescaled by any positive factors by scaling the weights in the
kernel and bias of L1 (1) and the output of the network willremain unchanged if we inversely scale the corresponding

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

weight in L2 (2). We term this endomorphism an inverselyproportional factorization and it is shown schematically in
Figure 1.

at the output of each layer. For a given tensor T we denoteT̂ the noisy version of that tensor. In addition, we denotewith T̃ ∆N the additive noise source to T due to ∆N . We
start by calculating the output of the first layerYˆ1 = X1 ~ (W1 + ∆W1 ) + ∆Y1= X1 ∗ W1 + X1 ~ ∆W1 + ∆Y1≡ Y1 + Y˜1

Figure 1. A schematic showing inversely proportional factorization
in a pair of convolutional layers. An output channel i in K1 ,the kernel of L1, is scaled by a positive factor ci . Assuming
the activation function A(·) is homogeneous, the post activation
channel is also scaled by ci . All the weights in K2 , the kernelof L2 that interact with channel i are inversely scaled. Y2 , theoutput of L2 remains unchanged. The bias terms were omitted forclarity.

With the exception of the last layer, we can scale the individual channels withing each layer in a full network by
iteratively factorizing pairs of layers. Finally, we note thatthe commonly used ReLU(Nair & Hinton, 2010), PRelu(Heet al., 2015a) and linear activations all satisfy the homogeneous property from equation (3). Thus our scheme isapplicable to most commonly used CNNs.
3.2. Quantization Noise Analysis
Understanding the formation and propagation of quantization noise across the network is an essential step in the design of better quantization algorithms. In this section we analyze the effects of weight and activations quantization using
the same two-layers setting depicted in Figure 1. We model
the effect of limited precision by adding noise terms ∆W1 ∈IRK1 ×K1 ×Fin1 ×Fout1 , ∆W2 ∈ IRK2 ×K2 ×Fin2 ×Fout2 , and∆Y1 ∈ IRFout1 to W1 , W2 , and Y1 respectively. The noisy
model is shown in Figure 2. For simplicity we also assume
that all the biases are zero and that all activations are linear.

∆W1

+ Y˜1

(4)

∆Y1

∆W1
The output of the first layer has two noise sources. Y˜1is due to the interaction of the weight quantization noise∆Y1
with the input X1 and Y˜1is the intrinsic quantizationnoise. Assuming the noises are independent, their varianceis:no
E (Ỹ1∆W1 )2 = K12 · Fin1 · E{(X1 )2 } · E{∆(W1 )2 }n ∆Y o1 2
E (Y˜1) = E{∆Y12 }(5)
For uniform quantization, the noise terms ∆W 1, ∆Y 1 distribution can be approximated as uniform, zero-centered,i.i.d processes (Lin et al., 2015; Marco & Neuhoff, 2005).
Denoting by W1r , Y1r the dynamic ranges of W1 , Y1 we get
W1r
W1r∆W 1 ∼ U (− N +1
, N +1
)
2
2
r
r
Y
Y
∆Y 1 ∼ U (− N1+1 , N1+1 )
2
2

(6)

The dynamic range of a tensor is determined by the extreme
values across all the channels within it and so the noise
distribution is determined by, at most, two channels - thechannel with the largest value and the channel with smallest
value. We term these channels the dominant channels andnote that there is substantial variance between the extermum
values of different channels. Next, we calculate the outputof the second layer. Explicit calculation of Y2 shows fournoise sources, three are due to the quantization of ∆W1 ,∆W2 , ∆Y1 , and one rooted in the multiplication of Y1∆W1 ,Y1∆Y1 by ∆W2 . The last component can be neglected inmost practical scenarios and the variances of the others are

Figure 2. A simple model showing the effects of quantization on
two layers model. ∆W1 , ∆W2 , and ∆Y1 emulate the quantization
effect.

n ∆W o Xn ∆W o1 2
1 2
E (Y˜2) =W22 · E (Y˜1)
n ∆Y o Xn ∆Y o1 2
2 2
E (Y˜2) =W22 · E (Y˜1)
n ∆W o2 2
E (Y˜2) = K22 · Fin2 · E{(Y1 )2 } · E{(∆W2 )2 }We now show how each noise term affects the overall noise(7)

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

3.3. Effect of inversely-proportional factorization on
SQNR
We use SQNR to quantify the effect of the quantizationnoise.


E Y2
E Y2
o
n
n
o
=
SQN RY ≡(8)
E (Y − Ŷ )2
E Ỹ 2
We calculate SQN RY 1 and SQN RY 2 , the SQNRs at theoutputs of layer 1 and 2 respectively, by plugging (5),(7)into (8).

E Y12
n
o
SQN RY 1 =∆W1∆Y1E (Y˜1
+ Y˜1
)2

E Y12
o
SQN RY 2 = n ∆W1∆Y1∆W2
E (Y˜2
+ Y˜2
+ Y˜2
)2

E{(Y20 )2 } = E{(Y2 )2 }

(9)

(10)

Amplifying the channels of Y1 haphazardly might increase
the layer’s extremum values which will increases the variance of the noise sources ∆W1 , ∆Y1 (6). On the other
hand, amplification Y1 is compensated by attenuation of W2
which may only decrease the variance of ∆W2 if it results
in a reduction of W2 ’s extremum values. Thus for the noise
sources we have
E{(∆W10 )2 } ≥ E{(∆W1 )2 }E{(∆Y10 )2 } ≥ E{(∆Y1 )2 }(11)

E{(∆W20 )2 } ≤ E{(∆W2 )2 }We now make the crucial assumption that the channels of
Y1 are amplified in such a manner that the variances of∆W1 , ∆Y1 remain unchanged while the variance of ∆W2∆W1∆Y1
decreases. Under these assumptions, Y˜1
, Y˜1
are
unaffected by the amplification of Y1 and we get thatSQN RY0 1 ≥ SQN RY 1

depends on E{(Y1 )2 } · E{(∆W2 )2 }, i.e. whether theamplification of Y1 is more dominant than the attenuationof ∆W2 . Thus there is no guarantee that SQN RY 2
improves. Undaunted, in the next section we present
a greedy algorithm that through iterative application of
inversely-proportional factorization improves the SQNR
across the network and reduces the post-quantization
degradation.

4. Equalization Algorithm

We now show how inversely proportional factorization affects the SQNR of both layers. We start by looking at theeffect on the signal components. We denote the scaling
vector with C ∈ IRFout1 and the scaled version of tensor
T by T 0 . For simplicity, since scaling the whole layer by aconstant has no effect on the SQNR, we can assume without
loss of generality that C ≥ 1. Therefore, we can say that
all channels in Y1 are either amplified or unchanged. In
addition, we showed in Section 3.1 that the factorization has
no effect on Y2 . Thus for the signal components we haveE{(Y10 )2 } ≥ E{(Y1 )2 }The
on SQN(9)Y2n effecton R∆Yo is more tricky.
∆W 1 21 2
E (Y˜2) , E (Y˜2)
are decreased byn ∆W o2 2
the attenuation of W2 . What happens to E (Y˜2)

(12)

Building on the analysis in Section 3 we propose twoalgorithms designed around the idea of applying a prequantization factorization that increases the energy of channels without changing the variance of the noise. This isachieved by amplifying non-dominant channels such that
their extremum values are matched with those of the dominant channels. As shown in Figure 3, these algorithms tends
to equalize the channels’ energy and therefore got the namechannel equalization.
4.1. One-Step Equalization
Algorithm 1 One Step Equalization
1: while 1 do
2:layer ← getN extLayer()3:
if layer is last layer then4:
return5:
end if
6:
kerOutChM ax ← getKerOutChM ax(layer)
7:
actOutChM ax ← getActOutChM ax(layer)
8:
kerOutM ax ← max(kerOutChM ax)
9:actOutM ax ← max(actOutChlM ax)
kerOutM ax
10:
kerScale ← kerOutChM
ax
actOutM ax
11:
actScale ← actOutChM ax
12:scale ← min(kerScale, actScale, Smax )13:
scaleLayer(layer, scale)
14: end while
Algorithm 1 explains a simple, one-step channel equalization method. We assume that the network can be represented as direct acyclic graph with layers being represnted by nodes and that it is topologicaly sorted. The algorithm is then applied iteratively beginning at the firstlayer(node) and continues until we reach all of the network’s output layers. At each iteration the layer’s channels are equalized by employing inversely-proportional
factorization with its successor layers. A layer is eligible to be equalized only once all of its predecessor layers
were equalized. The function getN extLayer() returns theSame, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization(a) One Step Equalization(b) Two Steps Equalization

Figure 3. Example of the effect of channel equalization on the two layers model. Each bar indicates the maximum values of a channel. The
blue graphs show the data for weights, the green for the activations, and the red the weights of the successor layer. Figure (a) demonstrates
the one step algorithm. On the left is the initial state and on the right is the state after equalization. Figure (b) demonstrates the two steps
algorithm. On the left is the state after is the next layer weights equalization and on the right is the final statenext layer. The functions getKerOutChM ax(layer) andgetActOutChM ax(layer) return the maximum values perchannel for the weights and activation respectively. Eachone of them results in a vector of length Fout . For eachchannel, we calculate the ratio between the layer’s extermum and the channel’s extremum for the activations andweights, these ratios are defined as the activation and weightscales. When a layer is scaled, each channel is scaled by
the minimum between the activation and weight scales. We
further limit the scale by a pre-defined maximum to preventthe over-scaling of channels with small activation values.
scaleLayer(layer, scale) is the scaling of the layer andits successors according to (1), (2). It is easy to see thatthe maximum values of the weights and the activation postequalization won’t change and that all the scales are ≥ 1.
An example of the results of one-step equalization on thechannel scales within a layer is shown in Figure 3(a). We
see that, post equalization, channels have much less variancein scales which in turn implies that they tend to have similarenergy. As explained in section 3.3 for each iteration there
is no guarantee that SQN RY 2 improves but in most cases
we witnessed that it did. Moreover, even if the SQN RY 2
decreases it will improved in next iteration when the channelof Y2 will be equalized.
4.2. Two Steps Equalization
We define optimal equalization (OE) as the state where theextremum values of all the channels are equal. OE canbe done in terms of weights only, activations only or both.
OE for activations or weights can always be achieved but

equalization of one will be sub-optimal for the other. OE
for both, on the other hand, is out of reach in most cases
because we have only one scale per channel. The two steps
equalization tries to make a step toward the optimal OE.
Algorithm 2 Two Steps Equalization
1: while 1 do
2:layer ← getN extLayer()3:
if layer is last layer then4:
return5:
end if6:
sucInChM ax ← getSucInChM ax(layer)
7:
sucInM ax ← max(sucInChM ax)
8:
kerOutChM ax ← getKerOutChM ax(layer)
9:
actOutChM ax ← getActOutChM ax(layer)
10:
kerOutM ax ← max(kerOutChM ax)
11:
actOutM ax ← max(actOutChM ax)
KerOutM ax
sucInChM ax
12:
kerScale ← kerOutChM
ax · sucInM ax
actOutM ax
sucInChM ax
13:
actScale ← actOutChM ax · sucInM ax
14:scale = min(kerScale, actScale, Smax )15:
scale = scale/min(scale)16:
scaleLayer(layer, scale)
17: end while
Algorithm 2 explains the two steps equalization process.
The basic idea is to diminish the layers extremum values before the equalization. This is done by applying
proportionally-inverse factorization in reverse - we attenuate the channels of the first layer and compensate by amplifying the values of the second layer. To avoid increasing the weight noise in the second layer the compensating

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

amplification is not allowed to change the extremum values of the second layer. This is done by using the function getSucInChM ax that returns the maximum per input
channel of the successor layer weights. Dividing each chanax
nel by sucInChM
sucInM ax will equalize the next layer and attenuate all the channels in the current layer. The second step of
the algorithm is the same as in the one-step equalization. At
the end of the algorithm we normalized all the scales so they
will all be ≥ 1. Figure 3(b) shows an example of two steps
equalization. We can see that the channels are equalized alittle bit better and that the maximum value of the next layer
is lower. Therefore, we can expect the noise in the network
after this equalization will be attenuated and indeed ourtests showed that this algorithm can produce better channel
equalization. Our intuition is that if a dominant channel
can be attenuated than it means that the weights of second
layer multiplying it are small. In other words - before thefactorization the second layer was ”naturally” attenuating
the channel, signaling that its scale is too large comparedto the other channels. In a limited precision setting it isimportant that this ”gain control” be done beforehand sincequantization is adversely affected by channels with outlier
scales.

5. Experiments and Results
In this section we perform experiments analyzing the performance of our proposed algorithms. We first verify our
analysis in previous sections by measuring the noise acrosstest networks with and without equalization. We then showthat a reduction in noise translates to a reduction in the
post-quantization degradation of classifier networks trainedon the ImageNet dataset(Russakovsky et al., 2015) and finally we show that our algorithm can also be applied to
MobileNets(Howard et al., 2017) with some modifications.
In all our tests we used layer wise quantization. Activations
were encoded using 8-bit unsigned integers and weights
were encoded using 8-bit integers. Biases were encoded using 16-bits integers. We used passive quantization, meaningthat no retrain was used and there is no need for labeled
data. For all experiments we extract the activation extremum
values using 64 images.
5.1. SQNR measurements
We designed a test that shows the noise of each layer separately. Moreover, the test can differentiate between activations and weights noises. For each layer we measured
three quantities: the layer output where only the weights∆W1
are quantized (Y1 + Y˜1), the layer output where only
∆Y 1
the activations are quantized(Y1 + Y˜1), and the layer
output where both are quantized (Yˆ1 ). To measure the noise
we compared these quantities to those of the original full
precision layer output.

We measured the SQNR of each layer and compared it tothe one predicted by (5) to verify our assumptions. The
results of this experiment are shown in in Figure 4 forResNet-152(He et al., 2015b) and Inception-V3(Szegedy
et al., 2015). The results show good agreement between
predicted and measured noise.

Figure 4. Noise estimation verification, A Comparison between theestimated SQNR according to (5) and measured SQNR. The layer
index is sorted by the estimated SQNR from largest to smallest.

We now analyze the quality of our two-steps equalization algorithm. To that end we look at a simple
setting where only the weight or only the activations
are quantized. PFor weight
P quantization, we compare
SQN RW 1 ≡(W1 )2 / (∆W1 )2 , a measure of how
well the layers weights are equalized to the weight OE.
And
quantization, we compare SQN RY 1 ≡P for2 activationP(Y1 ) / (∆Y1 )2 , a measure of how well the layers output channels are equalized, to the activation OE. This gives
us an idea how far we are from the overall OE of both activations and weights. Figure 5 shows the results of thiscomparison on Inception-V3. We see that the method improves significantly the SQNR of the activations and almost
reached the performance of the OE. For weights, the effectis smaller and the gap to the OE is larger.
Finally, we measured the effect of equalization on noises
throughout the network. We measured the SQNR in threecases: without equalization, with one- and two-steps equalization. Observing the results, as shown on Figure 6, several conclusions can be drawn:(1) overall the SQNR is improved by equalization (2) The greedy nature of the algorithm means that for some intermediate layers the weight
induced SQNR decreases. This is due to the fact that weight
induced noise is increased when the layer input is amplified.

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Figure 5. A comparison to the optimal SQNR. Demonstrating theeffect of equalization algorithm on the SQNR for inception-V3.
The figure shows how the algorithm tries to close the gap between
the original state and optimal equalization. In these graphs theenergy of the signal is measured and noise is estimated based on a
uniform distribution assumption. The layer index is sorted by theestimated SQNR from largest to smallest.

5.2. ImageNet Quantization Performance
We now show that the overall reduction in SQNR achievedby the equalization algorithms results in improved performance of the quantized networks. Table 1 summarizes our
results. We compared the classification performances of thequantized networks to their floating point version. Equalization improves quantization performance for almost all
nets and for some, like inception-V1 (Szegedy et al., 2014)and DenseNet121 , a considerable improvement is observed.
An examination of these networks revealed layers whichsuffer from an extreme imbalance between the channels.
As we showed, this significantly increase the noise within
the layer, triggering an avalanche effect throughout the network. Equalization eliminates this effect and enables good
performance post quantization. Overall we see that twosteps equalization gives better performance than one-stepequalization.
Figure 6 shows that the noise stemming from activation
quantization is always reduced while the noise stemmingfrom weight quantization occasionally increases after equalization. To quantify the effect we have on both noise sources
we repeated the performance measurements for weightsonly and activations-only quantization. We see that equalization has a positive effect for both scenarios and that which
noise term is dominant is network dependent. For example, for the Inception architectures, weights quantization isdominant, while for ResNet-152 and DenseNet activation
quantization is dominant. In addition, our results indicatethat the total degradation ≈ weight degradation + activation
degradation.
MobileNet is a challenging architecture for quantization andmany passive quantization schemes result in large degrada-(a) Matched(b) Sorted

Figure 6. Noise measurement acrossnetwork for Inception-V3.
P the
2 P
MeasurePthe activation
SQNR(Y(Ỹ1∆Y1 )2 and weights
1) /
∆W1 2
2 P
SQNR (Y1 ) / (Ỹ1
) for three cases: pre-equalization,
post one step equalization, and post two steps equalization. In figure (a), for better visibility, the layers are sorted by pre-equalization
SQNR values. In figure (b), for even better visibility, the postequalization lines are sorted as well.

tion (Jacob et al., 2017; Lee et al., 2018). It also employs
ReLU6(Jacob et al., 2017) activations which require special treatment for the equalization algorithm to work since
it does not satisfy (3). One-step equalization can be usedalmost without change if it only amplifies channels withextremum activation value below 6. One-step equalization
gives limited improvement (Table 2). To enable Two-steps
equalization we disable the division by min(scale) at theend of algorithm 2 and in doing so allow the algorithm toattenuate channels (C < 1). However, to prevent significant
modification to the full-precision network results, we limit
the attenuation to 70% of the original range. This method
shows negligible effect on the full-precision network but
shows a significant improvement for the quantized network(Table 2).
In addition, we found that due to the use of depthwise convolutions which have a small number of weights in each kernel
the mean of the quantized weights might be different fromthe original value which results in a shift of the distribution.
As a remedy to this problem we use knowledge distillation(Hinton et al., 2015) and fine-tune only the biases to compensate for the shift so that the distribution means are thesame for both the original and quantized network. Since
only the biases are being updated 1000 unlabeled images are
all that is needed and the fine-tuning process is very short.
Used in conjunction with equalization we get competitive
results (Table 2) with the state-of-the art (Jacob et al., 2017;
Lee et al., 2018; Krishnamoorthi, 2018; Sheng et al., 2018;
Google). However, our result is unique in the following
regards: it doesn’t require channel-wise quantization which

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization
Table 1. A comparison of the performance of the two proposed
algorithms. For each network we tabulate the post-quantization
degradation without equalization and with one- and two-steps
equalization. The degradation is measured relative to the networks
top-1 accuracy which is estimated using the full ILSCRC2015
validation set of 50K images. For Resnet-152 and Inception-V3 wealso measure the degradation when only the weights or activations
are quantized. To facilitate reproducibility we used open source
pre-trained networks. All networks are taken from TF-slim models(TF-slim) with the exception of DenseNet(GitHub)

Original
One
No
Top-1
Step
Equal
accuracy
Equal
Weights and Activations Quantization
ResNet-V1-50
75.2%
0.25% 0.38%
ResNet-V1-152 76.8%
1.27% 0.80%
Inception-V3
77.9%
0.66% 0.47%
Inception-V1
69.8%
2.23% 0.41%
DenseNet-121
74.3%
5.78% 0.42%
Weights Quantization Only
ResNet-V1-152 76.8%
0.23% 0.19%
Inception-V3
77.9%
0.55% 0.36%
Activations Quantization Only
ResNet-V1-152 76.8%
0.98% 0.66%
Inception-V3
77.9%
0.09% 0.03%
Network
Name

Two
Steps
Equal
0.25%
0.78%
0.35%
0.39%
0.35%
0.16%
0.32%
0.62%
0.05%

Table 2. MobileNet degradation of the different quantization
schemes compared to the floating point implementation. All networks are taken from TF-slim models (TF-slim).

Net
VersionNo
Equal

V1-1.0V2-1.0V2-1.4

7.89%
42.68%
8.06%

One
Step
Equal
6.12%
4.07%
6.21%

Two
Steps
Equal
3.2%
2.1%
1.9%

Bias
Only

Equal+
Bias

1.3%
1.5%
1.4%

0.95%
0.61%
0.55%

has significant overhead for hardware implementation as
well as additional storage requirements. It uses only ∼ 1000
unlabeled images allowing it to be used with off-the-shelfpre-trained models and the quantization process is simple
and very fast to implement.

6. Discussion
This paper highlights a property of convolutional neuralnetworks that is often overlooked which allows inverselyproportional factorization. We showed methods to harness
this property to generate equivalent networks that are much
more robust to quantization noises. Our intuition was that
networks have implicit ”gain control” mechanisms that canbe made explicit through channel equalization. When thechannel are equalized, outliers are removed and quantization

performance is improved. Given the same constrains of 8bits
quantization, layer-wise scaling, and without re-training ouralgorithms reached state-of-the-art performance.
Our focus was on passive quantization allowing rapid deployment, however, equalization should benefit other quantization methods. When fine tuning or quantization-aware
training are used, equalization can be integrated as a preprocessing step to reduce noise prior to training. We believethat most current quantization methods will benefit fromapplying proper equalization.
There is much more to explore towards realizing the fullpotential of inversely proportional factorizations. We suggested a greedy equalization algorithm that performs well
but advanced equalization algorithms can push the improvement even further. For example, we showed that the impact
of weights and activations quantization might change between layers. This property can be exploited for better
equalization. In addition, advanced prediction methods ofthe noise’s effect on the network performance like thosesuggested in Sakr et al. (2017) or Choi et al. (2016) can beused for equalization optimization.
More generally, this work is a first attempt to utilize equivalent net factorizations. The approach should find meritin other applications as well. For pruning, activation only
equalization can be employed to make the interpretation ofweight importance more natural. After equalization, small
weights have less effect on the network and therefore are
more likely to be pruned by methods that rely on the relative
weight size(Han et al., 2015). During training, inverselyproportional factorization can be used to scale the gradientsof different channels within a layer allowing for faster convergence or avoiding vanishing/exploding gradients.

References
Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable Methods for 8-bit Training of Neural Networks.
may 2018a. URL http://arxiv.org/abs/1805.
11046.
Banner, R., Nahshan, Y., Hoffer, E., and Soudry, D. ACIQ:
Analytical Clipping for Integer Quantization of neuralnetworks. oct 2018b. URL http://arxiv.org/
abs/1810.05723.
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J.,
Srinivasan, V., and Gopalakrishnan, K. PACT: Parameterized Clipping Activation for Quantized Neural Networks.
may 2018. URL http://arxiv.org/abs/1805.
06085.
Choi, Y., El-Khamy, M., and Lee, J. Towards the limitof network quantization. CoRR, abs/1612.01543, 2016.
URL http://arxiv.org/abs/1612.01543.

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Courbariaux, M., Bengio, Y., and David, J.-P. Training
deep neural networks with low precision multiplications.
dec 2014. URL http://arxiv.org/abs/1412.
7024.
GitHub, I.
pudae/tensorflow-densenet.
https://
github.com/pudae/tensorflow-densenet.
Google.
Tensorflow lite.
URL https://www.
tensorflow.org/lite/performance/
model_optimization.
Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan,
P. Deep Learning with Limited Numerical Precision.
feb 2015. URL https://arxiv.org/abs/1502.
02551.
Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained
Quantization and Huffman Coding. oct 2015. URL
http://arxiv.org/abs/1510.00149.
He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep
into Rectifiers: Surpassing Human-Level Performanceon ImageNet Classification. feb 2015a. URL http://arxiv.org/abs/1502.01852.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015b.
URL http://arxiv.org/abs/1512.03385.
Hinton, G., Vinyals, O., and Dean, J. Distilling the Knowledge in a Neural Network. mar 2015. URL https://arxiv.org/abs/1503.02531.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang,
W., Weyand, T., Andreetto, M., and Adam, H. MobileNets: Efficient Convolutional Neural Networks for
Mobile Vision Applications. apr 2017. URL http://arxiv.org/abs/1704.04861.
Huang, G., Liu, Z., van der Maaten, L., and Weinberger,
K. Q. Densely Connected Convolutional Networks.
aug 2016. URL http://arxiv.org/abs/1608.
06993.
Ignatov, A., Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., and Van Gool, L. AI Benchmark: Running Deep
Neural Networks on Android Smartphones. oct 2018.
URL http://arxiv.org/abs/1810.01109.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
CoRR, abs/1502.03167, 2015. URL http://arxiv.
org/abs/1502.03167.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard,
A., Adam, H., and Kalenichenko, D. Quantization

and Training of Neural Networks for Efficient IntegerArithmetic-Only Inference. 2017. doi: 10.1109/CVPR.
2018.00286.
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal,G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers,A., Boyle, R., Cantin, P.-l., Chao, C., Clark, C., Coriell, J.,
Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami,
T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho, C. R.,
Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Koch,
A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D.,
Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean,
G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R.,
Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick,M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek,
A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M.,
Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G.,
Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R.,
Wang, W., Wilcox, E., and Yoon, D. H. In-Datacenter Performance Analysis of a Tensor Processing Unit. apr 2017.
URL http://arxiv.org/abs/1704.04760.
Krishnamoorthi, R. Quantizing deep convolutional networks
for efficient inference: A whitepaper. Technical report,
2018.
Lee, J. H., Ha, S., Choi, S., Lee, W., and Lee, S. Quantization for rapid deployment of deep neural networks.
CoRR, abs/1810.05488, 2018. URL http://arxiv.
org/abs/1810.05488.
Lin, D. D., Talathi, S. S., and Annapureddy, V. S. Fixed
Point Quantization of Deep Convolutional Networks.
nov 2015. URL https://arxiv.org/abs/1511.
06393.
Marco, D. and Neuhoff, D. L. The validity of the additivenoise model for uniform scalar quantizers. IEEE Transactions on Information Theory, 51(5):1739–1755, May
2005. ISSN 0018-9448. doi: 10.1109/TIT.2005.846397.
McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani,D., Arthur, J. V., Yildiz, I. B., and Modha, D. S. Discovering Low-Precision Networks Close to Full-Precision
Networks for Efficient Embedded Inference. sep 2018.
URL http://arxiv.org/abs/1809.04191.

Migacz, S. 8-bit Inference with TensorRT. 2017.
URL
http://on-demand.gputechconf.
com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.
pdf{%}0Ahttp://on-demand.
gputechconf.com/gtc/2017/video/s7310-szymon-migacz-8-bit-inference-with-tensorrt
mp4.

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedingsof the 27th International Conference on International
Conference on Machine Learning, ICML’10, pp. 807–
814, USA, 2010. Omnipress. ISBN 978-1-60558-9077. URL http://dl.acm.org/citation.cfm?
id=3104322.3104425.
Park, J., Naumov, M., Basu, P., Deng, S., Kalaiah, A., Khudia, D., Law, J., Malani, P., Malevich, A., Nadathur, S.,
Pino, J., Schatz, M., Sidorov, A., Sivakumar, V., Tulloch,A., Wang, X., Wu, Y., Yuen, H., Diril, U., Dzhulgakov,
D., Hazelwood, K., Jia, B., Jia, Y., Qiao, L., Rao, V.,
Rotem, N., Yoo, S., and Smelyanskiy, M. Deep Learning
Inference in Facebook Data Centers: Characterization,
Performance Optimizations and Hardware Implications.
nov 2018. URL http://arxiv.org/abs/1811.
09886.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,
Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale
Visual Recognition Challenge. International Journal ofComputer Vision (IJCV), 115(3):211–252, 2015. doi:
10.1007/s11263-015-0816-y.
Sakr, C., Kim, Y., and Shanbhag, N. Analytical guarantees on numerical precision of deep neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3007–3016, International
Convention Centre, Sydney, Australia, 06–11 Aug
2017. PMLR. URL http://proceedings.mlr.
press/v70/sakr17a.html.
Sheng, T., Feng, C., Zhuo, S., Zhang, X., Shen,
L., and Aleksic, M.
A Quantization-Friendly
Separable Convolution for MobileNets.
mar
2018.
doi: 10.1109/EMC2.2018.00011.
URL
http://arxiv.org/abs/1803.08607http://dx.doi.org/10.1109/EMC2.2018.00011.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E.,
Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. CoRR,
abs/1409.4842, 2014. URL http://arxiv.org/
abs/1409.4842.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna,
Z. Rethinking the inception architecture for computer
vision. CoRR, abs/1512.00567, 2015. URL http://
arxiv.org/abs/1512.00567.
TF-slim, G.
Tensorflow slim models.
URL
https://github.com/tensorflow/models/tree/master/research/slim.

Vanhoucke, V., Senior, A., and Mao, M. Improving
the speed of neural networks on CPUs. Technicalreport, 2011. URL http://research.google.
com/pubs/archive/37631.pdf.
Zhou, A., Yao, A., Guo, Y., Xu, L., and Chen, Y. Incremental Network Quantization: Towards Lossless CNNs
with Low-Precision Weights. feb 2017. URL https://arxiv.org/abs/1702.03044.
Zhou, A., Yao, A., Wang, K., and Chen, Y. Explicit losserror-aware quantization for low-bit deep neural networks.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.