Mixed Precision DNNs: All you need is a good parametrization
Authors: Stefan Uhlich
Lukas Mauch
Fabien Cardinaux
Kazuki Yoshiyama
Javier Alonso Garcia
Stephen Tiedemann
Thomas Kemp
Akira Nakamura
Venue:  (2020)
Keywords: Deep Neural Network Compression
Quantization
Straight through gradients

Abstract: Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.

Published as a conference paper at ICLR 2020
MIXED PRECISION DNNS:
ALL YOU NEED IS A GOOD PARAMETRIZATION
Stefan Uhlich, Lukas Mauch; Fabien Cardinaux; Kazuki Yoshiyama Javier Alonso García, Stephen Tiedemann, Thomas Kemp Sony Europe B. Germany firstname. last name@ sony com
Akira Nakamura
Sony Corporate, Japan akira. b. nakamura@sony.comm
ABSTRACT
Efficient deep neural network (DNN) inference mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's s parameters using gradient methods. We show that a suited parametrization of the quantizer propose to
the key achieve stable training and good final performance. Specifically, parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicilly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving fate-of-the-asd performance.
1 INTRODUCTION
Quantized DNNs apply quantizers Q lR { T) (7) to discretize the weights and/or activations
of a DNN (Han al. 2015 Zhou al. 2017 7, Li al. 2016 Liu & Mattina 2019 Cardinaux et al. 2018 Jain et al. 2019 Bai et al. 2018). They require considerably less memory and have
lower computational complexity, since discretized values (71 (q.,..... 4t can be stored, multiplied and
accumulated efficiently. This is particularly relevant for inference mobile or embedded devices with limited computational power.
However, gradient based training of quantized DNNs is difficult, as the gradient of quantization function vanishes almost everywhere, i.e., backpropagation through a quantized DNN almost always returns a zero gradient. Different solutions to this problem have been proposed in the literature: first possibility is to use DNNs with stochastic weights from a categorical distribution and to optimize the evidence lower bound (ELBO) to obtain estimate of the posterior distribution of the weights. As proposed in (Jang et al. 2016 Maddison et al. 2016 Louizos et al., 2019), the categorical distribution can be relaxed categorical distribution such that the ELBO becomes differentiable under reparametrization.
concrete distribution smoothed approximation of the
second possibility is use the straight through estimator (STE) (Bengio et al. 2013). STE allows the gradients to be backpropagated through the quantizers and, thus, the network weights can be adapted with standard gradient descent (Hubara et al.) 2016). Compared to STE based methods, stochastic methods suffer from large gradient variance, which makes training of large quantized DNNs difficult. Therefore, STE based methods are more popular in practice.
More recent research (Jain et al.) 2019; Esser et al. 2019, Wang et al. 2018 Elthakeb et al. 2018 focuses on methods which can also learn the optimal quantization parameters, e.g., the stepsize, dynamic range and bitwidth, in parallel to the network weights. This is promising approach as DNNs with learned quantization parameters almost always outperform DNNs with handcrafted ones.
"Equal contribution.

---

Published as a conference paper at ICLR 2020
Recently, and in parallel to work, (Jain et al. 2019) explored the use of STE to define the gradient with respect to the quantizers's 's dynamic range. The authors applied a per-tensor quantization and used the dynamic range Similarly, (Esser et al. 2019) learned the stepsize using gradient descent. However, neither of them learned the optimal bitwidth of the quantizers.
additional trainable parameter also learned with gradient descent.
One approach was proposed in (Wang et al. 2018 Elthakeb et al. 2018). They learn the bitwidth with reinforcement learning, i.e., they learn an optimal bitwidth assignment policy. Their experiments show that a DNN with learned and heterogeneous bitwidth assignment outperforms quantized DNNs with homogeneous bitwidth assignment. However, such methods have high computational complexity as the bitwidth policy must be learned, which involves training many quantized DNNs.
In this paper, we will use the STE approach and show that the quantizer's parameters, including the bitwidth, can be learned with gradient methods if good parametrization is chosen. Specifically, we show that directly learning the bitwidth not optimal. Instead, we propose to learn the stepsize and dynamic range. The bitwidth can then be inferred from them. Compared to (Wang al. 2018 Elthakeb et al. 2018), our method has the advantage that training quantized DNNs has nearly the same computational complexity standard float32 training.
The contributions of this paper are:
1. We show that there are three different parametrizations for uniform and power-of quantiza- tion and that, in both cases, one of them has gradients particularly well suited to train quantized DNNs. The other parametrizations have the problem of yielding gradients with unbounded gradient norm and coupled components.
2. Using this parametrization, we are able learn all quantization parameters for DNNs with per-tensor quantization and global memory constraints. We formulate the training as a constrained optimization problem, where the quantized DNN is constrained not to exceed given overall memory budget, and show how to solve it in penalty framework.
3. We confirm our findings with experiments on CIFAR-10 and ImageNet. For example, we train a heterogeneously quantized MobileNetV2 on ImageNet requiring total of only 1.65MB to store the weights and only 0.57MB to store its largest feature map. This is equivalent to a homogenous 4bit quantization of both weights and activations. However, network learns to allocate the bitwidth het- erogeneously in an optimal way. Our MobileNetV2 achieves an error of 30.26% compared to 29.82% for the floating point baseline. This is state-of-the-ar for such heavily quantized MobileNetV2.
We use the following notation throughout this paper: x, x, X and X' denote a scalar, a (column) vector, matrix and tensor with three or four dimensions, respectively; [-] and [-] are the floor and ceiling operators. Finally, (.) denotes the Dirac delta function.
2 CHOOSING A QUANTIZATION PARAMETRIZATION
Let (æ;; O) be a quantizer with the parameters 0, which maps x E to discrete values {qi, }- In this section, compare different parametrizations of Q(:; 0) for uniform quantization and power- of-two quantization and analyze how well the corresponding straight-through gradient estimates (2r; 0) and V (a:; 0) are suited optimize the quantizer parameters 0. Our key result is, that the training of quantized DNNs which learns both, the optimal quantized weights and the optimal quantization parameters 0, is very sensitive the choice of the parametrization of the quantizers. From an optimization point of view, it is best to parametrize the quantizer Q(2; 0) with the stepsize d and the dynamic range 4max as it leads to gradients with stable norms. Doing so, we can use standard gradient descent to learn the quantization parameters and do not need to use stochastic or reinforcement based algorithms, which are computationally expensive.
2.1 PARAMETRIZATION AND STRAIGHT THROUGH GRADIENT ESTIMATES
A symmetric uniform quantizer (2; 0) which maps real value x E IR to one of I = 2k + 1 quantized values q E (-kd, ..., O,... kd} computes
d
| S 4max
= (2; 0) = sign(x)
2
(I)
qmax
[x]
qmax
2

---

Published as a conference paper at ICLR 2020
1,000
2
500
5
6
10
d
5
Qmax
Guax
0
(a) Case U1: 0 = [b. df
(b) Case U2: = [b, ymas]
(c) Case U3: = [d, 9max]
Figure 1: Maximum gradient norm max, V PQQu O)l For "U1" and "U2" the maximum gradient
norm can grow exponentially with varying bitwidth b whereas it is bounded for "U3".
where the parameter vector 0 [d, 9max, 61Sh consists of the step size d E R. the maximum value Jmax E R and the number of bits E N. b 2 used to encode the quantized values q.
When training quantized DNNs, want optimize (2; 0) with respect to the input r and the quantization parameters 8, meaning that we need the gradients VuQu (z; 0) and VoQ(2; 0). A common problem is, that the exact gradients
useful for training. For example, Qu:
2 d d 28 2-1111 8 (Ir d (k ))) vanishes almost everywhere. A solution is to define the derivative using STE (Bengio et al. 2013), which ignores the floor operation in [[] This leads to
0, 0,Qu(z)
{
|as [ac]
4max
fmax
which is non-zero in the interesting region La|| S Qmax and which turned out to be very useful to
train quantized DNNs in practice (Ynn et al. 2019). In this work, follow this idea and define the gradients V v (x; 0) and V VnQ (r; 0), using STE whenever we need to differentiate a floor operation.
We refer this differentiable quantization (DQ).
An important observation from ① is that the parameters = [d, Imax, bj" of a quantizer depend on each other, i.c., 9max (2 1)d. This means, that we can choose from three equivalent parametrizations of Qu 0): Case "U1" with 0 = [b, df case "U2" with 0 = [b, {max] and case
"U3" with e [d. 9max] I". Interestingly, they differ their gradients:
Case UI: Parametrization with respect to A = [b, d)",, using Jmax
Imax (b, d) gives
(Qu 0) x)
|x|| S (2 - 1)d
VoQu(1; Tu;Qu 0]
(3a)
(0QQu(;;
2- log(2)d
2h sign(x) |x|| (26-1 1)d
Case U2: Parametrization with respect to 0
[b. mmax]? I", using d = d(b, Imax) gives
2h- log
2
(Qu(x:0) - -x)
|0
Jmax
O O Ommn Qu (x;
V V (QQu(z;0) 0)
(3b)
|o||
Tmax
sign(:r)
Case U3: Parametrization with respect to 0 [d, 9max] IT.
using 6 = b(d, mmax) gives
daQu Oeaa Qu (2:
(Qu - x))
|oc|
Tmax
VoQu(x;
(3c)
O sign(x)
4max
Fig.1 shows the maximum gradient norm max, V T (z; 0) for the three parametrizations "U1"
to U3". For the parametrizations "U1" and "U2". max V (r; 0)]l can grow exponentially with varying bitwidth as Q2Qu (x; 0) E 2- 2, 1 for "UI" and 0,Qu (z; 0)
Iman log 2, L log 2) for "U2". This not desirable when training quantized DNNs. 28-1-1
because it will lead to large changes of the gradient norm and forces to small learning rates
3

---

Published as a conference paper at ICLR 2020
0p Qu Oa Qule)
Quip rmax Quia
DaQulp amee Q
Ad Qu(x)
Gmax
(a) Case U1
(b) Case U2
(c) Case U3
(d) Input derivative
Figure 2: Partial derivatives of (u 0) with respect to the input and the quantization parameters d, Qmax and b. Partial derivatives are coupled for "U1" and "U2" but are decoupled for "U3".
avoid divergence. However, parametrization "U3" does not suffer from such an unbounded gradient norm both partial derivatives QaQu (2; 0) [-2: bl and Orna Quauqu((ee (x; 0) {-1,1} are bounded.
Fig. shows the gradients for the parametrization "UI" to "U3". For parametrization "U3", the partial derivatives in VoQu ()) decoupled, i.e., VoQu(<;; is unit vector, which either points only in the direction of d if |z] S 9max or only in the direction of qmax if | > Qmax We will show in Sec. 2..3 that this implies diagonal Hessian, which results in a better convergence behavior of gradient descent. In contrast, both parametrizations "UI" and "U2" have partial derivatives that are coupled. summary, this implies that parametrization "U3" is the best DQ parametrization.
Similar considerations can be made for the power-of-two quantization @2(x; 0), which maps real-valued number x e IR a quantized value q E {=2# k 2} by
Umin
[ar] S Qmin
q = Qr(:;; 0) sign(x)
200.5+1oggle11
Jmin
< |oo| S Qmax
(4)
Gmax
[ac] 4max
where Qmin and [max are the minimum and maximum absolute values of the quantizer for a bitwidth of bit. Power-of-two quantization is an especially interesting scheme for DNN quantization, since multiplication of quantized values can be implemented as an addition of the exponents. Using the STE for the floor operation, the derivative p_Qp(x; 0) given by
| S Qmin
0₂Qp(z)=
Imin |a|| 4max
(5)
|o]| >
9max
The power-of-two quantization has the three parameters [b, 9min Imax] =: 0, which depend on each other with the relationship 9max 22n-1 Qmin- Therefore, have again three different parametrizations with 0 = [b, Jmin = [b, {mux] or = [qmin: (max]. respectively. Similarly to the
uniform case, one parametrization (0- 1mmin 4max]) leads to gradient of very simple form
[1, 0)T
|x]| S qmin
0gma Qu (x; 0) 2 p Qu 0)
veQp(r;0)
=
[0, 0
Amin < | S 9max
(6)
[0, 1)T
[x]| >> 9max
which has again a bounded gradient magnitude and independent components and is, hence, best suited for first order gradient based optimization.
2.2 CONSTRAINTS ON
In practice, for an efficient hardware implementation, we need to ensure that the quantization parameters only take specific discrete values: for uniform quantization, only integer values are allowed for the bitwidth b, and the stepsize d must be power-of-two, see e.g. (Jain et al.) ; for power-of-two quantization, the bitwidth must be an integer, and the minimum and maximum absolute values Qmin and (max must be powers-of-two.
We fulfill these constraints by rounding the parameters the forward pass to the closest integer or power-of-two value. In the backward pass we update the original float values, i.e., we used again the STE to propagate the gradients.
2.3 EXPERIMENTAL COMPARISON OF DQ PARAMETRIZATIONS
In the following compare the parametrizations using experiments.
4

---

Published as a conference paper at ICLR 2020
b, (U1)
Mean squazed aror
0.55
b, (P))
10
Smuo (U2)
6, Rain (P2)
Staax (U13
Grine Tinax (P3)
Mein 10-3
500
1,000 1,500 2,000
2,000
Ienttion
Iierationn
(a) Uniform quantization
(b) Power-of-twe quantization
Figure 3: MSE for quantizing Gaussian data x ~ [(, 1) with uniform and power-of-twe swwer-of-two quantiza- tion. Parametrizations "U3" and "P3" converge the lowest MSE without any oscillations.
1) Quantization of Gaussian data In our first experiment we use DQ to learn the optimal quantiza- tion parameters 0 which minimize the mean squared error (MSE) E 2 (Q(x; 0) x) with
gradient descent and compare the convergence speed for three possible parametrizations of uniform and power-off of-two quantizer. We choose this example as the gradient VoQ(2; 0) = [[Q((;;; 0) x> V (2; 0)] is just scaled version of (oQ(a; 0), i.e., the gradient direction de- pends directly on the parametrization of Q (x;; 0) and thus the effects of changing the parametrization
can be observed.
It is interesting to study the Hessian H
V V (Q(x; ))
E 2x of the MSE:
H = E V (x; 0) (); (Q(x;)) V 0)] R E V0Q 0) V)
Note that we use the outer-product approximation (Bihhop 2006) in order to simplify our con- siderations. From this equation is apparent that the Hessian will be diagonal for the case U3 as (xQ(I; V 0)² only contains element in either (1, or (2 and, therefore, E oz; V T)]] is diagonal matrix. From this, we can see that gradient descent with an individual learning rate for each parameter is equivalent to Newton's method and, therefore, efficient. In general this will not be the case for U1 and U2.
We conduct an experiment, using ADAM to optimize the mean squared quantization error on artificially generated data, which is generated by drawing 10dt samples from NO. 1). Please note that the same example was studied in (Jain et al. 2019). The results in Fig. 3 clearly show that the parametrizations "U3" and "P3" are best suited to optimize the uniform and power-of-two quantization parameters, respectively. Indeed, these quantizers converge without oscillation to the lowest MSE. It is interesting to see, that even adaptive gradient methods like ADAM not solve the scaling issue described in Sec. 2.1 In the Appendix [A.4we give further empirical evidence to support this claim and compare the different parametrizations for the training of quantized ResNet-20 on CIFAR- 10 using ADAM. Note that all cases use the same learning rate. For the interested reader, more detailed visualization of the error surfaces over the quantization parameters can be found in Appendix A.3
2) CIFAR-10 In our second experiment we train a ResNet-20 (He et al. 2016) with quantized parameters and activations on CIFAR-10 10 (Krizhevsky & Hinton 2009) using the same settings as proposed by (He et al., 2016). Fig. | shows the evolution of the training and validation error during training for the case of uniform quantization. The plots for power-of-tee quantization can be found the appendix (Fig. TOP. We initialize this network from random parameters or from a pre-trained float network. The quantized DNNs are trained for 160 epochs, using SGD with momentum 0.9 and a learning rate schedule starting with 0.01 and reducing it by a factor of 10 after 80 and 120 epochs, respectively. We use random flips and crops for data augmentation. Each epoch takes about 2.5 min on a single GTX 1080 Ti.
In case of randomly initialized weights, we an initial stepsize d; = - for the quantization of weights and activations. Otherwise, we initialize the weights using a pre-trained floating point
network and the initial stepsize for layer is chosen to be d, 2 2 (max W/1/(22 - The remaining quantization parameters are chosen such that we start from an initial bitwidth of = 4 bit. This is reasonable upper limit for b, as in practice no performance degradation can be observed for > 4bit. Even simple offline algorithms like min/max quantization result in networks with good accuracies. We define memory constraints during training, i.... the network can learn to number of bits to quantize weights and activations of each layer. From Fig. 4 we again observe that the parametrization 0 = [d, {max] is best suited to train uniformly quantized DNN as it converges
large
5

---

Published as a conference paper at ICLR 2020
Table Comparison of different DQ parametrizations for ResNet-20 on CIFAR-10. (validation error with "random" float net" initialization)
Uniform quanfization
Puwer-uf-two quantization
Quantization
Fluat32
[b, d
[b, gead]
[d. good
[b, 9ows
[6, Qoiel
(R Beas]
Weights
8.50%/7 29%
17.8% 17.8%/8.18 18%
8.80%/7 9.43%/7.74%
8..0%
11.70%
53.07%/23.018
10.61%
WeiitissAActiiationn
9%.0%%
9.23 4..%%
22.91%/11.68
divenging/35.00%
15.10%
Kandom initialization
Pre-trained inttialiaaion
10
Volidaiiin crnir
10
Training creor
10
ermi
10
Taining
10-1
30 30-1
10-2
10-2
10
-0
-104
-10
100
Dension
Tieration
Deration
Teraaion
b, dl (UT)
b. (U2)
d. 9raxe (U3)
Figure 4: ResNet-20 with uniformly quantized weights and activations.
to the best local optimum. Furthermore, observe the smallest oscillation of the validation error for this parametrization.
comparess the best validation error for all parametrizations of the uniform and power-of-two quantizations. We trained networks either with quantized weights and full precision activations or with both being quantized. In case of activation quantization with power-of-two, we use bit to explicitly represent the value x = 0. This advantageous as the ReLU nonlinearity will map many activations to this value. We can observe that training the quantized DNN with the optimal parametrization of DQ, i.e., using either 0 = [d, 9max] 1 or = [Tmin [max]" results in a network with the lowest validation error. This result again supports our theoretical considerations from Sec. 2.1
TRAINING QUANTIZED DNNs WITH MEMORY CONSTRAINTS
We now discuss how train quantized DNNs with memory constraints. Such constraints appear in many applications when the network inference is performed on an embedded device with limited computational power and memory resources.
A quantized DNN consists of layers which compute
X = fiQQW, 0/)) Q(X Q(c;; with (= 1,.., L,
where fi(-) denotes the nonlinear activation function of layer I and Q(:; 0) is per-tensor quantization with parameters 0 applied separately to the input and output tensors [-] E I₁ and X; E I,, and also to both the weight tensors Wi E Pi and the bias vector e₁ E Rd For a fully connected layer, Z₁-1 = M: It MM are vectors, P: = RM M are matrices and A * B is a matrix- vector product. In case of convolutional layer, I
-MM11XN,-11 - In M, Pi = R - x; K, are tensors and is a set of M 2D convolutions, where the square-sized feature maps of size N Ni-1 using square-sized
convolution is performed kernels of size K₁ x Kj.
DNNs with quantized weights and activations have smaller memory footprint and are also com- putationally cheaper to evaluate since Q(a: 0) ( 0) for a E R requires only an integer multiplication for the case of uniform quantization of-two quantization. Furthermore, ((a; 0) + (@;; 0) for a, B E R only requires an integer addition. Table 2 compares the computational complexity and the memory footprint of layers which apply uniform or power-of-two quantization to weights and activations.
integer addition of the exponents for power-
We consider the following memory characteristies of the DNN, constraining them during training:
1. Total memory S (0) O = E[= S (0:") to store all weights: We use the constraint
5t = - 1=1
gi ( 0 = 5 (0 0%
(0
S < 0.
(8a)
In this paper, we "weights" refer W and c.

---

Published as a conference paper at ICLR 2020
Table 2: Number of multiplications CTTaa additions Cradd as well as required memory to store the weights St" and activations Sf of fully connected and convolutional layers.
Layer
Quantization
Cr
C
ST"
ST
uniform
My Mi
Mt M, 2M,
Fully connected
Mi 1700
pow-2
Convolutional
uniform
Mi Mr
NN7 K?
M, N? M IN
(M 196
MiN
pow-2
to ensure that the total weight memory requirement (")" ...., 4 is smaller than certain maximum weight memory size Si gives S (01) for the case of fully connected and convolutional layers. Each layer's s memory requirement S/2 (0.") depends on the bitwidth lif: reducing SV (01²) will reduce the bitwidth by"
2. Total activation memory (0 ..... 0)) = El=1 (0 to store all feature maps: We use the constraint
02(0 07. = 5th (0f, (Of...., 0 5 Σ 7
ST (0T
5/ 0,
(8b)
to ensure upper limit on the total activation memory size Sg- Table 2 gives S (0F) for the case of fully connected and convolutional layers. Such a constraint is important if we use pipelining for accelerated inference, i.c., if we evaluate multiple layers with several consecutive inputs in parallel. This can, c.g., be the case for FPGA implementations (Guo et al. 2017).
3. Maximum activation memory (TT(
0 = max I..... S to store the largest feature
map: We use the constraint
g .... ,i, STx (0), 07) S6
max (=1,..,
(S
S0 S 0,
(8c)
to ensure that the maximum activation size S does not exceed a given limit Sg This constraint is relevant for DNN implementations where layers are processed sequentially.
To train the quantized DNN with memory constraints, we need to solve the optimization problem
min W..c..ffoo .0f E [J( s.t. (0) ...., 0% 0 .... 0 S 0 for all L,..., 3 (9)
where J ,,, V') is the loss function for yielding the DNN output X' although the ground truth is V. Eq. (9) learns the weights Wr, G as well as the quantization parameters 0.,, 0.. In order to use simple stochastic gradient descent solvers, we use the penalty method (Bertsekas 2014) to convert into the unconstrained optimization problem
3)1 1., Σ X max(0, gj (OT, ..... 0 "i,0,,.., OD);
min Ep(x, [] W. of
(10)
where A E R+ are individual weightings for the penalty terms. Hence, training with weight and activation size constraints requires choosing two penalty weightings Aj. one for (8a) and one for either (8b) or (8ch.
Note, that the optimization problem (0) does not necessarily give a quantized DNN which fulfills the memory constraints. The probability to fulfill the constraint gj depends on the choice of Ay. In particular, this probability increases with larger Ay- However, choosing a too large X, will yield penalty term that dominates over the network loss decreasing the network performance. In our experiments, we choose X, such that the initial loss and the penalty term have approximately the same magnitude. Using this simple heuristic, we optained quantized DNNs that reached a high accuracy and at the same time fulfilled the constraints at the end of training.
4 EXPERIMENTS
In the following, we will use the best parametrizations for uniform and power-of-two DQ, i.e., Ou = [d. ymax] 1 and 0p [ymin: 9max] 12" that we found in Sec. 2 Both parametrizations do not directly depend on the bitwidth b. Therefore, we compute it by using b(Ov:) [lgg2
Jas + 1
d
and b(0p) logg (l0g2
)+1 +1]. All quantized networks
pre-trained float32
grin
network for initialization and all quantizers are initialized as described in Sec.
2.3 For our experiments
on CIFAR-10, use the same training setup described in Sec. 2.3 For the experiments
7

---

Published as a conference paper at ICLR 2020
Table Homogeneous vs. heterogeneous quantization of ResNet-20 on CIFAR-IO.
Bitwidth Weight/Activ.
Groee
Size
Uniform quant. Power-of-two quant.
Weight/Actv..
Validation error
Validation error
Baseline
32bit/3bbit
1048KB/64KB7736KB
7.29%
Fixed
2bit/322bit
fixed/-
65, 5/.5KB//6KB/3366B
10.81%
8.99%
TQT Jain
2019
2bib/32bitt
learned--
65.5KB/64KBB7366BBBB
9,47%
8.79%
Ours (W7 constE K3
learne/322ii
learned/-
7UKBB66KKBB/366E
8,59%
8,53%
Fixed
2bio/4bit
Excd/fixed
65. (5.5KB/3KB/22KB
11.30%
11.62%
TQT Jain et al.
2019
2hic/4bit
learnedMeammed
65.5KR/SKR/99KR
9.62%
1 11.29%
Ours constr
learned/lcamed
learned/icamed
70KB/ /92KB
9.38%
11.29%
Ours (W/ constz.
and 8c
learne/leameed
learned/leameed
70KB/8KB/
8,58%
11.23%
ImageNet, we train the quantized DNNs for 50 epochs, using SGD with momentum 0.9 and learning rate schedule starting with 0.01 and reducing it by a factor of 10 after 16 and 32 epochs, respectively. Please note that we quantize all layers opposed to other papers which use a higher precision for the first and/or last layer.
In our experiments, we noticed that the performance of DQ is not sensitive to the choice of X, in (1O). For the CIFAR-10 experiments, we use À = 0.1 for both constraints (for sizes in kB). For the ImageNet experiments, we kept the same regularization level by scaling x with the square of the size ratio between the ImageNet model and the CIFAR-10 10 model. We scale with the square-ratio as the constraints in (0) are squared penalty terms.
First, in Table 3 top, train ResNet-20 CIFAR-10 with quantized weights and float32 activa- tions. We start with the most restrictive quantization scheme with fixed Jmax and b 2bit ("Fixed""). Then, we allow the model to learn Jmax while b = 2bit remains fixed as done in (Jain al. 2019) ("TQT"). Finally, we learn both gmax and b with the constraint that the weight size is at most 70KB ("Ours"), which just 4.5kB larger that the previous 2Bit networks. This allows the model allocate more than two bits to some layers. From Table too,, we observe that the error is smallest when we learn all quantization parameters.
In Table Bbottom, weights and activations are quantized. For activation quantization, we consider two cases as discussed in Sec. 3 The first one constrains the total activation memory S while the second constrains the maximum activation memory S such that both have the same size as a homogeneously quantized model with 4bit activations. Again, we observe that the error is smallest when we learn quantization parameters.
We also use DQ to train quantized 18 (He et al. 2016) and MobileNetV2 (Sandler et al. 2018) on ImageNet (Deng et al. 2009) with 4bit uniform weights and activations or equivalent-size networks with learned quantization parameters. This is quite aggressive and, thus, fixed quantization scheme loses more than 6% accuracy while our quantization scheme loses less than 0.5% compared to float32 precision network.
Our results compare favorably to other recent quantization approaches. To our knowledge, the best result for 4bit ResNet-18 was reported by (Esser et al. 2019) (29.91 % error). This is very close to our performance (29.92 229.92% error). Importantly, (Esser et al. 2019) did not quantize the first and last layers, meaning that their network is much bigger. Specifically, compared to our quantized R their model with high precision input and output layers requires 37% more memory to store the weights. Moreover, (Esser et al., 2019) learns stepsizes which are not restricted to powers-of-two. As explained in Sec. 2.2 uniform quantization with power-of-two stepsize leads to more efficient inference, effectively allowing to efficiently compute any multiplication with an integer multiplication and bit-shift. To our knowledge only (Wang et al.) 2018) reported results of MobileNerV2 quantized to 4bit. They keep the baseline performance constraining the network to the same size as the 4bit network. However, they do not quantize the activations in this case. In addition, DQ training is efficient since is comparable to the training of unquantized network. Specifically, one epoch on ImageNet takes 37min for MobileNetV2 and 18min for ResNet- 18 on four Nvidia Tesla V100.
Fig. 5 shows the weight bitwidth assignment over layers. We observe that small bitwidths are used for layers with many parameters, i.e., pointwise convolutions and fully connected layers. However, the resulting bitwidth assignments are complex, meaning that there is no simple heuristic. Therefore, it is important learn the optimal bitwidth assignment.
8

---

Published as a conference paper at ICLR 2020
Table 4: Homogeneous vs. heterogeneous quantization of MobileNetV2 and ResNet-18 on ImageNet.
MobileNet
ResNet-18
Bitwidth Weight/Activ.
Siran
Size
Validation Error
Size Weight/Acti((mas)
Validation
Weight/Activ.
Weight/Activ(mmx)
Emor
Baseline
32biu/2bii
13.23MB/4
29.82%
44.56MBB3.04MBB
29.72%
Fixed TQT Jain
4bit/4bit
fixed/fixed
L65MBA0.57MB
36.27%
5,57MB/0,38ML
34.15%
2019)
4bir/4bit
leamed/learned
165MBA0557MB
32.21%
5.57MB/0,38MB
30.49%
Ours (w/ consir.
and Rc
learned/leameed
learne//eareed
LSSMB/I.57MB
30.26%
5.40MB/0.38MB
29.92%.
Ours (wlo constr.)
learne//eaareed
learned/icareed
3.14MB/1.58MB
29.41%
10.50MBB/1
29.34%
ResNet-18
MobilaNeiV2
bi
hit
Divellit
Birohlit
bit
bit
2 bit
2 bii
Layer
Layer
Convolution
Degthwise
Figure 5: Weight bitwidth assignment over layers for ResNet-18 and MobileNetV2 on ImageNet with weights constrained to a maximum size of 5,57MB. Our method has learned a heterogencous bitwidth distribution, which gives better performance than homogeneous (see Table 4).
5 CONCLUSIONS
In this paper we discussed differentiable quantization and its application to the training of compact DNNs with memory constraints. order fulfill memory constraints, we introduced penalty functions during training and used stochastic gradient descent to find the optimal weights well as the optimal quantization values in joint fashion. We showed that there are several possible parametrizations of the quantization function. In particular, learning the bitwidth directly is not optimal; therefore, we proposed to parametrize the quantizer with the stepsize and dynamic range instead. The bitwidth quantization methods while does not require to retrain the network multiple times in contrast to reinforcement learning approaches (Wang al. 2018 Elthakeb al. 2018
then be inferred from them. This approach is competitive to other recent
ACKNOWLEDGEMENTS
We would like to thank Masato Ishii for many helpful comments during the preparation of this manuscript.
REFERENCES
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. CoRR, abs/ 10111080000111 2018. URL httpp://aaxxvv..gg/bbs////////// 00861
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.343: 2013,
Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Fabien Cardinaux, Stefan Uhlich, Kazuki Yoshiyama, Javier Alonso Gareía, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Iteratively training look-up tables for network quantization. arXiv preprint arXiv: 101::vv:111000 11181115155, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.

---

Published as a conference paper at ICLR 2020
Ahmed T. Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Esmaeilzadeh. Releq: A reinforcement learning approach for deep quamtization of neural networks. CoRR, 0bs//811.001004 2018. /RL:t////aaxxx
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S. Modha. Learned step size quantization. CoRR, abs/1902.08153, 2019. URL
http://arxiv/ wwtpoorg//////oo// /ttp:///xxiV.orrggaa///2222 08153
Kaiyuan Guo, Shulin Zeng, Jincheng Yu, Yu Wang, and Huazhong Yang. A survey of fpga-based neural network accelerator. arXiv preprint arXiv:1712.008: 1714.0834, 2017.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network CoRR, abs/1510.001499, 2015. URL
with pruning, trained quantization and huffman coding.
http:// /ttp:///arrvv..m org/ abs htth/bbb//5155
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. CoRR, abs/ (oss/169.00061,, 2016. URL http://axxxv.. www.aawwwwwwi.oaaaaoooaooooooeot
abs/1609. 07061
Sambhav R. Jain, Albert Gural, Michael Wu, and Chris Dick. Trained uniform quantization for accurate and efficient neural network inference fixed-point hardware. CoRR, abs/1903.08006, 2019. URLpttp: //axxiv www.vv.rrgrggo.o org/abs a1903 08066
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1.01 144, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks, arXiv preprint arXiv: 2016.
Zhi-Gang Liu and Matthew Mattina. Learning low-precision neural networks without straight-through estimator (ste). arXiv preprint /903.00 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019, URL lhtpp:: httptt:::p//pppeeeiiieee net
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. CoRR, abs//611.00777 2016. URL httt:://axx/i
org htl//bb/1150 /111 00712
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh
Chieh Chen. Mo-
bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Sony. Neural Network Libraries (NNabla).
www http:/wwww..oooooo/nnaaaaaaaaaaa httppp//ggg.hhh....o/ooooo//aaaa
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quantization. CoRR, abs//811..0 1.08886., 2018. URL
0888
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher. Yingyong Qi, and Jack Xin. Under- standing ttraight estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044, 2017.
10

---

Published as a conference paper at ICLR 2020
A DERIVATION OF THE GRADIENTS FOR DIFFERENTIABLE QUANTIZATION (DQ)
In the following sections, we will give the derivatives & (a:; 0) and gradients VoQ(=; 0) for the uniform and the power-of-two quantizers. The results are summarized in Sec. 2.
82
We use the straight-through gradient estimate whenever we need to differentiate a non-differentiable floor function, i.e., we assume
x = 1.
(11)
A.1
DERIVATIVES OF THE UNIFORM QUANTIZER
Fig.
a. shows a symmetric uniform quantizer (a; 0) which maps a real value æ R to of quantized values q E { (-kd, O, O,......d kd] by computing
2k
d
2 S 9max
= Qu (z; 0) = sign(x)
(12)
Qmax
a > Gmax
using the parameters 0 [dl 4max bjTT where d the stepsize, 9max E R is the maximum value
and E N is the number of bits that we use to encode the quantized values q. The elements of O are dependent as there is the relationship 9max = (2h-1 1)d.
A.1.1 CASE U1: PARAMETRIZATION WITH RESPECT TO b AND d
For the parametrization with respect to the bitwidth b and steps size d.
12 is given by
a S (20-1 11dd
q = Qu(z; = sign(r)d
(13)
20-1
x > (2b-1 1)d
and the derivatives are given by
aQu(z; sign(x)
26 1 log 2 26-1 -
10
x I (2 1)d
(14a)
0b
(22-- -1)d
| > (2b-1 1)d'
aQu(z; sign(x) Od
1 d |x||
|
| I (2 1)d
(14b)
(2b-1 l)d
|o| (2 1)d
A.1.2 CASE U2: PARAMETRIZATION WITH RESPECT TO b AND 9MAX
For the parametrization with respect to the bitwidth b and maximum value 1max.
12 is given by
|x]| 2h +
1 2
q = (u b, (max) = sign(ar) (max 26-1-1
Quee
Gmax
(15)
Gmax
and the derivatives are given by
aQu (2; b, (mnx)
sign(x)
26 log 2
25-1
|xc]
2--1-1
Br
|oc|
Gmax
(16a)
Øb
20-1
4max
aQu (z; b, (mnx))
sign(x:)
26 I-1
|
20
2
+
32
|20 <
Gmax
(16b)
8guax
Umax
9mnx
x
Qmax
20
+
where 31 straigh gradient estimate for the floor function.
Smax
|x]| and B2
-|2x], if we the
2b-1 log 2
Bqax
11

---

Published as a conference paper at ICLR 2020
(a)
9max [Q((2)
Omxx
Omax
9min
9max
(a) Uniform quantization
(b) Power-of quantization
Figure 6: Examples of uniform quantizer (2) and power-of-two quantizer (p(z) for b = 3 bits
A.1.3 CASE U3: PARAMETRIZATION WITH RESPECT TO d AND 4MAX
Eq. (12 gives the quantization with respect the step size d and maximum value 4max derivatives are
The
(Qu d, (max)
sign
1 d
2
|0||
ar
4max
(17a)
Od
d
qmax
UQu d, 9max)
Qmax
sign(z)
(17b)
0qmax
qmax
Amax
[2]
9max
A.2 DERIVATIVES OF THE POWER-OF-TWO QUANTIZER
Power-of-twe quantization Qp(x; 0) maps real-valued number E R quantized value g E (+2*; k E 2} by
Jmin
[x|| < Qmin
q = Qp(:; 0) (ign(x)
210.5+logg/zt]
Umin
|
S
9max
(18)
Imax
Amax
where amin and Ymax the minimum and maximum (absolute) values of the quantizer for bitwidth of bits. Fig. 6b shows the quantization curve for this quantization scheme.
Using the STE for the floor operation, the derivative (xQp(x; 0) is given by
|x|| S Amin
0,Qp(x) 2 1i
9min
|x|
9max
(19)
|x| >
Qmax
The power-of-two quantization has the three parameters 0 [b, qmins [max], which are dependent on
each other, i.c., mmax 22h-1-1 *minn- Therefore, we have again three different parametrizations with [b, gmin] 0 = [b. [max] or [min [max] respectively. The resulting partial derivatives for each parametrization are shown in Fig. and summarized in the following sections. Similar the uniform case, one parametrization (0 Amin (max]) leads to gradient with the nice form
[1, 0j
|x|| < qmin
2gmm Qu
V TpQp(z; 0)
O Qui 0)
[0, 0
Amin
| S 9max
(20)
[o, 1J"
|x]] 4max
which has bounded gradient magnitude and independent components and is, hence, well suited for first order gradient based optimization.
A.2.1 CASE P1: PARAMETRIZATION WITH RESPECT TO AND MMAX
For the parametrization with 0
[b, (max].
(8)
is given by
2
x < 2-2 2-20-11
4max
qmax
Qr(2;b. b, 9mmx) = sign(x)
2 2l054+
2 2-2"
141
qmax
S 4max
(21)
Omax
x Qmax
12

---

Published as a conference paper at ICLR 2020
8Q (m) 56
[a) 0mas
aQ 3b
8Q p (a) Omin
8 (
8Q p
amin
mmax
aQ aQp(2)
20
(a) Casc Pl
(b) Case P2
(c) Case P3
(d) Input derivative
Figure 7: Derivatives for the three different parametrizations of (x(;; 0)
and the partial derivatives are
2 2" +b (log 2)2 Imax
-2 2-22
aQr (x;; b, Amux)
7max
Öb
sagn(x)
-2 -2-2" 2" ++1
Umax
< Qmax
Gmax
(22a)
2 2-22-1+1
-2 2 22-1+1
3Qp(r; 30; 9max)
Amax
Amax
sign(x)
-2 -2-20
20-2011-
+1
9max
4max
(22b)
Quax
A.2.2 CASE P2: PARAMETRIZATION WITH RESPECT TO AND JMIN
For the parametrization with 0 = [b, m.nn]
18
is given by
Omin
|x|| I Umin
Qp(x; b, 9min) sign( az)
0.5 10g2/111==1
Gmin
< |x| < 2
Amin
(23)
22
[x| > 22
Umin
Tmin
and the partial derivatives are
qmin
8Q ÖQr(r; (min)
= sign(x)
Tuin
x < 22
Mmin
(24a)
bb
2 40-2 2log 2)2 Imin
x > 2
Mmin
[x|| < Umin
aQ b1; 4min)
Oquin
sign(x)
220
Amin
<
[ 2
[x] < 22
qmin
(24b)
Amin
A.2.3 CASE P3: PARAMETRIZATION WITH RESPECT TO ISIN AND JMAX
Eq. (18) gives the parametrization of (( 0) with respect the minimum value 4min and maximum value Jmax. The derivatives are
aQ (x; Umin 4max)
| S
Imin
sign(x)
Amin
2
< 4max
(25a)
Dquin
|
(fmax
eQ (2) quin, {max)
0
[
<
Qmin
sign(x))
Gmin
Gmax
(25b)
0qpax
x
Gmax
13

---

Published as a conference paper at ICLR 2020
0.3
15
0
d
Yuur
10
10
d
Quan
(a) Case U1
(b) Case U2
(c) Case U3
Figure MSE surfaces for uniform quantization. Only U3 reaches the optimum 0".
0.5
0.5
0.5
2
0.5
b
Gun
10
Gmin
10
Guxx
(a) Case P1
(b) Case P2
(c) Case P3
Figure 9: MSE surfaces for power-of-two quantization. Only P3 reaches the optimum 0*,
A.3 VISUALIZATION THE ERROR SURFACE FOR THE QUANTIZATION OF GAUSSIAN DATA
In Sec. 2.3 of the paper, we compared the three different parametrizations of the uniform quantizer at the example of optimal quantization of Gaussian data. To get better understanding of Fig we show how the eπrоr surfaces look like for this example problem. The experimental setup the same as in Sec. 2.3, i.e., we use DQ to learn the optimal quantization parameters of a uniform and
power-of-ree two quantizer, which minimize the expected quantization error ming (xr - Qz; 0))²
We three different parametrizations, adapt the quantizer's parameters with gradient descent and compare the convergence speed as well as the final quantization error. As an input, we generate 10d samples from NO,, 1).
Fig. showw the corresponding error surfaces for the three different parametrizations of the uniform quantization. The red curve shows the path through the parameter space taken by gradient descent in order to optimize the MSE, starting with the initial values = 2, = 9max 1. The optimum 0" is located largest sample magnitude in our dataset is max (11. ØN S 4. In each of the cases U1-U3, the error surface is composed of steep ridges and large flat regions. The steep ridges force us use small learning rates to avoid divergence. For cases U1 and U2, the optimal 0" can not be reached. However, for U3, 0* lies the border of flat region and can be easily reached. Furthermore, case U3 shows much faster and more stable convergence without oscillation, since the gradient magnitudes are bounded and the error surface has fewer steep ridges where gradient descent starts oscillating.
16, dg2-- 4max 4, since allow maximal bitwidth of 16bit and the
Fig. 9 shows the corresponding error surfaces for the three different parametrizations of the power-of- two quantization. Again, the optimum 0" is not attained for two parametrizations, namely P1 and P2. as 0" is surrounded by a large, mostly flat region. For these two cases, gradient descent tends to oscillate at steep ridges and tends to unstable. However, gradient descent converges to a point close to 0* for parametrization P3, where
Amins [mux)-
Finally, we also did a comparison the different power-of of-two quantizations on CIFAR-10. Fig. 10 shows the evolution of the training and validation error if start from random pre-trained float network initialization. We can observe that = [ymin: qmax] has the best convergence behavior and thus also results in the smallest validation error (cf. Table 1). The unstable behavior of P2 is expected as the derivative 8Qp take very large (absolute) values.
Squi
A.4 FURTHER EXPERIMENTS WITH ADAM
Finally, did experiment to verify that the parametrization is important, even if adaptive gradient descent methods like ADAM are used for optimization. glivesslgives the results for a ResNet-20 trained CIFAR-10. We observe, that again U3 and P3 are the best parametrizations.
14

---

Published as a conference paper at ICLR 2020
(3)
Initualzation
Validiooo erter
10
Training etaur
100
10
Training
10-2
10-1
10
-20
leraionn
Ferraion
leration
Deration
((U))
Gree
Oree (U3)
Figure 10: ResNet-20 with power-of-two quantized weights and activations.
Table 5: Error rate of ResNet-20 on CIFAR-10 10 using different quantization parametrizations. Training done either by SGD with momentum or ADAM.
Parametrization
SGD momentum
ADAM
U1
11.74
7.61
U2
7.44
7.85
7.32
7.36
P1
15.35
7.54
P2
7.74
7.79
P3
7.40
7.40
B IMPLEMENTATION DETAILS FOR DIFFERENTIABLE QUANTIZATION
The following code gives our differentiable quantizer implementation in NNabla (Sony). The source code for reproducing our results will be published after the review process has been finished.
15

---

Published as a conference paper at ICLR 2020
B.1 UNIFORM QUANTIZATION
B.1.1 CASE UI: PARAMETRIZATION WITH RESPECT TO b AND d
del parametris Foint
iiitt, Lintt, dinia, duexx, aappaaanteesseaaae "erarrantrii version of "cinos_peint, pein___pps__ttttttttse quantizne whare the bitwidth sispsise 'a" learnable paraneless.
Returns:
10
...
scap_cca, min_value, nax_value);
12
rebuen E.nininanssalat i..aaiiinn ninuvaluw), nan_valus)
13
14
shape!:
reburn brooddestt (P.cashape (1,) lentshape), ImplacenFaleet, shapeeshappe
16
17
def ieentiieppppe www..pepeeeeeee (v):
18
P.roundiP, log (2g
20
get pww.aaater.ooooooeeeoo.. ("n"" (),
21
CunsLani Ioiiinitiin Iniiialie (Liinil), gradeTrue, Lredaded_gr Eix.perameters) pat potaeeeroooo orevaate ("d", (h,
24
25
Constant initialiserdd initl, need_gradeTrue, gradeed_r cixpparaneteral
28
29
enxucw that bitwidth as apeci fled range and an inceges F.cound(alip sts.ppsssiiiin Pumin, n_max]1 signs
32
n
33
stepai speci fiedd range and power quant poe_ ipuauelar dinin, dLtax))
36
37
ensure that dynamic is in speat fied
40
comрute ni n/nex va value that we can represent
41
elaez
44
xnin na. Variabbe((tll geod_grad=Faisse xnin.d 0,
45
47
broadcast variahles to correct size
48
brosdcest aos larid,, shape brosdcest_ccskk omin, brosscest__oonlee Demox, shape=x.enape)
49
52
apply tixed-point quantizaticn Foround clipppppp..
53
smin, xmaxl
16

---

Published as a conference paper at ICLR 2020
B.1.2 CASE U2: PARAMETRIZATION WITH RESPECT TO b AND 4MAX
porametria hiee___iee___ddee_e__
sig,
init, min, nax,
iaax_ xeax_min, xnxx_nax, sx..prrmeteree-Psie sFixed point quantize' bitxicth 'b" and dynanic parge 'xnax" aro leatrable porametors.
Paraanetrii
Returns:
nnabbl, Veriahio: arcey.
10
I1
def dl ip sos Mallarie bs, min_valus, nax value:
12
return wninimmm_scaaarr itt.smmm_rrrrr..otommooooeoooo moxxmmun_ccmior(( min_valuo), nax_value)
13
14
sot brasdiest scclar shapsi:
15
return b.broddcastt (F. iy, (1)) imnisazpo), inplace-Falsel, chape
16
quantize__evt (2)
18
return ** E.rounsiE..ogiv) / (2.1
19
20
orw.rrrrre""""""" (),
necd_gadd-Teus,
23
as_noed_eraddmot cix_parametorsi
24
need_grad geea,
27
graddnot pirpperattt
28
ensere Fitwisth
range an inceger
30
P. (a Runia, nunax]]
31
1t sign:
32
34
easuze that dynanic zonge is in specicted rarge xraa scaipscaaer (NmaN, xnax_min, xmox_nax)
35
36
compute step
dycanic zange and sure pox2 (anax / (2 ** 1))
38
quant quant pow2 powd
39
40
nia/max that
sign:
42
xnin -xnax
43
c1oc:
44
Variaababllll neenggrad=False)
46
47
broadcast variables to correct sise
48
broadazst (d,
broedoost_aeesle benan, shape)
30
seax
avoadosst_acalle tanax, shape)
51
apply rixed point quait secarn F.round by (s, snin, smaxt
17

---

Published as a conference paper at ICLR 2020
B.1.3 CASE U3: PARAMETRIZATION WITH RESPECT TO d AND îMAX
porametric patee__ooo___gantttteeeeee
sig,
init, min, max, imax_ xnax_min, xnxx_nax, sxx_premetere--asiee sFixed point quantize' stepsize 'd" and dynanic parge 'xnax" aro leatrable porametors.
Returns:
Veriahio: arcey.
10
I1
def dl ip. soa coalarty, is, min_valus, nax value:
12
return wnintmmm_ccaaar ott.ammm_rrrrr..otommooooeoooo moxxmmun_cccior(( min_valuo), nax_value)
13
14
sot braadzest brasditst scclar shapsi:
15
return F.broddcastt (F.reshape iy, (1)) imnisazpo), inplace-Falset, chape
16
Srontizs_pev2000
18
return ** E.rounif...ogvv) / (2.1
19
20
gct_parameter_.c or_erestee""""
Consraninntttallzorrrd....
necd_gadd-Teus,
23
as_noed_grad cix_parametorsi
24
htt__parmmterr.om cremte""""xx"""
aeed_grad Ieen,
27
ac_nerd_grad-mot pix.parattters
28
enseze stopsize is specitied tange power
30
quantize_ (cl ac_aaaa (é, d_nin, d_nax))
31
32
dynanic clip_acalar (wrax, amax_nin, xлax_nax)
specidied
34
35
compute nia/max value that ve can represent
36
xnin -xaxx
38
elae:
39
xnin na. (a.Variable((L,ll meed_arad=Falae)
40
xnin.d
42
broadcast variahles to correct size
43
broadiest_ sctpsstttccllrd, shape shape)
44
baoadoast axoadczat_pcaale tmaax, shape)
46
47
asply fixed-polas quant ization F.zound (t.clip__pp by
48
18

---

Published as a conference paper at ICLR 2020
B.2 POWER-OF-TWO QUANTIZATION
B.2.1 CASE Pl: PARAMETRIZATION WITH RESPECT TO AND TMAX
del pasmetrrig..oooogueeeee xnax (4, aign, wich_zero, iintt, init, miexmni, smaxunee, aesaaannteraaaaaia "*"Paranttric version of "por2_quancize Khoze the bitwidth range
leatnable paranelecs.
Returns:
"nnebla. vunable.Verablle:
10
...
scip_scalar(y, min_value, nax_value);
12
rebuen E.mininan (F. i.naaiinnnsceaas ninuvaluw), nan_valus)
13
14
shape!:
15
reburn beoodoest (P. to, (1)) lentalape), ImplacenFaleel, shapeeshapp)
16
17
def www..ppeeeeeee.. (v):
18
P.roundiP. (Fg (2.10g(2.1)
20
pr.pppaaaerr.o...eee.. ("n", (),
21
an Initialiei (Linnil), gradeTrue, ax_nnedd_gradend fix.parmmtters)
24
s0ax gel wawaaateeeeeeeeee create (*anax", (),
25
Constant lnitialiteeeeeeee init), gredggradsTrue,
as_need piapparanetersl
28
29
нпxосH that bitwidth is specifled range and an inceges F.cound(alipp cooomntclipp_ooseerr Punin, n_maxi1 sign: ==n
32
33
it
36
ensuce that dynanic zange is in specicied rarge and an integer Ixnax, xnax,nir, nex))
37
quant ise pou2 (el
comрute nia value that we can represent xnin 12 44 1- as n) F. 11) saax
40
41
variables
broedcast acs soslan benin, shape) xrax brosscest_scalleecc www.eriiieeeee. chapo-x.caapel
44
45
unsigned, quantize nogative velues to zoro it not sign:
47
48
(xrelu(x)
19
compute absolute value/sign et input
*x F.abs|xk
52
(.
53
54
55
prune snallest clenets (in magnitude) to zero it they are smaller than min Neget (2)
56
57
K_threshoid
idx:
F.grester, equal x thresholsd; Fyless tax, xmin)
60
1dx2
s.croater_equate xnini F. (2X, xnax)
1dx3
F.groster_oqutil
63
1dx:
(ex, xnini
64
10x2
Sonater_egualine xnini 2.less xnax]
r.cratter_quulld
67
not beckpropogste gradient through indices
68
idx1.naed daxl_naed_rra False
69
neod__radd
neod_geed_
21
72
not bacckpooagate gradiant through sige sx.need_grad Palse
75
cake caze of velues putsida of dynanic rarge
76
zecurn Denin idx1 quantize_ 1dx2 xnax 1dx3)
19

---

guantize laxi Published as a conference paper at ICLR 2020
B.2.2 CASE P2: PARAMETRIZATION WITH RESPECT TO AND JMIN
poranetris pOw2_ quantize. Xman (N, sigh, zeth_
init, mim, nex, inin,init, xhir_nin, xnia_nex, tty.paraneter==oolse) "poo2_qquane quancize" sitxicth 'n' and the smollest value 'minn are lezrnabic parameters.
Paraanttrii
Returns:
nnabbl, Veriahio: arrey.
10
I1
def dl ip coalarite bs, htw.:mnm.roorr..aoooooeeeeoooo moximmmn.cmmlor( min_valuo), nax_velue)
nox valuee
12
return wninimmm_ccaaan
13
14
sot brasdiest scclar shapsi:
15
return b.broddcastt (F.ceshape Wy (1)) ienisarpo), inplace-Falset, chape-shape)
16
18
return ** CroonsiE,LogiE.aat / 1p.11112.11
19
20
gct_parameter__o prr_errtte""" ("a", (),
sonsantiittaalieerrr.ttttt
22
necLggadd-Teus,
23
as_noed_grad cix_parametorsi
24
wwt__parmmter_rom www.orrrrreeeeeeeeeeeeeee ("mmin"
need_grad Ieea,
27
grad-not pirpperattters
28
ensere Fitwisth is speciried range an inceger
30
= P. 2.sound(c2ig Runia, ([nax)]
31
1t sign:
32
sach
34
35
36
mininur. dynanic is specified and power-of-tao quantizc pow2 (clap_ocalar tmiin xnia_mir, xmir_nax))
38
39
compute nia/max value that we can sopresent
40
42
broadcast variahles to correct size
43
Mrin bracdccst__cclkee omni, shape aepeem..caaa caape) broadoast
44
shape)
15 unxigned, then quant ine all negative values to Zero 46
47
if not sign:
48
F.relu(K)
50
conpute absolute value/sign of input F.absix)
51
F.siign(s)
54
12 xich_aoro:
55
prune snallwat wlementa (in magnilude) aecu 1C Ubey smalle lwz Aaget
"_theeshald xnin op.agrt (2)
58
59
idx:
P.croates_egual (ax, k-theeshold) P.lesa (axy smin) snax)
P.grester
1dx3
F.groales fax, snax!
62
else:
63
idx1
(axs seint regeeater_euuee
1dx2
F.
18x3
F.grester egual fax, xnaxl
66
67
do not Sackpropagate gradient Chrough indiows .nedd grad
ids2.newd_gradd
70
idx3, nradd Falae
71
backpropagete gradient through sige. ex.need_gred Felse
73
74
75
Lake of veluwa out wide of dynamic
20

---

Published as a conference paper at ICLR 2020
B.2.3 CASE P3: PARAMETRIZATION WITH RESPECT TO JMIN AND îMAX
poranetric quantizo & eign, kith_zero, min init, smir nin, mex, mex_innt,, max,nnn, xnax_nex, sixpparoeeots-allls 'pos2 quancize' min value "xmin" and max value "xnxx" are learrable porametors.
"**aranetrii
Returns:
-nnbble, Veriahio: arcey.
10
I1
def dl ip sos scalar bs, return wwinimms_ccaaase mmxximun_ccciao(( min_valuo), nax_value)
nax value)
12
13
14
sot braadzest scclar shapsi:
15
return b.braddcast (Feceshape iy (1y) lenisaspo), inplace-Falset, chape
16
18
return 2, ** F. Fuzound (Fg.g (F / rp.logi2.))
19
20
gct_ or_crretteeeeeeooooooooooooeo ('mmii""
22
nead_arad-Teea,
23
ac_necd_grad eixpporametors!
24
("xmax",
Consraciitttaalloocccooeo (xnaX_int)))
necd_grad Ieeo,
27
ao_need_grad-mot por.marattt
28
ensere
sycanis zange specified range power-ot-cxo
30
sain quant pov2 aclip_acabar tanin, aniaumin, xmin_nax))
31
32
ensure that minimun synants is specified and power-of-txo
quant pouz aclap_ txnax, anax_mir, xmxx_nax)
34
35
broadcast veriaties to correct cize
36
broadoist brocdcest_pcaale scaloo tmox, shape s.shspe)
38
39
12 unsigned, then quancize all negative values to zezo sign:
40
F.zelu(x)
42
43
compute absolute value/sign et input F.absis)
44
(.sign(x)
46
47
1E with_xero:
48
snallest eleniat nagnitude)
smaller
Aaget (2) "_theeshoid anin / 12)
50
51
idxi
P.greates_
s_threshold) P.less snin)
1dx2
P.ereates_egual anint F.lesa (ax, anax)
54
1dx3
F. P.creater_equllce rax, anax!
55
llees
idx:
P.less
anint
18x2
P.greeter equell fax, snint V.lens snax]
58
18x3
P.geente: tax, soaxt
59
beckpropagete gradient through indi pred_grad idx2. naod_orad False
62
63
ids3.newd_graat Falxs
not backpropagste gradient through sigr nxedeed_ Falme
66
67
dynanic rarge
secuen (enán guent _ee (ax]
xnex
21


Reviews:
[{"review_id": "ryl5_mkRKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of training mixed-precision models. \nSince quantization involves non-differentiable operations, this paper discusses how to use the straight-through estimator to estimate the gradients,  and how different parameterizations of the quantized DNN affect the optimization process. The authors conclude that using the parameterization wrt the stepsize d and quantization range q_max has the best performance.\n\nIn the discussion for the three parameterization choices in section 2.1. \nIt is not clear how the range for U2 is obtained. Given d an integer, the gradient wrt b is also bounded. In this case, why is case U3 better than U2? In Table 1, it is shown that U2 also has good performance for uniform quantization.\n\nIndeed, the gradient of any of the three parameters (stepsize, bitwidth and quantization range) can be derived by using chain rule given the gradients of the other two. It is not clear to me why some of them can be unbounded while others do not. In addition, It is not clear to me why having different gradient scales is a big problem. Adaptive learning rate methods like Adam should be able to help deal with the different scale of the gradients for three parameters. Can the authors compare the three parameterizations using Adam and see if similar empirical results can still be observed.\n\nAt the end of Section 2.1, the authors said that \"similar considerations can be made for power-of-two quantization\".  However, from table 1, these three parameterizations indeed have quite different performances for uniform and power-of-two quantization.  E.g., for uniform quantization, U2 and U3 perform significantly better than U1, while for power-of-two quantization, U1 and U3 perform significantly better than U2. Can the authors elaborate more on the difference?\n\nIs the proposed differential quantization method used for both weight and activation? If so, how are the gradients w.r.t. the weights propagated through the quantized activations?\n\n---------- post-rebuttal comment -------------\nI thank the authors for their detailed response. It has solved most of my concerns and I accordingly raised my score.\n---------------------------------------------------------", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxa1xZHcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose learning a quantizer for mixed precision DNNs. They do so using a suitable parameterization on the quantizer's step size and dynamic range using gradient descent, and where the quantizer's bitwidth are inferred from the former two rather than also learned jointly.\n\nAs a non-expert in the field, I found the paper well-written and interesting in their analysis of their proposed parameterization. They explain well how quantizers work, and the intuition and relationships of the parameters behind two popular types of quantizers: uniform and power-of-two. Equation (3) is especially explicit in understanding how the choice of 2 of the 3 parameters makes an impact on the choice of gradients. My understanding is that this is the core contribution.\n\nNovelty-wise, I don't have enough background to tell if this is much of a leap from related work that has already proposed learning certain parameters of quantizers (but different parameters, or not the exact 2 proposed by the authors). I do like the discussion of related quantizer literature noted in the introduction.\n\nI don't know if there is already previous work in the paper's follow-up section of learning quantized DNN under a constraint involving maximum total memory, total activation memory, and maximum activation memory. The solution of a Laplace multiplier seems fairly naive and hard to work in practice as it is not a hard constraint. As a naive question, how does the scale of these values compare to the original loss function? For example, if we think of the original loss function as a negative log-likelihood which computes bits/example, does it make sense to add a constraint penalty in kB as in the experiments, which is a completely different unit scale? Do you also backpropagate through the constraint function g?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJehYBRCFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work studies differentiable quantization of deep neural networks with straight-through gradient (Bengio et. al., 2013). The authors find that a proper parametrization of the quantizer is critical to stable training and good quantization performance and demonstrated their findings to obtain mixed precision DNNs on two datasets,  i.e., CIFAR-10 and Imagenet.\n\nThe paper is clearly written and easy to follow. The idea proposed is fairly straight-forward. Although the argument the authors used to support the finding is not very rigorous, the finding itself is still worth noting. \n\nOne of the arguments that the authors used to support the specific form of parametrization is that it leads to diagonal Hessian. From optimization perspective, what matters is the condition number, i.e., max/min of the eigenvalues of the Hessian. Could this explain the small difference between the three different parametrization forms with uniform quantization and the big difference for power-of-two quantization? \n\nThe penalty method used to address the memory constraints will not necessarily lead to solutions that satisfy the constraints. The authors noted that the algorithm is not sensitive to the choice of the penalty parameters. Have the authors tried to tackle problems of hard memory constraints?\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers uniformly vote to accept this paper. Please take comments into account when revising for the camera ready. I was also very impressed by the authors' responsiveness to reviewer comments, putting in additional work after submission.
Decision: Accept (Poster)