OPTIMAL BINARY QUANTIZATION FOR DEEP NEURAL NETWORKS
Authors: Hadi Pouransari
Oncel Tuzel
Venue:  (2020)
Keywords: Binary Neural Networks
Quantization

Abstract: Quantizing weights and activations of deep neural networks results in significant improvement in inference efficiency at the cost of lower accuracy. A source of the accuracy gap between full precision and quantized models is the quantization error.
In this work, we focus on the binary quantization, in which values are mapped to -1 and 1. We introduce several novel quantization algorithms: optimal 2-bits, optimal ternary, and greedy. Our quantization algorithms can be implemented efficiently on the hardware using bitwise operations. We present proofs to show that our proposed methods are optimal, and also provide empirical error analysis. We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed optimal quantization algorithms.

Under review as a conference paper at ICLR 2020
OPTIMAL BINARY QUANTIZATION FOR DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Quantizing weights and activations of deep neural networks results in significant improvement inference efficiency at the cost of lower accuracy. source the accuracy gap between full precision and quantized models is the quantization error. In this work, we focus on the binary quantization, in which values are mapped to -1 and 1. We introduce several novel quantization algorithms: optimal 1-bit, ternary, 2-bits, and greedy. Our quantization algorithms can be implemented efficiently the hardware using bitwise operations. We present proofs to show that our proposed methods are optimal, and also provide empirical error analysis. We conduct experiments on the ImageNet dataset and show reduced accuracy gap when using the proposed optimal quantization algorithms.
INTRODUCTION
A major challenge in the deployment of Deep Neural Networks (DNNs) is their high computational cost. Finding effective methods to improve run-time efficiency is still an area of research. We can group various approaches taken by researchers into the following three categories.
Specifically designed hardwares are deployed to efficiently perform com- putations in ML tasks. Compiler optimization: Compression and fusion techniques coupled with efficient hardware-aware implementations, such as dense and sparse matrix-vector multiplication, are used. Model optimization: Run-time performance can also be gained by modifying the model structure and the underlying arithmetic operations. While hardware and compiler optimization are typically lossless (i.e. incur no loss in model accuracy), model optimization trades-off computa- tional cost (memory, runtime, or power) for model accuracy. For example, by scaling the width of the network (Zagoruyko & Komodakis 2016). The goal of model optimization is improve the trade-off between computational cost and model accuracy. This work falls into this category.
Hardware optimization:
1.1 ARCHITECTURE OPTIMIZATION
One strategy to construct efficient DNNs is to define template from which efficient computational blocks can be generated. Multiple instantiations of these blocks are then chained together to form
DNN. SqueezeNet (landola et al. 2016), MobileNets (Howard et al. 2017 Sandler et al. 2018)
ShuffleNets (Zhang |al.b2008 Ma al. 2018), and ESPNets (Mehta et al. 2018 2019) fall into this category. Complementary to these methods, NASNet (oph et al.) 2018)) and EfficintNeet (Tan 2019) search for an optimal composition blocks restricted to computational budget (e.g., FLOPS) by changing the resolution, depth, width, or other parameters of each layer.
1.2 PRUNING AND COMPRESSION
Several methods have been proposed to improve runtime performance by detecting and removing computational redundancy. Methods in this category include low-rank acceleration (Jaderberg et al., 2014), the use of depth-wise convolution in Inception (Szegedy al. 2015), sparsification of kernels in deep compression (Han et al. 2015), re-training redundant neurons DSD (Han et al. 2016b), depth-wise separable convolution in Xception (Chollet 2017). pruning redundant filters in PFA (Suau et al.) 2018), finding an optimal sub-network in lottery ticket hypothesis (Frankle Carbin 2018), and separating channels based the features resolution in octave convolution (Chen et al.

---

Under review as a conference paper at ICLR 2020
2019). While some of these compression methods can be applied to trained network, most add taming-time time constraints to create a computationally efficient model.
training
1.3
LOW-PRECISION ARITHMETIC AND QUANTIZATION
Another avenue improve runtime performance (and the focus of this work) is to use low-precision arithmetic. The idea is to use fewer bits to represent weights and activations. Some instances of these strategies already exist in Al compilers, where it is common to cast weights a trained model from 32 bits to 16 or 8 bits. However, in general, post-training quantization reduces the model accuracy. This can addressed by incorporating lower-precision arithmetic into the training process fluring-training quantization), allowing the resulting model to better adapt to the lower precision. For example, Gupta et al. (2015); Jacob et al. (2018) the authors use 16 and 8 bits fixed-poine representation train DNNs.
Using fewer bits results in dramatic memory savings. This has motivated research into methods that single bit to represent scalar weight: In Courbariaux et al. (2015) the authors train models with weights quantized to the values in 1-1,1 1. While this results in high level of compression, model accuracy drop significantly. Li et al. (2066) (uu et al. (2016) reduce the accuracy gap between full precision and quantized models by considering ternary quantization (using the values in {-1, 1)), at the cost of slightly less compression.
To further improve the computational efficiency, the intermediate activation tensors (feature maps) can also be quantized. When this is the case, an implementation can use high-performance operators that act on quantized inputs, for example convolutional block depicted in Figure left). This idea has been explored (Courbariaux
al.)
2016; Rastegari
al. 2016 Zhou et al., 2016 Hubara
et al. et al.
2017, Mishra et al. 2017] (0017:Lin et al.
2017 Cai et al. 2017 Ghasemzadeh et al. 2018 7Zhan
2018a Choi et al. 2018), and many other works.
We call mapping from tensor with full precision entries to tensor with the same shape but with values {-1, 1} a binary quantization. When both weights and activations of a DNN are quantized using binary quantization, called Binary Neural Network (BNN), fast and power-efficient kernels which use bitwise operations can be implemented. Observe that the inner proeuttr between two vectors with entries in {-1,1} }} can be written bitwise XNor operations followed by bit- counting (Courbariaux et al., 2016). However, the quantization of both weights and activations further reduces the model accuracy. In this work, focus on improving the accuracy of the quan- tized model through improved quantization. The computational cost remains similar to the previous BNNs RRastegari al. 2016; Hubara al. (017).
W
ReLU
BN
Clip
W
KiNor Conv
+1-1-141
Quantizer
Quantizer
Nmm
Canv
+1-1---
XNor
Conv
Figure 1: Left: The convolutional block used in this paper. Right: When both weights and activa- tions quantized using binary quantization, the convolution can be implemented efficiently using bitwise XNor and bit-counting operations. See Section 3.2 for more details.
1.4 MAIN CONTRIBUTIONS
In this work, we analyze the accuracy of binary quantization when applied to both weights and activations of a DNN, and propose methods to improve the quantization accuracy:
We present an analysis of the quantization error and show that scaled binary quantization is good approximation (Section
2

---

Under review as a conference paper at ICLR 2020
We derive the optimal 1-bit (Section 3.1.1. 2-bits (Section 3.2.2). and ternary (Section
3.2.3 scaled binary quantization algorithms.
We propose a greedy k-bits quantization algorithm (Section 3.2.4).
Experiments the ImageNet dataset show that the optimal algorithms have reduced quan- tization error, and lead to improved classification accuracy (Section 5)
2 LOW-RANK BINARY QUANTIZATION
Binary quantization (that maps entries of a tensor to (-1, of weights and activation tensors of a neural network can significantly reduce the model accuracy. A remedy to retrieve this accuracy loss to scale the binarized tensors with few full precision values. For example, [uubar et al. (2017) learn scaling for each channel from the parameters of batch-normalizatioo, and Rastegari et al. (016) scale the quantized activation tensors using the channel-wise average of pixel values.
In this section, using low-rank matrix analysis, we analyze different scaling strategies. We conclude that multiplying the quantized tensor by single scalar, which is computationally the most efficient option, has approximately the same accuracy the more expensive alternatives.
We introduce the rank-1 binary quantization- an approximation to a matrix X E Rmxx:
X x 1105. S
(1)
where X, E R is a rank-1 matrix, S (-1, 1)mxx, and @ is element-wise multiplication (Hadamard product). Note that this approximation is also defined for tensors, after appropriate reshaping. For example, for an image classification task, we can reshape the output of layer of DNN with shape h X X n, where h, w, and are height, width, and number of channels, respectively, into an m X matrix with m = hw rows and one column per channel.
We define the error of rank-1 binary quantization as 1X-Xx Slle, where || || is the Frobenius norm. Entries of S are in (-1, 13. therefore, the quantization error is equal to IXoS-XX x1 Note that || @ 3 (the total energy). which is equal to sum of the squared singular values, is the same for any S. Different choices of S change the distribution of the total energy among components of the Singular Value Decomposition (SVD) of X @ S. The optimal rank-1 binary quantization achieved when most of the energy of X @ S is in its first component.
In Rastegari et al. 2016), the authors proposed to quantize the activations by applying the sign function and scale them by their channel-wise average. We can formulate this scaling strategy special rank- binary quantization X ~ al @ sign(X), where
ΣE= X for 1sis sign(x) f-1
if i
(2)
and 1 is an n--dimensional vector with all entries 1.
In Appendix A we show that the optimal rank-1 binary quantization is given by S sign(X) X) and X = (runcatdd--SVD(1 truncated, -SV)((( = L is the first component the SVD of X X sign(X)) | More- over, we empirically analyze the accuracy of the optimal rank-1 binary quantization for a random matrix X, where its entries are i.i.d. ~ (0, 1). This is a relevant example since after the application of Batch Normalization (BN) (loffe & Szegedy 20155) activation tensors are expected to have simi- lar distribution. The first singular value of X captures most of the energy of (| X I ~ 0.64, and the first left and right singular vectors
where sign(X) is the element-wiss wise sign of X, and
almost constant vectors. Therefore, scalar multiple of sign( X) approximates X well: X ~ a o o, & sign( X) z v 11 o sign(X = vsign(X). where Rz0- We use this computationally efficient approximation called scaled binary quantization.
SCALED BINARY QUANTIZATION
In Section showed that scaled binary quantization is good approximation to activation and weight tensors a DNN. Next we show how we can further improve the accuracy of scaled binary quantization using bits. To simplify the presentation (1) we flatten matrix X E IR in to
3

---

Under review as a conference paper at ICLR 2020
vector RN with N mn, and (2) assume the entries of are different realizations of a random variable x with am underlying probability distribution p(z). In practice, we compute all statistics using their unbiased estimators from vector (e.g.. Σ, I; is unbiased estimator of Ex)p(x). Furthermore, for f R + R, we denote entrywise application of to a: by f(x)- The quantized approximation of a: is denoted by æ², and the error (loss) of quantization is | x=I22 All optimal solutions are with respect to this error and hold for an arbitrary distribution p(æ).
3.1
1-BIT QUANTIZATION
I-bit scaled binary quantization is:
x = x = vs()),
(3)
which is determined by scalar E R R20 and function s: IR + (-1,1). Finding the optimal 1-bit scaled binary quantization can be formulated as the following optimization problem:
minimize 6,5
p(x)(us(x)) -x)
(4)
s.t.
IR + (-1,),, R22
3.1.1 OPTIMAL 1-BIT ALGORITHM
The solution of problem is given by v = Expp [x]] and s(x) = sign(x) (for the proofs see Appendix B). Therefore, for vector the optimal scaled binary quantization is given by
Σ, [rg] N
sign(x))
(5)
where (aal is an unbiased estimator of Ex<p[[x]]-
3.2 B-BISS QUANTIZATION
We can further improve the accuracy of scaled binary quantization by adding more terms to the approximation 3 k-bits scaled binary quantization of æ is
v,,((x),
(6)
which is determined by a set lo pairs of scalars vs's and functions s; : R - (-1,1). Observe that any permutation of (vi, s:)'s results in the same quantization. To remove ambiguity, assume Vi 2 2 Uk 2 O.
When both weights, w, and activations, æ, quantized using scaled binary quantization (6) their inner-product can be written as:
(
(8 87)
where ": = Ek- 42
of and :" = Σド
R. u. 8. are quantized activations and weights with ke and
tsse bits, respectively, Se = of (z), and 5/ = 5 (w). This imner-produc can be computed efficiently using bitwise XNors followed by bit-counting (see Figure (right with ke 2 and htd 1).
Finding the optimal k-bits scaled binary quantization can be formulated as:
k Σ
2
minimize
p(z)
(1))(a))
dx
(8)
S.I. 11 Visisk k
8, R (-1,1},
V1
2 20
This is optimization problem with nonvec domain for all & 1. We solve the optimization for & = 1 in Section 3.1 and & = 2 in Section 3.2.2 for arbitrary distribution p(x). We also provide approximate solution B in Section 3.2.4 using greedy algorithm.
4

---

Under review as a conference paper at ICLR 2020
Discussion: A general k--bit quantizer maps full precision values to arbitrary set of 2 numbers, not necessarily in the form of (6). The optimal quantization in this case can be computed using the Lloyd's algorithm [Lloyd 1982). While a general b-bits quantization has more representation power compared to k-bits scaled binary quantization, it does not allow an efficient implementation based on bitwise operations. Fixed-point representation (as opposed to floating point) is also in the form of (6) with an additional constant term. However, fixed-point quantization uniformly quantizes the space, therefore, can be significantly inaccurate for small values of k.
3.2.1 FOLDABLE QUANTIZATION
In this section, we introduce special family of k-bits scaled binary quantizations that allow fast computation of the quantized values. We name this family of quantizations foldable. k-bits scaled binary quantization given by (0%, st'ss is foldable if the following conditions are satisfied:
(x sign(x x Σ (x)) for 1 S sisk k
When the foldable condition is satisfied, given u/ s, we can compute the (;(z))ss in (6) efficiently by applying the sign function.
3.2.2 OPTIMAL 2-BITS ALGORITIIM
In this section, present the optimal 2-bits binary quantization algorithm, the solution of (8) for 2. In Appendix Ce we show that the optimal 2-bits binary quantization is foldable and the scalars
2 and U2 should satisfy the following optimality conditions:
U1 =
(E (E.p p
|x||
|
en + Exp |X|
|x| S v)))
(10)
U/2 (Exxp
|x|
|x|
> 01 - Ex-pl==l | S vil)
(11)
In Figure 2 we visualize the conditional expectations that show up in 10 for a random variable x with standard normal distribution. The optimal 21 lies the intersection of the identity line and average of the conditional expectations in (TO).
For given vector E R.V we solve for in 100) efficiently. We substitute the conditional
expectations in (T0) by conditional average operators as their unbiased estimators. (TO) implies that
for the optimal U1 the average of the entries in |ax] smaller than U1 (an estimator of [xx=p[[|] | [x s
vi] ) and the average of the entries greater than ei (an estimator of [Ex=pIXx [|x] |x] > vi() should be equidistant form U1. Note that (10) may have more than one solution, which are local minima of the objective function in (8). We find all the values that satisfy this condition in O log.N) N) time. We first sort entries of based on their absolute value and compute their cumulative sum. Then with one pass we can check whether (T)) is satisfied for each element of a. We evaluate the objective function in (8) for each local minima, and retain the best. After "1 calculated 02 is simply computed from III As explained in Section 4 this process is only done during the training. In our experiments, finding the optimal 2-bits quantization increased the training time by 35% compared to the 2-bits greedy algorithm (see Section 3.2.4). Sine the optimal 2-bits binary quantization is foldable, after recovering i1 and 222 we have $1 (x) = =sign(>) and s2 (2 = sign(x U1 Uzgig)x:)).
3.2.3 OPTIMAL TERNARY ALGORITHM
The optimization domain of g for k over the scalars is illustrated in Figure righht) The boundaries of the domain, 1/2 = 0 and tj = 02 = correspond to 1-bit binary and ternary (Li al, 2016) quantizations, respectively. The scaled ternary quantization maps each full precision value to 2v.00 0, 2r). Ternary quantization needs 2-bits for representation. However, when a hardware with sparse calculation support is available, for example as in EIE (Han et al.) (0T6a), using ternary quantization can be more efficient compared to general 2-bits quantization. In Appendix D we show that the optimal scaled ternary quantization is foldable and the scalar v should satisfy:
Ex-plle [x|
|x|
>
(12)
The process of solving for in 12) is similar to that of solving for en in (1)p described above.
5

---

Under review as a conference paper at ICLR 2020
Fy [[[H]
M <
B. = B|K b
Vi Terrary Quarrtiattion
(R) F.//2
2
1-
Pyp 1/2 12 bits Ouantization)
Py Quantization)
0.0
0.5
1.0
1.5
2.0
25
F
Figure 2: Left: The conditional expectations in (10) for random variable x with standard normal distribution. The optimal value for 2-bits quantization is shown with a solid dot. Right: Optimiza- tion domain of (8) for k=2. The boundaries correspond 1-bit and ternary quantizations.
3.2.4 k-BITS GREEDY ALGORITIM
In this section, we propose a greedy algorithm to compute k-bits scaled binary quantization, which call Greedy Foldable (GF). given in Algorithm
Algorithm 1: k-bits Greedy Foldable (GF) binary quantization: compute x given <
for f-1 to do 0 + mean(abs(r))
8; sign(r) // element-wise sign. For gradient sign use STE.
r +- r U.8, // compute new residual.
end
return æ
In GF algorithm we compute sequence of residuals. At each step, we greedily find the best and o for the current residual using the optimal 1-bit binary quantization (5). Note that for & the GF is the same as the optimal 1-bit binary quantization.
Few of the other papers that have tackled the k-bits binary quantization train quantized DNNs are as follows. In ReBNet (Ghasemzadeh et al., 2018), the authors proposed algorithm similar to Algorithm [] but considered v as trainable parameters to learned by pack-propagation. [in al. (2017) and Zhang et al. (2018a) find k-bits binary quantization via alternating optimization for st and 0. s. Note that, all these methods produce sub-optimal solutions.
TRAINING BINARY NETWORKS
The loss functions in our quantized neural networks are non-differentiable due the sign function in the quantizers. To address this challenge we use the training algorithm proposed in Courbariaux jet al. (0155). To compute the gradient of the sign function we use the Straight Through Estimator (STE) (Bengio al. 2013): d/dz sign(x) | During the training we keep the full precision weights and use Stochastic Gradient Descent (SGD) to gradually update them in back mack-propagation. In the forward- -pass, only the quantized weights used.
During the training we compute quantizers (for both weights and activations) using the online statis- tics, i.e., the scalars in k-bits scaled binary quantization O are computed based the observed values. During the training we also store the running average these scalars. During inference we the stored quantized scalars to improve the efficiency. This procedure is similar the update of the batch normalization parameters in a standard DNN training (loffe & Szegedy 2015).
5 EXPERIMENTS
We conduct experiments on the ImageNet dataset (Deng et al., 2009) using the ResNet-18 architec- ture (He et al. 2016). The details of the architecture and training provided in Appendix

---

Under review as a conference paper at ICLR 2020
We conduct three sets of experiments: (I) evaluate quantization error of activations of pre- traitrained DNN, (2) evaluate the quantization error based on the classification accuracy of post-training quan- tized network, and (3) evaluate the classification accuracy of suring-traiinigg quantized networks. We report the quantization errors of the proposed binary quantization algorithms (optimal 1-bit, 2- bits, ternary, and the greedy foldable quantizations) and compare with the state-of-the-art algorithms BWN-Net (Rastegari et al. 2016)) XNor-Net (Rastegari et al.) 2016). TWN-Net (Li et al. 2016), DoReFa-Net (Zhou al. 2016), ABC-Net (Lin et al. 2017), and LQ-Net (Zhang 2018a).
5.1 QUANTIZATION ERROR OF ACTIVATIONS
To quantify the errors of the introduced binary quantization algorithms we adopt the analysis per- formed by Anderson & Berg (2017). They show that the angle between æ and æ can be used as measure of the accuracy of a quantization scheme. They prove that when x = sign(æ) and elements of x are i.i.d. ~ (0, 1), Z(x,, æ²) converges to ~ 37 degrees for large N.
Here use the real data distribution. We trained a full precision network. We compute the activa- tion tensors each layer for set of 128 images. In Figure B we show the angle between the full precision and quantized activations for different layers. When the optimal quantization is used, significant reduction in the angle observed compared to the greedy algorithm. The optimal 2-bits quantization even better than the greedy 4-bits quantization for later layers of the network, for which activation tensors have more skewed distribution, make it harder for quantization form of (6). Furthermore, the accuracy of the optimal quantization has less variance with respect different input images and different layers of the network.
70
Method
ka
k
Top-1
Top-5
60-
Post-GF
32
0.1
0.5
Post-GF
32
0.3
1.1
Post-GF
32
1.4
4.6
Post-GF
32
5.3
14.1
Post-Opt
32
5.3
13.9
Opt
54.3
77.3
30
GF
2
59.7
81.7
GF
3
61.1
82.7
20-
GF
4
61.3
82.8
Opt
T
58.3
80.6
10-
SF/Opt-1
GF-3
Opt-2
Opt
2
60.4
82.2
GF-2
GF-4
FP
32
32
69.6
89.2
2
4
10
12
16
layer index
Table 1: Validation accuracy of quantized ResNet-18 trained on ImageNet. k and k are number of bits to quantize activations and weights, respectively. T, Opt, GF, and FP refer to ternary, optimal, Greedy Fold- able, and full precision, respectively.
Figure 3: The angle between the full precision and the quantized activations for different layers of trained full precision ResNet- architecture on ImageNet. The 95% confidence interval over different imput images is shown.
5.2 POST-TRAINING QUANTIZATION
In this section we apply post- quantization to the weights of pre-trained full precision network. We then use the quantized network for inference and report the classification accuracy. This procedure can result in an acceptable accuracy for a moderate number of bits (e.g., 16 or 8). However, the error significantly grows with a lower number of bits, which is the case in this experiment. Therefore, we only care about the relative differences between different quantization strategies. This experiment demonstrates the effect of quantization errors the accuracy of the quantized DNNs.
The results shown in the top half of Table I When the optimal 2-bits quantization is used, significant accuracy improvement (more than one order of magnitude) is observed compared to the greedy 2-bits quantization, which illustrate the effectiveness of the optimal quantization.

---

Under review as a conference paper at ICLR 2020
5.3 DURING-TRAINING QUANTIZATION
To achieve higher accuracy we apply quantization during the training, so that the model can adapt to the quantized weights and activations. In the bottom half of TT,le we report the accuracies of the during-training quantized DNNs, all trained with the same setup. We use 1-bit binary quantization for weights, and use different quantization algorithms for activations. When quantization is applied during-training, significantly higher accuracies are achieved. Similar to the previous experiments the optimal quantization algorithm achieves better accuracy compared to the greedy.
In Table 2 we report results from the related works in which ResNet- 18 architecture with quantized weights and/or activations is trained on the ImageNet dataset for the classification task. We report the mean and standard deviation of the model accuracy over runs when our algorithms are used. Note that for 1-bit quantization the Greedy Foldable (GF) algorithm is the same with the optimal 1-bit binary quantization. In Opt* we used 2 larger batch-size compared to Opt but with the same number of optimization steps. As shown in the Table 2 the proposed quantization algorithms match or improve the accuracies of the state-of-the-art BNNs.
Method (ResNet- 18 on ImageNet)
kn
hnd
Val.
top-1
Val. top-5
XNor-Net (Rastegari et al. 2016)
1
51.2
73.2
Opt
1
54.3
0.2
77.4 + 0.2
Opt
T
58.3
0.1
80.6 + 0.0
DoReFa-Net (Zhou et al. 2016
1
53.4
LQ-Net (Zhang et al. 201 8a
2
1
62.6
84.3
HWGQ (Cai et al.
2017
2
1
59.6
82.2
GF
2
59.7 = 0.2
81.7 + 0.1
Opt
2
60.4 0.2
82.2 + 0.1
Opt#
62.4 0.1
83.6 + 0.1
GF
3
61.1 0.1
82.7 #.1 0.1
NeC-Net (Lin et al. (017)
61.0
83.2
DoReFa-Net (Zhou et al. 2016
59.2
81.5
GF
4
61.3 = 0.2
82.8 + 0.1
BeN-Net (Rategaari et al. 2016)
32 1
60.8
83.0
Opt
32
1
64.2
0.4
85.2 + 0.1
TWN-Net (Li et al. 2016)
32
T
61.8
84.2
Opt
32
T
64.4
0.1
85.4 + 0.1
FP
32
32
69.6
89.2
Table 2: Comparison with state-of-the-art quantization. Opt and GF are the proposed optimal and greedy foldable quantization algorithms, respectively. T and FP refer to ternary and full precision network, respectively.
"This result taken from (Zhang 2018a).
CONCLUSION
In this work, we analyze the accuracy of binary quantization train DNNs with quantized weights and activations. We discuss methods to improve the accuracy of quantization, namely scaling and using more bits.
We introduce the rank-1 binary quantization, as a general scaling scheme. Based on singular value analysis we motivate using the scaled binary quantization, a computationally efficient scaling strategy. We define a general k-bits scaled binary quantization. We provide provably optimal 1-bit, addition, we propose a greedy k-bits quantization algorithm.
2-bits, and ternary quantizations. We show results for post and during-training quantization, and demonstrate significant improvement in accuracy when optimal quantization is used. We compare the proposed quantization algorithms with state-of-the-an BNNs the ImageNet dataset and show improved classification accuracies.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander G Anderson and Cory P Berg. The high-dimensional geometry of binary neural networks. arXiv preprint arXiv: 1705.07199, 2017.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 13X.ii:300..3430 2013
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half- wave gaussian quantization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5918-5926, 2017.
Yunpeng Chen, Haoqi Fang, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. arXiv preprint 1904v:/904.05000 2019.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint 1807 069.4.694 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 1251-1258, 2017.
Matthieu Courbariaux, Yoshua Bengio, and Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: 162.:0/00..02300 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803:0880033333 2018
Mohammad Ghasemzadeh, Mohammad Samragh, and Farinaz Koushanfar. Rebnet: Residual binarized neural network. In 2018 IEEE 26th Annual International Symposium on Field- Programmable Custom Computing Machines (FCCM), pp. 57-64. IEEE, 2018
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint /rXiv: 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William Dally. Eie: efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243-254. IEEE, 2016a.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Va- jda, Manohar Paluri, John Tran, et al. Dsd: Dense-sparse-denss training for deep neural networks. arXiv preprint arXiv:1607 0003800........ 2016b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: ha0l:/774.0000 1704.04861, 2017.

---

Under review as a conference paper at ICLR 2020
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):6869-689)) 2017.
Forrest N landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. 2016.
arXiv preprint
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:/502.0316. 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-rithmmeti-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pp. 2704-2713, 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3888 2014.
Fengfu Li, Bo Zhang, and Bin Liu. Temary weight networks. arXiv preprint drXiv:1605.04711. 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions information theory, 28(2): 129-137, 1982.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 552-568, 2018.
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: light- weight, power efficient, and general purpose convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9190-9200, 2019.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: wide reduced-precision networks. arXiv preprint arXiv: 1at.000094.1444 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 522-542. Springer, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Xavier Suau, Luca Zappella, and Nicholas Apostoloff. Network compression using correlation analysis of layer responses. 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.1194. 2019.
Sergey Zagoruyko and Nikos Komodakis. 165.011665 2016.
Wide residual networks.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference Computer Viston (ECCV), pp. 365-382, 2018a.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6850 2018b.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1eXiv:600.06111 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.
Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.

---

Under review as a conference paper at ICLR 2020
A OPTIMAL RANK-1 BINARY QUANTIZATION
In this section, we find the optimal rank-1 binary quantization of an m by n matrix X that is discussed in Section 2
minimize X1.S
|X - X1 ② S
s.t.
S E {-1,1)mxn
(13)
X1 E Rm
rank(XX)
First, observe that the element-win wise multiplication by -1 and +1 does not change the Frobenius norm. Therefore:
min X S.X: X
F
min = || - X1 ④ O S.X:
min S.X1
X
c - -11le
(14)
Furthermore, note that
min XX11
X c = S) +
O S
(15)
Here (X @ S) is the i'th singular value of @ S and is its rank. In addition for any S:
Σ o S) = |X o S
X|
(16)
Hence, minimize the sum in 15 we need find an for which of S) is maximized:
min S,X;
X S 11 X max (X @ S)
(17)
s
01 o 5)= |X + S is the 2-norm of X C S. Therefore:
maxof X
5) max |||||==|
(X O S)rll2
(18)
For any S and r R² we have (X @ S)r
r I|2 since for S S m we have IΣ, Xigry < Σ, |Xiy lryl. Here |X| X @ sign X) is the element-wise absolute value
of X. Note that for S XXg and with positive values the inequality becomes equality. Therefore:
[XX
S)rl2²
max |r
X |r|l
(19)
|r]==
Observe that the deement-wise absolute value does not change the vector norm, i.e. ||| [2 = || I|2 and hence |r| is unit vector when is. Also for any have || 113
<
X | since for
1 < i < m we have Σ, |X,5|rs| < E, |Xrsllrrs- So we have
X 1 max max |||||==
X
(
(20)
11:=1
Therefore, we showed that sign(X (X) and X₁ equal to the best rank- approximation of X (i.e. the first term in its SVD) is a solution of 1
For an X with i.i.d. (O, 1) entries show the singular values and first left and right singular vectors of X in Figure Observe that the first singular value of X captures most of the energy 0.64. The fraction of energy captured by the first component of SVD converges to the squared mean of the standard folded normal distribution (2/Tr) for large square matrices. Also note that the first left and right singular vectors are almost constant, i.e., they can written as for some R.
| x
B OPTIMAL SCALED BINARY QUANTIZATION
In this section, we solve the following optimization problem corresponding to the optimal 1-bit scaled binary quantization discussed in Section 3.1
minimize 8,8
((=)((s(r) - 2r
(21)
s.t.
8 ARR- (-1,11
U E Rx0
12

---

Under review as a conference paper at ICLR 2020
1.0
0.6
1
a
x2( [
0.4
0.0
0.2
00.5
0.0
-1.0
10
20
30
10
20
30
Figure 4: Left: Distribution of energy for [X] where X IR
30x30 is standard normal random
matrix. Right: Entries of the first left and right singular vectors of X (shown in green and blue) almost constant.
Here p is probability distribution function. First, observe that:
Vx E R (v- - z²² < (-u x)² iff x: > 0
(22)
Therefore, the optimal choice for function s is s(x) = sign(x). So we can rewrite 21 as follows:
minimize
p(z)(v (x1)2do dx
(23)
S.t.
U E R20
Setting the gradient of the objective function in
23
with respect to to zero, get:
too p( |x
Ex<p[[x|]
(24)
1_ p(æ)dr
Hence, we showed that the optimal 1-bit scaled binary quantization maps x to Ex<p[[x|] sign(x).
C OPTIMAL 2-BITS BINARY QUANTIZATION
In this section, we solve the following optimization problem corresponding to the optimal 2-bits binary quantization discussed in Section 3.2.2
minimize 01,09,81.88
p(x) (1181(x) + (282(() x)² dr
(25)
s.t.
$1 82
(-1,11
02
First, we show that the optimal 2-bits binary quantization is foldable, i.e., Vo RR s1 sign(x) and 82 (z) = sign(æ - (z)). Observe that
$1(x)x) 2
f(x) = (1) + (282(x) x²²= = o7 1 (z()))2(1)
1/ U1
(26)
of
1 Up
(s1(z)
g(x)
11
The inequality in (6 holds because e1 2 02, and therefore, 1 1 (z))s((())) 2 The objective function in (25) a weighted average f(x) with non-negative weights. For r E R the inequality is strict if $1 (x) # sign(x)). In that case, flipping the value of both $1 (æ) and 52(x)) reduces f(x) to
a strictly smaller value ((r). Hence, the optimal solution of (25) should satisfy 8, (x) = sign(x) for all IR.
13

---

Under review as a conference paper at ICLR 2020
For any and 81 if we consider quantization for y. Based on the result showed in Appendix B for the optimal solution we have $2(2) sign(y) sign(z - U1 (x)). This completes the proof to show that the optimal 2-bits binary quantization is foldable.
U1 (x), the problem reduces to the 1-bit binary
Next, find the optimal values for U1 and U2. If we substitute (1 sign(x) and 82(2)
sign(x - (1s1 (x)) in 25 we can decompose I into four segments and write:
e(U1,
p(a) (1)81()) + (252(x) - x)² do
px + + u2) dx + p(z)(x + - -22) +
(27)
(( 1 + 2z + p(z)(x - 11 - 122)2)2 22)
(xx)(x- - + d + |
g(x)(xx 01 (
Here e(V), 02) is the error as a function of i₁ and u2 and g(x) p(-2) + p(x)) is the folded distribution function. Assuming the optimal point occurs in the interior of the domain, it should satisfy the zero gradient condition: de/Ov1 0e/0uz = O. Taking derivative from 27 with respect i and 02 and set zero we get:
U1 [
+000
+u2 (f." q(z)dr- f (())dx)
zq(x)dx
zq(x)dr
(28)
/ aq(x)do + fove rq(x)dx
+V1 (L, ((x)do f. q(x)dr)
0/2
We can simplify (28 and rewrite ii and in terms of the following conditional expectations: 1
2 (Ex~p || |a|| > vi] Ex~p[1>1 [x|| S v1l)
(29)
02 (Eapp ||||||
|a|| > wn - E |x|| S vi))
Hence, the optimal values of U1 and D2 can be obtained by solving for U1 and U2 in 29 The optimization domain has two boundaries. One is when di2 = 0. This reduces the problem to 1- bit binary quantization. The optimal solution in that case is discussed in Appendix B The other boundary is when 11 = 02. This results in the ternary quantization. The optimal solution in this case discussed in Appendix D
D OPTIMAL TERNARY QUANTIZATION
In this section, we find the optimal symmetric ternary quantization, that to map a full precision value x discrete set (-2v, 0, 20) as discussed in Section 3.2.3 Finding the optimal mapping can be formulated as the following optimization problem:
minimize 2,81,82
p(x) (us) + v82(x) - x)² dx
(30)
s.t.
81,82
(-1,1}
20
First, the above form is special case of 25 hence, using the argument as in Appendix C we can show that there is a foldable optimal solution: 81 (z = sign(x) and 82(x) = sign(( 08 (x)))
Then the total error function of can be written as:
e(v) q(x)=²dr +
q(=)(x 22)) 20)²
(31)
where q(x) is the folded probability distribution function. Taking derivative from (31 and setting zero, we get:
1 zg(x)da Expplar |x|| > v]
(32)
2 2 q(x)dxx
2
14

---

Under review as a conference paper at ICLR 2020
E DETAILS OF TRAINING RESNET ON IMAGENET
In this section, we explain the details of how the DNN results reported in this paper are produced. All results correspond the ResNet-18 architecture trained on the ImageNet dataset for the classifi- cation task. We use the standard training and validation splits of the ImageNet dataset. We followed a similar architecture as XNor-Net (Rastegari al. 2016). The convolutional block that we use is depicted in Figure [left). We use ReLU non linearity before the batch normalization suggested by (Rastegani al. 2016). Also, we find it important to use bounded dynamic range, and therefore clip the values to d, dj. We use d = 2, 3, 5, and 8 for k 1, 2, 3, and 4 bits quantizations, respectively. Similar to the other BNNs for the first and last layers we use full precision. Also, as suggested by Choi ct al. (2018) use full precision short-cuts in ResNet architecture, which adds a small computational/memory overhead. We quantize weights per filter and activations per layer. first-order polynomial learning-rate rate annealing schedule (from 10 to
As Cai al. (017)) 10 and tram for 120 epochs. We do not use weight decay. For the data augmentation we use the standard methods used train full precision ResNet architecture. For training we apply random resize and crop to x 224, followed by random horizontal flipping, color jittering, and lightening. For test resize the images to 256 256 followed by center cropping 224
15


Reviews:
[{"review_id": "SygD-H4pKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proves seval binary quantization guarantee for the 1 or 2 bit cases, and shows some empirical error analysis. \n\nI am not an expert on neural network compression so I am not quite sure how the proposed method compares with the state-of-the-art algorithms. On the other hand, I checked several proofs provided by the authors for the 1-bit and 2-bit quantization cases. The proofs look good to me.\n\nSome minor comments:\n1. For the definition (9) can the authors make it clear that it is for all v_j s.t. v_1>= v_2>=.... instead of \\exits v_j?\n2. Can authors provide some explanation why in (8) we want to have v1>=v2>=vk in the constraint? I understand we need that in the proof, but is there any reason this is also the case in empirical evaluation? To me we may also have cases such that v1 < v2, is there any guarantee for those cases?\n3. Feels like the draft can be compressed into 8 pages, even if the work looks nice.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklAmT2cKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new family of quantized neural networks, where the weights are quantized based on fixed precisions. The authors proposed the optimal 1-bit/2-bit/ternary quantization schemes based on minimizing the L_2 loss. The authors proposed a greedy algorithm to approximate the optimal k-bit quantization.  The authors showed through extensive experiments on real DNNs that the proposed optimal quantization (Opt in the tables) can give better generalization as compared to several state-of-the-art alternative methods.\n\nThe reviewer appreciates that this quantized DNN-paper has a theory. This theory is based on minimizing the L_2 loss between the original data and quantized data. It unifies several quantization schemes into one framework. This is in contract with technical papers in the area which propose a single quantization method. The proposed quantization can further be implemented efficiently using XNOR operations and bit-counting.\n\nThe writing is of good quality. The length is a bit larger than the recommended length of 8 pages.\n\nThe reviewer has the following concerns,\n\n- This paper motivates from rank-k quantization but implemented as a scaled quantization. At the end of page 3, the authors showed that the scaled quantization can somehow approximate the rank-k quantization. However, the approximation is loose without any guarantee. Overall, I don't understand how the low-rank quantization in section 2 is related to the proposed method and fits in the overall picture. Ideally, in section 2, the authors can have some theoretical statements to state the optimality of the rank-k quantization. Then, the authors can say that, because of practical difficulties to implement the SVD, the use the scaled binary quantization instead. Or, the authors can simply remove section 2, and add a paragraph to introduce rank-k quantization, which is in contract to the scaled quantization they used in the paper.\n\nSomewhere in the text, the authors have to explain the optimality is with respect to the L_2 loss. There can be alternative quantization based on different losses.\n\nFigure 4, could you explain why the angle trends up as the layer index increases?\n\nAfter equation(4): mention the choice of p(x)\n\nIn conclusion, analyz -> analyze, introduc->introduce", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkgXuMtDnB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to quantize the weights of neural networks that can minimize the L_2 loss between the quantized values and the full-precision ones. The authors propose solutions for optimal 1-bit/ternary/2-bit quantization, as well as a greedy algorithm to approximate the optimal k-bit quantization.  Experiments are performed on image classification data set ImageNet using ResNet-18.\n\nFirst of all, the authors should explicitly state in the paper that the optimality is in terms of what?  Indeed the authors obtain the solution of (1) quantization with a scaling parameter, and (2) in terms of minimizing quantization error (L_2 loss between the quantized value and the full-precision one), which is quite restricted to be a universally optimal one.\n\nSince the number of weights and activations are limited in the network, it is not appropriate to formulate quantization error the weights and activations in the format of continuous distribution in (4) and (8). \n\nIt is also not clear to me why this paper begins with rank-1 quantization but ends up with scaled quantization. What kind of assumption are used in this approximation, and can the optimality still be guaranteed?\n\nOne of my main concerns is the novelty of this paper. Many of the solutions in the paper have already been discovered in literature. For instance, the optimal 1-bit solution in (5) was already obtained in Binary-Weight-Network [1] in 2016. The optimal ternary solution (i.e., the ternarization threshold should be 1/2 of the scaling parameter) in (12)  was also already obtained in Corollary 3.1 in  [2] in 2018, as a special case when curvature information is dropped.\n\nYet another concern is about the experiments. Since the proposed optimal binarization has the same solution as BWN, where does the performance gain in Table 2 come from? Moreover, some of the recent quantization methods are not compared. For instance, in PACT [3], 2-bit weight&acitvation quantization already achieves 64.4% top-1 accuracy of Resnet18 on ImageNet, while the proposed method achieves the same accuracy with full-precision activation and 2-bit weight (Table 2). In addition, according to Table 2, the proposed method also can not beat LQ-Net. \n\n\n[1] Rastegari, Mohammad, et al. \"Xnor-net: Imagenet classification using binary convolutional neural networks.\" ECCV, 2016.\n[2] Hou, Lu, and James T. Kwok. \"Loss-aware weight quantization of deep networks.\" ICLR 2018.\n[3] Choi, Jungwook, et al. \"Pact: Parameterized clipping activation for quantized neural networks.\" arXiv preprint arXiv:1805.06085 (2018).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxkiR7o_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "this paper looks at different quantization schemes (replace values of vector in \\R to, basically, plus or minus v, for well chosen v).\n\nDifferent schemes are considered, namely the low rank binary quantization (a matrix is approximated by the componentwise product of a low rank matrix and a +/-1 matrix) and k-bit binary  quantization.\nFinding the best quantization ends up in solving a program, which is convex and quite straightforward fo k=1 and 2, and unfortunately (apparently) non-convex for k>2. So the authors suggest a greedy approach for k>2.\n\nThe motivation of this work is DNN, arguing that quantized vectors should improve computations cost, hence some experiments are provide on DNN, yet they only illustrate the fact that quantization does not deteriorate too much the learning & test error.\n\nThe main criticisms is therefore the lack of concrete evidences that those schemes are actually helpful and, on the other hand, the relative simplicity (so the theoretical part of the paper are not sufficient by itself)", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to quantize the weights of neural networks that can minimize the L_2 loss between the quantized values and the full-precision ones. The paper has limited novelty, as many of the solutions presented in the paper have already been discovered in the literature. During the discussion, the reviewers agree that it is an incremental contribution. Parts of the paper can also be clarified, particularly on the optimality of the solution, assumptions used in the approximation, and some of the experimental results. Experimental results can also be made more convincing by adding comparision with the more recent quantization methods.
Decision: Reject