Smart Ternary Quantization
Authors: Gregoire Morin
Ryan Razani
Vahid Partovi Nia
Eyyub Sari
Venue:  (2020)

Abstract: Neural network models are resource hungry. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary quantization provides a more flexible model and often beats binary quantization in terms of accuracy, but doubles memory and increases computation cost. Mixed quantization depth models, on another hand, allows a trade-off between accuracy and memory footprint. In such models, quantization depth is often chosen manually (which is a tiring task), or is tuned using a separate optimization routine (which requires training a quantized network multiple times). Here, we propose Smart Ternary Quantization (STQ) in which we modify the quantization depth directly through an adaptive regularization function, so that we train a model only once. This method jumps between binary and ternary quantization while training. We show its application  on image classification.

Under review as a conference paper at ICLR 2020
SMART TERNARY QUANTIZATION
Anonymous authors
Paper under double-bling blind review
ABSTRACT
Neural network models are resource hungry. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource require- ments. Ternary quantization provides a more flexible model and often beats binary quantization in terms of accuracy, but doubles memory and increases computation cost. Mixed quantization depth models, on another hand, allows a trade-off be- tween accuracy and memory footprint. In such models, quantization depth is often chosen manually (which is a tiring task), or is tuned using a separate optimization routine (which requires training a quantized network multiple times). Here, we propose Smart Ternary Quantization (STQ) in which we modify the quantiza- tion depth directly through an adaptive regularization function, that we train a model only once. This method jumps between binary and ternary quantization while training. We show its application on image classification.
INTRODUCTION
Deep Neural Networks (DNN) models have achieved tremendous attraction because of their success on a wide variety of tasks including computer vision, automatic speech recognition, natural language processing, and reinforcement learning (Goodfellow et al., 2016). More specifically, in computer vision DNN have led to a series of breakthrough for image classification (Krizhevsky et al., 2017), (Simonyan & Zisserman, 2014), (Szegedy et al., 2015), and object detection (Redmon et al., 2015), (Liu et al., 2015), (Ren et al., 2015). DNN models are computationally intensive and require large memory to store the model parameters. Computation and storage resource requirement becomes an impediment to deploy such models in many edge devices due to lack of memory, computation power, battery, etc. This motivated the researchers to develop compression techniques to reduce the cost for such models.
Recently, several techniques have been introduced in the literature to solve the storage and compu- tational limitations of edge devices. Among them, quantization methods focus on representing the weights of neural network in lower precision than the usual 32-bits float representation, saving the memory footprint of the model. Binary quantization (Courbariaux et al., 2015), (Hubara et al., 2016), (Rastegari et al., 2016), (Zhou et al., 2016), (Lin et al., 2017) represent weights with 1 bit precision and ternary quantization (Lin et al., 2015). (Li & Liu, 2016). (Zhu et al., 2016) with 2 bits precision. While the latter frameworks lead to significant memory reduction compared to their full precision counterpart, they are constrained to quantize the model with 1 bit or 2 bits, on demand. We relax this constraint, and present Smart Ternary Quantization (STQ) that allows mixing 1 bit and 2 bits layers while training the network. Consequently, this approach automatically quantizes weights into binary or ternary depending upon a trainable control parameter. We show that this ap- proach leads to mixed bit precision models that beats ternary networks both in terms of accuracy and memory consumption. Here we only focus on quantizing layers because it is casier to implement layer-wise quantization at inference time after training. However, this method can be adapted for mixed precision training of sub-network, block, filter, or weight easily. To the best of our knowledge this is the first attempt to design single training algorithm for low-bit mixed precision training.
RELATED WORK
There are two main components in DNN's, namely, weight and activation. These two components usually computed in full precision, i.e. floating point 32-bits. This work focuses quantizing

---

Under review as a conference paper at ICLR 2020
the weights of the network, i.e. generalizing BinaryConnect (BC) of Courbariaux et al. (2015) and Ternryy Weight Network (TWN) of Li & Liu (2016) towards automatic or bits mixed- -rexisspe using a single training algorithm.
In BC the real value weights w are binarized to w E {-1, +1} during the forward pass. To map a full precision weight to a binary weight, the deterministic sign function is used,
wh sign(w) [+1
W
The derivative of the sign function is zero on \ (0). During back propagation, this cancels out the gradient of the loss with respect to the weights after the sign function. Therefore, those weights cannot get updated. To bypass this problem, Courbariaux al. (2015) use a clipped straight-through estimator
ac ac
dw Owb (w)
(2)
where is the loss function and (.) is the indicator function on the set A. In other words (2) approximates the sign function by the linear function f(x) = x within [-1, and by a constant elsewhere. During back propagation, the weights are updated only within [11++1] The binarized weights are updated with their corresponding full precision gradients. Rastegari et al. (2016) add scaling factor to reduce the gap between binary and full-precision model's accuracy, defining Binary Weight Network (BWN), The real value weights W in each layer are quantized as {-1, +1} where E [[]] E R. Zhou et al. (2016), generalize the latter work and approximates the full precision weights with more than one bit while Lin et al. (2017) approximate weights with linear combination of multiple binary weight bases.
2.1 TERNARY WEIGHT NETWORKS
Ternary Weight Network (TWN) (Li & Liu, 2016) is neural network with weights constrained to {-1,0, +1). The weight resolution is reduced from 32 bits to bits, replacing full precision weights with ternary weights. TWN aims to fill the gap between full precision and binary precision weight. Compared to binary weight networks, ternary weight networks have stronger expressive capability. As pointed out (Li & Liu, 2016), for a 3x3 weight filter in a convolutional neural network, there 512 possible variation with binary precision and 3ax3 19683 with ternary precision.
is 2
Li Liu (2016) find the closest ternary weights matrix W to its corresponding real value weight matrix W using
à. W arg min |W 1 1
a. Wt
(3)
s.t. μ p> 0, w/y (- o, 1}, i,j 1.2,..., n. The ternary weight W is achieved by applying symmetric threshold A
Wij > A, |uy - A, 0
+1
W
Wiy < <-
Li Liu (2016) define weight dependant threshold A and a scaling factor μ that approximately solves (3). TWN is trained using stochastic gradient descent. Similar to BC and BWN schemes; ternary-value weights are only used for the forward pass and back propagation, but not for the parameter updates. At inference, the scaling factor can be folded with the input X
W ② (+ (μX) W
(5)
where denotes the convolution.
Zhu et al. (2016) proposed a more general ternary method which reduces the precision of weights in neural network to ternary values. However, they quantize the weights to asymmetric values {-p1, 0, } using two full-precision scaling coefficients μ1 and μ2 for each layer of neural net- work. While the method achieve better accuracy opposed to TWN, its hardware implementation becomes a challenge, because there are two unequal full precision scaling factors to deal with.
Our method provides compromise between BC and TWN and trains weights with single trainable scaling factor μ. Weights jumps between ternary {-p, p4, 0, +4} and binary {-4, +u}. This provides single algorithm for 1 or 2 bits mixed precision.
2

---

Under review as a conference paper at ICLR 2020
2.2 REGULARIZATION
Regularization term is the key to prevent over-fitting problem and to obtain robust generalization for unseen data. Standard regularization functions, such as Le or L1 encourage weights to be concen- trated about the origin. However, in case of binary network with binary valued weights, it is more appropriate to have a regularization function to encourage the weights about p x {-1, +1}, with scaling factor 1s 0 such
Ri(w, p)) = =||w| 14
(6)
proposed in Belbahri et al. (2019).
straightforward generalization for ternary quantization
R2(w, = ||0|
11 2
Regularizer (6) encourages weights about {-u, + and (7) about {-4, 0, +u]. The two functions depicted in Figure These regularization functions are only useful when the quantization depth is set before training start. We propose more flexible version to smoothly move between these two functions using shape parameter.
1.
1.0
0.4
0.4
02
(a) Rí (wi, p 1)
(b) Ra Haww, 1)
Figure 1: Binary and ternary regularizers; Rí encourages binary weights, with minimums at
{-1, T-1,+pu +1}, and R₂ encourages ternary weights, with minimums at {-1, 0, +1}.
3

---

Figure 2: Adaptive regularization function. When 3 + the regularization function switches from Under review as a conference paper at ICLR 2020
3 SMART TERNARY QUANTIZATION
Here we propose an adaptive regularization function that switches between binary regularization of (6) and ternary regularization of (7)
R(w p. B) min
|||| pl, tan(3)
in which z a trainable scaling factor, and (#: 3) controls the transition between (6) and (7). As a special case B - 3 converges to the binary regularizer (6) and 3 - coincide with the ternary regularizer (7), depicted in Figure 2. A large value of tan(3) repels estimated weights away from zero thus yielding binary quantization, and small tan( values encourage zero weights. The shape parameter controls the quantization depth. Quantization is done per layer, therefore we let | very per layer. We recommend to regularize about i.e. preferring binary quantization apriori
R(w, p. B) min (Iwe) - s4/, tan(())ull + + cot(3)).
in which y controls the proportion of binary to ternary layers.
For single filter W the regularization function is sum over its elements
ΣΣ min (1res) - nl, tan(8) + + + cot(
R(W, 34,
(10)
Large values y encourage binary layers. In each layer, weights are pushed to binary or ternary values, depending on the trained value of the corresponding 3. A generalization of (9) towards Ip norms of Belbahri et (2019) is also possible. However, here we only focus on regularizer constructed using the In norm as the accuracy did not change significantly by using T.p norm with different values p.
The introduced regularization function is added to the empirical loss function L(). The objective function defined on weights W, scaling factors μ,, and quantization depths 3 is optimized using back propagation
Kr
P(W,,,3)= +
Σ
R(W kis (t),
(11)
where & indexes the channel, and I indexes the layer. One may use a different regularization constant X for each layer to keep the impact of the regularization term balanced across layers, indeed different layers may involve different number of parameters, We set \ #W; where À constant, and # W, 1 is the number weights layer L.
10:
36.
44-
42
66
(a) R(w, /
1,3 4
(b) R(( (1, P 1,9 (c) R(w, 14 1, 1.5707)
ternary binary.
We propose to the same threshold based function of Li & Liu (2016) (3), but with fixed threshold A per layer Note that Li Liu (2016) propose a weight-dependant threshold. We let the possibility for the weights only accumulate about {-p, +44 and not about depending 3.
4

---

Under review as a conference paper at ICLR 2020
One may set A to have the same balanced weights in {-p4, 0, +/] at initialization for all layers and let the weights evolve during training. Formally, if or is the standard deviation of the initial Gaussian weights in layer I, propose A = 0.2 d). The probability that single weight lies in the range [- Ar] is z 0.16. All the weights falling in this range will be quantized as zeros after applying the threshold function.
Weights are naturally pushed to binary or ternary values depending on Bo during training. Eventually, threshold ō close _ ~ 1.57 defines the final quantization depth for each layer.
Binary
31 2 8
Final quantization depth of layer I:
Ternary
31
4 EXPERIMENT
We run experiments on two simple image classification tasks MNIST (LeCun et al., 1998) and CIFAR1O (Krizhevsky & Hinton, 2009) datasets. We compare our method (STQ) with BinaryCon- nect (BC) of Courbariaux et al. (2015), Binary Weight Networks (BWN) of Rastegari et al. (2016), Ternary Weights Network (TWN) of Li & Liu (2016) and also with a Full Precision network (FP). The quality of the compression is measured only in terms of memory, it is difficult to compare mixed precision models, with binary and ternary, in terms of consumed energy. Assume that n; the quan- tization depth for the layer and # W, is the number of weights in layer therefore the compression >> #3,x32
The compression ratio for a binary network is 32, for a ternary network 16, and our approach falls in between.
ratio is
21-1 # W
STQ network generalizes binary and ternary regularization in single regularization function. First we show how to control the proportion of binary and ternary layers using ? in (8). Figure clarifies the effect of r on the weight distribution. When y is large, /3 is encouraged towards ž which corre- sponds to binary quantization. Consequently, weights are pushed about {-,t, +/ and removed from the trained values, see Figure 3a. On the contrary, when y is small, 3 tends to z and the weights started including in their values, see Figure 3b.
(a) 1x 10
1. 10-1
1.57
(b) 1 1x 10 B 0.79
Figure 3: Effect of y on the weights distribution of a layer while training.
4.1 MNIST
MNIST is an image classification benchmark dataset with 28 x 28 gray-scales images representing digits ranging from 0 to 9. The dataset is split into 60k training images and 10k testing images. We used the LeNet-5 (LeCun et al., 1998) architecture consisting of 5 layers, 2 convolution followed by maxpooling, stacked with two fully connected layers and softmax layer at the end. We train the network for 60 epochs using Adam optimizer. We used the initial learning rate of 0.01, but divided by 10 in epoch 15 and 30 to stabilize training. The batch size is set to 64 with L2 weight decay constant 10 only for BC, BWC and TWN. The full precision L is trained with no regularization it provided superior accuracy. STQ is trained with À 0.1 and y 1 x 10 and the effective regularization constant is divided by the number of weights in each layer to compensate for the layer size. The best validation accuracy for each method is reported in Table 1, as well as
5

---

Under review as a conference paper at ICLR 2020
the quantization depth, and the overall compression ratio. We observe that STQ network quantized the first two convolutional layers bit, and the last fully-commectdd layers 2 bits. The accuracy improvement and the compression ratio is marginal for simple task and simple architectures. The effect of smart training becomes more visible for more complex tasks with more layers.
Table I: Smart ternary quantization (STQ) compared with Binary Connect (BC), Binary Weight Network (BWN), Temeary Weight Network (TWN), and Full Precision (FP) MNIST dataset.
Quantization depth per layer (-bits)
Compression ratio
Accuracy (top-1)
BC
1-1-1-1-1
32
99.35
BWN
1-1-1-1-1
32
99.32
TWN
2-2-2-2-2
16
99.38
STQ
1-1-2-2-2
16.3
99.37
FP
1
99.44
4.2 CIFAR10
CIFAR 10 is an image classification benchmark that contains 32 32 RGB images from ten classes. The dataset is split into 50k training images and 10k testing images. All images are normalized using mean (0.4914, 0.4822, 0.4465) and std (0.247, 0.243, 0.261). For the training session, we pad the sides of the images with 4 pixels, then a 32 x 32 crop is sampled, and flipped horizontally at random.
We use two VGG-like architectures, i) VGG-7 architecture defined in Li & Liu (2016) in which we apply batch normalization after each layer and use ReLU activations, ii) a standard VGG-16 architecture. Deep architectures are sensitive to early layer quantization. As commonly practiced, we did not quantize the first and the last layers in VGG-16 for all competing methods.
We train the network for 150 epochs, using Adam optimizer with the initial learning rate 0.001 divided by 10 at epochs 40 and 80. The batch size is set to 64 with L2 weight decay constant 10 10 for STQ. The best validation accuracy for each method is reported in Table 2. STQ beats pure 2 bits network TWN, even in terms of accuracy. recommends three 1 bit layers for VGG-7 and seven bit layers for VGG-16. The compression ratio is significantly higher than a ternary network. The weight distribution of each layers are depicted in Figure for the VGG-7 architecture. Weights are pushed {-,, + parameter B.
moreover À 0.1,
} or {-4, 0, +p} depending on the shape
Table 2: Smart ternary quantization (STQ) compared with Binary Connect (BC), Binary Weight Network (BWN), Temeary Weight Network (TWN), and Full Precision (FP) CIFAR10 dataset.
Quantization depth per layer (-bits)
Compression ratio
Accuracy (top-1)
Architecture
Method
BC
1-1-1-1-1-1-1
32
92.49
BWN
1-1-1-1-I-I-I
32
92.42
VGG-7
TWN
2-2-2-2-2---2
16
92.74
STQ
2-1-1-1-2-2-2
18.3
92.94
FP
93.72
BC
31.5
91.92
BWN
$2.1:1111111111111111111.
31.5
91.85
VGG-16
TWN
32.22212222222222222333
15.9
92.14
STQ
12221-1222-2111-1112233
25.1
92.38
FP
92.53

---

Under review as a conference paper at ICLR 2020
(a) Layer
(b) Layer 2, 32
(c) Layer 3, Ba
(d) Layer 4,
1.56
1.57
1.57
1.57
(e) Layer 5, 933g 1.00
(1) Layer 33 = 0.86
(g) Layer 7, 7, 0.98
Figure 4: Layer wise weights distribution in VGG-7 for STQ The weights are pushed to binary when the shape parameter 3 is close to = 1.57.
5 CONCLUSION
Smart ternary quantization (STQ) is training method to build 1 and 2 bits mixed quantized layers. Depth optimization requires training network multiple times which is costly, specially the network is complex. This approach successfully combines quantization with different depths, while training the network only once. We tried layer-wise quantization, because it is more suitable for mixed-precision inference implementation. However, subnetwork, block, filter, or weight mixed quantization is feasible using similar algorithm.
STQ makes manual tuning of quantization depth unnecessary. It allows to improve the memory consumption, by automatically quantizing some layers with smaller precision. This method some- times even outperforms pure ternary networks in terms of accuracy thank to formal regularization function that shapes trained weights towards mixee-precision
is well-known that some layers are more resilient to aggressive quantization, Our introduced methodology trains network similar to pure ternary but give an insight about which layers can be simplified further by going to binary quantization depth.
REFERENCES
Mouloud Belbahri, Eyyüb Sari, Sajad Darabi, and Vahid Partovi Nia. Foothill: A quasiconvex regu- larization for edge computing of deep neural networks. In Image Analysis and Recognition 16th International Conference (ICIAR), pp. 341 2019.
Matthieu Courbariaux, Yoshua Bengio, and Yean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems (NIPS), pp. 3123-3131 2015.
Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Learning. Adaptive computation and machine learning. MIT Press, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-ian and Yoshua Bengio. Binarized neural networks. In Advances neural information processing systems (NIPS), pp. 4107-4115, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep convo- lutional neural networks. Commun. ACM, 60(6):84 90, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11) (6(11):277823244 Nov 1998.
7

---

Under review as a conference paper at ICLR 2020
Fengfu Li and Bin Liu. Temary weight networks. CoRR, abs/1605.04711, 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in neural information processing systems (NIPS), pp. 344-352, 2017.
Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. CoRR, abs/1510.03009, 2015.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. CoRR, abs/1512.00322 2015.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. CoRR, abs/1603.05279, 2016.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. CoRR, abs/1506.026440, 2015.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, Abs/1500.00197, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 0bt//109.11.. 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.0616/ 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. CoRR, abs/1612.01064, 2016.
8


Reviews:
[{"review_id": "rJgg6GphYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach where the regularisation is used to optimise whether each layer of a DNN is binary or ternary. The paper presents an equation for performing this along with two examples of the process in use.\n\nThe paper is inconsistently written with work described at different levels in different sections and has an inconsistent feeling about it. For example the introduction seems to stop abruptly before it describes all the parts of the paper.\n\nThe paper seems to contain an idea which might have merit. However, the idea just does not seem to have been developed enough.\n\nMajor concerns:\n1) The authors claim that this is the first attempt at a training algorithm for mixed precision training. However, a simple google search throws up many papers in this area. Many of which are not mentioned in the related work.\n\n2) Equations are not discussed in enough detail, nor are the parameters defined. Or if they are defined they are done so much later in the work.\n\n3) There doesn’t seem to be enough material here to reproduce the work.\n\n4) In the results you talk about ‘the best’. Given that there has been much criticism over the last two years about good academic practice the fact that you don’t say at least ‘best out of …’ is worrying. \n\n5) You have magic parameters lambda and gamma. You say that these effect the outcome of the work but in your examples you only state values these are set to. One would expect to see at least some analysis of how varying these values effect the outcome. But better would be to show that you have identified good values for both of these parameters. Ideally would be an evaluation of how others could identify the best values.\n\n6) Figures 3 and 4 are difficult to interpret. They need a clear explanation.\n\nSome more generic comments:\n- The abstract seems to assume a huge prior knowledge by the reader.\n\n- ‘1 bit precision’ - precision seems to have no meaning in this context. Surely just ‘1 bit’\n\n- The related work contains a lot of equations, but no real explanation of what they are.\n\n- ‘we let β very per layer’ -> ‘we let β vary per layer’\n\n- In equation 10 what do I and J represent?\n\n- ’28 × 28 gray-scales images’ -> ’28 × 28 gray-scale images’\n\n- ‘For the training session, we pad the sides of the images with 4 pixels, then a 32 × 32 crop is sampled, and flipped horizontally at random.’ - why?\n\n- ‘As commonly practiced’ - by whom?\n\n- ‘which is costly, specially if’ -> ‘which is costly, especially if’\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eEnyfrYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses a generalization to low bit quantization and combines the approaches of binary and ternary quantization methods. Past methods such as Binary Connect and Binary Weights Network have shown that you can train a network efficiently with 1-bit quantization, and methods such as Ternary Weights Network demonstrate 2-bit quantization with weights taking one of {-1, 0, 1} * mu, with mu being a scale computed per weight tensor. The authors generalize these two methods so that the choice of binary vs ternary weights can be made per layer automatically during training. The primary contribution to make that work is by incorporating a generic regularizer with addition hyper-parameters to trade-off between the binary weight regularizer and ternary weight regularizer. In addition to that, the regularization also includes a prior to make the layers prefer binary weights by default. This is done by adding a cost that penalizes the choice of ternary weights for each layer.\n\nOverall, the paper is well written and explained, with supporting experiments to show on MNIST and on CIFAR10 that this method performs quite competitively compared to an all-binary or all-ternary weights network. Low-bit quantization is an important research area and this paper makes a strong contribution by studying mixed-precision low-bit quantization. The mathematical explanations of the generalized regularizer are well justified. For instance, the regularization constant lambda is normalized for each layer by the total number of weights to evenly weight all layers.\n\nAlthough the experiments cover MNIST and CIFAR10, it's not clear how mixed-precision low-bit methods perform on models more prevalent in the real-world. Supporting the experiments with ResNet variants on ImageNet would help clarify that further.  The paper very well explains the fundamentals of 1-bit and 2-bit methods, and the contribution of this paper (generalization of these two methods) is a somewhat natural extension without significant novelty. Moreover, in addition to the accuracy, it would also be better to understand how the run-time performance of such models (on existing software and hardware implementations) compares to pure binary and ternary networks, as knowledge of that would reveal insights into systems optimizations to be made in future work in this field. Given all of this, my rating is a weak accept.\n\nPros:\n- The problem and the fundamentals (prior work) are very well laid out.\n- The regularization component that generalizes the two types of quantization is sound.\n- Experiments on CIFAR10 strongly show improved accuracy and higher compression ratio compared to ternary weights network.\n\nCons:\n- Lacking more realistic experiments on larger datasets such as ImageNet and models like ResNets.\n- The runtime performance of how STQ compares to 1-bit and 2-bit quantization variants isn't shown.\n- The regularizer introduces more hyper-parameters to tweak and it's not clear how sensitive these are to the choice of the architecture. Different values of gamma are used in the two experiments, and further analysis on how the performance varies for various values of lambda and gamma would shed further insight.\n\nMinor comments:\n- In equation (3), the term under argmin should be mu and not alpha.\n- Section 3, line above equation (9) reads \"very per layer\" instead of \"vary per layer\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S6PlF27iLaS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies mixed-precision quantization in deep networks where each layer can be either binarized or ternarized. The authors propose  an adaptive regularization function that can be pushed to either 2-bit or 3-bit through different parameterization, in order to automatically determine the precision of each layer. Experiments are performed on small-scale image classification data sets MNIST and CIFAR-10.\n\nThe proposed regularization method is simple and straightforward. However, many details are not stated clearly enough for reproduction. E.g, since the proposed regularization already promotes binary or ternary weights, whey is there still a thresholding operation at the end of Section 3? Is it because the proposed regularization can not provide strict binary or ternary weights? Does the method require one more hard binarization/ternarization step after \\beta is learned. Indeed, tan(x) is not well-defined when x=pi/2, and the derivative tan'(x)= 1+tan^2(x) can be large when x is near pi/2, and does gradient descent work well in this case?\n\nThe experiments are only performed on small-scale data sets. Thus it is hard to tell if the proposed method also works for larger networks or data sets? Moreover, it is not fair to use \"best validation accuracy\" for comparison with other methods since the validation set is seen during training and it is not clear if the hyper-parameters of the proposed methods are tuned for best performance on the seen validation set. It would be more fair to compare the test accuracy like in the BinaryConnect (BC) paper. Yet another concern is that many recent methods that can train mixed-precision networks are not compared. For instance, the HAQ method [1] searches for precision for each layer using the reinforcement learning method, how does the proposed method perform when compared with it?\n\n[1]. Wang, Kuan, et al. \"HAQ: Hardware-Aware Automated Quantization with Mixed Precision.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeJHadbcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The Paper talks about the Smart Ternary Quantization method that improves the quantization over binary and ternary quantizations by specifying an adaptive quantization. The proposed regularization function is covered in detail and the results are evaluated on MNIST and CIFAR10 datasets\n\nThe authors can improve the submission by \n1. evaluating more modern networks with bigger datasets, as opposed to the ones demonstrated.\n2. describing gamma (eq 10) , it wasn't clear why that parameter was introduced (in addition to the beta) and it's significance, what was more confusing is coverage for this instead of beta in the experimental setup\n3. why the LR needed to be modified for the described method", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies mixed-precision quantization in deep networks where each layer can be either binarized or ternarized. The proposed regularization method is simple and straightforward. However, many details and equations are not stated clearly. Experiments are performed on small-scale image classification data sets. It will also be more convincing to try larger networks or data sets. More importantly, many recent methods that can train mixed-precision networks are not cited nor compared. Figures 3 and 4 are difficult to interpret, and sensitivity on the new hyper-parameters should be studied. The use of "best validation accuracy" as performance metric may not be fair. Finally, writing can be improved. Overall, the proposed idea might have merit, but does not seem to have been developed enough.
Decision: Reject